Image processing method, device, electronic device and storage medium

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, and relates to the field of computer technology. The method comprises: extracting features of N characters contained in a structured image, obtaining multimodal features corresponding to each of the N characters, and obtaining initial reference information based on the obtained N multimodal features; then inputting the initial reference information and the N multimodal features into a document multimodal model, and performing the following operations in a loop iteration manner until the structured data before the output termination character is obtained: reading a multimodal feature and obtaining a historical structured data; finally, based on the current reference information, a multimodal feature and a historical structured data, the structured data of the character corresponding to the multimodal feature and the updated reference information can be obtained. In this way, since the document multimodal model supports the structuring of multiple documents, the accuracy of the character structuring results is improved.

Description

Background Art

[0002] With the rapid development of computer technology, objects can detect and recognize text in various scenarios based on optical character recognition (OCR) technology. For example, objects can structure unstructured images into semi-structured data such as text based on optical character recognition technology.

[0003] In the related technology, when performing structured processing on images based on optical character recognition technology, the following solutions are generally adopted:

[0004] Solution 1: The electronic device receives the image to be structured, determines the template image corresponding to the image to be structured, and maps the image to be structured to the template image, thereby extracting the structured result of the corresponding field in the image to be structured according to the anchor point features in the template image to obtain the recognition result. The aforementioned anchor point features are, for example, fixed text, field distribution, etc.

[0005] However, when the image to be structured is rotated or deformed, that is, when the layout of the image to be structured is changed, the matching template image may not be found using solution 1, resulting in an inability to obtain accurate recognition results.

[0006] Solution 2: The electronic device receives the image to be structured, calls a dedicated text field detector to detect the position of the required structured field, and then calls a text recognizer to recognize the field at the corresponding position to obtain a character recognition result.

[0007] However, when the layout of the image to be structured is complex and has many field types, solution 2 may not be able to accurately locate the position of the required structured field, resulting in the inability to obtain a highly accurate character structuring result.

[0008] In summary, under the related art, when electronic devices perform structured processing on images, there is a problem that they cannot process images with variable layouts and various field types, which leads to poor accuracy of character structuring results. Summary of the invention

[0009] An image processing method, device, electronic device and storage medium are provided in the embodiments of the present application for providing a character structuring solution without restrictions on layout type and field type, so as to accurately obtain structured data of characters in an image.

[0010] In a first aspect, an image processing method is provided, comprising:

[0011] Perform feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters; wherein each modality represents a character attribute, and N is a positive integer;

[0012] Based on the obtained N multimodal features, initial reference information is obtained, wherein the reference information includes the attention weight of each layer of the network in the document multimodal model and the comprehensive multimodal features corresponding to the image to be structured;

[0013] The initial reference information and the N multimodal features are input into the document multimodal model, and the following operations are performed in a loop iteration manner until the structured data before the output termination character is obtained:

[0014] Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty;

[0015] Based on the current reference information, the one multimodal feature and the one historical structured data, the structured data of the character corresponding to the one multimodal feature and the updated reference information are obtained.

[0016] In a second aspect, an image processing device is provided, comprising:

[0017] An extraction unit, configured to perform feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters; wherein each modality represents a character attribute, and N is a positive integer;

[0018] A determination unit, configured to obtain initial reference information based on the obtained N multimodal features, wherein the reference information includes an attention weight of each layer of the network in the document multimodal model and a comprehensive multimodal feature corresponding to the image to be structured;

[0019] The processing unit is used to input the initial reference information and the N multimodal features into the document multimodal model, and perform the following operations in a loop iteration manner until the structured data before the output termination character is obtained:

[0020] Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty;

[0021] Based on the current reference information, the one multimodal feature and the one historical structured data, the structured data of the character corresponding to the one multimodal feature and the updated reference information are obtained.

[0022] Optionally, the device further comprises a training unit, configured to:

[0023] Determining an overall loss function; the overall loss function includes a loss function for a sequence-to-sequence task;

[0024] After training the preset document multimodal model, performing a convergence test on the trained preset document multimodal model using the overall loss function;

[0025] When it is determined that the preset document multimodal model after training has converged, the document multimodal model is obtained.

[0026] In a third aspect, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method described in any one of the first aspects is implemented.

[0027] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method described in any one of the first aspects is implemented.

[0028] In a fifth aspect, a computer program product is proposed, comprising a computer program, wherein when the computer program is executed by a processor, the image processing method described in any one of the first aspects is implemented.

[0029] The beneficial effects of this application are as follows:

[0030] In an embodiment of the present application, an image processing method, device, electronic device and storage medium are proposed, and feature extraction is performed on N characters contained in a structured image to obtain multimodal features corresponding to each of the N characters, and initial reference information is obtained based on the obtained N multimodal features; then the initial reference information and the N multimodal features are input into a document multimodal model, and the following operations are performed in a loop iteration manner until the structured data before the output termination character is obtained: a multimodal feature is read, and a historical structured data is obtained; finally, the structured data of a character corresponding to a multimodal feature and the updated reference information can be obtained based on the current reference information, a multimodal feature and a historical structured data. In this way, the present application can support the structuring of multiple documents based on the document multimodal model, thereby improving the accuracy of the character structuring results.

[0031] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the overall structure of a document multimodal model in an embodiment of the present application;

[0035] Figure 3 Schematic diagram of the structure of a stacked layered cross-modal encoder in an embodiment of the present application;

[0036] Figure 4 Schematic diagram of the structure of a modality-aware mask module MAMM in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a process of training a document multimodal model in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an OCR error correction in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a flow chart of an image processing method in an embodiment of the present application;

[0040] Figure 8 is a schematic diagram of an implementation of an image processing method in an embodiment of the present application;

[0041] Fig. 9 is a schematic diagram of an image processing result in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of the structure of an image processing device in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0045] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0046] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0047] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0049] NER-LM: (NER model with language model pre-training), NER model with language model pre-training. Named entity recognition (NER) (also known as entity recognition, entity chunking, and entity extraction) is a subtask of information extraction that aims to locate and classify named entities in text into pre-defined categories, such as people, organizations, locations, time expressions, quantities, monetary values, percentages, etc.

[0050] MD-Bert: (Modal Deupling Bert), that is, stacked layered cross-modal encoder, in which BERT (Bidirectional Encoder Representations from Transformers), that is, the Encoder of the bidirectional Transformer, describes character-level, word-level, sentence-level and even inter-sentence relationship features.

[0051] Unidirectional LM: left-to-right LM or right-to-left LM, where the left-to-right LM can be understood as using the left word of the mask word to predict the masked word; the right-to-left LM can be understood as using the right word of the mask word to predict the masked word.

[0052] Bidirectional LM: Both the left and right words of the current mask can be seen.

[0053] Sequence-to-sequence LM: The input is the source sentence and the target sentence, and the mask word is on the target. Then the context of the current mask is all the words in the source sentence and the words on the left of the mask word in the target sentence that can be seen.

[0054] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0055] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0056] The solution provided in the embodiment of this application involves technologies such as document structuring in computer vision of artificial intelligence, and the document multimodal model implemented in this application is constructed and trained based on machine learning technology. The embodiment of the present invention can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0057] The following is a brief introduction to the design concept of the embodiment of the present application:

[0058] As mentioned above, the OCR structured methods provided by the relevant technology have the following problems: the template matching method based on image features has high requirements on image quality and text recognition results, and it is difficult to deal with problems such as rotation, perspective, and distortion; and the customized detection method based on text features can reduce the dependence on image quality to a certain extent, but the detection effect on complex layouts is poor.

[0059] Moreover, none of the above methods can cope with OCR structured scenarios with complex or no layouts, and none of them can handle OCR recognition errors, so their scope of application is limited.

[0060] In view of this, the present application provides an image processing method, device, electronic device and storage medium. Based on this solution, a generative OCR structuring method can be provided, that is, for the structured image, a document multimodal model determined by a multimodal sequence-to-sequence generation method based on large-scale corpus pre-training is processed, so that structured data corresponding to the image to be structured can be obtained, thereby solving the problems that the OCR structuring method under the related technology cannot handle variable layouts, diverse field types and OCR error processing, and improving the accuracy of the character structuring results.

[0061] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0062] See also Figure 1 , which is a schematic diagram of a possible application scenario in an embodiment of the present application, includes a terminal device 110 (including a terminal device 1101 of object 1, a terminal device 1102 of object 2, ..., a terminal device 110n of object n), and an electronic device 120.

[0063] In the embodiment of the present application, the terminal device 110 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, smart TVs, vehicle-mounted devices, personal digital assistants (PDAs), etc.

[0064] The electronic device 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It may also be an electronic device such as a desktop computer, a mobile phone, a mobile computer, a tablet computer, etc.

[0065] In the embodiment of the present application, a communication connection is established between the terminal device 110 and the electronic device 120 through a communication network using a wired connection or a wireless connection.

[0066] In a possible technical solution of the present application, the object can determine the image to be structured on the terminal device 1101, and then send the image to be structured to the electronic device 120 through the terminal device 1101; or, the electronic device 120 can send a request to obtain the image to be structured to the terminal device 110, thereby receiving the image to be structured sent by the terminal device 110. That is to say, in the embodiment of the present application, either the terminal device or the electronic device can determine the image to be structured, and the present application does not limit this.

[0067] The technical solution proposed in this application can be applied to various non-fixed-layout or fixed-layout OCR structuring scenarios, such as Youtu intelligent structured OCR, receipt recognition OCR and other businesses.

[0068] Specifically, based on the determination rule corresponding to the object or electronic device, the image to be structured is determined, wherein the image to be structured may be one or more images according to actual processing needs, and the following description will take one image to be structured as an example.

[0069] Scenario 1: An image to be structured is determined based on an image selected or input by an object on the terminal device 110 .

[0070] Specifically, the object can select or input an image on the terminal device 110 and trigger a request for structuring the image, so that the terminal device 110 can send the image to the electronic device 120, and then the electronic device 120 can determine the image to be structured.

[0071] For example, after the subject selects an image including receipt information on the terminal device 1102 , the terminal device 1102 may send the image as an image to be structured to the electronic device 120 , and then the electronic device 120 may determine the image to be structured.

[0072] Scenario 2: The electronic device 120 determines the image to be structured based on the corresponding determination rule.

[0073] Specifically, the electronic device 120 may determine the images to be structured in sequence based on the corresponding determination rule, for example, according to the order in which the images are input to the electronic device 120. For example, the electronic device 120 stores 30 images, and then determines the image with order 1 as the image to be structured according to the storage order of the 30 images. Of course, other determination rules may also be used, which are not limited in the embodiments of the present application.

[0074] In the embodiment of the present application, after the electronic device 120 determines the image to be structured, the electronic device 120 performs feature extraction on the N characters contained in the structured image to obtain the multimodal features corresponding to each of the N characters, and obtains the initial reference information based on the obtained N multimodal features; then the initial reference information and the N multimodal features are input into the document multimodal model, and the following operations are performed in a loop iteration manner until the structured data before the output termination character is obtained: read a multimodal feature and obtain a historical structured data; finally, based on the current reference information, a multimodal feature and a historical structured data, the structured data of the character corresponding to the multimodal feature and the updated reference information can be obtained. In this way, the present application can support the structuring of multiple documents based on the document multimodal model, thereby improving the accuracy of the character recognition results.

[0075] In order to better illustrate the image processing method provided in the present application, the training process of the document multimodal model in the present application is first explained by taking the training of the image model completed only by the electronic device 120 as an example.

[0076] The following first describes the structure of the document multimodal model in conjunction with the accompanying drawings:

[0077] See also Figure 2 FIG. 1 is a schematic diagram of the overall structure of a possible document multimodal model in an embodiment of the present application, wherein the constructed document multimodal model includes two parts: a stacked hierarchical cross-modal encoder and an iterative generation.

[0078] In the embodiment of the present application, the electronic device can obtain the multiple multimodal features obtained to obtain initial reference information, which includes the historical memory state of each layer of the document multimodal model and the comprehensive multimodal features corresponding to the image to be structured. Specifically, the initial reference information can be understood as Figure 2 Then, the electronic device can input the initial reference information and the multimodal features of a character into the document multimodal model, and then the document multimodal model processes it to obtain the corresponding structured data and the updated reference information, that is, Figure 2 M1 in.

[0079] In the embodiments of this application, please refer to Figure 3 As shown, Figure 3 Schematic diagram of a possible stacked layered cross-modal encoder in an embodiment of the present application. Specifically, Figure 3 The stacked hierarchical cross-modal encoder shown in can be understood as Figure 2 The part of the document multimodal model that processes the initial reference information and the multimodal features of a character.

[0080] In the embodiments of the present application, Figure 3 As shown, Figure 3 The stacked hierarchical cross-modal encoder in includes multiple concats, which can be understood as the concat() method, which is used to connect two or more arrays. This method does not change the existing arrays, but only returns a copy of the connected arrays.

[0081] In the embodiments of the present application, Figure 3As shown, the stacked layered cross-modal encoder includes multiple modal decoupling blocks Modal decoupling Block. Specifically, each modal decoupling block includes a modal perception mask module MAMM, Add and Norm connected to MAMM, Feed Forward connected to Add and Norm, and another Add and Norm connected to Feed Forward. Among them, Feed Forward can be understood as a feedforward network layer or a feedforward neural network layer, Add and Norm can be understood as an additive normalization network layer and a Layer Norm network layer with residual connections, wherein the additive normalization network layer can smoothly integrate the input and the output of other network layers, and the Layer Norm network layer calculates the last dimension. In addition, normalizing the network layer can prevent the numerical changes within the network layer from being too large, which is beneficial to speed up the training speed and improve the generalization performance.

[0082] Specifically, Figure 3 In the above, Multi modal embeddings can be understood as multimodal embedding features, i.e., multimodal features of a character, Token Prediction can be understood as token prediction, and Decoder output can be understood as decoder output, so that the aforementioned structured data of a character can be obtained. Figure 3 The t in is used to represent the time step. In this application, the time step can be understood as the time interval between characters; and, Figure 3 S is used to represent the text features of characters, X and Y are used to represent the layout features of characters, V is used to represent the visual information of characters, F is used to represent historical structured data, and M is used to represent reference information.

[0083] Obviously, the stacked hierarchical cross-modal encoder in the document multimodal model provided in this application takes the memory M, i.e., the historical memory state and the historical structured data F, as input, generates the output, i.e., Dt, and updates the memory M, wherein the memory includes the historical state of each network layer and the previously embedded multimodal features of the document multimodal model.

[0084] In the specific implementation process, in the first time step, that is, the time interval between processing the zeroth character and the first character, M, that is, the historical memory state is empty, that is, there is no historical memory state, and the historical structured data is empty, and what is input is the multimodal features of all characters, so the initial reference information and the comprehensive multimodal features corresponding to the image to be structured can be obtained.

[0085] In the specific implementation process, at any time step after the first time step, the electronic device can input the output of the previous time step or the termination character into the document multimodal model, so that the document multimodal model can obtain the structured data of the characters corresponding to any time step based on the output of the previous time step or the termination character and the updated reference information.

[0086] For example, see Figure 2 As shown, at the third time step, the input to the document multimodal model is M1, Date0, and the multimodal features of a character, and the document multimodal model outputs Date1 and M2.

[0087] Obviously, in this application, in order to learn more general features and make full use of pre-training data, a unified encoder-decoder module called MD-Bert, namely the aforementioned stacked hierarchical cross-modal encoder, is proposed. The module consists of a stacked hierarchical multimodal Transformer encoder.

[0088] Specifically, in the stacked layered cross-modal encoder, during the encoding phase, the OCR texts can allocate attention to each other, which is equivalent to a self-encoding model. During the decoding phase, the decoded text field can not only see the entire OCR input, but also pay attention to the decoded text part, which is equivalent to a one-way decoder.

[0089] In the embodiment of the present application, considering the multimodal model under the related art, each modality is usually fused by adding or connecting different modal features together, which inevitably introduces unnecessary correlations between each other. However, these correlations are interfering with the ability to enhance the model, because different data modes, i.e., modalities, are actually orthogonal. Therefore, a customized process is designed for the processing of each modal feature in the present application.

[0090] In the specific implementation, in order to solve the problem of different attention of the attention module, this application uses a mask to control the context part that the token should pay attention to when calculating its context representation. Figure 4 As shown, it can be seen that in the present application, the text features corresponding to the characters, namely the semantics of the characters, the layout features, namely the layout of the characters, and the visual features, namely the input features of the computer vision of the characters, are mapped to the hidden state.

[0091] Specifically, this application proposes a MAMM (Modality-Aware Mask Module) encoder, which is a hierarchical multimodal Transformer model that jointly models different modalities in a decoupled manner. In this way, when the modal content corresponding to text features, layout features, and visual features is input into MAMM, MAMM first calculates the attention score of each modality separately, then adds these attention scores together to obtain a fusion score, and finally uses this fusion score to apply masking and subsequent operations to the semantic content.

[0092] In the embodiment of the present application, MAMM follows the design of the basic module in BERT, but replaces the multi-head attention with modality-aware multi-head attention. And it MAMM also includes a feed-forward (FF) layer, residual connection and layer normalization (LN), and does not share modality parameters.

[0093] See also Figure 5 As shown, it is a schematic diagram of the training process of the document multimodal model in the embodiment of the present application. Figure 5 , the training process of the document multimodal model in the embodiment of the present application is described:

[0094] Step 501: pre-training an initial document multimodal model based on preset tasks to obtain a preset document multimodal model; the preset tasks include unidirectional generation tasks, bidirectional generation tasks, and sequence-to-sequence generation tasks.

[0095] In an embodiment of the present application, the initial document multimodal model can be pre-trained based on three cloze tasks, namely, unidirectional LM, bidirectional LM and Sequence-to-Sequence LM.

[0096] Specifically, in the aforementioned cloze task, some text can be randomly selected in the input and replaced with a special tag [MASK]. Then, the corresponding output vector calculated by the Transformer network can be input into a softmax classifier to predict the masked tag. In other words, the parameter learning of the document multimodal model is implemented by minimizing the cross entropy loss, and the adjustment parameters of the document multimodal model are calculated using the predicted tags and the original tags.

[0097] Step 502: Create a sample set according to a plurality of sample images and the marked structured data corresponding to the characters included in the sample images.

[0098] In the embodiment of the present application, a large number of sample images can be selected based on the actual implementation process. Optionally, hundreds of millions of OCR texts can be selected. Specifically, a large number of sample images can be selected first, and then optical character recognition OCR processing can be performed on these sample images, so that a large number of OCR texts can be obtained.

[0099] In the specific implementation process, images of different formats can be selected from databases corresponding to various business scenarios, and images of different fields can also be selected from databases corresponding to various business scenarios, and then OCR processing can be performed on the selected images to obtain OCR text. Furthermore, a sample set can be established based on the OCR text and the marked structured data corresponding to the characters in the OCR text.

[0100] Step 503: Input the sample images in the sample set into the preset document multimodal model for training to obtain multiple structured data, and compare the multiple structured data with the corresponding labeled structured data to obtain multiple comparison results; each structured data is determined based on the multimodal features of a character, current reference information, and the structured data of the previous character of the character.

[0101] In an embodiment of the present application, the electronic device can input the sample images in the sample set into the preset document multimodal model for training, so that the electronic device can obtain the corresponding multiple structured data. Further, the electronic device can compare the multiple structured data with the corresponding labeled structured data respectively to obtain multiple comparison results. In other words, the electronic device can obtain the comparison result of the predicted label, i.e., the obtained structured data, and the original label, i.e., the labeled structured data.

[0102] Step 504: According to the multiple comparison results, the preset document multimodal model is adjusted to obtain a document multimodal model that meets the preset convergence condition.

[0103] In an embodiment of the present application, the electronic device may adjust the preset document multimodal model based on the aforementioned determined multiple comparison results. Specifically, the electronic device may determine an overall loss function; wherein the overall loss function includes a loss function for a sequence-to-sequence task. Furthermore, after training the preset document multimodal model, the electronic device may perform a convergence check on the trained preset document multimodal model through the overall loss function; when the electronic device determines that the trained preset document multimodal model has converged, the document multimodal model is obtained.

[0104] Specifically, the NER-LM task sequence to sequence LM is extended to better constrain the integrity of entities. For example, please continue to refer to Figure 4As shown, the text in the first (i.e., source OCR text) segment can pay attention to each other from two directions within the segment, while the text in the second (i.e., target structured data) segment can only pay attention to the left context in the target and itself, as well as the tokens in the source segment. Specifically, given a source segment s_{1}, s_{2} containing entity values, and the corresponding entity types n_{1}, n_{2} and some background sentences b_{1}, b_{2}, the input format is then formed into A``[BEG]s_{1}b_{1}s_{2}b_{2}[SEP]” and B``[BEG]n_{1}s_{1}n_{2}s_{2}[SEP]”. In this way, each token in A can access all other tokens of A, and each token in B can access all tokens in A and the previous tokens in B. That is, the target entity in B is masked for prediction during training.

[0105] In the embodiment of the present application, since hundreds of millions of different versions and / or different fields of OCR text are used in the model pre-training, the document multimodal model obtained by training has context-awareness. In this way, when encountering input containing typical OCR error text, it can be automatically corrected, thereby improving the accuracy of character recognition.

[0106] For example, see Figure 6 As shown, when the input OCR text lacks space characters or fixed fields, the document multimodal model in the present application can automatically correct it, thereby inputting structured data of characters with higher accuracy.

[0107] In the embodiments of the present application, after introducing the training process of the document multimodal model, the image processing method provided by the present application is described.

[0108] See also Figure 7 , which is a flowchart of a possible implementation of an image processing method in an embodiment of the present application.

[0109] Step 701: extract features of N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters; wherein each modality represents a character attribute, and N is a positive integer.

[0110] Optionally, before executing step 701, the electronic device may further perform optical character recognition (OCR) processing on the structured image to obtain a character recognition result.

[0111] In an embodiment of the present application, after the electronic device obtains a character recognition result or determines an image to be structured, it can perform feature extraction on the N characters contained in the image to be structured or the character recognition result to obtain multimodal features corresponding to each of the N characters.

[0112] In the embodiment of the present application, the electronic device may perform feature extraction on N characters, and the multimodal features corresponding to each of the N characters may be obtained by, but not limited to, the following steps:

[0113] Step a: Perform text information encoding processing on the N characters included in the structured image or character recognition result to obtain text features corresponding to each of the N characters.

[0114] In the specific implementation process, considering that semantic content is the most reliable information for structured extraction of information, in the actual implementation process, the semantic content of each text segment can be obtained from the character recognition results.

[0115] In the embodiment of the present application, the text information encoding process is performed on each character to obtain the text features corresponding to each of the N characters, which can be understood as converting the text corresponding to the character recognition result into the input of the network.

[0116] In a specific implementation process, the electronic device can divide the text corresponding to the character recognition result according to a predetermined batch length L to determine at least one token sequence. That is to say, in the embodiment of the present application, the text corresponding to the character recognition result can be processed with the token sequence as a processing unit. In this way, in a specific implementation process, the structured data corresponding to a token sequence can be determined first, and then the structured data corresponding to the next token sequence can be determined, so that the structured data corresponding to the character recognition result can be determined in an orderly and efficient manner.

[0117] Specifically, a start indication tag [BEG] may be added in front of each token sequence, and an end indication tag [SEP] may also be attached to the end of the sequence before each token. In addition, when the length of the token sequence is different from the predetermined batch length L, an additional padding tag [PAD] may be added. It can be seen that the padding tag [PAD] is used to align the length of the sequence with the predefined batch length L.

[0118] Based on the above method, the electronic device can obtain a token sequence input into the document multimodal model, which is represented as follows:

[0119] S=[[BEG],w1,w2,…,w n ,[SEP],[PAD],…],|S|=L.

[0120] It should be noted that in the specific implementation process, although a fixed length was used during training, due to the use of a novel position embedding method cord2vec, the document multimodal model can also effectively handle variable lengths that are longer than the training time during the specific implementation.

[0121] Specifically, the text features corresponding to the characters included in the image to be structured in the present application can be understood as w in the token sequence.

[0122] Step b: Perform layout information encoding processing on the N characters included in the structured image to obtain layout features corresponding to each of the N characters; the layout features are used to characterize the space and sequence order information corresponding to the characters.

[0123] In the embodiment of the present application, considering that the structured task is a typical two-dimensional scene, the relative position of the characters is crucial to their meaning, and the traditional sequence annotation method requires reading order sorting, which is cumbersome and complicated. Therefore, the present application provides a unified two-dimensional & one-dimensional encoding method, based on which spatial and sequence order information can be encoded simultaneously, thereby avoiding serialization operations.

[0124] In an embodiment of the present application, the electronic device can standardize and discretize the coordinates corresponding to N characters, respectively, obtain the processed coordinates of the N characters, and then determine the layout features corresponding to the N characters based on the processed coordinates of the N characters and a preset bounding box, wherein the layout features include the upper left corner coordinates, the lower right corner coordinates, and at least one of the width and height of the preset bounding box.

[0125] In a specific implementation process, the electronic device can standardize and discretize the coordinates into integers within a range, and the range is, for example, [0, α], where α is the maximum input ratio, that is, the maximum ratio of the coordinates of the input character. In addition, the electronic device can use the vertex coordinates and width and height of the character to correspond to the bounding box.

[0126] In an embodiment of the present application, the electronic device may represent the layout information of the characters based on the tuple. Optionally, the layout information may be represented as: (x0, x1, w), (y0, y1, h), where (x0, y0) and (x1, y1) are used to represent the coordinates of the upper left corner and the lower right corner of each character, and w and h are respectively used to represent the width and height of the character. In this way, the position information and word order information of the characters may be determined based on the tuple of the characters.

[0127] Optionally, the electronic device may determine a standardized bounding box based on the Cord2Vec method, and thereby determine tuple information corresponding to each character based on the standardized bounding box, and further determine layout features corresponding to each character.

[0128] In a specific implementation, the electronic device may assume that each character is distributed in a grid of [W, H], and each character occupies a pixel with a width and height of 1 according to the row priority principle. Then, two embedding layers may be used to embed the x-axis features and the y-axis features respectively, so that the normalized bounding box of the i-th character mark can be determined as:

[0129] x i =PosEmb2D x (x0,x1,w),y i =PosEmb2D y (y0,y1,h).

[0130] Among them, PosEmb2Dx and PosEmb2Dy are position embedding functions, which take coordinates as input, embed each input element separately, and then add each embedding using the element-by-element function.

[0131] In the specific implementation process, considering the possible existence of placeholders, the placeholders can be regarded as some evenly divided grids, so that the bounding box coordinates corresponding to the placeholders can be determined, and then the layout features corresponding to the placeholders can be determined. For example, an empty bounding box X_{PAD}=(0,0,0), Y_{PAD}=(0,0,0) is attached to [PAD], and X_{SEP}=(0,w,w), Y_{SEP}=(0,h,h) is attached to other special tags [BEG], [SEP].

[0132] Step c: Perform image information encoding processing on the N characters included in the structured image to obtain visual features corresponding to each of the N characters.

[0133] In the specific implementation process, ResNet-18 can be used as the basic model of the visual encoder. Specifically, the electronic device can adjust the image to be structured to a W*H image, and then input the W*H image into a visual encoder based on ResNet-18, so that the visual encoder feature map scales the image to a fixed size through average pooling, for example, the width of the image is W / n and the height is H / n. Finally, RoI align is applied to extract a visual embedding of a fixed size for each tag, and then the visual features corresponding to N characters can be obtained. It should be noted that, considering that the visual backbone of the CNN network cannot capture position information, in the actual implementation process, the electronic device also adds layout position embedding to the image tag embedding, so that the visual features of each character can be accurately determined.

[0134] It should be noted that, in the actual implementation process, the aforementioned steps a, b, and c may be executed simultaneously or separately, and there is no particular restriction on the order of execution.

[0135] Step d: Based on the text features, layout features, and visual features corresponding to the N characters, obtain the multimodal features corresponding to the N characters.

[0136] In an embodiment of the present application, after the electronic device obtains the text features, layout features, and visual features corresponding to each of the N characters, it can determine the multimodal features corresponding to each of the N characters based on the text features, layout features, and visual features corresponding to each of the N characters.

[0137] That is to say, the multimodal features corresponding to each of the N characters can be understood as multimodal features including text features, layout features, and visual features. In addition, it should be noted that in the actual implementation process, the multimodal features of the characters may also include other features, which are not limited in this application.

[0138] Step 702: Based on the obtained N multimodal features, initial reference information is obtained, where the reference information includes the attention weight of each layer of the network in the document multimodal model and the comprehensive multimodal features corresponding to the image to be structured.

[0139] In an embodiment of the present application, the electronic device can input the obtained N multimodal features into the document multimodal model respectively, and then fuse the N multimodal features through the normalization network layer of the first encoder in the document multimodal model, and process the fused features through the residual connection network layer of the first encoder to obtain initial reference information.

[0140] Step 703: Input the initial reference information and N multimodal features into the document multimodal model, and perform the following operations in a loop iteration manner until the structured data before the output termination character is obtained:

[0141] Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty;

[0142] Based on current reference information, a multimodal feature and historical structured data, structured data of a character corresponding to a multimodal feature and updated reference information are obtained.

[0143] In an embodiment of the present application, the electronic device may input the initial reference information and N multimodal features into a document multimodal model, and perform corresponding operations in a loop iteration manner until structured data before the output termination character is obtained.

[0144] See also Figure 2As shown, the electronic device can read a multimodal feature and obtain a historical structured data, thereby inputting a multimodal feature, current reference information and a historical structured data into a document multimodal model, and then the document multimodal model processes them to obtain structured data of characters corresponding to a multimodal feature and updated reference information.

[0145] For example, assume that the multimodal feature read by the electronic device is the multimodal feature 3 of the third character, that is, the text feature w3, the layout feature, i.e., two tuple information, and the visual feature V3 of the third character, and reads the structured data Date1 corresponding to the second character, and then inputs the feature w3, the layout feature, i.e., two tuple information, the visual feature V3, F2 and the current reference information M2 into the document multimodal model, so that the document multimodal model can output the structured data Date2 and the updated reference information M3.

[0146] It can be seen that the image processing method provided by the present application can support the structuring of images of unknown layouts and unknown fields without the need for pre-layout of word order. This not only improves the accuracy of character recognition in images, but also improves the processing speed of image structuring, thereby improving the structuring effect of non-fixed layout images in Youtu OCR.

[0147] See also Figure 8 FIG. 1 is a schematic diagram of an exemplary image processing method in an embodiment of the present application. Figure 8 The characters included in the image to be structured shown in the figure are: "Date: 11 / 12 / 96 Attention: John J. Mulderigc / o Mike Baker Company: Philip Morris Management OCRp". Then, the electronic device can perform OCR processing on the image to be structured, and then obtain the character recognition result. Further, the electronic device can perform feature extraction on the multiple characters included in the character recognition result, and obtain multimodal features corresponding to each of the multiple characters.

[0148] Specifically, the electronic device can input the multimodal features corresponding to each of the multiple characters, namely T0, into the document multimodal model, so as to obtain the initial reference information, namely M0. Then, the initial reference information and the multimodal features of the first character are input into the document multimodal model to obtain the structured data, namely Date0, and the updated reference information, namely M1. Furthermore, the multimodal features of the second character, the historical structured data, namely Date0, and the current reference information, namely M1, can be input into the document multimodal model, so as to obtain the structured data, namely Date1, and the updated reference information, namely M2. The loop iteration method is continuously used to execute until the structured data before the output termination character is obtained, so as to obtain the structured data corresponding to the image to be structured, namely "Date: 11 / 12 / 96 [DESP] Attention: John J. Mulderigc / o Mike Baker [DESP] Company: Philip Morris Management OCRp [SEP]".

[0149] It can be seen that compared with the solutions in the related art that must be based on a fixed format, the image processing method provided by the present application does not need to limit the version style of the image and the style of the field, and can perform structured processing on OCR documents of various formats, thereby enhancing the versatility of the image processing solution and improving the accuracy of character recognition, thereby improving the user experience of the object.

[0150] See also Fig. 9 The figure is a schematic diagram of an image processing result in an embodiment of the present application. Obviously, the image processing method provided in the present application can clearly determine the structured data corresponding to each character, and the structured data corresponding to the image can be output or presented in the form of key-value pairs such as fields and recognition results, thereby improving the user experience of the object.

[0151] Based on the same inventive concept, see Fig.10 As shown, it is a schematic diagram of the logical structure of the image processing device in the embodiment of the present application. The image processing device 1000 includes an extraction unit 1001, a determination unit 1002, and a processing unit 1003, wherein:

[0152] The extraction unit 1001 is used to perform feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters; wherein each modality represents a character attribute, and N is a positive integer;

[0153] A determination unit 1002 is used to obtain initial reference information based on the obtained N multimodal features, wherein the reference information includes the attention weight of each layer of the network in the document multimodal model and the comprehensive multimodal features corresponding to the image to be structured;

[0154] The processing unit 1003 is configured to input the initial reference information and the N multimodal features into the document multimodal model, and perform the following operations in a loop iteration manner until the structured data before the output termination character is obtained:

[0155] Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty;

[0156] Based on the current reference information, the one multimodal feature and the one historical structured data, the structured data of the character corresponding to the one multimodal feature and the updated reference information are obtained.

[0157] Optionally, before extracting features from each character contained in the structured image, the device further includes an obtaining unit, which is used to:

[0158] Perform optical character recognition (OCR) processing on the image to be structured to obtain a character recognition result.

[0159] Optionally, the extraction unit 1001 is used to:

[0160] Performing text information encoding processing on the N characters included in the image to be structured to obtain text features corresponding to each of the N characters; and

[0161] Performing layout information encoding processing on the N characters included in the image to be structured to obtain layout features corresponding to each of the N characters; the layout features are used to represent the space and sequence order information corresponding to the characters; and,

[0162] Performing image information encoding processing on the N characters included in the image to be structured to obtain visual features corresponding to each of the N characters;

[0163] Based on the text features, layout features, and visual features corresponding to each of the N characters, multimodal features corresponding to each of the N characters are obtained.

[0164] Optionally, the extraction unit 1001 is specifically used for:

[0165] Standardizing and discretizing the coordinates corresponding to the N characters to obtain the processed coordinates of the N characters;

[0166] Based on the processed coordinates of the N characters and the preset bounding box, layout features corresponding to the N characters are determined, wherein the layout features include upper left corner coordinates, lower right corner coordinates, and at least one of width and height of the preset bounding box.

[0167] Optionally, the determining unit 1002 is specifically configured to:

[0168] Inputting the obtained N multimodal features into the document multimodal model respectively;

[0169] The N multimodal features are fused through the normalization network layer of the first encoder in the document multimodal model, and the fused features are processed through the residual connection network layer of the first encoder to obtain initial reference information.

[0170] Optionally, the device further comprises a training unit, configured to:

[0171] Pre-training the initial document multimodal model based on the preset task to obtain a preset document multimodal model;

[0172] Establishing a sample set according to a plurality of sample images and the marked structured data corresponding to the characters included in the sample images;

[0173] Inputting sample images in the sample set into a preset document multimodal model for training to obtain a plurality of structured data, and respectively comparing the plurality of structured data with corresponding labeled structured data to obtain a plurality of comparison results; each of the structured data is determined based on a multimodal feature of a character, current reference information, and structured data of a character preceding the character;

[0174] According to the multiple comparison results, the preset document multimodal model is adjusted to obtain a document multimodal model that meets a preset convergence condition.

[0175] Optionally, the device further comprises a training unit, configured to:

[0176] Determining an overall loss function; the overall loss function includes a loss function for a sequence-to-sequence task;

[0177] After training the preset document multimodal model, performing a convergence test on the trained preset document multimodal model using the overall loss function;

[0178] When it is determined that the preset document multimodal model after training has converged, the document multimodal model is obtained.

[0179] After introducing the image processing method and apparatus according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0180] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0181] Based on the same inventive concept and the same technical concept as the above method embodiment, the present application embodiment provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Fig.11 As shown, it includes at least one processor 1101 and a memory 1102 connected to the at least one processor. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Fig.11 For example, the processor 1101 and the memory 1102 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0182] In the embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one processor 1101, and the at least one processor 1101 can perform the steps of the above-mentioned image processing method by executing the instructions stored in the memory 1102.

[0183] Among them, the processor 1101 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and identify the video tag of the target video by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 may include one or more processing units, and the processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0184] Processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0185] Memory 1102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0186] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned image processing method.

[0187] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the above-mentioned image processing method.

[0188] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable digital electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable digital electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable electronic device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable electronic device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: The method comprises: Performing feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to the N characters; wherein each modality represents a character attribute, and N is a positive integer; the multimodal features include: text features, layout features, and visual features obtained by performing feature extraction on each character attribute of the N characters; Based on the obtained N multimodal features, initial reference information is obtained, wherein the reference information includes the attention weight of each layer of the network in the document multimodal model and the comprehensive multimodal features corresponding to the image to be structured; The initial reference information and the N multimodal features are input into the document multimodal model, and the following operations are performed in a loop iteration manner until the structured data before the output termination character is obtained: Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty; Based on the current reference information, the one multimodal feature and the one historical structured data, the structured data of the character corresponding to the one multimodal feature and the updated reference information are obtained.

2. The method according to claim 1, characterized in that Before extracting features of each character contained in the structured image, the method further includes: Perform optical character recognition (OCR) processing on the image to be structured to obtain a character recognition result.

3. The method according to claim 1 or 2, characterized in that Performing feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters, including: Performing text information encoding processing on the N characters included in the image to be structured to obtain text features corresponding to each of the N characters; and Performing layout information encoding processing on the N characters included in the image to be structured to obtain layout features corresponding to each of the N characters; the layout features are used to represent the space and sequence order information corresponding to the characters; and, Performing image information encoding processing on the N characters included in the image to be structured to obtain visual features corresponding to each of the N characters; Based on the text features, layout features, and visual features corresponding to each of the N characters, multimodal features corresponding to each of the N characters are obtained.

4. The method according to claim 3, characterized in that Performing layout information encoding processing on the N characters included in the image to be structured to obtain layout features corresponding to each of the N characters includes: Standardizing and discretizing the coordinates corresponding to the N characters to obtain the processed coordinates of the N characters; Based on the processed coordinates of the N characters and the preset bounding box, layout features corresponding to the N characters are determined, wherein the layout features include upper left corner coordinates, lower right corner coordinates, and at least one of width and height of the preset bounding box.

5. The method according to claim 1 or 2, characterized in that: Based on the N multimodal features obtained, the initial reference information is obtained, including: Inputting the obtained N multimodal features into the document multimodal model respectively; The N multimodal features are fused through the normalization network layer of the first encoder in the document multimodal model, and the fused features are processed through the residual connection network layer of the first encoder to obtain initial reference information.

6. The method according to claim 1 or 2, characterized in that: The document multimodal model is trained based on the following method: Pre-training the initial document multimodal model based on the preset task to obtain a preset document multimodal model; Establishing a sample set according to a plurality of sample images and the marked structured data corresponding to the characters included in the sample images; Inputting sample images in the sample set into the preset document multimodal model for training to obtain a plurality of structured data, and respectively comparing the plurality of structured data with corresponding labeled structured data to obtain a plurality of comparison results; each of the structured data is determined based on a multimodal feature of a character, current reference information, and structured data of a character preceding the character; According to the multiple comparison results, the preset document multimodal model is adjusted to obtain a document multimodal model that meets a preset convergence condition.

7. The method according to claim 6, characterized in that According to the multiple comparison results, the preset document multimodal model is adjusted to obtain a document multimodal model that meets a preset convergence condition, including: Determining an overall loss function; the overall loss function includes a loss function for a sequence-to-sequence task; After training the preset document multimodal model, performing a convergence test on the trained preset document multimodal model using the overall loss function; When it is determined that the preset document multimodal model after training has converged, the document multimodal model is obtained.

8. An image processing device, characterized in that: The device comprises: An extraction unit is used to perform feature extraction on N characters contained in the structured image to obtain multimodal features corresponding to each of the N characters; wherein each modality represents a character attribute, and N is a positive integer; the multimodal features include: text features, layout features, and visual features obtained by performing feature extraction on each character attribute of the N characters; A determination unit, configured to obtain initial reference information based on the obtained N multimodal features, wherein the reference information includes an attention weight of each layer of the network in the document multimodal model and a comprehensive multimodal feature corresponding to the image to be structured; The processing unit is used to input the initial reference information and the N multimodal features into the document multimodal model, and perform the following operations in a loop iteration manner until the structured data before the output termination character is obtained: Read a multimodal feature and obtain a historical structured data, wherein the historical structured data is: structured data of a character corresponding to the multimodal feature read last time, and if the multimodal feature is read for the first time, the historical structured data is empty; Based on the current reference information, the one multimodal feature and the one historical structured data, the structured data of the character corresponding to the one multimodal feature and the updated reference information are obtained.

9. The device according to claim 8, characterized in that Before extracting features from each character contained in the structured image, the device further includes an obtaining unit, which is used to: Perform optical character recognition (OCR) processing on the image to be structured to obtain a character recognition result.

10. The device according to claim 8 or 9, characterized in that The extraction unit is used for: Performing text information encoding processing on the N characters included in the image to be structured to obtain text features corresponding to each of the N characters; and Performing layout information encoding processing on the N characters included in the image to be structured to obtain layout features corresponding to each of the N characters; the layout features are used to represent the space and sequence order information corresponding to the characters; and, Performing image information encoding processing on the N characters included in the image to be structured to obtain visual features corresponding to each of the N characters; Based on the text features, layout features, and visual features corresponding to each of the N characters, multimodal features corresponding to each of the N characters are obtained.

11. The device according to claim 10, characterized in that The extraction unit is specifically used for: Standardizing and discretizing the coordinates corresponding to the N characters to obtain the processed coordinates of the N characters; Based on the processed coordinates of the N characters and the preset bounding box, layout features corresponding to the N characters are determined, wherein the layout features include upper left corner coordinates, lower right corner coordinates, and at least one of width and height of the preset bounding box.

12. The device according to claim 8 or 9, characterized in that The determining unit is specifically configured to: Inputting the obtained N multimodal features into the document multimodal model respectively; The N multimodal features are fused through the normalization network layer of the first encoder in the document multimodal model, and the fused features are processed through the residual connection network layer of the first encoder to obtain initial reference information.

13. The device according to claim 8 or 9, characterized in that The device also includes a training unit, which is used to: Pre-training the initial document multimodal model based on the preset task to obtain a preset document multimodal model; Establishing a sample set according to a plurality of sample images and the marked structured data corresponding to the characters included in the sample images; Inputting sample images in the sample set into a preset document multimodal model for training to obtain a plurality of structured data, and respectively comparing the plurality of structured data with corresponding labeled structured data to obtain a plurality of comparison results; each of the structured data is determined based on a multimodal feature of a character, current reference information, and structured data of a character preceding the character; According to the multiple comparison results, the preset document multimodal model is adjusted to obtain a document multimodal model that meets a preset convergence condition.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image processing method according to any one of claims 1 to 7 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images