Image processing method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose an image processing method, apparatus, computer equipment, and storage medium. The image processing method includes: obtaining an image to be identified, performing character recognition processing on the image to be identified, and obtaining a recognition result; wherein the recognition result includes a recognized data sequence, and the data sequence includes one or more of a character sequence, an image sequence, and a position sequence; performing sequence annotation processing on the data sequence based on a multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence; and creating a structured document corresponding to the image to be identified based on the data sequence and the structured category of each data in the data sequence. By adopting this application, the efficiency and accuracy of converting images into structured documents can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] Nowadays, more and more resource information is stored in the form of images. Converting characters in images into structured documents is beneficial for document retrieval, document analysis, text editing, and other intelligent services. For example, converting photos of business licenses into structured documents can improve retrieval efficiency.

[0003] Currently, the conversion of images into structured documents is done manually. The main process includes: manually locating character regions, manually recognizing text within these regions, and manually inputting text to generate structured documents. This manual process is inefficient and susceptible to subjective factors, which reduces accuracy. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, storage medium, and computer program product, which can improve the efficiency and accuracy of converting images into structured documents.

[0005] An embodiment of the present application provides an image processing method, including:

[0006] Acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence, the character sequence being a sequence composed of characters in the image to be recognized, the image sequence being a sequence composed of slices of each character in the character sequence in the image to be recognized, and the position sequence being a sequence composed of the positions of each character in the character sequence in the image to be recognized;

[0007] performing sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence;

[0008] A structured document corresponding to the image to be recognized is created according to the data sequence and the structured category of each data in the data sequence.

[0009] An embodiment of the present application provides an image processing device, including:

[0010] an acquisition module, configured to acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence; the character sequence is a sequence composed of characters in the image to be recognized; the image sequence is a sequence composed of slices of each character in the character sequence in the image to be recognized; and the position sequence is a sequence composed of the positions of each character in the character sequence in the image to be recognized;

[0011] an identification module, configured to perform sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence;

[0012] A creating module is used to create a structured document corresponding to the image to be recognized according to the data sequence and the structured category of each data in the data sequence.

[0013] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the methods in the above embodiments.

[0014] On one hand, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the methods in the above embodiments are executed.

[0015] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program / computer instructions. The computer program / computer instructions are stored in a computer-readable storage medium. When the computer program / computer instructions are executed by a processor of a computer device, the methods in the above embodiments are executed.

[0016] The present application automatically converts images into structured documents by the terminal device without the need for human intervention, which can improve the efficiency of converting images into structured documents. The terminal device automatically executes the process, which can eliminate the interference of subjective factors and improve the accuracy of converting images into structured documents. Furthermore, multimodal features characterize data sequences from multiple dimensions, which can improve the recognition accuracy of structured categories and thus improve the accuracy of structured documents. Furthermore, by identifying the structured categories of each data in the data sequence and then generating a structured document, it is not restricted by the layout of characters in the image to be identified, which can ensure the versatility of the present application in the field of image to structured documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a system architecture diagram of image processing provided by an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of an image processing scenario provided by an example of this application;

[0020] Figure 3 is a schematic diagram of an image processing provided by an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a structured category provided in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of document recognition based on a graph neural network provided in an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of generating a structured document provided by an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of an image processing provided by an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of an image processing provided by an embodiment of the present application;

[0026] Figure 9 This is a flow chart of a multimodal feature extraction model training process provided by an embodiment of the present application;

[0027] Figure 10 is a schematic diagram of a replacement task provided in an embodiment of the present application;

[0028] Figure 11 This is a flow chart of a multimodal feature extraction model training process provided by an embodiment of the present application;

[0029] Figure 12 is a schematic diagram of a matching task provided in an embodiment of the present application;

[0030] Figure 13 This is a flow chart of a multimodal feature extraction model training process provided by an embodiment of the present application;

[0031] Figure 14is a schematic diagram of a classification task provided in an embodiment of the present application;

[0032] Figure 15 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0033] Figure 16 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0036] This application relates to computer vision (CV) and natural language processing (NLP) under artificial intelligence, specifically to OCR (Optical Character Recognition) technology in computer vision technology and sequence labeling technology in natural language processing.

[0037] OCR technology involves an electronic device (such as a scanner or digital camera) examining characters printed on paper, determining their shape by detecting patterns of light and dark, and then using character recognition methods to translate these shapes into computer text. Sequence labeling technology involves classifying each component in a sequence.

[0038] The conversion of images into structured documents proposed in this application can be applied to scenarios where paper documents are stored electronically; it can be applied to scenarios where certificates are stored in information databases to improve the efficiency of certificate information retrieval; it can also be applied to scenarios where key information of bill images is identified, etc.

[0039] See Figure 1 , is a system architecture diagram of image processing provided by an embodiment of the present invention. The server 10f establishes a connection with a terminal device cluster through a switch 10e and a communication bus 10d. The terminal device cluster may include: terminal device 10a, terminal device 10b, ..., terminal device 10c. Taking terminal device 10a as an example, terminal device 10a obtains an image to be recognized, performs character recognition processing on the image to be recognized, and obtains a recognition result. The recognition result includes a data sequence, and the data sequence includes one or more of a character sequence, an image sequence, and a position sequence. The terminal device 10a performs sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data (character and / or slice and / or position) in the data sequence. A structured document of the image to be recognized is created based on the structured category and the data sequence.

[0040] The step of generating a structured document can also be performed by server 10f. Terminal device 10a obtains an image to be recognized and sends it to server 10f. Server 10f performs character recognition and sequence labeling to generate a structured document. Server 10fa then sends the generated structured document to terminal device 10a. Terminal device 10a can perform subsequent operations based on the structured document, such as document retrieval and editing.

[0041] A terminal device may also be referred to as a terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent, or user device. A terminal device may be, but is not limited to, a smart home appliance, a handheld device with wireless communication capabilities (e.g., a smartphone or tablet), a computing device (e.g., a personal computer (PC)), an in-vehicle terminal, an intelligent voice interaction device, a wearable device, or other smart device.

[0042] Server 10f can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0043] The following Figure 2 As an example, we will explain how to convert an image into a structured document. Figure 2, which is a schematic diagram of an image processing scenario provided by an example of the present application. The terminal device obtains image 20a, performs character recognition processing on image 20a, and obtains a character sequence 20b, an image sequence 20c, and a position sequence 20d. The character sequence 20b is a sequence composed of characters contained in image 20a, the image sequence 20b is a sequence composed of slices of each character in the character sequence 20b in image 20a, and the position sequence 20c is a sequence composed of the positions of each character in the character sequence 20b in image 20a. In general, the characters in the character sequence 20b, the images in the image sequence 20c, and the positions in the position sequence 20d are one-to-one corresponding.

[0044] The terminal device performs feature conversion on the above three sequences. First, each character in the character sequence 20b is converted into a character feature. Here, a pre-trained word vector model can be used to convert each character into a word vector, and the converted word vector can be used as the character feature.

[0045] Each slice in image sequence 20c is then converted into an image feature. A pre-trained graph vector model can be used to convert each slice into a graph vector, and the converted graph vector is used as the image feature. An image classification model can be trained based on large-scale images. The image classification model includes a hidden layer and an output layer, and the hidden layer of the image classification model is used as the graph vector model.

[0046] Finally, each position in the position sequence 20d is converted into a position feature. Each position can be standardized and the standardized position can be used as the position feature.

[0047] There is no limitation on the order of converting each character in the character sequence 20b into a character feature, converting each image in the image sequence 20c into an image feature, and converting each position in the position sequence 20b into a position feature.

[0048] The terminal device combines all character features into a character feature sequence, all image features into an image feature sequence, and all position features into a position feature sequence. By aligning and splicing the above three feature sequences, a multimodal feature sequence can be obtained. It can be seen that each multimodal feature in the multimodal feature sequence is a fusion of character features, image features and position features.

[0049] The terminal device sequentially inputs the multimodal feature sequence into a trained annotation model. The annotation model can be an RNN (Recurrent Neural Network) structure, an LSTM (Long Short-Term Memory) structure, or a Transformer. The annotation model outputs structured categories for each multimodal feature. The structured categories can include K (Key) categories and V (Value) categories. Furthermore, the K category can be further subdivided into KB (Begin), KI (Inner), and KE (End). Similarly, the V category can be further subdivided into VB, VI, and VE categories.

[0050] Of course, the identified structured categories can also be considered as structured categories of characters, slices, and positions.

[0051] The terminal device can divide the character sequence 20b into multiple K fields and multiple V fields according to the structured category of each character. A K field only includes one KB category and one KE category, and a V field only includes one VB category and one VE category. In general, the character sequence is divided into multiple structured fields. Since there are multiple K fields and multiple V fields, it is also necessary to pair these multiple K fields and multiple V fields to obtain multiple KV field pairs. A document 20e containing multiple KV field pairs is generated. Document 20e is the structured document corresponding to image 20a. The terminal device can generate a structured document according to a certain format. For example, a KV field pair corresponds to a row (or a column) of the structured document.

[0052] Subsequently, the user may perform further business processing based on the generated structured document 20e, such as performing document retrieval based on the structured document 20e, editing the structured document 20e, or extracting key fields from the structured document 20e.

[0053] In which, character recognition processing is performed on the image to be recognized (such as the image 20a in the above embodiment) to obtain recognition results (such as the character sequence 20b, image sequence 20c and position sequence 20e in the above embodiment), and the data sequence is annotated based on the multimodal feature sequence of the data sequence (such as the multimodal feature sequence formed by splicing the character feature sequence, image feature sequence and position feature sequence in the above embodiment) to obtain structured categories (such as the K category and V category in the above embodiment). The specific process of creating a structured document (such as the structured document 20e in the above embodiment) can be seen below. Figure 3-Figure 14 Corresponding embodiments.

[0054] See Figure 3, which is a schematic diagram of an image processing provided by an embodiment of the present application. The image processing may include the following steps:

[0055] Step S101, obtain an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, and the data sequence includes one or more of a character sequence, an image sequence, and a position sequence. The character sequence is a sequence composed of characters in the image to be recognized, the image sequence is a sequence composed of slices of characters in the character sequence in the image to be recognized, and the position sequence is a sequence composed of positions of characters in the character sequence in the image to be recognized.

[0056] Specifically, the terminal device obtains the image to be identified (as described above Figure 2 Corresponding to the image 20a in the embodiment, character recognition processing is performed on the image to be recognized to obtain the recognition result (as shown above Figure 2 This corresponds to the character sequence 20b, image sequence 20c and position sequence 20e in the embodiment.

[0057] The recognition result includes a recognized data sequence, which includes one or more of a character sequence, an image sequence, and a position sequence. That is, the data sequence may include only a character sequence, or only an image sequence, or only a position sequence; the data sequence may include both a character sequence and an image sequence, or both a character sequence and a position sequence, or both an image sequence and a position sequence; and the data sequence may include both a character sequence, an image sequence, and a position sequence.

[0058] The character sequence can be a sequence of characters in the image to be recognized, the image sequence is a sequence of slices of each character in the character sequence in the image to be recognized, and the position sequence is a sequence of positions of each character in the character sequence in the image to be recognized. In general, there is a one-to-one correspondence between the characters in the character sequence, the slices in the image sequence, and the positions in the position sequence.

[0059] The number of character sequences can be one or more. If there is one, then the characters in the character sequence are all the characters in the image to be recognized. If there are multiple character sequences, then one character sequence can correspond to a paragraph or a line of the image to be recognized. Of course, the number of character sequences = the number of image sequences = the number of position sequences. These multiple character sequences, multiple image sequences, and multiple position sequences are in one-to-one correspondence, and the elements (characters, images, and positions) in the one-to-one correspondence of one character sequence, one image sequence, and one position sequence are also in one-to-one correspondence.

[0060] The following describes the specific process of how to perform character recognition processing to obtain a data sequence:

[0061] Target detection is performed on the image to be recognized. The purpose of target detection is to find the position of the character area in the image to be recognized and to recognize the characters in the character area. All recognized characters are combined into the original text (that is, the original text contains all the characters in the image to be recognized), and the recognized positions are combined into the original position sequence. The original position sequence is a sequence composed of the positions of all the characters in the image to be recognized. Of course, the number of characters in the original text = the number of positions in the original position sequence, and there is a one-to-one correspondence between the characters in the original text and the positions in the original position sequence.

[0062] Each position includes the upper left corner coordinates, lower right corner coordinates, length and width of the character area. The character area is a rectangular area.

[0063] The terminal device extracts slices corresponding to each position in the original position sequence from the image to be recognized, and combines the extracted slices into an original image sequence. The original image sequence, original text and original position sequence are all in one-to-one correspondence.

[0064] The terminal device divides the original text into K character sequences according to the original position sequence, where one character sequence corresponds to one paragraph. The original image sequence and the original position sequence are also divided in the same way as the K character sequences, obtaining K image sequences and K position sequences respectively.

[0065] For example, according to the principle of one character sequence per paragraph, the original text can be divided into three character sequences. The 1st to 5th characters in the original text are character sequence 1, the 6th to 15th characters are character sequence 2, and the 16th to 20th characters are character sequence 3; similarly, the 1st to 5th slices in the original image sequence are image sequence 1, the 6th to 15th slices are image sequence 2, and the 16th to 20th slices are image sequence 3; the 1st to 5th positions in the original position sequence are position sequence 1, the 6th to 15th positions are position sequence 2, and the 16th to 20th positions are position sequence 3.

[0066] Step S102: performing sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence.

[0067] Specifically, when the data sequence includes multiple sequences among character sequences, image sequences and position sequences, the sequences contained in the data sequence are converted into feature sequences respectively, and the multiple feature sequences are aligned and spliced into a multimodal feature sequence of the data sequence.

[0068] The feature sequence obtained by converting a character sequence is a character feature sequence, the feature sequence obtained by converting an image sequence is an image feature sequence, and the feature sequence obtained by converting a position sequence is a position feature sequence. The terminal device can use a word vector model to convert a character sequence into a character feature sequence. That is, the word vector model is used to encode each character in the character sequence to obtain the character feature of each character. All character features are then concatenated to obtain a character feature sequence.

[0069] The terminal device can call a graph vector model to convert the image sequence into an image feature sequence. Specifically, the graph vector model is called to encode each slice in the image sequence, obtaining the image features of each slice. All image features are then concatenated to obtain the image feature sequence. Alternatively, when performing target recognition, the regional features of each slice can be obtained and used as image features.

[0070] The terminal device may normalize each position in the position sequence (L1 normalization or L2 normalization) to obtain the position feature of each position.

[0071] For example, if the data sequence includes a character sequence and an image sequence, the character sequence is converted into a character feature sequence, the image sequence is converted into an image feature sequence, and the two feature sequences are aligned and spliced into a multimodal feature sequence.

[0072] In general, the data modality corresponding to the data sequence can determine the modal components of the multimodal features. If the data sequence includes a character sequence and an image sequence, the multimodal features in the multimodal feature sequence include character features and image features; if the data sequence includes a character sequence and a position sequence, the multimodal features include character features and position features; if the data sequence includes an image sequence and a position sequence, the multimodal features include image features and position features; if the data sequence includes a character sequence, an image sequence, and a position sequence, the multimodal features include character features, image features, and position features.

[0073] When the data sequence only contains a character sequence, or only contains an image sequence, or only contains a position sequence, the terminal device can obtain an auxiliary sequence of the data sequence, convert the data sequence into a feature sequence, convert the auxiliary sequence into an auxiliary feature sequence, and align and splice the feature sequence and the auxiliary feature sequence into a multimodal feature sequence of the data sequence.

[0074] For example, the auxiliary sequence of the character sequence may include a pinyin sequence composed of pinyin, an audio sequence composed of pronunciation, etc.; the auxiliary sequence of the image sequence may include a frequency domain image sequence (a sequence composed of images after the slices are converted to the frequency domain) and a time domain image sequence (a sequence composed of images after the slices are converted to the time domain).

[0075] The terminal device calls the trained multi-modal feature extraction model to encode the multi-modal feature sequence, obtaining a multi-modal structured feature sequence, where one multi-modal feature corresponds to one multi-modal structured feature. The multi-modal structured feature sequence is a sequence composed of the multi-modal structured features of each data (each character and / or each slice and / or each position) in the data sequence.

[0076] The terminal device then calls the structured category recognition model to decode the multi-modal structured feature sequence, obtaining the structured category of each data (each character and / or each slice and / or each position) in the data sequence. The structured category can indicate structured information, and the structured category can specifically include a key structure category (such as the K category in the corresponding Figure 2 embodiment) and a value structure category (such as the V category in the corresponding Figure 2 embodiment). The key structure category can further include a key start category, a key middle category, and a key end category, and the value structure category can further include a value start category, a value middle category, and a value end category.

[0077] The multi-modal feature extraction model and the structured category recognition model can be an RNN structure, an LSTM structure, a Bert structure, or a Transformer structure, etc.

[0078] Furthermore, the structured category can also include a title structure category, a footnote structure category, etc.

[0079] Please refer to Figure 4 , Figure 4 which is a schematic diagram of a structured category provided by an embodiment of this application. In addition to the key structure category and the value structure category, the structured category can also include a title structure category T, and the title structure category T can also be subdivided into a title start category T-B, a title middle category T-I, and a title end category T-E. As Figure 6 shown, the structured category of the character "驾" is the title start category T-B, the structured category of the character "驶" is the title middle category T-I, and the structured category of the character "证" is the title end category T-E.

[0080] The structured category can also include a footnote structure category F, and the footnote structure category F can also be subdivided into a footnote start category F-B, a footnote middle category F-I, and a footnote end category F-E. As Figure 6 shown, the structured category of the character "车" is the footnote start category F-B, the structured categories of the characters "管" and "所" are both the footnote middle category F-I, and the structured category of the character "发" is the footnote end category F-E.

[0081] Step S103: creating a structured document corresponding to the image to be recognized according to the data sequence and the structured category of each data in the data sequence.

[0082] Specifically, when the data sequence includes at least a character sequence, the structured category of each character in the character sequence can be determined by the above steps.

[0083] If there are multiple character sequences (K is greater than 1), the structured category of each character in each character sequence can be determined in the above manner.

[0084] The terminal device may divide the K character sequences into N key fields according to the structured category of each character in the K character sequences, where one key field corresponds to a key start category and a key end category.

[0085] For example, the terminal device may start traversing from the first character of the character sequence, find the first key start category, continue traversing to find the first key end category, and combine all characters between the first key start category and the first key end category into a key field; continue traversing to find the second key start category and the second key end category, and combine all characters between the second key start category and the second key end category into a key field.

[0086] Similarly, the terminal device may divide the character sequence into N value fields according to the structured category of each character in the character sequence, where one value field corresponds to a value start category and a value end category.

[0087] The N key fields and the N value fields are key fields and value fields corresponding to the K character sequence.

[0088] If the structured category also includes other categories, characters belonging to the same category can be combined into a single field. For example, if the structured category also includes the title structure category, characters belonging to the title structure category can be combined into a title field; if the structured category also includes the footnote structure category, characters belonging to the footnote structure category can be combined into a footnote field, and so on.

[0089] At this point, the terminal device has obtained N key fields and N value fields. It is also necessary to pair these N key fields and N value fields to obtain N key-value pairs, and create a document according to the preset layout structure. The document includes these N key-value pairs. The document is the structured document of the image to be recognized (as mentioned above). Figure 2 Corresponding to 20e in the embodiment. Of course, if there are other fields (eg, title field, footer field, etc.), these fields can also be added to the structured document according to the preset format.

[0090] For example, N key-value pairs are arranged in the center of the structured document, and one key-value pair occupies a row (or column) of the structured document, the title field is arranged at the top of the structured document, the footer field is arranged at the bottom of the structured document, etc.

[0091] There are many ways to pair N key fields with N value fields. This application uses a graph neural network model or a classification model for pairing. These two methods are explained below.

[0092] The process of pairing based on the graph neural network model is as follows:

[0093] The terminal device constructs a topology graph based on N key fields and N value fields. The topology graph includes 2N nodes, with one node corresponding to one key field or one value field. Any two of the 2N nodes are designated as the first node and the second node, respectively. The weight of the edge connecting the first node and the second node is determined based on the distance between the field corresponding to the first node and the field corresponding to the second node in the image to be recognized. Each node also has a corresponding node feature. The node feature of the first node is the feature obtained by superimposing the multimodal structured features of all characters contained in the field corresponding to the first node.

[0094] The following formula (1) can be used to calculate the edge weight between the first node and the second node:

[0095]

[0096] Among them, h max and w max Refers to the height and width of the image to be identified, i and j refer to the first node and the second node respectively. x and j x Refers to the horizontal coordinates of the fields corresponding to the first node and the second node in the image to be identified, i y and j y They respectively refer to the vertical coordinates of the fields corresponding to the first node and the second node in the image to be identified.

[0097] Call the graph neural network to match the topological graph and obtain the relationship category between the first node and the second node. The matching process of the graph neural network is to obtain the output Z (Z is an N×F matrix, F is the dimension of the output feature) by only taking the feature X of the topological graph (X is an N×m matrix, N is the number of nodes in the topological graph, and m is the dimension of the node feature) as input without changing the adjacency relationship of the nodes. A represents the adjacency matrix of the topological graph. The graph neural network (GCN) can obtain the information of the neighbors in the graph through multi-layer convolution operations. For example, for a layer of GCN, the new F-dimensional vertex feature matrix is: D is the degree matrix of the topological graph.

[0098] In this application, the dimension F of the output feature is N, and each row represents the matching probability between a node and the remaining N-1 nodes. If the matching probability is greater than the preset probability threshold, then the relationship category between the two nodes can be considered as a matching relationship category.

[0099] If the relationship type between the first node and the second node is a matching relationship type, the key field and the value field corresponding to the first node and the second node are combined into one key-value pair.

[0100] In summary, the terminal device can match N key-value pairs based on the graph neural network model.

[0101] See Figure 5 , Figure 5 This is a schematic diagram of document recognition based on a graph neural network provided by an embodiment of the present application. Through the inclusion relationship between phrases and documents, a character document topology map is constructed. The hidden layer based on the graph neural network can extract the feature representation of each phrase and the feature representation of each document. The output layer based on the graph neural network identifies the type of each document. Figure 5 As shown, it is identified that document 1 and document 2 are both type A, and document 4 is type B.

[0102] The process of pairing based on the classification model is:

[0103] Combine N key fields and N value fields into N 2 Field combinations, a field combination includes a key field and a value field. 2 For any field combination in the field combinations (referred to as the target field combination), the terminal device obtains the multimodal structured features of all characters contained in the key field in the target field combination, and superimposes the obtained multimodal structured features as the key field features of the key field in the target field combination. The terminal device obtains the multimodal structured features of all characters contained in the value field in the target field combination, and superimposes the obtained multimodal structured features as the value field features of the value field in the target field combination.

[0104] The terminal device concatenates the key field features and the value field features into the combined features of the target field combination, and inputs the combined features into the trained classification model. The classification model classifies the combined features to obtain the matching probability of the target field combination. If the matching probability is greater than the preset probability threshold, the target field combination is used as a key-value pair.

[0105] The same method can be used to determine whether the remaining field combinations are key-value pairs.

[0106] Alternatively, if the data sequence does not include a character sequence but includes an image sequence, the terminal device can use OCR technology to identify the characters in each slice of the image sequence. Similarly, the structured category corresponding to the slice is also used as the structured category of the character corresponding to the slice. Subsequently, the key field and value field can be further divided and paired according to the above method to generate a structured document.

[0107] If the data sequence does not include either a character sequence or an image sequence, and only includes a position sequence, the terminal device first extracts the slices corresponding to each position in the position sequence from the image to be recognized, combining them into an image sequence. The terminal device then uses OCR technology to identify the characters contained in each slice in the image sequence. Similarly, the structured category corresponding to the position is used as the structured category of the character corresponding to that position. Subsequently, the key field and value field can be further divided and paired according to the above method to generate a structured document.

[0108] See Figure 6 , Figure 6 This is a schematic diagram of generating a structured document provided by an embodiment of the present application, such as Figure 6 As shown, first, character recognition is performed on the image to obtain the text (the text can correspond to the character sequence of this application), the position of each character in the text, and the sub-image of each character in the text in the image (which can correspond to the slice in this application). The terminal device converts each character in the text into a word vector, and then combines them into a character feature sequence. Similarly, each sub-image is converted into an image feature, and then all image features are combined into an image feature sequence, each position is converted into a position feature, and all position features are combined into a position feature sequence. The terminal device aligns and splices the above-mentioned character feature sequence, image feature sequence and position feature sequence into a multimodal feature sequence. The multimodal feature sequence is input into the multimodal feature extraction model to extract the multimodal structured features of each character / each sub-image / each position, and the multimodal structured feature sequence is input into the structured recognition model to obtain whether each character is a K category or a V category.

[0109] Among them, the multimodal feature extraction model is pre-trained using a self-supervised model, and through multiple training tasks, the multimodal feature extraction model can extract structured features.

[0110] The terminal device then divides the characters in the text into K fields (which may correspond to the key fields in this application) and V fields (which may correspond to the value fields in this application) according to whether each character is in the K category or the V category. By pairing the K fields and the V fields, structured text can be obtained.

[0111] See Figure 7 , Figure 7This is a schematic diagram of image processing provided by an embodiment of the present application. After the terminal device obtains the image to be identified, it performs KV field extraction, that is, identifies the K field and V field in the image. The specific process is: perform character recognition processing on the image to obtain all the characters contained in the image, the slices containing the characters, and the positions of the characters in the image. The above data are all converted into features and spliced into a multimodal feature sequence. The multimodal feature sequence is identified by a multimodal feature extraction model and a structured category recognition model to determine whether each character is in the K category or the V category, and then the corresponding characters are combined into K fields and V fields. KV field matching is to pair multiple K fields with multiple V fields, and the paired K fields and V fields are output in pairs according to the preset format to obtain structured text.

[0112] See Figure 8 , Figure 8 This is a schematic diagram of an image processing provided by an embodiment of the present application. Figure 8 As shown, characters and positions are identified from an image, and slices are cut from the image. Characters are converted into character features, slices can be converted into image features, and positions can be converted into position features. By correspondingly splicing character features, image features, and position features, a multimodal feature sequence can be obtained. Image features can be image region features extracted during character recognition. The multimodal feature sequence is input into a multimodal feature extraction model to obtain a multimodal structured feature representation of each character. Based on the multimodal structured feature representation of each character, the structured category of each character is identified. In general structured task scenarios, structured categories include key structure categories and value structure categories. In vertical scenario structured tasks, structured categories can also include footnote structure categories, title structure categories, ID structure categories, etc. The terminal device can create a structured document of the image based on the structured category of each character.

[0113] From the above, it can be seen that the present application converts the OCR structuring problem into a sequence labeling problem, and creates a structured document by identifying the structured categories of all characters in the image to be identified. It is not restricted by the layout of the characters in the image to be identified, nor does it require setting the document template in advance, which can improve versatility. Furthermore, the present application pairs the fields through a graph neural network model or a classification model to ensure the accuracy of the generated structured document.

[0114] The multimodal feature extraction model in this application is based on a feature extraction model trained using pre-training tasks. Pre-training involves designing different tasks for multiple application scenarios, each of which corresponds to a corresponding model. The feature extraction portion of the model is shared, but the feature recognition portion is unique to each model. This allows the feature extraction portion to extract universal features, and subsequent fine-tuning using a small number of samples from the target application scenario is sufficient to obtain a model applicable to a specific scenario.

[0115] Since the training model of this application is to identify the structured category of each data (i.e., sequence labeling scenario), after the pre-training is completed, fine-tuning of a small number of samples in the sequence labeling scenario of the structured category can obtain the trained multimodal feature extraction model and structured category recognition model, which can reduce the sample requirement in the sequence labeling scenario and improve the robustness of the multimodal feature extraction model.

[0116] The pre-training task may include one or more of the replacement task, matching task and classification task. When the pre-training task includes multiple of the replacement task, matching task and classification task, in one iteration process, only one task may participate (randomly selected), or multiple tasks may participate together, or some of the multiple tasks may participate.

[0117] For example, the pre-training task includes Task 1 and Task 2. The first iteration is based on Task 1, and the second iteration is based on Task 2. Alternatively, any iteration is based on Task 1 + Task 2.

[0118] The replacement task is to perform data replacement processing on the first modality sample sequence, and predict the replacement result of each sample in the first modality sample sequence after processing based on the second modality sample sequence (in layman's terms, it is to predict whether each sample in the first modality sample sequence after replacement processing is replaced). The first modality sample sequence and the second modality sample sequence are two different sequences among the sample character sequence, sample image sequence and sample position sequence, and the sample character sequence is the first modality sample sequence or the second modality sample sequence (it can be understood that the sample character sequence must participate in the replacement task, and only one of the sample image sequence and the sample position sequence may participate, or both may participate). The replacement task can achieve semantic and image alignment, or semantic and position alignment, or both semantic and image alignment and semantic and position alignment.

[0119] The sample character sequence, the sample image sequence and the sample position sequence are in a one-to-one correspondence, that is, the characters in the sample character sequence, the slices in the sample image sequence and the positions in the sample position sequence are derived from the same image.

[0120] The matching task is to predict the contextual relationship between the sample image subsequence and the sample character subsequence, where the sample image subsequence is a subsequence of the sample image sequence and the sample character sequence is a subsequence of the sample character sequence. The matching task can enhance field-level multimodal alignment.

[0121] The classification task is to predict the document category of a sample character sequence (including handwriting category, letter category, form category, etc.). The classification task can be performed at the document level so that the model can capture more global features.

[0122] It is worth noting that replacement tasks, matching tasks, and classification tasks are all self-supervised tasks. Labels can be generated during the pre-training process. The self-supervisory approach can expand the sample size. The multimodal feature extraction model trained with a large number of samples has better feature extraction capabilities.

[0123] See Figure 9 , Figure 9 This is a flow chart of a method for training a multimodal feature extraction model provided in an embodiment of the present application. This embodiment of the present application mainly describes how to train a multimodal feature extraction model based on a replacement task:

[0124] Step S201: Acquire a first modal sample sequence and a second modal sample sequence, and perform replacement processing on the first modal sample sequence to obtain a replacement sample sequence.

[0125] Specifically, the first modality sample sequence and the second modality sample sequence are two different sequences among the sample character sequence, the sample image sequence and the sample position sequence, and the sample character sequence is the first modality sample sequence or the second modality sample sequence.

[0126] The terminal device replaces the first modal sample sequence and uses the replaced first modal sample sequence as the replacement sample sequence. The replacement can replace some samples in the first modal sample sequence with data of other modalities, or set some samples to be empty. For example, if the first modal sample sequence is a sample character sequence, some characters in the sample character sequence are replaced with random characters to obtain a replacement sample sequence.

[0127] Step S202: convert the replacement sample sequence into a replacement feature sequence, and convert the second modality sample sequence into a second modality sample feature sequence, and align and splice the replacement feature sequence and the second modality sample feature sequence into a first sample multimodal feature sequence.

[0128] Specifically, the terminal device converts the replacement sample sequence into a replacement feature sequence, and converts the second modality sample sequence into a second modality sample feature sequence. The conversion method here is the same as the previous method of obtaining the character feature sequence, image feature sequence and position feature sequence.

[0129] Similarly, the replacement feature sequence and the second modal sample feature sequence are aligned and spliced into the first sample multimodal feature sequence.

[0130] Step S203: Call the multimodal feature extraction model to encode the first sample multimodal feature sequence to obtain a first sample structured feature sequence.

[0131] Step S203: calling a replacement discrimination model to decode the first sample structured feature sequence to obtain a predicted replacement result sequence.

[0132] Specifically, the predicted replacement result sequence is whether each sample in the replacement sample sequence predicted by the model is replaced.

[0133] Step S204: obtaining a replacement result label sequence of the replacement sample sequence, and training the multimodal feature extraction model according to the replacement result label sequence and the predicted replacement result sequence.

[0134] Specifically, the terminal device obtains the replacement result label sequence (i.e., the actual replacement result) of the replacement sample sequence, determines the replacement loss based on the replacement result label sequence and the predicted replacement result sequence, and adjusts the model parameters of the multimodal feature replacement model based on the replacement loss.

[0135] Because the first modality sample sequence and the second modality sample sequence are two different sequences among sample character sequences, sample image sequences, and sample position sequences, multiple combinations are possible. For example, the first modality sample sequence is a sample character sequence, and the second modality sample sequence is a sample position sequence, or the first modality sample sequence is a sample position sequence, and the second modality sample sequence is a sample character sequence, or the first modality sample sequence is a sample image sequence, and the second modality sample sequence is a sample character sequence, and so on.

[0136] The terminal device can select one or more target combinations to participate in training according to business requirements. Each target combination can determine a replacement loss. All replacement losses can be superimposed to obtain the total replacement loss, and then the model parameters of the multimodal feature replacement model can be adjusted based on the total replacement loss.

[0137] For example, the first modality sample sequence is a sample character sequence, and the second modality sample sequence is a sample position sequence. This combination can determine a loss; the first modality sample sequence is a sample image sequence, and the second modality sample sequence is a sample character sequence. This combination can also determine a loss. These two losses are superimposed, and the superimposed loss is used to train the multimodal feature replacement model.

[0138] It should be noted that the modal components of the samples participating in the training will directly determine the modal components of the multi-modal features and also determine which sequences are included in the data sequence. This is because the modality of the samples processed by the multi-modal feature extraction model during the training process is the same as the modality of the data (i.e., the multi-modal feature sequence) processed by the multi-modal feature extraction model during the model usage process.

[0139] If only the sample position sequence does not participate in the training, then the data sequence includes the character sequence and the image sequence; if only the sample image sequence does not participate in the training, then the data sequence includes the character sequence and the position sequence; if the sample character sequence, the sample image sequence, and the sample position sequence all participate in the training, then the data sequence includes the character sequence, the image sequence, and the position sequence.

[0140] Please refer to Figure 10 , Figure 10 which is a schematic diagram of a replacement task provided by an embodiment of the present application. In the Bert model in the field of natural language processing, the Bert model is also obtained through pre-training. Specifically, as Figure 10 shown, the pre-training of the Bert model includes a masking task. Specifically, by masking some characters, the model is allowed to predict the masked characters. In Figure 10 , masking method 1 is to mask the characters "份" and "号" in the string "身份证号码", and the model predicts the masked characters. Masking method 2 is to mask the characters "身" and "份" in the string "身份证号码", and the model predicts the masked characters.

[0141] The replacement task included in the pre-training task of the multi-modal feature extraction model can be further divided into a character replacement task and an image replacement task. The character replacement task can be to retain the sample position sequence and replace the characters in the sample character sequence, and let the multi-modal feature extraction model + replacement discriminant model predict whether the characters in the replaced sample character sequence are replaced. The character replacement task can align semantics and structure (i.e., position). The image replacement task can be to retain the sample character sequence and replace the images in the sample image sequence, and let the multi-modal feature extraction model + relationship discriminant model predict whether the images in the replaced sample image sequence are replaced. The image replacement task can align semantics and images.

[0142] Please refer to Figure 11 , Figure 11 which is a schematic diagram of a process for training a multi-modal feature extraction model provided by an embodiment of the present application. This embodiment mainly describes how to train a multi-modal feature extraction model based on a matching task:

[0143] Step S301, obtain a sample image sequence, extract a sample image subsequence from the sample image sequence, obtain a sample character sequence, and extract a sample character subsequence from the sample character sequence.

[0144] Specifically, the terminal device obtains a sample image sequence, extracts a plurality of consecutive images from the sample image sequence as a sample image subsequence, and obtains a sample character sequence, extracts a plurality of consecutive characters from the sample character sequence as a sample character subsequence.

[0145] Step S302 : converting the sample image subsequence into a sample image feature subsequence, and aligning and splicing the sample image feature subsequence and the sample character feature subsequence of the sample character subsequence into a second sample multimodal feature sequence.

[0146] The conversion method here is the same as the previous method of obtaining character feature sequence, image feature sequence and position feature sequence.

[0147] Step S303: Call the multimodal feature extraction model to encode the second sample multimodal feature sequence to obtain a second sample structured feature sequence.

[0148] Step S304: calling a relationship discrimination model to decode the second sample structured feature sequence to obtain a predicted contextual relationship.

[0149] Specifically, predicting the contextual relationship means that the model predicts the contextual relationship between the sample character subsequence and the sample image subsequence. The predicted contextual relationship includes contextual matching relationship and contextual mismatching relationship.

[0150] Step S305 , obtaining a contextual relationship label between the sample image subsequence and the sample character subsequence, and training the multimodal feature extraction model according to the predicted contextual relationship and the contextual relationship label.

[0151] Specifically, the contextual relationship label is the actual contextual relationship between the sample character subsequence and the sample image subsequence. The matching loss is determined according to the contextual relationship label and the predicted contextual relationship predicted by the model, and the model parameters of the multimodal feature replacement model are adjusted based on this matching loss.

[0152] See Figure 12 , Figure 12It is a schematic diagram of a matching task provided by an embodiment of the present application, which randomly samples from a sample image sequence and randomly samples from a sample character sequence. The image obtained by random sampling is converted into image features, and then spliced into a sample image feature sequence. The characters obtained by random sampling are converted into character features, and also spliced into a sample character feature sequence, and the sample image feature sequence and the sample character feature sequence are input into a multimodal feature extraction model (assuming that the multimodal feature extraction model needs to input features of three modalities (character features, image features and position features). If the input data has only one modal feature, the remaining two modal features are set to empty) to extract the multimodal structured features of the input image and the multimodal structured features of the input characters. The multimodal structured feature sequence is identified by a relational discrimination model to obtain the contextual relationship R between the character and the image. The multimodal feature extraction model is then trained by the predicted contextual relationship and the actual contextual relationship.

[0153] See Figure 13 , Figure 13 This is a flow chart of a multimodal feature extraction model training process provided by an embodiment of the present application. This embodiment mainly describes how to train a multimodal feature extraction model based on a classification task:

[0154] Step S401: Acquire a sample character sequence, and convert the sample character sequence into a sample character feature sequence.

[0155] Specifically, the terminal device obtains a sample character sequence, calls a word vector model to convert each character in the sample character sequence into a word vector, and combines the converted word vectors into a sample character feature sequence.

[0156] Step S402: Call the multimodal feature extraction model to encode the sample character feature sequence to obtain a third sample structured feature sequence.

[0157] Step S403: calling a type discrimination model to decode the third sample structured feature sequence to obtain a predicted category.

[0158] The predicted category is the document category to which the sample character sequence predicted by the model belongs.

[0159] Step S404: Obtain the category label of the sample character sequence, and train the multimodal feature extraction model according to the predicted category and the category label.

[0160] Among them, the category label of the sample character sequence is the true category of the sample character sequence, the classification loss is determined according to the category label and the predicted category predicted by the model, and the model parameters of the multimodal feature replacement model are adjusted based on this classification loss.

[0161] See Figure 14 , Figure 14 This is a schematic diagram of a classification task provided by an embodiment of the present application. The character sequence in the document is converted into character features and input into a multimodal feature extraction model. Assume that the multimodal feature extraction model requires input of features of three modalities (character features, image features, and position features). Since there are only features of the character modality, the features of the other two modalities are set to empty. The multimodal feature extraction model extracts the multimodal structured features of each character, and the category discrimination model identifies the multimodal structured features to obtain a predicted document category C. The multimodal feature extraction model is trained based on the predicted document category C and the actual document category.

[0162] From the above, it can be seen that the multimodal feature extraction model in this application is trained based on the pre-training task. The multimodal feature extraction model can be trained by fine-tuning a small number of samples in the sequence labeling scenario of structured categories. This can reduce the sample requirement in the sequence labeling scenario and improve the robustness of the multimodal feature extraction model.

[0163] For further information, see Figure 15 , which is a structural diagram of an image processing device provided by an embodiment of the present application. Figure 15 As shown, the image processing device 1 can be applied to the above Figure 3-Figure 14 The terminal device in the corresponding embodiment. Specifically, the image processing device 1 can be a computer program (including program code) running in a computer device, for example, the image processing device 1 is an application software; the image processing device 1 can be used to execute the corresponding steps of the method provided in the embodiment of the present application.

[0164] The image processing device 1 may include: an acquisition module 11 , a recognition module 12 and a creation module 13 .

[0165] An acquisition module 11 is configured to acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence; the character sequence is a sequence composed of characters in the image to be recognized; the image sequence is a sequence composed of slices of each character in the character sequence in the image to be recognized; and the position sequence is a sequence composed of the positions of each character in the character sequence in the image to be recognized;

[0166] an identification module 12, configured to perform sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence;

[0167] The creation module 13 is configured to create a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence.

[0168] In a possible implementation, when the data sequence includes multiple characters, images, and positions, the recognition module 12 is configured to perform sequence labeling on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category for each data in the data sequence, specifically:

[0169] Converting multiple sequences contained in the data sequence into feature sequences respectively, and aligning and splicing the multiple feature sequences into a multimodal feature sequence of the data sequence;

[0170] Calling a multimodal feature extraction model to encode the multimodal feature sequence to obtain a multimodal structured feature sequence;

[0171] A structured category recognition model is called to decode the multimodal structured feature sequence to obtain a structured category of each data in the data sequence.

[0172] In a possible implementation, the multimodal feature extraction model is a feature extraction model trained based on a pre-training task, wherein the pre-training task includes one or more of a replacement task, a matching task, and a classification task;

[0173] The replacement task is to perform data replacement processing on a first modality sample sequence and predict the replacement result of each sample in the first modality sample sequence after the processing based on a second modality sample sequence, wherein the first modality sample sequence and the second modality sample sequence are two different sequences among a sample image sequence, a sample character sequence, and a sample position sequence, and the sample character sequence is a first modality sample sequence or a second modality sample sequence;

[0174] The matching task is to predict the contextual relationship between a sample image subsequence and a sample character subsequence, wherein the sample image subsequence is a subsequence of the sample image sequence, and the sample character subsequence is a subsequence of the sample character sequence;

[0175] The classification task is to predict the document category of the sample character sequence, and there is a corresponding relationship between the sample image sequence, the sample character sequence and the sample position sequence.

[0176] In a possible implementation, the image processing device 1 may further include: a first training module 14 .

[0177] A first training module 14 is configured to obtain a first modality sample sequence and a second modality sample sequence, and perform a replacement process on the first modality sample sequence to obtain a replacement sample sequence;

[0178] The first training module 14 is further configured to convert the replacement sample sequence into a replacement feature sequence, and convert the second modality sample sequence into a second modality sample feature sequence, and align and splice the replacement feature sequence and the second modality sample feature sequence into a first sample multimodal feature sequence;

[0179] The first training module 14 is further configured to call the multimodal feature extraction model to encode the first sample multimodal feature sequence to obtain a first sample structured feature sequence;

[0180] The first training module 14 is further configured to call a replacement discrimination model to decode the first sample structured feature sequence to obtain a predicted replacement result sequence;

[0181] The first training module 14 is further configured to obtain a replacement result label sequence of the replacement sample sequence, and train the multimodal feature extraction model according to the replacement result label sequence and the predicted replacement result sequence.

[0182] In a possible implementation, the image processing module may further include: a second training module 15 .

[0183] The second training module 15 is configured to obtain a sample image sequence and extract a sample image subsequence from the sample image sequence;

[0184] The second training module 15 is further configured to obtain a sample character sequence and extract a sample character subsequence from the sample character sequence;

[0185] The second training module 15 is further configured to convert the sample image subsequence into a sample image feature subsequence, and to align and splice the sample image feature subsequence and the sample character feature subsequence of the sample character subsequence into a second sample multimodal feature sequence;

[0186] The second training module 15 is further configured to call the multimodal feature extraction model to encode the second sample multimodal feature sequence to obtain a second sample structured feature sequence;

[0187] The second training module 15 is further configured to call a relationship discrimination model to decode the second sample structured feature sequence to obtain a predicted contextual relationship;

[0188] The second training module 15 is further configured to obtain a contextual relationship label between the sample image subsequence and the sample character subsequence, and train the multimodal feature extraction model based on the predicted contextual relationship and the contextual relationship label.

[0189] In a possible implementation, the image processing device 1 may further include: a third training module 16 .

[0190] The third training module 16 is used to obtain a sample character sequence and convert the sample character sequence into a sample character feature sequence;

[0191] The third training module 16 is further configured to call the multimodal feature extraction model to encode the sample character feature sequence to obtain a third sample structured feature sequence;

[0192] The third training module 16 is further configured to call a type discrimination model to decode the third sample structured feature sequence to obtain a predicted category;

[0193] The third training module 16 is further configured to obtain a category label of the sample character sequence and train the multimodal feature extraction model according to the predicted category and the category label.

[0194] In one possible implementation, the structured category includes a key structure category and a value structure category. When the data sequence includes at least a character sequence, the creation module 13, when used to create a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence, is specifically configured to:

[0195] Combining characters belonging to the key structure category into key fields, and combining characters belonging to the value structure category into value fields, wherein the number of the key fields and the number of the value fields are both N, where N is an integer greater than 1;

[0196] Pair the N key fields with the N value fields to obtain N key-value pairs, and create the structured document containing the N key-value pairs.

[0197] In one possible implementation, the bond structure categories include a bond start category and a bond end category;

[0198] When the creation module 13 is used to combine characters belonging to the key structure category into a key field, it is specifically used to:

[0199] According to the structured category of each character, the character sequence is divided into N key fields, and a key field includes a key start category and a key end category.

[0200] In a possible implementation, when the creation module 13 is used to pair N key fields with N value fields to obtain N key-value pairs, it is specifically used to:

[0201] Constructing a topological graph based on N key fields and N value fields, wherein the topological graph includes 2N nodes, one node corresponds to one key field or one value field, a connection edge weight between a first node and a second node is determined based on a distance between the first node and the second node in the image to be recognized, the first node and the second node are two different nodes among the 2N nodes, each node corresponds to a node feature, and the node feature of the first node is a feature obtained by superimposing multimodal structured features of all characters contained in the field corresponding to the first node;

[0202] Calling a graph neural network model to perform matching processing on the topological graph to obtain a relationship category between the first node and the second node; the relationship category includes a matching relationship category;

[0203] If the relationship category between the first node and the second node is a matching relationship category, the key field and the value field corresponding to the first node and the second node are combined into a key-value pair.

[0204] In a possible implementation, when the creation module 13 is used to pair N key fields with N value fields to obtain N key-value pairs, it is specifically used to:

[0205] Divide N key fields and N value fields into N 2 field combinations, each field combination includes a key field and a value field;

[0206] Obtaining the key field feature of the key field in the target field combination, and obtaining the value field feature of the value field in the target field combination, wherein the target field combination is any field combination of the N2 field combinations, the key field feature is a feature obtained by superimposing the multimodal structured features of all characters contained in the key field in the target field combination, and the value field feature is a feature obtained by superimposing the multimodal structured features of all characters contained in the value field in the target field combination;

[0207] The key field features and the value field features are combined into the combined features of the target field combination, and the classification model is called to classify the combined features to obtain the matching probability of the target field combination;

[0208] If the matching probability is greater than the probability threshold, the target field combination is used as a key-value pair.

[0209] In a possible implementation, when the data sequence includes a character sequence, an image sequence, and a position sequence, and the number of the character sequences, the number of the image sequences, and the number of the position sequences are all K, K is a positive integer, the acquisition module 11, when performing character recognition processing on the image to be recognized and obtaining a recognition result, is specifically configured to:

[0210] Performing object detection on the image to be recognized to obtain an original text and an original position sequence, wherein the original text is a text composed of all characters in the image to be recognized, and the original position sequence is a sequence composed of positions of all characters in the image to be recognized in the image to be recognized;

[0211] Extracting slices corresponding to the original position sequence from the image to be identified, and combining the extracted slices into an original image sequence;

[0212] Divide the original text into K character sequences, where each character sequence corresponds to a paragraph;

[0213] The original image sequence is divided into K image sequences according to K character sequences, and the original position sequence is divided into K position sequences.

[0214] According to an embodiment of the present invention, Figure 3-Figure 14 Each step involved in the method shown can be performed by Figure 15 The image processing apparatus shown in FIG. 1 is executed by each module. For example, Figure 3 The steps S101-S103 shown in FIG can be respectively Figure 15 The acquisition module 11, the identification module 12 and the creation module 13 shown in are executed; for example, Figure 9 Steps S201 to S205 shown in FIG can be replaced by Figure 15 The first training module 14 shown in FIG. Figure 11 Steps S301 to S305 shown in FIG can be replaced by Figure 15 The second training module 15 shown in FIG. Figure 13 Steps S401 to S404 shown in FIG can be replaced by Figure 15 The third training module 15 shown in FIG.

[0215] Further, see Figure 16 , is a structural diagram of a computer device provided in an embodiment of the present application. Figure 3-Figure 14 The terminal device in the corresponding embodiment may be a computer device 1000. Figure 16As shown, the computer device 1000 may include: a user interface 1002, a processor 1004, an encoder 1006 and a memory 1008. A signal receiver 1016 is used to receive or send data via a cellular interface 1010, a WIFI interface 1012, ..., or an NFC interface 1014. The encoder 1006 encodes the received data into a data format for computer processing. A computer program is stored in the memory 1008, and the processor 1004 is configured to perform the steps of any of the above method embodiments through the computer program. The memory 1008 may include a volatile memory (e.g., a dynamic random access memory DRAM) and may also include a non-volatile memory (e.g., a one-time programmable read-only memory OTPROM). In some instances, the memory 1008 may further include a memory remotely located relative to the processor 1004, and these remote memories may be connected to the computer device 1000 via a network. The user interface 1002 may include: a keyboard 1018 and a display 1020.

[0216] exist Figure 16 In the computer device 1000 shown, the processor 1004 can be used to call the computer program stored in the memory 1008 to implement:

[0217] Acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence, the character sequence being a sequence composed of characters in the image to be recognized, the image sequence being a sequence composed of slices of each character in the character sequence in the image to be recognized, and the position sequence being a sequence composed of the positions of each character in the character sequence in the image to be recognized;

[0218] performing sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence;

[0219] A structured document corresponding to the image to be recognized is created according to the data sequence and the structured category of each data in the data sequence.

[0220] In one embodiment, when the data sequence includes multiple characters, images, and positions, the processor 1004 performs the following steps when performing sequence labeling on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category for each data in the data sequence:

[0221] Converting multiple sequences contained in the data sequence into feature sequences respectively, and aligning and splicing the multiple feature sequences into a multimodal feature sequence of the data sequence;

[0222] Calling a multimodal feature extraction model to encode the multimodal feature sequence to obtain a multimodal structured feature sequence;

[0223] A structured category recognition model is called to decode the multimodal structured feature sequence to obtain a structured category of each data in the data sequence.

[0224] In one embodiment, the multimodal feature extraction model is a feature extraction model trained based on a pre-training task, wherein the pre-training task includes one or more of a replacement task, a matching task, and a classification task;

[0225] The replacement task is to perform data replacement processing on a first modality sample sequence and predict the replacement result of each sample in the first modality sample sequence after the processing based on a second modality sample sequence, wherein the first modality sample sequence and the second modality sample sequence are two different sequences among a sample image sequence, a sample character sequence, and a sample position sequence, and the sample character sequence is a first modality sample sequence or a second modality sample sequence;

[0226] The matching task is to predict the contextual relationship between a sample image subsequence and a sample character subsequence, wherein the sample image subsequence is a subsequence of the sample image sequence, and the sample character subsequence is a subsequence of the sample character sequence;

[0227] The classification task is to predict the document category of the sample character sequence, and there is a corresponding relationship between the sample image sequence, the sample character sequence and the sample position sequence.

[0228] In one embodiment, the processor 1004 further performs the following steps:

[0229] Acquire a first modal sample sequence and a second modal sample sequence, and perform replacement processing on the first modal sample sequence to obtain a replacement sample sequence;

[0230] Converting the replacement sample sequence into a replacement feature sequence, and converting the second modal sample sequence into a second modal sample feature sequence, and aligning and splicing the replacement feature sequence and the second modal sample feature sequence into a first sample multimodal feature sequence;

[0231] Calling the multimodal feature extraction model to encode the first sample multimodal feature sequence to obtain a first sample structured feature sequence;

[0232] Calling the replacement discrimination model to decode the first sample structured feature sequence to obtain a predicted replacement result sequence;

[0233] A replacement result label sequence of the replacement sample sequence is obtained, and the multimodal feature extraction model is trained according to the replacement result label sequence and the predicted replacement result sequence.

[0234] In one embodiment, the processor 1004 further performs the following steps:

[0235] Acquire a sample image sequence, and extract a sample image subsequence from the sample image sequence;

[0236] Obtaining a sample character sequence, and extracting a sample character subsequence from the sample character sequence;

[0237] Converting the sample image subsequence into a sample image feature subsequence, and aligning and splicing the sample image feature subsequence and the sample character feature subsequence into a second sample multimodal feature sequence;

[0238] Calling the multimodal feature extraction model to encode the second sample multimodal feature sequence to obtain a second sample structured feature sequence;

[0239] Calling the relationship discrimination model to decode the second sample structured feature sequence to obtain a predicted contextual relationship;

[0240] A contextual relationship label between the sample image subsequence and the sample character subsequence is obtained, and the multimodal feature extraction model is trained according to the predicted contextual relationship and the contextual relationship label.

[0241] In one embodiment, the processor 1004 further performs the following steps:

[0242] Obtaining a sample character sequence, and converting the sample character sequence into a sample character feature sequence;

[0243] Calling the multimodal feature extraction model to encode the sample character feature sequence to obtain a third sample structured feature sequence;

[0244] Calling a type discrimination model to decode the structured feature sequence of the third sample to obtain a predicted category;

[0245] Obtain a category label of the sample character sequence, and train the multimodal feature extraction model based on the predicted category and the category label.

[0246] In one embodiment, the structured category includes a key structure category and a value structure category. When the data sequence includes at least a character sequence, the processor 1004, when creating a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence, specifically performs the following steps:

[0247] Combining characters belonging to the key structure category into key fields, and combining characters belonging to the value structure category into value fields, wherein the number of the key fields and the number of the value fields are both N, where N is an integer greater than 1;

[0248] Pair the N key fields with the N value fields to obtain N key-value pairs, and create the structured document containing the N key-value pairs.

[0249] In one embodiment, the bond structure categories include a bond start category and a bond end category;

[0250] When the processor 1004 combines the characters belonging to the key structure category into a key field, it specifically performs the following steps:

[0251] According to the structured category of each character, the character sequence is divided into N key fields, and a key field includes a key start category and a key end category.

[0252] In one embodiment, when the processor 1004 performs pairing of N key fields and N value fields to obtain N key-value pairs, the processor 1004 specifically performs the following steps:

[0253] Constructing a topological graph based on N key fields and N value fields, wherein the topological graph includes 2N nodes, one node corresponds to one key field or one value field, a connection edge weight between a first node and a second node is determined based on a distance between the first node and the second node in the image to be recognized, the first node and the second node are two different nodes among the 2N nodes, each node corresponds to a node feature, and the node feature of the first node is a feature obtained by superimposing multimodal structured features of all characters contained in the field corresponding to the first node;

[0254] Calling a graph neural network model to perform matching processing on the topological graph to obtain a relationship category between the first node and the second node; the relationship category includes a matching relationship category;

[0255] If the relationship category between the first node and the second node is a matching relationship category, the key field and the value field corresponding to the first node and the second node are combined into a key-value pair.

[0256] In one embodiment, when the processor 1004 performs pairing of N key fields and N value fields to obtain N key-value pairs, the processor 1004 specifically performs the following steps:

[0257] Divide N key fields and N value fields into N 2 field combinations, each field combination includes a key field and a value field;

[0258] Obtaining the key field feature of the key field in the target field combination, and obtaining the value field feature of the value field in the target field combination, wherein the target field combination is any field combination of the N2 field combinations, the key field feature is a feature obtained by superimposing the multimodal structured features of all characters contained in the key field in the target field combination, and the value field feature is a feature obtained by superimposing the multimodal structured features of all characters contained in the value field in the target field combination;

[0259] The key field features and the value field features are combined into the combined features of the target field combination, and the classification model is called to classify the combined features to obtain the matching probability of the target field combination;

[0260] If the matching probability is greater than the probability threshold, the target field combination is used as a key-value pair.

[0261] In one embodiment, when the data sequence includes a character sequence, an image sequence, and a position sequence, and the number of the character sequences, the number of the image sequences, and the number of the position sequences are all K, K is a positive integer, the processor 1004 performs the following steps when performing character recognition processing on the image to be recognized and obtaining a recognition result:

[0262] Performing object detection on the image to be recognized to obtain an original text and an original position sequence, wherein the original text is a text composed of all characters in the image to be recognized, and the original position sequence is a sequence composed of positions of all characters in the image to be recognized in the image to be recognized;

[0263] Extracting slices corresponding to the original position sequence from the image to be identified, and combining the extracted slices into an original image sequence;

[0264] Divide the original text into K character sequences, where each character sequence corresponds to a paragraph;

[0265] The original image sequence is divided into K image sequences according to K character sequences, and the original position sequence is divided into K position sequences.

[0266] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3-Figure 14The description of the image processing method in the corresponding embodiment can also be performed as described above. Figure 15 The description of the image processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0267] In addition, it should be noted that: the embodiment of the present application also provides a computer storage medium, and the computer storage medium stores the computer program executed by the image processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3-Figure 14 Therefore, the description of the image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device for execution, or executed on multiple computer devices located at one location, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can be combined into a blockchain network.

[0268] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the above-mentioned Figures 3 to 14 The method in the corresponding embodiment will therefore not be described in detail here.

[0269] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0270] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence, the character sequence being a sequence composed of characters in the image to be recognized, the image sequence being a sequence composed of slices of each character in the character sequence in the image to be recognized, and the position sequence being a sequence composed of the positions of each character in the character sequence in the image to be recognized; performing sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence; Creating a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence, wherein the structured document includes N key-value pairs, where the N key-value pairs are obtained by pairing N key fields with N value fields; the N key fields and the N value fields are paired based on a graph neural network model; Among them, the pairing process based on the graph neural network model includes: constructing a topological graph according to the N key fields and the N value fields, the topological graph includes 2N nodes, one node corresponds to a key field or a value field, the connection edge weight between the first node and the second node is determined according to the distance between the first node and the second node in the image to be identified, the first node and the second node are two different nodes among the 2N nodes, each node corresponds to a node feature, and the node feature of the first node is the feature after superimposing the multimodal structured features of all characters contained in the field corresponding to the first node; calling the graph neural network model to perform matching processing on the topological graph to obtain the relationship category between the first node and the second node; the relationship category includes a matching relationship category; if the relationship category between the first node and the second node is a matching relationship category, the key field and value field corresponding to the first node and the second node are combined into a key-value pair.

2. The method according to claim 1, characterized in that When the data sequence includes multiple of a character sequence, an image sequence, and a position sequence, performing sequence labeling on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence includes: Converting multiple sequences contained in the data sequence into feature sequences respectively, and aligning and splicing the multiple feature sequences into a multimodal feature sequence of the data sequence; Calling a multimodal feature extraction model to encode the multimodal feature sequence to obtain a multimodal structured feature sequence; A structured category recognition model is called to decode the multimodal structured feature sequence to obtain a structured category of each data in the data sequence.

3. The method according to claim 2, characterized in that The multimodal feature extraction model is a feature extraction model trained based on a pre-training task, wherein the pre-training task includes one or more of a replacement task, a matching task, and a classification task; The replacement task is to perform data replacement processing on a first modality sample sequence and predict the replacement result of each sample in the first modality sample sequence after the processing based on a second modality sample sequence, wherein the first modality sample sequence and the second modality sample sequence are two different sequences among a sample image sequence, a sample character sequence, and a sample position sequence, and the sample character sequence is a first modality sample sequence or a second modality sample sequence; The matching task is to predict the contextual relationship between a sample image subsequence and a sample character subsequence, wherein the sample image subsequence is a subsequence of the sample image sequence, and the sample character subsequence is a subsequence of the sample character sequence; The classification task is to predict the document category of the sample character sequence, and there is a corresponding relationship between the sample image sequence, the sample character sequence and the sample position sequence.

4. The method according to claim 3, characterized in that The method further comprises: Acquire a first modal sample sequence and a second modal sample sequence, and perform replacement processing on the first modal sample sequence to obtain a replacement sample sequence; Converting the replacement sample sequence into a replacement feature sequence, and converting the second modal sample sequence into a second modal sample feature sequence, and aligning and splicing the replacement feature sequence and the second modal sample feature sequence into a first sample multimodal feature sequence; Calling the multimodal feature extraction model to encode the first sample multimodal feature sequence to obtain a first sample structured feature sequence; Calling the replacement discrimination model to decode the first sample structured feature sequence to obtain a predicted replacement result sequence; A replacement result label sequence of the replacement sample sequence is obtained, and the multimodal feature extraction model is trained according to the replacement result label sequence and the predicted replacement result sequence.

5. The method according to claim 3, characterized in that The method further comprises: Acquire a sample image sequence, and extract a sample image subsequence from the sample image sequence; Obtaining a sample character sequence, and extracting a sample character subsequence from the sample character sequence; Converting the sample image subsequence into a sample image feature subsequence, and aligning and splicing the sample image feature subsequence and the sample character feature subsequence into a second sample multimodal feature sequence; Calling the multimodal feature extraction model to encode the second sample multimodal feature sequence to obtain a second sample structured feature sequence; Calling the relationship discrimination model to decode the second sample structured feature sequence to obtain a predicted contextual relationship; A contextual relationship label between the sample image subsequence and the sample character subsequence is obtained, and the multimodal feature extraction model is trained according to the predicted contextual relationship and the contextual relationship label.

6. The method according to claim 3, characterized in that The method further comprises: Obtaining a sample character sequence, and converting the sample character sequence into a sample character feature sequence; Calling the multimodal feature extraction model to encode the sample character feature sequence to obtain a third sample structured feature sequence; Calling a type discrimination model to decode the structured feature sequence of the third sample to obtain a predicted category; Obtain a category label of the sample character sequence, and train the multimodal feature extraction model based on the predicted category and the category label.

7. The method according to claim 2, characterized in that The structured category includes a key structure category and a value structure category. When the data sequence includes at least a character sequence, creating a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence includes: Combining characters belonging to the key structure category into key fields, and combining characters belonging to the value structure category into value fields, wherein the number of the key fields and the number of the value fields are both N, where N is an integer greater than 1; Pair the N key fields with the N value fields to obtain N key-value pairs, and create the structured document containing the N key-value pairs.

8. The method according to claim 7, characterized in that The bond structure categories include a bond start category and a bond end category; Combining characters belonging to the key structure category into a key field includes: According to the structured category of each character, the character sequence is divided into N key fields, and a key field includes a key start category and a key end category.

9. The method according to claim 1, characterized in that When the data sequence includes a character sequence, an image sequence, and a position sequence, and the number of the character sequences, the number of the image sequences, and the number of the position sequences are all K, K is a positive integer, performing character recognition processing on the image to be recognized to obtain a recognition result includes: Performing object detection on the image to be recognized to obtain an original text and an original position sequence, wherein the original text is a text composed of all characters in the image to be recognized, and the original position sequence is a sequence composed of positions of all characters in the image to be recognized in the image to be recognized; Extracting slices corresponding to the original position sequence from the image to be identified, and combining the extracted slices into an original image sequence; Divide the original text into K character sequences, where each character sequence corresponds to a paragraph; The original image sequence is divided into K image sequences according to K character sequences, and the original position sequence is divided into K position sequences.

10. An image processing device, characterized in that: include: an acquisition module, configured to acquire an image to be recognized, perform character recognition processing on the image to be recognized, and obtain a recognition result; wherein the recognition result includes a recognized data sequence, the data sequence including one or more of a character sequence, an image sequence, and a position sequence; the character sequence is a sequence composed of characters in the image to be recognized; the image sequence is a sequence composed of slices of each character in the character sequence in the image to be recognized; and the position sequence is a sequence composed of the positions of each character in the character sequence in the image to be recognized; an identification module, configured to perform sequence labeling processing on the data sequence based on the multimodal feature sequence of the data sequence to obtain a structured category of each data in the data sequence; a creation module, configured to create a structured document corresponding to the image to be recognized based on the data sequence and the structured category of each data in the data sequence, wherein the structured document includes N key-value pairs, where the N key-value pairs are obtained by pairing N key fields with N value fields; the N key fields and the N value fields are paired based on a graph neural network model; Among them, the pairing process based on the graph neural network model includes: constructing a topological graph according to the N key fields and the N value fields, the topological graph includes 2N nodes, one node corresponds to a key field or a value field, the connection edge weight between the first node and the second node is determined according to the distance between the first node and the second node in the image to be identified, the first node and the second node are two different nodes among the 2N nodes, each node corresponds to a node feature, and the node feature of the first node is the feature after superimposing the multimodal structured features of all characters contained in the field corresponding to the first node; calling the graph neural network model to perform matching processing on the topological graph to obtain the relationship category between the first node and the second node; the relationship category includes a matching relationship category; if the relationship category between the first node and the second node is a matching relationship category, the key field and value field corresponding to the first node and the second node are combined into a key-value pair.

11. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

12. A computer storage medium, characterized in that The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, a computer device having the processor executes the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images