Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium, and the method comprises the steps: obtaining a to-be-processed image, and determining a code element sequence corresponding to at least one text line in the to-be-processed image; predicting a category corresponding to each code element in the code element sequence, and determining a text line feature corresponding to at least one text line in the to-be-processed image according to the category; and according to the text line features, extracting an entity in the to-be-processed image and an entity category of the entity. Therefore, the text line is detected to obtain the text line detection result by using the existing character detection and recognition engine, the text line feature corresponding to at least one text line in the to-be-processed image is determined under the text line detection result, and then the entity and the entity category of the entity in the to-be-processed image are extracted according to the text line feature. And a complete entity can be extracted.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] The current method for extracting entities in document images often uses a text detection and recognition engine to detect the position of a text line in a document image, identify the text line at the text line position, and determine the entity in the document image based on the code element sequence obtained from the identified text line.

[0003] However, the current entity extraction method in document images has certain limitations. It requires that the extracted entities must be continuous and complete in the code element sequence in order to extract the complete entities in the document image. Summary of the invention

[0004] In order to solve the technical problem that the above-mentioned current method for extracting entities from document images has certain limitations and requires that the extracted entities must be continuous and complete in the code element sequence in order to extract the complete entities from the document image, the embodiment of the present application provides an image processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0006] Acquire an image to be processed, and determine a code element sequence corresponding to at least one text line in the image to be processed;

[0007] Predicting a category corresponding to each codeword in the codeword sequence, and determining text line features corresponding to at least one text line in the image to be processed according to the category;

[0008] Entities in the image to be processed and entity categories of the entities are extracted according to the text line features.

[0009] In an optional implementation, determining the code element sequence corresponding to at least one text line in the image to be processed includes:

[0010] Detecting at least one text line position in the image to be processed, and for any of the text line positions, identifying a text line at the text line position;

[0011] Splitting the text line by a word segmenter to obtain a code element subsequence corresponding to the text line;

[0012] All the code element subsequences are concatenated to obtain a code element sequence corresponding to at least one text line in the image to be processed.

[0013] In an optional implementation, predicting the category corresponding to each symbol in the symbol sequence includes:

[0014] Determine a code element coordinate sequence corresponding to the code element sequence;

[0015] Determining a multimodal code element feature sequence according to the code element sequence, the code element coordinate sequence and the image to be processed;

[0016] Wherein, the multimodal codeword features in the multimodal codeword feature sequence correspond one-to-one to the codewords in the codeword sequence;

[0017] According to the multimodal codeword feature sequence, the category corresponding to each codeword in the codeword sequence is predicted.

[0018] In an optional implementation, determining, according to the category, text line features corresponding to at least one text line in the image to be processed includes:

[0019] For any text line in the image to be processed, searching for a code element subsequence corresponding to the text line from the code element sequence;

[0020] According to the category, selecting a first codeword from the codeword subsequence, and selecting a target multimodal codeword feature corresponding to the first codeword from the multimodal codeword feature sequence;

[0021] Determine a text line feature corresponding to the text line according to the target multimodal code element feature.

[0022] In an optional implementation, the selecting the first symbol from the symbol subsequence according to the category comprises:

[0023] From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity;

[0024] The step of selecting a target multimodal symbol feature corresponding to the first symbol from the multimodal symbol feature sequence includes:

[0025] Selecting the multimodal code element feature corresponding to the starting code element of the entity from the multimodal code element feature sequence;

[0026] The determining, according to the target multimodal code element feature, a text line feature corresponding to the text line comprises:

[0027] The multimodal code element feature corresponding to the start code element of the entity is determined as the text line feature corresponding to the text line.

[0028] In an optional implementation, the selecting the first symbol from the symbol subsequence according to the category comprises:

[0029] From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity;

[0030] From the codeword subsequence, screen the codewords of the end category, and determine the codewords of the end category as the end codewords of the entity;

[0031] From the codeword subsequence, select the intermediate codeword between the start codeword of the entity and the end codeword of the entity, and determine it as the intermediate codeword of the entity;

[0032] The step of selecting a target multimodal symbol feature corresponding to the first symbol from the multimodal symbol feature sequence includes:

[0033] Selecting a first multimodal code element feature corresponding to a start code element of the entity from the multimodal code element feature sequence;

[0034] Selecting a second multimodal symbol feature corresponding to the intermediate symbol of the entity from the multimodal symbol feature sequence;

[0035] Selecting a third multimodal code element feature corresponding to the end code element of the entity from the multimodal code element feature sequence;

[0036] The determining, according to the target multimodal code element feature, a text line feature corresponding to the text line comprises:

[0037] Determine a text line feature corresponding to the text line according to the first multimodal code element feature, the second multimodal code element feature, and the third multimodal code element feature.

[0038] In an optional implementation, determining the text line feature corresponding to the text line according to the first multimodal code element feature, the second multimodal code element feature, and the third multimodal code element feature includes:

[0039] Determining an average multimodal symbol feature of the first multimodal symbol feature, the second multimodal symbol feature, and the third multimodal symbol feature;

[0040] The average multimodal code element feature is determined as the text line feature corresponding to the text line.

[0041] In an optional implementation, when the image to be processed includes a plurality of text lines, extracting entities and entity categories of the entities in the image to be processed according to the text line features includes:

[0042] Performing concatenation processing on the text line features corresponding to each of the plurality of text lines to obtain a text line feature sequence;

[0043] Aggregating the plurality of text lines according to the text line feature sequence to obtain a text line aggregation result;

[0044] Predicting entity line categories corresponding to each of the plurality of text lines according to the text line feature sequence;

[0045] Entities in the image to be processed and entity categories of the entities are extracted according to the categories, the text line aggregation results, and the entity line categories.

[0046] In an optional implementation, aggregating the plurality of text lines according to the text line feature sequence to obtain a text line aggregation result includes:

[0047] For a first text line feature in the text line feature sequence, determining any of the remaining text line features in the text line feature sequence;

[0048] Concatenating the first text line feature with any of the remaining text line features to obtain a paired text line feature;

[0049] Determining a relative position feature between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature;

[0050] Adding the paired text line feature to the relative position feature to obtain a fused text line feature;

[0051] Predicting, based on the fused text line feature, a pairing result between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature;

[0052] According to the pairing result, the plurality of text lines are aggregated to obtain a text line aggregation result.

[0053] In an optional embodiment, the relative position feature is multiple;

[0054] The step of adding the paired text line feature to the relative position feature to obtain a fused text line feature includes:

[0055] splicing a plurality of the relative position features to obtain a target relative position feature, and deforming the target relative position feature;

[0056] Wherein, the size of the deformed target relative position feature is the same as the size of the paired text line feature;

[0057] The paired text line feature is added to the deformed target relative position feature to obtain a fused text line feature.

[0058] In an optional implementation, predicting, based on the fused text line feature, a pairing result between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature includes:

[0059] Inputting the fused text line features into a first linear classifier to obtain a text line aggregation category;

[0060] In a case where the text line aggregation category is a preset text line aggregation category, determining that a text line corresponding to the first text line feature is successfully paired with a text line corresponding to any other text line feature;

[0061] In a case where the text line aggregation category is not a preset text line aggregation category, it is determined that the text line corresponding to the first text line feature fails to be paired with the text line corresponding to any other text line feature.

[0062] In an optional implementation, predicting the entity line categories corresponding to each of the plurality of text lines according to the text line feature sequence includes:

[0063] Inputting the text line feature sequence into a second linear classifier to obtain an entity line category sequence;

[0064] Wherein, the entity line categories in the entity line category sequence correspond one-to-one to the text line features in the text line feature sequence;

[0065] For any of the text lines, searching for a text line feature corresponding to the text line from the text line feature sequence;

[0066] The entity row category corresponding to the text row feature is searched from the entity row category sequence, and is determined as the entity row category corresponding to the text row.

[0067] In an optional implementation, extracting entities and entity categories of the entities in the image to be processed according to the categories, the text line aggregation results, and the entity line categories includes:

[0068] For any of the text lines, searching the code element subsequence corresponding to the text line from the code element sequence;

[0069] According to the category, selecting a second codeword from the codeword subsequence, and sequentially splicing the second codewords to obtain an initial entity corresponding to the text line;

[0070] Entities in the image to be processed and entity categories of the entities are extracted according to the initial entities, the text line aggregation results, and the entity line categories.

[0071] In an optional implementation, extracting entities and entity categories of the entities in the image to be processed according to the initial entity, the text line aggregation result, and the entity line category includes:

[0072] Determine at least one text row set according to the text row aggregation result, wherein the text row set includes one or more text rows;

[0073] For any of the text line sets, when the text line set contains one text line, the following processing is performed:

[0074] Determine the initial entity corresponding to the text line contained in the text line set as the first entity in the image to be processed;

[0075] Determine the entity row category corresponding to the text row included in the text row set as the entity category of the first entity;

[0076] For any of the text line sets, when the text line set contains multiple text lines, the following processing is performed:

[0077] Sequentially concatenate the initial entities corresponding to the plurality of text lines contained in the text line set to obtain a second entity in the image to be processed;

[0078] Determine an entity row category with the highest occurrence frequency among the entity row categories corresponding to the plurality of text rows included in the text row set;

[0079] The entity row category with the highest occurrence frequency is determined as the entity category of the second entity.

[0080] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0081] An image acquisition module, used for acquiring an image to be processed;

[0082] A code element sequence determination module, used to determine a code element sequence corresponding to at least one text line in the image to be processed;

[0083] A category prediction module, used to predict the category corresponding to each codeword in the codeword sequence;

[0084] A text line feature determination module, used to determine text line features corresponding to at least one text line in the image to be processed according to the category;

[0085] The entity and entity category extraction module is used to extract the entities and entity categories of the entities in the image to be processed according to the text line features.

[0086] In a third aspect of the embodiments of the present application, there is further provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0087] Memory, used to store computer programs;

[0088] The processor is used to implement any image processing method described in the first aspect when executing the program stored in the memory.

[0089] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image processing method described in any one of the first aspects above.

[0090] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0091] The technical solution provided by the embodiment of the present application obtains an image to be processed, determines a codeword sequence corresponding to at least one text line in the image to be processed, predicts the category corresponding to each codeword in the codeword sequence, and determines text line features corresponding to at least one text line in the image to be processed based on the category, and extracts entities and entity categories of entities in the image to be processed based on the text line features.

[0092] By determining the code element sequence corresponding to at least one text line in the image to be processed and predicting the category corresponding to each code element in the code element sequence, the text line features corresponding to at least one text line in the image to be processed are determined according to the category, and then the entities in the image to be processed and the entity categories of the entities are extracted according to the text line features. In this way, the text lines are detected using the existing text detection and recognition engine to obtain the text line detection results. Based on the text line detection results, the text line features corresponding to at least one text line in the image to be processed are determined, and then the entities in the image to be processed and the entity categories of the entities are extracted according to the text line features, so that the complete entities can be extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0094] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0095] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0096] Figure 1 A schematic diagram of an implementation flow of an image processing method shown in an embodiment of the present application;

[0097] Figure 2 A schematic diagram of an image to be processed shown in an embodiment of the present application;

[0098] Figure 3 It is a schematic diagram of an implementation flow of another image processing method shown in an embodiment of the present application;

[0099] Figure 4 A schematic diagram of a text line detection result shown in an embodiment of the present application;

[0100] Figure 5 A schematic diagram of an entity extraction result shown in an embodiment of the present application;

[0101] Figure 6 A schematic diagram of the structure of an image processing device shown in an embodiment of the present application;

[0102] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0104] The disclosure below provides many different embodiments or examples to implement the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0105] At present, the entity extraction method in document images has certain limitations. It requires that the extracted entities must be continuous and complete in the codeword sequence in order to extract the complete entity in the document image. However, this is difficult to achieve in reality because the entities to be extracted in the document image may span multiple lines of text, and for the text detection and recognition engine, the detection and recognition results of a single line of text are usually given. In this way, the entities to be extracted in the document image are scattered across multiple text lines. The text lines are divided into codeword sequences by the word segmenter, which cannot guarantee that the entities required to be extracted are continuous and complete in the codeword sequence, so the complete entities in the document image cannot be extracted.

[0106] To this end, the embodiment of the present application determines the code element sequence corresponding to multiple text lines in the image to be processed, and predicts the category corresponding to each code element in the code element sequence, and determines the text line features corresponding to each of the multiple text lines in the image to be processed according to the category, and then extracts the entities and the entity categories of the entities in the image to be processed according to the text line features. In this way, the text lines are detected using the existing text detection and recognition engine to obtain text line detection results. Based on this text line detection result, the text line features corresponding to each of the multiple text lines in the image to be processed are determined, and then the entities and the entity categories of the entities in the image to be processed are extracted according to the text line features, so that the complete entities can be extracted.

[0107] Based on this, Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0108] S101, obtaining an image to be processed, and determining a code element sequence corresponding to at least one text line in the image to be processed.

[0109] In an embodiment of the present application, an image to be processed may be obtained, and the image to be processed may be, for example, a document image, wherein the document image contains at least one text line. Figure 2 shown.

[0110] For the image to be processed, a code element sequence corresponding to at least one text line in the image to be processed can be determined. The code element sequence consists of at least one code element. The code element can be, for example, a Chinese character or an English word, etc. This embodiment of the present application does not limit this.

[0111] S102, predicting the category corresponding to each codeword in the codeword sequence, and determining text line features corresponding to at least one text line in the image to be processed according to the category.

[0112] In the embodiment of the present application, for a symbol sequence, the category corresponding to each symbol in the symbol sequence can be predicted, wherein the category can be specifically divided into three categories: starting category, ending category, and other category.

[0113] For the start category, the corresponding code element represents the start code element of the entity, for the end category, the corresponding code element represents the end code element of the entity, and for other categories, the corresponding code element does not belong to any entity. The embodiments of the present application do not limit this.

[0114] For the categories corresponding to each code element in the code element sequence, the text line features corresponding to at least one text line in the image to be processed can be determined according to these categories, which means that each text line has a corresponding text line feature.

[0115] S103, extracting entities and entity categories of entities in the image to be processed according to the text line features.

[0116] In an embodiment of the present application, for text line features corresponding to at least one text line in the image to be processed, entities and entity categories of entities in the image to be processed can be extracted based on these text line features.

[0117] For entity categories, for example, it can be a name, date of birth, birthday, or ID number, etc., and the entity category varies depending on the image to be processed and needs to be determined based on business needs. The embodiments of the present application do not limit this.

[0118] In this way, the existing text detection and recognition engine is used to detect text lines and obtain text line detection results. Based on this text line detection result, the text line features corresponding to multiple text lines in the image to be processed are determined. Then, based on the text line features, the entities in the image to be processed and the entity categories of the entities are extracted, so that the complete entities can be extracted.

[0119] Through the above description of the technical solution provided in the embodiment of the present application, by determining the code element sequence corresponding to at least one text line in the image to be processed, and predicting the category corresponding to each code element in the code element sequence, the text line features corresponding to at least one text line in the image to be processed are determined according to the category, and then the entities and the entity categories of the entities in the image to be processed are extracted according to the text line features. In this way, the text lines are detected using the existing text detection and recognition engine to obtain the text line detection results. Under this text line detection result, the text line features corresponding to at least one text line in the image to be processed are determined, and then the entities and the entity categories of the entities in the image to be processed are extracted according to the text line features, so that the complete entities can be extracted.

[0120] like Figure 3 FIG. 1 is a schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application. The method is applied to an electronic device. When the image to be processed contains multiple text lines, the method may specifically include the following steps:

[0121] S301, obtaining an image to be processed, detecting multiple text line positions in the image to be processed, and identifying a text line at any text line position.

[0122] S302, splitting the text line through a word segmenter to obtain a code element subsequence corresponding to the text line.

[0123] S303, concatenating all code element subsequences to obtain code element sequences corresponding to multiple text lines in the image to be processed.

[0124] In an embodiment of the present application, an image to be processed is obtained, and the image to be processed may be, for example, a document image. For the image to be processed, multiple text line positions in the image to be processed are detected, where the text line position generally refers to the minimum circumscribed rectangular frame of the text line, which is usually embodied in the form of coordinates of four boundaries.

[0125] For any text line position in the image to be processed, for example Figure 4 The dotted box shown identifies the text line at the text line position, splits the text line through a word segmenter, and obtains a code element subsequence corresponding to the text line. In this way, the code element subsequence corresponding to each text line in the image to be processed can be obtained.

[0126] The above-mentioned code element subsequences may be concatenated, wherein the code element subsequences may be concatenated in the order of text lines, so that code element sequences corresponding to multiple text lines in the image to be processed may be obtained.

[0127] For example, if there are two lines of text in the image to be processed, the code element sequence can be specifically (t11, t12, t13, t21, t22, t23), where tij represents the jth code element split from the i-th line of text, t11, t12, and t13 constitute the code element subsequence corresponding to the first line of text, and t21, t22, and t23 constitute the code element subsequence corresponding to the second line of text. The two are concatenated in the order of the text lines to obtain the above code element sequence.

[0128] It should be noted that, for the word segmenter, different word segmenters are used to split the text line, and the code element subsequences corresponding to the obtained text line are different. For example, each code element in the code element subsequence can be a Chinese character or an English word, which is not limited in the embodiment of the present application.

[0129] S304, determining a codeword coordinate sequence corresponding to the codeword sequence, and determining a multimodal codeword feature sequence according to the codeword sequence, the codeword coordinate sequence and the image to be processed.

[0130] In the embodiment of the present application, for a symbol sequence, a symbol coordinate sequence corresponding to the symbol sequence can be determined. For any symbol in the symbol sequence, a text line corresponding to the symbol can be found, and the text line position of the text line can be placed at the position of the symbol in the symbol sequence to obtain a symbol coordinate sequence corresponding to the symbol sequence.

[0131] For example, for the code element sequence (t11, t12, t13, t21, t22, t23), the text line positions of the two text lines are B1 and B2 respectively. For the code element tij in the code element sequence, the text line corresponding to the code element can be found, and the text line position of the text line can be placed to the position of the code element in the code element sequence to obtain the code element coordinate sequence corresponding to the code element sequence, that is, (B1, B1, B1, B2, B2, B2).

[0132] After the above steps, a symbol sequence and a symbol coordinate sequence can be obtained, so that a multimodal symbol feature sequence can be determined according to the symbol sequence, the symbol coordinate sequence and the image to be processed, wherein the multimodal symbol features in the multimodal symbol feature sequence correspond one-to-one to the symbols in the symbol sequence.

[0133] Among them, the codeword sequence, the codeword coordinate sequence and the image to be processed can be input into the existing multimodal fusion backbone network to obtain a multimodal codeword feature sequence. For example, the multimodal fusion backbone network can be LiLT, LayoutLMv1, LayoutLMv2, LayoutLMv3, etc., which is not limited in the embodiment of the present application.

[0134] S305: predict the category corresponding to each codeword in the codeword sequence according to the multimodal codeword feature sequence.

[0135] In an embodiment of the present application, for a multimodal codeword feature sequence, the category corresponding to each codeword in the codeword sequence can be predicted based on the multimodal codeword feature sequence, and based on the category, the starting codeword of the entity constituting the codeword sequence can be known.

[0136] Specifically, a linear classifier can be set to classify the codewords into three categories: starting category, ending category, and other categories. To this end, the multimodal codeword feature sequence is input into the linear classifier to obtain the category corresponding to each codeword in the codeword sequence.

[0137] S306: for any text line in the image to be processed, search the codeword subsequence corresponding to the text line from the codeword sequence.

[0138] S307, selecting a first codeword from the codeword subsequence according to the category, and selecting a target multimodal codeword feature corresponding to the first codeword from the multimodal codeword feature sequence.

[0139] S308: Determine a text line feature corresponding to the text line according to the target multimodal code element feature.

[0140] In an embodiment of the present application, for any text line in the image to be processed, a codeword subsequence corresponding to the text line is searched from the codeword sequence, the first codeword is filtered from the codeword subsequence according to the category, and the target multimodal codeword feature corresponding to the first codeword is filtered from the multimodal codeword feature sequence, and the text line feature corresponding to the text line is determined based on the target multimodal codeword feature.

[0141] Among them, the multimodal code element features corresponding to the starting code element of the entity can be selected as the text line features corresponding to the text line, or the multimodal code element features corresponding to the starting code element, the middle code element, and the ending code element of the entity can be selected and then averaged as the text line features corresponding to the text line.

[0142] To this end, for any text line in the image to be processed, a codeword subsequence corresponding to the text line is searched from the codeword sequence, and codewords of the starting category are screened from the codeword subsequence, and the codewords of the starting category are determined as the starting codewords of the entity. The multimodal codeword features corresponding to the starting codewords of the entity are screened from the multimodal codeword feature sequence, and the multimodal codeword features corresponding to the starting codewords of the entity are determined as the text line features corresponding to the text line.

[0143] Alternatively, for any text line in the image to be processed, a codeword subsequence corresponding to the text line is searched from the codeword sequence, and from the codeword subsequence, codewords of the starting category are screened, and the codewords of the starting category are determined as the starting codewords of the entity, and from the codeword subsequence, codewords of the ending category are screened, and the codewords of the ending category are determined as the ending codewords of the entity, and from the codeword subsequence, intermediate codewords between the starting codeword of the entity and the ending codeword of the entity are screened and determined as the intermediate codewords of the entity.

[0144] A first multimodal code symbol feature corresponding to the start code symbol of an entity is selected from the multimodal code symbol feature sequence, a second multimodal code symbol feature corresponding to the middle code symbol of the entity is selected from the multimodal code symbol feature sequence, a third multimodal code symbol feature corresponding to the end code symbol of the entity is selected from the multimodal code symbol feature sequence, and a text line feature corresponding to the text line is determined according to the first multimodal code symbol feature, the second multimodal code symbol feature and the third multimodal code symbol feature.

[0145] Among them, for the first multimodal symbol feature, the second multimodal symbol feature and the third multimodal symbol feature, an average is calculated as the text line feature corresponding to the text line. Specifically, the average multimodal symbol feature of the first multimodal symbol feature, the second multimodal symbol feature and the third multimodal symbol feature is determined, and the average multimodal symbol feature is determined as the text line feature corresponding to the text line.

[0146] S309: concatenate text line features corresponding to the plurality of text lines to obtain a text line feature sequence.

[0147] In an embodiment of the present application, after the above steps, when there are multiple text lines in the image to be processed, the multiple text lines in the image to be processed each have their own corresponding text line features, and then the text line features corresponding to the multiple text lines can be spliced ​​to obtain a text line feature sequence.

[0148] Among them, the text line features corresponding to multiple text lines can be concatenated according to the order of the text lines to obtain a text line feature sequence for subsequent text line aggregation and text line classification.

[0149] S310, aggregating multiple text lines according to the text line feature sequence to obtain a text line aggregation result.

[0150] In the embodiment of the present application, for a text line feature sequence, multiple text lines in the image to be processed may be aggregated according to the text line feature sequence to obtain a text line aggregation result.

[0151] Among them, the text line features in the text line feature sequence can be paired in pairs and binary classified through a linear classifier to determine whether the two text lines corresponding to the paired text line features are the previous and next adjacent text lines in the same entity. If so, the subsequent two text lines should be aggregated together, and the entities corresponding to the two text lines should also be aggregated together, so that the complete entity can be extracted.

[0152] Based on this, for the first text line feature in the text line feature sequence (the first text line feature is any text line feature in the text line feature sequence), determine any remaining text line features in the text line feature sequence (any remaining text line features refer to any text line features in the text line feature sequence except the first text line feature), concatenate the first text line feature with any remaining text line features to obtain a paired text line feature, determine the relative position feature between the text line corresponding to the first text line feature and the text line corresponding to any remaining text line feature, add the paired text line feature and the relative position feature to obtain a fused text line feature, predict the pairing result between the text line corresponding to the first text line feature and the text line corresponding to any remaining text line feature based on the fused text line feature, and aggregate multiple text lines based on the pairing result to obtain a text line aggregation result.

[0153] It should be noted that, in the process of predicting the pairing results between the text line corresponding to the first text line feature and the text line corresponding to any other text line feature, introducing the relative position feature can significantly improve the prediction accuracy of the pairing results between the text line corresponding to the first text line feature and the text line corresponding to any other text line feature, which will help with text line aggregation later.

[0154] Among them, there are generally multiple relative position features, which can specifically be to determine the first minimum bounding rectangular box of the text line corresponding to the first text line feature, and the second minimum bounding rectangular box of the text line corresponding to any other text line feature, calculate the first X-axis coordinate difference between the X-axis coordinate of the center point of the first minimum bounding rectangular box and the X-axis coordinate of the center point of the second minimum bounding rectangular box, calculate the first Y-axis coordinate difference between the Y-axis coordinate of the center point of the first minimum bounding rectangular box and the Y-axis coordinate of the center point of the second minimum bounding rectangular box, and calculate the first length between the length of the first minimum bounding rectangular box and the length of the second minimum bounding rectangular box. ratio, calculating a first width ratio between the width of the first minimum circumscribed rectangular frame and the width of the second minimum circumscribed rectangular frame, calculating a second X-axis coordinate difference between the X-axis coordinate of the center point of the second minimum circumscribed rectangular frame and the X-axis coordinate of the center point of the first minimum circumscribed rectangular frame, calculating a second Y-axis coordinate difference between the Y-axis coordinate of the center point of the second minimum circumscribed rectangular frame and the Y-axis coordinate of the center point of the first minimum circumscribed rectangular frame, calculating a second length ratio between the length of the second minimum circumscribed rectangular frame and the length of the first minimum circumscribed rectangular frame, and calculating a second width ratio between the width of the second minimum circumscribed rectangular frame and the width of the first minimum circumscribed rectangular frame.

[0155] In this way, for the relative position feature, it can generally be the above-mentioned first X-axis coordinate difference, first Y-axis coordinate difference, first length ratio, first width ratio, second X-axis coordinate difference, second Y-axis coordinate difference, second length ratio, second width ratio, etc., and multiple relative position features are spliced ​​to obtain the target relative position feature, and the target relative position feature is deformed, wherein the target relative position is deformed through linear operation, and the size of the deformed target relative position feature is the same as the size of the paired text line feature, and the paired text line feature and the deformed target relative position feature are added to obtain the fused text line feature.

[0156] The fused text line feature is input into the first linear classifier to obtain a text line aggregation category. When the text line aggregation category is a preset text line aggregation category (for example, 1), it is determined that the text line corresponding to the first text line feature is successfully paired with the text line corresponding to any other text line feature, indicating that the text line corresponding to the first text line feature and the text line corresponding to any other text line feature should be aggregated together; when the text line aggregation category is not a preset text line aggregation category (for example, 0), it is determined that the text line corresponding to the first text line feature and the text line corresponding to any other text line feature fail to be paired, indicating that the text line corresponding to the first text line feature and the text line corresponding to any other text line feature do not need to be aggregated together.

[0157] After the above steps, the pairing results of the text line corresponding to the first text line feature in the text line feature sequence and the text line corresponding to any other text line feature in the text line feature sequence can be known. Therefore, based on the pairing results, multiple text lines can be aggregated to obtain the text line aggregation result, and the text line aggregation result can be reflected in the form of a text line set.

[0158] For example, assuming that there are text line 1, text line 2, and text line 3, after the above processing, text line 1 is successfully paired with text line 2, and text line 2 is successfully paired with text line 3, then text line 1, text line 2, and text line 3 can be aggregated together to obtain a text line aggregation result, that is, a text line set contains text line 1, text line 2, and text line 3.

[0159] S311, predicting entity line categories corresponding to each of the plurality of text lines according to the text line feature sequence.

[0160] In the embodiment of the present application, for a text line feature sequence, the entity line categories corresponding to each of the multiple text lines can be predicted based on the text line feature sequence, wherein a linear classifier can be used to determine the entity line categories corresponding to each of the multiple text lines.

[0161] Specifically, for a text line feature sequence, the text line feature sequence is input into the second linear classifier to obtain an entity line category sequence, wherein the entity line category in the entity line category sequence corresponds one-to-one to the text line feature in the text line feature sequence, and for any text line, the text line feature corresponding to the text line is searched from the text line feature sequence, and the entity line category corresponding to the text line feature is searched from the entity line category sequence, and determined as the entity line category corresponding to the text line. The entity line category may be, for example, a name, birthday, ID number, etc.

[0162] S312, extracting entities and entity categories of entities in the image to be processed according to the categories corresponding to the code elements, the text line aggregation results, and the entity line categories.

[0163] In the embodiment of the present application, after the uploading step, the category corresponding to each code element in the code element sequence, as well as the text line aggregation results of the text lines in the image to be processed and the entity line category can be obtained, so that the category, text line aggregation results and entity line category can be comprehensively analyzed to extract the entities in the image to be processed and the entity category of the entity. The extracted entities can be as follows: Figure 5 The content within the dotted box is shown.

[0164] Specifically, for any text line, the codeword subsequence corresponding to the text line is searched from the codeword sequence; according to the category, the second codeword is selected from the codeword subsequence, and the second codeword is sequentially spliced ​​to obtain the initial entity corresponding to the text line; according to the initial entity, the text line aggregation result and the entity line category, the entity and the entity category of the entity in the image to be processed are extracted. Among them, the second codeword generally refers to the codeword with the category of the starting category, the codeword with the category of the ending category, and the intermediate codeword between the codeword with the category of the starting category and the codeword with the category of the ending category in the codeword subsequence.

[0165] Among them, the text line aggregation result is usually reflected in the form of a text line set. There are one or more text lines in the text line set. When there is only one text line in the text line set, the entity corresponding to the text line in the text line set is directly taken as the final entity. When there are multiple text lines in the text line set, the entities corresponding to the text lines in the text line set are spliced ​​to obtain the final entity.

[0166] Based on this, according to the text line aggregation results, at least one text line set is determined, and the text lines in the text line set are one or more; for any text line set, when the text line set contains one text line, the following processing is performed: the initial entity corresponding to the text line contained in the text line set is determined as the first entity in the image to be processed; the entity line category corresponding to the text line contained in the text line set is determined as the entity category of the first entity; when the text line set contains multiple text lines, the following processing is performed: the initial entities corresponding to the multiple text lines contained in the text line set are spliced ​​in order (the order here refers to the text line position) to obtain the second entity in the image to be processed; the entity line category with the highest occurrence frequency is determined among the entity line categories corresponding to the multiple text lines contained in the text line set; the entity line category with the highest occurrence frequency is determined as the entity category of the second entity.

[0167] For example, Figure 4 As shown, the text line aggregation results are usually presented in the form of text line sets, where text line set 1 contains 1 text line "Zhang San", and text line set 2 contains 2 text lines "Electronic Information and Science", "College of Science and Technology". For text line set 1, the initial entity "Zhang San" corresponding to the text line "Zhang San" contained in text line set 1 is determined as the first entity in the image to be processed, and the entity type "name" corresponding to the text line "Zhang San" contained in text line set 1 is determined as the entity category of the first entity; for text line set 2, the initial entities "Electronic Information and Science", "College of Science and Technology" corresponding to the 2 text lines "Electronic Information and Science", "College of Science and Technology" contained in text line set 2 are spliced ​​to obtain the second entity "College of Electronic Information and Science and Technology" in the image to be processed, that is, Figure 5The content within the dashed box shown realizes the aggregation of entities, obtaining complete entities. Among the entity types corresponding to the two text lines "Electronic Information and Science" and "College of Science and Technology" included in text line set 2, the entity line category "School Name" with the highest frequency of occurrence is determined and designated as the entity category of the second entity.

[0168] In addition, for the text detection and recognition engine, the detection and recognition results of single text lines are usually given, and entities with adjacent positions will fall into the same text line. For example, Figure 4 as shown, the two entities "Telephone" and "159..." fall into the same text line, causing entity adhesion. Therefore, entity separation is required for this situation.

[0169] Based on this, for each symbol in the symbol sequence, three classifications are made, namely the start category, the end category, and other categories. For the start category and the end category, the start symbol and the end symbol of the entity can be known, so that a unique entity can be determined, realizing the separation of adhered entities.

[0170] For example, as Figure 4 shown, the two entities "Telephone" and "159..." fall into the same text line. According to the start category and the end category, the start symbol and the end symbol of the entity can be known. Therefore, for this text line, two initial entities, namely "Telephone" and "159...", can be determined. Subsequently, these 2 initial entities are determined as the first entities in the image to be processed, that is, as Figure 5 the content within the dashed box shown.

[0171] In this way, using the existing text detection and recognition engine, the text line detection results are obtained by detecting the text lines. Under these text line detection results, the text line features corresponding to each of the multiple text lines in the image to be processed are determined, the text line features are concatenated to obtain a text line feature sequence, and according to the text line feature sequence, the multiple text lines are aggregated to obtain a text line aggregation result. According to the text line feature sequence, the entity line categories corresponding to each of the multiple text lines are predicted. According to the category, the text line aggregation result, and the entity line category, the entities and the entity categories of the entities in the image to be processed are extracted, realizing the aggregation of entities and the separation of adhered entities, thereby extracting complete entities.

[0172] Corresponding to the above method embodiment, the embodiment of the present application also provides an image processing device, as Figure 6 shown, the device may include: an image acquisition module 610, a symbol sequence determination module 620, a category prediction module 630, a text line feature determination module 640, an entity and entity category extraction module 650.

[0173] The image acquisition module 610 is used to acquire the image to be processed;

[0174] A code element sequence determination module 620, configured to determine a code element sequence corresponding to at least one text line in the image to be processed;

[0175] A category prediction module 630, configured to predict a category corresponding to each symbol in the symbol sequence;

[0176] A text line feature determination module 640 is used to determine text line features corresponding to at least one text line in the image to be processed according to the category;

[0177] The entity and entity category extraction module 650 is used to extract the entities and entity categories of the entities in the image to be processed according to the text line features.

[0178] In an optional implementation, the symbol sequence determination module is specifically used to:

[0179] Detecting at least one text line position in the image to be processed, and for any of the text line positions, identifying a text line at the text line position;

[0180] Splitting the text line by a word segmenter to obtain a code element subsequence corresponding to the text line;

[0181] All the code element subsequences are concatenated to obtain a code element sequence corresponding to at least one text line in the image to be processed.

[0182] In an optional implementation, the category prediction module is specifically used to:

[0183] Determine a code element coordinate sequence corresponding to the code element sequence;

[0184] Determining a multimodal code element feature sequence according to the code element sequence, the code element coordinate sequence and the image to be processed;

[0185] Wherein, the multimodal codeword features in the multimodal codeword feature sequence correspond one-to-one to the codewords in the codeword sequence;

[0186] According to the multimodal codeword feature sequence, the category corresponding to each codeword in the codeword sequence is predicted.

[0187] In an optional implementation, the text line feature determination module specifically includes:

[0188] A codeword subsequence search submodule, used for searching, for any text line in the image to be processed, a codeword subsequence corresponding to the text line from the codeword sequence;

[0189] A codeword screening submodule, configured to screen a first codeword from the codeword subsequence according to the category;

[0190] A code element feature screening module, used for screening the target multimodal code element feature corresponding to the first code element from the multimodal code element feature sequence;

[0191] The text line feature determination submodule is used to determine the text line feature corresponding to the text line according to the target multimodal code element feature.

[0192] In an optional implementation, the symbol screening submodule is specifically used for:

[0193] From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity;

[0194] The code element feature screening module is specifically used for:

[0195] Selecting the multimodal code element feature corresponding to the starting code element of the entity from the multimodal code element feature sequence;

[0196] The text line feature determination submodule is specifically used for:

[0197] The multimodal code element feature corresponding to the start code element of the entity is determined as the text line feature corresponding to the text line.

[0198] In an optional implementation, the symbol screening submodule is specifically used for:

[0199] From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity;

[0200] From the codeword subsequence, screen the codewords of the end category, and determine the codewords of the end category as the end codewords of the entity;

[0201] From the codeword subsequence, select the intermediate codeword between the start codeword of the entity and the end codeword of the entity, and determine it as the intermediate codeword of the entity;

[0202] The code element feature screening module is specifically used for:

[0203] Selecting a first multimodal code element feature corresponding to a start code element of the entity from the multimodal code element feature sequence;

[0204] Selecting a second multimodal symbol feature corresponding to the intermediate symbol of the entity from the multimodal symbol feature sequence;

[0205] Selecting a third multimodal code element feature corresponding to the end code element of the entity from the multimodal code element feature sequence;

[0206] The text line feature determination submodule is specifically used for:

[0207] Determine a text line feature corresponding to the text line according to the first multimodal code element feature, the second multimodal code element feature, and the third multimodal code element feature.

[0208] In an optional implementation, the text line feature determination submodule is specifically used to:

[0209] Determining an average multimodal symbol feature of the first multimodal symbol feature, the second multimodal symbol feature, and the third multimodal symbol feature;

[0210] The average multimodal code element feature is determined as the text line feature corresponding to the text line.

[0211] In an optional implementation, when the image to be processed contains multiple text lines, the entity and entity category extraction module specifically includes:

[0212] A text line feature concatenation submodule, used for concatenating the text line features corresponding to the plurality of text lines to obtain a text line feature sequence;

[0213] A text line aggregation submodule, used for aggregating a plurality of the text lines according to the text line feature sequence to obtain a text line aggregation result;

[0214] An entity line category prediction submodule, used for predicting the entity line category corresponding to each of the plurality of text lines according to the text line feature sequence;

[0215] The entity and entity category extraction submodule is used to extract the entities and entity categories of the entities in the image to be processed according to the categories, the text line aggregation results and the entity line categories.

[0216] In an optional implementation, the text line aggregation submodule specifically includes:

[0217] A text line feature determination unit, configured to determine any of the remaining text line features in the text line feature sequence based on the first text line feature in the text line feature sequence;

[0218] A text line feature concatenation unit, configured to concatenate the first text line feature with any of the remaining text line features to obtain a paired text line feature;

[0219] A relative position feature determining unit, used to determine a relative position feature between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature;

[0220] A feature adding unit, used for adding the paired text line feature and the relative position feature to obtain a fused text line feature;

[0221] A result prediction unit, configured to predict, based on the fused text line feature, a pairing result between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature;

[0222] The text line aggregation unit is used to aggregate the multiple text lines according to the pairing result to obtain a text line aggregation result.

[0223] In an optional implementation, the relative position feature is multiple; and the feature adding unit is specifically used for:

[0224] splicing a plurality of the relative position features to obtain a target relative position feature, and deforming the target relative position feature;

[0225] Wherein, the size of the deformed target relative position feature is the same as the size of the paired text line feature;

[0226] The paired text line feature is added to the deformed target relative position feature to obtain a fused text line feature.

[0227] In an optional implementation, the result prediction unit is specifically used to:

[0228] Inputting the fused text line features into a first linear classifier to obtain a text line aggregation category;

[0229] In a case where the text line aggregation category is a preset text line aggregation category, determining that a text line corresponding to the first text line feature is successfully paired with a text line corresponding to any other text line feature;

[0230] In a case where the text line aggregation category is not a preset text line aggregation category, it is determined that the text line corresponding to the first text line feature fails to be paired with the text line corresponding to any other text line feature.

[0231] In an optional implementation, the entity row category prediction submodule is specifically used to:

[0232] Inputting the text line feature sequence into a second linear classifier to obtain an entity line category sequence;

[0233] Wherein, the entity line categories in the entity line category sequence correspond one-to-one to the text line features in the text line feature sequence;

[0234] For any of the text lines, searching for a text line feature corresponding to the text line from the text line feature sequence;

[0235] The entity row category corresponding to the text row feature is searched from the entity row category sequence, and is determined as the entity row category corresponding to the text row.

[0236] In an optional implementation, the entity and entity category extraction submodule is specifically used for:

[0237] A codeword subsequence searching unit, configured to search, for any of the text lines, from the codeword sequence for a codeword subsequence corresponding to the text line;

[0238] an initial entity concatenation unit, configured to select a second codeword from the codeword subsequence according to the category, and concatenate the second codewords in sequence to obtain an initial entity corresponding to the text line;

[0239] The entity and entity category extraction unit is used to extract the entity and the entity category of the entity in the image to be processed according to the initial entity, the text line aggregation result and the entity line category.

[0240] In an optional implementation, the entity and entity category extraction unit is specifically used to:

[0241] Determine at least one text row set according to the text row aggregation result, wherein the text row set includes one or more text rows;

[0242] For any of the text line sets, when the text line set contains one text line, the following processing is performed:

[0243] Determine the initial entity corresponding to the text line contained in the text line set as the first entity in the image to be processed;

[0244] Determine the entity row category corresponding to the text row included in the text row set as the entity category of the first entity;

[0245] When the text line set contains multiple text lines, the following processing is performed:

[0246] Sequentially concatenate the initial entities corresponding to the plurality of text lines contained in the text line set to obtain a second entity in the image to be processed;

[0247] Determine an entity row category with the highest occurrence frequency among the entity row categories corresponding to the plurality of text rows included in the text row set;

[0248] The entity row category with the highest occurrence frequency is determined as the entity category of the second entity.

[0249] The present application also provides an electronic device, such as Figure 7 As shown, it includes a processor 71, a communication interface 72, a memory 73 and a communication bus 74, wherein the processor 71, the communication interface 72, and the memory 73 communicate with each other through the communication bus 74.

[0250] A memory 73, for storing computer programs;

[0251] The processor 71 is used to execute the program stored in the memory 73 to implement the following steps:

[0252] Acquire an image to be processed, and determine a codeword sequence corresponding to at least one text line in the image to be processed; predict a category corresponding to each codeword in the codeword sequence, and determine text line features corresponding to at least one text line in the image to be processed based on the category; extract entities in the image to be processed and entity categories of the entities based on the text line features.

[0253] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0254] The communication interface is used for communication between the above electronic device and other devices.

[0255] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0256] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0257] In another embodiment provided in the present application, a storage medium is provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes any one of the image processing methods in the above embodiments.

[0258] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the image processing methods in the above embodiments.

[0259] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0260] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0261] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0262] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, and determine a code element sequence corresponding to at least one text line in the image to be processed; Predicting a category corresponding to each codeword in the codeword sequence, and determining text line features corresponding to at least one text line in the image to be processed according to the category; Entities in the image to be processed and entity categories of the entities are extracted according to the text line features.

2. The method according to claim 1, characterized in that The step of determining a symbol sequence corresponding to at least one text line in the image to be processed includes: Detecting at least one text line position in the image to be processed, and for any of the text line positions, identifying a text line at the text line position; Splitting the text line by a word segmenter to obtain a code element subsequence corresponding to the text line; All the code element subsequences are concatenated to obtain a code element sequence corresponding to at least one text line in the image to be processed.

3. The method according to claim 1, characterized in that The predicting the category corresponding to each symbol in the symbol sequence includes: Determine a code element coordinate sequence corresponding to the code element sequence; Determining a multimodal code element feature sequence according to the code element sequence, the code element coordinate sequence and the image to be processed; Wherein, the multimodal codeword features in the multimodal codeword feature sequence correspond one-to-one to the codewords in the codeword sequence; According to the multimodal codeword feature sequence, the category corresponding to each codeword in the codeword sequence is predicted.

4. The method according to claim 3, characterized in that The step of determining text line features corresponding to at least one text line in the image to be processed according to the category includes: For any text line in the image to be processed, searching for a code element subsequence corresponding to the text line from the code element sequence; According to the category, selecting a first codeword from the codeword subsequence, and selecting a target multimodal codeword feature corresponding to the first codeword from the multimodal codeword feature sequence; Determine a text line feature corresponding to the text line according to the target multimodal code element feature.

5. The method according to claim 4, characterized in that The step of selecting the first symbol from the symbol subsequence according to the category comprises: From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity; The step of selecting a target multimodal symbol feature corresponding to the first symbol from the multimodal symbol feature sequence includes: Selecting the multimodal code element feature corresponding to the starting code element of the entity from the multimodal code element feature sequence; The determining, according to the target multimodal code element feature, a text line feature corresponding to the text line comprises: The multimodal code element feature corresponding to the start code element of the entity is determined as the text line feature corresponding to the text line.

6. The method according to claim 4, characterized in that The step of selecting the first symbol from the symbol subsequence according to the category comprises: From the codeword subsequence, select the codeword of the category of the starting category, and determine the codeword of the category of the starting category as the starting codeword of the entity; From the codeword subsequence, screen the codewords of the end category, and determine the codewords of the end category as the end codewords of the entity; From the codeword subsequence, select the intermediate codeword between the start codeword of the entity and the end codeword of the entity, and determine it as the intermediate codeword of the entity; The step of selecting a target multimodal symbol feature corresponding to the first symbol from the multimodal symbol feature sequence includes: Selecting a first multimodal code element feature corresponding to a start code element of the entity from the multimodal code element feature sequence; Selecting a second multimodal symbol feature corresponding to the intermediate symbol of the entity from the multimodal symbol feature sequence; Selecting a third multimodal code element feature corresponding to the end code element of the entity from the multimodal code element feature sequence; The determining, according to the target multimodal code element feature, a text line feature corresponding to the text line comprises: Determine a text line feature corresponding to the text line according to the first multimodal code element feature, the second multimodal code element feature, and the third multimodal code element feature.

7. The method according to claim 6, characterized in that The determining, according to the first multimodal code element feature, the second multimodal code element feature, and the third multimodal code element feature, a text line feature corresponding to the text line includes: Determining an average multimodal symbol feature of the first multimodal symbol feature, the second multimodal symbol feature, and the third multimodal symbol feature; The average multimodal code element feature is determined as the text line feature corresponding to the text line.

8. The method according to claim 1, characterized in that In a case where the image to be processed includes a plurality of text lines, extracting entities and entity categories of the entities in the image to be processed according to the text line features includes: Performing concatenation processing on the text line features corresponding to each of the plurality of text lines to obtain a text line feature sequence; Aggregating the plurality of text lines according to the text line feature sequence to obtain a text line aggregation result; Predicting entity line categories corresponding to each of the plurality of text lines according to the text line feature sequence; Entities in the image to be processed and entity categories of the entities are extracted according to the categories, the text line aggregation results, and the entity line categories.

9. The method according to claim 8, characterized in that The step of aggregating the plurality of text lines according to the text line feature sequence to obtain a text line aggregation result includes: For a first text line feature in the text line feature sequence, determining any remaining text line features in the text line feature sequence; Concatenate the first text line feature with any other text line feature to obtain a paired text line feature; Determining a relative position feature between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature; Adding the paired text line feature to the relative position feature to obtain a fused text line feature; Predicting, based on the fused text line feature, a pairing result between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature; According to the pairing result, the plurality of text lines are aggregated to obtain a text line aggregation result.

10. The method according to claim 9, characterized in that The relative position features are multiple; The step of adding the paired text line feature to the relative position feature to obtain a fused text line feature includes: splicing a plurality of the relative position features to obtain a target relative position feature, and deforming the target relative position feature; Wherein, the size of the deformed target relative position feature is the same as the size of the paired text line feature; The paired text line feature is added to the deformed target relative position feature to obtain a fused text line feature.

11. The method according to claim 9, characterized in that The step of predicting, based on the fused text line feature, a pairing result between a text line corresponding to the first text line feature and a text line corresponding to any other text line feature includes: Inputting the fused text line features into a first linear classifier to obtain a text line aggregation category; In a case where the text line aggregation category is a preset text line aggregation category, determining that a text line corresponding to the first text line feature is successfully paired with a text line corresponding to any other text line feature; In a case where the text line aggregation category is not a preset text line aggregation category, it is determined that the text line corresponding to the first text line feature fails to be paired with the text line corresponding to any other text line feature.

12. The method according to claim 8, characterized in that The step of predicting entity line categories corresponding to each of the plurality of text lines according to the text line feature sequence comprises: Inputting the text line feature sequence into a second linear classifier to obtain an entity line category sequence; Wherein, the entity line categories in the entity line category sequence correspond one-to-one to the text line features in the text line feature sequence; For any of the text lines, searching for a text line feature corresponding to the text line from the text line feature sequence; The entity row category corresponding to the text row feature is searched from the entity row category sequence, and is determined as the entity row category corresponding to the text row.

13. The method according to claim 8, characterized in that The extracting entities in the image to be processed and entity categories of the entities according to the categories, the text line aggregation results and the entity line categories includes: For any of the text lines, searching the code element subsequence corresponding to the text line from the code element sequence; According to the category, selecting a second codeword from the codeword subsequence, and sequentially splicing the second codewords to obtain an initial entity corresponding to the text line; Entities in the image to be processed and entity categories of the entities are extracted according to the initial entities, the text line aggregation results, and the entity line categories.

14. The method according to claim 13, characterized in that The extracting the entity in the image to be processed and the entity category of the entity according to the initial entity, the text line aggregation result and the entity line category includes: Determine at least one text row set according to the text row aggregation result, wherein the text row set includes one or more text rows; For any of the text line sets, when the text line set contains one text line, the following processing is performed: Determine the initial entity corresponding to the text line contained in the text line set as the first entity in the image to be processed; Determine the entity row category corresponding to the text row included in the text row set as the entity category of the first entity; When the text line set contains multiple text lines, the following processing is performed: Sequentially concatenate the initial entities corresponding to the plurality of text lines contained in the text line set to obtain a second entity in the image to be processed; Determine an entity row category with the highest occurrence frequency among the entity row categories corresponding to the plurality of text rows included in the text row set; The entity row category with the highest occurrence frequency is determined as the entity category of the second entity.

15. An image processing device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; A code element sequence determination module, used to determine a code element sequence corresponding to at least one text line in the image to be processed; A category prediction module, used to predict the category corresponding to each codeword in the codeword sequence; A text line feature determination module, used to determine text line features corresponding to at least one text line in the image to be processed according to the category; The entity and entity category extraction module is used to extract the entities and entity categories of the entities in the image to be processed according to the text line features.

16. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images