Form recognition method, device and storage medium based on self-attention mechanism

Abstract

The present invention relates to the field of artificial intelligence and digital medicine, and provides a form recognition method, device and storage medium based on a self-attention mechanism, the method comprising: obtaining a form image to be recognized, and performing text detection and text recognition on the form image to obtain text position information and text content information of the form image; reading the pixel value of each pixel in the form image to obtain the pixel information of the form image; according to the pixel information, text position information and text content information of the form image, a structured processing result of the form image is obtained through a pre-trained form data processing model, wherein the form data processing model is obtained by training an encoding and decoding model based on a self-attention mechanism; performing form data post-processing on the structured processing result of the form image to obtain structured form data. The above method can improve the adaptability of different form images in automatic recognition and improve the accuracy of form image recognition.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and digital medicine, and in particular to a form recognition method, device, storage medium and computer equipment based on a self-attention mechanism. Background Art

[0002] With the development and progress of science and technology, artificial intelligence technology is becoming more and more perfect, and the automatic recognition technology of form lists has gradually developed and begun to be applied in people's daily lives. For example, in the business scenario of medical insurance reimbursement in different places, it is often necessary to manually enter the details of the medical list. Generally, there are dozens of details to be entered for each case, which is undoubtedly a tedious, time-consuming and labor-intensive task for medical insurance staff. In order to reduce manpower consumption and improve reimbursement timeliness, some solutions for automatic form recognition have emerged to save manpower investment costs and improve the efficiency of form recognition.

[0003] In the prior art, the method for automatically identifying various forms is usually to perform text recognition and text content extraction on the form image, and then perform regular post-processing on the recognized text to form structured data. However, this regular post-processing method is difficult to handle complex situations such as no table lines and multiple line merges. Moreover, for more complex layouts, the regular processing method does not have good robustness and adaptability, resulting in low accuracy in form recognition. Another common form automatic recognition method is a form reconstruction method based on template matching. This method requires first designing a template image, then matching and calibrating the input image and the template image, and finally recognizing the input image based on the structure of the template image. However, this method must set a template for each form separately, has poor versatility, and has low accuracy in form recognition. Summary of the invention

[0004] In view of this, the present application provides a form recognition method, device, storage medium and computer equipment based on the self-attention mechanism, the main purpose of which is to solve the technical problems in the prior art that the form automatic recognition method has poor versatility and low recognition result accuracy.

[0005] According to a first aspect of the present invention, a form recognition method based on a self-attention mechanism is provided, the method comprising:

[0006] Acquire a form image to be identified, and perform text detection and text recognition on the form image to obtain text position information and text content information of the form image;

[0007] Read the pixel value of each pixel in the form image to obtain the pixel information of the form image;

[0008] According to the pixel information, text position information and text content information of the form image, a structured processing result of the form image is obtained through a pre-trained form data processing model, wherein the form data processing model is trained through an encoding and decoding model based on a self-attention mechanism;

[0009] The structured processing result of the form image is subjected to form data post-processing to obtain structured form data.

[0010] According to a second aspect of the present invention, a form recognition device based on a self-attention mechanism is provided, the device comprising:

[0011] The text information acquisition module is used to acquire the form image to be identified, and perform text detection and text recognition on the form image to obtain the text position information and text content information of the form image;

[0012] The image information acquisition module is used to read the pixel value of each pixel in the form image to obtain the pixel information of the form image;

[0013] The structured processing module is used to obtain the structured processing result of the form image according to the pixel information, text position information and text content information of the form image through the pre-trained form data processing model, wherein the form data processing model is obtained by training the encoding and decoding model based on the self-attention mechanism;

[0014] The data post-processing module is used to perform form data post-processing on the structured processing results of the form image to obtain structured form data.

[0015] According to a third aspect of the present invention, there is provided a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-mentioned form recognition method based on the self-attention mechanism.

[0016] According to a fourth aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned form recognition method based on the self-attention mechanism when executing the program.

[0017] The present invention provides a form recognition method, device, storage medium and computer equipment based on a self-attention mechanism. First, the text position information and text content information of the form image are obtained through text detection and text recognition technology, and then the pixel information of the form image is obtained by reading the pixel value of the form image. Finally, according to the pixel information, text position information and text content information, the form image is subjected to structured processing and data post-processing through a pre-trained form data processing model to obtain structured form data. By using the form data processing model, the method can be applied to form images of various layouts and formats at the same time, effectively improving the robustness and adaptability of the form automatic recognition method for different form images. In addition, the form data processing model in the method is implemented by a coding and decoding model based on the self-attention mechanism, wherein the self-attention mechanism can perform fusion learning on various types of information in the form image, thereby learning the mutual connection between various types of information, and the coding and decoding model can accurately predict the hidden information in the form image through fusion feature information. Therefore, the form data processing model can effectively improve the accuracy of automatic recognition of form images.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 A schematic diagram of a process flow of a form recognition method based on a self-attention mechanism provided by an embodiment of the present invention is shown;

[0021] Figure 2 A schematic diagram of pre-training of a form data processing model provided by an embodiment of the present invention is shown;

[0022] Figure 3 A training schematic diagram of a form data processing model provided by an embodiment of the present invention is shown;

[0023] Figure 4 A structural schematic diagram of a form recognition device based on a self-attention mechanism provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0025] In one embodiment, Figure 1 As shown, a form recognition method based on the self-attention mechanism is provided, and the method is applied to a computer device such as a server as an example for explanation, wherein the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The above method includes the following steps:

[0026] 101. Obtain a form image to be identified, and perform text detection and text recognition on the form image to obtain text position information and text content information of the form image.

[0027] Among them, form images refer to images formed by photographing or scanning various paper forms. In actual work and life, paper forms are often manually filled out in Excel, Word, or manually drawn forms. For example, paper forms can be production records in a factory workshop or medical records in a hospital. The information in such paper forms usually needs to be further counted and analyzed by computer equipment. Therefore, it is necessary to extract the information in such form images in a structured manner.

[0028] Specifically, first, the computer device can extract the form image to be identified from a database or from other storage devices, or can directly scan the paper form to be identified to obtain the form image to be identified, and then perform text detection and text recognition on the form image to be identified, so as to obtain the text position information and text content information in the form image to be identified. Among them, text detection refers to locating the text area in the image, and usually marking the text line in the form of a bounding box to form a text box; and text recognition is to identify the text information and the position information corresponding to the text in the text box obtained by text detection after text detection. In this embodiment, text recognition and text detection are the basis of form recognition, and the accuracy of their recognition and detection is crucial. Therefore, this embodiment can be implemented based on the methods with higher accuracy of text recognition and text detection in the prior art.

[0029] 102. Read the pixel value of each pixel in the form image to obtain pixel information of the form image.

[0030] Among them, image pixel information generally refers to the pixel value in the image. The image read by the computer device is composed of small squares. Each small square has a clear position and an assigned color value. The pixel value is a digital sequence used to represent a minimum unit of a small square in the image. The pixel value is usually read by the computer device through an algorithm. Common algorithms include python, matlab, opencv, etc.

[0031] Specifically, the computer device can extract pixel values ​​from the form image through an algorithm for reading pixel values ​​to obtain pixel information of the form image. The pixel information of the form image may include color value information of each basic unit in the form image, and may also include position information of each basic unit. The pixel information of the form image, the text content information, and the text position information together constitute the overall information of the form image. There is also a certain relationship between the pixel information and the text content information and the text position information. Certain pixel information corresponds to corresponding text content information and text position information, and the training of the subsequent form data processing model can be realized based on the relationship between various information.

[0032] 103. According to the pixel information, text position information and text content information of the form image, a structured processing result of the form image is obtained through a pre-trained form data processing model, wherein the form data processing model is trained by an encoding and decoding model based on a self-attention mechanism.

[0033] Among them, the self-attention mechanism is a variant of the attention mechanism. The attention mechanism imitates the internal process of biological observation behavior, that is, it combines internal experience and external sensation to increase the observation precision of some areas, while the self-attention mechanism reduces the dependence on external information and is better at capturing the internal correlation of data or features. The form data processing model in this implementation is obtained by training the encoding and decoding model based on the self-attention mechanism, wherein the encoding and decoding model is a transformer model, which is composed of an encoder and a decoder and is commonly used in the field of translation. This embodiment innovatively applies the encoding and decoding model to the structured processing process of automatic recognition of form images, and can use the prediction ability of the encoding and decoding model to effectively improve the data processing ability of the form data processing model, thereby improving the accuracy of recognition.

[0034] Specifically, the computer device can process the pixel information, text position information, text content information and other information of the form image to be identified through the pre-trained form data processing model to obtain the structured processing result of the form image. In this embodiment, the structured processing result can set different output tasks according to the training tasks of the form data processing model, so as to obtain different output results, wherein the output result of the form data processing model can include the starting coordinate determination information of the form image, the form column name determination information and the text position information on at least two sides, etc., which is not specifically limited in this embodiment.

[0035] In this embodiment, in the process of processing multimodal information including image pixel information, text position information and text content information through the form data processing model, the feature expression of multimodal information is an important part of the processing process. Among them, multimodal information refers to information with multiple different sources or forms. For example, one kind of information can be expressed by video, text and voice respectively, then these three forms of expression are three modes of the information. In this embodiment, multimodal information refers to the image pixel information, text position information and text content information in the form image. The multimodal information can express different features of the form image from different angles, and the features of different modes are related to each other, so that the overall information of the form image can be highly restored according to the correlation between the multimodal information. In this embodiment, the self-attention mechanism is used to perform multimodal information fusion learning on various information. Compared with the learning method based on a single information, this fusion learning method based on the self-attention mechanism can better express features around the complementarity between multiple information, thereby improving the accuracy of the form data processing model.

[0036] 104. Perform form data post-processing on the structured processing result of the form image to obtain structured form data.

[0037] Among them, the post-processing process is the process of forming the final structured form data based on the structured processing results of the form image. Structured form data refers to the data that arranges various information in the form image according to certain categories and orders to form a table structure with clear structure and clear information.

[0038] Specifically, the computer device can further perform position estimation and information arrangement processing on the structured processing results of various form images obtained by the form data processing model, and generate structured form data according to information such as form column names and text content position information. In this embodiment, post-processing the structured processing results of the form images is the last step in form recognition, so the accuracy of the post-processing also directly determines the accuracy of the form recognition results. This embodiment further improves the accuracy of form recognition by performing data post-processing based on the accurate output results of the form data processing model.

[0039] The form recognition method based on the self-attention mechanism provided in this embodiment first obtains the text position information and text content information of the form image through text detection and text recognition technology, then obtains the pixel information of the form image by reading the pixel value of the form image, and finally performs structured processing and data post-processing on the form image through the pre-trained form data processing model according to the pixel information, text position information and text content information, so as to obtain structured form data. By using the form data processing model, the above method can make the method applicable to form images of various layouts and formats at the same time, and effectively improves the robustness and adaptability of the form automatic recognition method for different form images. In addition, the form data processing model in the above method is implemented by the encoding and decoding model based on the self-attention mechanism, wherein the self-attention mechanism can perform fusion learning on various types of information in the form image, thereby learning the mutual connection between various types of information, and the encoding and decoding model can accurately predict the hidden information in the form image through the fusion feature information, therefore, the above form data processing model can effectively improve the accuracy of automatic recognition of form images.

[0040] In one embodiment, the above step 101 can be implemented by the following method: first, the computer device can obtain the form image to be identified from the database, and then perform text detection on the form image to be identified to obtain all text boxes in the form image to be identified. Further, the computer device can obtain all text position information and text content information in the form image to be identified by performing text recognition on all detected text boxes. In this embodiment, the text position information can include one-dimensional position information and two-dimensional position information, wherein the one-dimensional position information refers to arranging the text in the form image to be identified in a certain order, and the serial number information corresponding to each text is the one-dimensional position information, such as the one-dimensional position information of the first text in the text is No. 1, the one-dimensional position information of the second text is No. 2, and so on; the two-dimensional position information refers to the position coordinate information of each text in the form image to be identified, such as the two-dimensional position information of the first text is (0, 0), the two-dimensional position information of the second text is (0, 1), and so on. This embodiment enriches the text position information expression of the form image by adding one-dimensional position information and two-dimensional position information, and the expression of one-dimensional position information is simpler and more direct, while the expression of two-dimensional position information is more accurate and clear. The combination of the two can improve the efficiency of form image recognition while also ensuring the accuracy of recognition. In addition, in this embodiment, text detection can use the DBnet algorithm based on text segmentation, and text recognition can use the CRNN+CTC algorithm. In addition, text detection and text recognition can also use other algorithms commonly used in the prior art, which are not limited in this embodiment.

[0041] In one embodiment, the above step 102 can be implemented in the following manner: first, before reading the pixel value of each pixel in the form image, the form image can be segmented according to the preset image segmentation rule to obtain multiple form image modules. For example, the entire form image can be cross-divided into four equal parts to obtain the upper left area form image module, the lower left area form image module, the upper right area form image module and the lower right area form image module. In addition, the number and method of segmentation of the form image can obtain different segmentation results according to different preset rules. For example, the form image can also be vertically divided into three equal parts, etc. This embodiment does not specifically limit the image segmentation rules. Then, read the pixel value of each pixel of all form image modules to obtain the pixel information of each form image module. In this embodiment, the pixel information also includes the one-dimensional position information and the two-dimensional position information of each form image module. This embodiment further divides the form image into multiple form image modules, thereby achieving more accurate positioning of the pixel information of the form image, so that the relationship between the pixel information, the text position information and the text content information can be more accurately expressed in the subsequent learning process based on the self-attention mechanism.

[0042] In one embodiment, the above step 103 can be implemented in the following manner: First, the computer device can perform data conversion preprocessing on the pixel information, text position information and text content information of the form image to obtain a feature vector combination of the form image. In this embodiment, the feature vector combination of the form image can be obtained in the following manner: First, the pixel information and text content information of the form image are classified and encoded to obtain the classification coding information of the form image, thereby distinguishing the image pixel information and the text content information according to the classification coding information. Then, the pixel information, text position information, text content information and classification coding information of the form image are respectively data converted to obtain the image feature vector, text position feature vector, text content feature vector and classification coding feature vector of the form image. Further, the image feature vector, text position feature vector, text content feature vector and classification coding feature vector are fused and spliced ​​to obtain the feature vector combination of the form image. For example, Figure 2 and Figure 3 As shown, "C" in the input part of the two figures represents the classification coding feature vector of image pixel information, and "A" represents the classification coding feature vector of text content information. What is spliced ​​with the classification coding feature vector are the one-dimensional position feature vector in the image feature vector and the one-dimensional position feature vector in the text position feature vector. What is spliced ​​next are the two-dimensional position feature vector in the image feature vector and the two-dimensional position feature vector in the text position feature vector. What is spliced ​​with the two-dimensional position feature vector are the image feature vectors of the four form image modules obtained after segmentation and the text content feature vector.

[0043] Further, the computer device can pre-process the feature vector combination of the form image through the self-attention mechanism in the form data processing model to obtain the fused feature vector of the form image. In this embodiment, by inputting the feature vector combination of the form image into the form data processing model, the self-attention mechanism in the form data processing model can learn the mutual relationship between the multiple feature vectors in the feature vector combination, and the fused feature vector of the form image can be obtained through the learned mutual relationship between the multiple feature vectors. Finally, the computer device can obtain the structured processing result of the form image through the encoding and decoding network in the form data processing model according to the fused feature vector of the form image. In this embodiment, the structured processing result of the form image may include the starting coordinate determination information of the form image, the form column name determination information, and the text position information on at least two sides, etc.

[0044] In one embodiment, the above step 104 can be implemented in the following manner: first, according to the structural processing result of the form image, the text content information of the starting coordinates of the form image, the text content information of each form column name, and the text position information of at least two sides of each text content information are determined; secondly, according to the text content information of the starting coordinates and the text position information of at least two sides of each text content information, the coordinate information of each text content information is determined. Finally, according to the text content information of each form column name and the coordinate information of each text content information, structured form data is generated. In this embodiment, the text position information of at least two sides of the text content information can be one-dimensional position information on the right and bottom of the text content information, or one-dimensional position information on the left and top of the text content information, etc. This embodiment does not specifically limit the text position information. Specifically, first, the starting coordinates of the form are found according to the starting coordinate determination result in the output result of the form data processing model, which can be marked as the coordinates (0, 0); then, the column names of the form are found according to the form column name determination information in the output result of the form data processing model, and the text content information of each form column name is obtained; further, according to the right one-dimensional position information of each text content feature vector in the output result of the form data processing model, it can be marked as the coordinates (0, 1), (0, 2) and so on, and then, according to the lower one-dimensional position information of each text content feature vector in the output result of the form data processing model, the position of each text content feature vector is located, and it can be marked as the coordinates (1, 0), (2, 0) and so on, so as to obtain the coordinate position of each text content; finally, according to the text content information of each form column name and the coordinate information of each text content information, orderly arrangement and combination are performed to generate structured form data.

[0045] In a specific embodiment, the training method of the form data processing model includes the following steps:

[0046] First, perform text detection and text recognition on a large number of form sample images of different versions and containing complex content, and obtain pixel information, text position information, and text content information of multiple form image samples. Secondly, respectively perform data conversion on the pixel information, text position information, and text content information of each of the form image samples using a conversion algorithm suitable for images, position information, and content information, extract feature vectors, and obtain the image feature vector, text position feature vector, and text content feature vector of each form image sample. Furthermore, fuse and splice the image feature vector, text position feature vector, and text content feature vector of each form image sample to obtain a feature vector combination of each form image sample. For example, Figure 2 and Figure 3As shown, the splicing process of the feature vector combination is to first splice the classification coding feature vector of the pixel information represented by "C" in the two pictures, and the classification coding feature vector of the text content information represented by "A", with the one-dimensional position feature vector in the picture feature vector and the one-dimensional position feature vector in the text position feature vector, and then splice the one-dimensional position feature vector with the two-dimensional position feature vector in the picture feature vector and the two-dimensional position feature vector in the text position feature vector, and then splice the two-dimensional position feature vector with the picture feature vectors of the four form picture modules obtained by segmentation and the text content feature vector, and finally obtain the feature vector combination.

[0047] Furthermore, the feature vector combination of each form image sample is used as input, and the image-text matching judgment information, text blackening judgment information and text occlusion prediction information of each form image sample is used as output to pre-train the encoding and decoding model based on the self-attention mechanism to obtain a pre-trained encoding and decoding model. The schematic diagram of the pre-training is shown in FIG. Figure 2 In this embodiment, before the feature vector group is input into the encoding and decoding model, the text content feature vector of each form image sample can be randomly blackened and blocked, and then the text information is predicted to match the image information, whether the text information is blackened, and the blocked text content feature vector is predicted. Figure 2 As shown, each form image sample is divided into equal parts crosswise, and the segmentation result of each form image sample is encoded to obtain pixel information V1, V2, V3 and V4 of the form image module of each form image sample, and each pixel information includes one-dimensional position information and two-dimensional position information of the sample form image module, and the text position information also includes one-dimensional position information and two-dimensional position information.

[0048] Figure 2The pre-trained form data processing model in is an encoding and decoding model based on a spatially aware self-attention mechanism. The input part is a feature vector combination that integrates the image feature vector, the text position feature vector, the text content feature vector, and the classification encoding feature vector. The output part is the image-text matching judgment information, the text blackening judgment information, and the text occlusion prediction information of each form image sample. For example, firstly, the text content feature vector of each form image sample is randomly blackened and occluded, and then the image feature vector, the text position feature vector, the text content feature vector, and the classification encoding feature vector of each form image sample are fused and spliced ​​to obtain the feature vector combination of each form image sample, and the feature vector combination of all form image samples is used as input to judge the image-text matching information. If the judgment result is a match, matched is output, and if not matched, not matched is output; then the text blackening is judged. If the judgment result is blackened, covered is output, and if not blackened, not covered is output; finally, the text information at the occlusion is predicted and the prediction result is output. Finally, by pre-training the encoding and decoding model based on the self-attention mechanism, the model parameters were optimized and a pre-trained form data processing model was obtained.

[0049] Furthermore, the pre-trained encoding and decoding model is trained with the feature vector combination of each form image sample as input, the starting coordinate determination information, the form column name determination information and the text position information on at least two sides of each form image sample as output, and a form data processing model is obtained. The training diagram of the form data processing model is shown in FIG. Figure 3 As shown in the figure, the input part is a feature vector combination of the form image, and the output part is divided into four parts, namely the starting coordinate determination information (if it is the starting coordinate, True is output, if it is not the starting coordinate, False is output), the form column name determination information (if it is the form column name, True is output, if it is not the form column name, False is output), the one-dimensional position information on the right side of each text content feature vector, and the one-dimensional position information below each text content feature vector. For example, firstly, the feature vector combination that integrates the image feature vector, the text position feature vector, the text content feature vector, and the classification coding feature vector is input into the pre-trained encoding and decoding model, and then according to the output content, the first task, the second task, the third task, and the fourth task are set for the training process of the form data processing model, where,

[0050] The first task is to determine whether each vector is the starting coordinate of the form. If it is the starting coordinate, the first task output is True, if it is not the starting coordinate, the first task output is False;

[0051] The second task is to determine whether each feature vector is a table column name. If it is a table column name, the second task output is True, if not, the second task output is False;

[0052] The third task is to predict the right position information of each text content feature vector, and the output of the third task is the one-dimensional position information on the right side of the feature vector;

[0053] The fourth task is to predict the position information below the feature vector of each text content, and the output of the fourth task is one-dimensional position information below the feature vector.

[0054] In addition, for the third and fourth tasks, the left position information and the upper position information of each text content feature vector can also be predicted, and the left position information and the lower position information of each text content feature vector can also be predicted, etc. Finally, the results of the four tasks are used as output to train the encoding and decoding model to obtain the form data processing model.

[0055] In a specific embodiment, the form image may be a medical list image, wherein the medical list image refers to a medical list in an image format formed by photographing or scanning the medical list, which can be used in medical insurance reimbursement, payment, etc. The medical list image may include detailed information such as the name of the drug, the price of the drug, the name and age of the person making the reimbursement. In this embodiment, the automatic recognition of the medical list image first obtains the medical list image to be recognized, and performs text detection and text recognition on the medical list image to obtain the text position information and text content information of the medical list image, then reads the pixel value of each pixel in the medical list image to obtain the pixel information of the medical list image, further, according to the pixel information, text position information and text content information of the medical list image, through the pre-trained form data processing model, the structured processing result of the medical list image is obtained, and finally, the structured processing result of the medical list image is subjected to form data post-processing to obtain structured medical list data. In the business scenario of medical insurance reimbursement in other places, it is often necessary to manually enter the details of the medical list. Generally, there are dozens of details to be entered for each case, which is undoubtedly a tedious, time-consuming and labor-intensive task for medical insurance staff. The form recognition method based on the self-attention mechanism provided in this embodiment can save input manpower and improve the timeliness of medical insurance reimbursement in other places.

[0056] The form recognition method based on the self-attention mechanism provided in this embodiment obtains a fused feature vector that integrates multimodal features by pre-training the encoding and decoding model, obtains a more relevant feature vector through multimodal feature fusion learning of the self-attention mechanism, and further trains the pre-trained encoding and decoding model according to the fused feature vector, thereby obtaining a form data processing model, which further improves the accuracy of form recognition.

[0057] Further, as Figure 1 The specific implementation of the method shown in the embodiment provides a form recognition method device based on the self-attention mechanism, such as Figure 4 As shown, the device includes: a text information acquisition module 21, a graph information acquisition module 22, a structured processing module 23 and a data post-processing module 24.

[0058] The text information acquisition module 21 can be used to acquire the form image to be identified, and perform text detection and text recognition on the form image to obtain the text position information and text content information of the form image;

[0059] The image information acquisition module 22 can be used to read the pixel value of each pixel in the form image to obtain the pixel information of the form image;

[0060] The structured processing module 23 can be used to obtain the structured processing result of the form image according to the pixel information, text position information and text content information of the form image through a pre-trained form data processing model, wherein the form data processing model is obtained by training the encoding and decoding model based on the self-attention mechanism;

[0061] The data post-processing module 24 can be used to perform form data post-processing on the structured processing result of the form image to obtain structured form data.

[0062] In a specific application scenario, the text information acquisition module 21 can be used to perform text detection on a form image to obtain a text box in the form image, and then perform text recognition on the text box to obtain text position information and text content information of the form image, wherein the text position information includes one-dimensional position information and two-dimensional position information.

[0063] In a specific application scenario, the image information acquisition module 22 can be used to segment the form image according to a preset image segmentation rule to obtain multiple form image modules, and then read the pixel value of each pixel of each form image module to obtain the pixel information of each form image module, wherein the image pixel information also includes one-dimensional position information and two-dimensional position information of each form image module.

[0064] In a specific application scenario, the structured processing module 23 can be specifically used to perform data conversion preprocessing on the pixel information, text position information and text content information of the form image to obtain a feature vector combination of the form image. Then, the feature vector combination of the form image is preprocessed through the self-attention mechanism in the form data processing model to obtain a fused feature vector of the form image. Further, based on the fused feature vector, the form data processing model is used to obtain a structured processing result of the form image, wherein the structured processing result of the form image includes the starting coordinate determination information of the form image, the form column name determination information and the text position information on at least two sides.

[0065] In a specific application scenario, the structured processing module 23 can also be used to classify and encode the pixel information and text content information of the form image to obtain the classification coding information of the form image, and then perform data conversion on the pixel information, text position information, text content information and classification coding information of the form image to obtain the image feature vector, text position feature vector, text content feature vector and classification coding feature vector of the form image; further, the image feature vector, text position feature vector, text content feature vector and classification coding feature vector are fused and spliced ​​to obtain a feature vector combination of the form image.

[0066] In a specific application scenario, the data post-processing module 24 can be specifically used to determine the text content information of the starting coordinates of the form image, the text content information of each form column name, and the text position information on at least two sides of each text content information according to the structured processing result of the form image, and then determine the coordinate information of each text content information according to the text content information of the starting coordinates and the text position information on at least two sides of each text content information, and finally, generate structured form data according to the text content information of each form column name and the coordinate information of each text content information.

[0067] It should be noted that for other corresponding descriptions of the functional units involved in the form recognition device based on the self-attention mechanism provided in this embodiment, please refer to Figure 1 The corresponding description in will not be repeated here.

[0068] Based on the above Figure 1 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 The form recognition method based on self-attention mechanism is shown.

[0069] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product. The software product to be identified can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0070] Based on the above Figure 1 The method shown, and Figure 4 The embodiment of the form recognition device based on the self-attention mechanism shown in the figure, in order to achieve the above purpose, this embodiment also provides a physical device of the form recognition method based on the self-attention mechanism, which can be a personal computer, a server, a smart phone, a tablet computer, a smart watch, or other network devices, etc., and the physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above Figure 1 The method shown.

[0071] Optionally, the physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0072] Those skilled in the art will understand that the physical device structure of the form recognition method based on the self-attention mechanism provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0073] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware of the physical device and the software resources to be identified, and supports the operation of the information processing program and other software and / or programs to be identified. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms, or by hardware. By applying the technical solution of the present application, first, a form image to be identified is obtained, and text detection and text recognition are performed on the form image to obtain text position information and text content information of the form image, and the pixel value of each pixel in the form image is read to obtain the pixel information of the form image, and then, according to the pixel information, text position information and text content information of the form image, a structured processing result of the form image is obtained through a pre-trained form data processing model, wherein the form data processing model is obtained by training an encoding and decoding model based on a self-attention mechanism, and finally the structured processing result of the form image is post-processed with form data to obtain structured form data. Compared with the prior art, the problem that the regular form data post-processing method in the prior art has low robustness and poor adaptability when facing form images with poor image quality or very complicated content is solved. In addition, through the use of the form data processing model, the present method can be applied to form images of various layouts and formats, and the adaptability to different form images is improved. In addition, the present invention learns image information, text content information and text position information based on the self-attention mechanism, thereby obtaining a close relationship between multimodal information, and accurately trains the encoding and decoding model according to the mutual relationship between multimodal information to obtain a form data processing model, thereby improving the accuracy of automatic recognition of form images.

[0075] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0076] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A form recognition method based on self-attention mechanism, characterized in that: The method comprises: Acquire a form image to be identified, and perform text detection and text recognition on the form image to obtain text position information and text content information of the form image; Reading the pixel value of each pixel in the form image to obtain pixel information of the form image; According to the pixel information, text position information and text content information of the form image, a structured processing result of the form image is obtained through a pre-trained form data processing model, wherein the form data processing model is obtained by training an encoding and decoding model based on a self-attention mechanism, and the structured processing result of the form image includes starting coordinate determination information of the form image, form column name determination information and text position information on at least two sides; The structured processing result of the form image is subjected to form data post-processing to obtain structured form data, specifically including: determining text content information of the starting coordinates of the form image, text content information of each form column name, and text position information on at least two sides of each text content information according to the structured processing result of the form image; determining coordinate information of each text content information according to the text content information of the starting coordinates and the text position information on at least two sides of each text content information; generating structured form data according to the text content information of each form column name and the coordinate information of each text content information.

2. The method according to claim 1, characterized in that The performing text detection and text recognition on the form image to obtain text position information and text content information of the form image includes: Performing text detection on the form image to obtain a text box in the form image; Text recognition is performed on the text box to obtain text position information and text content information of the form image, wherein the text position information includes one-dimensional position information and two-dimensional position information.

3. The method according to claim 1, characterized in that Before reading the pixel value of each pixel in the form picture to obtain pixel information of the form picture, the method further includes: Segment the form image according to a preset image segmentation rule to obtain multiple form image modules; Then, the step of reading the pixel value of each pixel in the form image to obtain pixel information of the form image includes: The pixel value of each pixel of each form picture module is read to obtain the pixel information of each form picture module, wherein the pixel information also includes the one-dimensional position information and the two-dimensional position information of each form picture module.

4. The method according to claim 1, characterized in that: The method of obtaining a structured processing result of the form image by using a pre-trained form data processing model according to the pixel information, text position information and text content information of the form image includes: Performing data conversion preprocessing on pixel information, text position information and text content information of the form image to obtain a feature vector combination of the form image; Preprocessing the feature vector combination of the form image through the self-attention mechanism in the form data processing model to obtain a fused feature vector of the form image; According to the fused feature vector, a structured processing result of the form image is obtained through the form data processing model, wherein the structured processing result of the form image includes starting coordinate determination information of the form image, form column name determination information and text position information on at least two sides.

5. The method according to claim 4, characterized in that The step of performing data conversion preprocessing on the pixel information, text position information and text content information of the form picture to obtain a feature vector combination of the form picture includes: Classify and encode the pixel information and text content information of the form image to obtain classification encoding information of the form image; Respectively convert the pixel information, text position information, text content information and classification coding information of the form image to obtain a graphic feature vector, a text position feature vector, a text content feature vector and a classification coding feature vector of the form image; The image feature vector, text position feature vector, text content feature vector and classification coding feature vector are fused and spliced ​​to obtain a feature vector combination of the form image.

6. The method according to any one of claims 1 to 5, characterized in that: The training method of the form data processing model includes: Obtain pixel information, text position information and text content information of multiple form image samples; Respectively convert the pixel information, text position information and text content information of each form picture sample to obtain a picture feature vector, a text position feature vector and a text content feature vector of each form picture sample; The image feature vectors, text position feature vectors and text content feature vectors of each form image sample are merged and spliced ​​to obtain a feature vector combination of each form image sample; Taking the feature vector combination of each form image sample as input, and taking the image-text matching judgment information, text blackening judgment information and text occlusion prediction information of each form image sample as output, pre-training the encoding and decoding model based on the self-attention mechanism to obtain a pre-trained encoding and decoding model; Taking the feature vector combination of each form image sample as input, taking the starting coordinate determination information, form column name determination information and text position information on at least two sides of each form image sample as output, the pre-trained encoding and decoding model is trained to obtain a form data processing model.

7. A form recognition device based on self-attention mechanism, characterized in that: The device comprises: A text information acquisition module is used to acquire a form image to be identified, and perform text detection and text recognition on the form image to obtain text position information and text content information of the form image; An image information acquisition module is used to read the pixel value of each pixel in the form image to obtain the pixel information of the form image; A structured processing module, used to obtain a structured processing result of the form image through a pre-trained form data processing model according to pixel information, text position information and text content information of the form image, wherein the form data processing model is obtained by training an encoding and decoding model based on a self-attention mechanism, and the structured processing result of the form image includes starting coordinate determination information of the form image, form column name determination information and text position information on at least two sides; The data post-processing module is used to perform form data post-processing on the structured processing result of the form image to obtain structured form data, specifically including: determining the text content information of the starting coordinates of the form image, the text content information of each form column name and the text position information on at least two sides of each text content information according to the structured processing result of the form image; determining the coordinate information of each text content information according to the text content information of the starting coordinates and the text position information on at least two sides of each text content information; generating structured form data according to the text content information of each form column name and the coordinate information of each text content information.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images