Certificate image OCR (optical character recognition) method and device based on semantic correction

Abstract

The embodiment of the invention discloses a certificate image OCR (Optical Character Recognition) method and device based on semantic correction. According to the method, a certificate image OCR recognition model based on semantic correction is provided, the model can simulate the recognition process of human for text content on a low-quality picture, that is, the final recognition result is corrected in the mode that semantic information is combined with visual features, and the accuracy of OCR recognition is improved. The model can be suitable for a certificate image labeling task and an OCR (Optical Character Recognition) task, and is particularly suitable for a low-quality certificate image labeling task and an OCR task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a document image OCR recognition method and device based on semantic correction. Background Art

[0002] The OCR recognition of ID images mainly includes three stages: text line detection, text line recognition and structured parsing. Among them, text line recognition is usually achieved through pre-trained recognition models, which need to be trained with a large number of ID images. However, in the field of ID recognition, there are a large number of low-quality ID images, and it is very difficult to annotate and extract OCR information from such low-quality ID images. Summary of the invention

[0003] One or more embodiments of the present specification provide a method and device for OCR recognition of document images based on semantic correction, which can accurately perform OCR recognition on document images, especially low-quality document images.

[0004] In a first aspect, a training method for an OCR recognition model of a document image based on semantic correction is provided, wherein the OCR recognition model comprises: a visual feature extraction module, a decoder, a parallel visual prompt module and a multimodal semantic reasoning module; the method comprises:

[0005] Inputting the acquired sample image of the document text line into the visual feature extraction module to obtain a visual feature image;

[0006] Inputting the visual feature image into the decoder to obtain semantic information;

[0007] Inputting the visual feature image into the parallel visual prompt module to obtain aligned one-dimensional character features;

[0008] Inputting the one-dimensional character features and semantic information of the sample image of the document text line into the multimodal semantic reasoning module to obtain an OCR prediction result;

[0009] Determining the confidence level of the OCR prediction result;

[0010] When the confidence of the OCR prediction result is less than a preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted; otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

[0011] As an optional implementation manner of the method of the first aspect, obtaining the sample image of the text line of the certificate specifically includes:

[0012] Obtain a sample image of the document;

[0013] Perform text line detection on the certificate sample image to obtain the certificate text line sample image.

[0014] As an optional implementation of the method described in the first aspect, the method may further include:

[0015] A loss function is constructed according to the semantic information of the sample image of the document text line and the real text information of the sample image of the document text line to update the parameters of the visual feature extraction module and the decoder.

[0016] Specifically, the loss function may be a CTC loss function.

[0017] As an optional implementation manner of the method of the first aspect, determining the confidence of the OCR prediction result specifically includes:

[0018] The OCR prediction result is matched with the real text information of the document text line sample image, and the confidence of the OCR prediction result is determined according to the matching result.

[0019] As an optional implementation manner of the method of the first aspect, determining the confidence of the OCR prediction result specifically includes:

[0020] Performing a quality evaluation on the sample image of the certificate text line to obtain an image quality score of the sample image of the certificate text line;

[0021] Matching the semantic information with the real text information of the document text line sample image, and determining a first confidence level of the OCR prediction result according to the matching result;

[0022] Matching the OCR prediction result with the real text information of the document text line sample image, and determining a second confidence level of the OCR prediction result according to the matching result;

[0023] The confidence of the OCR prediction result is determined according to the image quality score, the first confidence and the second confidence.

[0024] As an optional implementation manner of the method of the first aspect, before inputting the semantic information of the sample image of the document text line into the multimodal semantic reasoning module, the method further includes:

[0025] Masking or modifying some characters in the semantic information to obtain tampered semantic information;

[0026] The tampering semantic information and the one-dimensional character features of the document text line sample image are input into the multimodal semantic reasoning module to train the error correction capability of the multimodal semantic reasoning module.

[0027] In a second aspect, a training device for a document image OCR recognition model based on semantic correction is provided, wherein the OCR recognition model comprises: a visual feature extraction module, a decoder, a parallel visual prompt module and a multimodal semantic reasoning module; the device comprises:

[0028] A first data acquisition module is used to acquire a sample image of a text line of a certificate;

[0029] A model training module is used to input the visual feature image into the decoder to obtain semantic information; input the visual feature image into the parallel visual prompt module to obtain aligned one-dimensional character features; input the one-dimensional character features and semantic information of the document text line sample image into the multimodal semantic reasoning module to obtain OCR prediction results;

[0030] The model modification module is used to determine the confidence of the OCR prediction result; when the confidence of the OCR prediction result is less than a preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted; otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

[0031] As an optional implementation manner of the device described in the second aspect, the first data acquisition module is specifically used to acquire a sample image of a certificate; perform text line detection on the sample image of the certificate to obtain the sample image of the certificate text line.

[0032] As an optional implementation of the device described in the second aspect, the model training module is also used to construct a loss function based on the semantic information of the sample image of the certificate text line and the real text information of the sample image of the certificate text line to update the parameters of the visual feature extraction module and the decoder.

[0033] Specifically, the loss function is a CTC loss function.

[0034] As an optional implementation of the device described in the second aspect, the model modification module is specifically used to match the OCR prediction result with the actual text information of the document text line sample image, and determine the confidence of the OCR prediction result based on the matching result.

[0035] As an optional implementation manner of the device according to the second aspect, the model modification module is specifically used to:

[0036] Performing a quality evaluation on the sample image of the certificate text line to obtain an image quality score of the sample image of the certificate text line;

[0037] Matching the semantic information with the real text information of the document text line sample image, and determining a first confidence level of the OCR prediction result according to the matching result;

[0038] Matching the OCR prediction result with the real text information of the document text line sample image, and determining a second confidence level of the OCR prediction result according to the matching result;

[0039] The confidence of the OCR prediction result is determined according to the image quality score, the first confidence and the second confidence.

[0040] As an optional implementation manner of the device according to the second aspect, the device further includes:

[0041] a tampering module, the tampering module being used to cover or modify some characters in the semantic information of the sample image of the document text line before the semantic information is input into the multimodal semantic reasoning module to obtain tampered semantic information;

[0042] The model training module is also used to input the tampering semantic information and the one-dimensional character features of the document text line sample image into the multimodal semantic reasoning module to train the error correction capability of the multimodal semantic reasoning module.

[0043] In a third aspect, a method for OCR recognition of a document image based on semantic correction is provided, the method comprising:

[0044] Obtain the target document image;

[0045] Segmenting a text line image from the target document image;

[0046] Inputting the text line image into an OCR recognition model based on semantic correction to obtain an OCR recognition result of the text line image; the OCR recognition model based on semantic correction is pre-trained according to the training method of the document image OCR recognition model based on semantic correction mentioned above;

[0047] Based on the OCR recognition results of all the text line images of the target document image, the OCR recognition result of the target document image is determined.

[0048] In a fourth aspect, a document image OCR recognition device based on semantic correction is provided, the device comprising:

[0049] A second data acquisition module, used to acquire a target document image;

[0050] A preprocessing module, used for segmenting a text line image from the target document image;

[0051] A recognition module, used for inputting the text line image into an OCR recognition model based on semantic correction to obtain an OCR recognition result of the text line image; the OCR recognition model based on semantic correction is pre-trained according to the training method of the document image OCR recognition model based on semantic correction mentioned above;

[0052] The parsing module is used to determine the OCR recognition result of the target document image based on the OCR recognition results of all text line images of the target document image.

[0053] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned training method of the document image OCR recognition model based on semantic correction, or executes the above-mentioned document image OCR recognition method based on semantic correction.

[0054] In a sixth aspect, an electronic device is provided, including:

[0055] at least one memory for storing a program;

[0056] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned training method of the document image OCR recognition model based on semantic correction, or execute the above-mentioned document image OCR recognition method based on semantic correction.

[0057] The beneficial effect of the training method of the document image OCR recognition model based on semantic correction described in the embodiments of this specification is that the method simulates the human recognition process of text content on low-quality images, that is, it uses semantic information combined with visual features to correct the final recognition result. The document image OCR recognition model trained by this training method can be applied to document image annotation tasks and OCR recognition tasks, especially to low-quality document image annotation tasks and OCR recognition tasks.

[0058] The training device of the document image OCR recognition model based on semantic correction, the document image OCR recognition method and device based on semantic correction described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 The schematic diagram of the structure of an OCR recognition model framework based on semantic correction described in the embodiments of this specification is exemplarily shown.

[0061] Figure 2 The schematic diagram of the structure of a multimodal semantic reasoning module described in the embodiments of this specification is exemplarily shown.

[0062] Figure 3 The flowchart of a method for training a document image OCR recognition model based on semantic correction described in an embodiment of this specification is exemplarily shown.

[0063] Figure 4 The structural diagram of a training device for a document image OCR recognition model based on semantic correction described in an embodiment of this specification is exemplarily shown.

[0064] Figure 5 The flowchart of a document image OCR recognition method based on semantic correction described in an embodiment of this specification is exemplarily shown.

[0065] Figure 6 The schematic diagram of the structure of a document image OCR recognition device based on semantic correction described in the embodiment of this specification is exemplarily shown.

[0066] Figure 7 The structural diagram of an electronic device described in an embodiment of this specification is exemplarily shown. DETAILED DESCRIPTION

[0067] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.

[0068] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0069] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0070] Extracting OCR information from low-quality images has always been a challenge. On the one hand, this is because current OCR recognition models are usually based on visual information recognition, which has poor performance on low-quality image data. On the other hand, it is difficult to annotate low-quality images, which leads to a shortage of such training samples, and ultimately causes the poor performance of OCR recognition models on low-quality images.

[0071] In view of this, one or more embodiments of the present specification propose a document image OCR recognition method and device based on semantic correction, which can use semantic information to correct visual information, thereby improving the accuracy of OCR recognition, and is particularly suitable for OCR recognition and labeling tasks of low-quality images.

[0072] The following will further describe in detail the privacy erasure model training method, privacy erasure method and device described in one or more embodiments of this specification in combination with the drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation on the embodiments of this specification.

[0073] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a structure of an OCR recognition model framework based on semantic correction proposed in one or more embodiments of this specification. Figure 1 The input of the OCR recognition model framework is a text line image, and the output is the OCR recognition result of the text line image, that is, the text information. The OCR recognition model framework mainly includes: a visual feature extraction module, a decoder, a parallel visual prompt module and a multimodal semantic reasoning module.

[0074] The visual feature extraction module is used to extract a two-dimensional visual feature image from the input text line image.

[0075] In some embodiments, the visual feature extraction module can use the Resnet50+FPN structure to extract visual information, and use the Transformer unit to extract and decode the sequence information of the visual information, and finally obtain the two-dimensional visual feature image of the above-mentioned text line image. In the Transformer unit, a self-attention mechanism can also be introduced to realize the multi-layer and multi-directional cascading of sequence information, so as to obtain a visual feature image that can represent the global visual information.

[0076] The decoder is used to decode the two-dimensional visual feature image extracted by the visual feature extraction module to obtain semantic information corresponding to the two-dimensional visual feature image, and the semantic information is in text form.

[0077] The parallel visual cue module is used to obtain the corresponding character features of each target character in the text line image in parallel through the attention mechanism. Specifically, after the visual feature extraction module outputs the two-dimensional visual feature image, the parallel visual cue module will calculate the corresponding attention feature (attention map) for each character in the text line image, and by performing pixel-by-pixel weighted summation with the two-dimensional visual feature image, the one-dimensional character feature corresponding to each target character can be obtained. In addition, the parallel visual cue module also uses the reading order of the characters to replace the hidden variables of the previous moment to guide the calculation of the attention features of the current moment, thereby achieving the purpose of parallel extraction of visual features. Because the calculation method is independent of time, the parallel visual cue module can output the one-dimensional aligned character features of all time in parallel.

[0078] The multimodal semantic reasoning module is used to predict the OCR prediction results based on the semantic information output by the decoder and the one-dimensional character features output by the parallel visual cue module.

[0079] The structure of the multimodal semantic reasoning module can be adaptively selected according to the requirements, and this embodiment does not limit this. For example, the multimodal semantic reasoning module can be implemented by using a dual-tower structure Transformer, a semantic network, etc.

[0080] The following takes the semantic network as an example to illustrate the implementation principle of the multimodal semantic reasoning module.

[0081] In some embodiments, the multimodal semantic reasoning module can adopt Figure 2The semantic network (SemanticNetwork) shown in Figure 1 is implemented. The semantic network uses the Multiway Transformer structure to process text modal data and visual modal data, and performs cross-modal modeling and information fusion. In the Multiway Transformer structure, each layer of Transformer contains a shared self-attention module, a visual expert module, a language expert module and a visual-language expert module. Each expert module is a feedforward neural network, in which the visual expert module is used to extract visual modal features from the input one-dimensional character features, the language expert module is used to extract text modal features from the input semantic information, and the visual-language expert module is used to use a fusion encoder to fuse semantic information feature information and one-dimensional character feature information. The shared self-attention module can effectively learn the alignment of different modal information, and deeply fuse the different modal information to make it better applied to multimodal understanding tasks. The visual expert module, the language expert module and the visual-language expert module perform attention interaction between modalities through the shared information output by the shared self-attention module, and perform information fusion by learning the correlation between modalities.

[0082] visible, Figure 1 The OCR recognition model framework based on semantic correction shown can extract semantic information and visual features of the input text image respectively, and use the semantic information to prompt the model to learn visual features related to semantics, thereby making the OCR recognition result of the model more accurate.

[0083] The above-mentioned semantic correction-based OCR recognition model framework can be applied to the OCR recognition task of text images, and can also be applied to the text image annotation task.

[0084] In the OCR recognition business scenario of ID images, there is a large amount of low-quality, unlabeled image data, and each field of the ID image contains semantic information, such as name, date, address, etc. Therefore, applying the above-mentioned semantic correction-based OCR recognition model framework in the OCR recognition business scenario of ID images can achieve good results.

[0085] Below, taking the document image OCR recognition task as an example, the application principle of the OCR recognition model framework based on semantic correction is explained.

[0086] Please refer to Figure 3 , Figure 3 The flowchart of a method for training a document image OCR recognition model based on semantic correction is shown as an example. Figure 3 As shown, the training method includes steps S300 to S308.

[0087] S300: Inputting the acquired sample image of the document text line into the visual feature extraction module to obtain a visual feature image.

[0088] In some implementations, the above-mentioned certificate text line sample image may be obtained by acquiring a certificate sample image and then performing text line detection on the acquired certificate sample image.

[0089] The above-mentioned document sample images may all be low-quality images, or may partially be low-quality images and the other partially be high-quality images.

[0090] S302: Input the visual feature image into the decoder to obtain semantic information.

[0091] In some implementations, the decoder may be pre-trained based on a small amount of training data so that the decoder learns the ability to parse semantic information.

[0092] Specifically, the sample images of the certificate text lines obtained above can be used as training data, and the real text information of the sample images of the certificate text lines can be used as labels of the sample images of the certificate text lines. A loss function can be constructed based on the difference between the semantic information of the sample images of the certificate text lines and the real text information of the sample images of the certificate text lines to update the parameters of the visual feature extraction module and the decoder.

[0093] The above loss function can be implemented using the CTC loss function, and accordingly, the decoder can be implemented using a softmax layer. The calculation objects of CTC Loss are the output matrix of the visual feature image after softmax normalization and the label matrix corresponding to the real text information. CTC Loss does not require the output matrix to be the same size as the label matrix corresponding to the real text information. The dimension of the output matrix after softmax layer normalization is N×T, where N is the number of categories and T is the number of frames of the input visual feature image.

[0094] S304: Input the visual feature image into a parallel visual prompt module to obtain aligned one-dimensional character features.

[0095] Specifically, the parallel visual cue module has two inputs, one is the visual feature image, and the other is the sample image of the ID text line. The parallel visual cue module will calculate the corresponding attention feature (attention map) for each character in the ID text line sample image, and obtain the one-dimensional character feature corresponding to each character in the ID text line sample image by performing pixel-by-pixel weighted summation with the corresponding visual feature image. Since the calculation method of the one-dimensional character feature by the parallel visual cue module is independent of time, the parallel visual cue module can output the one-dimensional aligned character features of all time in parallel.

[0096] S306: Input the one-dimensional character features and semantic information of the sample image of the document text line into the multimodal semantic reasoning module to obtain the OCR prediction result.

[0097] The above-mentioned multimodal semantic reasoning module is specifically used to extract feature information of corresponding modalities for the input one-dimensional character features and semantic information respectively, and then fuse the extracted feature information of different modalities to predict the OCR prediction result based on the result of feature fusion.

[0098] S308: Determine the confidence of the OCR prediction result; when the confidence of the OCR prediction result is less than a preset confidence threshold, delete the reverse gradient conduction data of the sample image of the document text line, otherwise, retain the reverse gradient conduction data of the sample image of the document text line.

[0099] This training process is a semi-supervised training process. During the training process, the semantic information output by the decoder is used as a pseudo-label to prompt the OCR recognition model to learn semantically related visual features. Considering that the semantic information output by the decoder is not reliable, this embodiment also uses confidence to screen reliable reverse gradient conduction data.

[0100] In some embodiments, the above confidence level can be determined in the following manner:

[0101] The OCR prediction result is matched with the real text information of the document text line sample image, and the confidence of the OCR prediction result is determined based on the matching result.

[0102] Specifically, a corpus information database based on certificates can be constructed. This corpus information can be crawled from open source libraries such as wiki, and the corpus information database contains the real text information of the sample images of certificate text lines. Then the OCR prediction results output by the multimodal semantic reasoning module are matched with the corpus information in the corpus information database to determine the similarity between the OCR prediction results and each corpus information. Different confidence levels can be corresponded according to the similarity value, that is, the higher the similarity, the higher the confidence level.

[0103] After determining the confidence of the OCR prediction result, the reverse gradient conduction data can be screened according to the confidence. That is, when the confidence of the OCR prediction result is less than the preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted; otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

[0104] In some implementations, the above confidence level may also be determined in the following manner:

[0105] Build a corpus of real text information containing sample images of ID text lines.

[0106] The quality of the sample images of the ID text lines is evaluated to obtain the image quality scores of the sample images of the ID text lines.

[0107] Matching the semantic information with the real text information of the sample image of the document text line, and determining the first confidence of the OCR prediction result according to the matching result;

[0108] Matching the OCR prediction result with the real text information of the sample image of the document text line, and determining the second confidence level of the OCR prediction result according to the matching result;

[0109] Finally, the confidence of the OCR prediction result is determined by combining the image quality score, the first confidence level, and the second confidence level.

[0110] Specifically, the image quality score of the sample image of the document text line can be measured first, and this image quality score can be used as the third confidence level.

[0111] If the image quality score of the sample image of the document text line is greater than the preset quality score threshold, the image quality score of the sample image of the document text line will not be used to calculate the final confidence level, that is, the third confidence level is set to 0.

[0112] When calculating the confidence of the OCR prediction result, the maximum value of the first confidence, the second confidence and the third confidence can be taken as the confidence of the OCR prediction result. Alternatively, the confidence of the OCR prediction result can be weighted and summed to obtain the confidence of the OCR prediction result.

[0113] In some embodiments, before inputting the semantic information of the sample image of the certificate text line into the multimodal semantic reasoning module, some characters in the semantic information can also be masked or modified to obtain tampered semantic information, and then the tampered semantic information and the one-dimensional character features of the sample image of the certificate text line are input into the multimodal semantic reasoning module to further train the error correction capability of the multimodal semantic reasoning module.

[0114] Based on the above training method, an ID card image OCR recognition model suitable for the ID card image OCR recognition business scenario can be obtained, and the model is particularly suitable for low-quality ID card images.

[0115] It should be noted that the OCR recognition model obtained based on the above training method is also applicable to the document image annotation business scenario. The specific principle is the same as the above document OCR recognition business scenario and will not be repeated here.

[0116] Corresponding to the above-mentioned training method for the document image OCR recognition model based on semantic correction, this embodiment also provides a training device for the document image OCR recognition model based on semantic correction. Figure 4 , Figure 4The schematic diagram of the structure of a training device for a document image OCR recognition model based on semantic correction is shown as an example. Figure 3 The training method of the document image OCR recognition model based on semantic correction shown in Figure 4 The training device shown implements, but is not limited to, this training device.

[0117] like Figure 4 As shown, the training device of the document image OCR recognition model based on semantic correction includes:

[0118] The first data acquisition module 401 is used to acquire a sample image of a text line of a certificate.

[0119] The model training module 402 is used to input the visual feature image into the decoder to obtain semantic information; input the visual feature image into the parallel visual prompt module to obtain aligned one-dimensional character features; input the one-dimensional character features and semantic information of the document text line sample image into the multimodal semantic reasoning module to obtain OCR prediction results.

[0120] The model modification module 403 is used to determine the confidence of the OCR prediction result; when the confidence of the OCR prediction result is less than a preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted, otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

[0121] The first data acquisition module 401 is specifically used to acquire a sample document image and perform text line detection on the acquired sample document image to obtain a sample document text line image. The sample document image may be entirely low-quality images, or may be partially low-quality images and partially high-quality images.

[0122] For the model training module 402, before training the document image OCR recognition model based on semantic correction, the decoder can also be pre-trained using a small amount of training data to enable the decoder to learn the ability to parse semantic information.

[0123] Specifically, the sample images of the certificate text lines obtained by the above-mentioned first data acquisition module 401 can be used as training data, and the real text information of the sample images of the certificate text lines can be used as labels of the sample images of the certificate text lines. A loss function is constructed according to the difference between the semantic information of the sample images of the certificate text lines and the real text information of the sample images of the certificate text lines to update the parameters of the visual feature extraction module and the decoder.

[0124] The above loss function can be implemented using the CTC loss function, and accordingly, the decoder can be implemented using a softmax layer. The calculation objects of CTC Loss are the output matrix of the visual feature image after softmax normalization and the label matrix corresponding to the real text information. CTC Loss does not require the output matrix to be the same size as the label matrix corresponding to the real text information. The dimension of the output matrix after softmax layer normalization is N×T, where N is the number of categories and T is the number of frames of the input visual feature image.

[0125] As for the model modification module 403, the model modification module 403 is specifically used to match the OCR prediction result with the real text information of the document text line sample image, and determine the confidence of the OCR prediction result according to the matching result.

[0126] In some implementations, the above-mentioned model modification module 403 may specifically determine the confidence level in the following manner: matching the OCR prediction result with the actual text information of the document text line sample image, and determining the confidence level of the OCR prediction result based on the matching result.

[0127] Specifically, the model modification module 403 can pre-build a corpus information library based on the certificate. This corpus information can be crawled from open source libraries such as wiki, and the corpus information library also needs to contain the real text information of the sample image of the certificate text line. Then the OCR prediction result output by the multimodal semantic reasoning module is matched with the corpus information in the corpus information library to determine the similarity between the OCR prediction result and each corpus information. Different confidence levels can be corresponded according to the value of the similarity, that is, the higher the similarity, the higher the confidence level. After determining the confidence level of the OCR prediction result, the reverse gradient conduction data can be screened according to the confidence level, that is, when the confidence level of the OCR prediction result is less than the preset confidence level threshold, the reverse gradient conduction data of the sample image of the certificate text line is deleted, otherwise, the reverse gradient conduction data of the sample image of the certificate text line is retained.

[0128] In some implementations, the above-mentioned model modification module 403 may also specifically determine the confidence level in the following manner:

[0129] The model modification module 403 can pre-construct a corpus information library containing real text information of the sample images of the certificate text lines. Then, the quality of the sample images of the certificate text lines is evaluated to obtain the image quality score of the sample images of the certificate text lines. Next, the semantic information is matched with the real text information of the sample images of the certificate text lines, and the first confidence of the OCR prediction result is determined according to the matching result. The OCR prediction result is matched with the real text information of the sample images of the certificate text lines, and the second confidence of the OCR prediction result is determined according to the matching result. Finally, the confidence of the OCR prediction result is determined by combining the image quality score, the first confidence and the second confidence.

[0130] Specifically, the model modification module 403 may first measure the image quality score of the sample image of the document text line, and use the image quality score as the third confidence level. If the image quality score of the sample image of the document text line is greater than a preset quality score threshold, the image quality score of the sample image of the document text line is not used in calculating the final confidence level, that is, the third confidence level is set to 0.

[0131] When calculating the confidence of the OCR prediction result, the model modification module 403 may take the maximum of the first confidence, the second confidence and the third confidence as the confidence of the OCR prediction result. Alternatively, the confidence of the OCR prediction result may be weighted and summed to obtain the confidence of the OCR prediction result.

[0132] In addition, in some more specific embodiments, the above-mentioned training device may also include a tampering module, which is used to cover or modify some characters in the semantic information before the semantic information of the sample image of the document text line is input into the multimodal semantic reasoning module to obtain tampered semantic information.

[0133] Corresponding to the tampering module, the model training module 402 is also used to input the obtained tampering semantic information and the one-dimensional character features of the document text line sample image into the multimodal semantic reasoning module to further train the error correction capability of the multimodal semantic reasoning module.

[0134] For the above-mentioned training device of the document image OCR recognition model based on semantic correction, taking the module as an example of a software functional unit, the first data acquisition module 401 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first data acquisition module 401 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0135] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0136] As an example of a hardware functional unit, the first data acquisition module 401 may include at least one computing device, such as a server, etc. Alternatively, the first data acquisition module 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0137] The multiple computing devices included in the first data acquisition module 401 can be distributed in the same region or in different regions. The multiple computing devices included in the first data acquisition module 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first data acquisition module 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0138] In other embodiments, the first data acquisition module 401 can be used to execute any step in the training method of the document image OCR recognition model based on semantic correction, the model training module 402 can be used to execute any step in the training method of the document image OCR recognition model based on semantic correction, and the model modification module 403 can be used to execute any step in the training method of the document image OCR recognition model based on semantic correction. The steps that the first data acquisition module 401, the model training module 402, and the model modification module 403 are responsible for implementing can be specified as needed, and the first data acquisition module 401, the model training module 402, and the model modification module 403 respectively implement different steps in the training method of the document image OCR recognition model based on semantic correction to realize all the functions of the training device of the document image OCR recognition model based on semantic correction.

[0139] In this implementation, the training device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the training function of the document image OCR recognition model based on semantic correction.

[0140] Based on the training of the above-mentioned document image OCR recognition model based on semantic correction, the document image OCR recognition model based on semantic correction is trained, and this embodiment further proposes a document image OCR recognition method based on semantic correction.

[0141] like Figure 5 As shown, in some embodiments, the document image OCR recognition method based on semantic correction may include steps S500 to S506:

[0142] S500: Acquire a target document image.

[0143] The target certificate mentioned above may be a certificate that can indicate the identity of the user, specifically an ID card, passport, driver's license, etc.

[0144] The above-mentioned target document image can be obtained by collecting images of physical documents and crawling from public databases.

[0145] After acquiring the target document images, a large number of acquired target document images may be preprocessed and corrected to process all target document images into images of a standard size.

[0146] S502: Segment a text line image from the target document image.

[0147] Specifically, the text line image can be segmented from the target document image by means of text line detection.

[0148] S504: Input the text line image into the OCR recognition model based on semantic correction to obtain the OCR recognition result of the text line image.

[0149] The above-mentioned OCR recognition model based on semantic correction is pre-trained based on the above-mentioned training method of the document image OCR recognition model based on semantic correction.

[0150] S506: Determine the OCR recognition result of the target document image based on the OCR recognition results of all text line images of the target document image.

[0151] After obtaining the OCR recognition results of all the text line images of the target document image, these OCR recognition results can be structurally analyzed according to the positions of these text line images in the target document image, so as to obtain the OCR recognition result of the target document image.

[0152] Corresponding to the above-mentioned document image OCR recognition method based on semantic correction, this embodiment also provides a document image OCR recognition device based on semantic correction. Figure 6 , Figure 6 The exemplary embodiment shows a document image OCR recognition device based on semantic correction, which can be used to implement the document image OCR recognition method based on semantic correction. It should be noted that the document image OCR recognition based on semantic correction described in one or more embodiments of the present application can rely on Figure 6 The identification device shown implements, but is not limited to, this device.

[0153] like Figure 6 As shown, the identification device comprises:

[0154] The second data acquisition module 601 is used to acquire the target document image.

[0155] The pre-processing module 602 is used to segment the text line image from the target document image.

[0156] The recognition module 603 is used to input the text line image into the OCR recognition model based on semantic correction to obtain the OCR recognition result of the text line image.

[0157] The parsing module 604 is used to determine the OCR recognition result of the target document image based on the OCR recognition results of all text line images of the target document image.

[0158] For the second data acquisition module 601, the target certificate it acquires can be a certificate that can indicate the identity of the user, specifically an ID card, passport, driver's license, etc. The second data acquisition module 601 can acquire the above-mentioned target certificate image by collecting images of physical certificates and crawling from public databases.

[0159] After acquiring the target document images, the second data acquisition module 601 may also perform preprocessing and correction on a large number of acquired target document images, and process all target document images into images of a standard size.

[0160] As for the pre-processing module 602, the module can segment the text line image from the target document image by means of text line detection.

[0161] As for the recognition module 603, the OCR recognition model based on semantic correction adopted by the module is pre-trained based on the training method of the document image OCR recognition model based on semantic correction mentioned above.

[0162] The module can load a trained document image OCR recognition model from a training device for a document image OCR recognition model based on semantic correction, so as to recognize a text line image of a target document image.

[0163] With respect to the parsing module 604, after obtaining the OCR recognition results of all text line images of the target document image, the module can perform structured parsing on these OCR recognition results according to the positions of these text line images in the target document image, thereby obtaining the OCR recognition result of the target document image.

[0164] For the above-mentioned document image OCR recognition device based on semantic correction, taking the module as an example of a software functional unit, the second data acquisition module 601 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the second data acquisition module 601 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0165] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0166] As an example of a hardware functional unit, the second data acquisition module 601 may include at least one computing device, such as a server, etc. Alternatively, the second data acquisition module 601 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0167] The multiple computing devices included in the second data acquisition module 601 can be distributed in the same region or in different regions. The multiple computing devices included in the second data acquisition module 601 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the second data acquisition module 601 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0168] In other embodiments, the second data acquisition module 601 can be used to execute any step in the above-mentioned document image OCR recognition method based on semantic correction, the preprocessing module 602 can be used to execute any step in the above-mentioned document image OCR recognition method based on semantic correction, the recognition module 603 can be used to execute any step in the above-mentioned document image OCR recognition method based on semantic correction, and the parsing module 604 can be used to execute any step in the above-mentioned document image OCR recognition method based on semantic correction. The steps that the second data acquisition module 601, the preprocessing module 602, the recognition module 603 and the parsing module 604 are responsible for implementing can be specified as needed, and the second data acquisition module 601, the preprocessing module 602, the recognition module 603 and the parsing module 604 respectively implement different steps in the above-mentioned document image OCR recognition method based on semantic correction to realize all the functions of the above-mentioned document image OCR recognition device based on semantic correction.

[0169] In this implementation, the document image OCR recognition device based on semantic correction can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the document image OCR recognition function.

[0170] In some embodiments, an electronic device is also provided. Figure 7 , the electronic device includes: a bus 701, a processor 702, a memory 703 and a communication interface 704. The processor 702, the memory 703 and the communication interface 704 communicate with each other through the bus 701. The electronic device can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the electronic device.

[0171] The bus 701 may be a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The use of only one line does not mean that there is only one bus or one type of bus. Bus 701 may include a path for transmitting information between various components of the electronic device (eg, processor 702, memory 703, and communication interface 704).

[0172] The processor 702 may include any one or more of a processor such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0173] The memory 703 may include a volatile memory, such as a random access memory (RAM). The memory 703 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0174] The memory 703 stores executable program codes, and the processor 702 executes the executable program codes to respectively implement the functions of the aforementioned first data acquisition module 401, the model training module 402, and the model modification module 403, that is, to implement the functions of the aforementioned training device for the document image OCR recognition model based on semantic correction, thereby implementing the aforementioned training method for the document image OCR recognition model based on semantic correction. Alternatively, the processor 702 executes the executable program codes to respectively implement the functions of the aforementioned second data acquisition module 601, the preprocessing module 602, the recognition module 603, and the parsing module 604, that is, to implement the functions of the aforementioned document image OCR recognition device based on semantic correction, thereby implementing the aforementioned document image OCR recognition method based on semantic correction.

[0175] The communication interface 704 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the electronic device and other devices or a communication network.

[0176] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned training method of the document image OCR recognition model based on semantic correction, or implements the above-mentioned document image OCR recognition method based on semantic correction.

[0177] The computer-readable storage medium may be any available medium that can be stored in an electronic device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the electronic device to execute the above-mentioned training method of the document image OCR recognition model based on semantic correction, or to execute the above-mentioned document image OCR recognition method based on semantic correction.

[0178] It is to be understood that the structure illustrated in the embodiments of this specification does not constitute a specific limitation on the system of the embodiments of this specification. In other embodiments of the specification, the above system may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0179] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0180] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0181] It should be noted that the above examples are only specific embodiments of the present invention, and the present invention is obviously not limited to the above examples, and there are many similar variations. All variations directly derived or associated from the contents disclosed by the technicians in this field should fall within the protection scope of the present invention.

Claims

1. A training method for a document image OCR recognition model based on semantic correction, the OCR recognition model comprising: A visual feature extraction module, a decoder, a parallel visual cue module and a multimodal semantic reasoning module; the method comprises: Inputting the acquired sample image of the document text line into the visual feature extraction module to obtain a visual feature image; Inputting the visual feature image into the decoder to obtain semantic information; Inputting the visual feature image into the parallel visual prompt module to obtain aligned one-dimensional character features; Inputting the one-dimensional character features and semantic information of the sample image of the document text line into the multimodal semantic reasoning module to obtain an OCR prediction result; Determining the confidence level of the OCR prediction result; When the confidence of the OCR prediction result is less than a preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted; otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

2. The method according to claim 1, obtaining the sample image of the document text line, specifically comprising: Obtain a sample image of the document; Perform text line detection on the certificate sample image to obtain the certificate text line sample image.

3. The method of claim 1, further comprising: A loss function is constructed according to the semantic information of the sample image of the document text line and the real text information of the sample image of the document text line to update the parameters of the visual feature extraction module and the decoder.

4. The method according to claim 3, wherein the loss function is a CTC loss function.

5. The method of claim 1, determining the confidence of the OCR prediction result, specifically comprising: The OCR prediction result is matched with the real text information of the document text line sample image, and the confidence of the OCR prediction result is determined according to the matching result.

6. The method of claim 1, determining the confidence of the OCR prediction result, specifically comprising: Performing a quality evaluation on the sample image of the certificate text line to obtain an image quality score of the sample image of the certificate text line; Matching the semantic information with the real text information of the document text line sample image, and determining a first confidence level of the OCR prediction result according to the matching result; Matching the OCR prediction result with the real text information of the document text line sample image, and determining a second confidence level of the OCR prediction result according to the matching result; The confidence of the OCR prediction result is determined according to the image quality score, the first confidence and the second confidence.

7. The method according to claim 1, before inputting the semantic information of the sample image of the certificate text line into the multimodal semantic reasoning module, further comprises: Masking or modifying some characters in the semantic information to obtain tampered semantic information; The tampering semantic information and the one-dimensional character features of the document text line sample image are input into the multimodal semantic reasoning module to train the error correction capability of the multimodal semantic reasoning module.

8. A training device for a document image OCR recognition model based on semantic correction, the OCR recognition model comprising: Visual feature extraction module, decoder, parallel visual cue module and multimodal semantic reasoning module; The device comprises: A first data acquisition module is used to acquire a sample image of a text line of a certificate; A model training module is used to input the visual feature image into the decoder to obtain semantic information; input the visual feature image into the parallel visual prompt module to obtain aligned one-dimensional character features; input the one-dimensional character features and semantic information of the document text line sample image into the multimodal semantic reasoning module to obtain OCR prediction results; The model modification module is used to determine the confidence of the OCR prediction result; when the confidence of the OCR prediction result is less than a preset confidence threshold, the reverse gradient conduction data of the sample image of the document text line is deleted; otherwise, the reverse gradient conduction data of the sample image of the document text line is retained.

9. The device according to claim 8, wherein the first data acquisition module is specifically used to acquire a sample image of a certificate; and perform text line detection on the sample image of the certificate to obtain the sample image of the certificate text line.

10. In the device as described in claim 8, the model training module is also used to construct a loss function based on the semantic information of the sample image of the ID text line and the real text information of the sample image of the ID text line to update the parameters of the visual feature extraction module and the decoder.

11. The device of claim 10, wherein the loss function is a CTC loss function.

12. The device as claimed in claim 8, wherein the model modification module is specifically used to match the OCR prediction result with the real text information of the document text line sample image, and determine the confidence of the OCR prediction result according to the matching result.

13. The device according to claim 8, wherein the model modification module is specifically used for: Performing a quality evaluation on the sample image of the certificate text line to obtain an image quality score of the sample image of the certificate text line; Matching the semantic information with the real text information of the document text line sample image, and determining a first confidence level of the OCR prediction result according to the matching result; Matching the OCR prediction result with the real text information of the document text line sample image, and determining a second confidence level of the OCR prediction result according to the matching result; The confidence of the OCR prediction result is determined according to the image quality score, the first confidence and the second confidence.

14. The apparatus of claim 8, further comprising: a tampering module, the tampering module being used to cover or modify some characters in the semantic information of the sample image of the document text line before the semantic information is input into the multimodal semantic reasoning module to obtain tampered semantic information; The model training module is also used to input the tampering semantic information and the one-dimensional character features of the document text line sample image into the multimodal semantic reasoning module to train the error correction capability of the multimodal semantic reasoning module.

15. A document image OCR recognition method based on semantic correction, comprising: Obtain the target document image; Segmenting a text line image from the target document image; Inputting the text line image into an OCR recognition model based on semantic correction to obtain an OCR recognition result of the text line image; the OCR recognition model based on semantic correction is pre-trained according to the method according to any one of claims 1 to 7; Based on the OCR recognition results of all the text line images of the target document image, the OCR recognition result of the target document image is determined.

16. A document image OCR recognition device based on semantic correction, comprising: A second data acquisition module, used to acquire a target document image; A preprocessing module, used for segmenting a text line image from the target document image; A recognition module, used for inputting the text line image into an OCR recognition model based on semantic correction to obtain an OCR recognition result of the text line image; the OCR recognition model based on semantic correction is pre-trained according to the method according to any one of claims 1 to 7; The parsing module is used to determine the OCR recognition result of the target document image based on the OCR recognition results of all text line images of the target document image.

17. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7, or the method according to claim 15.

18. An electronic device, comprising: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7, or execute the method according to claim 15.

Images