Multi-modal certificate information collaborative identification method and system based on OCR technology

Abstract

The invention discloses a multi-mode certificate information collaborative identification method and system based on an OCR technology. The method comprises the steps that certificate image information is acquired; detecting feature information of an MRZ code and a two-dimensional code in the certificate image information through a detection model; classifying the feature information of the MRZ code and the two-dimensional code through a classification model, and outputting data images of the MRZ code and the two-dimensional code; and performing MRZ code identification and two-dimensional code identification through the identification model, and outputting a document image text identification result based on the position and the identification text. Synchronous detection and classification of two-dimensional codes, MRZ codes and characters are realized in a multi-code cooperation mode, so that processing steps are reduced, and the efficiency is improved; based on the reconstruction loss function, class weight and bounding box regression constraint are introduced, and the multi-target detection precision is remarkably improved; and a special image acquisition and processing module is designed to adapt to a complex imaging environment.

Description

Technical Field

[0001] The present invention relates to a multimodal document information collaborative recognition method and system based on OCR technology, and belongs to the technical field of image recognition. Background Art

[0002] With the rapid development of optical character recognition (OCR) technology, its application in document recognition is becoming increasingly widespread. Traditional documents (such as ID cards, passports, and business licenses) typically contain multiple information carriers, including text, QR codes, and machine-readable zones (MRZ codes). However, existing technologies still suffer from low efficiency, insufficient accuracy, and poor hardware and software interoperability when processing multimodal information. An efficient, high-precision, integrated recognition solution is urgently needed.

[0003] OCR technology, which extracts text through image processing and pattern recognition, has made significant progress in document recognition. For example, OCR document readers can quickly identify machine-readable codes on passports, ID cards, and other documents with an accuracy rate exceeding 99.9% and a word processing time of less than 2 seconds. However, existing technologies often focus on processing a single type of information:

[0004] 1. Text recognition: This method uses fixed template matching or deep learning algorithms, but is not adaptable enough to complex backgrounds or blurred images.

[0005] 2. QR code parsing: This relies on a dedicated decoding algorithm and deeply integrates the unit and text recognition processes. This results in distributed processing when multiple codes coexist, leading to low efficiency.

[0006] 3. MRZ code recognition: It relies on high-precision positioning and a dedicated character set model, and is sensitive to interference such as image tilt and uneven lighting.

[0007] Furthermore, existing OCR models, such as the EAST (Efficient and Accuracy Scene Text) text detection model, have limited performance in complex document images. Their loss function design is not fully optimized for multi-task scenarios (such as simultaneous detection of text, QR codes, and MRZ codes), resulting in insufficient positioning accuracy and high false positive rates.

[0008] Modern IDs often use a composite design of "text + QR code + MRZ code" to enhance information capacity and security. For example, a business license may include a unified social credit code and an anti-counterfeiting QR code, while a passport must simultaneously process an MRZ code and biometric information. However, existing solutions have the following problems:

[0009] 1. Low distributed processing efficiency: Traditional processes require calling independent modules such as text recognition and QR code decoding in sequence, resulting in increased processing delays.

[0010] 2. Insufficient classification and positioning accuracy: In mixed-code scenarios, the model is prone to misjudging overlapping areas, such as misclassifying a QR code as a text area or missing an MRZ code.

[0011] 3. Poor hardware adaptability: Most solutions rely on general-purpose computing devices and are unable to meet the real-time requirements of mobile or embedded scenarios.

[0012] While existing models (such as the EAST text detection model) perform well in text detection, their loss functions lack the ability to distinguish between multiple target classes (such as QR codes and MRZ codes), and the training data is not optimized for document scenarios, resulting in poor generalization in complex backgrounds. Furthermore, traditional training methods fail to fully utilize synthetic data augmentation techniques, making it difficult to cover display interference scenarios such as low light, stains, and tilt. Furthermore, existing OCR systems are mostly software-based, with relatively simple hardware designs. For example, while ID card scanners can quickly generate images, they cannot simultaneously adapt to multiple document types and lack dedicated image processing modules for DORMA recognition. Summary of the Invention

[0013] This invention provides a method and system for collaborative multimodal document information recognition based on OCR technology. By improving the model structure, it enables simultaneous detection and classification of QR and MRZ codes, reducing processing steps and improving efficiency. It also restructures the loss function, introduces class weights and bounding box regression constraints, and significantly improves multi-target detection accuracy. Furthermore, it designs a dedicated image acquisition and processing module to adapt to complex imaging environments. This method addresses the problems discussed in the background art.

[0014] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0015] The multimodal document information collaborative recognition method based on OCR technology includes:

[0016] Obtain document image information;

[0017] Input the document image information into the pre-built detection model to detect the feature information of the MRZ code and QR code in the document image information;

[0018] Input the feature information of the MRZ code and the QR code into a pre-built classification model, classify the feature information of the MRZ code and the QR code, and output the data images of the MRZ code and the QR code;

[0019] The data images of MRZ code and QR code are input into the pre-built recognition model, and MRZ code recognition and QR code recognition are performed respectively, and the document image text recognition results based on position and recognition text are output.

[0020] Furthermore, the detection model includes: a feature extraction layer, a feature fusion layer and an output layer; the method for detecting the feature information of the MRZ code and the QR code in the document image information includes:

[0021] The output channels of the feature extraction layer are 64, 128, 256, and 256, respectively, to construct multi-scale feature map information. Each layer is downsampled through the maximum pooling layer to obtain a list of four types of feature maps.

[0022] The feature fusion layer performs feature fusion through three upsampling operations, generating a unified high-resolution feature map from the feature map list information. Starting from the feature map obtained from the last step with 256 channels, it is upsampled through bilinear interpolation to align the spatial size with the previous feature map, and spliced ​​on the corresponding channel dimension to generate a high-dimensional feature map containing two scales. After that, two convolution and activation operations are performed. After completing the three upsampling operations, the convolution activation operation reduces the number of channels to 1 / 4 of the original size image.

[0023] The output layer sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function to obtain text score information; sets the feature map channel to 4 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and scales the value output by the Sigmoid function to the actual image size to obtain text position information, then sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and then maps the value of the Sigmoid output to the range of [-Π / 2, Π / 2] to obtain the rotation angle of the text box, and splices the position information and angle information to generate the geometric box information of the 5-channel text area.

[0024] Furthermore, the feature information of the MRZ code and the QR code in the document image information is detected, and further includes:

[0025] High-probability text information is retained through threshold filtering, and overlapping detection frames are removed through the non-maximum suppression algorithm to output the precise text area; by traversing each predicted polygon frame, based on the set threshold and calculating the overlapping area and geometric similarity between polygons in the group; if the threshold condition is met, the polygon frames are merged, otherwise the high-scoring frames are retained; and the geometric frame information of the text area is output.

[0026] Furthermore, the classification model includes a feature extraction layer and an output layer; the method for classifying the feature information of MRZ codes and QR codes includes:

[0027] The initial convolution layer of the feature extraction layer is a 3x3 convolution layer, which reduces the input image size to 112x112x16. An inverted residual structure is constructed based on modules such as the expansion layer, depthwise separable convolution, SE attention module, global average pooling layer, and fully connected layer, which processes the input image size to 1x1x1280. Two global average pooling layers are used to process the input features into a one-dimensional vector of dimension 1280.

[0028] The one-dimensional vector with a dimension of 1280 is predicted and classified to generate a probability distribution dimension based on the number of categories, and the corresponding category label and confidence level are output; the QR code rectangle in the original image is removed to obtain a single image of the text information image and the QR code.

[0029] Furthermore, the recognition model includes: a CNN model, an RNN model, and a CTC decoding module; and the method for performing MRZ code recognition and QR code recognition respectively and outputting an identification label based on the position and recognition result includes:

[0030] The CNN model sets the height of a single image of text information image and QR code to 1 and the width to 1 / 4 of the original image;

[0031] The RNN model expands the image adjusted by the CNN model into time steps along the width direction, and changes the output dimension to [time step, batch, character category] through the bidirectional LSTM and fully connected layers;

[0032] The CTC decoding module converts the time steps output by the RNN model into a probability distribution through Softmax normalization, removes blank characters, and selects the character sequence with the highest probability to generate the recognized text; it organizes the text in the form of position-recognition text and outputs the document image text recognition results.

[0033] Furthermore, the method for constructing the detection model includes:

[0034] A detection model is built based on the Pytorch platform. The detection layer uses the VGG model as the backbone network. The number of input channels of the first layer of the VGG model configuration is adjusted to 32, and the number of channels of the last layer is set to 256. The merging layer fuses the feature maps of different levels based on upsampling and downsampling operations, and splices the fused feature maps with the original feature maps. The output layer integrates the information of the detection layer and the merging layer, outputs the text area score and geometric information. The text area score is used to represent the confidence of the detected text area, removes overlapping polygon detection frames, and outputs the accurate text area. By traversing each predicted polygon detection frame, based on the set threshold, the overlapping area and geometric similarity between polygons in the group are calculated. If the threshold condition is met, the polygon detection frame is merged, otherwise the high-scoring frame is retained, thereby eliminating unreasonable polygon detection frames. The polygon vertex distance constraint formula of the polygon detection frame is:

[0035] ;

[0036] Where D vertex is the vertex distance under the constraint conditions, v is the calculated polygon; if D vertex < the set threshold, the two polygon detection frames are considered overlapping;

[0037] The formula for merging polygon detection boxes is:

[0038] ;

[0039] Where V merged is the vertex distance, G is the polygon set of the adjacent group, s i is the classification score of the i-th polygon, v i is the vertex coordinate of the i-th polygon; the confidence after updating and merging is the highest score in the group.

[0040] Furthermore, the method for constructing the detection model includes:

[0041] A classification model is built based on Pytorch, and the SE attention module is introduced. Channel weights are learned through global average pooling and two fully connected layers. The SE attention module is only used in some inverted residual blocks, and the number of parameters is reduced by reducing the number of channels in the intermediate layers. The h-swish activation function is selected to achieve the effect of the swish function while avoiding exponential operations. The dilated convolution, depthwise separable convolution, SE attention module and skip connection are combined through modular design.

[0042] Furthermore, the method for constructing the recognition model includes:

[0043] A recognition model is built based on pytorch; a CNN model is constructed, which has four maximum pooling layers. The window size of the last two pooling layers is changed from 2x2 to 1x2. The input image size is (1, 32, w), where the input image height is fixed at 32. After being processed by the CNN model, the image size is (512, 1, w / 4); an RNN model is constructed, which uses a bidirectional recurrent network and utilizes forward and backward context information to capture the contextual dependency of the sequence. The input image size is (w / 4, 512) and the output sequence is (w / 4, 5835). The character set selected for the training model includes Chinese, English, numbers, and characters, with a total of 5835 characters; a CTC decoding module is constructed to map the probability sequence output by the RNN model to recognized text. Duplicate characters are merged and whitespace is ignored through dynamic programming. At the same time, the loss function directly maximizes the probability of the correct label sequence without the need for labeling and output steps, thereby solving the alignment problem.

[0044] Furthermore, before training the detection model, classification model, and recognition model, balanced sample data is obtained by:

[0045] Based on the annotated labels, the inward bias is set to 0.2~0.3, and the Dice loss function is used to give more attention to pixels of fewer categories. In order to strengthen the foreground loss, a joint loss function is designed based on the weighted cross entropy and weighted Dice loss functions.

[0046] The weighted cross entropy function is: , where is the cross entropy loss function, Predict the probability of foreground pixels, is the true label of the foreground pixel, is the predicted probability of background pixels, is the true label of the background pixel;

[0047] The weighted Dice loss function is: , is the Dice loss function;

[0048] The joint loss function is:

[0049] , Expressed as the weight between loss functions, is the proportion of foreground pixel value.

[0050] A second aspect of the present invention provides a multimodal document information collaborative recognition system based on OCR technology, comprising:

[0051] An information acquisition module, used to obtain document image information;

[0052] A detection module is used to detect the characteristic information of the MRZ code and the QR code in the document image information;

[0053] A classification module is used to classify the feature information of MRZ codes and QR codes and output data images of MRZ codes and QR codes;

[0054] The recognition module is used to perform MRZ code recognition and QR code recognition respectively, and output the document image text recognition results based on the position and recognition text.

[0055] The beneficial effects achieved by the present invention are:

[0056] 1. By combining QR code detection with MRZ code detection, the interference of QR codes on text recognition is effectively avoided. The detected data is classified through the classification model, and combined with the QR code decoding model, multi-code collaborative detection and classification are achieved.

[0057] 2. Optimize and improve the deep learning model and deploy it through the ONNX (Open Neural Network Exchange) model to achieve detection, classification, and recognition of MRZ code and QR code information, greatly improving the speed and accuracy of document recognition containing QR codes. At the same time, the present invention is portable and can be deployed and reasoned based on domestic platforms.

[0058] 3. By adjusting the number of channels and feature merging layers of the detection model, the improved detection model is only 20M; the designed classification model is only 10M. The overall model size is relatively small, which allows for better deployment and application implementation.

[0059] In summary, the present invention achieves confidentiality and technical autonomy and controllability, and has good recognition effects for MRZ codes and QR codes, and has high robustness for document recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of the identification method of the present invention;

[0061] Figure 2 A schematic diagram of the model construction of the present invention;

[0062] Figure 3 It is a practical application device of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] Example 1:

[0065] like Figure 1 As shown, this embodiment provides a multimodal document information collaborative recognition method based on OCR technology, including:

[0066] Obtain document image information;

[0067] Input the document image information into the pre-built detection model to detect the feature information of the MRZ code and QR code in the document image information;

[0068] Input the feature information of the MRZ code and the QR code into a pre-built classification model, classify the feature information of the MRZ code and the QR code, and output the data images of the MRZ code and the QR code;

[0069] The data images of MRZ code and QR code are input into the pre-built recognition model, and MRZ code recognition and QR code recognition are performed respectively, and the document image text recognition results based on position and recognition text are output.

[0070] The detection model includes: a feature extraction layer, a feature fusion layer and an output layer; the method for detecting the feature information of the MRZ code and the QR code in the document image information includes:

[0071] The output channels of the feature extraction layer are 64, 128, 256, and 256, respectively, to construct multi-scale feature map information. Each layer is downsampled through the maximum pooling layer to obtain a list of four types of feature maps.

[0072] The feature fusion layer performs feature fusion through three upsampling operations, generating a unified high-resolution feature map from the feature map list information. Starting from the feature map obtained from the last step with 256 channels, it is upsampled through bilinear interpolation to align the spatial size with the previous feature map, and spliced ​​on the corresponding channel dimension to generate a high-dimensional feature map containing two scales. After that, two convolution and activation operations are performed. After completing the three upsampling operations, the convolution activation operation reduces the number of channels to 1 / 4 of the original size image.

[0073] The output layer sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function to obtain text score information; sets the feature map channel to 4 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and scales the value output by the Sigmoid function to the actual image size to obtain text position information, then sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and then maps the value of the Sigmoid output to the range of [-Π / 2, Π / 2] to obtain the rotation angle of the text box, and splices the position information and angle information to generate the geometric box information of the 5-channel text area.

[0074] The characteristic information of MRZ code and QR code in the document image information is also included:

[0075] High-probability text information is retained through threshold filtering, and overlapping detection frames are removed through the non-maximum suppression algorithm to output the precise text area; by traversing each predicted polygon frame, based on the set threshold and calculating the overlapping area and geometric similarity between polygons in the group; if the threshold condition is met, the polygon frames are merged, otherwise the high-scoring frames are retained; and the geometric frame information of the text area is output.

[0076] The classification model includes a feature extraction layer and an output layer; the method for classifying the feature information of MRZ codes and QR codes includes:

[0077] The initial convolution layer of the feature extraction layer is a 3x3 convolution layer, which reduces the input image size to 112x112x16. An inverted residual structure is constructed based on modules such as the expansion layer, depthwise separable convolution, SE attention module, global average pooling layer, and fully connected layer, which processes the input image size to 1x1x1280. Two global average pooling layers are used to process the input features into a one-dimensional vector of dimension 1280.

[0078] The one-dimensional vector with a dimension of 1280 is predicted and classified to generate a probability distribution dimension based on the number of categories, and the corresponding category label and confidence level are output; the QR code rectangle in the original image is removed to obtain a single image of the text information image and the QR code.

[0079] The recognition model includes: a CNN model, an RNN model, and a CTC decoding module; the method for performing MRZ code recognition and QR code recognition respectively, and outputting an identification label based on the position and recognition result includes:

[0080] The CNN model sets the height of a single image of text information image and QR code to 1 and the width to 1 / 4 of the original image;

[0081] The RNN model expands the image adjusted by the CNN model into time steps along the width direction, and changes the output dimension to [time step, batch, character category] through the bidirectional LSTM and fully connected layers;

[0082] The CTC decoding module converts the time steps output by the RNN model into a probability distribution through Softmax normalization, removes blank characters, and selects the character sequence with the highest probability to generate the recognized text; it organizes the text in the form of position-recognition text and outputs the document image text recognition results.

[0083] Example 2:

[0084] Based on the concept of Example 1, this embodiment provides a multimodal document information collaborative recognition system based on OCR technology, including:

[0085] An information acquisition module, used to obtain document image information;

[0086] A detection module is used to detect the characteristic information of the MRZ code and the QR code in the document image information;

[0087] A classification module is used to classify the feature information of MRZ codes and QR codes and output data images of MRZ codes and QR codes;

[0088] The recognition module is used to perform MRZ code recognition and QR code recognition respectively, and output the document image text recognition results based on the position and recognition text.

[0089] Example 3:

[0090] like Figure 2 As shown, this embodiment provides a method for constructing and training the model described in Example 1, including the following steps:

[0091] Step 1: Dataset annotation and organization: First, organize the MRZ code and QR code data, then create data labels based on the image information, and finally enhance the image data.

[0092] For the detection dataset labels, text and QR code flags were added to facilitate subsequent classification. For documents containing QR codes, multiple Hong Kong and Macau travel permits were collected and the dataset was produced through methods such as data enhancement. For data detection, the text area and QR code area of ​​the image were enhanced separately, such as text transformation, text distortion, background blending, QR code deformation, blur enhancement, etc. For data recognition, considering that documents may be folded or rotated in actual production testing, additional geometric transformations such as rotation, scaling, translation, and flipping are required on the data images. In order to address the problems of long time and small data volume in the production of text recognition datasets, a text image generation tool was developed. It supports Chinese and English, numbers, and characters, supports multiple fonts, and can realize text deformation and add complex backgrounds.

[0093] Step 11: Collect MRZ codes based on multiple documents. The characters mainly include uppercase letters AZ, numbers 0-9, and filler "<". The QR code data is mainly based on Hong Kong and Macau Pass.

[0094] Step 12: A Python-based tool for generating text images was developed. It supports Chinese, English, numbers, and characters. The generated font is OCRB. The generated image data consists of multiple lines of text, with a noise background and image background added. Data labels are generated in JSON format. To facilitate subsequent MRZ and QR code classification, a single image is cropped and labeled.

[0095] Step 13. For data detection, the text area and QR code area of ​​the image are enhanced separately, such as text transformation, text distortion, background blending and QR code deformation, blur enhancement, etc., and finally 11,000 training images and 1,000 test images are generated; for image classification, there are 20,000 training images and 2,000 test images based on images generated by the tool and actual documents; for data recognition, considering that documents may be folded, rotated, etc. in actual production testing, it is necessary to perform additional geometric transformations on the data images, such as rotation, scaling, translation and flipping, to generate 30,000 training images and 3,000 test images.

[0096] Step 2. Model framework design: The detection model performs MRZ code and QR code detection, adjusts the input channel of the network layer to obtain more feature information, and optimizes the post-processing part of the model, including removing overlapping detection boxes based on non-maximum suppression (NMS) and filtering low-confidence areas based on classification scores to obtain more accurate model output; for the classification of MRZ codes and QR codes, a lightweight classification model is used; MRZ code area and QR code recognition are realized based on the recognition model and QR code recognition module.

[0097] Step 21: Build a detection model based on the Pytorch platform. The detection layer uses the VGG model as the backbone network. The first layer of the VGG configuration is adjusted to 32 input channels, and the subsequent structure is preserved. The number of channels in the last layer is changed to 256. The merging layer fuses feature maps from different levels through upsampling and downsampling operations to ensure that the model can detect both large and small text. The fused feature maps are then concatenated with the original feature maps to further enrich the feature information. After multi-scale feature fusion, the model significantly improves the detection accuracy of text of different sizes. The output layer integrates the information from the detection and merging layers to output the following text region scores (indicating the confidence of the detected text region) and geometric information (including geometric attributes such as the coordinates and angles of the text bounding box). Post-processing techniques such as non-maximum suppression (NMS) are used to remove overlapping detection boxes and output the precise text region. Each predicted polygon box is traversed based on a set threshold, which in this embodiment is set to 0.3. The overlapping area and geometric similarity between polygons within the group are then calculated. If the threshold condition is met, the polygon boxes are merged, otherwise the high-scoring boxes are retained, and unreasonable polygon boxes are eliminated; the polygon vertex distance constraint formula involved is:

[0098] ;

[0099] Where D vertex is the vertex distance under the constraint conditions, and v is the calculated polygon. vertex <0.3 (the threshold value set in this embodiment), the two polygonal boxes are considered to overlap.

[0100] Weighted merging strategy formula:

[0101] ;

[0102] Where V merged is the vertex distance, G is the polygon set of the adjacent group. i is the classification score of the i-th polygon, v i is the vertex coordinate of the i-th polygon. The confidence score after updating is the highest score in the group.

[0103] Step 22. Build a classification model based on Pytorch. First, introduce a lightweight SE attention module, learn channel weights through global average pooling and two fully connected layers, use SE only in some inverted residual blocks, and reduce the number of channels in the intermediate layers to reduce the number of parameters. Secondly, select h-swish as the activation function, which can achieve the effect of the swish function while avoiding exponential operations. Finally, combine dilated convolution, depthwise separable convolution, SE attention module and skip connection through modular design to further improve the flexibility of the model, so that the network can be adjusted according to different computing requirements and task complexity.

[0104] Step 23: Build a recognition model based on pytorch, which mainly consists of a CNN model, an RNN model, and a CTC parsing model. First, a CNN model is constructed. There are four maximum pooling layers in total, but the window size of the last two pooling layers is changed from 2x2 to 1x2. This is because most text images are small in height and long in width. This can ensure that information in the width direction is not lost as much as possible. The input image size is (1, 32, w), where the input image height is fixed at 32. After processing by the CNN model, the image size is (512, 1, w / 4); secondly, an RNN model is constructed, using a bidirectional recurrent network and using forward and backward context information to capture the contextual dependency of the sequence. The input image size is (w / 4, 512) and the output sequence is (w / 4, 5835). Considering the subsequent recognition work, the character set selected for the training model includes Chinese, English, numbers, and characters, with a total of 5835; finally, a CTC decoding model is constructed to map the probability sequence output by the RNN model to the final text. Dynamic programming is used to merge repeated characters and ignore whitespace. At the same time, the loss function directly maximizes the probability of the correct label sequence without the need for labeling and output steps, thereby solving the alignment problem.

[0105] Step 3. Model training: All three models are built based on the Pytorch platform. The parameters suitable for the models are set, and the model performance is evaluated based on the Precision and F1-score evaluation indicators. At the same time, the model parameters and data augmentation strategies are adjusted according to the results.

[0106] In step 3, in order to obtain data with more realistic labels, the annotated labels are biased inward by 0.2~0.3. Based on actual production requirements and test results, the bias distance is 0.2. In order to prevent the problem of sample imbalance, the loss function Dice can give more attention to pixels of fewer categories, effectively alleviating the problem of dataset imbalance. At the same time, in order to strengthen the foreground loss, a joint loss function is designed based on the weighted cross entropy and weighted Dice loss functions. The weighted cross entropy function is: ;The weighted Dice loss function is: ; The joint loss function is: . Expressed as the weight between loss functions, is the proportion of foreground pixel values; in the post-processing part of model training, the polygon box merging in text detection is processed based on the improved non-maximum suppression algorithm, which effectively reduces redundant detection results and improves detection accuracy. In the specific implementation, by traversing each predicted polygon box, based on the set threshold, which is set to 0.3 in this embodiment, if the intersection of the polygon boxes is greater than the default threshold, the two adjacent boxes are merged, and then the unreasonable polygon boxes are eliminated based on the non-maximum threshold algorithm; in the detection model, when cropping the image based on the coordinate point, the original model uses a random selection position for cropping and determines whether it overlaps with the text area. This method will greatly increase the amount of calculation, thereby lengthening the training time and consuming a lot of resources. Therefore, this embodiment uses a calculated bounding box to limit the position of the cropped area, ensuring that the cropped area is more likely to contain the text area, thereby significantly reducing the training time; in the actual reasoning process, it is found that the detection effect of small fonts is better, while the detection effect of large fonts is poor. In order to enable the training process to include more data features, reference is made to the advanced EAST model training method, first training is performed on a small-size image, and then the trained model parameters are transferred to a large-size image to retain as much feature information as possible.

[0107] Step 31, improved detection model training. First, the data set is sorted. Based on the annotated label, the inward bias is 0.2~0.3. According to the actual production requirements and test results, the bias distance is 0.2; secondly, by traversing each predicted polygonal box, based on the set threshold of 0.3, if the intersection of the polygonal boxes is greater than the default threshold, the two adjacent boxes are merged, and then the unreasonable polygonal boxes are eliminated based on the non-maximum threshold algorithm; then, the calculated bounding box is used to limit the position of the cropped area to ensure that the cropped area is more likely to contain the text area, thereby greatly reducing the training time; finally, the joint loss function designed in this embodiment is used To make the training process include more data features, we refer to the advanced EAST model training method, first train on small-size images, and then migrate the trained model parameters to large-size images to retain as much feature information as possible.

[0108] Step 32: Classification model training. First, adjust the model based on the previous dataset, with the MRZ code and QR code labels set to 0 and 1, respectively, and the input image size set to 224x224. Second, select model hyperparameters such as Batch Size, Epochs, and Weight Decay based on computer performance and actual needs. Finally, optimize the model based on loss and accuracy. First, determine the learning rate and optimizer, and observe the changing trends of loss and accuracy. Second, adjust the regularization method (such as adding dropout or data augmentation) based on overfitting / underfitting. Finally, if convergence is slow, replace the optimizer or adjust the learning rate.

[0109] Step 33: Identify model training. First, model-related hyperparameters, such as the learning rate and batch size, are set. Second, during the forward propagation, residual blocks are introduced into the CNN model to enhance the model's feature extraction capabilities and training stability. Finally, the parameters of the CNN and RNN models are optimized using gradient descent. The CNN model optimization process calculates the gradient of the loss function with respect to the parameters and updates the parameters in the opposite direction of the gradient. This is specifically implemented by calculating the gradients of the convolutional, pooling, and fully connected layers, and then updating the optimizer and learning rate parameters. During the RNN model optimization process, during backpropagation, the RNN model is first expanded into a chain structure by time step. The error is then backpropagated from the last time step to the initial time step (gradient calculation), and finally the model parameters are updated. During the optimization process, it is important to avoid vanishing and exploding gradients (in long sequences, gradients can decay or grow exponentially).

[0110] Step 4: Model inference: Input a test image, and the detection model outputs the detection results of the text and QR code in the test image. The output results are post-processed to obtain the final detection results; based on the detection results, the corresponding data images are cropped and classified through the classification model; the classified results are sent to the recognition model for text recognition.

[0111] Step 41: Detection model inference. First, resize the test image to a fixed multiple of 32, such as 512, 256, etc., and feed it into the constructed detection model.

[0112] Next, the detection model's feature extraction, feature fusion, and output layers sequentially process the input image. The backbone network's feature extraction layer outputs channels of 32, 64, 128, 256, and 256, constructing multi-scale feature maps. Each layer is downsampled through a max pooling layer. Given the small number of features extracted in the first step and their large scale, features extracted with 32 channels are discarded. This process results in a list of four feature maps.

[0113] The feature fusion layer performs feature fusion through three upsampling operations, generates a unified high-resolution feature map from the feature list information, and starts from the feature map obtained from the 256 channels in the last step. It is upsampled by the bilinear interpolation method so that the spatial size is aligned with the previous feature map and spliced ​​on the corresponding channel dimension to generate a high-dimensional feature map containing two scales. Then, two convolution and activation operations are performed to reduce the number of channels of the feature map on the one hand and further extract features on the other hand. The subsequent upsampling and splicing operations are similar and will not be repeated. After completing the three upsampling operations, the last convolution activation operation reduces the number of channels to 1 / 4 of the original size image.

[0114] The model output layer processes the above feature map, sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, thereby obtaining text score information; for the geometric information of the text, the feature map channel is set to 4 through the convolution layer, and the convolution output is controlled within [0, 1] based on the Sigmoid function, and the value of the Sigmoid output is scaled to the actual image size, which is set to 256 in this embodiment, to obtain text position information, and then sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and then maps the value of the Sigmoid output to the range of [-Π / 2, Π / 2] to obtain the rotation angle of the text box, and splices the position information and angle information to generate 5-channel text geometric information.

[0115] Finally, the model output is post-processed. High-probability text information is retained through threshold filtering (set to 0.8 in this example), and overlapping detection boxes are removed through non-maximum suppression (NMS) to output the precise text region. Each predicted polygon is iterated over, based on a set threshold (set to 0.3 in this example). The overlapping area and geometric similarity between polygons within the group are calculated. If the threshold condition is met, the polygons are merged; otherwise, the high-scoring polygons are retained, eliminating the unreasonable polygons. The geometric box information of the text region is then output.

[0116] Step 42: Classification model inference. First, the data generated by the EAST model is processed. The images are segmented and cropped based on the data labels to form an image list. The images are then processed into 224x224 pixels with 3 image channels.

[0117] Next, the image is passed to the classification model and processed through the feature extraction and output layers. The initial convolutional layer for feature extraction is a 3x3 convolutional layer, which reduces the input image size to 112x112x16. An inverted residual architecture is then constructed based on modules such as an expansion layer (1x1 convolutional layer), depthwise separable convolution (3x3 convolution), a squeeze-and-excitation (SE) attention module, a global average pooling layer, and a fully connected layer, reducing the input image size to 1x1x1280. To meet the classification output requirements, two global average pooling layers are used to convert the input features into a one-dimensional vector of dimension 1280.

[0118] Finally, model post-processing is completed using Softmax normalization. The final one-dimensional vector output from the feature map is used for prediction and classification, generating a probability distribution dimension based on the number of categories (1000 in this example). The corresponding category label and confidence score are then output. The QR code rectangle in the original image is removed, and the classified text box is corrected to leave appropriate recognition space. This results in a single image of the text information image and the QR code.

[0119] Step 43: Recognition model inference. The QR code is recognized using specialized tools. For text region recognition, the original image processed by the classification model is first resized, with a fixed height of 32 and the width scaled proportionally.

[0120] Secondly, the recognition model is divided into CNN feature extraction, RNN sequence modeling, and CTC decoding. The CNN model architecture is constructed based on multiple convolutional and pooling layers, with the input image height set to 1 and the width set to 1 / 4 of the original image. The RNN model expands the CNN model output into time steps along the width direction. Through bidirectional LSTM and fully connected layers, the output dimension is changed to [time step, batch, character category], which is 20x1x5835 in this embodiment. The CTC decoding module converts the time step output into a probability distribution through softmax normalization, removes blank characters, and selects the character sequence with the highest probability to generate the recognized text.

[0121] Finally, the detection results are organized in the form of location-identification text to generate corresponding identification labels.

[0122] Example 4:

[0123] like Figure 3 As shown, this embodiment deploys the recognition system of the above embodiment in a document scanning and recognition device, including a first base (1) and a second base (2).

[0124] The first base (1) and the second base (2) are arranged side by side, and a transmission channel (A) for placing documents is formed between the first base (1) and the second base (2).

[0125] After the document enters the transmission channel (A), the following steps are carried out in sequence:

[0126] Document reading: Send the document into the document scanning and identification device for document identification and information reading.

[0127] ID scanning: Scan the ID card to obtain the scanned image.

[0128] Document information detection: The obtained image information is optimized and sent to the detection module to obtain the characteristic information of MRZ code and QR code.

[0129] Document information classification: Data classification is performed based on the obtained detection feature information, and data images of MRZ codes and QR codes are output.

[0130] Document information recognition: The obtained MRZ code image is data enhanced and sent to the recognition module for MRZ code recognition. The QR code image is recognized by the QR code recognition module. The two types of information are integrated and the final recognition information is output.

[0131] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0132] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform a multimodal document information collaborative recognition method based on OCR technology.

[0133] A computing device includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing a multimodal document information collaborative recognition method based on OCR technology.

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A multimodal document information collaborative recognition method based on OCR technology, characterized by: Obtain document image information; Input the document image information into the pre-built detection model to detect the feature information of the MRZ code and QR code in the document image information; Input the feature information of the MRZ code and the QR code into a pre-built classification model, classify the feature information of the MRZ code and the QR code, and output the data images of the MRZ code and the QR code; The data images of MRZ code and QR code are input into the pre-built recognition model, and MRZ code recognition and QR code recognition are performed respectively, and the document image text recognition results based on position and recognition text are output.

2. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized by: The detection model includes: a feature extraction layer, a feature fusion layer and an output layer; the method for detecting the feature information of the MRZ code and the QR code in the document image information includes: The output channels of the feature extraction layer are 64, 128, 256, and 256, respectively, to construct multi-scale feature map information. Each layer is downsampled through the maximum pooling layer to obtain a list of four types of feature maps. The feature fusion layer performs feature fusion through three upsampling operations, generating a unified high-resolution feature map from the feature map list information. Starting from the feature map obtained from the last step with 256 channels, it is upsampled through bilinear interpolation to align the spatial size with the previous feature map, and spliced ​​on the corresponding channel dimension to generate a high-dimensional feature map containing two scales. After that, two convolution and activation operations are performed. After completing the three upsampling operations, the convolution activation operation reduces the number of channels to 1 / 4 of the original size image. The output layer sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function to obtain text score information; sets the feature map channel to 4 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and scales the value output by the Sigmoid function to the actual image size to obtain text position information, then sets the feature map channel to 1 through the convolution layer, and controls the convolution output within [0, 1] based on the Sigmoid function, and then maps the value of the Sigmoid output to the range of [-Π / 2, Π / 2] to obtain the rotation angle of the text box, and splices the position information and angle information to generate the geometric box information of the 5-channel text area.

3. The multimodal document information collaborative recognition method based on OCR technology according to claim 2 is characterized in that: The characteristic information of MRZ code and QR code in the document image information is also included: High-probability text information is retained through threshold filtering, and overlapping detection frames are removed through the non-maximum suppression algorithm to output the precise text area; by traversing each predicted polygon frame, based on the set threshold and calculating the overlapping area and geometric similarity between polygons in the group; if the threshold condition is met, the polygon frames are merged, otherwise the high-scoring frames are retained; and the geometric frame information of the text area is output.

4. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: The classification model includes a feature extraction layer and an output layer; Methods for classifying feature information of MRZ codes and QR codes include: The initial convolution layer of the feature extraction layer is a 3x3 convolution layer, which reduces the input image size to 112x112x16. An inverted residual structure is constructed based on modules such as the expansion layer, depthwise separable convolution, SE attention module, global average pooling layer, and fully connected layer, which processes the input image size to 1x1x1280. Two global average pooling layers are used to process the input features into a one-dimensional vector of dimension 1280. The one-dimensional vector with a dimension of 1280 is predicted and classified to generate a probability distribution dimension based on the number of categories, and the corresponding category label and confidence level are output; the QR code rectangle in the original image is removed to obtain a single image of the text information image and the QR code.

5. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: The recognition model includes: a CNN model, an RNN model, and a CTC decoding module; the method for performing MRZ code recognition and QR code recognition respectively, and outputting an identification label based on the position and recognition result includes: The CNN model sets the height of a single image of text information image and QR code to 1 and the width to 1 / 4 of the original image; The RNN model expands the image adjusted by the CNN model into time steps along the width direction, and changes the output dimension to [time step, batch, character category] through the bidirectional LSTM and fully connected layers; The CTC decoding module converts the time steps output by the RNN model into a probability distribution through Softmax normalization, removes blank characters, and selects the character sequence with the highest probability to generate the recognized text; it organizes the text in the form of position-recognition text and outputs the document image text recognition results.

6. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: The method for constructing the detection model includes: A detection model is built based on the Pytorch platform. The detection layer uses the VGG model as the backbone network. The number of input channels of the first layer of the VGG model configuration is adjusted to 32, and the number of channels of the last layer is set to 256. The merging layer fuses the feature maps of different levels based on upsampling and downsampling operations, and splices the fused feature maps with the original feature maps. The output layer integrates the information of the detection layer and the merging layer, outputs the text area score and geometric information. The text area score is used to represent the confidence of the detected text area, removes overlapping polygon detection frames, and outputs the accurate text area. By traversing each predicted polygon detection frame, based on the set threshold, the overlapping area and geometric similarity between polygons in the group are calculated. If the threshold condition is met, the polygon detection frame is merged, otherwise the high-scoring frame is retained, thereby eliminating unreasonable polygon detection frames. The polygon vertex distance constraint formula of the polygon detection frame is: ； Where D vertex is the vertex distance under the constraint conditions, v is the calculated polygon; if D vertex < the set threshold, the two polygon detection frames are considered overlapping; The formula for merging polygon detection boxes is: ； Where V merged is the vertex distance, G is the polygon set of the adjacent group, s i is the classification score of the i-th polygon, v i is the vertex coordinate of the i-th polygon; the confidence after updating and merging is the highest score in the group.

7. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: The method for constructing the detection model includes: A classification model is built based on Pytorch, and the SE attention module is introduced. Channel weights are learned through global average pooling and two fully connected layers. The SE attention module is only used in some inverted residual blocks, and the number of parameters is reduced by reducing the number of channels in the intermediate layers. The h-swish activation function is selected to achieve the effect of the swish function while avoiding exponential operations. The dilated convolution, depthwise separable convolution, SE attention module and skip connection are combined through modular design.

8. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: The method for constructing the recognition model includes: A recognition model is built based on pytorch; a CNN model is constructed, which has four maximum pooling layers. The window size of the last two pooling layers is changed from 2x2 to 1x2. The input image size is (1, 32, w), where the input image height is fixed at 32. After being processed by the CNN model, the image size is (512, 1, w / 4); an RNN model is constructed, which uses a bidirectional recurrent network and utilizes forward and backward context information to capture the contextual dependency of the sequence. The input image size is (w / 4, 512) and the output sequence is (w / 4, 5835). The character set selected for the training model includes Chinese, English, numbers, and characters, with a total of 5835 characters; a CTC decoding module is constructed to map the probability sequence output by the RNN model to recognized text. Duplicate characters are merged and whitespace is ignored through dynamic programming. At the same time, the loss function directly maximizes the probability of the correct label sequence without the need for labeling and output steps, thereby solving the alignment problem.

9. The multimodal document information collaborative recognition method based on OCR technology according to claim 1 is characterized in that: Before training the detection model, classification model, and recognition model, it is also necessary to obtain balanced sample data. The method is as follows: Based on the annotated labels, the inward bias is set to 0.2~0.3, and the Dice loss function is used to give more attention to pixels of fewer categories. In order to strengthen the foreground loss, a joint loss function is designed based on the weighted cross entropy and weighted Dice loss functions. The weighted cross entropy function is: , where is the cross entropy loss function, Predict the probability of foreground pixels, is the true label of the foreground pixel, is the predicted probability of background pixels, is the true label of the background pixel; The weighted Dice loss function is: , is the Dice loss function; The joint loss function is: , Expressed as the weight between loss functions, is the proportion of foreground pixel value.

10. The multimodal document information collaborative recognition system based on OCR technology is characterized by: include: An information acquisition module, used to obtain document image information; A detection module is used to detect the characteristic information of the MRZ code and the QR code in the document image information; A classification module is used to classify the feature information of MRZ codes and QR codes and output data images of MRZ codes and QR codes; The recognition module is used to perform MRZ code recognition and QR code recognition respectively, and output the document image text recognition results based on the position and recognition text.

Images