Staged OCR model construction method based on interactive local attention control

Abstract

The invention discloses a staged OCR model construction method based on interactive local attention control, solves the problem of low recognition rate of a VIT model, and belongs to the field of computer vision. Comprising the following steps: taking an OCR image data set containing position information as a training set; constructing a mask matrix corresponding to the task demand; a ViT model is adopted as a basic model, and an ontology feature map is formed based on PAN and SPP; constructing an invalid region based on the mask matrix; the method comprises the following steps of: dividing an OCR model into Patch, adding a two-dimensional RoPE relative position code, sending into Ernie autoregressive Decoder through Transform Encoder as a key value, and generating a full-connection LM Head as a header to obtain an OCR model; the training set is trained in the first stage, the mask matrix is trained in the second stage, and a trained staged OCR model is obtained; according to the invention, the small target identification capability is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and machine learning, and relates to a phased OCR model construction method based on interactive local attention control. BACKGROUND

[0002] As a key application in computer vision, OCR (Optical Character Recognition) technology is widely used in fields such as bill recognition, certificate input, item tracking, and industrial scene information extraction. Its main task is to extract structured text information from images, requiring the model to have high accuracy, good scene adaptability, and fast response capability.

[0003] An OCR image dataset is a collection of data specifically used to train and evaluate OCR models, including image files and annotation files, which are obtained through academic datasets, open source platforms, or self-built datasets obtained by manual scanning / shooting and annotation.

[0004] In practical applications, users are usually only interested in a few key fields in the image, such as invoice headings and ID numbers, so most business processes expect to prioritize and quickly identify these key fields, and then perform full-image recognition. This "prioritize key, step-by-step identification" approach helps improve user experience and speed up response. However, existing OCR solutions have obvious shortcomings under such demands, mainly in the following two aspects:

[0005] 1. Image slicing-based processing method: manually or by rules to define key areas, cut the image and then input it into the recognition model; this method is simple to implement, but the character proportion is imbalanced after the image is cut, especially when the target character occupies a large proportion in the slice area, the model recognition is prone to error, affecting the accuracy.

[0006] 2. Dual-model structure processing method: use a lightweight model for preliminary screening, only recognize part of the area, and then call a main model with more parameters for full-text recognition after preliminary screening; however, the lightweight model usually has limited recognition ability, leading to a decline in overall accuracy; at the same time, the model structure is complex, and the deployment and maintenance cost is high.

[0007] In recent years, with the development of Transformer structure and attention mechanism, ViT model (Visual Transformer, ViT) has been gradually applied to OCR tasks, which uses self-attention mechanism to capture global information in images and improve character recognition ability. Although this type of model has high recognition accuracy and high attention to global features, its parameter size is higher than that of traditional model construction, and its recognition effect for objects of different sizes or across blocks is poor, for example, a car may occupy multiple blocks in a picture, or a smaller size image recognition effect is poor. In addition, the image does not support specific area recognition, which leads to poor actual application effect.

[0008] For example, patent application CN111027562A discloses an optical character recognition method based on multi-scale RNN and RNN combined with attention mechanism. The method proposes a recognition method that integrates multi-scale feature extraction and attention mechanism in an OCR system. The overall architecture includes a convolutional neural network (CNN) feature extraction module and a recognition module combining a recurrent neural network (RNN) with an attention mechanism. The technical problems of this method are as follows:

[0009] High model complexity and large computational overhead: multi-layer convolution, parallel branches, and repeated sequence modeling units, the network structure is relatively large; for example, the CNN part contains multi-scale convolution branches and seven layers of convolution stacking, and the RNN part integrates multi-head attention and multi-layer bidirectional LSTM network; although this design enhances the feature expression ability, it significantly increases the model parameter quantity and computational load.

[0010] Attention is uncontrollable: static self-attention allocation method, lack of explicit guidance ability to focus on the area; the model may pay attention to non-text areas or noise when processing complex scene pictures, resulting in invalid calculation of part of the scene.

[0011] Algorithm flow is long: the algorithm link of convolution feature extraction→attention+LSTM modeling→CTC decoding, which is not suitable for real-time scenarios; for example, it may be difficult to meet the efficiency requirements in low-latency industrial applications or mobile scenarios. SUMMARY

[0012] In order to solve the problems that the region recognized by the OCR model is uncontrollable by the user, and a large number of single characters in different regions will appear in the model segmentation of the VIT model, leading to poor recognition effect of tinytext (text smaller than 8px) in the image, the application provides a phased OCR model construction method based on interactive local attention control. On the basis of using SPP and PAN to improve the small target recognition ability of the model, the controllable attention mechanism and the phased training strategy are introduced, so that the model can actively focus on the specified region for text extraction without sacrificing the structural integrity, and the accuracy and execution efficiency are considered, effectively meeting the actual needs of 'priority recognition' in the landing scene.

[0013] The purpose of the application is achieved by the following technical solutions:

[0014] The application discloses a phased OCR model construction method based on interactive local attention control, which comprises the following steps:

[0015] Step one, the OCR image data set containing position information is divided into image pixel points, text regions and text contents through the Input Image layer as a training set; an Input Mask layer is introduced, and a mask matrix corresponding to the task demand is constructed from the size attribute of the text region;

[0016] Step two, a ViT model is used as a basic model, and the images in the training set are input into the spatial pyramid pooling module SPP after passing through three parallel convolution branches PAN of the basic model, and then the feature values of the specified regions of the body feature map are set to negative infinity based on the mask matrix, so as to construct the invalid regions of the body feature map and obtain the feature map;

[0017] Step three, after the feature map is divided into a plurality of non-overlapping local rectangular regions Patch, a sequence tensor is obtained by convolution mapping to a dimension embedding space, two-dimensional RoPE relative position coding is added, and then the sequence tensor is sequentially input into a hierarchical TransformerEncoder with a learnable [IMG] global token at the end to realize global context interaction, and then the sequence tensor is input into an Ernie autoregressive Decoder as a key value, and text is generated character by character according to the language modeling paradigm, and a full connection LMHead is generated at the head to obtain an OCR model;

[0018] Step four, the first stage of the OCR model generates a total loss function by combining a character autoregressive negative log-likelihood loss function with L2 regularization, and the training set is trained, and the second stage simultaneously enables a region overlap loss function, and the mask matrix is trained by a total loss function and a region overlap loss function in a preset proportion to obtain a trained phased OCR model.

[0019] In step one, the Input Mask layer is introduced, and the mask matrix corresponding to the task requirement is constructed by the size attribute of the text region:

[0020]

[0021] In the formula, M represents a size of 14x14, corresponding to 14x14 16x16 local rectangular regions patch uniformly cut from the whole 224x224 image; A is a function of arithmetic average of input vectors or matrices in block dimension, f(R) represents converting the text region into a vector vi with a length of 14, R is the text region, and the size attribute of R is R=(x, y, w, h), which are the horizontal coordinate, vertical coordinate, width and height of the top-left corner of the text region, respectively, and the default values are x=0, y=0, width=0, height=0, which are modified according to the task requirement; is the average coefficient, and i is the number of patch.

[0022] In step two, the images in the training set are processed by the three parallel convolution branches PAN of the basic model:

[0023] The 224x224x3 RGB images in the training set are processed by the three parallel first convolution branch, second convolution branch and third convolution branch of the basic model to extract features, respectively, and the three outputs maintain three kinds of receptive fields of fine-grained-mesoscale-large scale; all use rectified linear unit ReLU activation, and output three scale feature maps; wherein,

[0024] The first convolution branch retains the original image resolution feature and extracts fine-grained information, with a convolution kernel size of 3x3, a stride of 1, and a channel number expanded to 64;

[0025] The second convolution branch maintains the mesoscale field of view and extracts the middle layer semantic information, with a convolution kernel size of 5x5, a stride of 2, and a channel number expanded to 128;

[0026] The third convolution branch maintains the large-scale field of view and extracts the global context features, with a convolution kernel size of 7x7, a stride of 4, and a channel number expanded to 256.

[0027] In step two, the formation method of the body feature map is:

[0028] The spatial pyramid pooling module SPP concatenates the three scale feature maps output by the three parallel convolution branches PAN after up-sampling them to the same scale of the original resolution to form the body feature map.

[0029] In step three, the method of adding two-dimensional RoPE relative position encoding is:

[0030]

[0031] wherein, is the sequence tensor after adding two-dimensional RoPE relative position encoding, d is the position of the dimension, and θ is the rotation angle.

[0032] In step three, the method of sequentially adding a learnable [IMG] global token at the end of the hierarchical TransformerEncoder to realize global context interaction includes:

[0033] The sequence tensor after adding two-dimensional RoPE relative position encoding is subjected to 12-layer TransformerEncoder, each layer of TransformerEncoder is first subjected to one normalization LayerNorm, and then sequentially performs multi-head attention mechanism, regularization Dropout and residual connection; and then subjected to two normalization LayerNorm and feedforward neural network to realize global context interaction.

[0034] A learnable [IMG] global token is added at the end of the Encoder to provide an overall picture summary in the decoding stage, while retaining the mask information of the mask matrix for skip reading instructions, and outputting a 196x768 sequence.

[0035] wherein, the multi-head attention mechanism is:

[0036]

[0037] wherein, Attention(Q, K, V) is the output of the multi-head attention mechanism, Q, K, and V are the outputs of LayerNorm, wherein Q is the query, K is the key value, and V is the value; Softmax is a function that converts a real number vector into a probability distribution, so that the value of each element is between 0 and 1, and the sum of all elements is 1, T is transposition, d k is the dimension of each head.

[0038] In step four, the total loss function and the region overlap loss function are respectively:

[0039] L_total = 0.6L_AR + 0.05||θ|| 2 ;

[0040]

[0041] wherein, L_total is the total loss function value, L_AR is the character-level autoregressive negative log-likelihood loss function value, L_ROI is the region overlap loss function value, A is the position of the predicted frame, B is the position of the real frame, and C is the bounding box of A and B.

[0042] In step four, in the sub-stage, L_total accounts for 65%, and L_ROI accounts for 35%;

[0043]

[0044] In the formula, M is the length of the character sequence, m is the serial number of the character, p(y true |y) is the probability of the character, y true is the number of correct characters, and y is the total number of predicted characters.

[0045] In step four, the trained text region label value is also subjected to a label assignment strategy based on Gaussian distribution;The label assignment strategy based on Gaussian distribution is as follows:

[0046]

[0047] Wherein, p(x) is the output value of Gaussian distribution, x is a sample, mu is the mean, sigma is the standard deviation e is the natural constant.

[0048] The beneficial effects of the present application are:

[0049] 1. The present application aims to propose a strong landing, taking the ViT model as the basic model, through the controllable attention mechanism and the phased training strategy, to strengthen the practicality of the OCR model in the industrialization scene. In order to solve the problems of lack of key region recognition ability of traditional OCR model and high actual deployment threshold caused by complex structure of ViT model, the phased OCR model designed in the present scheme is a unified main body, light and efficient end-to-end architecture, which has good structural simplicity and execution efficiency.

[0050] 2. At the model structure level, by introducing an interactive local attention control mechanism, the model can explicitly specify the image area that needs to be identified first, so as to skip the feature extraction and identification process of non-key areas, and reduce invalid calculation. Under the action of this mechanism, the OCR system can realize fast feedback on the key field in the early stage of image upload, significantly shorten the response time, improve the user interaction experience, and improve the recognition efficiency.

[0051] 3. The phased training process is adopted, the first phase is based on full image input for character recognition ability pre-training, and a stable feature expression basis is constructed;The second stage is to guide the model to optimize the recognition ability of the specified area through local mask, to ensure that the model has the ability to efficiently process key areas while maintaining general recognition ability, and to enhance the adaptability of the model. In the sub-stage, L_total accounts for 65%, and L_ROI accounts for 35%;Guide the model to maintain very low response in the shielded area, maintain high resolution in the visible area, and ensure controllability during interactive inference.

[0052] 4, The model is particularly suitable for non-tabular text extraction, such as certificate information extraction and form review in industrial processes, and the process of "first identifying key fields and then processing the whole graph" can significantly improve the inference efficiency and deployment flexibility while ensuring accuracy, providing practical technical support for the actual industrialization of OCR systems based on the ViT model.

[0053] 5, By changing the ViT model-based learner in the scheme, the recognition accuracy of TinyTxt is greatly improved.

[0054] 6, By associating the construction relationship between user input and image mask matrix, interactive model text extraction special training is realized, image pre-auditing ability required in specific scenarios is realized, and interactive recognition is realized.

[0055] 7, Improve the training efficiency of the model, and use the open source language model to maintain the semantic understanding ability of the model, without training the decoder during the training process.

[0056] 8, SPP and PAN are integrated to improve the field of view during feature extraction and realize feature fusion after multiple angles, and different sizes of "convolution layer" information extraction modes are realized for different images; invalid areas can automatically ignore invalid areas in subsequent attention calculation, reducing redundant calculation from the source.

[0057] 9, Two-dimensional RoPE relative position coding is added to capture the relative relationship in the row and column directions. The problem that some characters may be placed in two region blocks due to fixed region division of the image in data blocking is solved.

[0058] 10, The label assignment strategy based on Gaussian distribution is used for the label value of the training text area, which solves the technical problem that the text occupies a small area in most images in the whole area training process. BRIEF DESCRIPTION OF DRAWINGS

[0059] The application will be further described in detail below according to the drawings and embodiments.

[0060] Figure 1 It is a kind of based on interactive local attention control's phased OCR model construction method process schematic diagram provided in the embodiment of the application.

[0061] Figure 2 It is a kind of area overlap loss function schematic diagram provided in the embodiment of the application.

[0062] Figure 3 It is a kind of OCR model schematic diagram provided in the embodiment of the application.

[0063] Figure 4is an efficiency experimental result schematic diagram provided by an embodiment of the present application.

[0064] Figure 5 is a TinyTxt recognition rate experimental result schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0066] As Figure 1 shown, the embodiment of the present application provides a phased OCR model construction method based on interactive local attention control, which comprises the following steps:

[0067] Step one, the OCR image data set containing position information is divided into image pixel points, text regions and text contents through the Input Image layer as a training set; the Input Mask layer is introduced, and the mask matrix corresponding to the task requirement is constructed by the size attribute of the text region;

[0068] Step two, the ViT model is used as a basic model, the images in the training set are input into the spatial pyramid pooling module SPP after passing through the three parallel convolution branches PAN of the basic model, and the feature maps are formed by concatenating the features of the original resolution after upsampling in the channel dimension; based on the mask matrix, the feature values of the specified region of the feature map are set to negative infinity to construct the invalid region of the feature map, and the feature map is obtained.

[0069] The SPP and PAN are integrated to improve the field of view during feature extraction and realize feature fusion after multiple angles, and the mode of extracting information of different sizes of "convolution layers" for different images is realized; the invalid region can make the subsequent attention calculation automatically ignore the invalid region, and reduce redundant calculation from the source.

[0070] Step three, after the feature map is divided into a plurality of non-overlapping local rectangular regions Patch, the sequence tensor is obtained by convolution mapping to the dimension embedding space, the two-dimensional RoPE relative position coding is added, and the global token level TransformerEncoder with a learnable [IMG] is sequentially appended to realize the global context interaction, and then the Ernie autoregressive Decoder is input as the key value, the text is generated character by character according to the language modeling paradigm, the generation head is the full connection LMHead, and the OCR model is obtained.

[0071] Step four, the first stage of the OCR model is trained with the total loss function generated by the character autoregressive negative log-likelihood loss function combined with L2 regularization on the training set, and the second stage synchronously enables the region overlap loss function, and the total loss function and the region overlap loss function are trained by a preset proportion to obtain the trained phased OCR model.

[0072] In step one, the Input Mask layer is introduced, and the mask matrix corresponding to the task requirement is constructed from the size attribute of the text region as follows:

[0073]

[0074] In the formula, M represents a size of 14x14, corresponding to 14x14 16x16 local rectangular regions patch uniformly cut from the whole 224x224 image; the value 0 indicates that it can participate in calculation, and -∞-\infty-\infty indicates that it is shielded (completely falls within the text box and no longer participates in attention or convolution); A is a function of performing arithmetic averaging on the input vector or matrix in the block dimension, f(R) represents converting the text region into a vector vi with a length of 14, which is equivalent to a 14x14 binary matrix expansion vector, taking 0 / 1 or an overlap ratio; R is the text region, and the size attribute of R is R=(x,y,w,h), which are the horizontal and vertical coordinates of the upper left corner of the text region, the width and height of the text region, respectively, all coordinates are in pixels, and the default values are x=0, y=0, width=0, height=0, which are modified according to the task requirement; is the average coefficient, and i is the number of the patch.

[0075] In step two, the images in the training set are processed by the three parallel convolution branches PAN of the basic model as follows:

[0076] The 224x224x3 RGB images in the training set are respectively processed by the three parallel first convolution branch, second convolution branch and third convolution branch of the basic model for feature extraction, and the three outputs maintain three kinds of receptive fields of fine-grained-mesoscale-large scale; all use the rectified linear unit ReLU activation, and output three scale feature maps; wherein,

[0077] The first convolution branch retains the original image resolution feature and extracts fine-grained information, with a convolution kernel size of 3x3, a stride of 1, and a channel number expanded to 64;

[0078] The second convolution branch maintains the mesoscale field of view and extracts the middle layer semantic information, with a convolution kernel size of 5x5, a stride of 2, and a channel number expanded to 128;

[0079] The third convolution branch keeps a large-scale view to extract global context features, with a convolution kernel size of 7x7, a stride of 4, and a channel number expanded to 256.

[0080] In step two, the method for forming the body feature map is:

[0081] The spatial pyramid pooling module SPP concatenates the three scale feature maps output by the three parallel convolution branches PAN to form a body feature map (224x224x448) after upsampling to the same scale of the original resolution.

[0082] The body feature map is divided into 14x14 16x16 patches and mapped to a 768-dimensional embedding space through convolution to obtain a sequence tensor of 196x768. Then, two-dimensional RoPE relative position encoding is added to capture the relative relationship in the row and column directions. This solves the problem of placing some characters in two area blocks due to fixed area division in data blocking. In order to improve the sensing effect of edge fonts, all 14 feature blocks are encoded based on their positions in the image as index (the first small picture is 0, the second picture is 1), and RoPE (rotational position encoding) is performed based on the index.

[0083] In step three, the method for adding two-dimensional RoPE relative position encoding is:

[0084]

[0085] In the formula, is the sequence tensor after adding two-dimensional RoPE relative position encoding, d is the position of the dimension, and θ is the rotation angle.

[0086] In step three, the method for realizing global context interaction by sequentially adding a learnable [IMG] global token layer to the end of the Transformer Encoder includes:

[0087] The sequence tensor after adding two-dimensional RoPE relative position encoding is passed through 12 layers of Transformer Encoder. Each layer of Transformer Encoder first performs a normalization LayerNorm, then sequentially performs 12 heads of attention mechanism, each head with 64 dimensions, regularization Dropout, and residual connection; then performs a second normalization LayerNorm and a feed-forward neural network Feed Forward Network to realize global context interaction.

[0088] An additional learnable [IMG] global token is added at the end of the encoder to provide a summary of the entire image during the decoding stage while retaining the mask information of the mask matrix for the skip instruction. The output is a 196x768 sequence. The Ernie autoregressive decoder (12 layers, shared with ERNIE weights) is used as the key-value to generate text character by character in the language modeling paradigm. The generation head is a fully connected LMHead, which completes the construction of the model.

[0089] The multi-head attention mechanism is:

[0090]

[0091] In the formula, Attention(Q, K, V) is the output of the multi-head attention mechanism, Q, K, and V are the outputs of LayerNorm, wherein Q is the query, K is the key value, and V is the value; Softmax is a function that converts a real number vector to a probability distribution, so that each element value is between 0 and 1, and the sum of all elements is 1, T is transposition, d k is the dimension of each head.

[0092] In step four, the total loss function and the region overlap loss function are respectively:

[0093] L_total = 0.6L_AR + 0.05||θ|| 2 ;

[0094]

[0095] In the formula, L_total is the total loss function value, L_AR is the character-level autoregressive negative log-likelihood loss function value, L_ROI is the region overlap loss function value, A is the position of the predicted frame, B is the position of the real frame, and C is the bounding box of A and B. As shown in Figure 2 , the positions of all frames are represented by (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the upper left corner and (x2, y2) represents the coordinates of the lower right corner.

[0096] In step four, in the second stage, L_total accounts for 65% and L_ROI accounts for 35%. The model is guided to maintain very low response in the shielded area and high recognition in the visible area to ensure controllability during interactive reasoning. Both stages use the Adam optimizer with an initial learning rate of 3x10 -4 , β = (0.9, 0.999), and gradually reduce δ and weight decay in the middle and later stages.

[0097]

[0098] In the formula, M is the length of the character sequence, m is the character number, and p(ytrue |y) is the probability of the character, y true The number of characters that are correct, y is the total number of predicted characters.

[0099] In step four, a label assignment strategy based on Gaussian distribution is further included for the trained text region label value, which solves the technical problem that the area occupied by the text in most images is less in the training process of the whole region; wherein the label assignment strategy based on Gaussian distribution is:

[0100]

[0101] Wherein, p(x) is the output value of Gaussian distribution, x is a sample, μ is the mean, σ is the standard deviation e is the natural constant. Improve the trainable area in a single sample, balance the number of positive and negative samples.

[0102] In order to illustrate the technical solutions of the present application, a specific example is provided:

[0103] The open source data set olmOCR is obtained from the Internet, and the data processing is shown in Table 1:

[0104] Table 1

[0105] Key operations Details / suggestions Dataset collection LaTex-OCR, olmOCR... Uniform resolution Short side -> 224px; mirror padding ROI mask generation Box projected to 14x14 grid, resulting in 0 / 1 mask Data augmentation RandAug(2,10) + rotation < 3° + lighting jitter

[0106] The multi-source data is unified in scale by the outside world and generates ROI (Region-Of-Interest) mask, and based on the technical solutions disclosed in the present application, the constructed stage-by-stage OCR model is as shown in Figure 3 and Table 2;

[0107] Table 2

[0108]

[0109]

[0110] The encoder is shown in Table 3:

[0111] Table 3

[0112]

[0113] Note A in Table 3: h heads, each head dimension d k , usually h = 12, d k = 64; d m = h x d k = 768;

[0114] Note B in Table 3: attention calculation: Q, K, V are the same input of LayerNorm.

[0115] The embedding layer guided by Erine determines that the model has basic text recognition ability, and then through the construction relationship between the associated user input and the mask matrix, the interactive model text extraction special training is realized, the image pre-audit ability required in specific scenes is realized, and the experimental data is derived from 3080Ti and 400 test data taken from olmOCR. The GOT-OCR model is the starting point of the present model (Patialattention OCR), and the technical solution of the present application is also used. The test effect of the olmOCR data set is as shown in Figure 4 and Figure 5 The spatial attention OCR carries the region where the text is located, and constitutes a mask matrix.

[0116] From Figure 4 It can be seen that the Patial Attention OCR model takes 0.7s in the case of no words in the picture (nowords), while the same type of model GOT-OCR (using VIT) takes 3.5s, and the MinerU v1.3.10 takes even 4.3s. The Patial Attention OCR model really achieves "the fewer the words, the faster; the empty graph almost zero overhead". In the case of increasing the multi-scale fusion path, the Patial attention OCR only increases 0.5s in the experimental background of 20 words, and the additional overhead is also within 1s in the background of 50 words.

[0117] From Figure 5 It can be seen that: although the Patial Attention OCR in the multi-scale image object, the average accuracy reaches 83.6%, far more than the open source model GOT OCR and MinerU v1.3.10 (62.0%, 67.8%), and the accuracy gap with the closed source model such as MistralOCR API is also within 5%.

[0118] To sum up, under the filling of the mask matrix, the technical scheme disclosed by the present application greatly improves the object recognition efficiency of the image with less text elements, and the Pan branch also enhances the adaptability of the network to objects of various scales; by introducing the interactive local attention control mechanism, the model can explicitly specify the image area that needs to be identified first according to the actual business requirements, thereby skipping the feature extraction and identification process of non-key areas, reducing invalid calculation, significantly shortening the response time, improving the user interaction experience, and improving the identification efficiency. Among them, under the support of the mask matrix, the running efficiency of the model is very high, and the place that needs to be identified can be controlled and confirmed by the customer, reducing the waste of resources; a phased training process is adopted, the first phase is based on full image input to pre-train the character recognition ability, and a stable feature expression basis is constructed; in the second phase, the local mask is used to guide the model to greatly improve the recognition accuracy of small objects and enhance the adaptability of the model; by establishing the relationship between the user input and the image mask matrix, the interactive model text extraction special training is realized, the image pre-audit ability required in specific scenarios is realized, and the interactive recognition is realized; improve the training efficiency of the model, while maintaining the semantic understanding ability of the model by using the open source language model, and the decoder does not need to be trained during the training process.

[0119] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for constructing a stage-based OCR model based on interactive local attention control, characterized in that, The method comprises: Step one, the OCR image data set containing position information is divided into image pixel points, text regions and text contents through the Input Image layer to serve as a training set; an Input Mask layer is introduced to construct a mask matrix corresponding to the task requirements according to the size attribute of the text region; Step two, a ViT model is used as a basic model, the images in the training set are input into a spatial pyramid pooling module SPP after being processed by three parallel convolution branches PAN of the basic model, and then the feature values in the specified regions of the body feature map are set to negative infinity based on the mask matrix to construct invalid regions of the body feature map, thereby obtaining a feature map; Step three, after the feature map is divided into a plurality of non-overlapping local rectangular regions Patch, the local rectangular regions are mapped to a dimension embedding space through convolution to obtain a sequence tensor, two-dimensional RoPE relative position encoding is added, and then the sequence tensor is sequentially input into a TransformerEncoder layer with a learnable [IMG] global token at the end to realize global context interaction, and then input into an Ernie autoregressive decoder to generate text character by character according to the language modeling paradigm, and finally a full-connection LMHead is generated at the head of the OCR model to obtain an OCR model; Step four, the first stage of the OCR model is trained by using a character autoregressive negative log-likelihood loss function combined with L2 regularization to generate a total loss function, and the second stage is trained by simultaneously enabling a region overlap loss function to train the mask matrix by using a total loss function and a region overlap loss function in a preset proportion, thereby obtaining a trained phased OCR model.

2. The method of claim 1, wherein, In step one, the Input Mask layer is introduced to construct a mask matrix corresponding to the task requirements according to the size attribute of the text region as follows: In the formula, M represents a size of 14*14, corresponding to 14*14 16*16 local rectangular regions patch which are uniformly cut from a whole 224*224 image; A is a function of arithmetic average of input vector or matrix in block dimension, f(R) represents converting the text region into a vector vi with length of 14, R is a text region, and the size attribute of R is R=(x,y,w,h), which are respectively the horizontal coordinate, vertical coordinate, width and height of the upper left corner of the text region, and the default values are x=0, y=0, width=0, height=0, which are modified according to the task requirements; is an average coefficient, and i is the number of patch.

3. The method of claim 2, wherein, In step two, the method for processing the images in the training set through the three parallel convolution branches PAN of the basic model is as follows: The RGB images with a size of 224x224x3 in the training set are processed through the three parallel first convolution branch, second convolution branch and third convolution branch of the basic model to extract features, and the three outputs maintain three kinds of receptive fields with fine-grained, mesoscale and large-scale, and all of them are activated by a rectified linear unit ReLU to output three scale feature maps; wherein, The first convolution branch retains the original image resolution feature and extracts fine-grained information, the convolution kernel size is 3x3, the stride is 1, and the channel number is expanded to 64; The second convolution branch maintains a mesoscale field and extracts middle-layer semantic information, the convolution kernel size is 5x5, the stride is 2, and the channel number is expanded to 128; The third convolution branch maintains a large-scale field and extracts global context features, the convolution kernel size is 7x7, the stride is 4, and the channel number is expanded to 256.

4. The method of claim 3, wherein, In step two, the formation method of the body feature map is as follows: The spatial pyramid pooling module SPP concatenates the three scale feature maps output by the three parallel convolution branches PAN along the channel dimension after up-sampling the three scale feature maps to the same scale of the original resolution to form a body feature map.

5. The method of claim 4, wherein, In step three, the method for adding two-dimensional RoPE relative position encoding is as follows: wherein is the sequence tensor after adding the two-dimensional RoPE relative position encoding, d is the position where the dimension is located, and θ is the rotation angle.

6. The method of claim 5, wherein, In step three, the method of realizing global context interaction by sequentially appending a learnable [IMG] global token layer to the end of the Transformer Encoder includes: The sequence tensor after adding the two-dimensional RoPE relative position encoding is input into the 12-layer Transformer Encoder. Each layer of the Transformer Encoder first performs normalization LayerNorm, and then sequentially performs multi-head attention mechanism, regularization Dropout and residual connection. Then, the global context interaction is realized by twice normalization LayerNorm and feedforward neural network. A learnable [IMG] global token is appended to the end of the Encoder to provide the whole image summary in the decoding stage, while the mask information of the mask matrix is reserved for the skip instruction. The output is a 196x768 sequence.

7. The method of claim 6, wherein, The multi-head attention mechanism is as follows: In the formula, Attention(Q, K, V) is the output of the multi-head attention mechanism, Q, K, and V are the outputs of LayerNorm, wherein Q is the query, K is the key value, and V is the value; Softmax is a function that converts a real number vector into a probability distribution, so that the value of each element is between 0 and 1, and the sum of all elements is 1, T is transposition, d k is the dimension of each head.

8. The method of claim 6, wherein, In step four, the total loss function and the region overlap loss function are as follows: L_total=0.6L_AR+0.05||θ|| 2 ; In the formula, L_total is the total loss function value, L_AR is the character-level autoregressive negative log-likelihood loss function value, L_ROI is the region overlap loss function value, A is the position of the predicted box, B is the position of the real box, and C is the external frame of A and B.

9. The method of claim 8, wherein, In step four, in the second stage, L_total accounts for 65 percent, and L_ROI accounts for 35 percent. where M is the length of the character sequence, m is the index of the character, p(y true |y) is the probability of the character, y true is the number of correct characters, and y is the total number of predicted characters.

10. The method of claim 9, wherein, In step four, it also includes a label assignment strategy based on Gaussian distribution for the training text region label value. The label assignment strategy based on Gaussian distribution is as follows: In the formula, p(x) is the output value of the Gaussian distribution, x is the sample, μ is the mean, σ is the standard deviation, and e is the natural constant.

Images