Scene text recognition method based on explicit supervised attention mechanism

Abstract

The present invention relates to a scene text recognition method based on an explicit supervised attention mechanism, and belongs to the field of scene text recognition. In the first part, the feature extraction part of the ResNet convolutional neural network with an attention mechanism is used to extract and encode features of a text image to obtain a feature map and a global representation. In the second part, the relationship between characters is modeled in combination with previously predicted character information, position information and global representation, and then attention weights are generated according to the feature map, and the weights are multiplied by the feature map to obtain the features of a single character, which is input into a feedforward neural network to obtain a predicted character, and then enters the prediction and recognition process of the next character, and so on, until the recognition end identifier is obtained. This method can automatically locate the features of the area at each prediction moment, thereby improving the recognition effect and solving the problem of poor recognition effect in curved or tilted conditions.

Description

Technical Field

[0001] The present invention belongs to the field of scene text recognition, and specifically proposes a method and system for text image recognition using a codec structure with an explicit supervised attention mechanism. The entire system uses a ResNet34 convolutional neural network with spatial attention and channel attention mechanisms to extract text image features, and uses a Transformer structure based on a self-attention mechanism for decoding and recognition. Background Art

[0002] Scene text recognition is an important challenge in the field of computer vision. Its task is to automatically detect and recognize text in natural images. Text, as a physical carrier of text, can be used to store and transmit information. With the help of text detection and recognition technology, important semantic information in visual images can be decoded. Due to the huge application value of scene text recognition, it has attracted a lot of research and exploration in industry and academia in recent years. However, most of the well-recognized ones are horizontal texts with simple backgrounds. However, in real scenes, due to the influence of factors such as lighting, occlusion, camera equipment, camera angle, and actual factors such as the bending, tilting, and artistic characters of the text itself, scene text recognition, especially irregular scene text recognition, has encountered a large bottleneck in recognition effect.

[0003] In order to address the problem of irregular text recognition, existing scene text recognition technologies (such as MORAN: A Multi-Object Rectified Attention Network and ASTER: An Attentional Scene Text Recognizer with Flexible Rectification for Scene Text Recognition) use a decoder based on the attention mechanism in the decoding stage, so that they can automatically focus on the character area in the image. These methods can better handle the problem of irregular text recognition, but because the scene images are too noisy, problems such as "attention drift" often occur, thereby reducing the accuracy of text recognition. Summary of the invention

[0004] Technical issues to be solved

[0005] In order to solve the problem of "attention drift" in the decoder based on the attention mechanism in the prior art, which leads to the problem of low text recognition accuracy, the present invention proposes a scene text recognition method based on an explicit supervised attention mechanism, which corrects the attention mechanism through explicit supervision to make the attention mechanism more accurate, thereby improving the accuracy and robustness of text recognition. It is used to solve the problem of curved and tilted scene text recognition.

[0006] Technical Solution

[0007] A scene text recognition method based on an explicit supervised attention mechanism is characterized by the following steps:

[0008] Step 1: Input the scene text image into the ResNet34 convolutional neural network to extract a feature map and record it as F, F∈R 25x8x512 ; Then input F into the global feature extraction layer of the six-layer Bottleneck to obtain the global feature representation G, G∈R 1×1×1024 ; At the same time, perform 1×1 convolution on F to obtain F' as the final extracted feature, F'∈R 25×8×1024 ; In each block of the four layers of ResNet34, channel attention and spatial attention mechanisms are added;

[0009] The spatial attention is explicitly supervised according to the character's annotation box. The calculation formula of this loss is as follows:

[0010]

[0011] where y pred is the attention weight at point i, j. When the point is within the character area y label is 1, otherwise it is 0; this supervision signal is only added to the last block of each layer;

[0012] Step 2: At the decoding time step t, the previously predicted character is added to the position information to obtain E, E∈R t×512 , and then concatenated with the global feature representation G to obtain the vector C, C∈R t×1024 ;

[0013] Step 3: Input C into the masked self-attention mechanism to model the dependencies between different characters in the output word. The self-attention mechanism is as follows:

[0014]

[0015] There are three steps to calculate attention. The first step is to calculate the similarity between the query and each key to obtain the weight. The second step is to use a softmax function to normalize these weights. Finally, the weights and the corresponding key values ​​are weighted and summed to obtain the final attention.

[0016] The encoder and decoder are connected using a two-dimensional attention module, whose structure is basically the same as the self-attention module, except that its K and V come from F' obtained in the encoding stage, and Q is the output of the masked self-attention module; the present invention performs explicit supervision training on the attention weights, and the loss function is defined as follows:

[0017]

[0018] where y pred is the attention weight at point i, j, when the point is within the character area y label is 1, otherwise it is 0;

[0019] Step 4: After passing through the two-dimensional attention module, we get an image feature vector. We pass this vector through a fully connected layer to get a vector with the same dimension as the number of letter types. We perform the argmax operation on this vector to get the prediction result at that moment.

[0020] The technical solution of the present invention further states: in step 1: the channel attention module inputs the feature map, passes through the global maximum pooling layer and the global average pooling layer based on width and height respectively, and then passes through the multilayer perceptron respectively; the features output by the multilayer perceptron are added based on bitwise multiplication, and then pass through the sigmoid activation operation to generate the final channel attention feature map; the map and the input feature map are bitwise multiplied to generate the input features required by the spatial attention module; the spatial attention module uses the feature map output by the channel attention module as the input feature map of this module, first performs a global maximum pooling and global average pooling based on the channel, and then splices the two results based on the channel; then undergoes a convolution operation to reduce the dimension to one channel; then generates the spatial attention feature through sigmoid; finally, the feature is multiplied by the input feature of the module to obtain the final generated feature.

[0021] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0022] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0023] A computer program, characterized by comprising computer executable instructions, which are used to implement the above method when executed.

[0024] Beneficial Effects

[0025] The scene text recognition method based on the explicit supervised attention mechanism proposed in the present invention can not only be used to recognize curved and tilted scene text images, but also use the two-dimensional attention mechanism to convert the information of the image into an attention weight matrix. At each prediction moment, the characteristics of the area can be automatically located, thereby improving the recognition effect and solving the problem of poor recognition effect in curved or tilted situations. The introduction of the explicit supervised attention mechanism can effectively solve the problem of attention drift, so that the model can find the key area of ​​the scene text letters at each moment of decoding, and combine the letter features to better recognize complex scene text images. At the same time, horizontal standard scene texts can also be recognized. The whole system has stronger practicality and can solve the recognition problems of scene texts in various situations including curved, tilted, horizontal, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 CBAM structure diagram

[0027] Figure 2 ResNet structure diagram DETAILED DESCRIPTION

[0028] The present invention will now be further described with reference to the embodiments and the accompanying drawings:

[0029] The system of the present invention includes two parts. The first part is the process of feature extraction and encoding of scene images based on spatial attention and channel attention mechanisms, and the second part is the decoding process of Transformer based on the self-attention mechanism. The recognition of scene text is realized through the structure of encoding and decoding and the attention mechanism. In the first part, the feature extraction part of the ResNet convolutional neural network with an attention mechanism is used to extract and encode the text image to obtain a feature map and a global representation. In the second part, the relationship between characters is modeled in combination with the previously predicted character information, position information and global representation, and then the attention weight is generated according to the feature map, and the weight is multiplied by the feature map to obtain the feature of a single character. The feature is input into the feedforward neural network to obtain the predicted character, and then enter the prediction and recognition process of the next character, and so on, until the recognition end identifier is obtained. It is worth noting that in order to ignore the background information of the picture as much as possible in the encoding stage and extract more text information, and to make the model pay attention to the corresponding feature map area in the decoding stage, the present invention explicitly supervises the attention mechanism used in the encoder and decoder according to the border information of the character. The steps are as follows:

[0030] (1) A scene text image is extracted through the ResNet34 convolutional neural network to obtain a feature map and record it as F, F∈R 25x8x512The present invention adds spatial attention and channel attention mechanism to each block of the four layers of ResNet. The attention mechanism is called CBAM, and its structure is shown in the figure Figure 1 shown.

[0031] The channel attention module passes the input feature map through the global maximum pooling layer and the global average pooling layer based on width and height, and then passes through the multi-layer perceptron. The features output by the multi-layer perceptron are added based on element-wise multiplication, and then through the sigmoid activation operation to generate the final channel attention feature map. This map is multiplied with the input feature map to generate the input features required by the spatial attention module.

[0032] The spatial attention module uses the feature map output by the channel attention module as the input feature map of this module. First, a global maximum pooling and global average pooling based on the channel are performed, and then the two results are spliced ​​based on the channel. Then, a convolution operation is performed to reduce the dimension to one channel. Then, a sigmoid operation is performed to generate the spatial attention feature. Finally, the feature is multiplied with the input feature of the module to obtain the final generated feature.

[0033] In order to extract as much text information as possible, the present invention explicitly supervises the spatial attention according to the character annotation box. The calculation formula of this loss is as follows:

[0034]

[0035] where y pred is the attention weight at point i, j. When the point is within the character area y label is 1, otherwise it is 0. This supervision signal is only added to the last block of each layer.

[0036] (2) In order to keep the dimension consistent in the two-dimensional attention calculation in the decoding stage, a 1x1 convolution is used to change the channel of F to obtain F', F'∈R 25x8x1024 At the same time, F is passed through the six-layer Bottleneck to obtain another feature G, which is called the global representation, G∈R 1x1x512, i.e. G is a 512-dimensional vector. Inspired by Transformer, the designed attention-based sequence decoder consists of three layers: ① a masked self-attention mechanism to model the dependencies between different characters in the output word; ② a two-dimensional attention module connecting the encoder and decoder; ③ a positional feed-forward layer applied to each decoding position separately. For each of the above three layers, a residual connection with addition operation is used, followed by layer-one. The above three layers constitute a module that can be stacked without sharing parameters.

[0037] (3) At time step t (t starts at 0), the previously predicted t characters are encoded and added to the position encoding position embedding to obtain a 512-dimensional vector, which is then concatenated with the global representation G to finally obtain t 1024-dimensional inputs C, C∈R t×1024 .

[0038] (4) C is input into a masked self-attention mechanism to model the dependencies between different characters in the output word. The self-attention mechanism is shown in the figure below.

[0039]

[0040] There are three main steps in calculating attention. The first step is to calculate the similarity between the query and each key to obtain the weight; then the second step is generally to use a softmax function to normalize these weights; finally, the weight and the corresponding key value are weighted and summed to obtain the final attention.

[0041] (5) A two-dimensional attention module is used to connect the encoder and decoder. Its structure is basically the same as that of the self-attention module, except that its K and V come from F' obtained in the encoding stage, and Q is the output of the masked self-attention module. In order to make the model pay attention to the area corresponding to the characters, this is reflected in the two-dimensional attention module by increasing the attention weight of the corresponding character area. The present invention performs explicit supervision training on the attention weight. The loss function is defined as follows:

[0042]

[0043] where y pred is the attention weight at point i, j, when the point is within the character area y label is 1 if the value is true, otherwise it is 0.

[0044] (6) After the two-dimensional attention module, we get an image feature vector, which is ∈R 1x1x1024After a fully connected layer, we get a vector with the same dimension as the number of letter types. We take the argmax operation on this vector to get the prediction result at that moment.

[0045] (7) Repeat the above (3-6) operations at the next moment to obtain the prediction results at multiple moments until the "END" terminator is obtained.

[0046] (8) During training, the decoder input is the vector obtained after the real label is embedded; during testing, since the real label information is unknown, the previous decoder output is used as the input at that moment. Backpropagation is only included in the training phase.

[0047] The specific process of the embodiment is as follows:

[0048] 1. Labeling of attention mechanism:

[0049] The synthetic scene text dataset SynthText provides the bounding box information of each character in the picture. The present invention produces a label of the attention mechanism based on the information for explicit supervision training of attention.

[0050] The image size of the model input is 400×128, with a width of 400 and a height of 128. In the encoding stage, the present invention adds an attention supervision signal to the last block of each layer of ResNet. The sizes of the spatial attention weight maps of these four stages are 100×32, 50×16, 25×8, and 25×8, respectively, and labels of corresponding sizes are made for this purpose. The present invention first scales the character bounding box to the corresponding size according to the ratio of the original image and the weight size of the attention map, and then generates an attention label, with the value inside the character bounding box being 1 and the value outside the character bounding box being 0. In the decoding stage, corresponding labels need to be made for each attention mechanism of the decoding step, and the size of each label is 25×8. First, the character bounding box is still scaled to the corresponding size, but the value of the attention label is only 1 if it belongs to the current character bounding box, otherwise it is 0.

[0051] 2. Scene text image preprocessing

[0052] In order to make the image size of the input model 400 × 128, the image size is adjusted to 400 × 128 using bilinear interpolation. The data augmentation methods used during training are random cropping, changing the brightness, contrast, saturation and hue of the image.

[0053] 3. Scene text image feature extraction based on ResNet with explicit supervised attention mechanism

[0054] The tensor (400×128×3) obtained after image preprocessing is input into the feature extraction layer of ResNet34. In each block of the four layers of ResNet34, channel attention and spatial attention mechanism CBAM are added, such as Figure 1 shown.

[0055] In order to increase the extracted feature map, the step size of the last layer of ResNet34 is changed from 2 to 1. The entire ResNet34 framework is shown in Figure 2 After feature extraction, we get F, F∈R 25×8×512 , and then input F into the global feature extraction layer of the six-layer Bottleneck to obtain the global feature representation G, G∈R 1×1×1024 , and perform 1×1 convolution on F to obtain F' as the final extracted feature, F'∈R 25×8×1024 The adjusted ResNet structure diagram is as follows:

[0056] 4. Feature decoding of Transformer structure based on explicit supervised attention mechanism

[0057] At the decoding time step t, the character embedding obtained previously is added to the position information position embedding to obtain E, E∈R t×512 , and then concatenated with the global representation G to obtain the vector C, C∈R t ×1024 , input to the self-attention module to model the relationship between characters. In the two-dimensional attention module, the output of the self-attention module is used as Q, and the encoded feature F' is used as K and V. The feature vector S that needs to be paid attention to is calculated at each time step. t , S t ∈R 1024 A position feed-forward layer is added to both the self-attention module and the 2D attention module, which can be regarded as a two-layer fully connected layer with 1024 input and output and 2048 hidden dimensions.

[0058] 5. Model Training

[0059] S t After a fully connected layer, the dimension of its output is equal to the number of all letter types. Then a softmax operation is performed to convert the output vector into the probability distribution of each letter. The letter with the largest probability distribution value is considered to be the prediction result of this layer. By analogy, the prediction results obtained at multiple time steps are all the letters in the scene text. The recognition loss function uses the cross entropy loss function:

[0060]

[0061] Where x is the predicted 94-dimensional vector and gt is the true character label. The final loss function is:

[0062]

[0063] Where α is a coefficient, and here α=0.1.

[0064] The optimizer uses ADADELTA to calculate gradients and perform back propagation. The batch size of the training is set to 112, one epoch requires 64638 iterations, and a total of 6 epochs are trained.

[0065] 8. Model Application

[0066] After the above training process, multiple models can be obtained. The best model (with the smallest loss function value) is selected for application. At this time, image data processing does not require data enhancement. It only needs to adjust the image to 400×128 size and normalize it as the input of the model. The parameters of the entire network model are fixed. Just input the image data and propagate forward. The feature map F'∈R is obtained in turn. 25×8×1024 and G∈R 1×1×512 . Then it is automatically passed to the decoding network for automatic decoding, and the recognition result can be directly obtained after the entire model. When a large number of scene text images need to be tested, all images need to be integrated into an lmdb format file to facilitate reading all images at one time.

[0067] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention are within the protection scope of the present invention.

Claims

1. A scene text recognition method based on explicit supervised attention mechanism, characterized in that Here are the steps: Step 1: Labeling of attention mechanism: The synthetic scene text dataset SynthText provides the bounding box information of each character in the image. Based on this information, the label of the attention mechanism is made for explicit supervision training of attention; specifically: The image size of the model input is 400×128, with a width of 400 and a height of 128. In the encoding stage, attention supervision signals are added to each block of each layer of ResNet. The sizes of the spatial attention weight maps of these four stages are 100×32, 50×16, 25×8, and 25×8, respectively, and labels of corresponding sizes are made for this purpose. First, the character bounding box is scaled to the corresponding size according to the ratio of the original image and the attention map weight size, and then the attention label is generated. The value inside the character bounding box is 1, and the value outside the character bounding box is 0. In the decoding stage, corresponding labels need to be made for each attention mechanism of the decoding step, and the size of each label is 25×8. First, the character bounding box is scaled to the corresponding size, but the value of the attention label is 1 only if it belongs to the current character bounding box, otherwise it is 0. Step 2: Scene text image preprocessing In order to make the image size of the input model 400×128, the image size is adjusted to 400×128 using the bilinear interpolation method; the data augmentation method used during training is random cropping, changing the brightness, contrast, saturation and hue of the image; Step 3: Scene text image feature extraction based on ResNet with explicit supervised attention mechanism The tensor 400×128×3 obtained after image preprocessing is input into the feature extraction layer of ResNet34; In each block of the four layers of ResNet34, channel attention and spatial attention mechanisms CBAM are added; Change the step size of the last layer of ResNet34 from 2 to 1; After feature extraction, we get F, F∈R 25×8×512 , and then input F into the global feature extraction layer of the six-layer Bottleneck to obtain the global feature representation G, G∈R 1×1×1024 , and perform 1×1 convolution on F to obtain F' as the final extracted feature, F'∈R 25×8×1024 ; Step 4: Feature decoding of the Transformer structure based on the explicit supervised attention mechanism At the decoding time step t, the character embedding obtained before is added to the position information positionembedding to obtain E, E∈R t×512 , and then concatenated with the global representation G to obtain the vector C, C∈R t×1024 , input to the self-attention module to model the relationship between characters; in the two-dimensional attention module, the output of the self-attention module is used as Q, and the encoded feature F' is used as K and V. The feature vector S that needs to be paid attention to is calculated at each time step t , S t ∈R 1024 ; A position feed-forward layer is added to both the self-attention module and the 2D attention module, which can be regarded as a two-layer fully connected layer with 1024 input and output and 2048 hidden dimensions; Step 5: Model training S t After a fully connected layer, the dimension of its output is equal to the number of all letter types. Then a softmax operation is performed to convert the output vector into the probability distribution of each letter. The letter with the largest probability distribution value is considered to be the prediction result of this layer. By analogy, the prediction results obtained in multiple time steps are all the letters in the scene text. The recognition loss function uses the cross entropy loss function: Where x is the predicted 94-dimensional vector, gt is the real character label; the final loss function is: Where α is the coefficient, here α=0.1; is the loss function for explicit supervision of spatial attention, loss dec_att Loss function for explicit supervised training of attention weights; The optimizer uses ADADELTA to calculate gradients and perform back propagation. The batch size of training is set to 112. One epoch requires 64638 iterations, and a total of 6 epochs are trained. After the above training process, multiple models can be obtained. The best model is selected for application. At this time, image data processing does not require data enhancement. It only needs to adjust the image to 400×128 size and normalize it as the input of the model. The parameters of the entire network model are fixed. Just input the image data and propagate forward. The feature map F'∈R is obtained in turn. 25×8×1024 and G∈R 1×1×512 ; Then it is automatically passed to the decoding network for automatic decoding, and the recognition result can be directly obtained after the entire model; when a large number of scene text pictures need to be tested, all pictures need to be integrated into an lmdb format file to facilitate reading all pictures at one time.

2. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

3. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

4. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images