Natural scene image text recognition method and system based on attention mechanism

Abstract

The application discloses a natural scene image Chinese text recognition method and system based on an attention mechanism, a method without correction or segmentation, which extracts rich global features by building a multi-level Efficient Swin Transformer network, realizes information interaction between different window features, models global information, and embeds channel attention to highlight the features of some important channels. The visual features obtained through the attention mechanism enable the network to focus on the text area of the feature map, improving the feature extraction capability of the network. Secondly, the semantic reasoning module considers the character context information, models the text sequence to obtain semantic features, and improves the prediction capability of the network. Finally, a double-feature fusion unit is used to fuse the visual and semantic features of different modalities, and the final text recognition result is obtained through classification, effectively avoiding the defects of segmented characters, and considering the internal relationship between characters.

Description

Technical Field

[0001] This invention belongs to the fields of digital image processing and computer vision technology, and relates to a method and system for text recognition in natural scene images, specifically a method and system for recognizing text in natural scene text images based on an attention mechanism. Background Technology

[0002] Textual information is abundant in daily life, and extracting its contents can improve quality of life. For example, autonomous driving relies heavily on speed limits, road information, and traffic light timings on signs; search engines categorize images based on textual information, allowing users to easily retrieve relevant images by inputting search terms. Therefore, text recognition is crucial. For text images with simple textual information and high image clarity, traditional OCR methods can be used for recognition. However, this method requires manual feature design before recognition. In natural scenes, text often lacks regularity in character shape, color, size, and position, and may even have low image clarity, making it difficult for traditional methods to achieve high recognition rates.

[0003] Currently, deep learning methods are mainly used for text recognition in natural scenes, which can be broadly divided into regular text recognition and irregular text recognition. Methods for regular text recognition mainly include CTC (connection-to-temporal classification) and sequence-to-sequence methods. These two methods have low accuracy for curved or rotated text. Methods for irregular text recognition mainly include: correction methods, which transform the irregular text image into a horizontal image using thin template interpolation or spatial transformation networks, and then process it as regular text; and segmentation methods, which first obtain the positional information of each character in the text image, segment individual character visual features, then classify and recognize the characters based on these features, and finally connect all the characters to obtain the recognition result. Correction methods often cause character distortion, leading to character recognition errors; segmentation methods require character-level annotation, and their performance is easily affected by the character segmentation results. Incorrect segmentation can lead to incorrect character recognition, or even failure to recognize characters. Currently, more and more researchers are using attention-based methods to recognize irregular text and avoid correction and segmentation. Summary of the Invention

[0004] To address the issue that regular text recognition methods have low accuracy when recognizing irregular text, and that methods such as correction or segmentation are often costly and neglect the inherent relationships between characters in the text sequence, this invention provides a natural scene text recognition method, system, and electronic device based on an attention mechanism.

[0005] The technical solution adopted by the method of the present invention is: a method for recognizing text in natural scene images based on an attention mechanism, which uses a natural scene text recognition network to recognize text in natural scene images;

[0006] The natural scene text recognition network includes a multi-level visual feature extraction module (efficient SwinTransformer), a visual attention module (DUnet Attention), a semantic reasoning module (Semantic ReasoningBlock), and a dual feature fusion unit (DFFU).

[0007] The multi-level visual feature extraction module, Efficient Swin Transformer, is used to output global features F. It includes a Patch Partition layer, a Linear Embedding layer, a Swin Transformer Block layer, and an ECA module. Several of the Efficient Channel Attention (ECA) modules and the Swin Transformer Block layer are cascaded, with the ECA modules embedded after the Swin Transformer Block layer. The Swin Transformer Block layer consists of a window-based multi-head self-attention module (W-MSA), a multilayer perceptron (MLP), a moving window-based multi-head self-attention module (SW-MSA), and a multilayer perceptron (MLP) connected sequentially. An LN module is set before the W-MSA module, MLP, and SW-MSA module, forming four small units. Each small unit is connected through a residual method.

[0008] The visual attention module DUnet Attention consists of three branches: V, Q, and K, which yield visual features F. v The V branch is obtained by linear transformation of the global feature F, and output after passing through a fully connected Linear layer; the Q branch is obtained by positional encoding, and output after passing through a fully connected Linear layer; the K branch is obtained by sequentially passing the global feature F through a Deformable Convolutional Network (DCV) and a DUnet network; the outputs of the Q and K branches after bitwise multiplication are combined with the output of the V branch after bitwise multiplication.

[0009] The visual feature F v The probability features F of the character are obtained by passing through a fully connected Linear layer and a normalization layer softmax. p Set the character mask matrix M according to the character prediction order. mask , will Fp and M mask The input is the Semantic Reasoning Block; the Semantic Reasoning Block includes a Masked Multi-Head Attention module, a Normalize layer, a FeedForward layer, a Normalize layer, a Linear fully connected layer, and a softmax normalization layer, to obtain the semantic features F. s The probability feature F p and mask matrix M mask Feature F is obtained through the Masked Multi-HeadAttention module. m The sequence passes through a normalization layer (Normalize), a feedforward layer (FeedForward), another normalization layer (Normalize), a fully connected layer (Linear), and a softmax layer (softmax), with the probability feature F... p It is connected to the normalized layer Normalize in a residual manner, and then connected to the next normalized layer Normalize in a residual manner before the feedforward connection layer;

[0010] The dual-feature fusion unit DFFU is used to fuse visual features F v and semantic features F s For visual features F v Semantic features F s After passing through fully connected Linear layers, the features are concatenated. The contribution weight of each feature is calculated using the Sigmoid function. Visual and semantic features are then fused, and the fused feature F is output. f .

[0011] The technical solution adopted by the system of the present invention is: a text recognition system for natural scene images based on an attention mechanism, comprising:

[0012] One or more processors;

[0013] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the attention-based text recognition method for natural scene images.

[0014] Compared with existing algorithms, the significant advantages and effects of this invention are as follows:

[0015] 1) This invention designs a multi-level efficient Swin Transformer feature extraction network to realize information interaction between features of different windows, model global information, and embed channel attention to highlight the features of certain important channels.

[0016] 2) The present invention designs an attention mechanism that enables the network to focus its attention on the text region of the feature map, thereby improving the network's feature extraction capability.

[0017] 3) This invention uses a semantic reasoning module to model text sequences, which improves the network's predictive ability, especially the character recognition accuracy of irregular text datasets.

[0018] 4) The average recognition accuracy of this invention is 93.5% for regular text datasets IC13, SVT, and IIIT5K; and 83.8% for irregular text datasets IC15, SVTP, and CUTE. Attached Figure Description

[0019] Figure 1 This is a diagram of the natural scene text recognition network structure according to an embodiment of the present invention;

[0020] Figure 2 This is a structural diagram of the multi-level efficient SwinTransformer according to an embodiment of the present invention;

[0021] Figure 3 This is a structural diagram of the DUnet Attention visual attention module according to an embodiment of the present invention;

[0022] Figure 4 This is a structural diagram of the Semantic Reasoning Block, an embodiment of the present invention.

[0023] Figure 5 This is a structural diagram of the dual-feature fusion unit (DFFU) according to an embodiment of the present invention. Detailed Implementation

[0024] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0025] This invention provides a method for text recognition in natural scene images based on an attention mechanism, which uses a natural scene text recognition network to recognize text in natural scene images;

[0026] Please see Figure 1The natural scene text recognition network in this embodiment includes a multi-level visual feature extraction module, Efficient Swin Transformer, a visual attention module, DUnet Attention, a semantic reasoning module, and a dual feature fusion unit, DFFU.

[0027] Please see Figure 2 The multi-level visual feature extraction module, Efficient Swin Transformer, in this embodiment is used to output global features F. It includes a Patch Partition layer, a Linear Embedding layer, a SwinTransformer Block layer, and an ECA module. Several efficient channel attention ECA modules and the SwinTransformer Block layer are cascaded, with the cascading method being that the efficient channel attention ECA modules are embedded after the SwinTransformer Block layer. The Swin Transformer Block layer in this embodiment consists of a window-based multi-head self-attention module W-MSA, a multilayer perceptron MLP, a moving window-based multi-head self-attention module SW-MSA, and a multilayer perceptron MLP connected in sequence. An LN module is set before the W-MSA module, MLP, and SW-MSA module, forming four small units. Each small unit is connected through a residual method.

[0028] In this embodiment, the input image of size H×W is divided into H / 4×W / 4 small blocks, each consisting of 4×4 pixels. The Linear Embedding layer transforms the channel dimension of the input image to a pre-set value of 512. The Swing Transformer Block extracts fine-grained features in the input image space. To extract important information from each channel, ECA is embedded as a residual after the Swing Transformer Block, allowing the network to focus on certain channel features. The Swing Transformer is an improvement on the Transformer, dividing the input image into windows and enabling remote information interaction between different window regions by moving the windows. Therefore, it can not only model global information but also has stronger feature extraction capabilities compared to the Transformer. The Swin Transformer in this embodiment consists of Layer Normalization (LN), Window-based Multi-head Self-attention (W-MSA), Multi-layer Perceptron (MLP), and Shifted Window-based Multi-head Self-attention (SW-MSA), as shown in the following structure. Figure 2 As shown on the right, an LN module is added before the W-MSA, MLP, and SW-MSA modules, forming four small units, each connected via residuals. The calculation formula for the input feature data X from level i to level i+1 is as follows:

[0029]

[0030]

[0031]

[0032]

[0033] In the formula, i represents the floor number. X i These represent the features of the outputs of the i-th layer W-MSA and MLP, respectively. X i+1 These represent the characteristics of the outputs of the (i+1)th layer SW-MSA and MLP, respectively.

[0034] Please see Figure 3The visual attention module DUnet Attention in this embodiment consists of three branches: V, Q, and K, which yield visual features F. v In this embodiment, the V branch is obtained by linear transformation of the global feature F and output after passing through a fully connected Linear layer; the Q branch is obtained by positional encoding and output after passing through a fully connected Linear layer; the K branch is obtained by sequentially passing the global feature F through a Deformable Convolutional (DVC) network and a DUnet network; the outputs of the Q and K branches are multiplied bit by bit and then multiplied bit by bit before being output from the V branch.

[0035] Please see Figure 3 In this embodiment of the DUnet network, the input features are sequentially downsampled and upsampled three times. After each downsampling, the residual is passed through a 1×1 convolutional layer (1×1 Conv) and then connected to the corresponding upsampling layer (Up-sampling). The downsampling consists of a dilated convolutional layer (DConv), batch normalization, and a ReLU activation layer (BN+ReLU) cascaded together. The upsampling consists of an upsampling layer (Up-sampling), a dilated convolutional layer (DConv), batch normalization, and a ReLU activation layer (BN+ReLU) cascaded together.

[0036] In this embodiment, the outputs of the three branches V, Q, and K are as follows:

[0037] V(F) = FW v (5)

[0038] Q = PW p (6)

[0039] K(F)=DUnet(DConv(F)+F) (7)

[0040]

[0041] Among them, W v and W p This is the matrix that the network needs to learn, where pos is the position of the character in the text sequence, and d is the matrix P(pos,d). i ) dimension, d i The position of the character vector, k is a natural number, 2k represents an even number, 2k+1 represents an odd number, P(pos,d) i ) is the position encoding matrix.

[0042] Please see Figure 4 The visual feature F in this embodiment v The probability features F of the character are obtained by passing through a fully connected Linear layer and a normalization layer softmax. pSet the character mask matrix M according to the character prediction order. mask , will F p and M mask Input the semantic reasoning block of this embodiment; the semantic reasoning block of this embodiment includes a masked multi-head attention module, a normalization layer, a feedforward connection layer, a normalization layer, a fully connected layer, and a softmax normalization operation layer to obtain semantic features F. s Probability feature F p and mask matrix M mask Feature F is obtained through the Masked Multi-Head Attention module. m The sequence passes through a normalization layer (Normalize), a feedforward layer (FeedForward), another normalization layer (Normalize), a fully connected layer (Linear), and a softmax layer (softmax), with the probability feature F... p It is residually connected to the normalized layer Normalize, and then residually connected to the next normalized layer Normalize before the feedforward layer.

[0043] The semantic reasoning block in this embodiment outputs F from the masked multi-head attention module. m ;

[0044] in:

[0045] F p =softmax(F v W1) (9)

[0046]

[0047]

[0048] Among them, F v For visual features, d represents the dimension of multi-head self-attention, and W represents the visual feature. i The transformation matrix obtained during training is given by i = {1, 2, 3}. When the network predicts the i-th character, i and j are equal, and the mask matrix M at this time is... maskAt position (i,j), the value is negative infinity. After adding the weights calculated from K and Q and passing them through the softmax function, the calculated weight coefficient is 0, meaning the information of that character itself is ignored. The prediction result is entirely derived from the information of other characters, thus avoiding interference from that character's information and improving the network's prediction ability. By predicting the character at each position sequentially, the semantically informative character feature F can be obtained. s .

[0049] Please see Figure 5 In this embodiment, the dual-feature fusion unit DFFU is used to fuse visual features F. v and semantic features F s For visual features F v Semantic features F s After passing through fully connected Linear layers, the features are concatenated. The contribution weight of each feature is calculated using the Sigmoid function. Visual and semantic features are then fused, and the fused feature F is output. f .

[0050] The fused feature F f for:

[0051] F f =W v F v *α+W s F s *(1-α) (12)

[0052] α=σ([W v F v W s F s (13)

[0053] Among them, W v W s α is the matrix that the network needs to learn, α is the calculated weight coefficient, and [·,·] represents the concatenation operation.

[0054] This invention is a method that does not require correction or segmentation. It extracts rich global features by building a multi-level Efficient SwinTransformer network, and then obtains visual features through an attention mechanism. Next, it considers the contextual information of characters through a semantic reasoning module and models the text sequence to obtain semantic features. Finally, it uses a dual-feature fusion unit to fuse visual and semantic features from different modalities and classifies them to obtain the final text recognition result. This method effectively avoids the defects of character segmentation and takes into account the inherent relationship between characters.

[0055] Experiments showed that the average recognition accuracy of this invention on regular text datasets IC13, SVT, and IIIT5K was 93.5%, and the average recognition accuracy on irregular text datasets IC15, SVTP, and CUTE was 83.8%. The recognition accuracies for IC13, SVT, and IIIT5K were 95.2%, 91.3%, and 93.9%, respectively; and the recognition accuracies for irregular text datasets IC15, SVTP, and CUTE were 80.3%, 85.4%, and 85.8%, respectively.

[0056] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for text recognition in natural scene images based on an attention mechanism, characterized in that: A natural scene text recognition network is used to recognize text in natural scene images; The natural scene text recognition network includes a multi-level visual feature extraction module (efficient SwinTransformer), a visual attention module (DUnet Attention), a semantic reasoning module (Semantic ReasoningBlock), and a dual feature fusion unit (DFFU). The multi-level visual feature extraction module, Efficient Swin Transformer, is used to output global features F; It includes a Patch Partition layer, a Linear Embedding layer, a Swin Transformer Block layer, and an ECA module. Several of the high-efficiency channel attention ECA modules and the Swin Transformer Block layer are cascaded, with the high-efficiency channel attention ECA modules embedded after the Swin Transformer Block layer. The Swin Transformer Block layer consists of a sequentially connected window-based multi-head self-attention module (W-MSA), a multilayer perceptron (MLP), a moving-window-based multi-head self-attention module (SW-MSA), and a multilayer perceptron (MLP). An LN module is placed before each of the W-MSA, MLP, and SW-MSA modules, forming four small units, each connected via a residual mechanism. The visual attention module DUnet Attention consists of three branches: V, Q, and K, which yield visual features F. v The V branch is obtained by linear transformation of the global feature F, and output after passing through a fully connected Linear layer; the Q branch is obtained by positional encoding, and output after passing through a fully connected Linear layer; the K branch is obtained by sequentially passing the global feature F through a Deformable Convolutional Network (DCV) and a DUnet network; the outputs of the Q and K branches after bitwise multiplication are combined with the output of the V branch after bitwise multiplication. The visual feature F v The probability features F of the character are obtained by passing through a fully connected Linear layer and a normalization layer softmax. p Set the character mask matrix M according to the character prediction order. mask , will F p and M mask Input the Semantic Reasoning Block; The Semantic Reasoning Block includes a Masked Multi-Head Attention module, a Normalize layer, a FeedForward layer, a Normalize layer, a fully connected Linear layer, and a softmax layer for normalization, to obtain semantic features F. s The probability feature F p and mask matrix M mask Feature F is obtained through the Masked Multi-Head Attention module. m The sequence passes through a normalization layer (Normalize), a feedforward layer (FeedForward), another normalization layer (Normalize), a fully connected layer (Linear), and a softmax layer (softmax), with the probability feature F... p It is connected to the normalized layer Normalize in a residual manner, and then connected to the next normalized layer Normalize in a residual manner before the feedforward connection layer; The dual-feature fusion unit DFFU is used to fuse visual features F v and semantic features F s For visual features F v Semantic features F s After passing through fully connected Linear layers, the features are concatenated. The contribution weight of each feature is calculated using the Sigmoid function. Visual and semantic features are then fused, and the fused feature F is output. f .

2. The text recognition method for natural scene images based on attention mechanism according to claim 1, characterized in that: Several of the aforementioned efficient channel attention (ECA) modules and the SwinTransformerBlock layer are cascaded together. The calculation formula for the input feature data X from level i to level i+1 is as follows: In the formula, i represents the layer number. X i These represent the features of the outputs of the i-th layer W-MSA and MLP, respectively. X i+1 These represent the characteristics of the SW-MSA and MLP outputs of the (i+1)th layer, respectively.

3. The text recognition method for natural scene images based on attention mechanism according to claim 1, characterized in that: The DUnet network input features are sequentially downsampled and upsampled three times. After each downsampling, the residual is passed through a 1×1 convolutional layer (1×1 Conv) and then connected to the corresponding upsampling layer (Up-sampling). The downsampling consists of a dilated convolutional layer (DConv), batch normalization, and a ReLU activation layer (BN+ReLU) cascaded together. The upsampling consists of an upsampling layer (Up-sampling), a dilated convolutional layer (DConv), batch normalization, and a ReLU activation layer (BN+ReLU) cascaded together.

4. The text recognition method for natural scene images based on attention mechanism according to claim 1, characterized in that: The outputs of the three branches V, Q, and K are as follows: V(F)=FW v (5) Q=PW p (6) K(F)=DUnet(DConv(F)+F) (7) Among them, W v and W p This is the matrix that the network needs to learn, where pos is the position of the character in the text sequence, and d is the matrix P(pos,d). i ) dimension, d i The position of the character vector, k is a natural number, 2k represents an even number, 2k+1 represents an odd number, P(pos,d) i ) is the position encoding matrix.

5. The text recognition method for natural scene images based on attention mechanism according to claim 1, characterized in that: The output F of the Masked Multi-Head Attention module of the Semantic Reasoning Block is... m ; in: F p =softmax(F v W1) (9) Where d is the dimension of multi-head self-attention, and W i The transformation matrix obtained during training is given by i = {1, 2, 3}. When the network predicts the i-th character, i and j are equal, and the mask matrix M at this time is... mask At position (i,j), the value is negative infinity. After adding the weights calculated from K and Q and passing them through the softmax function, the calculated weight coefficient is 0, meaning the information of the character itself is ignored, and the prediction result is obtained entirely from the information of other characters. By predicting the character at each position in turn, the semantically informative character feature F can be obtained. s .

6. The method for text recognition in natural scene images based on an attention mechanism according to any one of claims 1-5, characterized in that, The fused feature F f for: F f =W v F v *a+W s F s *(1-a) (12) α=σ([W v F v ,W s F s ]) (13) Among them, W v W s is the matrix that the network needs to learn, α is the calculated weight coefficient, [·,·] represents the concatenation operation, and σ is the sigmoid function.

7. A text recognition system for natural scene images based on an attention mechanism, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the attention-based text recognition method for natural scene images as described in any one of claims 1 to 6.

Images