Scene text recognition method based on cross-domain supervision signal guided attention generation

Abstract

The application discloses a scene text recognition method based on cross-domain supervision signal guided attention generation. In the encoding stage, the method extracts a text core area as a supervision signal to recursively guide attention, uses a gating mechanism to fuse the encoding information generated by the guided attention with the encoding information generated by the unguided attention, and enhances the robustness of the encoding information. In the decoding stage, the method combines an efficient parallel adaptive conversion decoder to decode, prevents attention drift in the decoding stage, and improves the recognition performance of the model. In addition, in the training stage, the method adopts a fusion strategy of artificial guidance and model adaptive learning to accurately learn the core area of the text, thereby providing correct supervision signals for attention guidance.

Description

Technical Field

[0001] This invention relates to a scene text recognition method based on attention generation guided by cross-domain supervision signals, and belongs to the field of scene text recognition. Background Technology

[0002] Scene text recognition is a crucial topic in computer vision, serving as the foundation for numerous downstream visual tasks such as scene understanding, autonomous driving, visual question answering, and key information extraction. In recent years, significant progress has been made in scene text recognition methods. While methods based on connected temporal sequences (CTC) and segmentation-based methods each have their advantages, attention-based text recognition methods have demonstrated superior vocabulary learning and observational abilities, becoming the mainstream approach in the field.

[0003] With the advent of Visual Transformer (ViT), attention-based methods have demonstrated superior recognition capabilities for both regular and irregular text compared to previous approaches. Some researchers have combined ViT and ResNet to extract rich textual information, subsequently using attention mechanisms to decode the feature-encoded sequences. Furthermore, to fully leverage the rich information in ViT feature-encoded sequences for text sequence prediction, researchers have proposed custom adaptive addressing and aggregation modules that select meaningful token combinations and aggregate them into a single token corresponding to a specific character. Nevertheless, attention-based methods, whether 1D or 2D, are susceptible to attention drift, ultimately leading to text recognition errors.

[0004] Attention drift during the decoding process has recently attracted widespread attention in academia. On the one hand, researchers have attempted to eliminate attention misalignment caused by decoding errors by separating the alignment operation from the decoding process and avoiding the use of historical decoding information. On the other hand, researchers have redefined text recognition tasks as character-level segmentation tasks, which involve segmenting text regions in an image and translating the segmentation results to obtain the final character sequence.

[0005] While researchers have implemented methods to mitigate attention drift and achieved significant results, they have overlooked the attention drift problem during the encoding process. However, the encoder plays an increasingly important role in various vision tasks, serving as the foundation for various business-specific and general-purpose large-scale vision models, and scene text recognition is no exception. Accumulated problems during the encoding process can directly lead to attention drift during decoding, further resulting in text recognition errors. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a scene text recognition method based on cross-domain supervised signal-guided attention generation. This method extracts the core region of the text as a supervised signal, recursively using supervised signals extracted from the CNN domain to guide the attention of the Transformer domain during feature encoding in a cross-domain manner for accurate learning. Guided attention is then used to correct the encoded information, and a gating mechanism is used to adaptively fuse it with the original attention-based encoded information to obtain accurate text feature encoding. For the obtained text feature encoding information, this invention employs a non-attentional adaptive decoder to decode the text feature encoding. This design avoids attention drift caused by using incorrect historical decoding information and improves decoding speed. Furthermore, the generation of the text core region integrates manual guidance and adaptive learning training strategies, enabling the model to independently learn and understand text core regions beneficial to text recognition from its own perspective, enhancing the model's learning ability for text core regions and further improving text recognition performance.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A scene text recognition method based on cross-domain supervised signal-guided attention generation is characterized by the following steps: In the encoding stage, the core region of the text is extracted as a supervised signal, and the supervised signal extracted from the CNN domain is recursively used in a cross-domain manner to guide the attention of the Transformer domain during feature encoding, thereby mitigating attention drift during encoding. Guided attention is then used to correct the encoded information. Finally, a gating mechanism is used to adaptively fuse the encoded information with the original attention-based encoding information to obtain the final text feature encoding. In the decoding stage, the obtained encoding result is decoded using a non-attention adaptive transformation decoder to mitigate attention shift during decoding. The method includes the following steps:

[0009] S1. Construct ground truth labels for manually guided text core region generation, which is achieved by using the smallest bounding box (x) surrounding the character. min ,y min ,x max ,y max It is constructed by scaling the center by a factor of γ;

[0010] S2. Construct a feature extractor, which is used for text feature extraction and text core region generation. The text core region is generated by the constructed feature pyramid network structure.

[0011] S3. In the encoding stage, multiple basic ViT blocks are first used for feature encoding. The encoded information is then input into the attention guidance module. Next, the core text region obtained in step S2 is used as a supervision signal to guide the attention generation in the Transformer. This guidance process recursively uses a cross-domain approach to guide the attention generation in the Transformer domain with the supervision signal obtained in the CNN domain. The guided attention is then used to correct the encoded information output from the ViT block. Finally, the encoded information generated by the guided attention and the encoded information generated by the unguided attention are input into the feature fusion module. This module uses a gating mechanism to fuse the two types of encoded information to obtain the final encoded information.

[0012] S4. In the decoding stage, a non-attention adaptive decoder is constructed using a series of 1×1 convolutional blocks and conventional matrix operations such as compression and transformation. This decoder accepts the output of the encoding stage as input and outputs the decoded result to obtain the character sequence.

[0013] S5. During the training phase, a combined training strategy of manual guidance and model adaptive learning is adopted. The manual guidance method uses the text core region constructed in S1 as supervision information to supervise the learning of the text core region in the feature extractor, enabling the model to extract the text core region. The model adaptive learning uses the model obtained by the above manual guidance method as the basis. In the case where there is no text core region as supervision, and only the text recognition result is used as supervision, the model adaptively learns the text core region that it understands and that is useful for text recognition.

[0014] Furthermore, step S1 specifically includes:

[0015] The construction of ground truth labels for manually guided text core region generation is achieved by using the smallest rectangle (x) surrounding the character. min ,y min ,x max y max The structure is formed by scaling the material γ times towards the center, as shown in the following formula:

[0016] w = x max -x min

[0017] h = y max -y min

[0018]

[0019] Among them, (x min ,y min ,x max ,y maxThe four variables in the diagram are the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner, which are the coordinates of the original character box. γ is the scaling ratio. The four variables are the x-coordinate of the top left point, the y-coordinate of the top left point, the x-coordinate of the bottom right point, and the y-coordinate of the bottom right point of the core area of ​​the text used for manual guidance.

[0020] Furthermore, step S2 specifically includes:

[0021] The feature extractor is constructed from a convolutional network, using a bottom-up chain to extract text features and a top-down feature pyramid structure to generate the core text region. The text feature extraction can be described as follows:

[0022]

[0023] Where, x f Represents text feature maps. This represents a bottom-up text feature extractor, where X represents the text image, H represents the height of the text image, W represents the width of the text image, and C represents the number of channels in the text feature map.

[0024] Furthermore, step S3 specifically includes:

[0025] In the encoding phase, firstly, feature encoding is performed using M basic ViT blocks. Specifically, for the feature map x in step S2... f Perform convolution and matrix transformation operations to form the input Z0 of the ViT block, and then encode features through M ViT blocks;

[0026] Secondly, an attention guidance module is constructed, which uses the acquired core text regions to generate a text guidance vector v. m And combined with the category vector v c Generate the final attention-guided vector v a The formula is as follows:

[0027]

[0028] Then the attention t of ViT a The guided attention is then used to correct the encoding results of ViT, as shown in the following formula:

[0029]

[0030] Among them, t a' It is the attention matrix for each head in ViT, t a The mean of the attention matrix of all attention heads is h, where h is the number of attention heads in ViT, N is the number of patches, and t is the mean of the attention matrix of all attention heads.ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of ViT block encoding, where D is the embedding dimension, sigmoid is the activation function, and norm represents layer normalization;

[0031] Finally, a feature fusion module is constructed. It maintains the original ViT encoding branch and outputs the encoding information generated without guided attention. Then, guided attention obtained from S3 is used to correct this encoding information, and it is initially fused to obtain the pre-fused encoding information v. f Finally, a gating mechanism is used to adaptively fuse the initially fused encoded information with the encoded information generated by unguided attention, resulting in the final fused encoded information F. The specific method is as follows:

[0032]

[0033] F=G⊙v ts +(1-G)⊙v f

[0034] Among them, W gated These are learnable weights, G is the gated weight, and t is the weight. ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of visual ViT block encoding, where N is the number of patches, D is the embedding dimension, sigmoid is the activation function, and norm indicates layer normalization.

[0035] Furthermore, step S4 specifically includes:

[0036] Construct an attention-free adaptive transformation decoder, represented as follows:

[0037]

[0038] Where Y is the decoded character information matrix, which can be post-processed to obtain the corresponding character sequence; ψ is an adaptive transformation function consisting of a series of 1×1 convolutional blocks and conventional matrix operations such as compression and transformation; T is the maximum length of the character sequence output by the decoder; and K is the number of character categories.

[0039] Furthermore, step S5 specifically includes:

[0040] The training process is divided into two stages, namely the manual guidance stage and the adaptive learning stage, depending on whether the text core region described in S1 is used to supervise the generation of the text core region in the feature extractor. In the manual guidance stage, the model is supervised using character sequences and manually set text core regions, while in the adaptive learning stage, only character sequences are used to supervise the model.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. Compared with the prior art, the present invention uses the core region of the text to guide attention learning in the encoding process in a cross-domain manner during the encoding stage, and uses a feature encoding fusion module to fuse the encoding information generated by guided attention with the encoding generated by unguided attention, thereby obtaining correct encoding information, preventing feature encoding errors caused by attention drift during the encoding process, improving the quality of text feature encoding, and providing a strong guarantee for obtaining the correct text sequence in the future.

[0043] 2. Compared with existing technologies, this invention employs a highly efficient parallel and non-attentional adaptive decoder to decode text feature encoding. This design avoids attention drift caused by using incorrect historical decoding information and improves decoding speed. In addition, the generation of text core regions integrates training strategies of manual guidance and adaptive learning, enabling the model to independently learn and understand text core regions that are beneficial to text recognition from its own perspective, thereby improving the model's learning ability of text core regions and further promoting text recognition performance. Attached Figure Description

[0044] Figure 1 This is a network structure diagram of the scene text recognition method based on cross-domain supervision signal-guided attention generation according to the present invention. Detailed Implementation

[0045] The scene text recognition method disclosed in this invention, based on cross-domain supervision signal-guided attention generation, includes the following steps:

[0046] Step S1 specifically includes:

[0047] Using the synthetic scene text dataset ST as the base dataset, ground truth labels for manually guided text core region generation are constructed. These labels are determined by the minimum bounding box (x) surrounding the character. min ,y min ,x max y max The structure is formed by scaling the material γ times towards the center, as shown in the following formula:

[0048] w = x max -x min

[0049] h = y max -y min

[0050]

[0051] Among them, (x min ,y min ,x max y max The coordinates of the top-left, top-left, bottom-right, and bottom-right points of the character are respectively the x-coordinates, y-coordinates, and y-coordinates of the original character box. γ is the scaling ratio, set to 0.5. This is the core region of the text used for manual guidance. At this point, the ST dataset can be used to supervise the training of the text recognition method and the process of generating the core text region.

[0052] Step S2 specifically includes:

[0053] Construct feature extractors, such as Figure 1 As shown in the dashed box of the feature extractor, this feature extractor consists of a convolutional network that uses a bottom-up chain to extract text features and a top-down feature pyramid structure to generate the core text region. Text feature extraction can be described as follows:

[0054]

[0055] Where, x f Represents text feature maps. This represents a bottom-up text feature extractor, where X represents the text image, H represents the height of the text image (set to 32), W represents the width of the text image (set to 128), and C represents the number of channels in the text feature map.

[0056] Step S3 specifically includes:

[0057] During the encoding phase, firstly, the feature map output in S2... The input Z0 is converted into a ViT block through operations such as convolution and matrix transformation. L-2 ViT blocks are then used to initially encode Z0. The specific implementation is as follows:

[0058]

[0059] z l =MSA(LN(z) l-1 ))+z l-1

[0060] Z l =MLP(LN(z) l '))+z l '

[0061] Where, x class It is a category embedding, E pos This is positional embedding, where r(·) represents the reshape operation, f(·) represents the flatten operation, α(·) represents a 1×1 convolutional block, l = 1...L, and L is the number of ViT blocks, which are set to 10 and 6 in our implementation, respectively. l The encoding result is represented by MSA, MLP, LN, and N is the number of patches, set to 256.

[0062] Secondly, an attention guidance module is constructed. In the last two ViT blocks, the core text region output from S2 is used as a supervision signal to guide the learning of attention in the ViT blocks, and the guided attention is used to correct the encoding results of ViT, such as... Figure 1 As shown in the attention guidance module, a text guidance vector v is generated using the acquired core text region. m And combined with the category vector v c Generate the final attention-guided vector v a The formula is as follows:

[0063]

[0064] v m =r(f(x) m ))

[0065] Where, x m The image shows the core text region, where r represents the reshape operation and f represents the flatten operation; then, attention is paid to the ViT block. a The guided attention is then used to correct the encoding results of the ViT block, as shown in the following formula:

[0066]

[0067] Among them, t a' It is the attention matrix for each head in ViT, t a is the mean of the attention matrix of all attention heads, h is the number of attention heads in ViT, set to 12, N is the number of patches, set to 256, t ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of visual ViT block encoding, where D is the embedding dimension, set to 768, sigmoid is the activation function, and norm represents layer normalization.

[0068] Next, construct a feature fusion module, such as Figure 1 As shown in the feature fusion module, the original ViT encoding branch is preserved, and the encoding information generated without guided attention is output. Then, the acquired guided attention is used to correct this encoding information and perform preliminary fusion to obtain the preliminarily fused encoding information v. f Finally, a gating mechanism is used to adaptively fuse the initially fused encoded information with the encoded information generated by unguided attention, resulting in the final fused encoded information F. The specific method is as follows:

[0069]

[0070] F=G⊙v ts +(1-G)⊙v f

[0071] Among them, W gated These are learnable weights, G is the gated weight, and t is the weight. ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of visual ViT block encoding, where N is the number of patches (set to 256), D is the embedding dimension (set to 768), sigmoid is the activation function, and norm indicates layer normalization.

[0072] Step S4 specifically includes:

[0073] In the decoding stage, an efficient, parallel, and non-attention-based adaptive transformation decoder is constructed, such as... Figure 1 The rightmost adaptive converter decoder module is shown below:

[0074]

[0075] Where Y is the decoded character information matrix, which can be post-processed to obtain the corresponding character sequence; ψ is an adaptive transformation function consisting of a series of 1×1 convolution operations and normalization operations; T is the maximum character sequence length output by the decoder, set to 27; and K is the number of character categories, set to 38.

[0076] By removing the attention calculation process during decoding, decoding speed can be improved by at least 30% compared to current state-of-the-art attention-based decoders.

[0077] Step S5 specifically includes:

[0078] For the training process, we used the Adadelta optimizer, the Cosine Annealing LR learning rate decay strategy, the RandAugment data augmentation method, ST and MJ scene text synthesis data, and a training strategy of manual guidance and model adaptive learning. The training process is divided into two stages. The first stage is the manual guidance learning stage, which uses the ST dataset generated in S1 with manually defined text core regions to supervise the generation of text core regions and text recognition sequences in the model. The second stage is the model adaptive learning stage, which only uses character sequences from the ST and MJ datasets to supervise the generation of text recognition sequences in the model. The loss function for the character sequences is the cross-entropy loss L. ce The loss function for the core text region uses the binary cross-entropy loss function L. bce Therefore, the loss function for the first stage is:

[0079] L=λ1L ce +λ2L bce

[0080] Where λ1 and λ2 are balance coefficients, set to 1 and 0.5 respectively, and the loss function for the second stage is:

[0081] L = L ce

[0082] At this stage, only the generation of character sequences needs to be supervised.

[0083] The experimental results are shown in Table 1. The present invention has achieved better performance than other methods on both regular and irregular text datasets. Among them, bolding the numbers represents the highest recognition accuracy on the dataset.

[0084] Table 1 Comparison of recognition accuracy of the present invention and other methods on a public test set.

[0085]

Claims

1. A scene text recognition method based on attention generation guided by cross-domain supervision signals, characterized in that: In the encoding stage, the core regions of the text are extracted as supervision signals. The supervision signals extracted from the CNN domain are recursively used across domains to guide the attention of the Transformer domain during feature encoding, mitigating attention drift during encoding. Guided attention is then used to correct the encoded information. Finally, a gating mechanism is used to adaptively fuse the encoded information with the original attention-based encoding to obtain the final text feature encoding. In the decoding stage, the obtained encoding results are decoded using a non-attention adaptive transformation decoder to mitigate attention shift during decoding. This method includes the following steps: S1. Construct real-value labels for manually guided text core region generation, which is achieved by using the smallest rectangle (x) surrounding the character. min ,y min ,x max ,y max It is constructed by scaling the center by a factor of γ; S2. Construct a feature extractor, which is used for text feature extraction and text core region generation. The text core region is generated by the constructed feature pyramid network structure. S3. In the encoding stage, multiple basic ViT blocks are first used for feature encoding. The encoded information is then input into the attention guidance module. Next, the core text region obtained in step S2 is used as a supervision signal to guide the attention generation in the Transformer. This guidance process recursively uses a cross-domain approach to guide the attention generation in the Transformer domain with the supervision signal obtained in the CNN domain. The guided attention is then used to correct the encoded information output from the ViT block. Finally, the encoded information generated by the guided attention and the encoded information generated by the unguided attention are input into the feature fusion module. This module uses a gating mechanism to fuse the two types of encoded information to obtain the final encoded information. S4. In the decoding stage, a non-attention adaptive decoder is constructed using a series of 1×1 convolutional blocks and conventional matrix operations such as compression and transformation. This decoder accepts the output of the encoding stage as input and outputs the decoded result to obtain the character sequence. S5. During the training phase, a combined training strategy of manual guidance and model adaptive learning is adopted. The manual guidance method uses the text core region constructed in S1 as supervision information to supervise the learning of the text core region in the feature extractor, enabling the model to extract the text core region. The model adaptive learning uses the model obtained by the above manual guidance method as the basis. In the case where there is no text core region as supervision, and only the text recognition result is used as supervision, the model adaptively learns the text core region that it understands and that is useful for text recognition.

2. The scene text recognition method based on cross-domain supervised signal-guided attention generation according to claim 1, characterized in that: Step S1 specifically includes: The construction of ground truth labels for manually guided text core region generation is achieved by using the smallest rectangle (x) surrounding the character. min ,y min ,x max y max The structure is formed by scaling the material γ times towards the center, as shown in the following formula: w=x max -x min h=y max -y min Among them, (x min ,y min ,x max ,y max The four variables in the diagram are the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner, which are the coordinates of the original character box. γ is the scaling ratio. The four variables are the x-coordinate of the top left point, the y-coordinate of the top left point, the x-coordinate of the bottom right point, and the y-coordinate of the bottom right point of the core area of ​​the text used for manual guidance.

3. The scene text recognition method based on cross-domain supervised signal-guided attention generation according to claim 1, characterized in that: Step S2 specifically includes: The feature extractor is constructed from a convolutional network, using a bottom-up chain to extract text features and a top-down feature pyramid structure to generate the core text region. The text feature extraction can be described as follows: Where, x f Represents text feature maps. This represents a bottom-up text feature extractor, where X represents the text image, H represents the height of the text image, W represents the width of the text image, and C represents the number of channels in the text feature map.

4. The scene text recognition method based on cross-domain supervised signal-guided attention generation according to claim 1, characterized in that: Step S3 specifically includes: In the encoding phase, firstly, feature encoding is performed using M basic ViT blocks. Specifically, for the feature map x in step S2... f Perform convolution and matrix transformation operations to form the input Z0 of the ViT block, and then encode features through M ViT blocks; Secondly, an attention guidance module is constructed, which uses the acquired core text regions to generate a text guidance vector v. m And combined with the category vector v c Generate the final attention-guided vector v a The formula is as follows: Then the attention t of ViT a The guided attention is then used to correct the encoding results of ViT, as shown in the following formula: Among them, t a' It is the attention matrix for each head in ViT, t a The mean of the attention matrix of all attention heads is h, the number of attention heads in ViT is N, and the number of patches is t. ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of ViT block encoding, where D is the embedding dimension, sigmoid is the activation function, and norm represents layer normalization; Finally, a feature fusion module is constructed. It maintains the original ViT encoding branch and outputs the encoding information generated without guided attention. Then, guided attention obtained from S3 is used to correct this encoding information, and it is initially fused to obtain the pre-fused encoding information v. f Finally, a gating mechanism is used to adaptively fuse the initially fused encoded information with the encoded information generated by unguided attention, resulting in the final fused encoded information F. The specific method is as follows: F=G⊙v ts +(1-G)⊙v f Among them, W gated These are learnable weights, G is the gated weight, and t is the weight. ga It is attention guided by attention vector, v ts' It uses encoded information for guided attention correction, v ts This is the result of visual ViT block encoding, where N is the number of patches, D is the embedding dimension, sigmoid is the activation function, and norm indicates layer normalization.

5. The scene text recognition method based on cross-domain supervision signal-guided attention generation according to claim 1, characterized in that: Step S4 is as follows: Construct an attention-free adaptive transformation decoder, represented as follows: Where Y is the decoded character information matrix, which can be post-processed to obtain the corresponding character sequence, ψ is an adaptive transformation function consisting of a series of 1×1 convolutional blocks and conventional matrix operations such as compression and transformation, T is the maximum character sequence length of the decoder output, and K is the number of character categories.

6. The scene text recognition method based on cross-domain supervision signal-guided attention generation according to claim 1, characterized in that: Step S5 is as follows: The training process is divided into two stages, namely the manual guidance stage and the adaptive learning stage, depending on whether the text core region described in S1 is used to supervise the generation of the text core region in the feature extractor. In the manual guidance stage, the model is supervised using character sequences and manually set text core regions, while in the adaptive learning stage, only character sequences are used to supervise the model.

Images