Scene text recognition method based on apparent difference guided sequence mixed mask

Abstract

The application discloses a sequence mixed mask scene text recognition method based on appearance difference guidance, comprising the following steps: 1) establishing a difference-guided sequence mixed mask DSHM pre-training framework; 2) appearance difference quantization; 3) sequence mixed mask generation; 4) difference condition decoder construction: modifying a student network decoder, introducing a difference condition token DC-Token, and realizing adaptive reconstruction; 5) multi-loss joint training; 6) migrating the pre-trained student network to various scene text recognition STR tasks, and utilizing the difficult area reasoning ability learned in the pre-training stage to realize robust text recognition in scenes such as occlusion, distortion and low resolution. The application can effectively solve the recognition challenge caused by the appearance difference caused by partial occlusion or deformation.

Description

Technical Field

[0001] This invention relates to artificial intelligence technology, and more particularly to a method for text recognition in sequence mixing mask scenes based on appearance differences. Background Technology

[0002] The CTC decoder-based approach directly converts image features into text sequences, using whitespace labels and post-processing to address alignment issues. This method offers fast inference speed but limited performance. Language-aware models utilize language information to aid recognition, typically employing an external language model (LM) or training an internal LM. DIG-SSTR integrates contrastive learning and reconstruction learning objectives, its design inspired by the reading and writing process. CCD employs character-level contrastive learning and is equipped with a spectral segmentation module. SSM enhances structural modeling by reconstructing directional signals in overlapping inputs. LMIM utilizes language-guided modeling to capture the internal structure of characters and inter-character dependencies. Furthermore, research has explored customized masking improvement schemes for STR: employing multilingual perceptual masking mechanisms (such as MLM and PLM) to present multi-layered textual cues; unifying masking and substitution language models to enhance the decoder's contextual reasoning capabilities.

[0003] In contrast, we propose a vision-based pre-training method. Unlike conventional masking techniques, our difference metric-guided hybrid masking strategy is specifically designed for sequential text recognition. This approach encourages the model to focus on structurally or semantically challenging regions during encoding, thus better meeting the requirements of robust sequential text recognition. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a text recognition method for sequence mixing mask scene based on appearance difference guidance, which addresses the deficiencies in the prior art.

[0005] The technical solution adopted by this invention to solve its technical problem is: a text recognition method for sequence mixing mask scenes based on appearance differences, comprising the following steps: 1) Establish a differential-guided sequence mixing mask DSHM pre-training framework; The pre-training framework is an asymmetric teacher-student architecture consisting of an Oracle teacher, a momentum teacher, and a student network. 2) Quantification of apparent differences The Apparent Difference Measure (ADM) is used to quantify and score each patch of the text image, and the difference score of each patch is obtained by combining the differences in local structure and global style. Each patch of the text image is a small rectangular image unit obtained by dividing the entire text image according to a set size. 3) Sequence mixing mask generation; Sort all patches by ADM score and select the most difficult patch as the mask seed; The seed patch is expanded into blocks to generate a coherent block-shaped differential guide mask, simulating a real-world scenario of continuous character occlusion. Apply random masks to the remaining unmasked areas to ensure training stability and the model's ability to perceive the global context; Merge the two types of masks to obtain the final mask tensor and reconstruction index; 4) Construction of the Difference Condition Decoder: The student network decoder is modified by introducing a difference condition token (DC-Token) to achieve adaptive reconstruction. 5) Multi-loss joint training; Gradient updates are performed on the student network, while the momentum teacher parameters are updated using an exponential moving average. 6) Transfer the pre-trained student network to various scene text recognition (STR) tasks, and use the reasoning ability of difficult regions learned in the pre-training stage to achieve robust text recognition in scenes including occlusion, distortion and low resolution.

[0006] According to the above scheme, in step 1), the Oracle teacher provides high-quality text image features for the appearance difference measure ADM, laying the foundation for subsequent difference measures; the Oracle teacher includes a mid-level semantic feature output layer and an attention map output module; Momentum teachers provide a stable, dynamically evolving target for reconstruction, avoiding the regression of training feedback loops.

[0007] According to the above scheme, step 2) specifically includes the following: 2.1) Calculate the anisotropic local difference (ALD): Measure the directional dependence of the local feature topology by the trace of the covariance matrix of the difference vectors of each patch and its eight neighboring features; 2.2) Calculate Intra-Instance Global Style Difference (IGSD): Use Oracle teacher attention maps to select text-related patches in images, construct instance-specific style prototypes, and measure the degree of deviation from the instance's dominant style by the L2 distance between the patch features and the prototypes. 2.3) Merge the ALD and IGSD scores to obtain the final ADM difference score for each patch, thus completing the localization of difficult regions.

[0008] According to the above scheme, step 4) specifically includes the following: The difference-aware information of the encoder in the student network is passed end-to-end to the decoder, so that the decoder can obtain the prior knowledge of the difficulty of each patch before reconstruction. For each masked position i, the decoder's fixed mask embedding is replaced with a shared base mask embedding. With difference embedding The dynamic combination of differences and embeddings is generated by the ADM scores of the corresponding mask patches through MLP mapping; For each masked position i, its input token It is no longer a fixed vector, but is embedded from a shared base mask. With an explicit difference embedding Dynamic composition; among which, The difference score of the mask patch corresponding to this position generate.

[0009] According to the above scheme, in step 4), the shared basic mask is embedded. With difference embedding The dynamic combination, the combination form is:

[0010] in, It is an MLP that maps scalar difference scores to a high-dimensional feature space. This represents the visible patch features from the encoder.

[0011] According to the above scheme, step 5) includes: Calculate the weighted reconstruction loss: Using the output of the momentum teacher as the reconstruction target, the loss weight is adjusted by the ADM score of the patch, and a stronger penalty is imposed on the reconstruction error in difficult regions; Calculate consistency auxiliary loss: Align the global output of the student network processing the masked image with the global output of the momentum teacher processing the original unmasked image to ensure the robust global representation capability of the model under the mask. The total loss is obtained by weighted fusion of the two types of losses. The student network is updated by gradient descent, and the momentum teacher is updated synchronously by EMA. The Oracle teacher is frozen throughout the process and does not participate in the update.

[0012] The beneficial effects of this invention are: 1. This invention uses frozen Oracle teachers for STR pre-training, providing stable, noise-free, high-quality features for appearance difference measurement, and solving the problem of inaccurate localization of difficult regions caused by feature fluctuations during difference quantification. 2. This invention proposes a multidimensional appearance difference measure (ADM) to achieve accurate quantification of difficult areas in complex scenarios; 3. By using differentiated conditional tokens, the prior knowledge of the difficulty of each patch is encoded in the decoder, guiding the decoder to focus on the reconstruction of difficult regions during the decoding process, so as to realize an adaptive reconstruction paradigm to improve the robustness of model recognition in partially occluded or text-distorted scenarios. Attached Figure Description

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0015] like Figure 1 As shown, a text recognition method for sequence mixing masking scenes guided by appearance differences includes the following steps: 1) Establish a differentially guided sequence mixing mask (DSHM) pre-training framework; An asymmetric teacher-student architecture is constructed, consisting of an Oracle teacher (frozen network), a momentum teacher (EMA updated network), and a student network (gradient descent updated unique network). Among them, Oracle Teacher provides high-quality text image features for the Apparent Dissimilarity Measure (ADM), laying the foundation for subsequent dissimilarity measures; Oracle Teacher includes a mid-level semantic feature output layer and an attention map output module; The momentum teacher provides a stable target for reconstruction that evolves dynamically, avoiding training feedback loop degradation; the momentum teacher acts as a dynamic supervisor for the student network, providing a stable and dynamically evolving high-quality feature target for the reconstruction task of the masked region, replacing the original image features in traditional MAE. The student network is the only network in the entire architecture that participates in gradient descent updates. It is the core of pre-training and is responsible for completing mask image feature encoding, difference conditional decoding, mask region reconstruction, and learning robust text feature extraction capabilities through the loss function. 2) Quantification of apparent differences The apparent difference measure (ADM) is used to quantify and score each patch of the text image, and the difference score of each patch is obtained by combining local structure and global style differences. Each patch of the text image is a small rectangular image unit obtained by dividing the entire text image according to a set size; as detailed below: 2.1) Calculate Anisotropic Local Difference (ALD): Measure the directional dependence of the local feature topology by the trace of the covariance matrix of the difference vectors of each patch and its eight neighboring features; 2.2) Calculate Intra-Instance Global Style Difference (IGSD): Use Oracle teacher attention map to select text-related patches in the image, construct instance-specific style prototypes, and measure the degree of deviation from the instance's dominant style by the L2 distance between the patch features and the prototypes. 2.3) Merge the ALD and IGSD scores to obtain the final ADM difference score for each patch, thus completing the localization of difficult regions; 3) Sequence mixing mask generation; First, based on the difference score of each patch All patches are sorted to identify the "most difficult" patch, which is then used as a seed in the subsequent masking process. Next, difficulty masks are generated sequentially according to the sorted index list. Building upon this, to simulate the occlusion of consecutive characters, block expansion is performed around these "seed" patches, forming several consecutive sequence mask blocks. This process ensures that we prioritize masking consecutive patch blocks containing high-scoring seeds.

[0016] Finally, to ensure training stability and maintain the model's awareness of the global context, we apply further random masks to the remaining unmasked patches. By merging the differentially guided mask and the random mask, we obtain the final mask tensor and the corresponding indices used for decoder reconstruction.

[0017] 4) Construction of the Difference Condition Decoder: The student network decoder is modified by introducing a difference condition token (DC-Token) to achieve adaptive reconstruction. For each masked position i, the decoder's fixed mask embedding is replaced with a shared base mask embedding. With difference embedding The dynamic combination of differences and embeddings is generated by the ADM scores of the corresponding mask patches through MLP mapping. For each masked position i, its input token It is no longer a fixed vector, but is embedded from a shared base mask. With an explicit difference embedding Dynamic composition. Among them, The difference score of the mask patch corresponding to this position Generate. Its combination form is:

[0018] in, It is an MLP that maps scalar difference scores to a high-dimensional feature space. This represents the visible patch features from the encoder.

[0019] The difference-aware information of the encoder in the student network is passed end-to-end to the decoder, so that the decoder can obtain the prior knowledge of the difficulty of each patch before reconstruction.

[0020] 5) Multi-loss joint training; Gradient updates are performed on the student network, while the momentum teacher parameters are updated using an exponential moving average. Calculate the weighted reconstruction loss: Using the output of the momentum teacher as the reconstruction target, the loss weight is adjusted by the ADM score of the patch, and a stronger penalty is imposed on the reconstruction error in difficult regions; Weighted Reconstruction Loss It is determined by the difference score of each masked patch. Adjust the corresponding loss weights This approach ensures that reconstruction errors are more severely penalized in more challenging areas: ; in, The goal is to reconstruct the model and provide stable and high-quality feature supervision. is the reconstructed feature output of the decoder for the j-th mask patch, and is the feature prediction result of the model for the mask region; Calculate consistency auxiliary loss: Align the global output of the student network processing the masked image with the global output of the momentum teacher processing the original unmasked image to ensure the robust global representation capability of the model under the mask. Consistency Auxiliary Loss : ; in, This indicates a stop-gradient operation, represented by the student vector. It is the output obtained after processing the masked image, while the teacher vector This comes from the processing of the original unmasked image and is used as an anchor point for global representation alignment; The total loss is obtained by weighted fusion of the two types of losses. The student network is updated by gradient descent, and the momentum teacher is updated synchronously by EMA. The Oracle teacher is frozen throughout the process and does not participate in the update. Total loss Represented as: ; 6) Transfer the pre-trained student network to various scene text recognition (STR) tasks, and use the reasoning ability of difficult regions learned in the pre-training stage to achieve robust text recognition in scenes including occlusion, distortion and low resolution.

[0021] This invention proposes a novel differential-guided sequence fusion masking (DSHM) pre-training framework. This framework employs a carefully designed asymmetric teacher-student architecture, comprising one student network and two independent teacher networks. The Oracle teacher, acting as a frozen network, provides stable, high-quality features for calculating appearance differences. It primarily calculates differential scores by measuring differences between image patches and local features, as well as instance-level global style differences. These scores are first used to optimize the sequence fusion masking strategy, prioritizing the masking of high-discrepancy patches in a coherent block format while retaining some random masks to ensure training stability, thus elevating the task from pixel-level reconstruction to more complex structural inference. The student network is the only network updated via gradient descent, aiming to reconstruct masked regions using visible context. During decoding, it uses differential score mapping to obtain differential conditional tokens, encoding the difficulty prior knowledge of each patch in the decoder. This guides the decoder to focus on reconstructing difficult regions during decoding, achieving an adaptive reconstruction paradigm to improve the model's robustness in partially occluded or text-distorted scenarios. The momentum teacher is updated through the exponential moving average of the student network, providing a stable and continuously evolving reconstruction target. This role separation helps avoid feedback loop degradation during training. This invention effectively addresses the recognition challenges posed by appearance differences caused by partial occlusion or deformation.

[0022] A systematic comparison of DSHM with existing methods was conducted on general benchmark datasets, the Union14M benchmark dataset, and Chinese benchmark datasets: Table 1 Comparison on a common benchmark dataset

[0023] Table 2 Comparison on the Union14M benchmark dataset

[0024] Table 3 Comparison on Chinese benchmark datasets

[0025] The specific analysis is as follows: Results on general benchmarks: Table 1 presents the performance comparisons on general benchmarks. The results show that, using pre-trained data of similar or comparable scale, our method achieves a superior average accuracy of 97.5%. Even compared to CLIP4STR, which utilizes large-scale pre-trained data, DSHM maintains competitive performance. Furthermore, our model performs equally well on irregular datasets, demonstrating that the proposed differential guidance mechanism helps the model better understand and infer challenging regions.

[0026] Results on the Union14M Benchmark: Considering that performance on conventional benchmarks is nearing saturation, we further evaluated our method on the more challenging Union14M benchmark, with results shown in Table 2. Our method achieves new state-of-the-art (SOTA) accuracy among the latest batch of baseline methods of similar scale. Specifically, DSHM outperforms LMIM by 1.3% and 0.7% in Curve and Artistic scenarios, respectively. These improvements validate the effectiveness of DSHM in guiding the model to allocate more attention to difficult text regions during the encoding and decoding phases, thereby significantly enhancing robustness in complex scenarios.

[0027] Results on Chinese benchmarks: Unlike English characters, which primarily rely on overall shape, Chinese characters depend more heavily on stroke structure. To verify the effectiveness of our method in recognizing more complex components such as radicals and densely packed strokes, we evaluated it on Chinese benchmarks. As shown in Table 3, the method of this invention achieved a relatively leading average accuracy, indicating that the model can accurately locate text regions in Chinese images based on inter-patch differences.

[0028] We conducted further experiments in more challenging scenarios, and the results are shown in Table 4 below. Table 4 shows the experimental results for five datasets of different sizes.

[0029] The specific analysis is as follows: We evaluated DSHM on five datasets of varying sizes and increasing difficulty, primarily including weakly / heavily occluded text, irregularly shaped text with diverse forms, and low-resolution scenes. The proposed method achieved robust performance across these challenging scenarios, demonstrating its effective ability to identify dissimilar or challenging regions and to perform targeted masking based on apparent dissimilarity metrics. This indicates that our model acquires the ability to infer structural and regional differences during pre-training, further validating the generalization and robustness of DSHM in handling complex textual conditions.

[0030] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for text recognition in a sequence-mixing mask scene based on appearance differences, characterized in that, Includes the following steps: 1) Establish a differential-guided sequence mixing mask DSHM pre-training framework; The pre-training framework is an asymmetric teacher-student architecture consisting of an Oracle teacher, a momentum teacher, and a student network. 2) Quantification of apparent differences The Apparent Difference Measure (ADM) is used to quantify and score each patch of the text image, and the difference score of each patch is obtained by combining the differences in local structure and global style. Each patch of the text image is a small rectangular image unit obtained by dividing the entire text image according to a set size. 3) Sequence mixing mask generation; Sort all patches by ADM score and select the most difficult patch as the mask seed; The seed patch is expanded into blocks to generate a coherent block-shaped differential guide mask, simulating a real-world scenario of continuous character occlusion. Apply random masks to the remaining unmasked areas to ensure training stability and the model's ability to perceive the global context; Merge the two types of masks to obtain the final mask tensor and reconstruction index; 4) Construction of the Difference Condition Decoder: The student network decoder is modified by introducing a difference condition token (DC-Token) to achieve adaptive reconstruction. 5) Multi-loss joint training; Gradient updates are performed on the student network, while the momentum teacher parameters are updated using an exponential moving average. 6) Transfer the pre-trained student network to various scene text recognition (STR) tasks, and use the reasoning ability of difficult regions learned in the pre-training stage to achieve robust text recognition in scenes including occlusion, distortion and low resolution.

2. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 1, characterized in that, In step 1), the Oracle teacher provides high-quality text image features for the appearance difference measure ADM, laying the foundation for subsequent difference measures; the Oracle teacher includes a mid-level semantic feature output layer and an attention map output module. Momentum teachers provide a stable, dynamically evolving target for reconstruction, avoiding the regression of training feedback loops.

3. The method for text recognition in a sequence-mixing masked scene based on appearance differences according to claim 1, characterized in that, In step 2), the specific details are as follows: 2.1) Calculate the anisotropic local difference (ALD): Measure the directional dependence of the local feature topology by the trace of the covariance matrix of the difference vectors of each patch and its eight neighboring features; 2.2) Calculate Intra-Instance Global Style Difference (IGSD): Use Oracle teacher attention maps to select text-related patches in images, construct instance-specific style prototypes, and measure the degree of deviation from the instance's dominant style by the L2 distance between the patch features and the prototypes. 2.3) Merge the ALD and IGSD scores to obtain the final ADM difference score for each patch, thus completing the localization of difficult regions.

4. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 1, characterized in that, In step 4), the specific details are as follows: The difference-aware information of the encoder in the student network is passed end-to-end to the decoder, so that the decoder can obtain the prior knowledge of the difficulty of each patch before reconstruction. For each masked position i, the decoder's fixed mask embedding is replaced with a shared base mask embedding. With difference embedding The dynamic combination of differences and embeddings is generated by the ADM scores of the corresponding mask patches through MLP mapping; For each masked position i, its input token It is no longer a fixed vector, but is embedded from a shared base mask. With an explicit difference embedding Dynamic composition; among which, The difference score of the mask patch corresponding to this position generate.

5. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 4, characterized in that, In step 4), the shared base mask embedding With difference embedding The dynamic combination, the combination form is: in, It is an MLP that maps scalar difference scores to a high-dimensional feature space. This represents the visible patch features from the encoder.

6. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 1, characterized in that, In step 5), the loss includes: Weighted Reconstruction Loss: Using the output of the momentum teacher as the reconstruction target, the loss weight is adjusted by the ADM score of the patch, and a stronger penalty is imposed on the reconstruction error in difficult regions.

7. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 6, characterized in that, In step 5), the loss also includes: consistency auxiliary loss: aligning the global output of the student network processing the masked image with the global output of the momentum teacher processing the original unmasked image to ensure the robust global representation capability of the model under the mask.

8. The method for text recognition in a sequence mixing mask scene based on appearance difference guidance according to claim 7, characterized in that, In step 5), the loss is a weighted fusion of the two types of losses to obtain the total loss. The student network is updated through gradient descent, and the momentum teacher is updated synchronously through EMA. The Oracle teacher is frozen throughout the process and does not participate in the update.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images