Image and text recognition methods, systems, storage media, and devices based on mask diffusion models.

Abstract

This invention provides an image text recognition method, system, storage medium, and device based on a mask diffusion model, belonging to the field of image or video recognition or understanding technology. This invention extracts multi-scale visual features of images through a visual encoder and combines them with a mask diffusion decoder. During the training phase, diverse masking strategies and random character replacement perturbations are employed to calculate denoising loss and self-reflective loss respectively, jointly optimizing the model. During the inference phase, starting from the fully masked state, the complete text sequence is recovered through multiple rounds of iterative denoising. This invention overcomes the unidirectional modeling limitations of traditional autoregressive models, achieving comprehensive context-dependent modeling, and introduces a self-reflective error correction mechanism and a block low-confidence masking strategy, significantly improving recognition accuracy and inference efficiency in complex scenarios such as occlusion and blurring. The method of this invention achieves leading levels on multiple public datasets, combining the advantages of high accuracy and high speed.

Description

Technical Field

[0001] This invention belongs to the field of image or video recognition or understanding technology, and relates to character recognition, error detection or correction, image preprocessing, and references that can be adjusted by adaptive methods. Specifically, it relates to an image text recognition method, system, storage medium, and device based on a mask diffusion model. Background Technology

[0002] Text recognition, as a fundamental task in optical character recognition systems, has always been a key research focus in the field of computer vision. In natural scenes, text recognition faces numerous complex challenges, including curved and deformed text, text in different orientations, occlusion, image blurring, and artistic fonts. These challenges severely impact the accuracy and robustness of text recognition systems in practical applications.

[0003] Existing text recognition methods can be mainly classified into the following categories:

[0004] (1) Architectures similar to BERT (Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C] / / Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.). These models first generate initial results in parallel and then improve predictions by integrating contextual information. While this strategy alleviates the lack of contextual understanding in purely parallel models, it can be sensitive to initial prediction errors that may propagate during the improvement process, thus limiting their performance.

[0005] (2) Autoregressive model (Shi B, Yang M, Wang X, et al. Aster: An attentionalscene text recognizer with flexible rectification[J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 41(9): 2035-2048.). This type of method predicts characters one by one through iterative decoding, explicitly modeling the contextual dependencies between characters, thus maintaining a high recognition accuracy when faced with complex text deformation, cluttered backgrounds and noise. However, the inherent sequential nature of the autoregressive model limits its decoding efficiency, requiring the generation of each string line, resulting in slow inference speed and difficulty in meeting the needs of real-time applications.

[0006] (3) Parallel decoding models (Yue X, Kuang Z, Lin C, et al. Robustscanner: Dynamically enhancing positional clues for robust text recognition[C] / / European conference on computer vision. Cham: Springer InternationalPublishing, 2020: 135-151.). To improve inference speed, these methods abandon inter-character dependency modeling and generate the entire character sequence simultaneously. Although the decoding speed is significantly improved, the recognition accuracy is generally lower than that of autoregressive models due to the lack of contextual modeling. To improve recognition accuracy, some methods use the results of parallel decoding prediction as initial values, and then improve the prediction results by introducing contextual information and combining the initial prediction confidence. However, these methods are sensitive to initial prediction errors, and errors may propagate during the improvement process, limiting their ability to surpass autoregressive models.

[0007] The masked diffusion model, as a novel generative model framework, works by progressively transforming data into a masked state through a masking process, and then generating new data through a reverse process. In recent years, this model has demonstrated performance comparable to autoregressive models in language modeling and image generation tasks, and has attracted widespread attention due to its parallel generation potential. Unlike the unidirectional left-to-right modeling of autoregressive models, the masked diffusion model can capture more flexible and comprehensive omnidirectional dependencies. Applying the masked diffusion model directly to text recognition tasks includes the following steps:

[0008] (1) Convert the text input to be recognized into a high-dimensional vector representation.

[0009] (2) Mask some elements in the input text to form partially missing input data.

[0010] (3) Noise is gradually removed from the completely noisy state through a backdiffusion process to recover some or all of the information in the original text. In the backdiffusion process, the goal of the model is to predict and recover the masked text elements. By calculating the difference between the prediction results and the original text, the model is optimized using a loss function to improve the accuracy of text recovery.

[0011] (4) In the above process, the model is trained by backpropagation algorithm and the parameters are updated by minimizing the loss function, so that the model can accurately predict the occluded text content given feature information.

[0012] (5) Input the completely obscured text into the trained model and predict the text content based on the given feature information.

[0013] However, applying the mask diffusion model to text recognition tasks still faces the following key challenges:

[0014] (1) Noise gap between training and inference: Mask diffusion models are usually trained with random noise, but scene text, as a special type of image, faces noise patterns with specific structures during inference. These patterns are rarely encountered during training, resulting in poor generalization performance and decreased recognition performance.

[0015] (2) Overconfident prediction in the reasoning process: Masked diffusion models tend to assign excessively high confidence scores to predictions, even if the predictions are wrong. This overconfidence hinders the effectiveness of confidence-based remasking mechanisms, making it difficult for the model to identify and correct early errors.

[0016] Therefore, there is an urgent need for a new text recognition method that can fully utilize the advantages of the mask diffusion model while addressing its inherent shortcomings in text recognition tasks, thereby achieving a dual improvement in recognition accuracy and inference efficiency. Summary of the Invention

[0017] This invention is made to solve the above-mentioned problems, and aims to provide an image text recognition method, system, storage medium and device based on a mask diffusion model.

[0018] This invention provides an image text recognition method based on a mask diffusion model, characterized by the following features: the mask diffusion model includes a visual encoder and a mask diffusion decoder; the image text recognition method is trained on a training set to achieve text recognition of the image text to be recognized; the training set contains multiple training samples, each training sample including an image text to be trained and its corresponding character sequence Y. training The image text recognition method includes the following steps: S10, using a visual encoder to extract multi-scale visual feature representations from the text in the image to be trained. S20, randomly select a masking strategy for Y. training Masking is performed to obtain a noisy character sequence and then converted into a token representation T. m Finally, T m and The input mask diffusion decoder is used for denoising to learn the text structure before calculating the denoising loss. S30, for Y training After generating a perturbation sequence by performing random character substitution perturbation, it is converted into a token representation T. r T r and Input mask diffusion decoder for error correction and calculate self-reflective loss S40, using and Perform joint optimization on the mask diffusion decoder and repeat steps S20-S30 until convergence; S50, use the visual encoder to extract multi-scale visual feature representations from the text in the image to be recognized. S60, the target sequence corresponding to the text in the image to be recognized starts from the full mask state, with... The tag representation corresponding to the initial mask sequence in the fully masked state is used as input, and the mask diffusion decoder trained in step S40 is used for multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

[0019] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: In step S10, the resolution of the text in the image to be trained is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. In step S50, the resolution of the text in the image to be recognized is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. H and W represent the height and width of the image text to be trained or recognized, respectively, and D represents the feature dimension of the image text to be trained or recognized.

[0020] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: wherein step S20 includes the following sub-step: S21, randomly selecting a mask strategy pair Masking yields a noisy character sequence. , y represents the total length of the sequence. i Y represents training The i-th primitive character, M T Indicates the selected masking strategy; S22, Y is masked through a character embedding layer. m Transformed into D-dimensional notation T m S23, T m and The input mask diffusion decoder outputs an L×D dimensional decoded feature tensor, which is then mapped to the vocabulary space by a linear classifier to obtain the character probability distribution at each position. S24, Calculate the denoising loss: In the above formula, I1 represents Y m The number of noise markers in the data is uniformly sampled from the range [0,L]. It is an indicator function, only when Y m The character at position i has a value of 1 only when it is marked by noise in the masking strategy. Y represents m The character at position i, MASK indicates that it is a noise marker imposed by the masking strategy. This indicates that the mask diffusion decoder is in a given Y m Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

[0021] The image text recognition method based on the mask diffusion model provided by this invention may also have the following features: the masking strategy includes random masking, full masking, forward autoregressive masking, backward autoregressive masking, fill-in masking, low-confidence masking, or block low-confidence masking.

[0022] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: wherein step S30 includes the following sub-steps: S31, for Perform random character replacement perturbation to generate the corresponding perturbation sequence. L represents the total length of the sequence, y i Y represents training The i-th original character, This represents the y-value after random character replacement perturbation. iS32, Y is embedded through a character embedding layer. r Transformed into D-dimensional notation T r S33, T r and The input mask diffusion decoder outputs an L×D dimensional decoded feature tensor, which is then mapped to the vocabulary space by a linear classifier to obtain the character probability distribution at each position. S34, Calculate the self-reflective loss: In the above formula, Y r The number of tags that are randomly replaced with other characters is I2, which is uniformly sampled from the range [0,L]. This indicates that the mask diffusion decoder is in a given Y r Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

[0023] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: wherein step S60 includes the following sub-steps: S61, setting all L positions of the target sequence corresponding to the image text to be recognized as mask markers [MASK] and generating a mask sequence. Its initial state was , 1≤i≤L, This indicates that [MASK] will be concatenated L times; S62, starting from i=1, i.e. Start; S63, through the character embedding layer Convert to the corresponding D-dimensional symbol representation S64, will and Input the trained mask diffusion decoder from step S40 and output an L×D dimensional decoding feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. S65 specifies a masking strategy to be applied to Generate a noisy character sequence. M T S66: Indicate the specified masking strategy; S67: Increment the value of i by 1; S68: Repeat steps S63-S66 for a total of K times until noise reduction is completed, and finally output the result. As a text sequence for identification.

[0024] The image text recognition method based on the mask diffusion model provided by the present invention may also have the following features: wherein the mask diffusion decoder is composed of N layers of the same structure stacked together, where N is a preset layer number parameter, and each decoding layer includes a multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module and a third residual normalization module connected in sequence.

[0025] This invention also provides an image text recognition system based on a mask diffusion model, characterized by using any of the aforementioned image text recognition methods based on a mask diffusion model, comprising: a visual encoder for extracting multi-scale visual feature representations from the image text to be trained or the image text to be recognized; a mask diffusion decoder for performing denoising or error correction learning based on the input visual feature representations and the labeled representations of the character sequences; and a training module configured to perform the following operation: calling the visual encoder to extract the visual feature representations of the image text to be trained. A random masking strategy is used for Y. training Perform masking to generate a noisy character sequence and convert it into a token representation T. m The common input mask diffusion decoder is then used to calculate the denoising loss. , for Y training Perform random character substitution perturbation to generate a perturbation sequence and convert it into a token representation T. r The common input mask diffusion decoder is then used to compute the self-reflective loss. ,joint and The mask diffusion decoder is optimized until it converges; the recognition inference module is configured to perform the following operations: call the trained visual encoder to extract the visual feature representation of the text in the image to be recognized. The target sequence corresponding to the text in the image to be recognized starts from the fully masked state, and... The label representation corresponding to the initial mask sequence in the fully masked state is used as input. The mask diffusion decoder trained by the training module performs multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

[0026] The present invention also provides a storage medium having the feature of storing program code internally, wherein the program code includes the aforementioned image text recognition system based on the mask diffusion model, for implementing the image text recognition method based on the mask diffusion model of any of the preceding claims.

[0027] The present invention also provides an image text recognition device based on a mask diffusion model, which includes a processor and the aforementioned storage medium, wherein the processor is connected to the storage medium and executes program code.

[0028] The present invention has the following beneficial effects:

[0029] (1) Overcoming the unidirectional modeling limitation of autoregressive models through comprehensive contextual modeling: This invention introduces the mask diffusion model into text recognition tasks for the first time. Traditional methods are limited by unidirectional autoregression or direct parallel prediction modeling, while the mask diffusion-based method achieves comprehensive contextual dependency modeling through an iterative denoising process. Unlike the line-by-line generation of autoregressive models, this invention can simultaneously consider forward, backward, and bidirectional contextual information, providing a new technical path for text recognition in complex scenarios.

[0030] (2) An innovative self-reflective error correction mechanism is proposed. A specialized self-reflective training strategy is designed, which constructs damaged sequences by randomly replacing characters, and trains the model to actively identify and correct high-confidence erroneous predictions. This mechanism enables the model to no longer blindly trust confidence scores, but to make judgments by comprehensively considering visual features and linguistic context, which significantly improves the robustness of the model in complex scenarios such as occlusion and blurring. Only three steps of denoising are required to achieve leading recognition accuracy, which makes the invented text recognition method based on the mask diffusion model have the advantages of fast inference speed.

[0031] (3) A diversified masking strategy training framework is proposed. By jointly training seven masking strategies, including random masking, full masking, forward / backward autoregression, cloze test, low confidence, and block low confidence masking, the noise distribution gap between the training and inference stages is effectively eliminated. This design ensures that the model is familiar with various re-masking patterns in the inference stage during training, thereby improving the model's generalization ability.

[0032] (4) Design a low-confidence remasking inference strategy. The sequence is divided into local blocks, and confidence evaluation and remasking are performed independently within each block. This successfully solves the "confidence trap" problem in traditional confidence strategies, namely, the generation stagnation caused by repeated masking at certain positions. This strategy significantly improves the recognition accuracy while ensuring inference efficiency.

[0033] (5) This invention is universal and scalable, and can flexibly configure different visual encoder and decoder architectures to adapt to the text recognition needs of different scenarios and languages. It can effectively handle text recognition tasks in various complex scenarios, including occlusion, blurring, artistic fonts, and bending deformation, and has broad application prospects in multiple fields such as document digitization, intelligent transportation, and augmented reality. The method of this invention has achieved leading levels on multiple public datasets, especially in the occluded text recognition task, where it has higher recognition accuracy than the state-of-the-art autoregressive models, while the inference speed is improved by 3 times. Attached Figure Description

[0034] Figure 1This is a flowchart of an image text recognition method based on a mask diffusion model, according to an embodiment of the present invention.

[0035] Figure 2 This is a diagram illustrating the masking strategy in step S21 of an embodiment of the present invention.

[0036] Figure 3 This is the result of a comparative test of the test examples of the present invention. Detailed Implementation

[0037] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the image text recognition method, system, storage medium and device based on the mask diffusion model of the present invention.

[0038] Example

[0039] This embodiment provides an image text recognition method based on a mask diffusion model. It is trained on a training set to achieve text recognition of the image text to be recognized. The training set contains multiple training samples, each including a text image to be trained and its corresponding character sequence Y. training .

[0040] The mask diffusion model used in this embodiment includes a visual encoder and a mask diffusion decoder.

[0041] The visual encoder uses SVTRv2 (Du Y, Chen Z, Xie H, et al. Svtrv2: Ctc beatsencoder-decoder models in scene text recognition[J]. arXiv preprint arXiv:2411.15858, 2024.), which is a visual encoder specifically designed for image text recognition to extract image features.

[0042] The mask diffusion decoder adopts the standard Transformer architecture, consisting of N stacked decoding layers with the same structure, where N is a preset layer number parameter. Each decoding layer includes a sequentially connected multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module, and a third residual normalization module. The N decoding layers are connected sequentially, with the output of the previous layer serving as the input of the next layer, effectively fusing visual and textual features and capturing contextual dependencies.

[0043] Figure 1 This is a flowchart of an image text recognition method based on a mask diffusion model, according to an embodiment of the present invention.

[0044] like Figure 1 As shown, an image text recognition method based on a mask diffusion model in this embodiment includes the following steps:

[0045] S10, using a visual encoder to extract multi-scale feature representations from the text in the image to be recognized. The specific steps are as follows:

[0046] The text in the training image is preprocessed to normalize its resolution to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. .

[0047] Where H and W are the height and width of the text in the image to be trained, and D is the feature dimension of the text in the image to be trained.

[0048] S20, After randomly selecting and implementing the masking strategy, use a mask diffusion decoder for denoising and calculate the denoising loss. Specifically, it includes the following sub-steps S21~S24:

[0049] S21, randomly select a masking strategy pair Masking yields a noisy character sequence. L represents the total length of the sequence, y i Y represents training The i-th character, M T This indicates the masking strategy selected.

[0050] Among them, masking strategy M T The options include any of the following, specifically as follows: Figure 2 As shown:

[0051] (a) Random Masking: For the input character sequence, a certain number of positions are randomly selected for masking. The number of masking positions is uniformly sampled from 0 to the sequence length L, and the original characters at the selected positions are replaced with the [MASK] marker.

[0052] (b) Full mask: Set all positions in the input character sequence to the [MASK] flag, which completely hides the original text information.

[0053] (c) Forward Autoregressive Mask: Simulates the stepwise decoding process from left to right. The decoded portion on the left side of the input character sequence is retained, and the undecoded portion on the right side is set as [MASK].

[0054] (d) Backward autoregressive mask: Simulates the stepwise decoding process from right to left. The decoded portion on the right side of the input character sequence is retained, and the undecoded portion on the left side is set as [MASK].

[0055] (e) Fill-in-the-blank mask: Preserve the characters at both ends of the input character sequence and mask the middle part, similar to a cloze test task. This strategy uses bidirectional contextual information to predict the missing characters in the middle.

[0056] (f) Low-confidence masking: Selective masking is performed based on the confidence scores predicted by the mask diffusion decoder. The average confidence score is calculated for all positions, and positions below the average confidence score are reset to [MASK].

[0057] (g) Block Low-Confidence Masking: The input character sequence is divided into K local blocks (K is the number of subsequent denoising steps), and low-confidence masking is performed independently within each block. This avoids the problem that global comparisons may cause some positions to be masked repeatedly.

[0058] S22, Y is embedded through a character embedding layer m Transformed into D-dimensional notation T m .

[0059] S23, T m and After being input into the mask diffusion decoder, its last decoding layer outputs an L×D-dimensional decoded feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. .

[0060] S24, Calculate the denoising loss :

[0061] .

[0062] In the above formula, I1 represents Y m The number of noise markers is uniformly sampled from the range [0,L].

[0063] It is an indicator function, only when Y m The value of the character at position i is 1 only when it is marked by noise in the masking strategy; unmasked positions do not participate in gradient calculation. Y represents m The character at position i, MASK indicates that it is a noise marker by the masking strategy.

[0064] This indicates that the mask diffusion decoder is in a given Y m Predict the original label Y at position i in the case of i The probability, For the original label at all positions i The set of probabilities.

[0065] The denoising process in steps S21-S24 aims to recover complete text from partially visible character sequences by systematically masking the mask diffusion decoder trained in step S40. It also aims to eliminate the distribution gap between training and inference by covering various masking patterns required for the inference stage in step S60, thereby improving the recognition capability of the mask diffusion decoder in complex scenes. Furthermore, the design of diverse masking strategies in step S21 ensures that the training distribution in steps S20 and S40 fully covers the masking patterns required for the inference stage in step S60. Specifically, a full mask corresponds to the initial state of inference, while the other strategies correspond to different iterative decoding paths.

[0066] S30, after performing random character replacement perturbation, use a masked diffusion decoder for error correction and calculate the self-reflective loss. Specifically, it includes the following sub-steps S31~S34:

[0067] S31, for Y training Perform random character replacement perturbation to generate the corresponding perturbation sequence Y. r Specifically, it includes the following sub-steps S311~S313:

[0068] S311, sample replacement quantity I2 from uniform distribution [0,L].

[0069] S312, randomly select I2 non-repeating set of position indices, and for each selected position, ... y at this position i Replace with other characters randomly sampled from the vocabulary. Unselected positions retain their original characters.

[0070] S313, Generate the corresponding perturbation sequence , This represents the y-value after random character replacement perturbation. i .

[0071] S32, Y is embedded through a character embedding layer r Transformed into D-dimensional notation T r .

[0072] S33, T r and After being input into the mask diffusion decoder, its last decoding layer outputs an L×D-dimensional decoded feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. .

[0073] S34, Calculate the self-reflection loss:

[0074] .

[0075] In the above formula, Y r The number of tags that are randomly replaced with other characters is I2, which is uniformly sampled from the range [0,L].

[0076] This indicates that the mask diffusion decoder is in a given Y r Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

[0077] The calculation of the self-reflective loss in step S30 and the calculation of the denoising loss in step S20 differ only in the supervision of the mask position. The self-reflective loss supervises the entire sequence, enabling the mask diffusion decoder to identify and correct errors at any position during inference in step S60.

[0078] S40, using and Perform joint optimization on the mask diffusion decoder and repeat steps S20~S30 until it converges.

[0079] S50 uses a visual encoder to extract multi-scale feature representations from text in an image to be recognized. The specific steps are as follows:

[0080] The text in the image to be recognized is preprocessed, and its resolution is normalized to H×W. Then, it is input into the visual encoder to extract features, and the output is a visual feature representation with dimensions H / 8×W / 4×D. .

[0081] Where H and W are the height and width of the text in the image to be recognized, and D is the feature dimension of the text in the image to be recognized. The above parameters are the same as in step S10.

[0082] S60, Inference Stage: The text in the image to be recognized is input into the trained mask diffusion decoder, and text recognition is achieved through K-step denoising. Specifically, it includes the following sub-steps S61~S67:

[0083] S61, set all L positions of the target sequence corresponding to the text in the image to be recognized as mask markers [MASK] and generate a mask sequence. Its initial state was , 1≤i≤L, This indicates that [MASK] will be concatenated L times.

[0084] S62, from i=1, that is start.

[0085] S63, through the character embedding layer Convert to the corresponding D-dimensional symbol representation .

[0086] S64, and Input the trained mask diffusion decoder from step S40 and output an L×D dimensional decoding feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. .

[0087] S65 specifies a masking strategy to be applied to Generate a noisy character sequence. M T Indicates the specified masking strategy.

[0088] S66, increment the value of i by 1.

[0089] S67, repeat steps S63~S66 K times for a total of K times until noise reduction is complete, final output As a text sequence for identification.

[0090] This embodiment also provides an image text recognition system based on a mask diffusion model, which uses the image text recognition method based on a mask diffusion model in this embodiment, including a visual encoder, a mask diffusion decoder, a training module, and a recognition inference module.

[0091] The visual encoder is used to extract multi-scale visual feature representations from the training image text or the recognition image text according to the methods in steps S10 and S50. and .

[0092] The mask diffusion decoder is used to perform denoising or error correction learning based on the visual feature representation of the input and the labeled representation of the character sequence according to the method in steps S20 to S40.

[0093] The training module is configured to perform the following operations: Following the method in step S10, the visual encoder is invoked to extract the visual feature representation of the text in the image to be trained. Following the method in step S20, a random masking strategy is used to apply Y. training Perform masking to generate a noisy character sequence and convert it into a token representation T. m The common input mask diffusion decoder is then used to calculate the denoising loss. Follow the method in step S30 to process Y training Perform random character substitution perturbation to generate a perturbation sequence and convert it into a token representation T. r The common input mask diffusion decoder is then used to compute the self-reflective loss. Combined according to the method in step S40 and The mask diffusion decoder is optimized until it converges.

[0094] The recognition and inference module is configured to perform the following operation: Following the method in step S50, the trained visual encoder is invoked to extract the visual feature representation of the text in the image to be recognized. Following the method in step S60, the target sequence corresponding to the text in the image to be recognized starts from the fully masked state, and... The label representation corresponding to the initial mask sequence in the fully masked state is used as input. The mask diffusion decoder trained by the training module performs multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

[0095] This embodiment also provides a storage medium that stores program code, which includes the image text recognition system based on the mask diffusion model in this embodiment, and is used to implement the image text recognition method based on the mask diffusion model in this embodiment.

[0096] This embodiment also provides an image text recognition device based on a mask diffusion model, including a processor and a storage medium as described in this embodiment. The processor is connected to the storage medium and executes the aforementioned program code.

[0097] Test case

[0098] This test case uses the image text recognition system based on the mask diffusion model in the embodiments, and performs actual testing according to the image text recognition method based on the mask diffusion model in the embodiments.

[0099] The usage method is as follows:

[0100] (1) Installation:

[0101] The operating system is Ubuntu 20.04 LTS, and the environment is CUDA 11.8.

[0102] Clone the project repository using Git, create a separate conda virtual environment named openocr, and set the Python version to 3.8.

[0103] Install PyTorch 2.2.0 and its companion torchvision 0.17.0 and torchaudio 2.2.0, ensuring compatibility with CUDA 11.8. Install the dependencies specified in requirements.txt via pip.

[0104] (2) Download the corresponding dataset.

[0105] Pre-trained model weight files are provided and can be loaded and used directly.

[0106] (3) Inference can be performed by specifying the configuration file and image path through the infer.py script.

[0107] Specifically in this test case:

[0108] (1) Use large-scale datasets for training and multiple standard datasets for model evaluation.

[0109] The training set for English scenarios uses the Union14M-Filter dataset, which contains 14 million filtered image-text samples.

[0110] The training set for Chinese scenarios uses the BCTR dataset, which covers four types: scenes, web pages, documents, and handwriting.

[0111] Subsequent evaluations used six regular datasets (ICDAR2013, SVT, IIIT5K, ICDAR2015, SVTP, CUTE80), seven challenging subsets of the Union14M-Benchmark dataset, and the occluded text dataset OST.

[0112] (2) Training settings:

[0113] The model was trained using the AdamW optimizer with a weight decay of 0.05. The English model underwent 40 training epochs, while the Chinese model underwent 100 training epochs. The learning rate was set to 5e-4, and a one-cycle learning rate scheduler was used. The English model used a linear warmup for the first 1.5 epochs, and the Chinese model used a linear warmup for the first 4.5 epochs, with an initial ratio of 0.001. The batch size was set to 1024.

[0114] All experiments were conducted using a distributed training architecture with four RTX 3090 GPUs within the PyTorch deep learning framework. For fair comparison, the same experimental settings and evaluation metrics were used consistently. The visual encoder SVTRv2 maintained a feature dimension of 512, and the mask diffusion decoder employed a 6-layer Transformer structure. Data augmentation strategies such as rotation, perspective distortion, motion blur, and Gaussian noise were applied during training.

[0115] The loss function employs a joint loss mechanism, comprising denoising loss and reflexive loss. The denoising loss is calculated using cross-entropy and is supervised only at the mask positions, while the reflexive loss is supervised across the entire sequence. The maximum text length is set to 25, the English vocabulary size is 94 (including uppercase and lowercase letters, numbers, and special symbols), and the Chinese vocabulary size is 6624.

[0116] (3) N=6 for mask diffusion decoder; D is 512 in step S10; K is 3 in step S67.

[0117] (4) Evaluation indicator: accuracy.

[0118] (5) Comparative experiments: Comparison with the current state-of-the-art models in three benchmark tests: com (ICDAR2013, SVT, IIIT5K, ICDAR2015, SVTP, CUTE80), Union14M-Benchmark and occludedscene text dataset.

[0119] For ease of system comparison, existing methods are divided into four decoding paradigms: connection-based temporal classification (CTC) model, ARM, PDM, and ReM, such as Figure 3 As shown.

[0120] like Figure 3 As shown, under the ARM, PDM, and REM paradigms, MDiff4STR-S and MDiff4STR-B consistently outperform competing models of similar size on Com, U14M, and OST. Notably, the one-step denoising version, MDiff4STR-B-PD, achieves state-of-the-art results, outperforming previous bests by 0.31%, 0.55%, and 1.30% on the three benchmarks, respectively. Further improvements using the dedicated decoding strategy MDiff4STR-B-BLC achieve improvements of 0.73%, 2.30%, and 4.30% on Com, U14M, and OST, respectively. In particular, the 4.30% gain on OST demonstrates the effectiveness of MDiff4STR's comprehensive language modeling in capturing complex contextual dependencies, providing a novel and powerful decoding paradigm for text detection.

[0121] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for image text recognition based on a mask diffusion model, characterized in that, The mask diffusion model comprises a visual encoder and a mask diffusion decoder, and the image text recognition method is trained based on a training set, so as to realize text recognition of an image text to be recognized, wherein the training set comprises a plurality of training samples, and each training sample comprises an image text to be trained and a corresponding character sequence Y training , the method for image text recognition based on a mask diffusion model comprises the following steps: S10, extracting multi-scale visual feature representations from the image text to be trained using the visual encoder ; S20, randomly select a mask strategy to Y training mask to get a noisy character sequence and convert it to token representation T m , finally T m and input the mask diffusion decoder to denoise to learn the text structure and calculate the denoising loss ; S30, to Y training S30, to Y r S30, to Y r S30, to Y S30, to Y ; S40, using and jointly optimizing the mask spreading decoder and repeating steps S20-S30 until it converges; S50, using the visual encoder to extract multi-scale visual feature representations from the image text to be recognized ; S60, starting from the full mask state, the target sequence corresponding to the image text to be recognized is iteratively denoised by the mask diffusion decoder trained in step S40, and the complete recognition text sequence of the target sequence is finally recovered. and the label representation corresponding to the initial mask sequence in the full mask state as input, the mask diffusion decoder trained in step S40 is iteratively denoised for multiple rounds, and the complete recognition text sequence of the target sequence is finally recovered.

2. The method for image text recognition based on a mask diffusion model according to claim 1, characterized in that: wherein In step S10, the resolution of the image text to be trained is normalized to HxW, and then input into the visual encoder to extract features, outputting a visual feature representation with a dimension of H / 8xW / 4xD , In step S50, the resolution of the image text to be recognized is normalized to HxW, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8xW / 4xD , H, W are the height and width of the to-be-trained image text or the to-be-recognized image text, and D is the feature dimension of the to-be-trained image text or the to-be-recognized image text.

3. The image text recognition method based on a masked diffusion model according to claim 1, characterized in that: wherein step S20 comprises the following sub-steps: S21, randomly selecting a mask strategy to mask to obtain a noisy character sequence , L represents the total length of the sequence, y i represents the i-th original character of Y training , M T represents the selected mask strategy; S22, converting Y into a D-dimensional token representation T by a character embedding layer m S22, converting Y into a D-dimensional token representation T by a character embedding layer m ; S23, T m With The mask diffusion decoder is inputted to output the decoded feature tensor of LxD dimension, and the linear classifier is used to map it to the vocabulary space to obtain the character probability distribution of each position ; S24, calculating a denoising loss: ， In the above formula, I1represents Y m The number of noise markers in Y is uniformly sampled from the range [0, L], is an indicator function that is 1 only when Y m is 1 if the character at position i of Y denotes the character at position i of Y m is marked by the mask policy noise, MASK denotes that represents the probability of the original label Y m at position i given the mask diffusion decoder Y i , is the set of probabilities of the original label Y i at all positions i.

4. The method for image text recognition based on a mask diffusion model according to claim 1 or 3, characterized in that: wherein the mask strategy comprises random mask, full mask, forward autoregressive mask, backward autoregressive mask, fill-in mask, low confidence mask or block low confidence mask.

5. The method for image text recognition based on a mask diffusion model according to claim 1, characterized in that: wherein step S30 comprises the following sub-steps: S31, to performing random character replacement scrambling to generate a corresponding sequence of perturbations , L denotes the total length of the sequence, y i denotes the i-th original character of Y training denotes the corresponding y i after the random character replacement perturbation​ S32, convert Y into D-dimensional label representation T by character embedding layer r r ;​ S33, T r With The mask diffusion decoder is inputted to output the decoded feature tensor of LxD dimension, and the linear classifier is used to map it to the vocabulary space to obtain the character probability distribution of each position ; S34, calculating a self-reflection loss: ， In the above formula, Y r The number of tokens in which the middle is randomly replaced by other characters is I2, which is uniformly sampled from the range [0, L], represents the probability of the original label Y r at position i given the mask diffusion decoder Y i is the set of probabilities of the original label Y i at all positions i.​ 6. The image text recognition method based on a masked diffusion model according to claim 1, characterized in that: wherein step S60 comprises the following sub-steps: S61, set all L positions of the target sequence corresponding to the image text to be identified as mask marks [MASK] and generate a mask sequence , which is initially , 1≤i≤L, denotes L times splicing of [MASK]; S62, from i = 1, i.e. start; S63, converting the input data into a corresponding D-dimensional token representation by a character embedding layer transformed into a corresponding D-dimensional token representation ; S64, to and In the trained mask diffusion decoder, the input step S40 outputs a decoded feature tensor of LxD dimensions and maps it to the vocabulary space through a linear classifier to obtain the character probability distribution for each position ; S65, specifying that a said mask policy applies to , generating a noisy character sequence , M T denotes a specified said mask policy; S66, adding 1 to the value of i; S67, repeating steps S63-S66 a total of K times until the noise reduction is completed, and finally outputting As a recognition text sequence.

7. The method for image text recognition based on a mask diffusion model according to claim 1, characterized in that: wherein, the mask diffusion decoder is composed of a stack of decoding layers with the same structure of N layers, N is a preset number of layers parameter, each of the decoding layers comprises a multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module and a third residual normalization module connected in sequence.

8. An image text recognition system based on a mask diffusion model, characterized in that, The method for image text recognition based on a mask diffusion model according to any one of claims 1-7 is used, comprising: a visual encoder configured to extract multi-scale visual feature representations from the to-be-trained image text or the to-be-recognized image text; a mask diffusion decoder configured to perform denoising or error correction learning according to the input visual feature representations and the tokenized representations of the character sequence; a training module configured to perform the following operations: calling the visual encoder to extract visual feature representation of the image text to be trained , generating a noisy character sequence by masking Y training with a random mask strategy and converting it into token representation T m , and jointly inputting the mask diffusion decoder to calculate the denoising loss , generating a perturbation sequence by random character replacement perturbation on Y training and converting it into token representation T r , and jointly inputting the mask diffusion decoder to calculate the self-reflection loss , optimizing the mask diffusion decoder jointly and until it converges; The recognition and inference module is configured to perform the following operation: call the trained visual encoder to extract the visual feature representation of the text in the image to be recognized. Starting from the full mask state, the target sequence corresponding to the text in the image to be identified is... The tag representation corresponding to the initial mask sequence in the full-mask state is used as input, and the mask diffusion decoder trained by the training module is used for multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

9. A storage medium, characterized by an internal storage having program codes, wherein the program codes comprise the image text recognition system based on a mask diffusion model according to claim 8, configured to implement the method for image text recognition based on a mask diffusion model according to any one of claims 1-7.

10. An image text recognition device based on a mask diffusion model, characterized in that, comprising a processor and the storage medium according to claim 9, the processor is connected with the storage medium and executes the program codes. the processor is connected with the storage medium and executes the program codes.

Images