Image text recognition method and system based on mask diffusion model, storage medium and equipment

Abstract

The invention provides an image text recognition method and system based on a mask diffusion model, a storage medium and equipment, and belongs to the technical field of image or video recognition or understanding. According to the method, multi-scale visual features of an image are extracted through a visual encoder, a mask diffusion decoder is combined, a diversified mask strategy and random character replacement disturbance are adopted in a training stage, denoising loss and auto-reflection loss are calculated respectively, and a model is optimized in a combined mode; in the reasoning stage, starting from a full mask state, a complete text sequence is recovered through multi-round iterative denoising. According to the method, the one-way modeling limitation of a traditional autoregression model is broken through, all-around context-dependent modeling is achieved, an autoreversion error correction mechanism and a block low-confidence mask strategy are introduced, and the recognition accuracy and reasoning efficiency in complex scenes such as shielding and fuzzy scenes are remarkably improved. The method provided by the invention reaches a leading level on a plurality of public data sets, and has the advantages of high precision and high speed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image or video recognition or understanding, relates to character recognition, relates to detecting or correcting errors, relates to image preprocessing, relates to a reference that can be adjusted through an adaptive method, and in particular relates to an image text recognition method, system, storage medium and equipment based on a mask diffusion model. BACKGROUND

[0002] Text recognition, as a basic task in optical character recognition systems, has always been a research focus in the field of computer vision. In natural scenes, text recognition faces many complex challenges, including curved and deformed text, text in different directions, occlusion, image blur, artistic fonts, etc. These challenges seriously affect the accuracy and robustness of text recognition systems in practical applications.

[0003] Existing text recognition methods can be mainly divided into the following categories:

[0004] (1) Similar to the architecture of BERT (Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [C] / / Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.). These models first generate initial results in parallel, and then improve predictions by integrating contextual information. Although this strategy alleviates the problem of lacking contextual understanding in purely parallel models, it can be very sensitive to initial prediction errors, which can propagate during the improvement process, thereby limiting their performance.

[0005] (2) Autoregressive model (Shi B, Yang M, Wang X, et al. Aster: An attentional scene text recognizer with flexible rectification[J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 41(9): 2035-2048.). This kind of method predicts characters one by one through iterative decoding, and explicitly models the context dependence between characters, so it can maintain high recognition accuracy when facing complex text deformation, cluttered background and noise. However, the inherent sequential characteristics of autoregressive model limit its decoding efficiency, which needs to generate one string line at a time, resulting in slow inference speed and difficulty in meeting real-time application requirements.

[0006] (3) Parallel decoding model (Yue X, Kuang Z, Lin C, et al. Robust scanner: Dynamically enhancing positional clues for robust text recognition[C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 135-151.). In order to improve the inference speed, this kind of method gives up the modeling of the dependence between characters, and generates the whole character sequence at the same time. Although the decoding speed is significantly improved, the recognition accuracy is generally lower than that of autoregressive model due to the lack of context modeling. In order to improve the recognition accuracy, some methods use the results of parallel decoding prediction as the initial value, and then improve the prediction results by introducing context information and combining the initial prediction confidence. However, these methods are sensitive to initial prediction errors, and errors may be propagated during the improvement process, limiting their ability to surpass autoregressive model.

[0007] As a new generation model framework, the core idea of mask diffusion model is to convert data into a masked state step by step through the masking process, and then generate new data through the reverse process. In recent years, this model has shown comparable performance to autoregressive model in language modeling and image generation tasks, and has received widespread attention due to its parallel generation potential. Unlike the one-way modeling of autoregressive model from left to right, mask diffusion model can capture more flexible and comprehensive omnidirectional dependence. The steps to apply mask diffusion model directly to text recognition task are as follows:

[0008] (1) The text to be recognized is converted into a high-dimensional vector representation.

[0009] (2) Masking some elements in the input text to form partially missing input data.

[0010] (3) Gradually remove noise from the completely noisy state through the back diffusion process to recover part or all of the original text information. In the back diffusion process, the goal of the model is to predict and recover the masked text elements. By calculating the difference between the prediction result and the original text, the model is optimized using a loss function to improve the accuracy of text recovery.

[0011] (4) In the above process, the model is trained through the backpropagation algorithm, and the parameters are updated by minimizing the loss function, so that the model can accurately predict the masked text content given the feature information.

[0012] (5) Completely masked text is input into the trained model to predict the text content based on the given feature information.

[0013] However, there are still the following key challenges in applying the mask diffusion model to the text recognition task:

[0014] (1) Noise gap between training and inference: Mask diffusion models are usually trained with random noise, but scene text as a special image faces specific noise patterns in inference, which are rarely encountered in training, leading to poor generalization performance and reduced recognition performance.

[0015] (2) Overconfident prediction in inference process: Mask diffusion models tend to assign excessively high confidence scores to prediction results, even if the prediction is wrong. This overconfidence hinders the effectiveness of the confidence-based re-masking mechanism, making it difficult for the model to identify and correct early errors.

[0016] Therefore, there is an urgent need for a new text recognition method that can fully utilize the advantages of the mask diffusion model while addressing its inherent shortcomings in the text recognition task, achieving dual improvements in recognition accuracy and inference efficiency. SUMMARY

[0017] The present application is made to solve the above problems, and aims to provide an image text recognition method, system, storage medium and device based on a mask diffusion model.

[0018] This invention provides an image text recognition method based on a mask diffusion model, characterized by the following features: the mask diffusion model includes a visual encoder and a mask diffusion decoder; the image text recognition method is trained on a training set to achieve text recognition of the image text to be recognized; the training set contains multiple training samples, each training sample including an image text to be trained and its corresponding character sequence Y. training The image text recognition method includes the following steps: S10, using a visual encoder to extract multi-scale visual feature representations from the text in the image to be trained. S20, randomly select a masking strategy for Y. training Masking is performed to obtain a noisy character sequence and then converted into a token representation T. m Finally, T m and The input mask diffusion decoder is used for denoising to learn the text structure before calculating the denoising loss. S30, for Y training After generating a perturbation sequence by performing random character substitution perturbation, it is converted into a token representation T. r T r and Input mask diffusion decoder for error correction and calculate self-reflective loss S40, using and Perform joint optimization on the mask diffusion decoder and repeat steps S20-S30 until convergence; S50, use the visual encoder to extract multi-scale visual feature representations from the text in the image to be recognized. S60, the target sequence corresponding to the text in the image to be recognized starts from the full mask state, with... The tag representation corresponding to the initial mask sequence in the fully masked state is used as input, and the mask diffusion decoder trained in step S40 is used for multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

[0019] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: In step S10, the resolution of the text in the image to be trained is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. In step S50, the resolution of the text in the image to be recognized is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. H and W represent the height and width of the image text to be trained or recognized, respectively, and D represents the feature dimension of the image text to be trained or recognized.

[0020] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: wherein step S20 includes the following sub-step: S21, randomly selecting a mask strategy pair Masking yields a noisy character sequence. , y represents the total length of the sequence. i Y represents training The i-th primitive character, M T S22 indicates the selected masking strategy; S22, Y is masked through a character embedding layer. m Transformed into D-dimensional notation T m S23, T m and The input mask diffusion decoder outputs an L×D dimensional decoded feature tensor, which is then mapped to the vocabulary space by a linear classifier to obtain the character probability distribution at each position. S24, Calculate the denoising loss: In the above formula, I1 represents Y m The number of noise markers in the data is uniformly sampled from the range [0,L]. It is an indicator function, only when Y m The character at position i has a value of 1 only when it is marked by noise in the masking strategy. Y represents m The character at position i, MASK indicates that it is a noise marker imposed by the masking strategy. This indicates that the mask diffusion decoder is in a given Y m Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

[0021] The image text recognition method based on the mask diffusion model provided by this invention may also have the following features: the masking strategy includes random masking, full masking, forward autoregressive masking, backward autoregressive masking, fill-in masking, low-confidence masking, or block low-confidence masking.

[0022] The image text recognition method based on the mask diffusion model provided by this invention may also have the following feature: wherein step S30 includes the following sub-steps: S31, for Perform random character replacement perturbation to generate the corresponding perturbation sequence. L represents the total length of the sequence, y i Y represents training The i-th original character, This represents the y-value after random character replacement perturbation. i; S32, converting Y r into a D-dimensional token representation T r ; S33, inputting T r and outputting a decoded feature tensor of LxD dimensions and mapping it to the vocabulary space through a linear classifier to obtain the character probability distribution of each position ; S34, calculating the self-reflection loss: , in the above formula, Y r is the number of tokens randomly replaced by other characters in Y , I2 is uniformly sampled from the range [0, L], r represents the probability of the mask diffusion decoder predicting the original token Y i at position i given Y is the set of probabilities of the original token Y i at all positions i.

[0023] In the image text recognition method based on the mask diffusion model provided by the application, the method can further have the following features: step S60 includes the following sub-steps: S61, setting all L positions of the target sequence corresponding to the image text to be recognized to a mask token [MASK] and generating a mask sequence , which is initially , 1≤i≤L, represents L times of splicing [MASK]; S62, starting from i=1, that is ; S63, converting to a corresponding D-dimensional token representation through a character embedding layer; S64, inputting and into the mask diffusion decoder trained in step S40, outputting a decoded feature tensor of LxD dimensions and mapping it to the vocabulary space through a linear classifier to obtain the character probability distribution of each position ; S65, applying a specified mask strategy to to generate a noisy character sequence , M T represents the specified mask strategy; S66, increasing the value of i by 1; S67, repeating steps S63-S66 a total of K times until the noise reduction is completed, and finally outputting as the recognized text sequence.

[0024] The image text recognition method based on the mask diffusion model provided by the present invention may also have the following features: wherein the mask diffusion decoder is composed of N layers of the same structure stacked together, where N is a preset layer number parameter, and each decoding layer includes a multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module and a third residual normalization module connected in sequence.

[0025] This invention also provides an image text recognition system based on a mask diffusion model, characterized by using any of the aforementioned image text recognition methods based on a mask diffusion model, comprising: a visual encoder for extracting multi-scale visual feature representations from the image text to be trained or the image text to be recognized; a mask diffusion decoder for performing denoising or error correction learning based on the input visual feature representations and the labeled representations of the character sequences; and a training module configured to perform the following operation: calling the visual encoder to extract the visual feature representations of the image text to be trained. A random masking strategy is used for Y. training Perform masking to generate a noisy character sequence and convert it into a token representation T. m The common input mask diffusion decoder is then used to calculate the denoising loss. , for Y training Perform random character substitution perturbation to generate a perturbation sequence and convert it into a token representation T. r The common input mask diffusion decoder is then used to compute the self-reflective loss. ,joint and The mask diffusion decoder is optimized until it converges; the recognition inference module is configured to perform the following operations: call the trained visual encoder to extract the visual feature representation of the text in the image to be recognized. The target sequence corresponding to the text in the image to be recognized starts from the fully masked state, and... The label representation corresponding to the initial mask sequence in the fully masked state is used as input. The mask diffusion decoder trained by the training module performs multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

[0026] The present invention also provides a storage medium having the feature of storing program code internally, wherein the program code includes the aforementioned image text recognition system based on the mask diffusion model, for implementing the image text recognition method based on the mask diffusion model of any of the preceding claims.

[0027] The present invention also provides an image text recognition device based on a mask diffusion model, which includes a processor and the aforementioned storage medium, wherein the processor is connected to the storage medium and executes program code.

[0028] The present application has the following beneficial effects:

[0029] (1) Overcome the one-way modeling limitation of autoregressive model through all-around context modeling: The present application first introduces the mask diffusion model into the text recognition task. Traditional methods are limited to one-way autoregressive or direct parallel prediction modeling, while the mask diffusion-based method realizes all-around context-dependent modeling through an iterative denoising process. Unlike the autoregressive model's string-by-string generation, the present application can simultaneously consider forward, backward and bidirectional context information, providing a new technical path for text recognition in complex scenarios.

[0030] (2) Innovatively propose a self-reflection error correction mechanism. A special self-reflection training strategy is designed to build damaged sequences by randomly replacing characters, training the model to actively identify and correct high-confidence errors. This mechanism makes the model no longer blindly believe in the confidence score, but considers visual features and language context for judgment, significantly improving the model's robustness in complex scenarios such as occlusion and blur. Only three steps of denoising are needed to achieve leading recognition accuracy, which makes the mask diffusion model-based text recognition method of the present application have the advantage of fast reasoning speed.

[0031] (3) Propose a diversified mask strategy training framework. Through the joint training of seven mask strategies, including random mask, full mask, forward / backward autoregressive, completion, low confidence and block low confidence mask, the noise distribution gap between training and reasoning phases is effectively eliminated. This design ensures that the model is familiar with various re-mask modes in the reasoning phase during training, improving the model's generalization ability.

[0032] (4) Design a block low confidence re-mask reasoning strategy. The sequence is divided into local blocks, and confidence evaluation and re-mask are performed independently within the block, successfully solving the "confidence trap" problem in traditional confidence strategies, i.e., generation stagnation caused by repeated masking at certain positions. This strategy significantly improves recognition accuracy while ensuring reasoning efficiency.

[0033] (5) The present application has universality and scalability, can flexibly configure different visual encoders and decoder architectures, and is suitable for text recognition needs in different scenarios and languages. It can effectively handle text recognition tasks in various complex scenarios, including occlusion, blur, artistic fonts, bending deformation, etc., and has broad application prospects in document digitization, intelligent transportation, augmented reality and other fields. The method of the present application has reached a leading level on multiple public data sets, especially in the occluded text recognition task, with higher recognition accuracy compared to the most advanced autoregressive model, while the reasoning speed is improved by 3 times. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1is a flow chart of the image text recognition method based on the mask diffusion model of an embodiment of the present application.

[0035] Figure 2 is an illustration of the mask strategy in step S21 of an embodiment of the present application.

[0036] Figure 3 is the result of the comparative test of the test example of the present application. DETAILED DESCRIPTION

[0037] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the following embodiments will specifically describe the image text recognition method, system, storage medium and device based on the mask diffusion model of the present application in conjunction with the accompanying drawings.

[0038] EMBODIMENT

[0039] The embodiment provides an image text recognition method based on a mask diffusion model, which is trained based on a training set to realize text recognition of an image text to be recognized. The training set contains a plurality of training samples, and each training sample includes an image text to be trained and a corresponding character sequence Y training .

[0040] The mask diffusion model used in the embodiment includes a visual encoder and a mask diffusion decoder.

[0041] The visual encoder adopts SVTRv2 (Du Y, Chen Z, Xie H, et al. Svtrv2: Ctc beats encoder-decoder models in scene text recognition[J]. arXiv preprint arXiv:2411.15858, 2024.), which is a visual encoder specially designed for image text recognition, used for extracting image features.

[0042] The mask diffusion decoder adopts a standard Transformer architecture, which is composed of N layers of decoding layers with the same structure, and N is a preset layer number parameter. Each decoding layer includes a multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module and a third residual normalization module connected in sequence. The N decoding layers are connected in sequence, and the output of the previous layer is taken as the input of the next layer, which can effectively fuse visual features and text features and capture context dependency.

[0043] Figure 1 is a flow chart of the image text recognition method based on the mask diffusion model of an embodiment of the present application.

[0044] AsFigure 1 As shown, the image text recognition method based on the mask diffusion model in this embodiment includes the following steps:

[0045] S10, using a visual encoder to extract a multi-scale feature representation from the image text to be recognized , the specific operation is as follows:

[0046] The to-be-trained image text is preprocessed, its resolution is normalized to HxW, and then input into the visual encoder to extract features, and the output dimension of the visual feature representation is H / 8xW / 4xD .

[0047] Wherein, H, W is the height, width of the to-be-trained image text, D is the feature dimension of the to-be-trained image text.

[0048] S20, after randomly selecting a mask strategy, using a mask diffusion decoder for denoising, calculating the denoising loss , specifically including the following sub-steps S21-S24:

[0049] S21, randomly select a mask strategy to Mask to get a noisy character sequence , L represents the total length of the sequence, y i represents the i-th character of Y training , M T represents the selected mask strategy.

[0050] Wherein, the selection of mask strategy M T includes any of the following, specifically as Figure 2 shown:

[0051] (a) Random mask: randomly select a certain number of positions to mask the input character sequence. The number of masks is uniformly sampled between 0 and the length of the sequence L, and the selected position replaces the original character with the [MASK] mark.

[0052] (b) Full mask: set all positions in the input character sequence to [MASK] mark, that is, completely hide the original text information.

[0053] (c) Forward autoregressive mask: simulate the step-by-step decoding process from left to right. Keep the left decoded part of the input character sequence, and set the right undecoded part to [MASK].

[0054] (d) Backward autoregressive mask: simulate the step-by-step decoding process from right to left. Keep the right decoded part of the input character sequence, and set the left undecoded part to [MASK].

[0055] (e) Filling-in mask: keep the characters at the beginning and end of the input character sequence, and mask the middle part, similar to a fill-in-the-blank task. This strategy utilizes the bidirectional context information to predict the missing characters in the middle.

[0056] (f) Low-confidence mask: selectively mask based on the confidence scores predicted by the mask diffusion decoder. Calculate the average confidence for all positions, and set the positions with lower confidence than the average to [MASK].

[0057] (g) Block low-confidence mask: divide the input character sequence into K local blocks (K is the number of subsequent denoising steps), and perform low-confidence masking independently within each block. This avoids the problem of repeated masking of certain positions caused by global comparison.

[0058] S22, convert Y m to a D-dimensional token representation T m through a character embedding layer.

[0059] S23, input T m into the mask diffusion decoder, and its last decoding layer outputs a LxD-dimensional decoding feature tensor . .

[0060] S24, calculate the denoising loss :

[0061] .

[0062] In the above formula, I1 represents the number of noisy tokens in Y m , which is uniformly sampled from the range [0, L].

[0063] is an indicator function, which is only 1 when the character at position i of Y m is masked by the masking strategy, and unmasked positions do not participate in gradient calculation. represents the character at position i of Y m , and MASK represents the token masked by the masking strategy.

[0064] represents the probability that the mask diffusion decoder predicts the original token Y m at position i of Y i given Y is the set of probabilities of the original tokens at all positions i.

[0065] ​The denoising process in steps S21-S24 aims to recover complete text from partially visible character sequences by systematically masking the mask diffusion decoder trained in step S40. It also aims to eliminate the distribution gap between training and inference by covering various masking patterns required for the inference stage in step S60, thereby improving the recognition capability of the mask diffusion decoder in complex scenes. Furthermore, the design of diverse masking strategies in step S21 ensures that the training distribution in steps S20 and S40 fully covers the masking patterns required for the inference stage in step S60. Specifically, a full mask corresponds to the initial state of inference, while the other strategies correspond to different iterative decoding paths.

[0066] S30, after performing random character replacement perturbation, use a masked diffusion decoder for error correction and calculate the self-reflective loss. Specifically, it includes the following sub-steps S31~S34:

[0067] S31, for Y training Perform random character replacement perturbation to generate the corresponding perturbation sequence Y. r Specifically, it includes the following sub-steps S311~S313:

[0068] S311, sample replacement quantity I2 from uniform distribution [0,L].

[0069] S312, randomly select I2 non-repeating set of position indices, and for each selected position, ... y at this position i Replace with other characters randomly sampled from the vocabulary. Unselected positions retain their original characters.

[0070] S313, Generate the corresponding perturbation sequence , This represents the y-value after random character replacement perturbation. i .

[0071] S32, Y is embedded through a character embedding layer r Transformed into D-dimensional notation T r .

[0072] S33, T r and After being input into the mask diffusion decoder, its last decoding layer outputs an L×D-dimensional decoded feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. .

[0073] S34, Calculate the self-reflection loss:

[0074] .

[0075] In the above formula, Y r The number of markers randomly replaced by other characters is I2, which is uniformly sampled from the range [0, L].

[0076] represents the probability of the original marker Y r at position i predicted by the mask diffusion decoder given Y i , is the set of probabilities of the original marker Y i at all positions i.

[0077] The calculation of the self-reflection loss in the above step S30 supervises the entire sequence, making the mask diffusion decoder able to identify and correct errors at any position when reasoning in step S60, which is different from the denoising loss calculated in step S20, which only supervises the mask position.

[0078] S40, jointly optimize the mask diffusion decoder using and and repeat steps S20-S30 until it converges.

[0079] S50, use the visual encoder to extract multi-scale feature representations from the image text to be recognized, which is done as follows:

[0080] The image text to be recognized is preprocessed to normalize its resolution to HxW, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8xW / 4xD.

[0081] where H and W are the height and width of the image text to be recognized, and D is the feature dimension of the image text to be recognized. The above parameters are the same as step S10.

[0082] S60, inference phase: input the image text to be recognized into the trained mask diffusion decoder, and implement text recognition through K-step denoising, which includes the following sub-steps S61-S67:

[0083] S61, set all L positions of the target sequence corresponding to the image text to be recognized to the mask marker [MASK] and generate a mask sequence , which is initially , 1≤i≤L, represents concatenating [MASK] L times.

[0084] S62, start from i=1, i.e. .

[0085] S63, the character embedding layer is used to convert the visual feature representation Y into a corresponding D-dimensional token representation T .

[0086] S64, the token representation T is input into the trained mask diffusion decoder to output a LxD-dimensional decoded feature tensor . .

[0087] S65, a mask strategy is specified to be applied to the decoded feature tensor to generate a noisy character sequence , M T denotes the specified mask strategy.

[0088] S66, the value of i is incremented by 1.

[0089] S67, steps S63-S66 are repeated a total of K times until the noise is removed, and the final output is taken as the recognized text sequence.

[0090] The embodiment also provides an image text recognition system based on a mask diffusion model, which uses the image text recognition method based on the mask diffusion model in the embodiment, and includes a visual encoder, a mask diffusion decoder, a training module, and an inference module.

[0091] The visual encoder is configured to extract multi-scale visual feature representations and from the image text to be trained or the image text to be recognized according to the method of step S10 and step S50.

[0092] The mask diffusion decoder is configured to perform denoising or error correction learning according to the input visual feature representation and token representation of the character sequence according to the method of steps S20-S40.

[0093] The training module is configured to perform the following operations: calling the visual encoder to extract the visual feature representation of the image text to be trained according to the method of step S10, generating a noisy character sequence by applying a random mask strategy to Y training and converting it into a token representation T m , and then jointly inputting the mask diffusion decoder to calculate the denoising loss , generating a perturbation sequence by randomly replacing characters according to the method of step S30, training and converting it into a token representation T r , and then jointly inputting the mask diffusion decoder to calculate the self-reflection loss​ , the method of step S40 is combined and Optimize the mask diffusion decoder until it converges.

[0094] The recognition inference module is configured to perform the following operations: calling the trained visual encoder to extract the visual feature representation of the image text to be recognized according to the method of step S50 , the method of step S60 is used to start from the full mask state to the target sequence corresponding to the image text to be recognized and the label representation corresponding to the initial mask sequence in the full mask state as input, through the mask diffusion decoder trained by the training module, multiple rounds of iterative denoising are performed, and finally the complete recognition text sequence of the target sequence is recovered.

[0095] The embodiment also provides a storage medium, which internally stores program code, and the program code comprises the image text recognition system based on the mask diffusion model in the embodiment and is used for implementing the image text recognition method based on the mask diffusion model in the embodiment.

[0096] The embodiment also provides an image text recognition device based on a mask diffusion model, which comprises a processor and the storage medium in the embodiment, the processor is connected with the storage medium and executes the aforementioned program code.

[0097] Test example

[0098] The test example uses the image text recognition system based on the mask diffusion model in the embodiment and performs actual testing according to the image text recognition method based on the mask diffusion model in the embodiment.

[0099] The method is as follows:

[0100] (1) Installation:

[0101] The operating system is Ubuntu 20.04 LTS, and the environment is CUDA 11.8.

[0102] Clone the project code repository using Git, create an independent conda virtual environment openocr, and set the Python version to 3.8.

[0103] Install PyTorch 2.2.0 and the matching torchvision 0.17.0, torchaudio 2.2.0, and ensure compatibility with CUDA 11.8. Install the dependency packages specified in requirements.txt through pip.

[0104] (2) Download the corresponding dataset.

[0105] Pre-training model weight files are provided to support direct loading and use.

[0106] (3) Through the infer.py script, specify the configuration file and image path to perform inference.

[0107] Specifically, in this test example:

[0108] (1) Use large-scale datasets for training, and use multiple standard datasets for model evaluation.

[0109] The training set in the English scenario uses the Union14M-Filter dataset, which contains 14 million filtered image text samples.

[0110] The training set in the Chinese scenario uses the BCTR dataset, which covers four types of scenes, web pages, documents, and handwriting.

[0111] Subsequent evaluation uses 6 regular datasets (ICDAR2013, SVT, IIIT5K, ICDAR2015, SVTP, CUTE80), 7 challenging subsets of Union14M-Benchmark, and the OST dataset of occluded text data.

[0112] (2) Training settings:

[0113] The model training uses the AdamW optimizer with a weight decay setting of 0.05. The English model training is set to 40 rounds, and the Chinese model training is set to 100 rounds. The learning rate is set to 5e-4, and the One-cycle learning rate scheduler is used. The English model uses linear warmup for the first 1.5 rounds, and the Chinese model uses linear warmup for the first 4.5 rounds, with an initialization ratio of 0.001. The batch size is set to 1024.

[0114] All experiments are distributed training using 4 RTX 3090 GPUs under the PyTorch deep learning framework. For fair comparison, the same experimental settings and evaluation indicators are used. The feature dimension of the visual encoder SVTRv2 is kept at 512, and the mask diffusion decoder uses a 6-layer Transformer structure. Data augmentation strategies such as rotation, perspective distortion, motion blur, and Gaussian noise are applied during training.

[0115] The loss function uses a joint loss, including a denoising loss and a self-reflection loss. The denoising loss uses cross-entropy calculation and is only supervised on the mask position, and the self-reflection loss is supervised on the entire sequence. The maximum text length is set to 25, the English vocabulary size is 94 (including uppercase and lowercase letters, numbers, and special symbols), and the Chinese vocabulary size is 6624.

[0116] (3) N=6 for the mask diffusion decoder; D=512 in step S10; K=3 in step S67.

[0117] (4) Evaluation metrics: Accuracy.

[0118] (5) Comparative experiments: Compared with the current state-of-the-art models in three benchmark tests: com (ICDAR2013, SVT, IIIT5K, ICDAR2015, SVTP, CUTE80), Union14M-Benchmark and occluded scene text dataset.

[0119] To facilitate system comparison, existing methods are divided into four decoding paradigms: connection-based temporal classification (CTC) model, ARM, PDM and ReM, as shown in Figure 3 .

[0120] As shown in Figure 3 , MDiff4STR-S and MDiff4STR-B are always superior to similar-sized competitive models on Com, U14M and OST under the ARM, PDM and REM paradigms. It is worth noting that the one-step denoising version MDiff4STR-B-PD achieves the state-of-the-art results, which is 0.31%, 0.55% and 1.30% higher than the previous best results in three benchmark tests, respectively. Further improvement is made using the dedicated decoding strategy MDiff4STR-B-BLC, which is 0.73%, 2.30% and 4.30% higher than the previous best results on Com, U14M and OST, respectively. In particular, the 4.30% gain on OST proves the effectiveness of the all-around language modeling of MDiff4STR in capturing complex contextual dependencies, providing a novel and powerful decoding paradigm for text detection.

[0121] Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An image text recognition method based on a mask diffusion model, characterized in that, The mask diffusion model includes a visual encoder and a mask diffusion decoder. The image text recognition method is trained based on a training set to achieve text recognition of the image text to be recognized. The training set contains multiple training samples, and each training sample includes an image text to be trained and its corresponding character sequence Y. training , The image text recognition method includes the following steps: S10, using the visual encoder to extract multi-scale visual feature representations from the text in the image to be trained. ; S20, randomly select a masking strategy for Y. training Masking is performed to obtain a noisy character sequence and then converted into a token representation T. m Finally, T m and The noise reduction loss is calculated after the mask diffusion decoder is input to learn the text structure. ; S30, for Y training After generating a perturbation sequence by performing random character substitution perturbation, it is converted into a token representation T. r T r and The input is given to the mask diffusion decoder for error correction and self-reflection loss is calculated. ; S40, using and Perform joint optimization on the mask diffusion decoder and repeat steps S20~S30 until it converges; S50, using the visual encoder, extract multi-scale visual feature representations from the text in the image to be recognized. ; S60, starting from the full mask state, the target sequence corresponding to the text in the image to be recognized is processed... The tag representation corresponding to the initial mask sequence in the full-mask state is used as input, and the mask diffusion decoder trained in step S40 is used for multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

2. The image text recognition method based on the mask diffusion model according to claim 1, characterized in that: in, In step S10, the resolution of the text in the image to be trained is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. , In step S50, the resolution of the text in the image to be recognized is normalized to H×W, and then input into the visual encoder to extract features, outputting a visual feature representation with dimensions H / 8×W / 4×D. , H and W are the height and width of the image text to be trained or the image text to be recognized, and D is the feature dimension of the image text to be trained or the image text to be recognized.

3. The image text recognition method based on the mask diffusion model according to claim 1, Its features are: Step S20 includes the following sub-steps: S21, randomly select a masking strategy pair Masking yields a noisy character sequence. L represents the total length of the sequence, y i Y represents training The i-th primitive character, M T This indicates the selected masking strategy; S22, Y is embedded through a character embedding layer m Transformed into D-dimensional notation T m ; S23, T m and The input to the mask diffusion decoder outputs an L×D dimensional decoded feature tensor, which is then mapped to the vocabulary space by a linear classifier to obtain the character probability distribution at each position. ; S24, Calculate the denoising loss: ， In the above formula, I1 represents Y m The number of noise markers in the data is uniformly sampled from the range [0,L]. It is an indicator function, only when Y m The character at position i has a value of 1 only when it is marked with noise by the masking strategy. Y represents m The character at position i, MASK indicates that it has been noise-marked by the masking strategy. This indicates that the mask diffusion decoder is in a given Y m Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

4. The image text recognition method based on the mask diffusion model according to claim 1 or 3, characterized in that: in, The masking strategies include random masking, full masking, forward autoregressive masking, backward autoregressive masking, fill-in masking, low-confidence masking, or block low-confidence masking.

5. The image text recognition method based on the mask diffusion model according to claim 1, characterized in that: in, Step S30 includes the following sub-steps: S31, for Perform random character replacement perturbation to generate the corresponding perturbation sequence. L represents the total length of the sequence, y i Y represents training The i-th original character, This represents the y-value after random character replacement perturbation. i ; S32, Y is embedded through a character embedding layer r Transformed into D-dimensional notation T r ; S33, T r and The input to the mask diffusion decoder outputs an L×D dimensional decoded feature tensor, which is then mapped to the vocabulary space by a linear classifier to obtain the character probability distribution at each position. ; S34, Calculate the self-reflection loss: ， In the above formula, Y r The number of tags that are randomly replaced with other characters is I2, which is uniformly sampled from the range [0,L]. This indicates that the mask diffusion decoder is in a given Y r Predict the original label Y at position i in the case of i The probability, For the original label Y corresponding to all positions i i The set of probabilities.

6. The image text recognition method based on the mask diffusion model according to claim 1, Its features are: Step S60 includes the following sub-steps: S61, set all L positions of the target sequence corresponding to the image text to be identified as mask markers [MASK] and generate a mask sequence. Its initial state was , 1≤i≤L, This indicates that [MASK] will be concatenated L times; S62, from i=1, that is start; S63, through the character embedding layer Convert to the corresponding D-dimensional symbol representation ; S64, and Input the mask diffusion decoder trained in step S40, and output an L×D dimensional decoding feature tensor. Then, a linear classifier is used to map the characters to the vocabulary space to obtain the character probability distribution at each position. ; S65, specifying one of the masking strategies to be applied to Generate a noisy character sequence. M T This indicates the specified masking strategy; S66, increment the value of i by 1; S67, repeat steps S63~S66 K times for a total of K times until noise reduction is complete, final output As a text sequence for identification.

7. The image text recognition method based on the mask diffusion model according to claim 1, characterized in that: in, The mask diffusion decoder is constructed by stacking N identical decoding layers. N is the preset number of layers. Each of the decoding layers includes a multi-head cross-attention module, a first residual normalization module, a multi-head self-attention module, a second residual normalization module, a feedforward network module, and a third residual normalization module, which are connected in sequence.

8. An image text recognition system based on a mask diffusion model, characterized in that, The image text recognition method based on the mask diffusion model according to any one of claims 1 to 7 includes: A visual encoder is used to extract multi-scale visual feature representations from the image text to be trained or the image text to be recognized; A mask diffusion decoder is used to perform denoising or error correction learning based on the input visual feature representation and the labeled representation of the character sequence; The training module is configured to perform the following operation: call the visual encoder to extract the visual feature representation of the text in the image to be trained. A random masking strategy is used for Y. training Perform masking to generate a noisy character sequence and convert it into a token representation T. m The denoising loss is then calculated by inputting the mask diffusion decoder together. , for Y training Perform random character substitution perturbation to generate a perturbation sequence and convert it into a token representation T. r The resulting input is then shared with the masked diffusion decoder to calculate the self-reflective loss. ,joint and The mask diffusion decoder is optimized until it converges; The recognition and inference module is configured to perform the following operation: call the trained visual encoder to extract the visual feature representation of the text in the image to be recognized. Starting from the full mask state, the target sequence corresponding to the text in the image to be identified is... The tag representation corresponding to the initial mask sequence in the full-mask state is used as input, and the mask diffusion decoder trained by the training module is used for multiple rounds of iterative denoising to finally recover the complete recognition text sequence of the target sequence.

9. A storage medium, characterized in that, The internal storage contains program code. The program code includes the image text recognition system based on the mask diffusion model as described in claim 8, used to implement the image text recognition method based on the mask diffusion model as described in any one of claims 1 to 7.

10. An image text recognition device based on a mask diffusion model, characterized in that, Includes a processor and the storage medium as described in claim 9. The processor is connected to the storage medium and executes the program code.

Images