Multilingual mosaic image text recognition method based on deep learning

Abstract

The present application is based on the recognition process of human visual cortex observation-exercise-perfection, and proposes a mosaic text recognition method: synthetic data and public benchmark data sets are used to construct original image-mosaic image data pairs; first, the mosaic image data is sent to the image encoder in the "observation" module to extract features hierarchically; then, the image is preliminarily restored through the reconstruction module; inspired by the mask autoregressive encoder, self-supervised learning is used to simulate the "exercise" process of human beings, and the visual self-attention module is used to predict the pixel value; finally, the denoising diffusion probability model of the "perfection" module trains the U-shaped neural network to predict the noise of each step to realize iterative optimization; the weight parameters are optimized by back propagation; and the open source recognition network or cloud service API is used to recognize the character sequence. The present application provides a thought and feasible scheme for the problem of multi-lingual mosaic text image recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and computer vision, and more particularly relates to a multi-lingual mosaic image text recognition method based on deep learning. BACKGROUND

[0002] Text is the most important communication tool and information carrier in human society, and is the basic element and distinctive mark of culture. "There is no history without text", which also shows the important position of text in the continuation of history. Nowadays, with the development and progress of instant messaging and video technology, people have invented mosaic, a processing method based on image pixel blurring, to protect or hide some private information. In mathematical logic, the information loss caused by mosaic is irreversible, which also poses great challenges to mosaic text recognition. With the promotion of deep neural networks, image text recognition technology has made significant progress in the past decade, promoting the development of file scanning, data archiving, ancient book recognition and other applications, which also makes it possible to recognize mosaic image text.

[0003] Traditional visual image restoration methods usually use digital image processing techniques such as interpolation, degradation model and sparse representation, which reconstruct images with the help of adjacent or dominant pixel values. Although such traditional methods have greatly promoted the development of this field, they often lack the expressiveness of complex dependencies when faced with mosaic images, resulting in poor results. In recent years, emerging deep learning technologies have shown amazing results in object detection and instance segmentation, gradually creating a wave in various visual fields, and also bringing new ideas and entry points to the difficult task of mosaic image text recognition. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application proposes a multi-lingual mosaic image text recognition method based on deep learning, which excavates the difficult task of mosaic image text recognition which has not been given due attention.

[0005] To achieve the above purpose, the present application provides a multi-lingual mosaic image text recognition method based on deep learning, which comprises:

[0006] Step one: data acquisition, synthetic text image data and public real scene text benchmark data set are used as original training data to form original image-mosaic image data pairs;

[0007] Step two: data preprocessing and data enhancement: first, adjust the input image to a uniform size and resolution; next, perform data enhancement on the adjusted image to improve the model's generalization ability to different input data.

[0008] Step three: the "observation" module is composed of an image encoder and a reconstruction module, which first sends the mosaic image data into the image encoder and then hierarchically extracts image features and text features using convolutional layers and mobile window-based visual self-attention models; then the sub-pixel convolution of the reconstruction module is further used to upsample the image to remove the mosaic and improve the image quality and resolution.

[0009] Step four: the "practice" module is composed of a decoder composed of an image encoder with the same shared parameters as the "observation" module and a series of visual self-attention modules, inspired by the masked autoregressive encoder, using a large amount of data for self-supervised learning to mimic the human "practice" process to predict the pixel value of the masked image patch;

[0010] Step five: the "refinement" module is composed of a denoising diffusion probability model, which mainly has two steps of forward noise addition and reverse noise removal: the forward noise addition process can be regarded as a Markov chain model that continuously adds Gaussian noise to the original image until it becomes a pure noise image, and the reverse noise removal process uses the original image and the noise image to train a U-shaped neural network to predict the noise at each step to achieve the effect of image restoration;

[0011] Step six: the "observation" module and the "refinement" module use the restored images output by the two to calculate the pixel loss and the peak signal-to-noise ratio with the original image; and the "practice" module uses the predicted masked image patch pixel to calculate the mean square error with the true pixel to assist and accelerate the training, and optimizes the recognition network weight parameters through back propagation;

[0012] Step seven: in the inference stage, the restored image is sent into the recognition network or cloud service API to complete the recognition of the mosaic multilingual text image and output the predicted character sequence, and the prediction accuracy and the normalized edit distance are used as the evaluation indicators.

[0013] In an embodiment of the present application, the certain number of pairs of original image-mosaic image data sets synthesized in step one need to be one-to-one corresponding, and the mosaic image will be sent into the "observation", "practice" and "refinement" modules of the subsequent network to gradually restore the image using visual self-attention models, masked autoencoders and denoising diffusion models.

[0014] In one embodiment of the present application, in the step one: since there is no public mosaic multilingual dataset at present, in order to verify the model generalization ability of the proposed method in real scene, we use the open source tool SynthTiger, which is a kind of synthetic text image generator, to generate multilingual mosaic synthetic training images, and adopt the public scene text recognition benchmark dataset, including (1) IIIT5kWords (IIIT); (2) Street View Text (SVT); (3) ICDAR 2013 (IC13); (4) ICDAR 2015 Incidental Text (IC15). The grid size of the generated mosaic image is 5x5, 10x10 and 15x15. In our experiment, we use the synthetic dataset of six languages, including English, Chinese, Bengali, Japanese, German and French data. At the same time, we also adopt the method of Gaussian noise, random blur, rotation clipping and various background fonts when generating data to make the synthetic data closer to the real scene data and make the test set and the training set diverse.

[0015] In one embodiment of the present application, the step two specifically comprises: given a color RGB original image-mosaic text picture pair Firstly, the height and width of the input image are obtained, and the scaling mode that is most suitable and can maximize the preservation of the shape of the original image text is selected according to the size relationship between the ratio of the two and 1 / 4 to adjust the input image to an image with a uniform height of 32 and a width of 128 resolution Next, we perform random horizontal or vertical flip data enhancement on the adjusted image to improve the model's generalization ability to different input data, and finally the enhanced image is sent as input into the image encoder of the "observation" module, the details of which will be introduced below.

[0016] In one embodiment of the present application, the step three specifically comprises: the "observation" module is composed of an image encoder and a reconstruction module, and the preprocessed mosaic image data is sent into the image encoder to further hierarchically extract the shallow and deep features of the image using convolutional layers and mobile window-based visual self-attention models to model and analyze image features and text features, which are respectively the representations of image low-frequency and high-frequency information, and then obtain F p =Encoder(X′;Θ enc ), Θ enc represents the parameters of the image encoder, Then the image is up-sampled by the sub-pixel convolution of the reconstruction module to remove the mosaic and improve the image quality and resolution, and the output is obtained Recons(·) represents a sub-pixel convolution image reconstruction module, which has a skip connection between shallow features and deep features, and the two are respectively the representations of low-frequency and high-frequency information of the image, Θ recons represents the parameters of the image reconstruction module, Then, according to the output and Y', the L1 pixel loss is calculated as The parameters Θ enc and Θ recons are used to optimize the parameters.

[0017] In an embodiment of the present application, the step four specifically comprises: in order to ensure the continuity of parameter optimization, the "practice" module is composed of an image encoder and a series of visual self-attention modules of a decoder sharing the same parameters with the "observation" module, inspired by the masked autoregressive encoder, we divide the image into non-overlapping image patch blocks with a size of 2x2 and randomly sample a subset of image patches The sampled visible patch subset is input into the image encoder after a linear mapping layer, while the remaining image patches are masked and removed, thereby greatly reducing the calculation demand and storage memory, wherein N and M respectively represent the number of visible and invisible patch blocks. We use a large amount of data for self-supervised learning to simulate the human "practice" process, and then the encoded visible patch F m =Encoder(Linear(Z′);Θ enc )(wherein Θ dec is the encoder parameter) and the shared and learnable mask image patch are spliced into and input into the decoder composed of a series of visual self-attention modules to predict the pixel value Z m =DecOder(F′ m ;Θ dec ), and the last layer of the decoder is a linear mapping layer so that the output satisfies Θ dec is the decoder parameter, and finally only the mask patch part Z′ m in the output Z m is used to calculate the mean square error L dec =MSE(Z′ m ,Z″) for optimizing the parameters Θ dec and Θ enc .

[0018] In one embodiment of the present application, the step five specifically comprises: inspired by the denoising diffusion probability model, we adopt the conditional denoising diffusion model to start from a pure Gaussian noise, based on the rough recovery image iterative optimization to simulate the "refinement" link of human cognitive process. Given the rough recovery image output by the image recovery module and the original normal image Y', the "refinement" iteration module aims to learn the mapping from the standard normal distribution to the empirical data distribution conditioned on the rough recovery image . Specifically, the U-shaped neural network (U-Net) model is trained conditioned on the rough recovery image , which gradually removes various degrees of noise from the pure noise image

[0019]

[0020] where T is the number of diffusion steps, set to 2000, and the noise added in each iteration of the forward process is according to the following formula:

[0021]

[0022] where α 1：T is a hyperparameter between 0 and 1, which determines the noise variable at each step, and I is the standard unit matrix. The forward process supports sampling at any time step t given the original image y0, which can be represented by the following formula:

[0023]

[0024] where This formula is very helpful for fast sampling y t at step t, which can be expressed by the following formula:

[0025]

[0026] For a given conditional rough image and a noisy target image y t , we train a U-Net as a denoising model, denoted as to predict the noise vector ∈, where is the conditional rough recovery image, and γ is the statistical quantity of the noise variance. Finally, we minimize the diffusion loss term, i.e.:

[0027]

[0028] where γ ~ p(γ) and set During training, we first uniformly sample a time step t ~ {0,..., T}, and then sample γ from the uniform distribution U(γ t-1 and γ t ~ U(γ t-1 , γ t ). Furthermore, using Bayes' rule, we can derive the posterior distribution y t | y0, y t-1 as follows:

[0029]

[0030]

[0031]

[0032] The reverse denoising process is given by parameters θ, defined as:

[0033]

[0034]

[0035]

[0036] where the reverse process converts the latent variable distribution p θ (y T ) to the data distribution p θ (y0), with x as the coarse recovered image. Combining the above equations and substituting y0 into the posterior distribution q(y t-1 | y0, y t ) parameterizes the mean of p θ (y t-1 | y t , x) as:

[0037]

[0038] Finally in the inference phase, the reverse process is reparameterized to obtain:

[0039]

[0040] where The model ultimately uses the predicted as output.

[0041] In an embodiment of the present application, step six specifically includes: we divide the training process of the model into two stages, first train the image encoder shared by "Observation" and "Practice", and then train the iterative optimization module denoising diffusion network. For the two training stages, we use the Adam optimizer and set the corresponding hyperparameters, where the learning rate of the first stage is 5e-5, 10 rounds of training are performed, and the batch size is set to 32; the learning rate of the second stage is 12e-6, 50 rounds of training are performed, and the batch size is 8. The cost function of network training is composed of the reconstruction loss function and the pixel loss function, and the formula is: L = L rec +L dec , wherein L rec is the image reconstruction module of the "Observation" module and the prediction output of the "Refinement" module and the L1 pixel loss calculated by Y' is: L dec is the mean square error L dec = MSE(Z′ m, Z″) calculated by the mask patch part Z′ m in the output Z m of the "Practice" module.

[0042] In an embodiment of the present application, step seven specifically includes: in order to measure the restoration effect of the mosaic image, we send the restored image into a recognition network such as the open source recognition framework provided by Baidu Paddle and the built-in mainstream text recognition model (including CRNN, MASTER, SAR and ABINet) or cloud service API (including domestic Ali Cloud, Baidu Intelligent Cloud and Tencent Cloud and foreign Microsoft cloud service Azure, Amazon cloud service Amazon and Google cloud service) to complete the recognition of multi-language text images of the mosaic and output the predicted character sequence, and use the prediction accuracy and the normalized edit distance as specific measurement evaluation indexes. In the experiment, we noticed that the recognition effects and recognition speeds of different recognizers are different, and in order to better evaluate the increase of the recognition performance of the restored image compared with the mosaic image, we also use the above recognizer or cloud service API to recognize the unprocessed mosaic image and the original high-pixel image respectively, and obtain the upper and lower limits of the indicators associated with the recognizer. Finally, we perform visual result display on the synthetic data of multiple languages and multiple public real scene data sets, and the specific effect is shown in the accompanying drawings Figure 2 .

[0043] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0044] The present application is inspired by the human cognitive process, uses the hierarchical Observation-Practice-Refinement module structure, and achieves significant improvement in recognition performance on both multilingual synthetic data and real scene data sets. Moreover, the present application generates a large amount of training data based on open source tools and adopts an efficient self-supervised learning method, thereby reducing the difficulty of data collection and data labeling. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The network model flowchart of the deep learning-based mosaic text recognition method in the embodiment of the present application is shown in the figure.

[0046] Figure 2 The visualization recovery graph of the present application on synthetic data and real scene data sets is shown in the figure. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application clearer and more understandable, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0048] In order to solve the problems existing in the prior art, the present application proposes a multilingual mosaic image text recognition method based on deep learning. Our goal is to propose a network model that simulates the human cognitive process of "observation-practice-perfection" and explores the security of existing mosaic technology on image information. As shown in Figure 1 , Figure 1 The three modules shown represent "observation-practice-perfection", the "observation" module is composed of an image encoder and a reconstruction module, the "practice" module is composed of a decoder composed of an image encoder with the same shared parameters as the "observation" module and a series of visual self-attention modules, and the "refined" module is composed of a denoising diffusion probability model. The core framework of the mosaic image pair generation, image encoding, mask autoencoder and diffusion model is also shown in the form of a flowchart (i.e. the main part of the mosaic image recovery module of the present application, the present application also includes the calculation of pixel loss, peak signal-to-noise ratio and recovery mean square error mentioned in step six). Image encoder, reconstruction module, self-supervised learning and diffusion model are the core parts of the mosaic text recovery method.

[0049] As shown in Figure 1 , the multilingual mosaic image text recognition method based on deep learning proposed by the present application includes the following steps:

[0050] Step one: data acquisition, using synthetic text image data and public real scene text benchmark data sets as original training data, to form the original image-mosaic image data pair;

[0051] Specifically, the number of pairs of original image-mosaic image data sets synthesized in step one need to be one-to-one. Since there is no public mosaic multilingual data set at present, in order to verify the model generalization ability of the proposed method in real scene, we use the open source tool SynthTiger, which is a synthetic text image generator, to generate multilingual mosaic synthetic training images and use public scene text recognition benchmark data sets, including (1) IIIT5kWords (IIIT); (2) Street View Text (SVT); (3) ICDAR 2013 (IC13); (4) ICDAR 2015 Incidental Text (IC15). The grid size of the generated mosaic image is 5x5, 10x10, and 15x15. In our experiments, we use synthetic data sets of six languages, including English, Chinese, Bengali, Japanese, German, and French data. At the same time, we also use methods such as Gaussian noise, random blur, rotation cropping, and various background fonts to process the generated data to make the synthetic data closer to real scene data and to make the test set and the training set diverse.

[0052] Step two: data preprocessing and data augmentation: first, adjust the input image to a uniform size and resolution; next, perform data augmentation on the adjusted image to improve the model's generalization ability to different input data.

[0053] Specifically, given a color RGB original image-mosaic text picture pair with a height of H, a width of W, and a color channel number of 3 First, get the height and width of the input image, and according to the size relationship between the ratio of the two and 1 / 4, select the most appropriate and maximum degree of original image text shape preservation scaling method to adjust the input image to a uniform height of 32, width of 128 resolution image pair Next, we perform random horizontal or vertical flipping data augmentation on the adjusted image to improve the model's generalization ability to different input data, and finally the enhanced image will be input into the image encoder of the "observation" module. The specific details will be introduced in the next step.

[0054] Step three: send the mosaic image data into the image encoder to extract image and text features; then further upsample the image through the subpixel convolution of the reconstruction module to remove the mosaic and improve the image quality and resolution;

[0055] Specifically, the "Observation" module consists of an image encoder and a reconstruction module, which first sends the pre-processed mosaic image data into the image encoder to hierarchically extract the shallow and deep features of the image using convolutional layers and a moving window-based visual self-attention model, which are respectively the representations of the low-frequency and high-frequency information of the image, and then models and analyzes the image features and text features, obtaining F p = Encoder(X'; Θ enc ), Θ enc represents the parameters of the image encoder, Then, the image is further upsampled by the sub-pixel convolution of the reconstruction module to remove the mosaic and improve the image quality and resolution, and the output is obtained Recons(·) represents the convolutional image reconstruction module with sub-pixel, which has a skip connection between the shallow and deep features, which are respectively the representations of the low-frequency and high-frequency information of the image, Θ recons represents the parameters of the image reconstruction module, Then, according to the output and Y', the L1 pixel loss is calculated as for optimizing the parameters Θ and Θ.

[0056] enc recons

[0057] Here, the image encoder input picture comes from the synthetic data set, and real scene data set and multi-language data set such as (1) IIIT5kWords (IIIT); (2) Street View Text (SVT); (3) ICDAR 2013 (IC13); (4) ICDAR 2015 Incidental Text (IC15) data set can also be used.

[0058] Step four: use a large amount of data for unsupervised learning to imitate the human "practice" process inspired by the masked autoregressive encoder. After sending the masked patch image block into the image encoder with the same shared parameters as the "observation" module, the decoder composed of a series of visual self-attention modules predicts the pixel value of the masked image patch block.

[0059] Specifically, to ensure the continuity of parameter optimization, the "practice" module consists of an image encoder with the same shared parameters as the "observation" module and a decoder composed of a series of visual self-attention modules, inspired by the masked autoregressive encoder, we divide the image into non-overlapping image patch blocks with a size of 2x2 and randomly sample a subset of image patches The sampled subset of visible patches is processed through a linear mapping layer and then input into the image encoder in the specified dimensions, while the remaining image patches... The visible patches are masked and removed, significantly reducing computational and memory requirements, where N and M represent the number of visible and invisible patches, respectively. We utilize a large amount of data for self-supervised learning to mimic the human "practice" process, and then encode the visible patches F. m =Encoder(Linear(Z′); Θ enc )(in Θ dec (For encoder parameters) and shared, learnable mask image patches are stitched together. The input is a decoder consisting of a series of visual self-attention modules to predict the pixel value Z of the mask image patch. m =Decoder(F′) m ;Θ dec Furthermore, the last layer of the decoder is a linear mapping layer such that the output satisfies... Θ dec For decoder parameters, only the output Z is used in the end. m The mask patch part Z′ in m To calculate the mean square error L dec =MSE(Z′) m Z″) is used to optimize parameter Θ dec and Θ enc .

[0060] Step 5: The “Refined” module consists of a denoising diffusion probability model, which mainly consists of two steps: forward denoising and reverse denoising. The forward denoising process can be regarded as a Markov chain model that continuously adds Gaussian noise to the original image until it becomes a pure noise image. The reverse denoising process uses the original image and the noise image to train a U-shaped neural network to predict the noise at each step, thereby achieving the effect of image restoration.

[0061] Specifically, inspired by the denoising diffusion probability model, we adopted a conditional denoising diffusion model that starts with pure Gaussian noise and iteratively optimizes the coarsely restored image to simulate the "refinement" stage in the human cognitive process. Given a coarsely restored image output by the image restoration module... Given the original normal image Y′, the “Refinement” iterative module aims to learn a mapping from a standard normal distribution to an empirical data distribution conditioned on the coarsely reconstructed image. Specifically, it learns the mapping from the standard normal distribution to an empirical data distribution conditioned on the coarsely reconstructed image. To conditionally train a U-shaped neural network (U-Net) model, gradually transforming it from purely noisy images... The diffusion model removes noise of varying degrees until a normal image Y' is obtained. It mainly consists of two steps: forward noise addition and reverse denoising. The forward noise addition process q can be viewed as a Markov chain model that continuously adds Gaussian noise to the original image y0 = Y' until it becomes a purely noisy image. This process can be represented by the following equation:

[0062]

[0063] Where T is the number of steps in the diffusion model, set to 2000, and the noise addition for each iteration of the forward process is performed according to the following formula:

[0064]

[0065] Where α 1：T The hyperparameters, ranging from 0 to 1, determine the noise variables at each step, and I is the standard identity matrix. The forward process, given the original image y0, supports sampling at any time step t, and can be expressed by the following equation:

[0066]

[0067] in This formula is applicable to fast sampling y in step t. t Very helpful, can be expressed by the following formula:

[0068]

[0069] For a given conditional coarse image and noisy target image y t We trained a U-Net as a denoising model, denoted as: Used to predict the noise vector ∈, where To conditionally coarsely recover the image, γ is a statistic of the noise variance. Finally, we minimize the diffusion loss term, i.e.:

[0070]

[0071] Where γ ~ p(γ) and set During training, we first uniformly sample a time step t ~ {0, ..., T}, and then from γ t-1 and γ t Uniform distribution U(γ) t-1 γ t Sampling γ in ). Furthermore, using Bayes' theorem, we can derive the formula for a given (y0, y...) t The posterior distribution y t-1 The derivation is shown in the following formula:

[0072]

[0073]

[0074]

[0075] The reverse denoising process is given by the parameter θ, defined as follows:

[0076]

[0077]

[0078]

[0079] where the reverse process converts the latent variable distribution p θ (y T ) to the data distribution p θ (y0), where x is the coarse recovered image. Combining the above equations and substituting y0 into the posterior distribution q(y t-1 |y0, y t ), the mean of the parameterization p θ (y t-1 |y t , x) is as follows:

[0080]

[0081] Finally in the inference phase, the reverse process is reparameterized to give the following:

[0082]

[0083] where The model ultimately uses the predicted as output.

[0084] Step six: the "observation" module and the "refinement" module use the output of the two to calculate the pixel loss and peak signal-to-noise ratio with the original image; the "practice" module uses the mean square error of the predicted mask image patch pixels to calculate the true pixels to assist and accelerate the training, and the network weight parameters are optimized through back propagation;

[0085] Specifically, we divide the training process of the model into two stages, first train the image encoder shared by the "Observation" and "Practice" at the same time, and then train the iterative optimization module denoising diffusion network. For the two training stages, we use the Adam optimizer and set the corresponding hyperparameters, where the learning rate of the first stage is 5e-5, trained for 10 rounds, and the batch size is set to 32; the learning rate of the second stage is 12e-6, trained for 50 rounds, and the batch size is 8. The cost function is composed of the reconstruction loss function and the pixel loss function, and the formula is: L = L rec +L dec , where L rec is the image reconstruction module of the "Observation" module and the predicted output of the "Refinement" module L dec 1 pixel loss calculated by Y' and Y: L m is the mask patch part Z' in the output Z m of the "Practice" module, and the mean square error L dec = MSE(Z' m , Z'') is calculated.

[0086] Here we specifically use the back propagation algorithm to calculate the gradient of the loss and let it propagate backward in the network to optimize the recognition network parameters.

[0087] Step seven: In the inference stage, the restored image is sent into the recognition network or cloud service API to complete the recognition of the mosaic multilingual text image and output the predicted character sequence, and the prediction accuracy and standardized edit distance are used as the evaluation indicators.

[0088] In the step, in order to measure the mosaic image restoration effect, we send the restored image into the recognition network such as the open source recognition framework provided by Baidu Paddle and the built-in commonly used recognition model (including CRNN, MASTER, SAR and ABINet) or cloud service API (including domestic Ali Cloud, Baidu Intelligent Cloud and Tencent Cloud and foreign Microsoft cloud service Azure, Amazon cloud service Amazon and Google cloud service) to complete the recognition of the mosaic multilingual text image and output the predicted character sequence, and use the prediction accuracy and standardized edit distance as specific measurement evaluation indexes. In the experiment, we noticed that the recognition effects and recognition speeds of different recognizers are different, and in order to better evaluate the increase of the recognition performance of the restored image compared with the mosaic image, we also use the above recognizer or cloud service API to recognize the unprocessed mosaic image and the original high-pixel image respectively, and obtain the upper and lower limits of the index associated with the recognizer. Finally, we perform visual result display on the synthetic data of multiple languages and multiple public real scene data sets, and the specific effect is shown in the accompanying drawings of the specification Figure 2 According to the schematic diagram, it can be known that the text information of multiple languages (Chinese, Japanese, German, French, English and Bengali) can be restored from the mosaic image to a recognizable effect, and the relatively complex Chinese, Japanese and Bengali also achieve good effects; in order to prove the generalization ability of the present application, the visual analysis is also performed on the real scene text benchmark data set, and the complex background noise and the text image of various styles and fonts can be better processed.

[0089] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A deep learning-based multilingual mosaic image text recognition method, characterized in that, The method comprises the following steps: Step one: data acquisition: adopt synthetic text image data and public real scene text benchmark data set as original training data to form original image-mosaic image data pairs; Step two: data preprocessing and data enhancement: first, adjust the input image to a uniform size and resolution; next, perform data enhancement on the adjusted image to improve the generalization ability of the model to different input data; Step three: the "observation" module is composed of an image encoder and a reconstruction module. First, the mosaic image data is sent into the image encoder, and then the image features and text features are extracted hierarchically using convolutional layers and a visual self-attention model based on a moving window; Then, the sub-pixel convolution of the reconstruction module is further used for image upsampling to remove the mosaic and improve the image quality and resolution; Step four: inspired by the masked autoregressive image encoder, a large amount of image data is used for self-supervised learning to simulate the human "practice" process. The "practice" module is composed of an image encoder with the same shared parameters as the "observation" module and a decoder composed of a series of visual self-attention modules to predict the pixel values of the mask image patch block; specifically including: To ensure continuity in parameter optimization, the "Practice" module consists of an image encoder sharing the same parameters as the "Observation" module and a decoder composed of a series of visual self-attention modules, processing the image... The image is divided into non-overlapping image patch blocks of size 2×2, and a subset of image patches is obtained by random sampling. The sampled subset of visible patches is processed through a linear mapping layer and then input into the image encoder in the specified dimension, while the remaining image patches... The mask will be covered and removed, thus greatly reducing computational requirements and storage memory. and These represent the number of visible and invisible patch blocks, respectively. Self-supervised learning is performed using a large amount of data to mimic the human "practice" process. Then, the coded visible patches... And obtained by stitching together shared and learnable mask image patches It inputs a decoder consisting of a series of visual self-attention modules to predict the pixel values ​​of the mask image patch. ,in , The encoder parameters are given, and the last layer of the decoder is a linear mapping layer such that the output satisfies... ; These are the decoder parameters; ultimately, only the output is used. The mask patch part To calculate the mean square error Used to optimize parameters and ; Step five: the "refinement" module is composed of a denoising diffusion probability model, which has two steps of forward noise addition and reverse noise removal: the forward noise addition process is regarded as a Markov chain model that continuously adds Gaussian noise to the original image until it becomes a pure noise image, while the reverse noise removal process uses the original image and the noise image to train a U-shaped neural network to predict the noise at each step to realize image recovery; Step six: the pixel loss and peak signal-to-noise ratio of the recovered image calculated by the "observation" module and the "refinement" module are calculated with the original image; while the "practice" module uses the mean square error of the predicted mask image patch block pixels with the true pixels to assist and speed up the training, and optimizes the recognition network weight parameters through back propagation; Step seven: in the inference stage, the recovered image is sent into the recognition network or cloud service API to complete the recognition of the mosaic multilingual text image and output the predicted character sequence, and the prediction accuracy and standardized edit distance are used as evaluation indicators.

2. The deep learning based multilingual mosaic image text recognition method according to claim 1, characterized in that: The pairs of original image-mosaic image data pairs synthesized in step one need to be one-to-one corresponding, and the mosaic image will be sent into the subsequent network "observation", "practice" and "refinement" three modules to gradually recover the image using visual self-attention model, mask autoencoder and denoising diffusion model.

3. The deep learning based multilingual mosaic image text recognition method according to claim 1 or 2, characterized in that, In the step one: Multilingual mosaic synthetic training images were generated using the open-source tool SynthTiger, a synthetic text image generator, and public scene text recognition benchmark datasets, including (1) IIIT5kWords (IIIT); (2) Street View Text (SVT); (3) ICDAR 2013 (IC13); (4) ICDAR 2015 Incidental Text (IC15). The grid size of the mosaic generated images was 5x5, 10x10, and 15x15. Synthetic datasets were used in six languages, including English, Chinese, Bengali, Japanese, German, and French data. Gaussian noise, random blur, rotation cropping, and various background font processing methods were used to generate the data to make the synthetic data closer to real scene data and to make the test set and training set diverse.

4. The deep learning-based multilingual mosaic image text recognition method according to claim 1 or 2, characterized in that, The step two specifically comprises: Given a color RGB original image-mosaic text picture pair with height H, width W, and 3 color channels , first get the height and width of the input image, and select the most appropriate scaling method that can maximize the preservation of the original image text shape according to the size relationship between the ratio of the two and 1 / 4 to adjust the input image to a uniform height of 32 and a width of 128 resolution image pair ; Next, the adjusted image is randomly horizontally or vertically flipped for data enhancement to improve the model's generalization ability to different input data, and finally the enhanced image will be input into the image encoder of the "observation" module.

5. The deep learning based multilingual mosaic image text recognition method according to claim 1 or 2, characterized in that, The step three specifically comprises: The "Observation" module consists of an image encoder and a reconstruction module. It first processes the pre-processed mosaic image data... The data is fed into an image encoder, where convolutional layers and a moving window-based visual self-attention model are used hierarchically to extract shallow and deep features of the image. These features are then used to model and analyze image and text features, representing low-frequency and high-frequency information in the image, respectively. This process yields... , This represents the parameters of the image encoder. Then, the image is further upsampled through sub-pixel convolution of the reconstruction module to remove mosaic effects and improve image quality and resolution, resulting in the output. , This represents a reconstruction module containing sub-pixel convolutional layers, with skip connections between shallow and deep features, representing low-frequency and high-frequency information in the image, respectively. This represents the parameters of the image reconstruction module. Then based on the output and calculate Pixel loss as Used to optimize parameters and .

6. The deep learning based multilingual mosaic image text recognition method according to claim 1 or 2, characterized in that, The step five specifically comprises: Inspired by the denoising diffusion probabilistic model, a conditional denoising diffusion model is adopted to start from a pure Gaussian noise, based on the coarse recovered image to iteratively optimize to simulate the "perfect refinement" part of the human cognitive process, given the coarse recovered image output by the image recovery module and the original normal image The "perfect refinement" iteration module aims to learn the mapping from the standard normal distribution to the empirical data distribution conditioned on the coarse recovered image.

7. The deep learning based multilingual mosaic image text recognition method according to claim 6, characterized in that, The "refinement" iteration module is specifically implemented as: Restoring an image with coarse-to-fine The U-shaped neural network model is trained as a condition, and gradually removes various degrees of noise from the pure noise image until a normal image is obtained The diffusion model has two steps: forward noise addition and reverse noise removal. The forward noise addition process is regarded as a Markov chain model that continuously adds Gaussian noise to the original image until it becomes a pure noise image, which is represented by the following formula: ， wherein The number of steps for the diffusion model, each iteration of the forward process adds noise according to the following equation: in The hyperparameters, ranging from 0 to 1, determine the noise variable at each step. Given a standard identity matrix, the forward noise addition process in a given original image... Under the condition of supporting any time step Sampling is performed, as expressed by the following formula: wherein , for the case of Fast sampling is very helpful, expressed by the formula: for a given condition coarse image and a noisy target image , a U-Net is trained as a denoising model, denoted as , to predict the noise vector , where is the condition coarse recovered image, is the statistic of noise variance, and finally the diffusion loss term will be minimized, i.e.: wherein and setting , during training, we first uniformly sample a time step , then sample and from uniform distributions over ; Furthermore, using Bayes' theorem, a given ( , posterior distribution of ) The derivation is shown in the following formula: The reverse denoising process is given by the parameter is given by the definition as follows: where the reverse noise addition process converts the latent variable distribution into a data distribution where Solving the above equations for the coarse recovered image and substituting into the posterior distribution parameterized in terms of the parameters the mean of which is given by Finally, in the reasoning stage, the reverse process is re-parameterized to obtain the following formula: wherein . The model ultimately uses the predicted as output.

8. The deep learning based multilingual mosaic image text recognition method according to claim 1 or 2, characterized in that, The step six specifically comprises: The training process of the model is divided into two stages. First, the image encoder shared by "observation" and "practice" is trained simultaneously, and then the iterative optimization module denoising diffusion network is trained. For the two training stages, the Adam optimizer is used and the corresponding hyperparameters are set. The learning rate of the first stage is 5e-5, 10 rounds of training are performed, and the batch size is set to 32. The learning rate of the second stage is 12e-6, 50 rounds of training are performed, and the batch size is 8.

9. The deep learning based multilingual mosaic image text recognition method according to claim 8, characterized in that, In the step six: The cost function is composed of a reconstruction loss function and a pixel loss function, the formula is: wherein is the image reconstruction module of the "Observation" module and the prediction output of the "Refinement" module and the calculated pixel loss: , is the "Practice" module using the output Mask patch part to calculate the mean square error .

Images