An image inpainting method for incomplete text image

Abstract

The application discloses an image restoration method for incomplete text images. The method mainly includes two stages: text structure reconstruction and text image reconstruction. In the text structure reconstruction stage, the incomplete text image is received as input, and through the cooperative work of the encoder, the intermediate module and the decoder, the accurate segmentation and prediction of the text structure are realized, so that the complete text structure binary image is obtained. Subsequently, in the text image reconstruction stage, the reconstructed complete text structure image and the original incomplete text image are jointly used as conditional input. This stage is composed of a recognition guide model and a diffusion model. The recognition guide model, i.e. the scene text recognizer, predicts the probability distribution of each character belonging to a specific character set by analyzing the incomplete text image, providing semantic guidance for text restoration. On the basis of semantic guidance, the diffusion model performs fine repair on the incomplete text image, significantly improving the readability of the text area and the fidelity of the image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and specifically relates to an image restoration method for incomplete text images. BACKGROUND

[0002] With the rapid development of information technology, text images have become increasingly important in various application scenarios, including but not limited to document scanning, electronic books, social media, etc. However, due to shooting conditions, storage medium aging or physical damage, etc., text images often appear incomplete, blurred or damaged, which seriously affects the readability of text information and the subsequent processing efficiency.

[0003] Currently, the restoration technology for incomplete text images mainly focuses on image enhancement and rule-based restoration methods. Image enhancement technology improves image quality by adjusting parameters such as contrast and brightness, but this method has limited effect on severely damaged text images. Rule-based restoration methods rely on pre-set rules to fill in missing text parts, but this method often lacks flexibility and is difficult to adapt to changing text image restoration needs.

[0004] In recent years, deep learning technology has made significant progress in the field of image processing, especially in scene text recognition and image restoration. Deep learning models can automatically learn image features and perform complex pattern recognition, providing new possibilities for the restoration of incomplete text images. However, existing deep learning-based text image restoration methods mostly focus on global features of images, ignoring local details of text structures, resulting in semantic and visual deficiencies in restored text images.

[0005] In addition, another problem of existing technology is the lack of effective semantic guidance in the restoration process, which may cause deviations in semantics between the generated text region and the original text, affecting the accuracy and readability of the restoration results. At the same time, existing technology often has difficulty in maintaining the original style and quality of the image when processing text images with complex backgrounds and diverse fonts. SUMMARY

[0006] To address the problem of incomplete text, the present application proposes an image restoration method for incomplete text images, which effectively solves the shortcomings of existing technology through two stages of text structure reconstruction and text image reconstruction, improving the accuracy, readability and fidelity of text image restoration.

[0007] Therefore, the technical solution adopted by the present application is: an image restoration method for incomplete text images, including text structure reconstruction and text image reconstruction, specifically including the following steps:

[0008] (1) The incomplete text image is used as the initial input, and the complete text structure image is used as the label. The encoder on the unet segmentation network is used to extract features from the input incomplete text image;

[0009] (2) The intermediate module explicitly injects high-level features containing more semantic information and low-level features containing more detailed information through simple Hadamard multiplication operations, thereby enhancing the semantics and details of the features;

[0010] (3) The optimized features are transmitted to the decoder for resolution reconstruction and segmentation to obtain a complete text structure binary image;

[0011] (4) The incomplete text image is input to the text image reconstruction part, and the recognition guide model in the text image reconstruction part is used as a scene text recognizer to extract semantic features from the incomplete text image;

[0012] (5) The complete text structure binary image and the original incomplete text image are used as inputs, and the diffusion model in the text image reconstruction part is guided by the text structure and semantic guidance to perform fine repair on the incomplete text image.

[0013] Further, the complete text structure binary image obtained in step (3) is S RG = R(x LQ ), where x LQ represents the incomplete text image, and R represents the structure reconstruction process.

[0014] Further, for the input incomplete text image x LQ in step (1), the encoder generates four features, denoted as f i , where 1≤i≤4. These features {f1,f2,f3,f4} are transmitted to the intermediate module for further optimization; based on the hierarchical feature maps generated by the encoder, first, a spatial and channel attention mechanism is applied to each feature f i , which is specifically expressed as:

[0015]

[0016] f i ' represents the i-th processed feature map, and represent the i-th spatial and channel attention parameters, respectively, and then a 1×1 convolution is applied to reduce the number of channels of f i ' to c, where c is a hyperparameter. The resulting feature map is defined as where H i , W i , and c represent f iwidth, height and channel number of the input image.

[0017] Further, the Hadamard multiplication operation in step (2) includes, for all feature maps f i applying element-wise Hadamard multiplication to enhance the i-th feature map, expressed as:

[0018]

[0019] where H(·) denotes Hadamard multiplication, and then f i are passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

[0020] Further, a quadratic convolution sn is introduced in the text structure reconstruction, so that the input vector is converted into 2 inner products and 1 sum norm term before being fed into the nonlinear activation function.

[0021] Further, the recognition guidance model in step (4) includes:

[0022] A scene text recognizer for analyzing incomplete text images and predicting the probability distribution of each character belonging to a specific character set; the recognition guidance representation is: where L denotes the maximum prediction length, |D| is the size of the character set, c RG represents the recognition guidance information, represents the attribute of the numerical value.

[0023] An attention mechanism for enhancing the recognition accuracy of the recognizer on key text regions.

[0024] Specifically, in the recognition guidance module, first, c RG is preprocessed and then input as the query (q). At the same time, the input feature f i-1 is obtained by linear transformation and convolution operation. The layer adopts dot product attention to capture the global correlation between pixels, calculates the correlation score between q and k, and weights the sum of values according to the correlation score, and finally obtains the weighted feature representation. Finally, the output convolution layer is used to obtain the final output feature, and the output feature is connected with the input data in residual connection.

[0025] Further, the diffusion model in step (5) includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, Gaussian noise is added according to the time step t for T steps, and given an incomplete text image x LQ denoted as x0, satisfying the data distribution q(x), the forward diffusion process can be expressed as:

[0026]

[0027] where I denotes the identity matrix, N denotes a normal distribution, x t , x t-1 denote data at time steps t and t-1 respectively, β t denotes a hyperparameter;

[0028] By using some reparameterization tricks, x t at an arbitrary time step t is sampled by:

[0029]

[0030] where, denotes a predefined noise parameter at time step t.

[0031] The reverse diffusion process p θ (x t-1 |x t ) is modeled by a neural network θ corresponding to the denoising network, the conditional form of the inverse process is written as:

[0032]

[0033] where, ∈ θ is the noise predicted by the neural network, μ θ is interpreted by the Bayesian rule as the mean, α t = 1-β t .

[0034] The method of the present application further comprises the following steps to optimize the repair result:

[0035] (1) The text recognition loss based on the content of the text area is taken as the text perception loss, and the overall optimization objective of the text structure reconstruction is expressed as:

[0036] L SR (S RG ,S GT ) = L bce (S RG ,S GT ) + L1(S RG ,S GT ) + L cha (S RG ,S GT ) + L sty (S RG ,S GT )

[0037] L SR (S RG ,S GT ), L bce (S RGS GT ), L1(S RG , GT ), L cha (S RG , GT ), L sty (S RG , GT ) respectively represent the total loss of text structure reconstruction, binary cross-entropy loss, L1 loss, text recognition loss and style loss.

[0038] (2) Adjust the parameters of the encoder, the intermediate module, the decoder, the recognizer or the diffusion model according to the evaluation result to improve the repair quality;

[0039] The recognizer is fine-tuned by minimizing the text recognition loss L rec :

[0040] L rec =||Rec(x LQ )-Rec(x HQ )||1

[0041] x HQ represents the complete image after image reconstruction.

[0042] In the training stage of the diffusion model, the whole process is supervised by the MSE loss:

[0043]

[0044] Wherein, L IR represents the image reconstruction loss, represents the expected value of x t sampled from 1 to T at all time steps t, ∈ represents the actual noise, ∈ θ (x t ,x LQ ,t) 2 represents the noise predicted by the model.

[0045] A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the steps of the image repair method for incomplete text image described above.

[0046] The method of the present application is generated and trained, comprising the following steps:

[0047] (1) Synthesize a data set of incomplete text images;

[0048] (2) Train the encoder, the intermediate module and the decoder using the data set;

[0049] (3) Feature extraction and reconstruction of new incomplete text images using the trained network;

[0050] (4) training the diffusion model using the dataset;

[0051] (5) using the trained network to reconstruct the incomplete text image.

[0052] As the damaged areas of text images are usually unpredictable, and the structure of text is composed of fixed strokes with relatively unique layout characteristics, traditional general image inpainting techniques often fail to achieve ideal results when dealing with text images. To address this challenge, the present invention proposes an innovative text structure reconstruction method based on the U-Net segmentation network design. This method, specifically targeting the structural characteristics and limitations of text images, effectively extracts features from damaged text images and reconstructs complete text structures using the deep learning capabilities of the U-Net segmentation network. This reconstruction process not only provides important image prior knowledge for subsequent text image inpainting tasks, but also ensures the semantic accuracy and structural consistency of the generated inpainting content, significantly improving the overall quality and effectiveness of text image inpainting.

[0053] The present invention proposes an advanced text image reconstruction method based on a diffusion model and integrated with an innovative recognition guidance mechanism. This method aims to improve the model's ability to recover unique character-shaped text regions by introducing a recognition guidance model to inject rich semantic information into the diffusion process. Specifically, the diffusion model incorporates incomplete text images, reconstructed complete text structures, and accurate recognition results from a scene text recognizer as its diffusion conditions. Furthermore, the method employs an attention mechanism to skillfully combine semantic-level guidance with pixel-level image information. This combination not only enhances the model's understanding of text content and the accuracy of generated results, but also significantly improves the fidelity of text inpainting, ensuring the visual coherence and semantic consistency of the inpainting results. Under the dual guidance of structural and semantic information, the method of the present invention can accurately generate and inpaint missing regions in text images, including text and its background. Even in the face of highly challenging scenarios, the method can exhibit excellent fidelity and diversity, effectively restoring the completeness and readability of text images, meeting high standards of image quality. The introduction of this technology brings new perspectives and solutions to the field of text image inpainting. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a general framework diagram of the method of the present invention;

[0055] Figure 2 is a structure diagram of the intermediate module;

[0056] Figure 3 is a structure diagram of the secondary neuron;

[0057] Figure 4 To identify the guiding model framework diagram. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with embodiments and drawings, the illustrative embodiments of the present application and the description thereof are only used to explain the present application, and do not limit the present application.

[0059] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one of ordinary skill in the art that the present application can be practiced without these specific details. In other instances, well-known structures have not been described in detail in order to avoid obscuring the present application.

[0060] An image inpainting method for incomplete text images, as shown in Figure 1 The method comprises the following steps:

[0061] Step 1: Text structure reconstruction includes three main modules: encoder, intermediate module and decoder. First, the multi-level feature map is extracted using the encoder. Next, for the i-th feature map, the high-level features containing more semantic information and the low-level features containing more detailed information are explicitly injected by a simple Hadamard multiplication operation, so as to enhance the semantics and details of the i-th feature. Subsequently, the optimized features are transmitted to the decoder for resolution reconstruction and segmentation. The structure reconstruction process can be represented as:

[0062] S RG = R(x LQ ) (1)

[0063] Where x LQ represents an incomplete text image. S RG is the output of text structure reconstruction, i.e. a binary mask image of complete text structure. R represents the structure reconstruction process.

[0064] For the input image The encoder generates 4 features, denoted as f i i-th feature, where 1≤i≤4. These features {f1,f2,f3,f4} are transmitted to the intermediate module for further optimization.

[0065] Based on the hierarchical feature maps generated using the encoder, first, a spatial and channel attention mechanism is applied to each feature f i This process enables the feature to integrate local spatial information and global channel information, which can be expressed as follows:

[0066]

[0067] Here, f i denotes the i-th feature map after processing, and denote the i-th spatial and channel attention parameters, respectively. In addition, the method also applies a 1x1 convolution to reduce the channel number of f i to c, where c is a hyper-parameter. The resulting feature map is defined as where H i , W i and c denote the width, height and channel number of f i , respectively. denotes the attribute of the numerical value.

[0068] Next, the feature map f i is smoothed by a 3x3 convolution and sent to the decoder for rescaling to the same resolution. Element-wise Hadamard multiplication is applied to all resampled feature maps to enhance the i-th feature map with both more semantic information and details, as follows:

[0069] f i ”'=H([f1”,f2”,f3”,f4”]) (3)

[0070] where H(·) denotes the Hadamard multiplication Figure 2 . Then, the enhanced f i ”' is passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

[0071] To further optimize the network, the method introduces a quadratic convolution sn, which converts the input vector into 2 inner products and 1 summation norm term before feeding into the nonlinear activation function, as shown in Figure 3 . After replacing the fully connected / ordinary convolution in the channel and spatial attention mechanism with the quadratic convolution, the method improves the segmentation reconstruction performance to some extent.

[0072] In addition to using the pixel-level binary cross-entropy L bce and L1 loss, the method also references the text recognition loss based on the content of the text region as the text-aware loss, which can more effectively make the image reconstruction focus on the text region rather than the irrelevant background region, thereby generating high-quality images and text features. The objective function of the text structure reconstruction is as follows:

[0073] L SR (S RG ,S GT )=L bce (S RG ,S GT )+L1(SRG ,S GT ) (4)

[0074]

[0075] L1(S RG ,S GT )=||S RG -S GT ||1 (6)

[0076]

[0077] L sty (S RG ,S GT )=||Gram(S RG )-Gram(S GT )||1 (8)

[0078] where N denotes the total number of pixels in the image, and Gram denotes the Gram matrix. L SR (S RG ,S GT ), L bce (S RG ,S GT ), L1(S RG ,S GT ), L cha (S RG ,S GT ), L sty (S RG ,S GT ) denote the total loss of text structure reconstruction, binary cross-entropy loss, L1 loss, text recognition loss and style loss, respectively, denotes a recognition function, S GT denotes an actual result, and S RG denotes a reconstruction result.

[0079] Therefore, the overall optimization objective of text structure reconstruction can be expressed as:

[0080] L SR (S RG ,S GT )=α1L bce (S RG ,S GT )+α2L1(S RG ,S GT )+L cha (S RG ,S GT )+L sty (S RG ,S GT )(9)

[0081] Step 2: Text image reconstruction is based on a U-Net structure diffusion model with attention mechanism, as shown in Figure 1 Text image reconstruction consists of 4 encoder blocks, 1 recognition guide module, and 4 decoder blocks, each block takes the features f i-1 and the time step of the embedding t e as input, where t e is embedded by sinusoidal position encoding. t e denotes the time embedding.

[0082] (1) The diffusion model can be decomposed into two processes: a forward diffusion process and a reverse diffusion process. In the forward process, Gaussian noise is added one by one according to the time step t, lasting T steps. The noise table is a pre-selected hyperparameter that controls the variance of the noise added at each step. Given a missing text image x LQ denoted as x0, satisfying the data distribution q(x), the forward process can be represented as:

[0083]

[0084] I denotes the identity matrix, N denotes the normal distribution, x t , x t-1 denote the data at time steps t and t-1, respectively, and β t denotes the hyperparameter;

[0085] By using some reparameterization tricks, x t at any time step t can be sampled as follows:

[0086]

[0087] where, denotes the predefined noise parameter at time step t.

[0088] Since β t is small enough, the distribution of the inverse step q(x t-1 |x t ) will also be a Gaussian distribution. Therefore, the inverse process p θ (x t-1 |x t ) can be modeled by a neural network θ corresponding to the denoising network. In the super-resolution task, the diffusion model generates the corresponding complete text image by taking x LQ image as a condition, and the conditional form of the inverse process can be written as:

[0089]

[0090]

[0091] where, ∈ θ is the noise predicted by the neural network, μ θ is the mean, and α t = 1 - β t , i.e., the predefined noise parameter at time step t - 1.

[0092] In the training phase, the whole process is supervised by the MSE loss as follows:

[0093]

[0094] L IR denotes the image reconstruction loss, denotes the expectation of x t sampled at all time steps t from 1 to T, ∈ denotes the actual noise, and ∈ θ (x t , x LQ , t) 2 denotes the noise predicted by the model.

[0095] In the inference phase, the final reconstruction result x HQ is obtained by sampling from random Gaussian noise through the reverse process for T steps, where each step can be written as:

[0096]

[0097] where ∈ t ~ N(0, I), ∈ t denotes the noise predicted by the model at time step t.

[0098] (2) The recognition guidance module is a scene text recognizer that takes incomplete text images as input and predicts the probability distribution of each character belonging to which class in the character set. Therefore, the recognition guidance can be represented as: where L denotes the maximum prediction length, |D| is the size of the character set, and c RG denotes the recognition guidance information, denotes the attribute of the data. Considering the importance of recognition guidance in improving the quality of the text region within the incomplete text image, it is particularly important to enhance the recognition accuracy of the recognizer. Therefore, in the training phase, the recognizer is fine-tuned by minimizing the text recognition loss L rec :

[0099] L rec = ||Rec(x LQ ) - Rec(x HQ )||1 (16)

[0100] x HQ represents the reconstructed complete text image.

[0101] As Figure 4 shown, the image features f i-1 enter the decoder after being processed by the recognition guidance module. In the recognition guidance module, the c RG is first pre-processed, and then taken as the input of the query (q). At the same time, the input feature f i-1 is taken as the input of the key (k) and the value (v) after linear transformation and convolution operation. This layer adopts dot-product attention to capture the global correlation between pixels, calculates the correlation score between q and k, and finally obtains the weighted feature representation by weighted sum of the value according to the correlation score. Finally, the output feature is obtained through the output convolution layer, and the output feature is connected with the input data in residual connection.

Claims

1. An image inpainting method for an incomplete text image, characterized in that: The method comprises text structure reconstruction and text image reconstruction, and specifically comprises the following steps: (1) An incomplete text image is taken as an initial input, and a complete text structure image is taken as a label. An encoder on an unet segmentation network is used to extract features of the input incomplete text image; (2) An intermediate module explicitly injects high-level features containing more semantic information and low-level features containing more detailed information through a simple Hadamard multiplication operation; (3) The optimized features are transmitted to a decoder for resolution reconstruction and segmentation to obtain a complete text structure binary image; (4) An incomplete text image is input into a text image reconstruction part. An identification guide model in the text image reconstruction part is taken as a scene text recognizer to extract semantic features of the incomplete text image; (5) The complete text structure binary image and the original incomplete text image are taken as inputs. Through text structure and semantic guidance, a diffusion model in the text image reconstruction part is guided to finely repair the incomplete text image.

2. The image inpainting method for an incomplete text image according to claim 1, wherein: The complete text structure binary image obtained in step (3) is wherein denotes the incomplete text image, and R denotes the reconstruction process.

3. The image inpainting method for an incomplete text image according to claim 1 or 2, characterized in that: In step (1), for the input incomplete text image The encoder generates four features, which are used here. Indicates the first One feature, among which These features The data is then transmitted to an intermediate module for further optimization; based on the hierarchical feature map generated by the encoder, each feature is first processed... The application of spatial and channel attention mechanisms is specifically expressed as follows: Indicates the processed number of... Each feature map and They represent the first Each space and channel attention parameter is then applied, and a 1×1 convolution is then applied to... The number of channels was reduced to ,in It is a hyperparameter, and the resulting feature map is defined as follows: ,in , and They represent Width, height, and number of channels.

4. The image inpainting method for an incomplete text image according to claim 3, wherein: The Hadamard multiplication operation of step (2) includes, for all feature maps Applying element-wise Hadamard multiplication to enhance the first feature map, the expression is: wherein, represents a Hadamard multiplication, then, are passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

5. The image inpainting method for an incomplete text image according to claim 1 or 2 or 4, characterized in that: In the text structure reconstruction, a secondary convolution sn is introduced, so that an input vector is converted into 2 inner products and 1 sum norm item before being fed to a nonlinear excitation function.

6. The image inpainting method for an incomplete text image according to claim 1, wherein: The identification guide model in step (4) comprises: A scene text recognizer for analyzing an incomplete text image and predicting a probability distribution of each character belonging to a particular character set; the recognition guidance is represented as: wherein, represents the maximum prediction length, is the size of the character set, represents the recognition guidance information, represents the properties of the number; An attention mechanism for enhancing the recognition accuracy of the recognizer to key text regions.

7. The image inpainting method for an incomplete text image according to claim 1, wherein: The diffusion model in step (5) includes a forward diffusion process and a reverse diffusion process Gaussian noise is added in sequence, lasting A given incomplete text image is denoted as satisfies the data distribution The forward diffusion process can be represented as: wherein, denotes the identity matrix, denotes a normal distribution, , denote data at time steps t and t-1, respectively, denotes a hyperparameter; By using some reparameterization tricks, we sample the arbitrary time step in the following way: ​ wherein denotes a predefined noise parameter at time step t; Reverse diffusion process By modeling the neural network corresponding to the denoising network The conditional form of the inverse process is written as: wherein , , is the noise predicted by the neural network, is interpreted as the mean by Bayes' rule.

8. The image inpainting method for an incomplete text image according to claim 1, wherein: The following steps are included to optimize the repair result: (1) A text recognition loss based on the content of a text region is taken as a text perception loss. The overall optimization objective of the text structure reconstruction is expressed as: , , , , respectively represent the total loss of text structure reconstruction, binary cross-entropy loss, L1 loss, text recognition loss and style loss. (2) According to the evaluation result, the parameters of the encoder, the intermediate module, the decoder, the recognizer or the diffusion model are adjusted to improve the repair quality; by minimizing the text recognition loss to fine-tune the recognizer: denotes the complete reconstructed image; In the training stage of the diffusion model, the whole process is supervised by an MSE loss: wherein, denotes the image reconstruction loss, denotes the sum over all time steps t from 1 to T denotes the expected value of the sampling, denotes the actual noise, denotes the model predicted noise.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to realize the steps of the image repair method for the incomplete text image according to any one of claims 1 to 8.

Images