Image restoration method for incomplete text image

Abstract

The invention discloses an image restoration method for an incomplete text image. The method mainly comprises two stages: text structure reconstruction and text image reconstruction. In a text structure reconstruction stage, an incomplete text image is received as input, and accurate segmentation and prediction of a text structure are realized through cooperative work of an encoder, an intermediate module and a decoder, so that a binary image of a complete text structure is obtained. Then, in a text image reconstruction stage, a reconstructed complete text structure image and an original incomplete text image are jointly used as condition input. The stage is composed of an identification guide model and a diffusion model. The recognition guide model, namely a scene text recognizer, predicts probability distribution of each character belonging to a specific character set by analyzing an incomplete text image, and provides semantic guidance for text repair. On the basis of semantic guidance, the diffusion model carries out refined restoration on an incomplete text image, so that the readability of a text region and the fidelity of the image are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular is an image restoration method for incomplete text images. Background Art

[0002] With the rapid development of information technology, text images are becoming increasingly important in various application scenarios, including but not limited to document scanning, e-books, social media, etc. However, due to shooting conditions, storage media aging or physical damage, text images are often incomplete, blurred or damaged, which seriously affects the readability of text information and subsequent processing efficiency.

[0003] At present, the restoration technology for incomplete text images mainly focuses on image enhancement and rule-based restoration methods. Image enhancement technology improves image quality by adjusting parameters such as contrast and brightness, but this method has limited effect on severely damaged text images. Rule-based restoration methods rely on preset rules to fill in the missing text parts, but this method often lacks flexibility and is difficult to adapt to the changing needs of text image restoration.

[0004] In recent years, deep learning technology has made significant progress in the field of image processing, especially in scene text recognition and image restoration. Deep learning models can automatically learn image features and perform complex pattern recognition, providing new possibilities for the restoration of incomplete text images. However, most existing deep learning-based text image restoration methods focus on the global features of the image and ignore the local details of the text structure, resulting in semantic and visual deficiencies in the restored text images.

[0005] In addition, another problem with the existing technology is the lack of effective semantic guidance during the restoration process, which may lead to semantic deviations between the generated text area and the original text, affecting the accuracy and readability of the restoration results. At the same time, the existing technology often has difficulty maintaining the original style and quality of the image when processing text images with complex backgrounds and diverse fonts. Summary of the invention

[0006] In response to the problem of incomplete text, the present invention proposes an image restoration method for incomplete text images. This method effectively solves the shortcomings of the existing technology through two stages: text structure reconstruction and text image reconstruction, and improves the accuracy, readability and fidelity of text image restoration.

[0007] In view of this, the technical solution adopted by the present invention is: an image restoration method for incomplete text images, including two parts: text structure reconstruction and text image reconstruction, specifically including the following steps:

[0008] (1) The incomplete text image is used as the initial input, and the complete text structure image is used as the label. The encoder on the UNET segmentation network is used to extract features from the input incomplete text image.

[0009] (2) The intermediate module explicitly injects high-level features containing more semantic information and low-level features containing finer details through a simple Hadamard multiplication operation, thereby enhancing the semantics and details of the features;

[0010] (3) The optimized features are transmitted to the decoder for resolution reconstruction and segmentation to obtain a complete binary image of the text structure;

[0011] (4) The incomplete text image is input to the text image reconstruction part. The recognition guidance model in the text image reconstruction acts as a scene text recognizer to perform semantic extraction on the incomplete text image and further obtain semantic features.

[0012] (5) The complete text structure binary image and the original incomplete text image are taken as input, and the diffusion model in the text image reconstruction is guided by the text structure and semantics to perform fine-tuning repair on the incomplete text image.

[0013] Furthermore, the complete text structure binary image obtained in step (3) is S RG =R(x LQ ), where x LQ represents an incomplete text image, and R represents the structure reconstruction process.

[0014] Furthermore, in step (1), for the input incomplete text image x LQ , the encoder produces 4 features, here we use f i represents the i-th feature, where 1≤i≤4, these features {f 1 ,f 2 ,f 3 ,f 4} is transferred to the intermediate module for further optimization; based on the hierarchical feature map generated by the encoder, each feature f i Apply the spatial and channel attention mechanism, which is specifically expressed as:

[0015]

[0016] f i ' represents the i-th feature map after processing, and denote the i-th spatial and channel attention parameters respectively, and then apply a 1×1 convolution to convert f i 'The number of channels is reduced to c, where c is a hyperparameter, so the resulting feature map is defined as Among them, H i , W i and c represent f i " width, height, and number of channels.

[0017] Furthermore, the Hadamard multiplication operation in step (2) includes performing i "Apply element-wise Hadamard multiplication to enhance the i-th feature map, expressed as:

[0018]

[0019] Where H(·) represents Hadamard multiplication, then, f i ”' is passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

[0020] Furthermore, a quadratic convolution sn is introduced in the text structure reconstruction, so that the input vector is converted into two inner products and one sum norm term before being fed into the nonlinear activation function.

[0021] Furthermore, the identification guidance model in step (4) includes:

[0022] A scene text recognizer that analyzes incomplete text images and predicts the probability distribution of each character belonging to a specific character set; the recognition guidance is expressed as: Where L represents the maximum prediction length, |D| is the size of the character set, and c RG Indicates identification guidance information. Represents the properties of the value.

[0023] An attention mechanism is used to enhance the recognition accuracy of the recognizer for key text regions.

[0024] Specifically, in the identification guidance module, firstly, c RG Perform preprocessing operations and then use it as the input of the query (q). At the same time, the input feature f i-1 The qkv obtained after linear transformation and convolution operation is used as the input of key (k) and value (v). This layer uses dot product attention to capture the global correlation between pixels, calculates the correlation score between q and k, and weighted sums the values ​​according to the correlation score to finally obtain the weighted feature representation. Finally, the final output feature is obtained through the output convolution layer, and the output feature is residually connected with the input data.

[0025] Furthermore, the diffusion model in step (5) includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, Gaussian noise is added sequentially according to the time step t for T steps. Given an incomplete text image x LQ Note: x 0, satisfying the data distribution q(x), the forward diffusion process can be expressed as:

[0026]

[0027] Among them, Ι represents the identity matrix, N represents the normal distribution, x t , x t-1 Represent the data at time step t and t-1, β t represents a hyperparameter;

[0028] By using some reparameterization tricks, we can rewrite x at any time step t in the following way t To sample:

[0029]

[0030] in, represents the predefined noise parameters at time step t.

[0031] Backward diffusion process θ (x t-1 |x t ) is modeled by a neural network θ corresponding to the denoising network, and the conditional form of the inverse process is written as:

[0032]

[0033] in, ∈ θ is the noise predicted by the neural network, μ θ As the mean is interpreted by Bayes' rule, α t =1-β t .

[0034] The method of the present invention further comprises the following steps to optimize the repair result:

[0035] (1) The text recognition loss based on the text region content is used as the text perception loss, and the overall optimization goal of text structure reconstruction is expressed as:

[0036] L SR (S RG ,S GT )=L bce (S RG ,S GT )+L 1 (S RG ,S GT )+L cha (S RG ,S GT )+L sty (S RG ,S GT )

[0037] L SR (S RG ,S GT ), L bce (S RG ,S GT ), L 1 (S RG ,S GT ), L cha (S RG ,S GT ), L sty (S RG ,S GT ) represent the total loss of text structure reconstruction, binary cross entropy loss, L1 loss, text recognition loss and style loss respectively.

[0038] (2) Adjust the parameters of the encoder, intermediate module, decoder, recognizer or diffusion model according to the evaluation results to improve the restoration quality;

[0039] By minimizing the text recognition loss L rec To fine-tune the recognizer:

[0040] L rec =||Rec(x LQ )-Rec(x HQ )|| 1

[0041] x HQ Represents the complete image after image reconstruction.

[0042] During the training phase of the diffusion model, the whole process is supervised by the MSE loss:

[0043]

[0044] Among them, L IR represents the image reconstruction loss, represents x for all time steps t from 1 to T t The expected value of sampling, ∈ represents the actual noise, ∈ θ (x t ,x LQ ,t) 2 represents the noise in the model predictions.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned image repair method for incomplete text images.

[0046] The method of generating and training the present invention comprises the following steps:

[0047] (1) A dataset of synthetic incomplete text images;

[0048] (2) Use the dataset to train the encoder, intermediate modules, and decoder;

[0049] (3) Use the trained network to extract and reconstruct features of new incomplete text images;

[0050] (4) Use the dataset to train the diffusion model;

[0051] (5) Use the trained network to reconstruct the incomplete text image.

[0052] Since the damaged areas of text images are usually difficult to predict, and the structure of the text is composed of fixed strokes and has relatively unique layout characteristics, traditional general image restoration techniques often find it difficult to achieve ideal results when processing text images. To address this challenge, the present invention proposes an innovative text structure reconstruction method based on the U-Net segmentation network design. This method specifically targets the structural characteristics and limitations of text images, and utilizes the deep learning capabilities of the U-Net segmentation network to effectively extract features from damaged text images and reconstruct the complete text structure. This reconstruction process not only provides important image prior knowledge for subsequent text image restoration tasks, but also ensures the semantic accuracy and structural consistency of the generated restoration content, thereby significantly improving the overall quality and effect of text image restoration.

[0053] The present invention proposes an advanced text image reconstruction method based on a diffusion model and integrates an innovative recognition guidance mechanism. This method aims to improve the model's ability to restore text regions with unique character shapes, and injects rich semantic information into the diffusion process by introducing a recognition guidance model. Specifically, the diffusion model integrates incomplete text images, reconstructed complete text structures, and accurate recognition results of scene text recognizers as its diffusion conditions. Furthermore, the present method adopts an attention mechanism to cleverly combine semantic guidance with pixel-level image information. This combination not only enhances the accuracy of the model's understanding and generation of text content, but also significantly improves the fidelity of text restoration, ensuring the visual coherence and semantic consistency of the restoration results. Under the dual guidance of structural and semantic information, the method of the present invention can accurately generate and repair missing areas in text images, including text and its background. Even in the face of highly challenging scenes, the present method can show excellent fidelity and diversity, effectively restore the integrity and readability of text images, and meet high standards of image quality requirements. The introduction of this technology brings new perspectives and solutions to the field of text image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall framework diagram of the method of the present invention;

[0055] Figure 2 It is the structure diagram of the middle module;

[0056] Figure 3 This is a diagram of the secondary neuron structure;

[0057] Figure 4 Framework diagram of the recognition guidance model. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0059] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is apparent to those of ordinary skill in the art that these specific details do not have to be employed to practice the present invention. In other embodiments, in order to avoid confusing the present invention, well-known structures are not specifically described.

[0060] An image restoration method for incomplete text images, such as Figure 1 As shown, the method comprises the following steps:

[0061] Step 1: Text structure reconstruction includes three main modules: encoder, intermediate module and decoder. First, the encoder is used to extract multi-level feature maps. Next, for the i-th feature map, high-level features containing more semantic information and low-level features containing finer details are explicitly injected through a simple Hadamard multiplication operation, thereby enhancing the semantics and details of the i-th feature. Subsequently, the optimized features are transmitted to the decoder for resolution reconstruction and segmentation. Its structure reconstruction process can be expressed as:

[0062] S RG =R(x LQ ) (1)

[0063] where x LQ Represents an incomplete text image. S RG is the output of text structure reconstruction, i.e., the binary mask image of the complete text structure. R represents the structure reconstruction process.

[0064] For the input image The encoder produces 4 features, which are represented by f i represents the i-th feature, where 1≤i≤4. These features {f 1 ,f 2 ,f 3 ,f 4} is transferred to the intermediate module for further optimization.

[0065] Based on the hierarchical feature map generated by the encoder, each feature f i Apply spatial and channel attention mechanisms. This process allows features to integrate local spatial information and global channel information, as shown below:

[0066]

[0067] Here, f i ' represents the i-th feature map after processing, and denote the i-th spatial and channel attention parameters respectively. In addition, this method also applies a 1×1 convolution to convert f i ' is reduced to c, where c is a hyperparameter. The resulting feature map is defined as Among them, H i , W i and c represent f i "The width, height and number of channels, Represents the properties of the value.

[0068] Next, the feature map f i "A 3×3 convolution smoothing optimization is performed and sent to the decoder for rescaling to the same resolution. Element-wise Hadamard multiplication is applied to all resampled feature maps to enhance the i-th feature map so that it has more semantic information and details at the same time. The expression is as follows:

[0069] f i ”'=H([f 1 ”,f 2 ”,f 3 ”,f 4 ”) (3)

[0070] Where H(·) represents the Hadamard multiplication ( Figure 2 ). Then, the enhanced f i ”' is passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

[0071] In order to further optimize the network, the method introduces a secondary convolution sn, so that the input vector is converted into 2 inner products and 1 sum norm term before being fed into the nonlinear activation function, as shown in Figure 3 As shown in Figure 2, this method replaces the fully connected / ordinary convolution in the channel and spatial attention mechanism with secondary convolution, which improves the segmentation and reconstruction performance to a certain extent.

[0072] In addition to using pixel-level binary cross entropy L bce and L1 This method also uses the text recognition loss based on the text area content as the text perception loss, which can more effectively make the image reconstruction focus on the text area rather than the irrelevant background area, thereby generating high-quality image and text features. The objective function of text structure reconstruction is as follows:

[0073] L SR (S RG ,S GT )=L bce (S RG ,S GT )+L 1 (S RG ,S GT ) (4)

[0074]

[0075] L 1 (S RG ,S GT )=||S RG -S GT || 1 (6)

[0076]

[0077] L sty (S RG ,S GT )=||Gram(S RG )-Gram(S GT )|| 1 (8)

[0078] Where N is the total number of pixels in the image and Gram is the Gram matrix. SR (S RG ,S GT ), L bce (S RG ,S GT ), L 1 (S RG ,S GT ), L cha (S RG ,S GT ), L sty (S RG ,S GT ) represent the total loss of text structure reconstruction, binary cross entropy loss, L1 loss, text recognition loss and style loss, respectively. represents the recognition function, S GT Indicates the actual result, S RG Represents the reconstruction result.

[0079] Therefore, the overall optimization goal of text structure reconstruction can be expressed as:

[0080] L SR (S RG ,S GT )=α 1 L bce (S RG ,S GT )+α 2 L 1 (S RG ,S GT )+L cha (S RG ,S GT )+L sty (S RG ,S GT )(9)

[0081] Step 2: Text image reconstruction is based on the U-Net structure diffusion model with attention mechanism, such as Figure 1 As shown in Figure 2, the text image reconstruction consists of 4 encoder blocks, 1 recognition guidance module, and 4 decoder blocks. Each block converts the feature f of the previous block into i-1 and embedding t e The time step is taken as input, where t e Embedded by sinusoidal position encoding. e Represents time embedding.

[0082] (1) The diffusion model can be decomposed into two processes: forward diffusion process and reverse diffusion process. In the forward process, Gaussian noise is added sequentially according to the time step t, and it lasts for T steps. Noise table is a pre-selected hyperparameter that controls the variance of the noise added at each step. Given a defective text image x LQ Note: x 0 , satisfies the data distribution q(x), then the forward process can be expressed as:

[0083]

[0084] Ι represents the identity matrix, Ν represents the normal distribution, x t , x t-1 Represent the data at time step t and t-1, β t represents a hyperparameter;

[0085] By using some reparameterization tricks, we can recalculate x at any time step t in the following way: t To sample:

[0086]

[0087] in, represents the predefined noise parameters at time step t.

[0088] Because β t Small enough, the reverse step q(x t-1 |x t ) will also be Gaussian. Therefore, the inverse step p θ (x t-1 |x t ) can be modeled by a neural network θ corresponding to the denoising network. In the super-resolution task, the diffusion model can be constructed by LQ The image is used as a condition to generate the corresponding complete text image. The conditional form of the inverse process can be written as:

[0089]

[0090]

[0091] in, ∈ θ is the noise predicted by the neural network, μ θ As the mean is interpreted by Bayes' rule, α t =1-β t , That is, the noise parameters predefined at time step t-1.

[0092] During the training phase, the whole process is supervised by the MSE loss as follows:

[0093]

[0094] L IR represents the image reconstruction loss, represents x for all time steps t from 1 to T t The expected value of sampling, ∈ represents the actual noise, ∈ θ (x t ,x LQ ,t) 2 represents the noise in the model predictions.

[0095] In the inference phase, we first sample from random Gaussian noise through the reverse process, and after T steps, we can get the final reconstruction result x HQ , where each step can be written as:

[0096]

[0097] where ∈ t ~N(0,Ι),∈ t represents the noise in the model prediction at time step t.

[0098] (2) The recognition guidance module is a scene text recognizer that accepts an incomplete text image as input and predicts the probability distribution of which class each character belongs to in the character set. Therefore, the recognition guidance can be expressed as: Where L represents the maximum prediction length and |D| is the size of the character set. RG Indicates identification guidance information. Represents the attribute of the data. Considering the importance of recognition guidance in improving the quality of the text area in the defective text image, it is particularly important to enhance the recognition accuracy of the recognizer. Therefore, in the training stage, by minimizing the text recognition loss L rec To fine-tune the recognizer:

[0099] L rec =||Rec(x LQ )-Rec(x HQ )|| 1 (16)

[0100] x HQ Represents the reconstructed complete text image.

[0101] like Figure 4 As shown, the image feature f after downsampling by the encoder i-1 After being processed by the identification and guidance module, it enters the decoder. In the identification and guidance module, c RG Perform preprocessing operations and then use it as the input of the query (q). At the same time, the input feature f i-1 The qkv obtained after linear transformation and convolution operation is used as the input of key (k) and value (v). This layer uses dot product attention to capture the global correlation between pixels, calculates the correlation score between q and k, and weighted sums the values ​​according to the correlation score to finally obtain the weighted feature representation. Finally, the final output feature is obtained through the output convolution layer, and the output feature is residually connected with the input data.

Claims

1. An image restoration method for incomplete text images, characterized by: It includes two parts: text structure reconstruction and text image reconstruction, which specifically include the following steps: (1) The incomplete text image is used as the initial input, and the complete text structure image is used as the label. The encoder on the UNET segmentation network is used to extract features from the input incomplete text image. (2) The intermediate module explicitly injects high-level features containing more semantic information and low-level features containing finer details through a simple Hadamard multiplication operation; (3) The optimized features are transmitted to the decoder for resolution reconstruction and segmentation to obtain a complete text structure binary image; (4) The incomplete text image is input to the text image reconstruction part. The recognition guidance model in the text image reconstruction acts as a scene text recognizer to perform semantic extraction on the incomplete text image and further obtain semantic features. (5) The complete text structure binary image and the original incomplete text image are taken as input, and the diffusion model in the text image reconstruction is guided by the text structure and semantics to perform fine-tuning repair on the incomplete text image.

2. The image restoration method for incomplete text images according to claim 1, characterized in that: The complete text structure binary image obtained in step (3) is S RG =R(x LQ ), where x LQ represents an incomplete text image, and R represents the reconstruction process.

3. The image restoration method for incomplete text images according to claim 1 or 2, characterized in that: In step (1), for the input incomplete text image x LQ , the encoder produces 4 features, here we use f i represents the i-th feature, where 1≤i≤4, these features {f1,f2,f3,f4} are transmitted to the intermediate module for further optimization; based on the hierarchical feature map generated by the encoder, each feature f is first i Apply the spatial and channel attention mechanism, which is specifically expressed as: f i ' represents the i-th feature map after processing, and denote the i-th spatial and channel attention parameters respectively, and then apply a 1×1 convolution to convert f i 'The number of channels is reduced to c, where c is a hyperparameter, so the resulting feature map is defined as Among them, H i , W i and c represent f i " width, height, and number of channels.

4. The image restoration method for incomplete text images according to claim 3, characterized in that: The Hadamard multiplication operation in step (2) includes: i "Apply element-wise Hadamard multiplication to enhance the i-th feature map, expressed as: f i ”'=H([f1”,f2”,f3”,f4”]) Where H(·) represents Hadamard multiplication, then, f i ”' is passed to the corresponding decoder for the next step of resolution reconstruction and segmentation.

5. The image restoration method for incomplete text images according to claim 1, 2, 4 or 5, characterized in that: A quadratic convolution sn is introduced in the text structure reconstruction, so that the input vector is converted into 2 inner products and 1 sum norm term before being fed into the nonlinear activation function.

6. The image restoration method for incomplete text images according to claim 1, characterized in that: The identification guidance model in step (4) includes: A scene text recognizer that analyzes incomplete text images and predicts the probability distribution of each character belonging to a specific character set; the recognition guidance is expressed as: Where L represents the maximum prediction length, |D| is the size of the character set, and c RG Indicates identification guidance information. The attribute representing the value; An attention mechanism is used to enhance the recognition accuracy of the recognizer for key text regions.

7. The image restoration method for incomplete text images according to claim 1, characterized in that: Step (5) The diffusion model includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, Gaussian noise is added sequentially according to the time step t for T steps. Given an incomplete text image x LQ Recorded as x0, satisfying the data distribution q(x), the forward diffusion process can be expressed as: Among them, Ι represents the identity matrix, N represents the normal distribution, x t , x t-1 Represent the data at time step t and t-1, β t represents a hyperparameter; By using some reparameterization tricks, we can rewrite x at any time step t in the following way t To sample: in, represents the predefined noise parameters at time step t; Backward diffusion process θ (x t-1 |x t ) is modeled by a neural network θ corresponding to the denoising network, and the conditional form of the inverse process is written as: in, α t =1-β t ,∈ θ is the noise predicted by the neural network, μ θ as the mean interpreted by Bayes' rule.

8. The image restoration method for incomplete text images according to claim 1, characterized in that: Include the following steps to optimize repair results: (1) The text recognition loss based on the text region content is used as the text perception loss, and the overall optimization goal of text structure reconstruction is expressed as: L SR (S RG ,S GT )=L bce (S RG ,S GT )+L1(S RG ,S GT )+L cha (S RG ,S GT )+L sty (S RG ,S GT ) L SR (S RG ,S GT ), L bce (S RG ,S GT ), L1(S RG ,S GT ), L cha (S RG ,S GT ), L sty (S RG ,S GT ) represent the total loss of text structure reconstruction, binary cross entropy loss, L1 loss, text recognition loss and style loss respectively; (2) Adjust the parameters of the encoder, intermediate module, decoder, recognizer or diffusion model according to the evaluation results to improve the restoration quality; By minimizing the text recognition loss L rec To fine-tune the recognizer: L rec =||Rec(x LQ )-Rec(x HQ )||1 x HQ represents the complete image after reconstruction; During the training phase of the diffusion model, the whole process is supervised by the MSE loss: Among them, L IR represents the image reconstruction loss, represents x for all time steps t from 1 to T t The expected value of sampling, ∈ represents the actual noise, ∈ θ (x t ,x LQ ,t) 2 represents the noise in the model predictions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image restoration method for incomplete text images described in any one of claims 1 to 8 are implemented.

Images