A Text Image Restoration Method Based on Generative Adversarial Networks

Abstract

This invention discloses a text image restoration method based on generative adversarial networks (GANs), comprising the following steps: S1: Collecting images containing inscriptions and random occluded images, performing scale normalization to generate k×k pixel images, and establishing an inscription image database, where k is a positive integer; S2: Constructing a generative adversarial network (GAN); S3: Using images from the inscription image database as training samples, training the GAN and optimizing the parameters of the generator and discriminator in the GAN; S4: Replacing the pixel values ​​of the occluded or damaged areas in the text image to be restored with the pixel values ​​of the corresponding areas in the generated text image. This invention effectively simulates the structural features of text images, improves the accuracy and quality of text image restoration results, and effectively restores damaged or contaminated text images, providing a new method and approach for image restoration.

Description

Technical Field

[0001] This invention belongs to the field of cultural relic restoration technology, specifically relating to a text image restoration method based on generative adversarial networks. Background Technology

[0002] Image restoration is a crucial part of image processing. It uses information from the intact parts of an image to fill in occluded, damaged, or redundant areas. Applications include removing occlusions from photographs, restoring damaged artifacts, and image data preprocessing.

[0003] Traditional image inpainting techniques often require specific shapes of the defects and simple, repetitive textures, which limits their application scenarios. With the improvement of computing power and the maturity of algorithm models, deep learning technology has achieved fruitful results in many areas of computer vision. In 2014, Ian Goodfellow, a scholar at the University of Montreal, proposed the concept of "Generative Adversarial Networks," which greatly promoted the development of generative models, and various evolved models have emerged in the past two years.

[0004] Generative Adversarial Networks (GANs) are based on game theory and consist of a generator and a discriminator, trained through adversarial learning. The generative model captures the distribution of sample data, while the discriminator acts as a binary classifier, distinguishing between real and generated samples. By learning from a large amount of similar data, it fits the probability distribution of the data and generates similar data, which aligns with the goal of image inpainting. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a text image restoration method based on generative adversarial networks. Its purpose is to overcome the drawbacks of traditional image restoration techniques, which require the shape of the part to be restored to be specific and the texture to be repetitive and simple. By introducing the advanced concept of generative adversarial networks, it can realize the restoration of ancient text images.

[0006] The technical solution adopted in this invention is:

[0007] A text image restoration method based on generative adversarial networks includes the following steps:

[0008] S1: Collect images containing inscriptions and random masked images, perform scale normalization to generate k×k pixel images, and establish an inscription image database, where k takes a positive integer;

[0009] S2: Construct a generative adversarial network;

[0010] S3: Use images from the inscription image database as training samples to train the generative adversarial network and optimize the parameters of the generator and discriminator in the generative adversarial network;

[0011] S4: Input a random vector that follows a normal distribution into the generator trained in S3 to generate a text image. Compare the intact area of ​​the text image to be repaired with the corresponding area of ​​the generated text image. Continuously adjust the input vector until the two are similar. Finally, replace the pixel values ​​of the occluded or damaged area in the text image to be repaired with the pixel values ​​of the corresponding area in the generated text image.

[0012] Optionally, S1 specifically includes:

[0013] Images of ancient characters were scanned from electronic versions of ancient books. An irregular mask image dataset was used, NVIDIA Irregular Mask Dataset: Testing Set. The two were stitched together to form an image of the damaged text to be repaired. Then, the edge images of the damaged text image were obtained by the Canny edge detection algorithm, and together they formed the input image information for training the network.

[0014] Optionally, the generative adversarial network constructed in S2 includes one generator G and one discriminator D. The output of the generator is connected to the input of the discriminator, and the discrimination result of the discriminator is fed back to the generator.

[0015] The generator part replaces all ordinary convolutions with gated convolutions and adopts a coarse and fine two-level network. The coarse network consists of gated convolutions and dilated gated convolutions, and uses an encoding and decoding structure for upsampling and downsampling to repair a coarse result. The fine network consists of two branches, with a context-based attention mechanism added to one branch.

[0016] The discriminator uses the SN-PatchGAN network to train a free-form image inpainting network.

[0017] Optionally, the loss function of the generative adversarial network constructed in S2 consists of an adversarial loss function, a reconstruction loss function, and a perceptual loss function;

[0018] The adversarial loss function L used in the GAN network part GAN Its generator G is:

[0019]

[0020] Discriminator D is:

[0021]

[0022] Where x represents a real data sample, z represents noise, Pdata(x) and Pz(z) represent the probability distribution of the real data x defined in the original image X in the dataset and the probability distribution of the latent variable z defined in the latent space Z, respectively, D(x) represents the value returned by feeding the data x into the discriminator D, and G(x) represents the value returned by feeding the noise z into the generator G;

[0023] The reconstruction loss function is:

[0024]

[0025] Where y i f(x) represents the ground truth value of the i-th sample. i ) represents the output of this model, i.e., the predicted value of the i-th sample; n is the number of samples;

[0026] The perceptual loss function is:

[0027]

[0028] Where φ represents the loss network, j represents the j-th layer of the network, and C j H j and W j These represent the number of channels, height, and width of the feature map at layer j, respectively.

[0029] The loss function is:

[0030] L=λ1L GAN +λ2L SmoothL1 +λ3L perceptual ;

[0031] λ1, λ2, and λ3 represent the weights of the adversarial loss function, the reconstruction loss function, and the perceptual loss function, respectively.

[0032] Optionally, the loss network φ uses a pre-trained VGG16 network to extract the features output by each convolutional layer in the VGG16 network for both the original and repaired images, thus constructing a perceptual loss.

[0033] Optionally, S3 and S4 specifically include:

[0034] The text image I to be processed is input into the coarse repair network, and the edge detection algorithm is used to obtain the text stroke edge image I-edge and the mask M that marks the position of missing pixels;

[0035] The mask M is concatenated in the channel dimension after the text image I to be processed and the text stroke edge image I-edge, forming 4-channel [I,M] and [I-edge,M], which are then used as input to the coarse network. The repair results output by the encoder and decoder are combined to obtain the coarse repair image I-rough. The coarse repair image I-rough is concatenated with the mask M to form [I-rough,M], which is then fed into the generator of the thinning network. After repair, the repaired image I-inpaint=G([I-rough,M]) is obtained.

[0036] The input to the discriminator D is the repaired image I-inpaint and the corresponding real complete image I-gt. The discriminator D scores and distinguishes between the repaired image I-inpaint and the corresponding real complete image I-gt.

[0037] Optionally, the size of the text image I to be processed is 512×512, the text stroke edge image I-edge is a binary image that corresponds one-to-one with the text image I to be processed, the part of the pixel value of 1 on the text stroke edge image I-edge corresponds to the edge part of the character shape in the text image I to be processed, the mask M is a binary single-channel mask with the same width and height as the text image I to be processed, and the value of each pixel on the mask M is 0 or 1, which correspond to the complete pixel and the missing pixel on the image, respectively.

[0038] The advantages of this invention are:

[0039] This invention introduces a generative adversarial network (GAN) method to better simulate the structural features of text images, thereby improving the accuracy and quality of text image restoration results. It effectively restores damaged or contaminated text images, providing a new method and approach for image restoration. It has significant application value in fields such as image restoration algorithms, archaeological excavation, and digital preservation of cultural relics. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 This is a flowchart of the text image restoration method based on generative adversarial networks of the present invention;

[0042] Figure 2 This is a structural diagram of the generator and discriminator of the present invention;

[0043] Figure 3 This is a screenshot of the text before repair in an application example;

[0044] Figure 4 This is the repaired text image in the application example;

[0045] Figure 5 This invention demonstrates the repair effect of the present invention on text with different degrees of damage;

[0046] Figure 6 This paper compares the restoration effects of the present invention with those of other mainstream natural image restoration algorithms. Detailed Implementation

[0047] A novel inscription dataset is used instead of the standard natural image dataset for network training. A coarse-fine two-stage generative adversarial network is constructed, incorporating image edge information as a priori to improve restoration quality. Perceptual loss is also added to enhance the detail information of the output features.

[0048] The present invention provides a text image restoration method based on generative adversarial networks. Its purpose is to overcome the shortcomings of traditional image restoration techniques, which require the shape of the area to be restored to be specific and the texture to be repetitive and simple. By introducing the advanced concept of generative adversarial networks, it achieves the restoration of ancient text images. The method includes the following steps:

[0049] (1) Collect images containing ancient inscriptions such as oracle bone inscriptions and bronze inscriptions, and randomly cover images, perform scale normalization to generate k×k pixel images, thereby establishing an inscription image database, where k takes a positive integer;

[0050] (2) Construct a generative adversarial network;

[0051] (3) Using images from the inscription image database as training samples, the generative adversarial network is trained to optimize the parameters of the generator and discriminator in the generative adversarial network.

[0052] (4) Input a random vector that follows a normal distribution into the generator trained in step (3) to generate a text image. Compare the intact area of ​​the text image to be repaired with the corresponding area of ​​the generated image. Continuously adjust the input vector until the two are similar. Finally, replace the pixel values ​​of the occluded or damaged area in the text image to be repaired with the pixel values ​​of the corresponding area of ​​the generated text image.

[0053] Step (1) is implemented as follows: Images of ancient characters are scanned and extracted from electronic versions of ancient books. The irregular mask image dataset used is the NVIDIA Irregular Mask Dataset: Testing Set, which is the most widely used mask data in the field of image restoration. The two are stitched together to form the image of the damaged text to be restored. Then, the edge image of the damaged text image is obtained by the Canny edge detection algorithm, and together they form the input image information for training the network.

[0054] Combination Figure 2The generative adversarial network constructed in step (2) includes a generator G and a discriminator D. The output of the generator is connected to the input of the discriminator, and the discriminator's judgment result is fed back to the generator. The generator part replaces all ordinary convolutions with gated convolutions and adopts a coarse and fine two-level network. The coarse network consists of gated convolutions and dilated gated convolutions, and uses an encoding and decoding structure for upsampling and downsampling to repair a coarse result. The fine network consists of two branches, one of which incorporates a context-based attention mechanism. The fine network with a context-based attention mechanism particularly improves the clarity of texture details, enabling more accurate repair. The discriminator part uses an SN-PatchGAN network to train a free-form image inpainting network.

[0055] The loss function of the generative adversarial network constructed in step (2) consists of three parts: adversarial loss function, reconstruction loss function, and perceptual loss function.

[0056] The adversarial loss function L used in the GAN network part GAN Its generator G is:

[0057]

[0058] Discriminator D is:

[0059]

[0060] Where x represents a real data sample, z represents noise, Pdata(x) and Pz(z) represent the probability distribution of the real data x defined in the data space X (i.e. the original image in the dataset) and the probability distribution of the latent variable z defined in the latent space Z (i.e., the random noise distribution), respectively, D(x) represents the value returned by feeding the data x into the discriminator D, and G(x) represents the value returned by feeding the noise z into the generator G.

[0061] Adversarial loss guides the generator to produce plausible restoration results to deceive the discriminator, while reconstruction loss guides the generator to produce restoration results that are closer to the real image. This paper adopts SmoothL1Loss, which combines some advantages of L2 Loss and L1 Loss. When the difference between the predicted value and the ground truth is small (absolute difference less than 1), the gradient is not too large (the loss function is smoother than L1 Loss); when the difference is large, the gradient value is small enough (more stable, less prone to gradient explosion).

[0062]

[0063] Where y i Let f(x) be the ground truth value of the i-th sample. i) represents the output of this model (i.e., the predicted value of the i-th sample), and n is the number of samples.

[0064] In style transfer research, perceptual loss is most commonly used and is highly suitable for measuring the effectiveness of style-consistent text image restoration because text images generally have high similarity, with differences between different characters manifested in stroke details; in other words, the styles of text images are relatively consistent. Perceptual loss uses a pre-trained network (typically VGG) to extract features from the outputs of different layers and calculates the feature loss between corresponding layers.

[0065]

[0066] Where φ is the loss network (in this invention, a pre-trained VGG16 network is used), j represents the j-th layer of the network, and Cj, Hj, and Wj represent the number of channels and width and height of the feature_map in the j-th layer.

[0067] This invention employs a pre-trained VGG16 network. Features from the outputs of each convolutional layer in the VGG16 network are extracted from both the original and repaired images to construct a perceptual loss.

[0068] Finally, the complete loss function of this algorithm consists of a weighted sum of adversarial loss, reconstruction loss, and perceptual loss:

[0069] L=λ1L GAN +λ2L SmoothL1 +λ3L perceptual ;

[0070] The specific implementation process of steps (3) and (4) is as follows: First, the text image I to be processed is input into the coarse repair network. The text stroke edge image I-edge and the mask M that marks the position of missing pixels are obtained through the edge detection algorithm. Since the number, position, shape and other factors of irregular holes are highly random, if only the text image I to be processed is input, the generator can easily confuse the holes, background noise and the actual parts to be repaired inside the text itself. Therefore, the text stroke edge image I-edge and the mask M are used as auxiliary condition information and input into the network. The size of the text image I to be processed is 512×512. The text stroke edge image I-edge is a binary image that corresponds one-to-one with the text image I to be processed. The part with a pixel value of 1 on the text stroke edge image I-edge corresponds to the edge of the character in the text image I to be processed. In the edge part, the mask M is a binary single-channel mask with the same width and height as the text image I to be processed. The value of each pixel on the mask M is 0 or 1, corresponding to a complete pixel and a missing pixel in the image, respectively. The mask M is concatenated after I and the text stroke edge image I-edge in the channel dimension to form 4-channel [I,M] and [I-edge,M], which are then used as input to the coarse network. The repair results output by the encoder and decoder are combined to obtain I-rough. I-rough is concatenated with M to form [I-rough,M], which is then fed into the generator of the thinning network. After repair, the image I-inpaint is obtained.

[0071] G([I-rough,M]). The input to the discriminator D is the inpainted image I-inpaint and the corresponding real complete image I-gt. The discriminator D scores both separately and attempts to distinguish them.

[0072] The text image to be processed, also known as the text image with irregularly shaped holes, is shown in the attached image. Figure 3 The text image before restoration. Figure 5 The middle input line refers to a text image with irregularly shaped holes. The true complete image (I-gt) is the original, complete image of the text image.

[0073] Example 1:

[0074] See Figure 1 The process of the image restoration method of the present invention is as follows:

[0075] Step (1) involves scanning and extracting a large number of images of ancient inscriptions from electronic versions of ancient books to train a generative adversarial network. The images are scaled to 512×512 pixels, named sequentially with numbers, and saved in the same folder to build a text image database containing 15,000 images.

[0076] Step (2) Construct a generative adversarial network model, including a generator G and a discriminator D. For example... Figure 2 As shown. The generator's output is connected to the discriminator's input, and the discriminator's judgment result is then fed back to the generator.

[0077] Step (3) Training the generative adversarial network uses a mini-batch method, dividing the images in the face image database into 4000 mini-batches, each containing 2 images. The training process for each mini-batch includes the following steps:

[0078] 3a) First fix the generator G and optimize the discriminator D to maximize the discrimination accuracy of D;

[0079] 3b) Fix the discriminator D and optimize the generator G so that the generated pseudo data G(z) behaves on D (G(z)) and the real data x behaves on D (x) in the same way, thus making it impossible to correctly determine the source of the data;

[0080] 3c) Repeat steps 3a and 3b, and repeatedly train the discriminator and generator alternately to continuously improve the performance of D and G. When the discriminative ability of D is improved to a certain extent and it can no longer correctly identify the source of the data, it can be considered that the generator G has learned the distribution of the real data, thus obtaining the final generative adversarial network parameter model.

[0081] Figure 3 This is a partial sample of the text dataset before repair. Figure 4 After being repaired by the present invention Figure 3 Corresponding text image examples.

[0082] Combination Figure 5 To investigate the repair effect of this invention on different damaged areas, test images were divided into five categories based on the proportion of the damaged area in the core character shape (rather than the proportion of the damaged area in the entire image): 0-15%, 15-30%, 30-45%, 45-60%, and over 60%. The results showed that when the character structure was basically intact, the proposed method could correctly repair the damaged area and restore the character shape to its recognizable form. Only when the damaged area increased to the point that the main structure of the character was severely damaged did the repair effect of this algorithm become less than ideal.

[0083] Table 1

[0084]

[0085] Combination Figure 6Table 1 compares the proposed method with current mainstream image inpainting methods such as CA (context attention), MED (mutual encoder-decoder), EC (edge ​​connect), and GC (gated convolution). Experimental results demonstrate that the proposed method achieves better inpainting results, outperforming existing methods on the most commonly used evaluation metrics in the field of image inpainting, SSIM and PSNR.

[0086] This invention discloses an improved image restoration method based on GAN (Generative Adversarial Network). The implementation steps are as follows: (1) collect images of ancient characters to establish a dataset; (2) construct a generative adversarial network; (3) train the generative adversarial network; and (4) use the trained generator to restore damaged text images. This invention, by introducing a generative adversarial network, better simulates the structural features of text images, improving the accuracy and quality of text image restoration results. It effectively restores damaged or contaminated text images, providing a new method and approach for image restoration. It has significant application value in image restoration algorithms, archaeological excavation, and digital preservation of cultural relics.

[0087] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0088] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0089] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A text image restoration method based on generative adversarial networks, characterized in that, Includes the following steps: S1: Collect images containing inscriptions and random masked images, perform scale normalization to generate k×k pixel images, and establish an inscription image database, where k takes a positive integer; S2: Construct a generative adversarial network; the generative adversarial network constructed in S2 includes one generator G and one discriminator D. The output of the generator is connected to the input of the discriminator, and the discrimination result of the discriminator is fed back to the generator; the generator part replaces all ordinary convolutions with gated convolutions and adopts a coarse and fine two-level network; the coarse network consists of gated convolutions and dilated gated convolutions, and uses an encoding and decoding structure for upsampling and downsampling to repair a coarse result; the fine network consists of two branches, and a context-based attention mechanism is added to one branch; the discriminator uses an SN-PatchGAN network to train a free-form image inpainting network; S3: Using images from the inscription image database as training samples, train the generative adversarial network and optimize the parameters of the generator and discriminator in the generative adversarial network; S4: Input a random vector that follows a normal distribution into the generator trained in S3 to generate a text image. Compare the intact area of ​​the text image to be repaired with the corresponding area of ​​the generated text image. Continuously adjust the input vector until the two are similar. Finally, replace the pixel values ​​of the occluded or damaged area in the text image to be repaired with the pixel values ​​of the corresponding area in the generated text image. Specifically, S3 and S4 include: The text image I to be processed is input into the coarse repair network, and the edge detection algorithm is used to obtain the text stroke edge image I-edge and the mask M that marks the position of missing pixels; The mask M is concatenated in the channel dimension after the text image I to be processed and the text stroke edge image I-edge, forming 4-channel [I,M] and [I-edge,M], which are then used as input to the coarse network. The restoration results from the encoder and decoder are combined to obtain the coarse restoration image I-rough. The coarse restoration image I-rough is then concatenated with the mask M to form... [I-rough,M] is fed into the generator of the thinning network, and the repaired image I-inpaint=G([I-rough,M]) is obtained after repair. The input to the discriminator D is the repaired image I-inpaint and the corresponding real complete image I-gt. The discriminator D scores and distinguishes between the repaired image I-inpaint and the corresponding real complete image I-gt.

2. The text image restoration method based on generative adversarial networks according to claim 1, characterized in that, S1 specifically includes: Images of ancient characters were scanned from electronic versions of ancient books. An irregular masking image dataset was used, NVIDIA Irregular Mask Dataset: Testing Set. The two were stitched together to form an image of the damaged text to be repaired. Then, the edge images of the damaged text image were obtained by the Canny edge detection algorithm, and together they formed the input image information for training the network.

3. The text image restoration method based on generative adversarial networks according to claim 1 or 2, characterized in that, The loss function of the generative adversarial network constructed in S2 consists of an adversarial loss function, a reconstruction loss function, and a perceptual loss function. The adversarial loss function L used in the GAN network part GAN Its generator G is: Discriminator D is: Where x represents a real data sample, z represents noise, Pdata(x) and Pz(z) represent the probability distribution of the real data x defined in the original image X in the dataset and the probability distribution of the latent variable z defined in the latent space Z, respectively, D(x) represents the value returned by feeding the data x into the discriminator D, and G(x) represents the value returned by feeding the noise z into the generator G; The reconstruction loss function is: Where y i f(x) represents the ground truth value of the i-th sample. i ) represents the output of this model, i.e., the predicted value of the i-th sample; n is the number of samples; The perceptual loss function is: Where φ represents the loss network, j represents the j-th layer of the network, and C j H j and W j These represent the number of channels, height, and width of the feature map at layer j, respectively. The loss function is: L=λ1LGAN+λ2LSm oothL1 +λ3L perceptual ; λ1, λ2, and λ3 represent the weights of the adversarial loss function, the reconstruction loss function, and the perceptual loss function, respectively.

4. The text image restoration method based on generative adversarial networks according to claim 3, characterized in that, The loss network φ uses a pre-trained VGG16 network to extract the features output by each convolutional layer in the original image and the restored image to construct a perceptual loss.

5. The text image restoration method based on generative adversarial networks according to claim 1 or 2, characterized in that, The size of the text image I to be processed is 512×512. The text stroke edge image I-edge is a binary image that corresponds one-to-one with the text image I to be processed. The part of the pixel value of I-edge is 1, which corresponds to the edge part of the character shape in the text image I to be processed. The mask M is a binary single-channel mask with the same width and height as the text image I to be processed. The value of each pixel on the mask M is 0 or 1, which corresponds to the complete pixel and the missing pixel on the image, respectively.

Images