A Chinese character image restoration algorithm based on skeleton extraction and adversarial learning

Abstract

The present invention relates to a Chinese character image restoration method based on skeleton extraction and adversarial learning, comprising: using a skeleton extraction network based on line thinning to process a Chinese character image data set, generating a corresponding text skeleton image data set, and using a random mask image for processing, and then dividing the paired Chinese character images and their text skeleton images into training sets and test sets; constructing a Chinese character image skeleton extraction and restoration network model, and using the created text skeleton image data set in batches to train the network model, so that the network has the ability to extract a complete text skeleton image from a defective Chinese character image; constructing a Chinese character image restoration network model based on a text skeleton, and using the previously extracted complete text skeleton image as prior knowledge to guide the restoration process of the Chinese character image; and using the model saved during the training process to test the images in the previously created test set. The method of the present invention can realize the task of restoring Chinese character images without knowing the mask information in advance, and compared with other restoration algorithms, a Chinese character image restoration result with clearer strokes can be obtained.

Description

Technical Field

[0001] The technologies involved in the present invention include computer vision, computer image processing and deep learning, among which the present invention focuses on the research of Chinese character image restoration algorithm based on frame extraction and adversarial learning. Background Art

[0002] With the development of the information age, the use of various multimedia devices has become widely popular. A large amount of information is stored in computers in the form of images. Images are also one of the most common forms of information in various fields of life and are closely related to people's daily lives. The image information disseminated in social media may contain many objects that are later added to these images, including signatures, watermarks, emoticons, etc. The addition of these objects can change the semantics of the image, so how to remove these objects from the image is a new direction in computer vision research. At the same time, the authenticity of image information is very important. The ability to edit images without leaving any traces poses a hidden danger to public information security. Therefore, the method of restoring a given image to the original image depends largely on the generation mechanism of the given image. Therefore, removing objects from images or image restoration technology is one of the focuses of current research and a hot topic in the field of information security [1].

[0003] In a broad sense, image inpainting technology[2][3] refers to repairing an image or removing an object in an image by filling the defective area or target area with an estimated value, so as to restore the pixel features of the damaged part of the incomplete image. Image inpainting technology plays a key role in many computer vision tasks[4][5]. Existing image inpainting technologies can be divided into two categories: copy-move methods and image inpainting methods based on machine learning. Copy-move methods inpaint images by extracting a part from a certain area of ​​the same or another image and then copying it to the object or defective area to be removed. Due to its simplicity, this technology is widely used in object removal tasks, but it is not suitable for complex situations such as face images or natural scenes. In order to repair damaged old images to eliminate scratches, noise and other problems. The target area can be filled with estimated values ​​by machine learning. This method can also be used to eliminate any type of distortion.

[0004] With the development of deep learning, image restoration based on machine learning can achieve good results, but most of the research is conducted on natural landscape images. These methods often produce random blurred lines in the task of text image restoration. At present, there are relatively few studies specifically targeting character images. Most of the research focuses on text with a small number of characters, such as English. For example, the classification idea can be introduced in languages ​​such as English, which has 52 uppercase and lowercase characters. However, the number of Chinese characters is huge, and such methods cannot be reused. There are very few studies specifically targeting text image restoration with a large number of characters such as Chinese characters. At the same time, traditional restoration algorithms often introduce mask information of the defective area as prior knowledge. These conditions limit the application of the above methods in Chinese character image restoration. In the field of cultural relics protection, digital construction is extremely important. Among them, the digital collection of information such as Chinese calligraphy, ancient books, and inscriptions usually obtains computer image information through scanning, photography, etc. The Chinese character image information not only carries content information, but also has extremely high cultural value and research value. In addition to the digitization of cultural relics, Chinese character images also exist in large quantities in handwriting input, document scanning, and daily photos. The text image information in these objects may be missing due to aging, damage, improper storage, etc. The process of filling in these missing information in the Chinese character image is called Chinese character image restoration. Repairing these Chinese information can show the text content more clearly and is more conducive to the text recognition task. Therefore, the research work on Chinese character image restoration has great significance in cultural relics protection and practical applications. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention provides a Chinese character image restoration algorithm based on skeleton extraction and adversarial learning, which solves the problems of insufficient existing open source data sets and the inapplicability of existing restoration algorithms to Chinese character image restoration.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is as follows:

[0007] A Chinese character image restoration method based on skeleton extraction and adversarial learning uses the character skeleton image as a priori knowledge and combines the idea of ​​text style transfer to fuse the stroke style in the Chinese character image with the structural information in the character skeleton image, and finally achieves the restoration task of defective Chinese character image restoration. Specifically, it includes the following steps:

[0008] (1) Use randomly generated pairs of line images (line images of varying thicknesses and line images of uniform thicknesses) to train a skeleton extraction network based on line thinning to produce a dataset of Chinese character images and their skeleton images. Then use random mask images to process the dataset and produce and divide the dataset for the Chinese character restoration task.

[0009] (2) Using the idea of ​​adversarial learning, we use the Chinese character images and their skeleton image datasets generated in the previous step to train a Chinese character image skeleton extraction and repair network. The input of this network is the defective Chinese character image. After being processed by the network model, a complete character skeleton image is generated. This character skeleton image removes the stroke style features of the Chinese character and only retains the structured Chinese character stroke information, which can be used as a priori knowledge to guide subsequent repair tasks.

[0010] (3) Using the defective Chinese character image and the complete text skeleton image generated in the previous step, a Chinese character image restoration network based on adversarial learning is trained. The network uses two encoders to extract features from the text skeleton image and the defective Chinese character image respectively, and then uses a decoder to fuse the two features, ultimately achieving the task of restoring the Chinese character image.

[0011] The beneficial effect of the present invention is that the method of the present invention can realize the task of Chinese character image restoration without knowing the mask area information in advance. For a writing system such as Chinese with a large number of characters, the restoration result of the algorithm can obtain a restoration result with smoother strokes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a structural diagram of the skeleton extraction network Skeleton-Net based on line refinement proposed in the present invention.

[0013] Figure 2 This is a structural diagram of the Chinese character image skeleton extraction and restoration network SRGAN proposed in the present invention.

[0014] Figure 3 This is a structural diagram of the Chinese character image restoration network SCGAN based on character skeleton proposed in this invention.

[0015] Figure 4 The Chinese character images and their character skeleton images in the data set proposed by the present invention are displayed (the Chinese character images and their character skeleton images are displayed. The first row is the complete Chinese character image, and the second row is the corresponding character skeleton image).

[0016] Figure 5 This is a display diagram of the Chinese character image restoration results of the method proposed in the present invention (a display diagram of the Chinese character image restoration results. The first row is the complete Chinese character image, the second row is the defective Chinese character image, the third row is the text skeleton extraction and restoration result after SRGAN, and the fourth row is the final Chinese character image restoration result of SCGAN). DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0018] The present invention is inspired by image restoration technology based on prior knowledge and text style transfer technology, and uses text skeleton images as prior knowledge to guide the restoration of text images. Text skeleton images can retain the structural information of text images while removing the style of text strokes, and have a guiding role in the restoration process of text images. In view of the above, the present invention proposes a Chinese image restoration algorithm based on skeleton extraction and adversarial learning, which is composed of a text skeleton extraction and restoration network SRGAN based on adversarial learning and a Chinese character image restoration network SCGAN based on text skeleton images. SRGAN uses the idea of ​​skeleton extraction to remove the style of text strokes, extracts and restores the skeleton of the input defective text image, and obtains a complete text skeleton image. SCGAN draws on the research on style transfer related algorithms, mixes the style information from the input defective text image and the structural information in the text skeleton image from SRGAN, and finally obtains a complete text image restoration result. At the same time, the present invention proposes a text skeleton extraction algorithm Skeleton-Net based on line thinning, and uses it to produce Chinese character images and their skeleton data sets, solving the problem of lack of data sets.

[0019] The specific implementation process of the present invention is as follows:

[0020] 1. Train the skeleton extraction network Skeleton-Net based on line refinement

[0021] Because text images and line images have similar line structures, inspired by the research on line vectorization, this paper uses randomly generated lines of varying thickness and corresponding uniform line images as training sets to train a line thinning network through adversarial training. The trained network model is then applied to the text image to generate the corresponding text skeleton image.

[0022] The whole network adopts adversarial training. The generator G adopts the long convolution mode, with a total of 32 layers, including 11 convolution blocks. The first 10 convolution blocks follow the Conv-BatchNorm-ReLU architecture. The edge padding size of the first layer is 4, the convolution kernel size is 9×9, and the step size is 1. This layer is to expand the receptive field of the convolution to retain the global features of the image and enhance the applicability of the network. The following 9 convolution blocks do not reduce the image size or change the number of channels. The feature map size remains unchanged, the number of channels is maintained at 64, the edge padding size of each convolution layer is 1, the convolution kernel size is 3×3, and the step size is 1. The last layer uses the Sigmoid activation function to map the result back to the range of (0, 1). In order to generate a clearer skeleton image, a binary classification discriminator D is used to distinguish the authenticity of the image. The discriminator uses the Conv-BatchNorm-LeakyReLu architecture.

[0023] The loss function of this network model is divided into two parts, including the L2 loss of the generator and the generative adversarial loss of GAN. Let x and x gt are the network input image and target image respectively, and the generator input is the image sequence x = {x1, x2, ...x n}, n is the training batch size, and the generator output is represented by x pred =G(x). The difference between the pixels of the paired images is calculated using the L2 norm, as follows:

[0024]

[0025] Where ||·||2 represents the Euclidean distance. The algorithm uses the idea of ​​generative adversarial network for training, and the adversarial loss function formula is as follows:

[0026]

[0027] The objective loss function formula of the Skeleton-Net network is as follows:

[0028] L G =L2+λ adv L adv

[0029] Among them, λ adv It is the weight parameter of the contribution of this item to the total loss. Here we hope to weaken the role of the discriminator and only play an auxiliary role. In this invention, a smaller value of 0.2 is selected as the discriminant loss weight.

[0030] 2. Preparation and division of Chinese character image and skeleton image datasets

[0031] The skeleton extraction algorithm based on line thinning is used to process the Chinese character image to produce the corresponding text skeleton image. When the original Chinese image in the present invention is collected, the text image of the corresponding style is generated according to the collected font file, and some handwritten Chinese character images from the CASIA-HWDB data set are added to enhance the universality of the data set. In the actual experiment, in order to balance the performance, the established paired Chinese character images and their skeleton data sets are divided, as shown in Table 1.

[0032] Table 1 Total amount and division table of Chinese character image and skeleton dataset

[0033] Training set Validation set Test Set Total quantity Print font 187750 56325 56325 300400 Calligraphy Fonts 37550 18775 18775 75100 CASIA-HWDB 18755 11265 11265 41305 total 244075 86365 86365 416805

[0034] The dataset covers 3755 Chinese characters in the national standard first-level font library. Different fonts are divided into training set, validation set and test set in a ratio of 6:2:2. The training set contains 50 printed fonts, 10 calligraphy and stylized fonts, and 5 handwritten fonts, totaling 244,075 images. Both the validation set and the test set contain 15 printed fonts, 5 calligraphy and stylized fonts, and 3 handwritten fonts, totaling 86,365 images. This article can be used to annotate images in data production and then be used in other tasks.

[0035] 3. Skeleton extraction and skeleton restoration algorithm SRGAN for Chinese character images

[0036] In this part, the present invention proposes a GAN-based text image skeleton extraction and skeleton restoration network SRGAN, which consists of a text skeleton extraction and restoration generator G r and the discriminator D r The network model structure is as follows: Figure 2 As shown. Generator G r It is divided into two parts. The network structure of the first part uses the same long convolution structure as that in Skeleton-Net, which is used to refine the input defective Chinese character image and obtain the defective text skeleton information. The second part adopts the encoder-decoder structure and adds the skip link structure in the U-Net network to generate a clearer text skeleton image.

[0037] G r The encoder part of is called E, which consists of 4 convolutional blocks φ and a total of 24 convolutional layers, which can be expressed as Each convolution block φ has the same structure, which contains two partial convolution blocks. Each partial convolution block follows the Conv-BatchNorm-ELU structure. Only increase the number of channels of the image without reducing the image size, use a convolutional kernel of size 3×3, and a convolutional layer with edge padding size 1, denoted as C_3_1_1, A convolutional layer with a kernel size of 2×2 and a stride of 2, denoted as C_2_0_2, is used to reduce the feature map size and further extract features. Therefore, the four convolutional blocks in the encoder part can be represented as φ l =(C_3_1_1, C_2_0_2). In the encoder part, the ELU activation function is used instead of the ReLU activation function to make the mean of the feature map distribution 0 to speed up the training. At the same time, the activation function is unilaterally saturated for better convergence.

[0038] In the generator G rThe decoder part is symmetrical with the encoder part and consists of 4 deconvolution blocks. Each block is connected to the feature map of the same size of the encoder part and then upsampled. The ReLU activation function is used, and the Sigmoid activation function is used in the last layer to map the image distribution back to (0, 1) to obtain the final output of the complete text skeleton image.

[0039] In the loss function design part, let the input of the network be x = {x1, x2, ...x n}, the target image of the generated network is x gt . Assume that the training batch size is n, and the final output of the generator is represented as x pred =G(x). Because the generator G r It is divided into two parts, which respectively perform the skeleton extraction and skeleton repair tasks of the defective text image. Therefore, it is necessary to set loss functions for the two parts to constrain the functions of the corresponding parts of the network. The output of the first half of the network is an intermediate result expressed as The target image is represented as The L2 norm is used to emphasize the difference between corresponding pixels between the skeleton image and the target image, as follows:

[0040]

[0041] Generator G r The purpose is to extract and repair the complete text skeleton image from the defective text image. Here, L2 loss is also used to calculate the difference between the corresponding pixel positions between the generated text skeleton image and the real target image. The formula is as follows:

[0042]

[0043] In addition, the generator G r A perceptual loss function is also used Guide the network to generate a text skeleton image that is closer to the target image in terms of line continuity and other features. The formula is as follows:

[0044]

[0045] where β l represents the contribution weight of each item to the total loss, φ l represents the trained VGG19 network used to extract the feature image after the first convolution block, and L is the number of convolution blocks. ||·||1 represents the L1 norm. In actual use, the performance of the generative adversarial network will be somewhat different from that in theory. If only the traditional GAN ​​loss function is used, the training of the GAN network will be unstable. Therefore, this paper adds the above additional loss function to stabilize the training. Therefore, the adversarial loss function of the network model is as follows:

[0046]

[0047] After defining the above different loss functions to evaluate the performance of the network, we can get the objective function equation for the overall optimization of the network. The formula is as follows:

[0048]

[0049] Here, different weights λ are used for each item. and adv Set to 0.05, 1.2, 0.5, and 0.1 respectively.

[0050] 4. SCGAN, a Chinese character image restoration algorithm based on character skeleton

[0051] The Chinese character image restoration network module based on the prior knowledge of the character skeleton also uses the adversarial training model SCGAN as a whole. The detailed structure of the network Figure 2 The text skeleton image can be regarded as a standard text structure that completely removes the stroke style information of Chinese characters, while the initial defective Chinese character image can provide certain stroke style feature information. Therefore, the network generator G c Two encoders E1 and E2 are used to extract features from the input defective Chinese character image and the text skeleton image as prior knowledge respectively.

[0052] The function of encoder E2 is to extract features from the text skeleton image. It consists of three convolution blocks φ, with a total of 24 layers, which can be expressed as N represents the number of convolutional blocks in encoder E2. Similar to SRGAN, each convolutional block contains two sub-blocks. This is to increase the depth of the network by separately expanding the number of channels and downsampling the image size. Responsible for expanding the number of channels without reducing the size of the input feature map of the previous layer, using a convolution kernel with a convolution kernel size of 3×3, an edge padding size of 1, and a step size of 1. This convolution layer can represent C_3_1_1, sub-block The convolution layer C_2_0_1 with a convolution kernel size of 2×2 and a stride of 2 is used to downsample the feature map, and the pooling layer in the traditional convolutional neural network is replaced by the convolution and ELU activation functions. Both sub-blocks use the Conv-BatchNorm-ELU structure.

[0053] The role of encoder E1 is to extract features from the input defective text image as the main encoder. It has a similar network structure to encoder E2, but slightly different in the number of layers. At the same time, E2 uses Instance-Normalization (IN) instead of Batch-Normalization (BN) for normalization.

[0054] The loss function design of the SCGAN network is similar to that of the SRGAN network, including L2 loss and perceptual loss. Suppose the input x of the entire network is x = {x1, x2, ...x n}, the target image of the generated network is x gt , by the generator G r The generated text skeleton image is represented as The final output of the entire network is expressed as Generator G c Through two encoders, the text skeleton image is The features of the input defective text image x are extracted, and the features from the two encoders are mixed and upsampled in the decoder to finally complete the task of repairing the defective text image. The L2 norm, i.e., mean square error loss, is used to calculate the difference between the generated Chinese character image and the real target image x. gt The difference between the corresponding pixels is as follows:

[0055]

[0056] In addition, the present invention also gives the generator G c Added a perceptual loss To enhance the ability to repair detail features, the Chinese font image repaired by the network is closer to the target image. The formula is as follows:

[0057]

[0058] where β l The contribution weight of each item to the total loss, φ l represents the VGG19 network that has been trained. DCGAN also adopts adversarial learning as a whole, and the formula for generating adversarial loss is as follows:

[0059]

[0060] The objective function formula of the text image based on the text skeleton in the overall restoration network is as follows:

[0061]

[0062] Here, different weights λ are used for each item. mse , and adv Set to 1.2, 1, and 0.1 respectively.

[0063] The network model in the present invention is built based on the Pytorch framework and the Python language. A Chinese character image restoration algorithm based on skeleton extraction and adversarial learning is provided. It first uses the line-thinning-based skeleton extraction network Skeleton-Net to process the Chinese character image dataset and generate a text skeleton image dataset, which solves the problem of the lack of Chinese character image datasets used for image restoration task research. Then a text image skeleton extraction and restoration network SRGAN is proposed, which realizes the task of removing stroke style features from defective Chinese character images and repairing the defective text skeleton. The text skeleton image results generated by the method model will be used as a priori knowledge to guide subsequent restoration tasks. Finally, a Chinese character image restoration algorithm network SCGAN based on text skeleton is proposed. Inspired by the text style transfer algorithm, two encoding networks are used to extract the structural features provided by the text skeleton image and the text stroke style features provided by the input defective Chinese character image, and then a decoder network is used for feature mixing, and finally the restoration of Chinese character images without providing defective area mask information is realized. Compared with other image restoration algorithms, the present invention is more challenging and more suitable for text image restoration tasks in language systems with a large number of characters such as Chinese characters. It is a relatively advanced Chinese character image restoration method.

[0064] It should be understood that the above embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. For those skilled in the art of the present invention, without departing from the concept of the present invention, several changes or substitutions can be made, which should be regarded as belonging to the protection scope of the present invention.

[0065] References:

[0066] [1]Isogawa M, Mikami D, Iwai D, et al.Mask Optimization for ImageInpainting[J]. IEEE Access, 2018, 6: 69728-69741.

[0067] [2]Criminisi A, P′erez P, Toyama K.Region filling and object removal by exemplarbased image inpainting[J]. IEEE Transactions on image processing, 2004, 13(9): 1200-1212.

[0068] [3]Pathak D,Krahenbuhl P,Donahue J,et al.Context encoders:Featurelearning by inpainting[C].In IEEE Conference on Computer Vision and PatternRecognition,2016:2536-2544.

[0069] [4]Radford A,Metz L,Chintala S.Unsupervised Representation Learningwith Deep Convolutional Generative Adversarial Networks[C].In InternationalConference on Learning Representations,2016.

[0070] [5]Yang C,Lu X,Lin Z,et al.High-resolution image inpainting usingmulti-scale neural patch synthesis[C].In Proceedings of the IEEE conferenceon computer vision and pattern recognition,2017:6721-6729.

Claims

1. A Chinese character image restoration algorithm based on skeleton extraction and adversarial learning, specifically including the following five steps: Step 1: Use randomly generated paired line images to train a line thinning-based skeleton extraction network, and use it to create a dataset of Chinese character images and their text skeleton images. This step can be divided into the following three steps; (i) Create random line image pairs, using straight lines, circles, and rectangles to generate random lines to simulate the horizontal, vertical, and left and right strokes in Chinese characters; the resolution of the input line image is 128×128 pixels, the line width is 1 to 20 pixels, and the lines in the target line image are all 2 pixels; a total of 5,000 line pair datasets are generated for training the skeleton extraction network; (ii) Perform the training operation of the skeleton extraction network, and use the data generated in the previous small step to train an adversarial learning network; the generator G in the network adopts a long convolution mode, with a total of 32 layers, including 11 convolution blocks. The first 10 convolution blocks all follow the Conv-BatchNorm-ReLU architecture. The first layer is padded with four layers of pixels "0" along the edge, that is, the padding size is 4, the convolution kernel size is 9×9, and the step size is 1. This layer is to expand the receptive field of the convolution to retain the global features of the image and enhance the applicability of the network; The 9 convolution blocks do not reduce the image size or change the number of channels. The feature map size remains unchanged, and the number of channels is maintained at 64. Each convolution layer is padded with an extra layer of pixels "0" along the edge, that is, the padding size is 1, the convolution kernel size is 3×3, and the step size is 1; the last layer uses the Sigmoid activation function to map the result back to the range of (0, 1); in order to generate a clearer skeleton image, a binary classification discriminator D is used to distinguish the authenticity of the image. The discriminator uses the Conv-BatchNorm-LeakyReLu architecture; (iii) Use the trained skeleton extraction network to process Chinese character images to create a text skeleton image dataset; Step 2: Using the idea of ​​adversarial learning, we use the Chinese character images and their character skeleton image datasets generated in the previous step to train a Chinese character image skeleton extraction and restoration network SRGAN. The input of the network is the defective Chinese character image. After being processed by the network model, a complete character skeleton image is generated. The whole network is a generative adversarial learning network, the generator G r The encoder part of is called E, which consists of 4 convolutional blocks φ, expressed as Each convolution block φ has the same structure with two partial convolution blocks. Each partial convolution block follows the Conv-BatchNorm-ELU structure; Only increase the number of channels of the image without reducing the image size, use a convolutional kernel of size 3×3, and a convolutional layer with edge padding size 1, denoted as C_3_1_1, A convolutional layer with a kernel size of 2×2 and a stride of 2, denoted as C_2_0_2, is used to reduce the size of the feature map and further extract features; the four convolutional blocks of the encoder part can be expressed as φ1=(C_3_1_1, C_2_0_2); the ELU activation function is used instead of the ReLU activation function in the encoder part; In the generator G r The decoder part is symmetrical with the encoder part, and consists of 4 deconvolution blocks. Each block is connected to the feature map of the same size of the encoder part and then upsampled. The ReLU activation function is used, and the Sigmoid activation function is used in the last layer to map the image distribution back to (0, 1) to obtain the final output complete text skeleton image. In the loss function design part, the input of the network is represented as x = {x1, x2, ...x n }, the target image of the generated network is x gt ; Assume that the training batch size is n, and the final output of the generator is represented as x pred =G(x); because the generator G r The network is divided into two parts, which respectively perform the skeleton extraction and skeleton repair tasks of the defective text image. Therefore, it is necessary to set loss functions for the two parts to constrain the functions of the relative parts of the network. The output of the first half of the network is an intermediate result expressed as The target image is represented as The L2 norm is used to emphasize the difference between corresponding pixels between the mentioned skeleton image and the target image, and the formula is as follows: Generator G r The purpose is to extract and repair the complete text skeleton image from the defective text image. Here, L2 loss is also used to calculate the difference between the corresponding pixel positions between the generated text skeleton image and the real target image. The formula is as follows: In addition, the generator G r A perceptual loss function is also used Guide the network to generate a text skeleton image that is closer to the target image in terms of line continuity features. The formula is as follows: where ||·||1 represents the L1 norm, β l represents the contribution weight of each item to the total loss, φ represents the pre-trained VGG19 network, and φ l It means obtaining the feature image output after cumulative calculation from the 1st convolution block to the lth convolution block in the VGG19 network; because in actual use, the performance of the generative adversarial network GAN will be somewhat different from that in theory. If only the traditional GAN ​​loss function is used, the training of the overall network will be unstable, resulting in the problem of blurred strokes of the generated text skeleton image. Therefore, the above-mentioned perceptual loss function is added. For stable training; for the GAN loss function, given the discriminator D r , the formula of the function is as follows: The objective function equation formula for overall network optimization is as follows: Here, different weights are used for each item and adv Set to 0.05, 1.2, 0.5, 0.1 respectively; Step 3: Use the defective Chinese character image and the complete character skeleton image generated in the previous step to train a Chinese character image restoration network SCGAN based on adversarial learning. The network uses two encoders E1 and E2 to extract features from the character skeleton image and the defective Chinese character image respectively, and then uses a decoder to fuse the two features, and finally achieves the task of restoring the Chinese character image; The encoder E2 consists of 3 convolutional blocks φ, with a total of 24 layers, which can be expressed as N represents the number of convolution blocks in encoder E2; each convolution block contains two sub-blocks. Responsible for expanding the number of channels without reducing the size of the input feature map of the previous layer, using a convolution kernel with a convolution kernel size of 3×3, an edge padding size of 1, and a step size of 1. This convolution layer can represent C_3_1_1, sub-block Use the convolution layer C_2_0_1 with a kernel size of 2×2 and a stride of 2 to downsample the feature map, and use convolution and ELU activation functions to replace the pooling layer in the traditional convolutional neural network; both sub-blocks use the Conv-BatchNorm-ELU structure; Encoder E1 and encoder E2 have similar network structures, but slightly different in the number of layers. At the same time, E2 uses Instance-Normalization (IN) instead of Batch-Normalization (BN) for normalization. The loss function design of the SCGAN network is similar to that of the SRGAN network, including L2 loss and perceptual loss. Suppose the input x of the entire network is x = {x1, x2, ...x n ), the target image of the generated network is x gt , by the generator G r The generated text skeleton image is represented as The final output of the entire network is expressed as The generator Gc uses two encoders to generate the text skeleton image The features of the defective text image x are extracted from the input, and the features from the two encoders are mixed and upsampled in the decoder to finally complete the task of repairing the defective text image; the L2 norm, i.e., mean square error loss, is used to calculate the difference between the generated Chinese character image and the real target image x gt The difference between the corresponding pixels is as follows: In addition, the generator G c Added a perceptual loss To enhance the ability to repair detail features, the Chinese font image repaired by the network is closer to the target image. The formula is as follows: where β l represents the contribution weight of each item to the total loss, φ represents the pre-trained VGG19 network, l is the number of convolutional blocks, and φ l It means obtaining the feature image output after the cumulative calculation of l convolution blocks in the VGG19 network; DCGAN as a whole also adopts the adversarial learning method. Given the discriminator D c , the formula for generating adversarial loss is as follows: The objective function formula of the text image in the repair network based on the text skeleton is as follows: Here, different weights λ are used for each item. mse , and adv Set to 1.2, 1 and 0.1 respectively; Step 4: Use the Nvidia GPU to train the constructed network model in batches using the newly created training set, and continuously optimize the model through the error back propagation algorithm; Step 5: Use the model saved during training to test it on the images in the newly created test set.

Images