Calligraphy Chinese character image restoration method based on double-flow generative adversarial network

Abstract

The invention discloses a calligraphy Chinese character image restoration method based on a double-flow generative adversarial network, and the method comprises the steps: collecting a plurality of calligraphy single character images of different styles, carrying out the preprocessing, generating a data set, and dividing the data set into a training set and a test set according to a proportion; constructing an image restoration model; taking the training set and the test set as input of an image restoration model, and designing a loss function to train and test the image restoration model; and inputting the to-be-restored Chinese character image into the trained image restoration model to obtain a calligraphy Chinese character image restoration result. The calligraphy Chinese character image restoration method based on the double-flow generative adversarial network has the characteristics of high character image restoration efficiency and high accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and restoration, and specifically relates to a calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network. Background Art

[0002] Chinese characters, a vital carrier of Chinese culture, boast a rich history spanning over 5,000 years. Throughout history, they have not only been used to record and pass on cultural heritage, but have also gradually evolved into the uniquely artistic form of calligraphy. Calligraphy, a treasure of Chinese culture, carries a rich tapestry of historical and cultural significance with its unique structural and formal beauty. The nation is currently digitizing its collections, photographing and preserving ancient books and other artifacts. However, many ancient works contain incomplete characters due to natural or human damage. To better preserve these artifacts and further promote traditional Chinese culture, the restoration of these damaged characters is both necessary and valuable.

[0003] Currently, the restoration of Chinese characters on damaged inscriptions, such as those on steles, is performed manually by restoration experts. However, due to the large number of inscriptions on ancient books and the time-consuming and labor-intensive nature of manual restoration, it is difficult to complete the restoration of a large number of ancient books in a short period of time, resulting in low efficiency. With the rapid development of artificial intelligence and computer vision technologies, deep learning is being used to assist in this task. Generative Adversarial Networks (GANs) consist of a generator and a discriminator, trained through adversarial learning. By having two neural networks (generator and discriminator) compete against each other, they learn the underlying distribution of the data and generate realistic samples, which aligns with the goal of image restoration. Summary of the Invention

[0004] The purpose of the present invention is to provide a calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network, which has the characteristics of high efficiency and high accuracy of text image restoration.

[0005] The technical solution adopted by the present invention is a calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network, which specifically includes the following steps: Step 1: Collect a number of calligraphy characters of different styles, generate a dataset after preprocessing, and divide it into training set and test set according to the ratio; Step 2: Build an image restoration model; Step 3: Use the training set and test set as the input of the image restoration model, and design a loss function to train and test the image restoration model; Step 4: Input the Chinese character image to be repaired into the trained image repair model to obtain the calligraphy Chinese character image repair result.

[0006] The present invention is also characterized in that: Step 1 is as follows: Several calligraphy character images of different styles are collected, and several masks with different covering degrees are fused with the calligraphy character images to generate damaged Chinese character images. The calligraphy character images and the damaged Chinese character images are paired to form a dataset, and then divided into training set and test set according to the proportion.

[0007] The image restoration model constructed in step 2, including the generator and the discriminator; The generator adopts a two-stream U-net variant architecture, including parallel texture branches and structure branches, which respectively extract texture features and structural features of the image. It also includes a bidirectional feature fusion module for fusing texture features and structural features to output fused features. It also includes a multi-scale dilated convolution fusion module for extracting multi-scale features from the fused features, performing channel splicing and weighted fusion on the multi-scale features, and outputting dilated fused features. The feature fusion features undergo a 1×1 convolution with a stride of 1 to output the restored image. The discriminator adopts a two-stream network structure, including parallel texture discrimination branches and structure discrimination branches, which are used to analyze the texture features and structural features of the restored image respectively to distinguish between real images and generated images.

[0008] The generator adopts a two-stream U-net variant architecture, including parallel texture branch and structure branch; The texture branch includes an encoder and a decoder. The encoder contains 7 layers of convolution. The first layer uses a 7×7 convolution layer with a stride of 2. The second and third layers use 5×5 convolution layers with a stride of 2. The fourth to seventh layers use 3×3 convolution layers with a stride of 2. The decoder contains 7 layers of transposed convolution layers with a convolution kernel of 3×3 and a stride of 1. The network structure of the structure branch is exactly the same as that of the texture branch; The 3-channel damaged Chinese character image and mask are input into the texture branch. After feature extraction and downsampling by the encoder of the texture branch, they enter the decoder of the structure branch and upsample through layer-by-layer transposed convolution. The decoder is connected with the features extracted by the encoder of the structure branch. The decoder of the structure branch outputs the structural feature map. ; The edge map, grayscale map and mask corresponding to the damaged Chinese character image are input into the structure branch. After feature extraction and downsampling by the encoder of the structure branch, they enter the decoder of the texture branch and upsample through transposed convolution. They are jump-connected with the features extracted from the corresponding layer of the encoder of the texture branch, and the decoder of the texture branch outputs the texture feature map. .

[0009] Bidirectional feature fusion module, for the input texture feature map and structural feature diagram , the processing process is as follows: For texture feature maps , first with the structural feature map Perform channel-level splicing, and then obtain structural gating after linear mapping and Sigmoid function activation , the formula is as follows:

[0010] Where, is a channel-level splicing operation; g is a linear mapping function, which is implemented by a convolution layer with a convolution kernel size of 3; is the Sigmoid activation function; Gating the structure and structural feature diagram Multiply them together to get the structural features retained after gated screening, and then combine the retained structural features with the texture feature map Add together to get the texture features optimized by structural features , which is expressed as follows:

[0011] Among them, is the training parameter initialized to 0; For the structural feature map , first with the texture feature map Perform channel-level splicing, and then obtain texture gating after linear mapping and Sigmoid function activation , the formula is as follows:

[0012] Gating the texture Texture feature map Multiply them to get the texture features retained after gated filtering, and then combine the retained texture features with the structural feature map Add together to get the structural features optimized by texture features , which is expressed as follows:

[0013] Finally, the texture features obtained after structural feature optimization are and structural features optimized by texture features Perform channel-level splicing to obtain fusion features , which is expressed as follows: .

[0014] Multi-scale dilated convolution fusion module, for the fusion features of the input , the processing process is as follows: First, after a 3×3 convolution operation, the input fusion features The number of channels is reduced to 64, and the fused features after dimensionality reduction are input into three dilated convolutional layers with different expansion rates to extract multi-scale features, which are expressed as follows:

[0015] Where, represents a 3×3 dilated convolution, represents the expansion rate of the i-th branch, BN represents batch normalization, and ReLU represents the activation function; Then multi-scale features and Perform channel-level addition, then enter a 3×3 convolution layer for feature fusion, and perform weighted adjustment through learnable weight coefficients to obtain weighted features , which is expressed as follows:

[0016] Where, is the weight coefficient; The obtained weighted features With multi-scale features The channel level is added, and then enters a 3×3 convolution layer for feature fusion, and weighted adjustment is performed through the learnable weight coefficient to obtain the weighted feature , which is expressed as follows:

[0017] Where, is the weight coefficient; The weighted features 、 Multi-scale features obtained with dilated convolution After channel splicing, it enters a 1×1 convolution for channel recovery and is then added to the original input to obtain the feature output of the multi-scale dilated convolution fusion module, which is expressed as follows: .

[0018] The discriminator adopts a two-stream network structure, including parallel texture discrimination branches and structure discrimination branches; The texture discrimination branch takes as input the restored image output by the generator. The texture discrimination branch consists of five convolution layers. The first to third layers use 4×4 convolution layers with a stride of 2 for downsampling, while the fourth and fifth layers use 4×4 convolution layers with a stride of 1 to maintain the resolution. The discrimination result is finally output through the Sigmoid function. The input of the structure discrimination branch is the edge map of the repaired image output by the generator and the grayscale image of the original damaged Chinese character image; the structure discrimination branch includes 5 layers of convolution, and its structure is exactly the same as that of the texture discrimination branch, and finally outputs the discrimination result through the Sigmoid function.

[0019] The loss function designed in step 3 is expressed as follows:

[0020] Where, To rebuild the losses, is the perceptual loss, is the style loss, To combat losses, is the intermediate supervision loss, are the weight coefficients of the corresponding losses respectively; Reconstruction losses Expressed as:

[0021] Where, Indicates the Euclidean distance, Indicates the expected value, represents the real image, Indicates the generated image; Perceptual loss Expressed as:

[0022] Where, and The real images are and generate images Feature representation extracted from the middle layer of the pre-trained network VGG-16, Indicates the number of feature layers in the i-th layer; Style Loss Expressed as:

[0023] Where, Represents Gram matrix operations; Fighting Losses Expressed as:

[0024] Where, Represents a real image The corresponding edge, Represents the generated image Corresponding edges, D represents the discriminator and G represents the generator; Intermediate supervision loss Expressed as:

[0025] Where, and Represents the projection function, respectively Mapping to edge graph ,Will Mapping to RGB image I g , represents the binary cross entropy, which is used to supervise edge generation.

[0026] The beneficial effects of the present invention are: The present invention is based on a calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network. The dual-stream generative adversarial network is used as the network model for image restoration. A dual-stream generator is designed for feature extraction. A feature fusion module and a multi-scale dilated convolution fusion module are used for feature fusion, so that texture features and structural features guide and complement each other, effectively avoiding problems such as information loss. A dual-stream discriminator is also designed for adversarial training. Finally, the trained model has a more natural restoration effect, more accurate results, and improved global and local consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the process of the calligraphy Chinese character image restoration method based on the dual-stream generative adversarial network of the present invention; Figure 2 This is a schematic diagram of the image restoration model structure in the calligraphy Chinese character image restoration method based on the dual-stream generative adversarial network of the present invention; Figure 3 It is a structural diagram of a bidirectional feature fusion module in the calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network of the present invention; Figure 4 It is a structural diagram of the multi-scale dilated convolution fusion module in the calligraphy Chinese character image restoration method based on the dual-stream generative adversarial network of the present invention; Figure 5 It is a structural diagram of the discriminator in the calligraphy Chinese character image restoration method based on the dual-stream generative adversarial network of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The present invention is based on a calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network. Figure 1 As shown, the specific steps include: Step 1: Collect a number of calligraphy images of different styles, generate a dataset after preprocessing, and divide it into training set and test set according to the ratio. Specifically: We collected several calligraphy character images of different styles and resized them to a consistent size. We then fused several masks with varying degrees of coverage (e.g., varying coverage ratios and damage shapes) with the calligraphy character images to generate damaged character images, simulating actual damage. We paired the calligraphy character images with the damaged character images to form a dataset, which was then divided into a training set and a test set in a 7:3 ratio.

[0030] Step 2: Build an image restoration model.

[0031] The image restoration model constructed by the present invention is based on an improved GAN network, including a generator and a discriminator.

[0032] The generator adopts a two-stream U-net variant architecture, including parallel texture branches and structure branches, which respectively extract texture features and structural features of the image; it also includes a bidirectional feature fusion module for fusing texture features and structural features and outputting fused features; it also includes a multi-scale dilated convolution fusion module for extracting multi-scale features from the fused features and performing channel splicing and weighted fusion on the multi-scale features to output dilated fused features; the feature fusion features undergo a 1×1 convolution with a step size of 1 to output the repaired image.

[0033] The generator adopts a two-stream U-net variant architecture, including parallel texture branch and structure branch; The texture branch includes an encoder and a decoder. The encoder contains 7 layers of convolution. The first layer uses a 7×7 convolution layer with a stride of 2. The second and third layers use 5×5 convolution layers with a stride of 2. The fourth to seventh layers use 3×3 convolution layers with a stride of 2. The decoder contains 7 layers of transposed convolution layers with a convolution kernel of 3×3 and a stride of 1. The network structure of the structure branch is exactly the same as that of the texture branch; The 3-channel damaged Chinese character image and mask are input into the texture branch. After feature extraction and downsampling by the encoder of the texture branch, they enter the decoder of the structure branch and upsample through layer-by-layer transposed convolution. The decoder is connected with the features extracted by the encoder of the structure branch. The decoder of the structure branch outputs the structural feature map. ; The edge map, grayscale map and mask corresponding to the damaged Chinese character image are input into the structure branch. After feature extraction and downsampling by the encoder of the structure branch, they enter the decoder of the texture branch and upsample through transposed convolution. They are jump-connected with the features extracted from the corresponding layer of the encoder of the texture branch, and the decoder of the texture branch outputs the texture feature map. .

[0034] Both the texture branch and the structure branch use partial convolution as the core operation. Each calculation is based only on the valid pixels marked by the current mask, that is, the area with a value of 1, and a weighted sum is performed and the output mask is automatically updated. In the decoding stage, the texture decoder upsamples through transposed convolution and introduces the corresponding layer structure features across branches for constraint; the structure decoder operates symmetrically and fuses the texture features. After the two features are finally reconstructed by the partial convolution layer, multi-scale information is integrated through skip layer connection to output the texture feature map. and structural feature diagram ,like Figure 2 shown.

[0035] The bidirectional feature fusion module realizes the information transfer between texture features and structural features through the information interaction mechanism, and introduces a gating mechanism to regulate the proportion of effective information in each feature, so that the features can be better fused. Figure 3 As shown, for the input texture feature map and structural feature diagram , the processing process is as follows: For texture feature maps , first with the structural feature map Perform channel-level splicing, and then obtain structural gating after linear mapping and Sigmoid function activation , the formula is as follows: , Where, is a channel-level splicing operation; g is a linear mapping function, which is implemented by a convolution layer with a convolution kernel size of 3; is the Sigmoid activation function; Gating the structure and structural feature diagram Multiply them together to get the structural features retained after gated screening, and then combine the retained structural features with the texture feature map Add together to get the texture features optimized by structural features , which is expressed as follows:

[0036] Among them, is the training parameter initialized to 0; For the structural feature map , first with the texture feature map Perform channel-level splicing, and then obtain texture gating after linear mapping and Sigmoid function activation , the formula is as follows:

[0037] Gating the texture Texture feature map Multiply them to get the texture features retained after gated filtering, and then combine the retained texture features with the structural feature map Add together to get the structural features optimized by texture features , which is expressed as follows:

[0038] Finally, the texture features obtained after structural feature optimization are and structural features optimized by texture features Perform channel-level splicing to obtain fusion features , which is expressed as follows: .

[0039] The fusion features output by the bidirectional feature fusion module Enter the multi-scale dilated convolution fusion module, which can expand the convolution receptive field and make the convolution kernel denser, so that the network can better capture the details of the image. Figure 4 As shown, for the input fusion features , the processing process is as follows: First, after a 3×3 convolution operation, the input fusion features The number of channels is reduced to 64, and the fused features after dimensionality reduction are input into three dilated convolutional layers with different expansion rates to extract multi-scale features, which are expressed as follows:

[0040] Where, represents a 3×3 dilated convolution, represents the expansion rate of the i-th branch, BN represents batch normalization, and ReLU represents the activation function; Then multi-scale features and Perform channel-level addition, then enter a 3×3 convolution layer for feature fusion, and perform weighted adjustment through learnable weight coefficients to obtain weighted features , which is expressed as follows:

[0041] Where, is the weight coefficient; The obtained weighted features With multi-scale features The channel level is added, and then enters a 3×3 convolution layer for feature fusion, and weighted adjustment is performed through the learnable weight coefficient to obtain the weighted feature , which is expressed as follows:

[0042] Where, is the weight coefficient; The weighted features 、 Multi-scale features obtained with dilated convolution After channel splicing, it enters a 1×1 convolution for channel recovery and is then added to the original input to obtain the feature output of the multi-scale dilated convolution fusion module, which is expressed as follows: .

[0043] In the multi-scale dilated convolution fusion module, except for the last 1×1 convolution layer, batch normalization and ReLU activation functions are added to the remaining layers to ensure training stability and output nonlinearity.

[0044] Finally, after a 1×1 convolution with a stride of 1, the number of feature map channels is reduced to 3, and the final repaired image is output.

[0045] In summary, the generator receives two inputs, respectively obtaining the texture features and structural features of the image. At the same time, through jump connections, the texture features guide the structural reconstruction, and the structural features can also guide the texture synthesis. The inputs of the structural features and texture features are entered into the bidirectional feature fusion module. The module uses gated convolution to retain and combine the important parts of the two features and finally outputs the fused features. The output fused features then enter the multi-scale dilated convolution fusion module. Through convolution and fusion of different scales, the correlation of local features of the image is enhanced, and the global and local consistency of the image is enhanced.

[0046] The discriminator adopts a two-stream network structure, such as Figure 5 As shown, it includes parallel texture discrimination branches and structure discrimination branches, which are used to analyze the texture features and structure features of the restored image respectively to distinguish the real image from the generated image.

[0047] Specifically, the texture discriminant branch takes as input the inpainted image output by the generator. It consists of five convolutional layers. The first three layers use 4×4 convolutional layers with a stride of 2 for downsampling, while the fourth and fifth layers use 4×4 convolutional layers with a stride of 1 to maintain resolution. The discriminant branch outputs the discriminant result using a sigmoid function. The structure discriminant branch takes as input the edge map of the inpainted image output by the generator (obtained after processing the inpainted image using a residual block and 1×1 convolution) and a grayscale image of the original damaged Chinese character image. This grayscale image serves as a conditional input to address the problem of sparse edges preventing the discriminator from learning effective information. The structure discriminant branch, consisting of five convolutional layers, has the same structure as the texture discriminant branch and outputs the discriminant result using a sigmoid function. The structure discriminant branch not only determines the authenticity of the generated edges but also ensures structural consistency by comparing the grayscale information with the real image. Furthermore, spectral normalization is used to constrain the weights of each layer, effectively alleviating the problem of GAN training instability.

[0048] The discriminator uses a global and local dual discriminator to simultaneously evaluate the authenticity of the texture and the rationality of the structure, increase the attention to the generated images originally belonging to the damaged area, and improve the repair accuracy and authenticity of the trained model.

[0049] Step 3: Use the training set and test set as the input of the image restoration model, and design a loss function to train and test the image restoration model.

[0050] The designed loss function is expressed as follows:

[0051] Where, To rebuild the losses, is the perceptual loss, is the style loss, To combat losses, is the intermediate supervision loss, are the weight coefficients of the corresponding losses respectively; Reconstruction losses It is used to constrain the similarity between the generated image and the real image at the pixel level. It is optimized using the L1 distance and is expressed as:

[0052] Where, Indicates the Euclidean distance, Indicates the expected value, represents the real image, Indicates the generated image; Perceptual loss The high-level semantic features are extracted through pre-trained convolutional neural networks to ensure that the generated image is consistent with the real image in semantic structure, which can be expressed as:

[0053] Where, and The real images are and generate images Feature representation extracted from the middle layer of the pre-trained network VGG-16, Indicates the number of feature layers in the i-th layer; Style Loss To maintain the similarity between the generated image and the real image in texture and style, the Gram matrix is ​​calculated at each layer of the network, and then the Euclidean distance between the corresponding layers is calculated. Finally, the Euclidean distances of different layers are added together, which is expressed as:

[0054] Where, Represents Gram matrix operation, which is used to capture style information between features; Fighting Losses It means that in the process of optimizing the model, the generator and the discriminator are trained against each other so that the generated results can be misjudged as true by the discriminator, which can be expressed as:

[0055] Where, Represents a real image The corresponding edge, Represents the generated image Corresponding edges, D represents the discriminator, G represents the generator; the discriminator goal is to transform the real image and the corresponding edge If the judgment is true, an image will be generated and the corresponding edge The judgment is false; the generator goal is to generate images and the corresponding edge Deceiving the discriminator.

[0056] Intermediate supervision loss Supervise the intermediate features of the generator to ensure that the structural feature Fs and the texture feature Ft accurately capture the edge structure and texture information respectively, which can be expressed as:

[0057] Where, and Represents the projection function, respectively Mapping to edge graph ,Will Mapping to RGB image I g , represents binary cross entropy, which is used to supervise edge generation; Used for supervised texture generation.

[0058] After approximately 300,000 training iterations, the model's performance was evaluated on a test set, demonstrating its high restoration accuracy. Ultimately, the optimal Chinese character image restoration model was obtained, demonstrating outstanding performance in the task of restoring damaged Chinese characters.

[0059] Step 4: Input the Chinese character image to be repaired into the trained image repair model to obtain the calligraphy Chinese character image repair result.

[0060] Example 1 This embodiment provides a calligraphy Chinese character image restoration method based on a two-stream generative adversarial network, which specifically includes the following steps: Step 1: Collect a number of calligraphy characters of different styles, generate a dataset after preprocessing, and divide it into training set and test set according to the ratio; Step 2: Build an image restoration model; Step 3: Use the training set and test set as the input of the image restoration model, and design a loss function to train and test the image restoration model; Step 4: Input the Chinese character image to be repaired into the trained image repair model to obtain the calligraphy Chinese character image repair result.

[0061] Example 2 Based on Example 1, step 1 is specifically as follows: Several calligraphy character images of different styles are collected, and several masks with different covering degrees are fused with the calligraphy character images to generate damaged Chinese character images. The calligraphy character images and the damaged Chinese character images are paired to form a dataset, which is then divided into training and test sets in a 7:3 ratio.

[0062] Example 3 Based on Example 2, the image restoration model constructed in step 2 includes a generator and a discriminator; The generator adopts a two-stream U-net variant architecture, including parallel texture branches and structure branches, which respectively extract texture features and structural features of the image. It also includes a bidirectional feature fusion module for fusing texture features and structural features to output fused features. It also includes a multi-scale dilated convolution fusion module for extracting multi-scale features from the fused features, performing channel splicing and weighted fusion on the multi-scale features, and outputting dilated fused features. The feature fusion features undergo a 1×1 convolution with a stride of 1 to output the restored image. The discriminator adopts a two-stream network structure, including parallel texture discrimination branches and structure discrimination branches, which are used to analyze the texture features and structural features of the restored image respectively to distinguish between real images and generated images.

[0063] Example 4 Based on Example 3, the generator adopts a two-stream U-net variant architecture, including parallel texture branch and structure branch; The texture branch includes an encoder and a decoder. The encoder contains 7 layers of convolution. The first layer uses a 7×7 convolution layer with a stride of 2. The second and third layers use 5×5 convolution layers with a stride of 2. The fourth to seventh layers use 3×3 convolution layers with a stride of 2. The decoder contains 7 layers of transposed convolution layers with a convolution kernel of 3×3 and a stride of 1. The network structure of the structure branch is exactly the same as that of the texture branch; The 3-channel damaged Chinese character image and mask are input into the texture branch. After feature extraction and downsampling by the encoder of the texture branch, they enter the decoder of the structure branch and upsample through layer-by-layer transposed convolution. The decoder is connected with the features extracted by the encoder of the structure branch. The decoder of the structure branch outputs the structural feature map. ; The edge map, grayscale map and mask corresponding to the damaged Chinese character image are input into the structure branch. After feature extraction and downsampling by the encoder of the structure branch, they enter the decoder of the texture branch and upsample through transposed convolution. They are jump-connected with the features extracted from the corresponding layer of the encoder of the texture branch, and the decoder of the texture branch outputs the texture feature map. .

[0064] Bidirectional feature fusion module, for the input texture feature map and structural feature diagram , the processing process is as follows: For texture feature maps , first with the structural feature map Perform channel-level splicing, and then obtain structural gating after linear mapping and Sigmoid function activation , the formula is as follows:

[0065] Where, is a channel-level splicing operation; g is a linear mapping function, which is implemented by a convolution layer with a convolution kernel size of 3; is the Sigmoid activation function; Gating the structure and structural feature diagram Multiply them together to get the structural features retained after gated screening, and then combine the retained structural features with the texture feature map Add together to get the texture features optimized by structural features , which is expressed as follows:

[0066] Among them, is the training parameter initialized to 0; For the structural feature map , first with the texture feature map Perform channel-level splicing, and then obtain texture gating after linear mapping and Sigmoid function activation , the formula is as follows:

[0067] Gating the texture Texture feature map Multiply them to get the texture features retained after gated filtering, and then combine the retained texture features with the structural feature map Add together to get the structural features optimized by texture features , which is expressed as follows:

[0068] Finally, the texture features obtained after structural feature optimization are and structural features optimized by texture features Perform channel-level splicing to obtain fusion features , which is expressed as follows: .

[0069] Multi-scale dilated convolution fusion module, for the fusion features of the input , the processing process is as follows: First, after a 3×3 convolution operation, the input fusion features The number of channels is reduced to 64, and the fused features after dimensionality reduction are input into three dilated convolutional layers with different expansion rates to extract multi-scale features, which are expressed as follows:

[0070] Where, represents a 3×3 dilated convolution, represents the expansion rate of the i-th branch, BN represents batch normalization, and ReLU represents the activation function; Then multi-scale features and Perform channel-level addition, then enter a 3×3 convolution layer for feature fusion, and perform weighted adjustment through learnable weight coefficients to obtain weighted features , which is expressed as follows:

[0071] Where, is the weight coefficient; The obtained weighted features With multi-scale features The channel level is added, and then enters a 3×3 convolution layer for feature fusion, and weighted adjustment is performed through the learnable weight coefficient to obtain the weighted feature , which is expressed as follows:

[0072] Where, is the weight coefficient; The weighted features 、 Multi-scale features obtained with dilated convolution After channel splicing, it enters a 1×1 convolution for channel recovery and is then added to the original input to obtain the feature output of the multi-scale dilated convolution fusion module, which is expressed as follows: .

[0073] Example 5 Based on Example 4, the discriminator adopts a dual-stream network structure, including parallel texture discrimination branches and structure discrimination branches; The texture discrimination branch takes as input the restored image output by the generator. The texture discrimination branch consists of five convolution layers. The first to third layers use 4×4 convolution layers with a stride of 2 for downsampling, while the fourth and fifth layers use 4×4 convolution layers with a stride of 1 to maintain the resolution. The discrimination result is finally output through the Sigmoid function. The input of the structure discrimination branch is the edge map of the repaired image output by the generator and the grayscale image of the original damaged Chinese character image; the structure discrimination branch includes 5 layers of convolution, and its structure is exactly the same as that of the texture discrimination branch, and finally outputs the discrimination result through the Sigmoid function.

[0074] Example 6 Based on Example 5, the loss function designed in step 3 is expressed as follows:

[0075] Where, To rebuild the losses, is the perceptual loss, is the style loss, To combat losses, is the intermediate supervision loss, are the weight coefficients of the corresponding losses respectively; Reconstruction losses Expressed as:

[0076] Where, Indicates the Euclidean distance, Indicates the expected value, represents the real image, Indicates the generated image; Perceptual loss Expressed as:

[0077] Where, and The real images are and generate images Feature representation extracted from the middle layer of the pre-trained network VGG-16, Indicates the number of feature layers in the i-th layer; Style Loss Expressed as:

[0078] Where, Represents Gram matrix operations; Fighting Losses Expressed as:

[0079] Where, Represents a real image The corresponding edge, Represents the generated image Corresponding edges, D represents the discriminator and G represents the generator; Intermediate supervision loss Expressed as:

[0080] Where, and Represents the projection function, respectively Mapping to edge graph ,Will Mapping to RGB image I g , represents the binary cross entropy, which is used to supervise edge generation.

Claims

1. A calligraphy Chinese character image restoration method based on a two-stream generative adversarial network, characterized by: The specific steps include: Step 1: Collect a number of calligraphy characters of different styles, generate a dataset after preprocessing, and divide it into training set and test set according to the ratio; Step 2: Build an image restoration model; Step 3: Use the training set and test set as the input of the image restoration model, and design a loss function to train and test the image restoration model; Step 4: Input the Chinese character image to be repaired into the trained image repair model to obtain the calligraphy Chinese character image repair result.

2. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 1 is characterized in that: Step 1 is as follows: Several calligraphy character images of different styles are collected, and several masks with different covering degrees are fused with the calligraphy character images to generate damaged Chinese character images. The calligraphy character images and the damaged Chinese character images are paired to form a dataset, and then divided into training set and test set according to the proportion.

3. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 2 is characterized in that: The image restoration model constructed in step 2, including the generator and the discriminator; The generator adopts a two-stream U-net variant architecture, including parallel texture branches and structure branches, which respectively extract texture features and structural features of the image; It also includes a bidirectional feature fusion module for fusing texture features and structural features and outputting fused features; it also includes a multi-scale dilated convolution fusion module for extracting multi-scale features from the fused features, performing channel splicing and weighted fusion on the multi-scale features, and outputting dilated fused features; the dilated fused features are subjected to a 1×1 convolution with a step size of 1 to output a restored image; The discriminator adopts a dual-stream network structure, including a parallel texture discrimination branch and a structure discrimination branch, which are respectively used to analyze the texture features and structure features of the restored image to distinguish between real images and generated images.

4. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 3 is characterized in that: The generator adopts a two-stream U-net variant architecture, including parallel texture branch and structure branch; The texture branch includes an encoder and a decoder. The encoder includes 7 convolution layers, the first layer uses a 7×7 convolution layer with a stride of 2, the second and third layers use 5×5 convolution layers with a stride of 2, and the fourth to seventh layers use 3×3 convolution layers with a stride of 2 respectively; the decoder includes 7 transposed convolution layers with a convolution kernel of 3×3 and a stride of 1; The network structure of the structure branch is completely consistent with that of the texture branch; The 3-channel damaged Chinese character image and mask are input into the texture branch. After feature extraction and downsampling by the encoder of the texture branch, they enter the decoder of the structure branch and upsample by layer-by-layer transposed convolution. The decoder is connected with the features extracted by the encoder of the structure branch, and the decoder of the structure branch outputs the structural feature map. ; The edge map, grayscale map and mask corresponding to the damaged Chinese character image are input into the structure branch. After feature extraction and downsampling by the encoder of the structure branch, they enter the decoder of the texture branch and upsample by transposed convolution. The decoder is connected with the features extracted by the encoder of the texture branch, and the texture feature map is output by the decoder of the texture branch. .

5. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 4 is characterized in that: The bidirectional feature fusion module, for the input texture feature map and structural feature diagram , the processing process is as follows: For texture feature maps , first with the structural feature map Perform channel-level splicing, and then obtain structural gating after linear mapping and Sigmoid function activation , the formula is as follows: Where, is a channel-level splicing operation; g is a linear mapping function, which is implemented by a convolution layer with a convolution kernel size of 3; is the Sigmoid activation function; Gating the structure and structural feature diagram Multiply them together to get the structural features retained after gated screening, and then combine the retained structural features with the texture feature map Add together to get the texture features optimized by structural features , which is expressed as follows: Among them, is the training parameter initialized to 0; For the structural feature map , first with the texture feature map Perform channel-level splicing, and then obtain texture gating after linear mapping and Sigmoid function activation , the formula is as follows: Gating the texture Texture feature map Multiply them to get the texture features retained after gated filtering, and then combine the retained texture features with the structural feature map Add together to get the structural features optimized by texture features , which is expressed as follows: Finally, the texture features obtained after structural feature optimization are and structural features optimized by texture features Perform channel-level splicing to obtain fusion features , which is expressed as follows: 。 6. The calligraphy Chinese character image restoration method based on a two-stream generative adversarial network according to claim 5 is characterized in that: The multi-scale dilated convolution fusion module is used to fusion features of the input , the processing process is as follows: First, after a 3×3 convolution operation, the input fusion features The number of channels is reduced to 64, and the fused features after dimensionality reduction are input into three dilated convolutional layers with different expansion rates to extract multi-scale features, which are expressed as follows: Where, represents a 3×3 dilated convolution, represents the expansion rate of the i-th branch, BN represents batch normalization, and ReLU represents the activation function; Then multi-scale features and Perform channel-level addition, then enter a 3×3 convolution layer for feature fusion, and perform weighted adjustment through learnable weight coefficients to obtain weighted features , which is expressed as follows: Where, is the weight coefficient; The obtained weighted features With multi-scale features The channel level is added, and then enters a 3×3 convolution layer for feature fusion, and weighted adjustment is performed through the learnable weight coefficient to obtain the weighted feature , which is expressed as follows: Where, is the weight coefficient; The weighted features 、 Multi-scale features obtained with dilated convolution After channel splicing, it enters a 1×1 convolution for channel recovery and is then added to the original input to obtain the feature output of the multi-scale dilated convolution fusion module, which is expressed as follows: 。 7. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 6 is characterized in that: The discriminator adopts a dual-stream network structure, including a parallel texture discrimination branch and a structure discrimination branch; The texture discrimination branch input is the restored image output by the generator; the texture discrimination branch contains 5 layers of convolution, the first to third layers use 4×4 convolution layers with a stride of 2 for downsampling, the fourth and fifth layers use 4×4 convolution layers with a stride of 1 to maintain the resolution, and finally output the discrimination result through the Sigmoid function; The input of the structure discrimination branch is the edge map of the repaired image output by the generator and the grayscale map of the original damaged Chinese character image; the structure discrimination branch includes 5 layers of convolution, and its structure is exactly the same as that of the texture discrimination branch, and finally the discrimination result is output through the Sigmoid function.

8. The calligraphy Chinese character image restoration method based on a dual-stream generative adversarial network according to claim 7 is characterized in that: The loss function designed in step 3 is expressed as follows: Where, To rebuild the losses, is the perceptual loss, is the style loss, To combat losses, is the intermediate supervision loss, are the weight coefficients of the corresponding losses respectively; Reconstruction losses Expressed as: Where, Indicates the Euclidean distance, Indicates the expected value, represents the real image, Indicates the generated image; Perceptual loss Expressed as: Where, and The real images are and generate images Feature representation extracted from the middle layer of the pre-trained network VGG-16, Indicates the number of feature layers in the i-th layer; Style Loss Expressed as: Where, Represents Gram matrix operations; Fighting Losses Expressed as: Where, Represents a real image The corresponding edge, Represents the generated image Corresponding edges, D represents the discriminator and G represents the generator; Intermediate supervision loss Expressed as: Where, and Represents the projection function, respectively Mapping to edge graph ,Will Mapping to RGB image I g , represents the binary cross entropy, which is used to supervise edge generation.

Images