Method for repairing damaged calligraphy Chinese character image based on T-former improvement

Abstract

The invention discloses a T-form improvement-based damaged calligraphy Chinese character image restoration method, which comprises the steps of collecting and generating single calligraphy Chinese character images of different styles, and preprocessing each calligraphy Chinese character image to serve as a data set; constructing an image restoration model; designing a loss function, and training and testing the constructed image restoration model by using the data set; and inputting a to-be-restored damaged calligraphy Chinese character image into the trained image restoration model to obtain a restored image. The method for repairing the damaged calligraphy Chinese character image based on T-form improvement has the characteristic of high damaged calligraphy Chinese character repairing accuracy, and meanwhile, the model has relatively low parameter quantity and calculation complexity.

Description

Technical Field

[0001] The invention belongs to the technical field of computer image processing, and in particular relates to a method for repairing damaged calligraphy Chinese character images based on an improved T-former. Background Art

[0002] The restoration of calligraphy is a complex and valuable task. It aims to restore the ancient calligraphy style, restore the stroke structure characteristics, explore the Chinese beauty expressed by calligraphy masters through words, and then preserve and transmit the cultural value carried by calligraphy. In addition, the research on the restoration of damaged Chinese characters can save the manpower, material resources and time costs brought by manual restoration. It not only provides a certain reference value for professional restorers, but also facilitates calligraphy enthusiasts to study the relics of calligraphy masters. With the rapid development of digital technology and the popularization of calligraphy culture, the use of scientific and technological methods to complete the restoration of damaged calligraphy Chinese characters has become a research hotspot in academia and practical application fields. Traditional image restoration technology uses the content near the damaged area to infer and fill the missing area by statistical probability. However, because the content of the damaged text is sparsely distributed, the restoration algorithm cannot obtain enough reference information, and often outputs irregular stroke structures, resulting in blurred or non-existent Chinese characters, and cannot complete the restoration of damaged calligraphy Chinese characters. Summary of the invention

[0003] The purpose of the present invention is to provide a method for repairing damaged calligraphy Chinese character images based on an improved T-former, which has the characteristics of high accuracy in repairing damaged calligraphy Chinese characters, and the model has a lower number of parameters and computational complexity.

[0004] The technical solution adopted by the present invention is a method for repairing damaged calligraphy Chinese character images based on the improved T-former, which specifically includes the following steps:

[0005] S1: Collect and generate individual calligraphy Chinese character images of different styles, and pre-process each calligraphy Chinese character image as a dataset;

[0006] S2: Build image restoration model;

[0007] S3: Design a loss function and use the dataset to train and test the constructed image restoration model;

[0008] S4: Input the damaged calligraphy Chinese character image to be repaired into the trained image repair model to obtain a repaired image.

[0009] The present invention is also characterized in that:

[0010] S1 is specifically:

[0011] S1.1: Select at least one font that matches the calligraphy style, generate several corresponding single Chinese character images, and adjust the font size;

[0012] S1.2: Download the mask dataset for the image restoration task. Select 10% to 50% of the mask images with different degrees of damage according to the mask images close to the boundary and not close to the boundary to simulate various damage conditions of images in reality. Put several single Chinese character images into the same folder, randomly add two types of masks with different degrees of damage, and generate a dataset.

[0013] S1.3: The resolution of the images in the dataset is uniformly adjusted to 64*64, and the dataset is divided into a training set and a test set in a ratio of 8:2.

[0014] The image restoration model constructed in S2 includes a 7×7 convolutional layer, a feature extraction network, an intermediate module, a feature recovery network, and a 7×7 convolutional layer connected in sequence;

[0015] The feature extraction network consists of three transformer_High modules connected in sequence. The input of the first transformer_High module is the feature of the original image after being processed by a 7x7 convolutional layer. The input of the second and third transformer_High modules are the output of the previous transformer_High module after downsampling. In addition to being input to the next stage after downsampling, the feature maps output by each transformer_High module are also input to the module of the corresponding layer of the feature recovery network, and feature fusion is performed with the corresponding module in the feature recovery network using a jump connection method.

[0016] The feature recovery network consists of an original transformer module and two transformer_High modules connected in sequence; the input of the original transformer module is the output of the intermediate module, which is upsampled and fused with the feature map input by the third transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the second transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the second transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the third transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the first transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer;

[0017] The middle module is an original transformer module.

[0018] The transformer_High module consists of three parts, including the first normalization layer, the receptive field enhancement module, the second normalization layer, the linear attention layer with a gating mechanism, the third normalization layer, and the improved feedforward neural network connected in sequence; and the output of the 7×7 convolution is spliced ​​with the output of the receptive field enhancement module, the output of the receptive field enhancement module is spliced ​​with the output of the linear attention layer with a gating mechanism, and the output of the linear attention layer with a gating mechanism is spliced ​​with the output of the improved feedforward neural network layer through residual connections;

[0019] The processing of the transformer_High module is as follows:

[0020] First, the input feature map X is normalized and used as the input of the receptive field enhancement module to obtain the output result X p , X p It is concatenated with the input feature X using a residual connection; then, after normalization, it is used as the input of the linear attention layer with a gating mechanism to obtain the output result X M , X M With X p Then perform residual connection; finally, after normalization, it is used as the input of the improved feedforward neural network to obtain the output result X F , X F With X M After splicing using the residual connection method, the final output result is obtained.

[0021] The receptive field enhancement module includes a 1×1 convolution, a 3×3 dilated convolution with dilation rates of 1, 2, and 3 connected to the 1×1 convolution, a Gelu activation function, a 3×3 convolution, and a Gelu activation function;

[0022] The processing process of the receptive field enhancement module is as follows:

[0023] First, the feature X of the input receptive field enhancement module is reduced to 1 / 2 of its original dimension by 1×1 convolution, and then X is processed by 3×3 dilated convolution with dilation=1, dilation=2, and dilation=3 respectively to obtain feature maps X1, X2, and X3. At this time, the number of channels of X1, X2, and X3 changes from 1 / 2channels to 1 / 3channels; then, X1, X2, and X3 are concatenated in the channel direction; finally, the enhanced features are output after being processed by Gelu activation function, 3×3 convolution, and Gelu activation function in sequence.

[0024] The feature map X is processed by 1×1 convolution and then the input hole convolution is processed. The output is:

[0025]

[0026] Where D(x, y) represents the value of the output feature map at position (x, y); I(x, y) represents the value of the input feature map at position (x, y); W(m, l) represents the weight value of the convolution kernel at position (m, l); M represents the size of the convolution kernel; d represents the hole rate, which controls the expansion of the receptive field during the convolution operation; m, l are the offsets of the convolution kernel relative to the center point, and the value range is x+m·d, y+l·d represent the sampling points selected by the dilated convolution in the input feature map, and the interval is determined by d.

[0027] For the input feature Y and the linear attention operator The output O of the attention with gating mechanism can be written as the Hadamard product of operators A and G, expressed as:

[0028] O=A⊙G

[0029]

[0030] G=φ g (W g Y)

[0031] Where W g is a learnable parameter, φ g is the activation function, ⊙ represents the Hadamard product;

[0032] The attention operator first performs three different transformations on the input features Y as new inputs and reshapes them into two-dimensional matrices to obtain the corresponding three embeddings: query Q, key K, and value V. Then the corresponding attention result O can be obtained as follows:

[0033]

[0034] In the formula, represents the transpose of K, C represents the number of channels; after the Softmax normalized dot product attention, we get:

[0035]

[0036] Where q, k, v correspond to query Q, key K, and value V respectively; N = H*W, which represents the product of image width and height;

[0037] By approximating the exponential function through Taylor expansion to linearize the attention, the Taylor formula of the exponential function that constitutes the Softmax operator is:

[0038] exp(x) = e x ≈1+x

[0039]

[0040] The improved feedforward neural network comprises a 1×1 point-by-point convolution and two parallel branches respectively connected to the 1×1 point-by-point convolution, one branch is a 3×3 depth-separable convolution, an activation function, a 3×3 depth-separable convolution, and an activation function connected in sequence; the other branch is a 5×5 depth-separable convolution, an activation function, a 5×5 depth-separable convolution, and an activation function connected in sequence; the two parallel branches are connected to a 1×1 convolution after a splicing operation; and the two parallel branches are connected with a splicing operation;

[0041] The processing of the improved feedforward neural network is as follows:

[0042] First, the input features are subjected to 1×1 point-by-point convolution to adjust the number of channels of the output features to twice the number of input channels while keeping the spatial size unchanged; then, they are processed by parallel 3×3 and 5×5 depthwise separable convolutions and activated by the activation function, and then split according to the channel direction to obtain feature maps X1_3, X2_3 and X1_5, X2_5 with the same number of channels; X1_3 and X1_5 are spliced ​​along the channel and input into 3×3 depthwise separable convolution, and X2_3 and X2_5 are spliced ​​along the channel and input into 5×5 depthwise separable convolution, and then spliced ​​along the channel after the activation function; finally, the output features are obtained by adjusting the number of channels through 1×1 convolution.

[0043] The loss function designed in S3 is expressed as:

[0044] L=λ r L re +λ p L perc +λ s L style +λ a L adv

[0045] In the formula, L represents the loss function, L re represents the reconstruction loss, L perc represents the perceptual loss, L style represents the style loss, L adv Represents adversarial loss; and sets the weight λ of each loss r =1,λ p =1,λ s =250 and λ a =0.1;

[0046] The reconstruction loss is defined as:

[0047] Lre =||I out -I g ||1

[0048] In the formula, I out represents the input image, I g represents the real image, ||·||1 represents the calculation of Euclidean distance;

[0049] Perceptual loss, defined as:

[0050]

[0051] In the formula, φ i (I out ) and φ i (I g ) represent the output image I out and the real image I g Feature representation extracted in the intermediate layers of the pre-trained network; Indicates expected value; N i Indicates the number of feature layers of the i-th layer;

[0052] Style loss, defined as:

[0053]

[0054] In the formula, Indicates that at layer j the corresponding activation map Φ j Constructed C j ×C j Gram matrix; represents the expected value of the jth layer;

[0055] Adversarial loss, defined as:

[0056]

[0057] In the formula, D(I g ) represents the discriminator's response to the real image I g is a true probability prediction; D(I out ) represents the discriminator's output image I out is a true probabilistic prediction; is a sample from the true data distribution, represents a noise vector sampled from the prior distribution.

[0058] The constructed image restoration model is trained and tested using the dataset in S3. During training, the batch size of the training set images is set to 8, the number of training iterations is 300,000, and the learning rate is 0.001. The Adam optimizer is used to train the constructed image restoration model, and the weight parameters of the connections between the network layers are continuously adjusted; the test set is used to test the accuracy of the trained image restoration model.

[0059] The beneficial effects of the present invention are:

[0060] The present invention is based on a method for repairing damaged calligraphy Chinese character images improved by T-former. The improved T-former network is used as the network for image repair, a three-layer network architecture is designed, and the transformer_High module with enhanced receptive field and the original transformer module are used as feature extraction modules. A convolution module with a parallel structure is designed as a feedforward neural network, and efficient feature extraction and expression are achieved through deep separable convolutions of different sizes and feature fusion. The texture pattern learning ability of CNN is combined with the ability of attention to capture long-range dependencies, effectively extracting local and global features, and combining information from different feature maps, thereby improving the feature representation ability, and ultimately improving the accuracy and authenticity of the repair results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of a method for repairing damaged calligraphy Chinese character images based on T-former improvement of the present invention;

[0062] Figure 2 It is a schematic diagram of the repair model structure in the repair method of damaged calligraphy Chinese character images based on the T-former improvement of the present invention;

[0063] Figure 3 It is a schematic diagram of the structure of the improved transformer_High module in the method for repairing damaged calligraphy Chinese character images based on the T-former improvement of the present invention;

[0064] Figure 4 It is a schematic diagram of the structure of the receptive field enhancement module in the method for repairing damaged calligraphy Chinese character images based on the T-former improvement of the present invention;

[0065] Figure 5 It is a schematic diagram of the improved feedforward neural network structure in the method for repairing damaged calligraphy Chinese character images based on the T-former improvement of the present invention;

[0066] Figure 6 It is a schematic diagram of the deep separable convolution structure in the method for repairing damaged calligraphy Chinese character images based on the T-former improvement of the present invention;

[0067] Figure 7 It is a schematic diagram of the point-by-point convolution structure in the method for repairing damaged calligraphy Chinese character images based on the improved T-former of the present invention. DETAILED DESCRIPTION

[0068] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] The present invention is based on the improved T-former method for repairing damaged calligraphy Chinese character images. Figure 1 As shown, the specific steps include:

[0070] S1: Collect and generate individual calligraphy Chinese character images of different styles, and pre-process each calligraphy Chinese character image as a data set. The specific steps include the following:

[0071] S1.1: Select at least one font that matches the calligraphy style, generate several corresponding single Chinese character images, and adjust the font size;

[0072] S1.2: Download the mask dataset for the image restoration task. Select mask images with different degrees of 10% to 50% based on the mask images close to the boundary and those not close to the boundary to simulate various damage conditions of images in reality. Put several single Chinese character images into the same folder, randomly add two types of masks with different degrees, and generate a dataset.

[0073] S1.3: The resolution of the images in the dataset is uniformly adjusted to 64*64, and the dataset is divided into a training set and a test set in a ratio of 8:2.

[0074] S2: Build an image restoration model.

[0075] The constructed image restoration model is in the form of a three-layer U-net network. The third-layer downsampling operation of the feature extraction network and the third-step upsampling operation of the feature recovery network are removed, which improves the resolution of the deepest layer features to deal with the problem of useless features in the deepest layer of low-resolution images due to low resolution. The improved transformer_High module and the original transformer module are used as feature extraction and feature recovery modules for each layer, respectively. Among them, convolutional layers, skip connection layers, fully connected layers, and linear attention layers with gating mechanisms are included; a new feedforward neural network is designed using a deep separable convolution block, and efficient feature extraction is achieved through deep separable convolutions of different sizes and feature fusion, combining information from different feature maps to improve feature representation capabilities; a receptive field enhancement module is added to the transformer_High module, and contextual information of different ranges is captured through multi-scale dilated convolutions, and channel compression and restoration are used to improve computational efficiency.

[0076] The specific network structure is as follows Figure 2 As shown, it includes a 7×7 convolutional layer, a feature extraction network, an intermediate module, a feature recovery network and a 7×7 convolutional layer connected in sequence;

[0077] The feature extraction network consists of three transformer_High modules connected in sequence. The input of the first transformer_High module is the feature of the original image after being processed by a 7x7 convolutional layer. The input of the second and third transformer_High modules are the output of the previous transformer_High module after downsampling. In addition to being input to the next stage after downsampling, the feature maps output by each transformer_High module are also input to the module of the corresponding layer of the feature recovery network, and feature fusion is performed with the corresponding module in the feature recovery network using a jump connection method.

[0078] The feature recovery network consists of an original transformer module and two transformer_High modules connected in sequence; the input of the original transformer module is the output of the intermediate module, which is upsampled and fused with the feature map input by the third transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the second transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the second transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the third transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the first transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer;

[0079] The middle module is an original transformer module.

[0080] Among them, the transformer_High module in the image restoration model consists of three parts, such as Figure 3 As shown, it includes a first normalization layer, a receptive field enhancement module, a second normalization layer, a linear attention layer with a gating mechanism, a third normalization layer, and an improved feedforward neural network connected in sequence; and the output of the 7×7 convolution is spliced ​​with the output of the receptive field enhancement module, the output of the receptive field enhancement module is spliced ​​with the output of the linear attention layer with a gating mechanism, and the output of the linear attention layer with a gating mechanism is spliced ​​with the output of the improved feedforward neural network layer through residual connections;

[0081] The processing of the transformer_High module is as follows:

[0082] First, the input feature map X is normalized and used as the input of the receptive field enhancement module to obtain the output result X p , X p It is concatenated with the input feature X using a residual connection; then, after normalization, it is used as the input of the linear attention layer with a gating mechanism to obtain the output result X M , X M With X p Then perform residual connection; finally, after normalization, it is used as the input of the improved feedforward neural network to obtain the output result X F , X F With X M After splicing using the residual connection method, the final output result is obtained.

[0083] In addition, the receptive field enhancement module in the image restoration model, such as Figure 4 As shown, it includes a 1×1 convolution, a 3×3 dilated convolution with dilation rates of 1, 2, and 3 connected to the 1×1 convolution, a Gelu activation function, a 3×3 convolution, and a Gelu activation function;

[0084] The processing process of the receptive field enhancement module is as follows:

[0085] First, the feature X of the input receptive field enhancement module is reduced to 1 / 2 of its original dimension by 1×1 convolution. Then, X is processed by 3×3 dilated convolution with dilation=1 (receptive field is 3x3), dilation=2 (receptive field is 5x5), and dilation=3 (receptive field is 7x7) to obtain feature maps X1, X2, and X3. At this time, the number of channels of X1, X2, and X3 changes from 1 / 2channels to 1 / 3channels. Subsequently, X1, X2, and X3 are concatenated in the channel direction to restore the number of feature channels. Finally, the enhanced features are output after being processed by Gelu activation function, 3×3 convolution, and Gelu activation function.

[0086] The dilated convolution can expand the receptive field of the convolution kernel without increasing the computational cost, thereby capturing more contextual information. The feature map X is processed by a 1×1 convolution and then the input dilated convolution is output as follows:

[0087]

[0088] Where D(x, y) represents the value of the output feature map at position (x, y); I(x, y) represents the value of the input feature map at position (x, y); W(m, l) represents the weight value of the convolution kernel at position (m, l); M represents the size of the convolution kernel; d represents the hole rate, which controls the expansion of the receptive field during the convolution operation; m, l are the offsets of the convolution kernel relative to the center point, and the value range is x+m·d, y+l·d represent the sampling points selected by the dilated convolution in the input feature map, and the interval is determined by d. The dilated convolution significantly increases the receptive field by increasing the interval between sampling points (i.e., expanding the effective size of the convolution kernel), and compared with pooling or strided convolution, the dilated convolution does not lose spatial information.

[0089] Then, the input features and the features after receptive field enhancement are concatenated through residual connections, normalized, and used as the input of the linear attention layer with a gating mechanism. The gating mechanism can be regarded as a neural network layer whose output M is the product of the two linear transformation components of the input X, and the calculation formula is as follows:

[0090] M=I⊙G

[0091] I=φ i (W u X)

[0092] G=φ g (W g X)

[0093] Where W u and W g is a learnable parameter, φ i ,φ g is the corresponding activation function and ⊙ is the Hadamard product.

[0094] For input feature Y and linear attention operator The output O of the attention with gating mechanism can be written as the Hadamard product of operators A and G, expressed as:

[0095] O=A⊙G

[0096]

[0097] G=φ g (W g Y)

[0098] Where W g is a learnable parameter, φ g is the activation function, ⊙ represents the Hadamard product;

[0099] The attention operator first performs three different transformations on the input features Y as new inputs and reshapes them into two-dimensional matrices to obtain the corresponding three embeddings: query Q, key K, and value V. Then the corresponding attention result O can be obtained as follows:

[0100]

[0101]

[0102] In the formula, represents the transpose of K, C represents the number of channels; after the Softmax normalized dot product attention, we get:

[0103]

[0104] Where q, k, v correspond to query Q, key K, and value V respectively; N = H*W, which represents the product of image width and height;

[0105] By approximating the exponential function through Taylor expansion to linearize the attention, the Taylor formula of the exponential function that constitutes the Softmax operator is:

[0106] exp(x) = e x ≈1+x

[0107]

[0108] Again, the improved feedforward neural network in the image restoration model, such as Figure 5 As shown, it includes a 1×1 point-by-point convolution and two parallel branches connected to the 1×1 point-by-point convolution respectively. One branch is a 3×3 depth-separable convolution, an activation function, a 3×3 depth-separable convolution, and an activation function connected in sequence; the other branch is a 5×5 depth-separable convolution, an activation function, a 5×5 depth-separable convolution, and an activation function connected in sequence; the two parallel branches are connected to a 1×1 convolution after a splicing operation; there is a splicing operation between the two parallel branches;

[0109] The processing of the improved feedforward neural network is as follows:

[0110] First, the input features are subjected to 1×1 point-by-point convolution to adjust the number of channels of the output features to twice the number of input channels while keeping the spatial size unchanged; then, they are processed by parallel 3×3 and 5×5 depthwise separable convolutions and activated by the activation function, and then split according to the channel direction to obtain feature maps X1_3, X2_3 and X1_5, X2_5 with the same number of channels; X1_3 and X1_5 are spliced ​​along the channel and input into 3×3 depthwise separable convolution, and X2_3 and X2_5 are spliced ​​along the channel and input into 5×5 depthwise separable convolution, and then spliced ​​along the channel after the activation function; finally, the output features are obtained by adjusting the number of channels through 1×1 convolution.

[0111] The structures of depth-wise separable convolution and point-wise convolution are as follows Figure 6-7 As shown, assuming that the number of input data channels is C, the number of required output data channels is N, and the number of parameters required for ordinary convolution is D K ×D K ×C×N, using depth-wise separable convolution data, it first undergoes channel-by-channel convolution, and the required number of parameters is D K ×D K ×C×1, and then through point-by-point convolution, the required parameters are 1×1×C×N, and the sum of the two parts of parameters is C×(D K ×D K +N) is smaller than the D of ordinary convolution K ×D K ×C×N, which can achieve the effect of reducing the number of parameters.

[0112] S3: Design a loss function and use the dataset to train and test the constructed image restoration model.

[0113] The designed loss function is expressed as:

[0114] L=λ r L re +λ p L perc +λ s L style +λ a L adv

[0115] In the formula, L represents the loss function, L re represents the reconstruction loss, L perc represents the perceptual loss, L style represents the style loss, L adv Represents adversarial loss; and sets the weight λ of each loss r =1,λ p =1,λ s =250 and λ a =0.1;

[0116] Reconstruction loss, output image I out Compared with the real image I g The L1 distance between them, which is a Euclidean measure of the difference between the generated image and the target image, is defined as:

[0117] L re =||I out -I g ||1

[0118] In the formula, I out represents the input image, I g represents the real image, ||·||1 represents the calculation of Euclidean distance;

[0119] Perceptual loss is to compare the difference between the generated image and the target image through the intermediate layer feature map of the pre-trained convolutional neural network. The smaller the perceptual loss value, the closer the two images are. It is defined as:

[0120]

[0121] In the formula, φ i (I out ) and φ i (I g ) represent the output image I out and the real image I g Feature representations extracted in the middle layers of a pre-trained network (such as a VGG network); Indicates expected value; N i Indicates the number of feature layers of the i-th layer;

[0122] Style loss, for both images, the Gram matrix is ​​calculated at each layer of the loss network, and the size of the feature_map generated is C j ×H j ×W j , then calculate the Euclidean distance between the corresponding layers, and finally add the Euclidean distances of different layers to get the final style loss, which is defined as:

[0123]

[0124] In the formula, Indicates that at layer j the corresponding activation map Φ j Constructed C j ×C i Gram matrix; represents the expected value of the jth layer;

[0125] Adversarial loss optimizes the model through the adversarial process between the generator and the discriminator. The goal of the generator is to generate images that are indistinguishable from real images. The loss of the generator is usually based on the output of the discriminator, encouraging the images generated by the generator to be visually indistinguishable from real images, and is defined as:

[0126]

[0127] In the formula, D(I g ) represents the discriminator's response to the real image I g is a true probability prediction; D(I out ) represents the discriminator's output image I out is a true probabilistic prediction; is a sample from the true data distribution, represents a noise vector sampled from the prior distribution.

[0128] The constructed image restoration model is trained and tested using the training set and test set in S1. During training, the batch size of the training set images is set to 8, the number of training iterations is 300,000, and the learning rate is 0.001. The Adam optimizer is used to train the constructed image restoration model, and the weight parameters of the connections between the network layers are continuously adjusted. The test set is used to test the accuracy of the trained image restoration model, and better restoration effects are obtained under different masking degrees.

[0129] S4: Input the damaged calligraphy Chinese character image to be repaired into the trained image repair model to obtain a repaired image.

[0130] Example 1

[0131] This embodiment provides a method for repairing damaged calligraphy Chinese character images based on T-former improvement, such as Figure 1 As shown, the specific steps include:

[0132] S1: Collect and generate individual calligraphy Chinese character images of different styles, and pre-process each calligraphy Chinese character image as a dataset;

[0133] S2: Build image restoration model;

[0134] S3: Design a loss function and use the dataset to train and test the constructed image restoration model;

[0135] S4: Input the damaged calligraphy Chinese character image to be repaired into the trained image repair model to obtain a repaired image.

[0136] Example 2

[0137] Based on Example 1, S1 is specifically:

[0138] S1.1: Select at least one font that matches the calligraphy style, generate several corresponding single Chinese character images, and adjust the font size;

[0139] S1.2: Download the mask dataset for the image restoration task. Select 10% to 50% of the mask images with different degrees of damage according to the mask images close to the boundary and not close to the boundary to simulate various damage conditions of images in reality. Put several single Chinese character images into the same folder, randomly add two types of masks with different degrees of damage, and generate a dataset.

[0140] S1.3: The resolution of the images in the dataset is uniformly adjusted to 64*64, and the dataset is divided into a training set and a test set in a ratio of 8:2.

[0141] Example 3

[0142] On the basis of Example 2, the image restoration model constructed in S2 includes a 7×7 convolutional layer, a feature extraction network, an intermediate module, a feature recovery network and a 7×7 convolutional layer connected in sequence;

[0143] The feature extraction network consists of three transformer_High modules connected in sequence. The input of the first transformer_High module is the feature of the original image after being processed by a 7x7 convolutional layer. The input of the second and third transformer_High modules are the output of the previous transformer_High module after downsampling. In addition to being input to the next stage after downsampling, the feature maps output by each transformer_High module are also input to the module of the corresponding layer of the feature recovery network, and feature fusion is performed with the corresponding module in the feature recovery network using a jump connection method.

[0144] The feature recovery network consists of an original transformer module and two transformer_High modules connected in sequence; the input of the original transformer module is the output of the intermediate module, which is upsampled and fused with the feature map input by the third transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the second transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the second transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the third transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the first transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer;

[0145] The middle module is an original transformer module.

[0146] Example 4

[0147] On the basis of Example 3, the transformer_High module consists of three parts, including a first normalization layer, a receptive field enhancement module, a second normalization layer, a linear attention layer with a gating mechanism, a third normalization layer, and an improved feedforward neural network connected in sequence; and the output of the 7×7 convolution is spliced ​​with the output of the receptive field enhancement module, the output of the receptive field enhancement module is spliced ​​with the output of the linear attention layer with a gating mechanism, and the output of the linear attention layer with a gating mechanism is spliced ​​with the output of the improved feedforward neural network layer through residual connections;

[0148] The processing of the transformer_High module is as follows:

[0149] First, the input feature map X is normalized and used as the input of the receptive field enhancement module to obtain the output result X p , X p It is concatenated with the input feature X using a residual connection; then, after normalization, it is used as the input of the linear attention layer with a gating mechanism to obtain the output result X M , X M With X p Then perform residual connection; finally, after normalization, it is used as the input of the improved feedforward neural network to obtain the output result X F , X F With X M After splicing using the residual connection method, the final output result is obtained.

[0150] The receptive field enhancement module includes a 1×1 convolution, a 3×3 dilated convolution with dilation rates of 1, 2, and 3 connected to the 1×1 convolution, a Gelu activation function, a 3×3 convolution, and a Gelu activation function;

[0151] The processing process of the receptive field enhancement module is as follows:

[0152] First, the feature X of the input receptive field enhancement module is reduced to 1 / 2 of its original dimension by 1×1 convolution, and then X is processed by 3×3 dilated convolution with dilation=1, dilation=2, and dilation=3 respectively to obtain feature maps X1, X2, and X3. At this time, the number of channels of X1, X2, and X3 changes from 1 / 2channels to 1 / 3channels; then, X1, X2, and X3 are concatenated in the channel direction; finally, the enhanced features are output after being processed by Gelu activation function, 3×3 convolution, and Gelu activation function in sequence.

[0153] The feature map X is processed by 1×1 convolution and then the input hole convolution is processed. The output is:

[0154]

[0155] Where D(x, y) represents the value of the output feature map at position (x, y); I(x, y) represents the value of the input feature map at position (x, y); W(m, l) represents the weight value of the convolution kernel at position (m, l); M represents the size of the convolution kernel; d represents the hole rate, which controls the expansion of the receptive field during the convolution operation; m, l are the offsets of the convolution kernel relative to the center point, and the value range is x+m·d, y+l·d represent the sampling points selected by the dilated convolution in the input feature map, and the interval is determined by d.

[0156] For the input feature Y and the linear attention operator The output O of the attention with gating mechanism can be written as the Hadamard product of operators A and G, expressed as:

[0157] O=A⊙G

[0158]

[0159] G=φ g (W g Y)

[0160] Where W g is a learnable parameter, φ g is the activation function, ⊙ represents the Hadamard product;

[0161] The attention operator first performs three different transformations on the input features Y as new inputs and reshapes them into two-dimensional matrices to obtain the corresponding three embeddings: query Q, key K, and value V. Then the corresponding attention result O can be obtained as follows:

[0162]

[0163] In the formula, represents the transpose of K, C represents the number of channels; after the Softmax normalized dot product attention, we get:

[0164]

[0165] Where q, k, and v correspond to the query Q, key K, and value V respectively; N = H*W, which represents the product of the image width and height:

[0166] By approximating the exponential function through Taylor expansion to linearize the attention, the Taylor formula of the exponential function that constitutes the Softmax operator is:

[0167] exp(x) = e x ≈1+x

[0168]

[0169] Example 5

[0170] On the basis of Example 4, the improved feedforward neural network includes a 1×1 point-by-point convolution and two parallel branches respectively connected to the 1×1 point-by-point convolution, one branch is a 3×3 depth-separable convolution, an activation function, a 3×3 depth-separable convolution, and an activation function connected in sequence; the other branch is a 5×5 depth-separable convolution, an activation function, a 5×5 depth-separable convolution, and an activation function connected in sequence; the two parallel branches are connected to a 1×1 convolution after a splicing operation; and a splicing operation is connected between the two parallel branches;

[0171] The processing of the improved feedforward neural network is as follows:

[0172] First, the input features are subjected to 1×1 point-by-point convolution to adjust the number of channels of the output features to twice the number of input channels while keeping the spatial size unchanged; then, they are processed by parallel 3×3 and 5×5 depthwise separable convolutions and activated by the activation function, and then split according to the channel direction to obtain feature maps X1_3, X2_3 and X1_5, X2_5 with the same number of channels; X1_3 and X1_5 are spliced ​​along the channel and input into 3×3 depthwise separable convolution, and X2_3 and X2_5 are spliced ​​along the channel and input into 5×5 depthwise separable convolution, and then spliced ​​along the channel after the activation function; finally, the output features are obtained by adjusting the number of channels through 1×1 convolution.

[0173] Example 6

[0174] Based on Example 5, the loss function designed in S3 is expressed as:

[0175] L=λ r L re +λ p L perc +λ s L style +λ a L adv

[0176] In the formula, L represents the loss function, L re represents the reconstruction loss, L perc represents the perceptual loss, L style represents the style loss, L adv Represents adversarial loss; and sets the weight λ of each loss r =1,λ p =1,λ s =250 and λ a =0.1;

[0177] The reconstruction loss is defined as:

[0178] L re =||I out -I g ||1

[0179] In the formula, I out represents the input image, I g represents the real image, ||·||1 represents the calculation of Euclidean distance;

[0180] Perceptual loss, defined as:

[0181]

[0182] In the formula, φ i (I out ) and φ i (I g ) represent the output image I out and the real image I g Feature representation extracted in the intermediate layers of the pre-trained network; Indicates expected value; N i Indicates the number of feature layers of the i-th layer;

[0183] Style loss, defined as:

[0184]

[0185] In the formula, Indicates that at layer j the corresponding activation map Φ j Constructed C j ×C j Gram matrix; represents the expected value of the jth layer;

[0186] Adversarial loss, defined as:

[0187]

[0188] In the formula, D(I g ) represents the discriminator's response to the real image I g is a true probability prediction; D(I out ) represents the discriminator's output image I out is a true probabilistic prediction; is a sample from the true data distribution, represents a noise vector sampled from the prior distribution.

[0189] Using the above loss function, the training set and test set of Example 2 are used to train and test the image restoration model of Example 5. During training, the batch size of the training set images is set to 8, the number of training iterations is 300000, the learning rate is 0.001, and the Adam optimizer is used to train the constructed image restoration model, and the weight parameters of the connections between the network layers are continuously adjusted; the accuracy of the trained image restoration model is tested using the test set.

Claims

1. A method for repairing damaged calligraphy Chinese character images based on T-former improvement, characterized in that: The specific steps include: S1: Collect and generate individual calligraphy Chinese character images of different styles, and pre-process each calligraphy Chinese character image as a dataset; S2: Build image restoration model; S3: Design a loss function and use the dataset to train and test the constructed image restoration model; S4: Input the damaged calligraphy Chinese character image to be repaired into the trained image repair model to obtain a repaired image.

2. The method for repairing damaged calligraphy Chinese character images based on the improved T-former according to claim 1 is characterized in that: S1 is specifically: S1.1: Select at least one font that matches the calligraphy style, generate several corresponding single Chinese character images, and adjust the font size; S1.2: Download the mask dataset for the image restoration task. Select 10% to 50% of the mask images with different degrees of damage according to the mask images close to the boundary and not close to the boundary to simulate various damage conditions of images in reality. Put several single Chinese character images into the same folder, randomly add two types of masks with different degrees of damage, and generate a dataset. S1.3: The resolution of the images in the dataset is uniformly adjusted to 64*64, and the dataset is divided into a training set and a test set in a ratio of 8:

2.

3. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 1 is characterized in that: The image restoration model constructed in S2 includes a 7×7 convolutional layer, a feature extraction network, an intermediate module, a feature recovery network, and a 7×7 convolutional layer connected in sequence; The feature extraction network is composed of three transformer_High modules connected in sequence; the input of the first transformer_High module is the feature of the original image after being processed by a 7x7 convolution layer, and the input of the second and third transformer_High modules are the results of the output of the previous transformer_High module after downsampling. In addition to being input to the next stage through downsampling, the feature maps output by each transformer_High module are also input to the module of the corresponding layer of the feature recovery network, and feature fusion is performed with the corresponding module in the feature recovery network using a jump connection method; The feature recovery network is composed of an original transformer module and two transformer_High modules connected in sequence; the input of the original transformer module is the output of the intermediate module, which is upsampled and fused with the feature map input by the third transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the second transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the second transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; the input of the third transformer_High module is the output of the previous layer, which is upsampled and fused with the feature map input by the first transformer_High module of the feature extraction network, and then processed by a 1×1 convolution layer; The intermediate module is an original transformer module.

4. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 3 is characterized in that: The transformer_High module consists of three parts, including a first normalization layer, a receptive field enhancement module, a second normalization layer, a linear attention layer with a gating mechanism, a third normalization layer, and an improved feedforward neural network connected in sequence; The output of the 7×7 convolution is concatenated with the output of the receptive field enhancement module, the output of the receptive field enhancement module is concatenated with the output of the linear attention layer with a gating mechanism, and the output of the linear attention layer with a gating mechanism is concatenated with the output of the improved feedforward neural network layer through residual connections. The processing of the transformer_High module is as follows: First, the input feature map X is normalized and used as the input of the receptive field enhancement module to obtain the output result X p , X p It is concatenated with the input feature X using a residual connection; then, after normalization, it is used as the input of the linear attention layer with a gating mechanism to obtain the output result X M , X M With X p Then perform residual connection; finally, after normalization, it is used as the input of the improved feedforward neural network to obtain the output result X F , X F With X M After splicing using the residual connection method, the final output result is obtained.

5. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 4 is characterized in that: The receptive field enhancement module includes a 1×1 convolution, a 3×3 dilated convolution with dilation rates of 1, 2, and 3 respectively connected to the 1×1 convolution, a Gelu activation function, a 3×3 convolution, and a Gelu activation function; The processing process of the receptive field enhancement module is as follows: First, the feature X of the input receptive field enhancement module is reduced to 1 / 2 of its original dimension by 1×1 convolution, and then X is processed by 3×3 dilated convolution with dilation=1, dilation=2, and dilation=3 respectively to obtain feature maps X1, X2, and X3. At this time, the number of channels of X1, X2, and X3 changes from 1 / 2channels to 1 / 3channels; then, X1, X2, and X3 are concatenated in the channel direction; finally, the enhanced features are output after being processed by Gelu activation function, 3×3 convolution, and Gelu activation function in sequence.

6. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 5 is characterized in that: The feature map X is processed by 1×1 convolution and then the input hole convolution is processed. The output is: Where D(x, y) represents the value of the output feature map at position (x, y); I(x, y) represents the value of the input feature map at position (x, y); W(m, l) represents the weight value of the convolution kernel at position (m, l); M represents the size of the convolution kernel; d represents the dilation rate, which controls the expansion of the receptive field during the convolution operation; m, l are the offsets of the convolution kernel relative to the center point, and their value range is x+m·d, y+l·d represent the sampling points selected by the dilated convolution in the input feature map, and the interval is determined by d.

7. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 4 is characterized in that: For input feature Y and linear attention operator The output O of the attention with gating mechanism can be written as the Hadamard product of operators A and G, expressed as: O=A⊙G G=φ g (IN g Y) Where W g is a learnable parameter, φ g is the activation function, ⊙ represents the Hadamard product; The attention operator first performs three different transformations on the input features Y as new inputs and reshapes them into two-dimensional matrices to obtain the corresponding three embeddings: query Q, key K, and value V. Then the corresponding attention result O can be obtained as follows: O=[o1,o2,...,o3] T In the formula, K T represents the transpose of K, C represents the number of channels; after the Softmax normalized dot product attention, we get: Where q, k, v correspond to query Q, key K, and value V respectively; N = H*W, which represents the product of image width and height; By approximating the exponential function through Taylor expansion to linearize the attention, the Taylor formula of the exponential function that constitutes the Soffmax operator is: exp(x)=e x ≈1+x 8. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 4 is characterized in that: The improved feedforward neural network comprises a 1×1 point-by-point convolution and two parallel branches respectively connected to the 1×1 point-by-point convolution, one branch is a 3×3 depth-separable convolution, an activation function, a 3×3 depth-separable convolution, and an activation function connected in sequence; the other branch is a 5×5 depth-separable convolution, an activation function, a 5×5 depth-separable convolution, and an activation function connected in sequence; the two parallel branches are connected to a 1×1 convolution after a splicing operation; and a splicing operation is performed between the two parallel branches; The processing process of the improved feedforward neural network is as follows: First, the input features are subjected to 1×1 point-by-point convolution to adjust the number of channels of the output features to twice the number of input channels while keeping the spatial size unchanged; then, they are processed by parallel 3×3 and 5×5 depthwise separable convolutions and activated by the activation function, and then split according to the channel direction to obtain feature maps X1_3, X2_3 and X1_5, X2_5 with the same number of channels; X1_3 and X1_5 are spliced ​​along the channel and input into 3×3 depthwise separable convolution, and X2_3 and X2_5 are spliced ​​along the channel and input into 5×5 depthwise separable convolution, and then spliced ​​along the channel after the activation function; finally, the output features are obtained by adjusting the number of channels through 1×1 convolution.

9. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 2 is characterized in that: The loss function designed in S3 is expressed as: L=λ r L re +λ p L perc +λ s L style +λ a L adv In the formula, L represents the loss function, L re represents the reconstruction loss, L perc represents the perceptual loss, L style represents the style loss, L adv Represents adversarial loss; and sets the weight λ of each loss r =1,λ p =1,λ s =250 and λ a =0.1; The reconstruction loss is defined as: L re =||I out -I g ||1 In the formula, I out represents the input image, I g represents the real image, ||·||1 represents the calculation of Euclidean distance; The perceptual loss is defined as: In the formula, φ i (I out ) and φ i (I g ) represent the output image I out and the real image I g Feature representation extracted in the intermediate layers of the pre-trained network; Indicates expected value; N i Indicates the number of feature layers of the i-th layer; The style loss is defined as: In the formula, Indicates that at layer j the corresponding activation map Φ j Constructed C j ×C j Gram matrix; represents the expected value of the jth layer; The adversarial loss is defined as: In the formula, D(I g ) represents the discriminator's response to the real image I g is a true probability prediction; D(I out ) represents the discriminator's output image I out is a true probabilistic prediction; is a sample from the true data distribution, represents a noise vector sampled from the prior distribution.

10. The method for repairing damaged calligraphy Chinese character images based on T-former improvement according to claim 9, characterized in that: The constructed image restoration model is trained and tested using the dataset in S3. During training, the batch size of the training set images is set to 8, the number of training iterations is 300,000, the learning rate is 0.001, and the Adam optimizer is used to train the constructed image restoration model, and the weight parameters of the connections between the network layers are continuously adjusted; Use the test set to test the accuracy of the trained image restoration model.

Images