A digital image restoration method based on structural information embedding and attention mechanism

Abstract

The invention discloses a digital image restoration method based on structural information embedding and attention mechanism, comprising the following steps: firstly, resizing and data enhancement are performed on a public data set image, and structural information is extracted by using an RTV method; then, an image restoration network based on structural information embedding and attention mechanism is constructed, and a generator sharing structural information and image features is used to simultaneously perform structural restoration and embedding-guided image feature reconstruction; a hinge-form structural information discriminator with spectral normalization is used to distinguish the authenticity of the restored smooth image; a joint loss constraint model is used to optimize the direction; and the restored task and effect evaluation of the damaged image can be performed by using the trained model weight parameters and the constructed restoration network; the invention can better characterize the structural information, so that the restored image has a more reasonable structure and a more realistic visual effect, and the subjective and objective test evaluation results and comparative experiments have proved the effectiveness of the method of the invention.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image restoration based on deep learning, and in particular to a digital image restoration method based on structural information embedding and attention mechanism. Background Art

[0002] Image restoration technology infers the content of the missing area based on the known information of the damaged image, and makes the restored image have a reasonable global semantic structure, fine texture details, and natural fusion of the missing area boundary. The comprehensive effect meets the visual requirements of the human eye. This technology has a wide range of application value in image editing, cultural film and television special effects production, image cultural relics protection, and military public security. Benefiting from the ability of neural networks to learn nonlinear complex mapping relationships between samples from a large number of training data sets and the powerful capture and representation capabilities of convolutional neural networks for image features, it can process complex semantic images. In recent years, image restoration methods based on deep learning have received increasing attention.

[0003] In order to generate a reasonable semantic structure, many methods try to use information such as rough repair images, semantic segmentation maps, edges, foreground contours or shapes as a priori guidance for the image repair process. Reference 1 (Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning [J]. arXiv, 2019, arXiv: 1901.00212) proposed using the Canny edge detection algorithm to obtain missing edges, first perform edge repair, and then input the repaired edges together with the damaged image and mask into the repair network to estimate the content of the missing area. This method proves that using structural information as guiding information can improve the quality of repair, but the edge detection process will cause a lot of meaningful information to be lost, resulting in a lack of vividness in the generated content.

[0004] In order to solve the problem that convolutional neural networks can only extract local features and cannot effectively utilize long-distance spatial feature information, many methods use attention mechanisms to solve the long-distance dependency problem. Reference 2 (Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention; in proceedings of the 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, [C]. IEEE: Salt Lake City, UT, USA, 2018.) proposed using contextual attention to calculate the similarity score between background area pixel blocks and foreground area pixel blocks to borrow background area features to reconstruct the missing area content. Reference 3 (Liao L, Xiao E, Wang Z, et al. Image inpainting guided by coherence priors of semantics and textures; in proceedings of the 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition [C]. IEEE: Nashville, TN, USA, 2021.) proposed semantic-level attention to perform attention exploration on non-local semantic coherence in different semantic segmentation maps to solve the problem of mixed semantic textures. These methods prove that the attention mechanism can effectively capture global information and generate finer texture details, but the contextual attention mechanism at the image block level will lead to invalid information extraction and similarity calculation when dealing with irregular mask restoration.

[0005] Current image restoration tasks tend to deal with complex semantic image restoration problems with irregular region loss. These restoration processes have problems such as invalid feature extraction and training ambiguity, which cause artificial artifacts such as color difference, blur and obvious boundaries in the restored images. Although some current methods use partial convolution, edge information and attention mechanisms to improve the restoration effect and generate interpretable structures and fine textures, there are also problems such as insufficient structural information of edge representation, unreasonable premise influence of sequence model architecture and invalid feature extraction. Summary of the invention

[0006] In order to solve the problems existing in the above-mentioned image restoration methods, the present invention proposes a digital image restoration method based on structural information embedding and attention mechanism. The method adopts an edge-smoothed image that removes high-frequency textures but retains clear edges and low-frequency structures to characterize the global structure of the image, and adopts an embedded architecture to effectively utilize structural information, and uses pixel-level contextual attention to solve the problem of invalid feature extraction in image restoration when irregular areas are missing.

[0007] A digital image restoration method based on structural information embedding and attention mechanism specifically includes the following steps:

[0008] S1. Obtain an experimental data set and perform longitudinal average segmentation on images in the image data set that are larger than the target size so that the preprocessed images have the same size, wherein the experimental data set includes an image data set and a mask data set; the image data set and the mask data set each include a training set and a test set, respectively.

[0009] The target size in this step can be determined as needed, for example, 256*256 is selected, and the vertical average segmentation includes left-right average segmentation preprocessing for images that are twice larger than 256*256, and left-center-right average segmentation preprocessing for images that are three times larger than 256*256, as long as the images after segmentation have the same size.

[0010] S2. The RTV smoothing method is used to extract the edge-preserving smoothed image as the structural information of the preprocessed image. The objective function of the RTV smoothing method is:

[0011]

[0012] in,

[0013]

[0014]

[0015]

[0016] In the formula, S is the output structure diagram, p is the image pixel, I is the input image, λ p is the weight, D x (p), D y (p) are the pixel-level window total variation measures of the output structure graph in the x and y directions, respectively, and L x (p), L y (p) are the window-specific changes in the output structure graph in the x and y directions, ε is a constant, q is a pixel point in the rectangular area R(p) centered at p, and g p,q is the weight function constructed according to the spatial relationship, x p 、xq are the x- and y-components of pixels p and q respectively. p ,y q are the components of pixels p and q in the y direction, respectively, and σ is the spatial scale parameter of the control window;

[0017] S3. Construct a digital image restoration network with structural information embedding and attention mechanism, wherein the digital image restoration network includes a generator for structural information and image sharing and a structural information discriminator; the generator for structural information and image sharing can simultaneously restore structural information and update mask, and use the restored structural information and mask information to guide the generation of image features; the structural information discriminator is used to distinguish the authenticity of the restored smooth image.

[0018] The structural information discriminator is a hinged structure with spectral normalization. Specifically, 5 convolutional layers with spectral regularization layers and activation function layers are set from front to back, the first 4 groups use LeakyReLU activation, and the last layer uses sigmoid activation; the generator is based on the U-Net architecture and sets 2 structural information repair and embedding coding layers, 6 partial convolution coding layers, a transposed convolution layer, 5 partial deconvolution decoding layers, a pixel-level attention layer, 1 partial deconvolution decoding layer, 2 structural information repair and embedding decoding layers, a bottleneck layer and a convolution layer in sequence from front to back; the structural information repair and embedding coding layer includes a structure generation layer, a partial convolution layer, a batch normalization layer, and a ReLU activation function layer from front to back, wherein the structure generation layer is composed of a partial convolution layer, a bottleneck layer and a 1*1 convolution layer; the 6 partial convolution coding layers are composed of a partial convolution layer, a ReLU activation layer, a batch normalization layer, and a 1*1 convolution layer. The six partial deconvolution decoding layers are composed of partial convolution layers, transposed convolution layers, leaky activation layers, and batch normalization layers, which upsample the complex features of the previous layer features and the jump-connected coding layer features in cascade in turn; the two structural information repair and embedding decoding layers are similar to the structural information repair and embedding coding layer structures, and are modified on the basis of them: the first one uses leaky activation and transposed convolution, and the second one removes the batch normalization layer and the activation function layer; the pixel-level attention layer reconstructs the missing part through the pixel-level context attention layer, and cascades the output features of the attention layer with the input features in the channel dimension, and then inputs the squeeze extraction module (SE module) to obtain the channel weights of the output features and input features of the aforementioned attention layer and perform adaptive feature fusion through the 1*1 convolution layer. The specific operations are as follows:

[0019] B1. First, from the input feature map F in The foreground missing area f x,y and background non-missing area b x,yExtract pixel points and use the regularized cosine similarity inner product to calculate the similarity score:

[0020]

[0021] B2. Use softmax operation to normalize the similarity to obtain the attention score:

[0022] Where λ is a constant;

[0023] B3. Perform attention score propagation with kernel size k to enhance attention consistency:

[0024]

[0025] B4, then use the background non-missing area b extracted in step B1 x,y The pixels in are used as convolution kernels to perform deconvolution to reconstruct the missing parts to generate the attention feature map F att :

[0026]

[0027] B5, then input feature map F in and attention feature map F att In channel-level cascading, it is represented as <F in ,F att >, then input the SE module to obtain different channel weights, and finally use 1*1 convolution adaptive fusion features:

[0028] F out =f conv (f SE ( <F in ,F att >)), where the height and width remain unchanged after performing 1*1 convolution, and the number of channels is halved.

[0029] The structural information repair and embedding coding layer is characterized in that the number of input channels of the structure generation layer of the first structural information repair and embedding coding layer is 6, and the output is 3, and the number of input channels of the partial convolution layer of the first structural information repair and embedding coding layer is 6, and the output is 64; the number of input channels of the structure generation layer of the second structural information repair and embedding coding layer is 67, and the output is 3, and the number of input channels of the partial convolution layer of the second structural information repair and embedding coding layer is 67, and the output is 128; the number of input channels of the structure generation layer of the first structural information repair and embedding decoding layer is 195, and the output is 3, and the number of input channels of the partial convolution layer of the first structural information repair and embedding decoding layer is 195, and the output is 64; the number of input channels of the structure generation layer of the second structural information repair and embedding decoding layer is 70, and the output is 3, and the number of input channels of the partial convolution layer of the second structural information repair and embedding decoding layer is 70, and the output is 64.

[0030] S4, input the images of the training set in the mask data set in step S1, the images of the training set in the preprocessed image data set, and the edge-retained smoothed images extracted in S2 into the digital image restoration network training constructed in step S3; set the optimization strategy and related hyperparameters to train the network and adopt a joint loss constraint network optimization direction; optimize the network through cyclic iterative optimization so that the loss function is continuously reduced and converged, and save the network training weight parameters.

[0031] The combined loss includes the in-hole reconstruction loss L hole , effective area reconstruction loss L valid , Perceptual loss L perceptual , style loss L style , variational loss L tv , structural resistance loss The specific calculation is as follows:

[0032] 1) In-hole reconstruction loss L hole Measure the generated image I in the mask area out Compared with the real image I gt Pixel difference, using L1 distance metric:

[0033] L hole =E[||(I out -I gt )⊙(1-M)||1], where E is the mathematical expectation and M represents the mask image;

[0034] 2) Effective area reconstruction loss L valid The L1 distance metric is used to measure the pixel difference between the generated image and the real image in the non-masked area:

[0035] L valid =E[||(I out -Igt )⊙M||1]

[0036] 3) Perceptual loss L perceptual Measure the distance between the feature map extracted by the pooling layer of the pre-trained VGG16 network and the real image generated by the image:

[0037]

[0038] in Represents the feature map of the generated image or the real image output by the i-th pooling layer of the VGG16 network, H i , W i , C i is the height, width, and channel of the feature map, and N is the number of pooling layers selected;

[0039] 4) Style loss L style Measuring the covariance distance between the feature maps extracted from the pooling layer of the pre-trained VGG16 network between the generated image and the real image:

[0040]

[0041] in represents the Gram matrix, and T represents the transposition of the matrix;

[0042] 5) Variational loss L tv To measure the difference between adjacent pixels of the restored image, use the L1 distance metric:

[0043] Where m x,y Represents the pixel at position (x, y);

[0044] 6) Structural resistance to loss Identify the edge-preserving smooth image S generated by using two structural information restoration and embedding the decoding layer restoration out The edge-preserving smoothed image S extracted from the real image gt The authenticity of each layer is determined by using the hinged adversarial loss to deal with the problem of gradient disappearance. Including the adversarial loss L for the discriminator D D and the adversarial loss L for the generator G G :

[0045]

[0046]

[0047] where p data 、p z S out and S gtThe probability distribution of , ReLU is the ReLU activation function;

[0048] 7) Joint loss L joint Calculated as:

[0049]

[0050] where λ hole , valid , perceptual , style , tv , adv is the loss term weight hyperparameter.

[0051] S5. Use the generator of the digital image restoration network constructed in step S3 and the network training weight parameters obtained in step S4 to construct a digital image restoration model based on structural information embedding and attention mechanism, and use the image restoration model to repair and complete the damaged image consisting of the test set in the image dataset and the test set in the mask dataset to obtain the repaired image and the completed smooth image.

[0052] Beneficial effects:

[0053] 1. The present invention proposes a digital image restoration method based on structural information embedding and attention mechanism. For various public data sets, the present invention first uses image cropping and data enhancement to process images, and then uses the RTV smoothing method to extract edge-preserved smooth images as structural information; the U-Net network is used as the basic network architecture, and the network is constructed by combining structural information recovery and embedding layer, pixel-level attention layer, and hinge form structure discriminator.

[0054] 2. The present invention adopts multiple joint losses to make the model converge better, and uses a smooth image that better represents the global structural information to guide image restoration, so that the restored image has a more reasonable structure; the use of pixel-level attention mechanism and discriminant network makes the generated restoration image more realistic and significantly improves the visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The overall diagram of the model structure used in the implementation of the present invention;

[0056] Figure 2 A schematic diagram of a flow chart of an embodiment of the present invention;

[0057] Figure 3 The restoration results of the restoration method designed by the present invention on damaged images with different mask ratios in Paris Street View;

[0058] Figure 4Figure 1 is a graph showing the restoration results of a damaged image in Paris Street View using the restoration method designed by the present invention and the method in document 1, wherein sub-images (a1)-(c1) are original images, sub-images (a2)-(c2) are damaged images to be restored, wherein white represents the area to be restored, sub-images (a3)-(c3) are restoration results of sub-images (a2)-(c2) using the method in document 1, and sub-images (a4)-(c4) are restoration results of sub-images (a2)-(c2) using the restoration method designed by the present invention. DETAILED DESCRIPTION

[0059] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, an embodiment of the present invention is further described in conjunction with the accompanying drawings. The embodiment is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the content of the present invention also fall within the scope of protection of the present invention.

[0060] A digital image restoration method based on structural information embedding and attention mechanism specifically includes the following steps:

[0061] S1. Obtain the experimental dataset: the image dataset uses the Paris Street View training set and test set, and the mask dataset uses the Irregular Mask Dataset; then the 936*537 images in the Paris Street View dataset are preprocessed by size segmentation, and each image is evenly segmented from left to right to obtain three images of the same size to expand the amount of training data; then all experimental data are adjusted to 256*256 (target size) and augmented during training; in the mask dataset, the mask M uses 0 to represent the missing area and 1 to represent the non-missing area.

[0062] S2. The RTV smoothing method is used to extract the edge-preserving smoothed image as the structural information of the preprocessed image. The objective function of the RTV smoothing method is:

[0063]

[0064] in,

[0065]

[0066]

[0067]

[0068] In the formula, S is the output structure diagram, p is the image pixel, I is the input image, λ p is the weight, D x(p), D y (p) are the pixel-level window total variation measures of the output structure graph in the x and y directions, respectively, and L x (p), L y (p) are the window-specific changes in the output structure graph in the x and y directions, ε is a constant, q is a pixel point in the rectangular area R(p) centered at p, and g p,q is the weight function constructed according to the spatial relationship, x p 、x q are the x- and y-components of pixels p and q respectively. p ,y q are the components of pixels p and q in the y direction, respectively, and σ is the spatial scale parameter of the control window;

[0069] S3, build a digital image restoration network with structural information embedding and attention mechanism, such as Figure 1 As shown, the digital image restoration network includes a generator for structural information and image sharing, a structural information discriminator, a structural extractor, and a VGG-16 network pre-trained on ImageNet.

[0070] The generator that shares the structural information and image can simultaneously realize the repair of structural information and the update of mask, and use the repaired structural information and mask information to guide the generation of image features; the structural information discriminator is SN-PatchGAN, which sets 5 convolutional layers with spectral regularization layers and activation function layers from front to back, the first 4 groups use LeakyReLU activation, and the last layer uses sigmoid activation; the generator is based on the U-Net architecture and sets 2 structural information repair and embedding coding layers, 6 partial convolution coding layers, a transposed convolution layer, 5 partial deconvolution decoding layers, a pixel-level attention layer, 1 partial deconvolution decoding layer, 2 structural information repair and embedding decoding layers, a bottleneck layer and a convolution layer from front to back; the structural information repair and embedding coding layer includes a structure generation layer, a partial convolution layer, a batch normalization layer, and a ReLU activation function layer from front to back, wherein the structure generation layer is composed of a partial convolution layer, a bottleneck layer and a 1*1 convolution layer; the 6 partial convolution coding layers are composed of a partial convolution layer, a bottleneck layer and a 1*1 convolution layer. The 6 partial deconvolution decoding layers are composed of partial convolution layers, transposed convolution layers, leaky activation layers, and batch normalization layers, which sequentially downsample the features, but the last 4 coding layers keep the number of channels unchanged at 512; the 6 partial deconvolution decoding layers are composed of partial convolution layers, transposed convolution layers, leaky activation layers, and batch normalization layers, which sequentially upsample the complex features of the previous layer features and the coding layer features of the jump connection; the 2 structural information repair and embedding decoding layers are similar in structure to the structural information repair and embedding coding layers, on which the first structural information repair and embedding decoding layer adopts leaky activation and transposed convolution, and the second structural information repair and embedding decoding layer removes the batch normalization layer and the activation function layer; the pixel-level attention layer reconstructs the missing part through the pixel-level context attention layer, and cascades the output features of the attention layer with the input features in the channel dimension, and then inputs the squeeze extraction module (SE module) to obtain the channel weights of the output features of the aforementioned attention layer and the input features, and performs adaptive feature fusion through a 1*1 convolution layer, for the input feature map F in , the specific operations are as follows:

[0071] B1, first from the foreground missing area f x,y and background non-missing area b x,y Extract pixel points and use the regularized cosine similarity inner product to calculate the similarity score:

[0072]

[0073] B2. Use softmax operation to normalize the similarity to obtain the attention score:

[0074] Where λ is a constant;

[0075] B3. Perform attention score propagation with kernel size k to enhance attention consistency:

[0076]

[0077] B4, then use the background non-missing area b extracted in step B1 x,y The pixels in are used as convolution kernels to perform deconvolution to reconstruct the missing parts to generate the attention feature map F att :

[0078]

[0079] B5, then input feature map F in and attention feature map F att In channel-level cascading, it is represented as <F in ,F att >, then input the SE module to obtain different channel weights, and finally use 1*1 convolution adaptive fusion features:

[0080] F out =f conv (f SE ( <F in ,F att >)), where the height and width remain unchanged after performing 1*1 convolution, and the number of channels is halved.

[0081] The structural information repair and embedding coding layer is characterized in that the number of input channels of the structure generation layer of the first structural information repair and embedding coding layer is 6, and the output is 3, and the number of input channels of the partial convolution layer of the first structural information repair and embedding coding layer is 6, and the output is 64; the number of input channels of the structure generation layer of the second structural information repair and embedding coding layer is 67, and the output is 3, and the number of input channels of the partial convolution layer of the second structural information repair and embedding coding layer is 67, and the output is 128; the number of input channels of the structure generation layer of the first structural information repair and embedding decoding layer is 195, and the output is 3, and the number of input channels of the partial convolution layer of the first structural information repair and embedding decoding layer is 195, and the output is 64; the number of input channels of the structure generation layer of the second structural information repair and embedding decoding layer is 70, and the output is 3, and the number of input channels of the partial convolution layer of the second structural information repair and embedding decoding layer is 70, and the output is 64.

[0082] S4, input the training set images in the mask dataset in step S1, the training set images in the preprocessed image dataset, and the edge-preserving smoothed images extracted in S2 into the digital image restoration network training constructed in step S3. We conducted the experiment on a machine configured with an Nvidia GTX 2080Ti GPU. The framework is based on Pytorch implementation. The parameters are set to use the Adam optimization strategy, and the learning rate of the generator training is 2×10-4 , the learning rate of the discriminator training is 2×10 -5 , batch_size is 6; relevant hyperparameters are set to train the network and the joint loss is used to constrain the network optimization direction; the loss function is continuously reduced and converged by performing cyclic iterative optimization on the network, and the network training weight parameters are saved.

[0083] The combined loss includes the in-hole reconstruction loss L hole , effective area reconstruction loss L valid , Perceptual loss L perceptual , style loss L style , variational loss L tv , structural resistance loss The specific calculation is as follows:

[0084] 1) In-hole reconstruction loss L hole Measure the generated image in the mask area (the image output by the network after the damaged image is repaired by the repair model) I out Compared with the real image I gt Pixel difference, using L1 distance metric:

[0085] L hole =E[||(I out -I gt )⊙(1-M)||1], where E is the mathematical expectation and M represents the mask image;

[0086] 2) Effective area reconstruction loss L valid The L1 distance metric is used to measure the pixel difference between the generated image and the real image in the non-masked area:

[0087] L valid =E[||(I out -I gt )⊙M||1]

[0088] 3) Perceptual loss L perceptual Measure the distance between the feature map extracted by the pooling layer of the pre-trained VGG16 network and the real image generated by the image:

[0089]

[0090] in Represents the feature map of the generated image or the real image output by the i-th pooling layer of the VGG16 network, H i , W i , C i is the height, width, and channel of the feature map, and N is the number of pooling layers selected;

[0091] 4) Style loss L styleMeasuring the covariance distance between the feature maps extracted from the pooling layer of the pre-trained VGG16 network between the generated image and the real image:

[0092]

[0093] in represents the Gram matrix, and T represents the transposition of the matrix;

[0094] 5) Variational loss L tv To measure the difference between adjacent pixels of the restored image, use the L1 distance metric:

[0095] Where m x,y Represents the pixel at position (x, y);

[0096] 6) Structural resistance to loss Identify the edge-preserving smooth image S generated by using two structural information restoration and embedding the decoding layer restoration out The edge-preserving smoothed image S extracted from the real image gt The authenticity of each layer is determined by using the hinged adversarial loss to deal with the problem of gradient disappearance. Including the adversarial loss L for the discriminator D D and the adversarial loss L for the generator G G :

[0097]

[0098]

[0099] where p data 、p z S out and S gt The probability distribution of , ReLU is the ReLU activation function;

[0100] 7) Joint loss L joint Calculated as:

[0101]

[0102] The loss term weight hyperparameter is set before each loss. In the embodiment, λ hole =50,λ valid =50,λ perceptual =0.05,λ style =150,λ tv =0.1,λ adv =0.1.

[0103] S5, constructing a digital image restoration model based on structural information embedding and attention mechanism based on the generator of the digital image restoration network constructed in step S3 and the network training weight parameters obtained in step S4, and using the image restoration model to repair and complete the damaged image consisting of the test set in the image dataset and the test set in the mask dataset, to obtain a restored image and a smooth image with edge preservation;

[0104] In order to objectively evaluate the restoration effect of the present invention, this embodiment tested the restoration effect of the irregular mask at different mask ratios by the restoration method designed by the present invention and the method of document 1 on Paris Street View and used multiple objective evaluation indicators such as PSNR, SSIM, MAE, and LPIPS for evaluation. PSNR and MAE reflect the difference between the real image and the restored image based on pixel statistical information, SSIM imitates the human visual system to extract high-order structural information to measure image similarity, and LPIPS uses deep visual representation to imitate human perception evaluation. The larger the PSNR and SSIM, the better the image restoration quality, and the smaller the MAE and LPIPS, the better the image restoration quality. The evaluation comparison results are shown in Table 1. It can be seen from the comparison of the experimental results that the restoration method designed by the present invention is superior to the method of document 1 in various evaluation indicators at various mask ratios.

[0105] The subjective repair effect of the present invention is as follows Figure 3 As shown, the first row shows the masked images with different mask ratios, the mask ratios are [0.01, 0.1), [0.1, 0.2), [0.2, 0.3), [0.3, 0.4), [0.4, 0.5), [0.5, 0.6], the second row shows the restored images, and the last image in the second row is the original image. Figure 3 It can be seen from the comparison between the repaired images and the original images with various mask ratios that the repair method designed in the present invention can handle the repair of damaged images with various mask ratios, and the repaired images can ensure the continuity of the overall semantic structure, the images are real and natural, and meet the visual requirements of the human eye.

[0106] Figure 4 The repair results of the damaged image in Paris Street View by the repair method designed by the present invention and the method in Document 1, wherein sub-images (a1)-(c1) are the original images, sub-images (a2)-(c2) are the damaged images to be repaired, and the white represents the area to be repaired, wherein sub-images (a3)-(c3) are the results of the repair of sub-images (a2)-(c2) by the method in Document 1, and sub-images (a4)-(c4) are the results of the repair of sub-images (a2)-(c2) by the repair method designed by the present invention. Comparison Figure 4 The repair results (a3) ​​and (a4) show that the repair method designed by the present invention can better achieve a continuous and consistent semantic structure repair effect than the method in reference 1. Figure 4 It can be seen from the repair results (a3) ​​and (a4) that the repair method designed in the present invention can achieve a finer-grained texture effect than the method in reference 1.

[0107] According to subjective and objective evaluations and comparative experiments, it can be seen that the method of the present invention can repair mask images of various proportions, and is superior to existing methods in ensuring the continuity and rationality of the overall structure, the authenticity and naturalness of the image, and the visual requirements of the human eye.

[0108] Table 1 Comparison of restoration evaluation with different mask ratios

[0109]

[0110] The above is a description of the relevant contents of the present invention, and those skilled in the art will be able to implement the present invention based on these descriptions. Based on the above contents of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

Claims

1. A digital image restoration method based on structural information embedding and attention mechanism, characterized in that: The following steps are involved: S1. Acquire an experimental data set including an image data set and a mask data set and perform longitudinal average segmentation on images larger than a target size in the image data set so that the preprocessed images have the same size, wherein the image data set and the mask data set each include a training set and a test set; S2. The RTV smoothing method is used to extract the edge-preserving smoothed image as the structural information of the preprocessed image. The objective function of the RTV smoothing method is: in, In the formula, S is the output structure diagram, p is the image pixel, I is the input image, λ p is the weight, D x (p), D y (p) are the pixel-level window total variation measures of the output structure graph in the x and y directions, respectively, and L x (p), L y (p) are the window-specific changes in the output structure graph in the x and y directions, ε is a constant, q is a pixel point in the rectangular area R(p) centered at p, and g p,q is the weight function constructed according to the spatial relationship, x p 、x q are the x- and y-components of pixels p and q respectively. p ,y q are the components of pixels p and q in the y direction, respectively, and σ is the spatial scale parameter of the control window; S3, constructing a digital image restoration network with structural information embedding and attention mechanism, the digital image restoration network includes a generator for structural information and image sharing and a structural information discriminator; the generator for structural information and image sharing is used to simultaneously restore structural information and update mask, and use the restored structural information and mask information to guide the generation of image features; the structural information discriminator is used to distinguish the authenticity of the restored smooth image; S4, inputting the images of the training set in the mask data set in step S1, the images of the training set in the preprocessed image data set, and the edge-retained smoothed images extracted by S2 into the digital image restoration network training constructed in step S3; setting the optimization strategy and related hyperparameters to train the network and adopting the joint loss constraint network optimization direction; optimizing the network by cyclic iterative optimization so that the loss function is continuously reduced and converged, and saving the network training weight parameters; S5, using the generator of the digital image restoration network constructed in step S3 and the network training weight parameters obtained in step S4 to construct a digital image restoration model based on structural information embedding and attention mechanism, and using the image restoration model to repair and complete the damaged image consisting of the test set in the image dataset and the test set in the mask dataset, to obtain a restored image and a completed smooth image; The structural information discriminator is SN-PatchGAN, which has five convolutional layers with spectral regularization layers and activation function layers from front to back. The first four convolutional layers use LeakyReLU activation, and the last convolutional layer uses sigmoid activation. The generator uses the U-Net architecture as the basic model and sets up 2 structural information repair and embedding coding layers, 6 partial convolution coding layers, a transposed convolution layer, 5 partial deconvolution decoding layers, a pixel-level attention layer, 1 partial deconvolution decoding layer, 2 structural information repair and embedding decoding layers, a bottleneck layer and a convolution layer in sequence from front to back; The structural information repair and embedding coding layer includes a structure generation layer, a partial convolution layer, a batch normalization layer, and a ReLU activation function layer from front to back, wherein the structure generation layer is composed of a partial convolution layer, a bottleneck layer, and a 1*1 convolution layer; The six partial convolutional coding layers are composed of a partial convolutional layer, a ReLU activation layer, and a batch normalization layer, which downsample the features in turn, but the last four partial convolutional coding layers keep the number of channels unchanged at 512; The six partial deconvolution decoding layers are composed of a partial convolution layer, a transposed convolution layer, a leaky activation layer, and a batch normalization layer, which sequentially upsample the complex features of the previous layer features and the skip-connected encoding layer features concatenated together; The first of the two structural information repair and embedding decoding layers includes, from front to back, a structure generation layer, a partial convolution layer, a transposed convolution layer, a batch normalization layer, and a leaky activation function layer, wherein the structure generation layer is composed of a partial convolution layer, a bottleneck layer, and a 1*1 convolution layer; the second of the two structural information repair and embedding decoding layers includes, from front to back, a structure generation layer and a partial convolution layer, wherein the structure generation layer is composed of a partial convolution layer, a bottleneck layer, and a 1*1 convolution layer; The pixel-level attention layer reconstructs the missing part through the pixel-level context attention layer, and concatenates the output features of the attention layer with the input features in the channel dimension, and then inputs the squeeze extraction module to obtain the channel weights of the output features of the attention layer and the input features, and performs adaptive feature fusion through a 1*1 convolution layer; The combined loss in step S4 includes the in-hole reconstruction loss L hole , effective area reconstruction loss L valid , Perceptual loss L perceptual , style loss L style , variational loss L tv , structural resistance loss The specific calculation is as follows: 1) In-hole reconstruction loss L hole Measure the generated image I in the mask area out Compared with the real image I gt Pixel difference, using L1 distance metric: L hole =E[(I out -I gt )⊙(1-M)1], where E is the mathematical expectation; M is the mask image; 2) Effective area reconstruction loss L valid The L1 distance metric is used to measure the pixel difference between the generated image and the real image in the non-masked area: THE valid =E[(I out -THE gt )⊙M1] 3) Perceptual loss L perceptual Measure the distance between the feature map extracted by the pooling layer of the pre-trained VGG16 network and the real image generated by the image: In the formula, Represents the feature map of the generated image or the real image output by the i-th pooling layer of the VGG16 network, H i , W i , C i is the height, width, and channel of the feature map, and N is the number of pooling layers selected; 4) Style loss L style Measuring the covariance distance between the feature maps extracted from the pooling layer of the pre-trained VGG16 network between the generated image and the real image: in represents the Gram matrix, T represents the transposition of the matrix; 5) Variational loss L tv To measure the difference between adjacent pixels of the restored image, use the L1 distance metric: Where m x,y Represents the pixel at position (x, y); 6) Structural resistance to loss Identify the edge-preserving smooth image S generated by using two structural information restoration and embedding the decoding layer restoration out The edge-preserving smoothed image S extracted from the real image gt The authenticity of each layer is determined by using the hinged adversarial loss to deal with the problem of gradient disappearance. Including the adversarial loss L for the discriminator D D and the adversarial loss L for the generator G G : where p data 、p z S out and S gt The probability distribution of , ReLU is the ReLU activation function; 7) Joint loss L joint for: In the formula, λ hole , valid , perceptual , style , tv , adv is the loss term weight hyperparameter.

2. According to claim 1, a digital image restoration method based on structural information embedding and attention mechanism is characterized in that: The specific operation of the pixel-level attention layer includes the following steps: B1. First, from the input feature map F in The foreground missing area f x,y and background non-missing area b x,y Extract pixel points and use the regularized cosine similarity inner product to calculate the similarity score of the corresponding pixel pairs: B2. Use softmax operation to normalize the similarity to obtain the attention score: Where λ is a constant; B3. Perform attention score propagation with kernel size k to enhance attention consistency: B4, then use the background non-missing area b extracted in step B1 x,y The pixels in are used as convolution kernels to perform deconvolution to reconstruct the missing parts to generate the attention feature map F att : B5, then input feature map F in and attention feature map F att In channel-level cascading, it is represented as <F in ,F att >, then input the squeeze extraction module to obtain different channel weights, and finally use 1*1 convolution adaptive fusion features, in which the height and width remain unchanged after executing 1*1 convolution, and the number of channels is halved.

3. According to claim 1, a digital image restoration method based on structural information embedding and attention mechanism is characterized in that: The number of input channels of the structure generation layer of the first structure information repair and embedding coding layer is 6, and the output is 3; the number of input channels of the partial convolution layer of the first structure information repair and embedding coding layer is 6, and the output is 64; the number of input channels of the structure generation layer of the second structure information repair and embedding coding layer is 67, and the output is 3; the number of input channels of the partial convolution layer of the second structure information repair and embedding coding layer is 67, and the output is 128; the number of input channels of the structure generation layer of the first structure information repair and embedding decoding layer is 195, and the output is 3; the number of input channels of the partial convolution layer of the first structure information repair and embedding decoding layer is 195, and the output is 64; the number of input channels of the structure generation layer of the second structure information repair and embedding decoding layer is 70, and the output is 3; the number of input channels of the partial convolution layer of the second structure information repair and embedding decoding layer is 70, and the output is 64.

Images