Image restoration method and electronic device based on texture and structure fusion prior

By integrating texture and structural features in the image repair model and combining spatial and linear attention, the artifact problem of image repair models in the prior art is solved when processing complex backgrounds and large-area missing, and the details of image repair are improved. Capability and processing efficiency.

CN118761936BActive Publication Date: 2024-11-29WEIFANG UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411237375.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-29
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

The existing image repair model is prone to artifacts when the processing background is complex and large-area areas are missing. Due to the limited size of the convolution kernel receptive field, it is difficult to effectively capture the long-distance dependence of image features, resulting in low processing efficiency. .

Method used

Using a priori image repair method based on texture and structure fusion priori, we train a model containing the detailed repair network and image content completion network, blend texture features and structural features into priori information, and combine spatial attention and linear attention to optimize Image repair effect. The specific steps include extracting texture features and Canny operators using Gabor filters, and weighting processing through the spatial attention module and linear attention module to generate detailed images and target images.

Benefits of technology

It significantly improves the detail restoration ability and processing efficiency of image repair, improves the accuracy and training efficiency of the model, can better capture the global and local features of the image, reduce artifacts, and improves the naturalness and consistency of the repair results. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761936B_ABST
    Figure CN118761936B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing technology, and specifically to an image restoration method and electronic device based on texture and structure fusion priors. In the present invention, the fused texture features and structural features are used as prior information to guide the restoration of damaged images, which plays a very important role in improving the image detail restoration effect and maintaining the rationality and consistency of the restoration content with the known content; by applying spatial attention and utilizing the local position relationship of features at different levels, the detail restoration capability of damaged images is improved; by applying linear attention, while the damaged image content is well restored, the image detail restoration effect is further enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image restoration method and electronic equipment based on texture and structure fusion priors. Background Art

[0002] In recent years, deep learning has demonstrated powerful capabilities in image restoration tasks, especially in filling pixels, which is very consistent with human vision. However, due to the complex background and large area missing of the image to be restored, artifacts still exist in the image restoration results. Most of the existing image restoration models use prior information to guide image content restoration, because it is very helpful to understand the overall structural details of the image and can synthesize content that matches the surrounding texture in the missing area, such as texture, structure, edge or shape. However, the existing models are easily limited by the receptive field size of the convolution kernel, which makes them unable to capture the long-distance dependencies of image features well. In contrast, Transformer-based methods have attracted widespread attention in image restoration tasks due to their ability to effectively capture global features. However, due to the increase in quadratic computational complexity, these models face great challenges in processing efficiency. Summary of the invention

[0003] In order to optimize the image restoration effect, the present invention provides an image restoration method and electronic device based on texture and structure fusion prior.

[0004] A first aspect of the present invention provides an image restoration method based on texture and structure fusion prior, which comprises the following steps:

[0005] Train an image inpainting model that includes a detail inpainting network and an image content completion network;

[0006] Inputting the image to be repaired into the image repair model, and the image repair model outputs the target image;

[0007] The texture features and structural features of the image to be repaired are fused and used as prior information, and are input into the detail repair network together with the image to be repaired, and spatial attention is applied to output a detail image;

[0008] The detail image is used as guiding information and is input into the image content completion network together with the image to be repaired, and linear attention is applied to output a target image (output image).

[0009] In the present invention, the fused texture features and structural features are used as prior information to guide the repair of damaged images, which plays a very important role in improving the image detail repair effect and maintaining the rationality and consistency of the repair content with the known content. Specifically, texture features can help understand the details and surface features of the image, such as fine textures, to ensure a natural transition between the repaired area and the surrounding area. At the same time, structural features can guide the overall layout and geometry of the repaired image to ensure that the repaired image can maintain better consistency and rationality with the human eye visually. In addition, by applying spatial attention and utilizing the local positional relationship of features at different levels, the detail repair ability of damaged images is improved. By applying linear attention, while the damaged image content is well repaired, the detail repair effect of the image is further enhanced.

[0010] In some embodiments, the texture features are extracted by Gabor filters. The frequency and direction of Gabor filters are very similar to the human visual system, and the texture features extracted by Gabor filters provide richer image details.

[0011] In some embodiments, the structural features are extracted by the Canny operator, which can find an optimal edge or the position with the strongest change in grayscale intensity in an image, and the structural features extracted by the Canny operator provide the overall shape information of the image.

[0012] In some embodiments, fusing the texture features and structural features of the image to be repaired and using them as prior information includes: fusing the texture features and structural features, and using the fused feature map for detail repair of the image.

[0013] In some embodiments, the detail repair network includes an encoder, a decoder, a jump connection module and a spatial attention module; the encoder performs DoubleConv operations, maximum pooling operations and downsampling operations on the input feature map; the decoder performs upsampling operations and DoubleConv operations on the feature map; the jump connection module transmits feature information between the encoder and the decoder; the spatial attention module performs average pooling operations and maximum pooling operations on the feature map in the channel dimension, and splices the average pooling results and the maximum pooling results together, and then performs a convolution operation on the spliced ​​result, and then generates an attention map through a Sigmoid activation function, and then multiplies the attention map and the input feature map element by element to obtain a weighted result of spatial attention. The spatial attention module plays a key role in accurately locating and processing damaged images. It can give priority to important areas in the image, such as high-frequency textures or key structural features, thereby significantly improving the effect and efficiency of image detail repair.

[0014] In some embodiments, the DoubleConv operation includes two convolution operations, two DropBlock operations, two BatchNorm2d function processes, and two ReLU activation function processes.

[0015] In some embodiments, the image content completion network includes an encoder, a decoder, a skip connection module and a linear attention module; the encoder performs a convolution operation on the input feature map; the decoder performs a deconvolution operation and ReLU function processing on the feature map; the feature information is transmitted between the encoder and the decoder through the skip connection module; the linear attention is calculated through the linear attention module, and a gating mechanism output is generated through 1×1 convolution and GELU activation function processing, and the gating mechanism output is multiplied by the linear attention to obtain a target image.

[0016] In some embodiments, the parameters of the detail restoration network are based on the loss L P Make updates;

[0017] ;

[0018] in,

[0019] ;

[0020] ;

[0021] ;

[0022] represents the reconstruction loss of the undamaged area of ​​the image to be repaired in the detail repair network, Represents the reconstruction loss of the damaged area of ​​the image to be repaired in the detail repair network, represents the adversarial loss of the detail restoration network; Represents the reconstruction loss weight of the damaged area of ​​the image to be repaired in the detail repair network, represents the adversarial loss weight of the detail restoration network; M represents the mask image, represents the number of non-zero elements in the mask image M, represents the real image, represents the output image, represents the Hadamard product; Represents the minimized generator The probability that the generated data is false, represents the maximum discriminator The probability of distinguishing real images from generated images, represents the generator of the detail restoration network, Represents the detail feature map of the real image, Represents all the real image data distribution The expected value, represents the discriminator of the detail restoration network, represents the inpainted detail image, Represents all the damaged image data under the distribution Expected value.

[0023] In some embodiments, the parameters of the image content completion network are based on the loss L G Make updates;

[0024] ;

[0025] in,

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] represents the reconstruction loss of the image to be repaired in the image content completion network, represents the adversarial loss of the image content completion network, represents the perceptual loss of the image content completion network, represents the style loss of the image content completion network; represents the adversarial loss weight of the image content completion network, represents the perceptual loss weight of the image content completion network, Represents the style loss weight of the image content completion network; Represents the minimized generator The probability that the generated data is false, represents the maximum discriminator The probability of distinguishing real images from generated images, Represents a real image expectations, represents the discriminator of the image content completion network, represents the generator of the image content completion network, represents the real image, represents the output image, Represents the output image expectations; represents the activation map of the i-th layer of the pre-trained network model of VGG-16 on the ImageNet dataset, ; .

[0031] A second aspect of the present invention provides an image restoration electronic device based on texture and structure fusion priors, which includes a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the image restoration method based on texture and structure fusion priors according to the computer program.

[0032] The technical solution of the embodiment of the present invention has the following beneficial effects:

[0033] (1) An image inpainting model consisting of a detail inpainting network and an image content completion network is proposed. This is an end-to-end image inpainting model with a dual attention mechanism guided by texture and structure. Through the multi-level fusion of global and local features, the accuracy and detail restoration ability of the model are effectively improved, while also greatly improving the training efficiency of the model.

[0034] (2) By applying spatial attention, the spatial relationship between different positions in the feature map is modeled and an accurate spatial attention map is generated. This not only improves the model's ability to determine the importance of features at different positions, but also significantly improves the model's ability and efficiency to repair local details of the image.

[0035] (3) By applying linear attention, the impact of the limited receptive field in traditional convolution on image restoration is effectively alleviated, the model’s ability to capture long-distance dependencies of features is improved, and the global detail information of the image is effectively restored. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the structure of the image restoration model in an embodiment of the present invention.

[0037] Figure 2 It is a schematic diagram of using Gabor filter and Canny operator to extract texture features and structural features respectively in an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the DoubleConv structure in an embodiment of the present invention.

[0039] Figure 4 Schematic diagram of the structure of the spatial attention module in an embodiment of the present invention.

[0040] Figure 5 Schematic diagram of the structure of a linear attention layer with gated convolution in an embodiment of the present invention.

[0041] Figure 6 Schematic diagram of the structure of the feedforward network layer in an embodiment of the present invention.

[0042] Figure 7 Schematic diagram of a linear Transformer encoding block in an embodiment of the present invention.

[0043] Figure 8 Shows the quantitative visual comparison results of different methods on the CelebA-HQ dataset.

[0044] Fig. 9 Shows the quantitative visualization comparison results of different methods on the ParisStreetView dataset.

[0045] Fig.10 Quantitative visual comparison results of different methods on the Place2 dataset are shown.

[0046] Fig.11 The figure shows the comparison results of the restoration effects of different methods on different defective images on different datasets, where, from top to bottom, they are the CelebA-HQ, Places2, and ParisStreetView datasets.

[0047] Fig.12 The ablation experiment results in the embodiment of the present invention are shown. DETAILED DESCRIPTION

[0048] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] In the following examples, the variables used and their meanings are shown in Table 1.

[0050] Table 1 Variables and their meanings

[0051]

[0052] In an embodiment of the present invention, image restoration is achieved by training an image restoration model. Figure 1As shown, in one embodiment of the present invention, the image restoration model includes a detail restoration network and an image content completion network, which correspond to the detail restoration stage (first stage) and the image content completion stage (second stage), respectively. Both stages follow the adversarial model, and each network is constructed based on the U-Net structure and the Markov discriminator (Patch-GAN discriminator). In the detail restoration network, the Gabor filter is first used to extract texture features, and the Canny operator is used to extract edge structure features, and then they are fused and input into the detail restoration network; the network uses the local position relationship of features at different levels to improve the detail restoration ability of the damaged image. In the image content completion network, the detail image obtained in the first stage is used as prior information to guide the content completion of the image, a linear Transformer encoding block is added, and the feature maps of each layer are fused through jump connections; the network uses the long dependency relationship of the features to fully capture the global feature information, and while repairing the damaged image content well, it further enhances the detail restoration effect of the image.

[0053] like Figure 2 As shown in the figure, Gabor filter and Canny operator are used to extract texture features and structural features respectively, and they are used as prior guidance information in the image restoration task. First, the texture features and structural features are effectively fused and used as input for image detail restoration, as shown in formula (1);

[0054] ;

[0055] in, represents a real image, Gabor() represents the use of Gabor filter to extract texture features, Canny() represents the use of Canny operator to extract structural features, Concat() represents feature fusion operation (addition operation), Represents the detail feature map of the fused real image.

[0056] from Figure 2 As can be seen in column (b), texture features provide richer image details. For example, the first row shows the details of the hair, and the third and fourth rows show the details of the windows and bricks. Figure 2 As can be seen in column (c), the structural features provide the overall shape information of the image. For example, the edge of the nose in the second row, the brick shape in the third row, and the edge of the snow mountain in the sixth row are all very clear. Figure 2 It can be seen from the (d) column that the fused feature map can well describe the rich details in the image.

[0057] The detail restoration network includes an encoder, a decoder, a skip connection module, and a spatial attention module. In the detail restoration network, the encoder stage uses a DoubleConv structure and a 2×2 maximum pooling operation, and is downsampled four times. Each DoubleConv structure contains two 3×3 convolution operations, 2 DropBlocks, 2 BatchNorm2d, and 2 ReLU activation functions, such as Figure 3 As shown. Among them, Dropblock discards the continuous pixels of adjacent areas of the feature map in the convolution layer together, that is, discards them in blocks, to prevent overfitting; BatchNorm2d helps to speed up the convergence of the network model. The decoder stage uses bilinear interpolation for upsampling, and uses the DoubleConv structure to gradually reduce the number of feature map channels. At the same time, a jump connection module is introduced between the encoder and decoder layers to effectively transfer feature information at different levels through the jump connection mechanism. Finally, a spatial attention module is introduced in the detail repair network to further enhance the model's ability to focus on important areas, effectively improving the performance and effect of the model in the repair task. The prior information P obtained in the first stage o It is expressed as follows:

[0058] ;

[0059] in, Representation detail repair network image generator; It represents the detail image output by the detail restoration network, that is, the prior information of the second stage; M represents the mask image.

[0060] The spatial attention module uses the spatial dimension between features to infer the spatial attention map, such as Figure 4 As shown in the figure. This module takes the feature map of the convolution layer as input, first performs average pooling and maximum pooling operations on it in the channel dimension, and concatenates the average pooling results and the maximum pooling results. Then, after a 7×7 convolution operation on the concatenated result, an attention map is generated through the Sigmoid activation function. Finally, the attention map is element-wise multiplied with the input feature map to obtain the weighted result of spatial attention. This spatial attention module plays a key role in accurately locating and processing damaged images. It can prioritize important areas in the image, such as high-frequency textures or key structural features, thereby significantly improving the effect and efficiency of image detail restoration.

[0061] The image content completion network includes an encoder, a decoder, a skip connection module, and a linear attention module. Both the encoder and the decoder contain 7 layers, and each layer contains a LeakyReLU activation function and an InstanceNorm2d function. In the image content completion network, the encoder uses a convolution operation with a step size of 2 to process the scale changes of each layer of features. In the decoder, deconvolution operations and ReLU functions are used to replace the original convolution operations and LeakyReLU functions. At the same time, in order to better model global features, 4 linear attention modules (linear Transformer encoding blocks) are added, which makes the image content completion network more focused on repairing global features, such as Figure 1 In addition, in order to better transfer the feature filling of the image content, the skip connection module (Skip-Connection) is used to connect each layer of the encoder features with the corresponding layer of the decoder. In the second stage, the real image , mask image M and detail image obtained by the first stage of restoration As input, the output obtained after the image restoration network is recorded as , as shown in formula (3);

[0062] ;

[0063] in, Represents an image content completion network image generator; Represents a real image; represents the detail image output by the detail restoration network; M represents the mask image.

[0064] The linear Transformer encoding block consists of a linear attention layer (LAG) with gated convolution and a simple feed-forward network layer (FFN). Figure 5 As shown in the linear attention layer with gated convolution, the input feature map is first Perform linear transformation to obtain matrices Q (query), K (key value) and V (value); then calculate the attention matrix based on matrices Q and K , use the Taylor expansion of the exponential function to normalize the attention matrix A; finally, perform weighted summation on the normalized attention matrix A and V to obtain the linearized attention, as shown in Figure 7 As shown in formula (4) to formula (9);

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] Where N = H × W, H and W represent the height and width of the feature map respectively; , and , C is the channel dimension of the feature, and N is much larger than C; , and are three trainable parameter matrices.

[0072] In a simple feedforward network layer, the linear attention calculated by formula (9) is input into the module of 1×1 convolution and GELU activation function to obtain the output G of the gating mechanism; finally, the output G of the gating mechanism is multiplied by the linear attention to obtain the output O, as shown in formulas (10) and (11):

[0073] ;

[0074] ;

[0075] in, represents 1×1 convolution, represents the GELU activation function, stands for Hadamard product.

[0076] Depend on Figure 7 It can be seen that the computational complexity of the linearized Transformer encoding block is , can significantly reduce the amount of calculation when processing large amounts of data, thereby improving calculation efficiency. At the same time, due to the reduction in complexity, the model in the embodiment of the present invention is more likely to process large-size inputs without significantly increasing the calculation cost.

[0077] In an embodiment of the present invention, the image restoration model is optimized using reconstruction loss, adversarial loss, perceptual loss and style loss.

[0078] The reconstruction loss is obtained by calculating the difference between the generated image and the real image pixel by pixel. The corresponding reconstruction loss functions in the detail restoration network and the image content completion network are defined as follows:

[0079] ;

[0080] ;

[0081] ;

[0082] in, represents the number of non-zero elements in the mask image M, and Represent the reconstruction loss of the undamaged area and the damaged area in the detail restoration network, represents the reconstruction loss of the entire image to be repaired in the image content completion network, and Represent the reconstruction loss of the undamaged area and the damaged area in the image content completion network, respectively. Represents the reconstruction loss weight of the damaged area.

[0083] In order to make the restored image details and image content more consistent and reasonable with the known areas of the damaged image, the corresponding adversarial losses in the detail restoration network and the image content completion network are defined as follows:

[0084] ;

[0085] ;

[0086] in, Represents the minimized generator The probability that the generated data is false, represents the maximum discriminator The probability of distinguishing real images from generated images, Represents all the real image data distribution The expected value, Represents all the damaged image data under the distribution Expectations of values; Represents the minimized generator The probability that the generated data is false, represents the maximum discriminator The probability of distinguishing real images from generated images, Represents a real image expectations, Represents the output image expectations.

[0087] In order to extract discriminative key features from the deep network structure and enhance the model's ability to repair image detail information, the perceptual loss is defined as follows:

[0088] ;

[0089] in, represents the activation map of the i-th layer of the pre-trained network model of VGG-16 on the ImageNet dataset, .

[0090] In order to reduce the difference between the generated image of the proposed model and the original image and make it more consistent with the visual characteristics of the human eye, the style loss is defined as follows:

[0091] ;

[0092] Among them, the Gram matrix .

[0093] Finally, the corresponding total loss functions in the detail restoration network and the image content completion network are defined as follows:

[0094] ;

[0095] ;

[0096] in, , , and Represents the weights for weighing different types of losses.

[0097] In order to verify the effect of the image restoration model in the embodiment of the present invention, a large number of experiments and analyses are carried out on three public image restoration datasets (Places2, CelebA-HQ, and ParisStreetView). The detailed information of the datasets is shown in Table 2.

[0098] Table 2 Detailed information of the dataset

[0099]

[0100] During model training, BatchSize is set to 1 and the Adam algorithm is used for optimization. The parameters in Adam are =0.5, =0.999, learning rate At the same time, in order to prove the superiority of the image restoration method of the embodiment of the present invention, it is compared with the cutting-edge methods (RN, RFR, CTSDG, DSNet, LGNet and CANet). The experimental environment is as follows: the operating system is 64-bit Ubuntu 20.04.6 LTS, the deep learning framework is PyTorch 1.6.0, and the GPU is NVIDIA GeForce RTX 4090. The comparison method is summarized as follows:

[0101] RN: This method achieves a more accurate and effective image restoration effect by calculating the feature normalization of the damaged area and the known area in the feature map, and using the learnable region normalization technology to process the feature representation of the damaged area and the known area respectively;

[0102] RFR: This method includes a recursive feature reasoning module and a knowledge consistent attention module. It repeatedly infers the defect boundary of the feature map and takes the repaired boundary as the known area, gradually strengthening the constraints of the defect center area to achieve continuous and natural image restoration.

[0103] CTSDG: This method generates realistic image restoration results by constraining texture synthesis and guiding structure reconstruction, using bidirectional gated feature fusion and contextual feature aggregation;

[0104] DSNet: This method dynamically selects appropriate spatial sampling positions during the convolution operation, combines multiple normalization methods, and selectively normalizes feature areas to achieve flexible image restoration;

[0105] LGNet: This method mainly includes a coarse inpainting network with a large receptive field and local and global refinement networks with different receptive fields. It improves the image inpainting results by constructing local inpainting and global inpainting models.

[0106] CANet: This method dynamically generates different weights for the irregular shapes and locations of damaged areas. The importance of information is determined by the size of the weights, allowing for flexible repair of various forms of damage.

[0107] In order to quantitatively evaluate the image restoration performance, Peak Signal-to-noise Ratio (PSNR), Structural Similarity Index (SSIM) and reconstruction loss are used. The performance of the image restoration method of the embodiment of the present invention is compared with the mainstream method. PSNR is used to evaluate the effect of image restoration from the perspective of human visual perception, SSIM and reconstruction loss are used to evaluate the effect of image restoration from the perspective of human visual perception, and SSIM and reconstruction loss are used to evaluate the effect of image restoration from the perspective of human visual perception. It measures the difference between the repaired image and the original image from the perspective of image pixels. Tables 3 to 5 provide quantitative comparison results of the image repair method of the embodiment of the present invention and the frontier method under different damage ratios in different data sets. The change trends of the three quantitative indicators under different damage ratios are shown in the figure. Figures 8 to 10 shown.

[0108] Table 3 Quantitative comparison results of image restoration effects of different methods on the CelebA-HQ dataset

[0109]

[0110] In the above table, ↑ means the higher the quantitative index, the better, and ↓ means the lower the quantitative index, the better.

[0111] From Table 3 and Figure 8It can be seen that the image restoration method of the embodiment of the present invention achieves the best PSNR, SSIM and reconstruction loss in the CelebA-HQ dataset. The PSNR and SSIM values ​​of LGNet are slightly lower than those of the image restoration method of the embodiment of the present invention, and the performances of CTSDG and DSNet are relatively close. In addition, as the damage ratio increases, the performance of the DSNet method is lower than that of the image restoration method of the embodiment of the present invention, but slightly better than that of the LGNet method. This is mainly because DSNet plays a key role in the dynamic selection of missing areas and known areas. It can be seen from the three indicators that the performance of the RN method is the worst.

[0112] Table 4 Quantitative comparison results of image restoration effects of different methods on the ParisStreetView dataset

[0113]

[0114] In the above table, ↑ means the higher the quantitative index, the better, and ↓ means the lower the quantitative index, the better.

[0115] From Table 4 and Fig. 9 It can be seen that as the defect area continues to increase, the image restoration method of the embodiment of the present invention achieves the optimal PSNR, SSIM and reconstruction loss in the ParisStreetView dataset. CTSDG achieves the second best PSNR, SSIM and reconstruction loss. The image restoration performance of DSNet is similar to that of CTSDG. As the defect area increases, the image restoration performance of RN is the worst.

[0116] Table 5 Quantitative comparison results of image restoration effects of different methods on the Place2 dataset

[0117]

[0118] In the above table, ↑ means the higher the quantitative index, the better, and ↓ means the lower the quantitative index, the better.

[0119] Similarly, from Table 5 and Fig.10 It can be seen that the image restoration method of the embodiment of the present invention achieves the best PSNR, SSIM and reconstruction loss on the Place2 dataset. From the PSNR and SSIM analysis, the LGNet method achieved the second best performance on the Place2 dataset; from the reconstruction loss According to the value analysis, the CTSDG method achieved the second best performance. CANet and RFR methods have similar image restoration performance, and the RN method has the worst image restoration performance.

[0120] Fig.11 The image restoration method of the embodiment of the present invention and the comparative method are compared in terms of the restoration effect of different defective images on different data sets. The first two rows are the image restoration effects on the CelebA-HQ data set, the third and fourth rows are the image restoration results of the Places2 data set, and the last three rows are the image restoration results of the ParisStreetView data set. It can be seen that the visual quality of the images restored by the RN and RFR methods is poor, which is also consistent with the quantitative results. By comparing the face restoration effect, it can be seen that the image restoration method of the embodiment of the present invention can better restore the eyes, mouth and other facial features. Compared with CTSDG and DSNet, it can be seen that the lips restored by the image restoration method of the embodiment of the present invention are more natural (the second row). Compared with the LGNet method, the image restoration method of the embodiment of the present invention can restore relatively complete image structures and details, such as windows (the sixth and seventh rows). This is because in the image restoration method of the embodiment of the present invention, texture and structural information are used as a priori guidance for image content restoration, which plays a very important role in restoring the detailed information of the image. Compared with CANet, obvious color difference and shadow appear in the restoration results of natural scenery (the fourth row) and face images (the second row). It can be seen from the observation of the above restoration effects that the image restoration method of the embodiment of the present invention relatively completely restores the structure and detail information of the face and the doors and windows.

[0121] The computational performance of the proposed method is evaluated by the algorithm complexity (FLOPs) and the number of parameters calculated by floating-point operations, as shown in Table 6. Among the listed comparative models, RN, RFR and CANet are single-stage image restoration models, CTSDG and DSNet are two-stage image restoration models, LGNet is a three-stage model, and the image restoration method of the embodiment of the present invention is a two-stage image restoration model. It can be seen that although the image restoration method of the embodiment of the present invention is two-stage, its computational complexity is relatively low, the number of parameters is moderate, and it has high stability. Although the CTSDG and DSNet models are computationally efficient and have a small number of parameters, it can be seen from experiments that their performance will also decline. In particular, in the image restoration method of the embodiment of the present invention, the number of parameters of the prior information restoration network in the first stage is only 7.74M. In order to further improve the quality of image restoration, the image restoration method of the embodiment of the present invention adds a linear Transformer module to the image content completion network in the second stage, which greatly reduces the computational complexity of the model and enhances the ability of the model to repair image details. The parameters of the image restoration model of the embodiment of the present invention mainly come from the depth sampling layer of the second stage, wherein the convolution layer of 512 channels requires approximately 47M parameters.

[0122] Table 6 Algorithm complexity (FLOPs) and parameter count (Params) of different models

[0123]

[0124] In order to verify the effectiveness of texture and structure features, semantic prior restoration module and linear Transformer module in image restoration, corresponding ablation experiments were carried out on the CelebA-HQ dataset. Fig.12 The comparison of image restoration effects of ablation experiments is listed; Table 7 gives the quantitative comparison results of ablation experiments, where "w / ogabor" means that Gabor texture features are not used in the model, "w / ocanny" means that edge features extracted by the Canny operator are not used in the model, "w / oprior" means that semantic priors are not used in the model, "w / oatt" means that the spatial attention module is not used in the first stage of the model, and "w / otrans" means that the linear Transformer module is not used in the second stage of the model.

[0125] Table 7 Ablation experiment results

[0126]

[0127] In the above table, ↑ means the higher the quantitative index, the better, and ↓ means the lower the quantitative index, the better.

[0128] from Fig.12 Middle (b) column - Fig.12 It can be seen from column (d) that without the structure and texture prior extracted by the Canny operator and Gabor filter, the details in the repaired result are relatively blurred. Fig.12 Column (e) and Fig.12 The results in column (f) are obtained by adding a dual attention module based on prior information. It can be seen that with the addition of the attention mechanism, the blurriness of the details in the repaired image is significantly reduced. In particular, after removing the linear Transformer module from the model, the effect of repairing the image is significantly reduced, which further proves the effectiveness of the module. It can be seen from the ablation experiment that the image repair model of the embodiment of the present invention achieves a relatively ideal image effect for image repair tasks with complex textures or large defect areas.

[0129] From the results listed in Table 7, it can be seen that removing only texture features or structural features has little effect on the image restoration performance of the model, but removing the prior fusion features of texture and structure will significantly reduce the performance of image restoration. From the two ablation experiments of "w / oatt" and "w / otrans", it can be seen that spatial attention and Transformer-based linear attention play a very important role in image restoration, especially Transformer-based linear attention. This result shows that the dual attention mechanism can effectively capture the long-distance dependency of features in the image restoration process, significantly improving the model's ability to repair image details.

[0130] The above is only a preferred embodiment of the present invention, and is not any formal or substantial limitation of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the method of the present invention, and these improvements and supplements should also be regarded as the protection scope of the present invention. Any technician familiar with this profession, without departing from the spirit and scope of the present invention, can make some changes, modifications and evolutions of the technical content disclosed above, which are equivalent embodiments of the present invention; at the same time, any changes, modifications and evolutions of any equivalent changes made to the above embodiments based on the essential technology of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An image restoration method based on texture and structure fusion priors, It is characterized in that The following steps are involved: Train an image inpainting model that includes a detail inpainting network and an image content completion network; Inputting the image to be repaired into the image repair model, and the image repair model outputs the target image; The texture features and structural features of the image to be repaired are fused and used as prior information, and are input into the detail repair network together with the image to be repaired, and spatial attention is applied to output a detail image; The detail image is used as guiding information and is input into the image content completion network together with the image to be repaired, and linear attention is applied to output a target image; The detail restoration network includes an encoder, a decoder, a skip connection module and a spatial attention module; the encoder performs DoubleConv operations, maximum pooling operations and downsampling operations on the input feature map; the decoder performs upsampling operations and DoubleConv operations on the feature map; the skip connection module transmits feature information between the encoder and the decoder; the spatial attention module performs average pooling operations and maximum pooling operations on the feature map in the channel dimension, and splices the average pooling results and the maximum pooling results together, and then performs a convolution operation on the spliced ​​result, and then generates an attention map through a Sigmoid activation function, and then multiplies the attention map and the input feature map element by element to obtain a weighted result of spatial attention; The image content completion network includes an encoder, a decoder, a skip connection module and a linear attention module; the encoder performs a convolution operation on the input feature map; the decoder performs a deconvolution operation and ReLU function processing on the feature map; the skip connection module transmits feature information between the encoder and the decoder; the linear attention module calculates the linear attention, and generates a gating mechanism output through 1×1 convolution and GELU activation function processing, and then multiplies the gating mechanism output and the linear attention to obtain a target image.

2. The method according to claim 1, It is characterized in that The texture feature is extracted by a Gabor filter.

3. The method according to claim 2, It is characterized in that The structural features are extracted by the Canny operator.

4. The method according to claim 3, It is characterized in that Fusing the texture features and structural features of the image to be repaired and using them as prior information includes: fusing the texture features and structural features, and using the fused feature map for detail repair of the image.

5. The method according to claim 1, It is characterized in that The DoubleConv operation includes two convolution operations, two DropBlock operations, two BatchNorm2d function processes and two ReLU activation function processes.

6. The method according to claim 1, It is characterized in that The parameters of the detail restoration network are updated according to the loss LP; ; in, ; ; ; represents the reconstruction loss of the undamaged area of ​​the image to be repaired in the detail repair network, Represents the reconstruction loss of the damaged area of ​​the image to be repaired in the detail repair network, represents the adversarial loss of the detail restoration network; Represents the reconstruction loss weight of the damaged area of ​​the image to be repaired in the detail repair network, represents the adversarial loss weight of the detail restoration network; M represents the mask image, represents the number of non-zero elements in the mask image M, represents the real image, represents the output image, represents the Hadamard product; Represents the minimized generator The probability that the generated data is false, Represents the maximum discriminator The probability of distinguishing real images from generated images, represents the generator of the detail restoration network, Represents the detail feature map of the real image, Represents all the real image data distribution The expected value, represents the discriminator of the detail restoration network, represents the inpainted detail image, Represents all the damaged image data under the distribution Expected value.

7. The method according to claim 1, It is characterized in that The parameters of the image content completion network are updated according to the loss LG; ; in, ; ; ; ; represents the reconstruction loss of the image to be repaired in the image content completion network, represents the adversarial loss of the image content completion network, represents the perceptual loss of the image content completion network, represents the style loss of the image content completion network; represents the adversarial loss weight of the image content completion network, represents the perceptual loss weight of the image content completion network, Represents the style loss weight of the image content completion network; Represents the minimized generator The probability that the generated data is false, Represents the maximum discriminator The probability of distinguishing real images from generated images, Represents a real image expectations, represents the discriminator of the image content completion network, represents the generator of the image content completion network, represents the real image, represents the output image, Represents the output image expectations; represents the activation map of the i-th layer of the pre-trained network model of VGG-16 on the ImageNet dataset, ; .

8. An electronic device for image restoration based on texture and structure fusion priors, It is characterized in that It comprises a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the image restoration method based on texture and structure fusion prior according to any one of claims 1 to 7 according to the computer program.

Citation Information

Patent Citations

  • GAN image restoration method based on jump connection multi-scale fusion

    CN114693565A

  • Automatic ionogram tracing method based on multi-scale attention enhancement U-Net

    CN114972760A