End-to-end image restoration method based on edge and structure prior joint guidance

Abstract

The invention relates to an end-to-end image restoration method based on edge and structure prior joint guidance, and the method comprises the following steps: S1, inputting a damaged image into a model, and carrying out the feature extraction and coding through multi-layer stacked gating convolution; s2, fully extracted context features are obtained; s3, splitting the extracted features into three parallel branches including an edge feature, a structural feature and an RGB image feature, and inputting an edge image and a structural image of a corresponding resolution into each branch; s4, the edge features and the structure features are sent to an adaptive prior feature fusion module, and fused prior features are obtained; s5, the prior features and the RGB image features are sent to a channel and region perception collaborative attention module together, and enhanced features are obtained; and S6, obtaining a trained network and a repaired complete image. According to the method, global complementary information is transmitted through prior features to assist in image restoration.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to an end-to-end image restoration method, particularly an end-to-end image restoration method based on edge and structure priors. Background Technology

[0002] Images contain rich information and can be presented intuitively, serving as one of the most important ways for humans to understand the outside world. With the advent of the digital age, image data has exploded, and massive amounts of images are widely used in various fields such as medicine, security, entertainment, and autonomous driving. However, during the storage, compression, and transmission of image information, images are often affected by noise, equipment failure, network transmission problems, and other human factors, leading to image loss. Therefore, image restoration technology has become a core component in ensuring image quality, restoring critical information, and improving the effectiveness of subsequent applications, playing an irreplaceable and vital role in numerous fields.

[0003] Traditional image inpainting methods are mainly divided into two categories: diffusion-based and patch-based. Diffusion-based methods repair the missing area by spreading information from surrounding pixels, but are only suitable for small-scale defects. Patch-based methods fill the missing area by searching for similar patches, but are computationally expensive and have limited success rates. Both methods lack an understanding of the global structure and semantic information of the image, resulting in images that are often blurry and riddled with artifacts, making it difficult to meet the requirements for high-quality inpainting.

[0004] In recent years, deep learning has made significant progress in the field of image inpainting. These methods typically employ an encoder-decoder structure, focusing on learning known visible content and filling in missing regions using unique convolutional methods and different contextual attention mechanisms. However, when dealing with images with large areas of occlusion or severe damage, generative networks alone struggle to capture sufficient semantic consistency information in a limited number of known regions. Therefore, some methods introduce prior information to guide image inpainting. By providing additional guidance, especially when the missing region lacks sufficient context, it helps the model infer relatively reasonable inpainting content, maintaining the structural consistency and visual coherence of the image, thereby improving the reliability of the results.

[0005] However, previous image inpainting methods guided by prior information were often multi-stage. The first stage only generated prior information, and the second stage combined the prior information generated in the first stage to perform the final image inpainting. However, obtaining reasonable priors from damaged images is itself a very challenging problem. In a serially coupled framework, using uncertain priors often leads to more misleading results in subsequent inpainting. Moreover, multi-stage models are very cumbersome and time-consuming to train.

[0006] Furthermore, some methods following the U-Net encoding / decoding architecture attempt to combine the modeling processes for texture and structural priors, typically using a shared generator to handle both simultaneously. However, this approach does not adequately consider the relationship between the two prior information types, making it difficult for prior features to convey globally complementary information to aid in image restoration.

[0007] Therefore, in order to solve the above-mentioned technical problems, this invention proposes an end-to-end image restoration method based on edge and structure prior guidance.

[0008] A search revealed no publicly available literature of the same or similar prior art as this invention. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an end-to-end image inpainting method based on joint guidance of edge and structural priors. First, a dataset of commonly used image inpainting tasks is selected as training samples. After feature extraction during the encoding stage, multi-branch collaborative learning and dynamic feature fusion are used during the decoding stage to improve the quality and efficiency of inpainting. After end-to-end training is completed, the test image to be inpainted is fed into the network for inpainting, and finally the inpainting result is output. This achieves the goal of using prior features to transmit global complementary information to assist in image inpainting.

[0010] The present invention solves its practical problem by adopting the following technical solution:

[0011] An end-to-end image inpainting method based on edge and structure priors jointly guided by the following steps:

[0012] S1. Input the damaged image into the model, and first perform feature extraction and encoding through multiple layers of stacked gated convolutions;

[0013] S2. Feed the features extracted in step S1 into the bottleneck layer composed of the multidimensional context extraction module MCFEM to obtain fully extracted context features.

[0014] S3. Enter the decoding stage. The features extracted in step S2 are split into three parallel branches: edge features, structural features, and RGB image features. Each branch is input with edge and structural images of the corresponding resolution.

[0015] S4. Input the edge features and structural features into the adaptive prior feature fusion module APFFM to obtain the fused prior features;

[0016] S5. The prior features output in step S4 are fed together with the RGB image features into the Channel and Region Aware Collaborative Attention Module (CRASA) to obtain the enhanced features F'. image ;

[0017] S6. Optimize the network parameters using the Adam optimizer and joint loss function to obtain the trained network;

[0018] S7. Feed the test set images into the trained network for testing to obtain the repaired complete images.

[0019] Furthermore, before step S1, the following steps are included: before the data is input into the model, the original image needs to be preprocessed and image enhancement operations are performed. The resolution of the image and the mask is uniformly constrained to a size of 256×256, and the corresponding edge image and structure smoothing image are generated. The kernel size of the Gaussian filter in the Canny algorithm for generating the edge image is set to 2. Then, the original image is subjected to random flipping, random cropping, and random rotation data enhancement operations. Finally, the pixel value range of the image is converted to [-1,1].

[0020] Furthermore, the specific steps of step S2 include:

[0021] S201. Let f be the feature map input to the Multidimensional Contextual Feature Extraction (MCFEM) module. h Its size is B×H×W×C;

[0022] Where B represents the training batch, C represents the number of feature map channels, and H and W represent the height and width of the feature map, respectively.

[0023] f h The features are divided into 4 groups on average according to the channel dimension, and the number of channels in each group is C / 4.

[0024] S202. The grouped features from step S201 are processed in parallel through a feature extraction branch based on dilated convolution. Each branch's dilated convolution is set with a different dilation coefficient r and a learnable weight parameter ω. i Subsequently, the four sets of contextual features from different receptive fields were concatenated and aggregated by splicing them together along the channel dimension.

[0025] This process can be represented as:

[0026]

[0027] Among them, f i DilatedConv represents the feature of the channel being averaged across all segments. i The dilated convolution on the i-th branch is shown. Let represent the output corresponding to the i-th branch, and [;] represent concatenation along the channel dimension.

[0028] S203, Feature map f of the Multidimensional Contextual Feature Extraction Module (MCFEM) hThe gate control value g is calculated using a 1×1 ordinary convolution and a sigmoid function, and then used to adjust f. h and f r Perform weighted summation to obtain the output features.

[0029] This process can be represented as:

[0030]

[0031] Furthermore, the specific steps of step S4 include:

[0032] S401. Let the edge features updated by the decoder be F. e The structural feature is F s Average pooling and max pooling operations are performed separately, and then the results of the two pooling operations are concatenated along the channel dimension to obtain the edge comprehensive pooling feature. Combined pooling features of the results

[0033] This process can be represented as:

[0034]

[0035] S402, Edge Pooling Feature Combined pooling features of the results First, the features are transformed in channel dimension using 1×1 convolution, and then a non-linear transformation is performed using the LeakyReLU activation function. Next, the features are further updated using 3×3 convolution, and the corresponding edge feature weight map G is calculated using the Sigmoid activation function. e and structural feature weight map G s ;

[0036] As shown below:

[0037] G e =Sigmoid(Conv2(ReLU(Conv1(F e p ))))

[0038] G s =Sigmoid(Conv2(ReLU(Conv1(F s p ))))

[0039] S403, Edge-sensing feature F e 'By analyzing the original structural features F s With weighted edge features G e ⊙F e Pixel-level addition is performed to obtain it; similarly, using G...s Calculate the structure-aware feature F s ', finally F e 'and F s 'Merge into F' prior As the final prior feature representation.

[0040] Furthermore, the specific steps of step S5 include:

[0041] S501. The SPADE normalization method is used to normalize the prior features F. prior Injected into RGB image features F image The formula is as follows:

[0042]

[0043] Where F * This represents the transformed feature map, where γ and β are modulation parameters learned from prior features, and μ... c and σ c These represent the mean and standard deviation of the feature at channel c, respectively.

[0044] S502, for F * Perform average pooling to scale its resolution from H×W to H'×W':

[0045]

[0046] Among them, AvgPool (7×7) This represents a pooling operation with a kernel size of 7×7.

[0047] S503, Use Instance Normalization(IN) to Perform a normalization operation, and then calculate the attention along the channel dimension:

[0048]

[0049] Among them, F proj This represents the mapping function that generates Q, K, and V, and DWConv represents depthwise separable convolution.

[0050] S504. The attention coefficients between channels are calculated using Softmax, adaptive pooling is used to scale the resolution to C×C, and then Sigmoid activation is applied. Finally, the enhanced channel features are multiplied channel by channel with the input features to obtain the enhanced channel features.

[0051] The specific formula is as follows:

[0052]

[0053] Where d is the dimension of vector K, which is usually used to scale the size of the dot product.

[0054] S505, the result obtained in step S504 The gated mask M is obtained after passing through a gated convolutional layer (GC). G Display the feature F that distinguishes the missing region. f and known region features F b :

[0055]

[0056]

[0057] S506. Extract 3×3 pixel patches and calculate their cosine similarity. Use softmax to weight the similarity and obtain the attention score for each patch. As shown below:

[0058]

[0059] S507, through coordinated attention of channels and space, yields the final output:

[0060]

[0061] Furthermore, the specific steps of step S6 include:

[0062] S601. During training, the number of samples fed in each time is: batch_size = 16, and a binary mask M is randomly generated, where 1 represents a known region and 0 represents a missing region.

[0063] S602. The loss function of the network is calculated as follows:

[0064] L total =L inpaint +L edge +L st

[0065] Among them, L inpaint L represents the image reconstruction loss. edge L represents the edge loss. st The reconstruction loss is represented by the Adam algorithm, which is used to minimize the loss function and continuously update the network parameters until convergence.

[0066] In step S602, L inpaint Pixel reconstruction loss L, including known and missing regions hole and L valid Perceived loss L perceptual and style loss L style :

[0067]

[0068] Where Ψ represents the activation feature map obtained using the pre-trained model VGG-16, and the expected value E is used to average the loss of all feature layers, G j This indicates that the feature map Ψ is activated at layer j. j The constructed Gram matrix. L edge Includes generative adversarial loss and multi-scale binary cross-entropy loss:

[0069]

[0070] Where, N s It is the number of edge map pixels at the s-th scale. and These are the edge prediction value and the true label at the s-th scale, respectively, G e For edge branch generator, D e For discriminator. L st Structural pixel reconstruction loss for known and missing regions and

[0071]

[0072] Advantages and beneficial effects of the present invention:

[0073] 1. This invention proposes an end-to-end image inpainting method based on joint guidance of edge and structural priors, applicable to image inpainting in numerous scenarios such as faces, buildings, and natural landscapes. First, datasets of commonly used image inpainting tasks are selected as training samples. After feature extraction during the encoding stage, multi-branch collaborative learning and dynamic feature fusion are used during the decoding stage to improve inpainting quality and efficiency. After end-to-end training, the test image to be inpainted is fed into the network for inpainting, and finally, the inpainted result is output.

[0074] 2. This invention designs a multi-dimensional context feature extraction module, which enhances the model's ability to reason about context in large-scale missing regions through feature weighted aggregation of multi-scale dilated convolution and gated residual fusion mechanism.

[0075] 3. This invention designs an adaptive prior feature fusion module, which achieves adaptive fusion of structural and edge features through a dynamic gating mechanism. This module can dynamically adjust the contribution of different priors to the missing region, effectively suppressing redundant and conflicting information while enhancing key prior features.

[0076] 4. This invention develops a channel and region perception collaborative attention module, which decouples channel and spatial processing: first, it compresses the spatial dimension to calculate channel correlation, thereby alleviating semantic differences between features; then, it introduces a region perception mechanism to model long-distance spatial dependencies and refine the generated content. Attached Figure Description

[0077] Figure 1 This is a flowchart of the processing of the present invention;

[0078] Figure 2 This is a schematic diagram of the multidimensional context feature extraction module of the present invention;

[0079] Figure 3 This is a schematic diagram of the adaptive prior feature fusion module of the present invention;

[0080] Figure 4 This is a schematic diagram of the channel and region perception collaborative attention of the present invention;

[0081] Figure 5 This is a graph showing the qualitative comparison results of the present invention on the Paris StreetView dataset;

[0082] Figure 6 This is a graph showing the qualitative comparison results of the present invention on the CelebA-HQ dataset;

[0083] Figure 7 This is a graph showing the qualitative comparison results of the present invention on the Places2 dataset. Detailed Implementation

[0084] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0085] An end-to-end image inpainting method guided by joint edge and structure priors, such as Figure 1 As shown, it includes the following steps:

[0086] S1. Input the damaged image into the model, and first perform feature extraction and encoding through multiple layers of stacked gated convolutions;

[0087] In this embodiment, the original image needs to be preprocessed and enhanced before the data is input into the model. In the experiment, the resolution of the image and the mask is uniformly constrained to 256×256, and corresponding edge images and structural smoothing images are generated. The kernel size of the Gaussian filter in the Canny algorithm for generating the edge image is set to 2. Then, random flipping, random cropping, and random rotation are performed on the original image to avoid overfitting during network training and enhance the generalization ability of the generator. Finally, the pixel value range of the image is converted to [-1, 1].

[0088] S2. Feed the features extracted in step S1 into the bottleneck layer composed of the multidimensional context extraction module MCFEM to obtain fully extracted context features.

[0089] like Figure 2 As shown, the specific steps of step S2 include:

[0090] S201. Let f be the feature map input to the Multidimensional Contextual Feature Extraction (MCFEM) module. h Its size is B×H×W×C;

[0091] Where B represents the training batch, C represents the number of feature map channels, and H and W represent the height and width of the feature map, respectively.

[0092] f h The features are divided into 4 groups on average according to the channel dimension, and the number of channels in each group is C / 4.

[0093] S202. The grouped features from step S201 are processed in parallel through a feature extraction branch based on dilated convolution. Each branch's dilated convolution is set with a different dilation coefficient r and a learnable weight parameter ω. i Subsequently, the four sets of contextual features from different receptive fields were concatenated and aggregated by splicing them together along the channel dimension.

[0094] This process can be represented as:

[0095]

[0096] Among them, f i DilatedConv represents the feature of the channel being averaged across all segments. i The dilated convolution on the i-th branch is shown. Let represent the output corresponding to the i-th branch, and [;] represent concatenation along the channel dimension.

[0097] S203, Feature map f of the Multidimensional Contextual Feature Extraction Module (MCFEM) h The gate control value g is calculated using a 1×1 ordinary convolution and a sigmoid function, and then used to adjust f. h and f r Perform weighted summation to obtain the output features.

[0098] This process can be represented as:

[0099]

[0100] S3. Enter the decoding stage. The features extracted in step S2 are split into three parallel branches: edge features, structural features, and RGB image features. Each branch is input with edge and structural images of the corresponding resolution.

[0101] S4. Input the edge features and structural features into the adaptive prior feature fusion module APFFM to obtain the fused prior features;

[0102] like Figure 3 As shown, the specific steps of step S4 include:

[0103] S401. Let the edge features updated by the decoder be F. e The structural feature is F s Average pooling and max pooling operations are performed separately, and then the results of the two pooling operations are concatenated along the channel dimension to obtain the edge comprehensive pooling feature. Combined pooling features of the results

[0104] This process can be represented as:

[0105]

[0106] S402, Edge Pooling Feature Combined pooling features of the results First, the features are transformed in channel dimension using 1×1 convolution, and then a non-linear transformation is performed using the LeakyReLU activation function. Next, the features are further updated using 3×3 convolution, and the corresponding edge feature weight map G is calculated using the Sigmoid activation function. e and structural feature weight map G s ;

[0107] As shown below:

[0108] G e =Sigmoid(Conv2(ReLU(Conv1(F e p ))))

[0109] G s =Sigmoid(Conv2(ReLU(Conv1(F s p ))))

[0110] S403, Edge-sensing feature F e 'By analyzing the original structural features F s With weighted edge features G e ⊙F e Pixel-level addition is performed to obtain it; similarly, using G... s Calculate the structure-aware feature F s ', and finally F' e and F′ s Merge into Fprior As the final prior feature representation.

[0111] S5. The prior features output in step S4 are fed together with the RGB image features into the Channel and Region Aware Collaborative Attention Module (CRASA) to obtain the enhanced features F'. image ;

[0112] like Figure 4 As shown, the specific steps of step S5 include:

[0113] S501. The SPADE normalization method is used to normalize the prior features F. prior Injected into RGB image features F image The formula is as follows:

[0114]

[0115] Where F * This represents the transformed feature map, where γ and β are modulation parameters learned from prior features, and μ... c and σ c These represent the mean and standard deviation of the feature at channel c, respectively.

[0116] S502, for F * Perform average pooling to scale its resolution from H×W to H'×W':

[0117]

[0118] Among them, AvgPool (7×7) This represents a pooling operation with a kernel size of 7×7.

[0119] S503, Use Instance Normalization(IN) to Perform a normalization operation, and then calculate the attention along the channel dimension:

[0120]

[0121]

[0122] Among them, F proj This represents the mapping function that generates Q, K, and V, and DWConv represents depthwise separable convolution.

[0123] S504. The attention coefficients between channels are calculated using Softmax, adaptive pooling is used to scale the resolution to C×C, and then Sigmoid activation is applied. Finally, the enhanced channel features are multiplied channel by channel with the input features to obtain the enhanced channel features.

[0124] The specific formula is as follows:

[0125]

[0126] Where d is the dimension of vector K, which is usually used to scale the size of the dot product.

[0127] S505, the result obtained in step S504 The gated mask M is obtained after passing through a gated convolutional layer (GC). G Display the feature F that distinguishes the missing region. f and known region features F b :

[0128]

[0129] S506. Extract 3×3 pixel patches and calculate their cosine similarity. Use softmax to weight the similarity and obtain the attention score for each patch. As shown below:

[0130]

[0131] S507, through coordinated attention of channels and space, yields the final output:

[0132]

[0133] S6. Optimize the network parameters using the Adam optimizer and joint loss function to obtain the trained network;

[0134] The specific steps of step S6 include:

[0135] S601. During training, the number of samples fed in each batch is 16, and a binary mask M is randomly generated, where 1 represents a known region and 0 represents a missing region.

[0136] S602. The loss function of the network is calculated as follows:

[0137] L total =L inpaint +L edge +L st

[0138] Among them, L inpaint L represents the image reconstruction loss. edge L represents the edge loss. st The reconstruction loss is represented by the Adam algorithm, which is used to minimize the loss function and continuously update the network parameters until convergence.

[0139] In step S602, L inpaintPixel reconstruction loss L, including known and missing regions hole and L valid Perceived loss L perceptual and style loss L style :

[0140]

[0141] Where Ψ represents the activation feature map obtained using the pre-trained model VGG-16, and the expected value E is used to average the loss of all feature layers, G j This indicates that the feature map Ψ is activated at layer j. j The constructed Gram matrix. L edge Includes generative adversarial loss and multi-scale binary cross-entropy loss:

[0142]

[0143] Where, N s It is the number of edge map pixels at the s-th scale. and These are the edge prediction value and the true label at the s-th scale, respectively, G e For edge branch generator, D e For discriminator. L st Structural pixel reconstruction loss for known and missing regions and

[0144]

[0145] All experiments used a uniform training setup: the generator and edge discriminator were trained using the Adam optimizer, with first-order gradient estimation β1 set to 0.9 and second-order gradient estimation β2 set to 0.999. Their initial learning rates were set to 1×10⁻⁶. -4 The learning rate was adjusted using a weight decay mechanism. The batch size was set to 16, and the weights of each loss function were configured as follows: the weight λ for image missing region reconstruction loss. hole The weight λ of the reconstruction loss for known regions of the image is set to 6. valid Set to 1, perceptual loss weight λ perceptual The style loss weight λ is set to 0.05. style Set to 120, binary cross-entropy loss weights Set to 0.5, edge adversarial loss weight Set to 0.1, the weight of the reconstruction loss in the structurally missing region. Set to 3, the weight of the reconstruction loss for the structurally known region. Set it to 0.5.

[0146] S7. Feed the test set images into the trained network for testing to obtain the repaired complete images.

[0147] The working principle of this invention is:

[0148] This invention provides an end-to-end image inpainting method based on joint guidance of edge and structural priors. First, the damaged image is input into the network, where features are extracted by encoding. Then, it is fed into a multi-branch decoding layer composed of edge features, structural features, and RGB image features, allowing for unbiased interaction of information among these three elements. After training, a test image to be inpainted can be fed into the network for inpainting, and finally, the inpainted result is output.

[0149] This invention repairs large-area missing images by combining edge and structural priors. The adaptive prior feature fusion module and the design of channel and region-aware collaborative attention fully utilize prior information, resulting in repaired images with better structural consistency and rationality.

[0150] The innovation of this invention lies in:

[0151] This invention presents an end-to-end image inpainting method guided by joint edge and structural prior knowledge. Based on an encoder-decoder architecture, the method first extracts features, which are then input into a decoder. The decoder consists of a main image inpainting branch and two auxiliary branches, used for structural and edge texture information processing, respectively. This network maximizes the integration of edge and structural prior knowledge, effectively guiding the image inpainting task. After model training, the test image to be inpainted can be fed into the network for inpainting, and the inpainted result is output.

[0152] The network of this invention combines prior information with RGB images, focusing on learning unbiased discriminative interactions between multiple branches. Under the joint supervision of structural and edge features, the entire network is trained efficiently in an end-to-end manner.

[0153] This invention designs an adaptive prior feature fusion module, which realizes information sharing and fusion of structural features and edge features. Through a dynamic gating mechanism, it adaptively adjusts the contribution weight of different features, which can not only highlight the importance of different priors in the damaged area, but also effectively suppress redundant and conflicting features.

[0154] This invention develops a channel- and region-aware collaborative attention module. By decoupling dimensions, it first compresses the space to calculate the contribution of the channels, thereby alleviating the semantic differences between prior features and the feature space structure of RGB images. Then, it uses region-aware learning to focus on relevant feature blocks located at distant spatial positions, thus refining the generated content.

[0155] In summary, this invention achieves unbiased guidance from prior information through multi-branch decoding, and designs adaptive prior feature fusion to make full and reasonable use of structural and edge priors. At the same time, it designs channel and region perception collaborative attention to further refine the repair content.

[0156] like Figures 5-7 As shown, to illustrate the effectiveness and adaptability of this invention, this experiment uses three commonly used datasets in the field of image inpainting: the ParisStreetView dataset, the CelebA-HQ dataset, and the Places2 dataset, to verify the effectiveness of this invention. The evaluation metrics for the test results include three items: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

[0157] Figure 5 This is a qualitative comparison result of the present invention on the Paris StreetView dataset; from left to right, the images are damaged, EC, RFR, CTSDG, MMT algorithms, and the restoration results of the present invention, with the last column showing the real images;

[0158] Figure 6 This is a qualitative comparison result of the present invention on the CelebA-HQ dataset; from left to right, the results are: damaged image, EC, RFR, MED, MMT algorithms, and the restoration result of the present invention; the last column is the real image.

[0159] Figure 7 This is a qualitative comparison result of the present invention on the Places2 dataset. From left to right, the results are: damaged image, EC, RFR, CTSDG, MMT algorithm, and the restoration result of the present invention. The last column is the real image.

[0160] Table 1 compares the evaluation metrics of this invention with other repair methods on the Paris StreetView dataset.

[0161]

[0162]

[0163] Table 2 compares the evaluation metrics of this invention with other repair methods on the CelebA-HQ dataset.

[0164]

[0165]

[0166] Table 3 compares the evaluation metrics of this invention with other repair methods on the Places2 dataset.

[0167]

[0168]

[0169] Through comparison, the present invention can achieve better results in image restoration, and all indicators are superior to other methods, which has certain practical value.

[0170] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. An end-to-end image inpainting method guided by joint edge and structure priors, characterized in that: Includes the following steps: S1. Input the damaged image into the model, and first perform feature extraction and encoding through multiple layers of stacked gated convolutions; S2. Feed the features extracted in step S1 into the bottleneck layer composed of the multidimensional context extraction module MCFEM to obtain fully extracted context features. S3. Enter the decoding stage. The features extracted in step S2 are split into three parallel branches: edge features, structural features, and RGB image features. Each branch is input with edge and structural images of the corresponding resolution. S4. Input the edge features and structural features into the adaptive prior feature fusion module APFFM to obtain the fused prior features; S5. The prior features output in step S4 are fed together with the RGB image features into the Channel and Region Aware Collaborative Attention Module (CRASA) to obtain the enhanced features F'. image ; S6. Optimize the network parameters using the Adam optimizer and joint loss function to obtain the trained network; S7. Feed the test set images into the trained network for testing to obtain the repaired complete images.

2. The end-to-end image inpainting method based on edge and structure prior joint guidance according to claim 1, characterized in that: Before step S1, the following steps are also included: Before the data is input into the model, the original image needs to be preprocessed and image enhancement operations are performed. The resolution of the image and the mask is uniformly constrained to a size of 256×256, and the corresponding edge image and structure smoothing image are generated. The kernel size of the Gaussian filter in the Canny algorithm for generating the edge image is set to 2. Then, the original image is subjected to random flipping, random cropping, and random rotation data enhancement operations. Finally, the pixel value range of the image is converted to [-1,1].

3. The end-to-end image inpainting method based on edge and structure prior joint guidance according to claim 1, characterized in that: The specific steps of step S2 include: S201. Let f be the feature map input to the Multidimensional Contextual Feature Extraction (MCFEM) module. h Its size is B×H×W×C; Where B represents the training batch, C represents the number of feature map channels, and H and W represent the height and width of the feature map, respectively. f h The features are divided into 4 groups on average according to the channel dimension, and the number of channels in each group is C / 4. S202. The grouped features from step S201 are processed in parallel through a feature extraction branch based on dilated convolution. Each branch's dilated convolution is set with a different dilation coefficient r and a learnable weight parameter ω. i Subsequently, the four sets of contextual features from different receptive fields were concatenated and aggregated by splicing them together along the channel dimension. This process can be represented as: Among them, f i DilatedConv represents the feature of the channel being averaged across all segments. i The dilated convolution on the i-th branch is shown. Let represent the output corresponding to the i-th branch, and [;] represent concatenation along the channel dimension. S203, Feature map f of the Multidimensional Contextual Feature Extraction Module (MCFEM) h The gate control value g is calculated using a 1×1 ordinary convolution and a sigmoid function, and then used to adjust f. h and f r We perform weighted summation to obtain the output features. This process can be represented as:

4. The end-to-end image inpainting method based on edge and structure prior jointly guided according to claim 1, characterized in that: The specific steps of step S4 include: S401. Let the edge features updated by the decoder be F. e The structural feature is F s Average pooling and max pooling operations are performed separately, and then the results of the two pooling operations are concatenated along the channel dimension to obtain the edge comprehensive pooling feature. Combined pooling features of the results This process can be represented as: S402, Edge Pooling Feature Combined pooling features of the results First, the features are transformed in channel dimension using 1×1 convolution, and then a non-linear transformation is performed using the LeakyReLU activation function. Next, the features are further updated using 3×3 convolution, and the corresponding edge feature weight map G is calculated using the Sigmoid activation function. e and structural feature weight map G s ; As shown below: G e =Sigmoid(Conv2(ReLU(Conv1(F e p )))) G s =Sigmoid(Conv2(ReLU(Conv1(F s p )))) S403, Edge-sensing feature F e 'By analyzing the original structural features F s With weighted edge features G e ⊙F e Pixel-level addition is performed to obtain it; similarly, using G... s Calculate the structure-aware feature F s ', and finally F' e and F′ s Merge into F prior As the final prior feature representation.

5. The end-to-end image inpainting method based on edge and structure prior joint guidance according to claim 1, characterized in that: The specific steps of step S5 include: S501. The SPADE normalization method is used to normalize the prior features F. prior Injected into RGB image features F image The formula is as follows: Where F * This represents the transformed feature map, where γ and β are modulation parameters learned from prior features, and μ... c and σ c These represent the mean and standard deviation of the feature at channel c, respectively. S502, for F * Perform average pooling to scale its resolution from H×W to H'×W': Among them, AvgPool (7×7) This represents a pooling operation with a kernel size of 7×7. S503, Use Instance Normalization(IN) to Perform a normalization operation, and then calculate the attention along the channel dimension: Among them, F proj This represents the mapping function that generates Q, K, and V, and DWConv represents depthwise separable convolution. S504. The attention coefficients between channels are calculated using Softmax, adaptive pooling is used to scale the resolution to C×C, and then Sigmoid activation is applied. Finally, the enhanced channel features are multiplied channel by channel with the input features to obtain the enhanced channel features. The specific formula is as follows: Where d is the dimension of vector K, which is usually used to scale the size of the dot product. S505, the result obtained in step S504 The gated mask M is obtained after passing through a gated convolutional layer (GC). G Display the feature F that distinguishes the missing region. f and known region features F b : S506. Extract 3×3 pixel patches and calculate their cosine similarity. Use softmax to weight the similarity and obtain the attention score for each patch. As shown below: S507, through coordinated attention of channels and space, yields the final output:

6. The end-to-end image inpainting method based on edge and structure prior joint guidance according to claim 1, characterized in that: The specific steps of step S6 include: S601. During training, the number of samples fed in each time is: batch_size = 16, and a binary mask M is randomly generated, where 1 represents a known region and 0 represents a missing region. S602. The loss function of the network is calculated as follows: L total =L inpaint +L edge +L st Among them, L inpaint L represents the image reconstruction loss. edge L represents the edge loss. st The reconstruction loss is represented by the Adam algorithm, which is used to minimize the loss function and continuously update the network parameters until convergence. In step S602, L inpaint Pixel reconstruction loss L, including known and missing regions hole and L valid Perceived loss L perceptual and style loss L style : Where Ψ represents the activation feature map obtained using the pre-trained model VGG-16, and the expected value E is used to average the loss of all feature layers, G j This indicates that the feature map Ψ is activated at layer j. j The constructed Gram matrix. L edge Includes generative adversarial loss and multi-scale binary cross-entropy loss: Where, N s It is the number of edge map pixels at the s-th scale. and These are the edge prediction value and the true label at the s-th scale, respectively, G e For edge branch generator, D e For discriminator. L st Structural pixel reconstruction loss for known and missing regions and

Images