A high-fidelity image inpainting method based on mask-aware attention network

Abstract

The application discloses a high-fidelity image restoration method based on a mask perception attention network, and steps are as follows: S1, receiving an original image and a binary mask, and merging image content and missing geometry information into unified features by using a gating embedding layer; S2, entering a three-level deep separable mask perception downsampling module, using deep convolution to reduce dimensions at each level, filtering invalid pixels by using a gating signal generated by a mask, finally using point-by-point convolution to double channels and adding residuals, and forming a pyramid feature; S3, entering an inflation convolution attention module, running inflation convolution with inflation rates of 1, 2, 4 and 8 at different layers, then using channel attention and spatial attention two-way weighting, and adding residuals of the result and original features; S4, using PixelShuffle to perform layer-by-layer upsampling, and jump splicing with the same scale encoder features, and finally outputting a restored image through 1*1 convolution; and S5, combining edge perception contrast and semantic contrast loss joint training to ensure sharp edges and consistent semantics.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and specifically to a high-fidelity image restoration method based on a mask-aware attention network. Background Technology

[0002] Image restoration technology aims to reconstruct missing or damaged parts of images using intelligent techniques, ensuring consistency between the restored area and the original image in terms of visual effect and semantic content. It has wide applications in fields such as cultural heritage restoration, medical imaging, and visual effects. Traditional methods mainly rely on manually designed prior knowledge and local feature matching, such as the PatchMatch algorithm. While it performs well in handling small-scale regular defects, it suffers from structural ambiguity and texture discontinuities under complex defects and struggles to adapt to diverse scenarios, such as poor restoration results on datasets with unique artistic styles like the Dunhuang murals.

[0003] The emergence of deep learning has driven the development of image inpainting technology. Generative Adversarial Network (GAN)-based models generate realistic details through adversarial training, but existing methods still have limitations: on the one hand, the limited receptive field of traditional convolution leads to insufficient fusion of multi-scale features, and details are easily blurred under complex and irregular masks; on the other hand, adversarial training has instability and pattern collapse problems.

[0004] The main reasons for the aforementioned shortcomings are: traditional convolutional methods struggle to effectively handle high-resolution, irregular masked regions, and are prone to losing multi-scale information during feature fusion, resulting in insufficient detail reconstruction capabilities. Existing deep learning methods separate global and local modeling; for example, traditional self-attention mechanisms have high computational complexity, limiting their application in high-resolution images. While local window attention mechanisms reduce computation, they disrupt the continuity between missing and visible regions. Models within the GAN framework are prone to mode collapse and gradient instability, are sensitive to the scale and quality of training data, and perform poorly on small-sample or low-quality mural data. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned technical deficiencies by providing a high-fidelity image restoration method based on a mask-aware attention network, applicable to fields such as cultural heritage restoration, facial image restoration, and photo editing. For example, in the restoration of Dunhuang murals, it can accurately locate damaged areas of the murals based on surrounding texture and color characteristics and automatically generate new content that conforms to the original style. In photo editing, it can effectively remove unwanted objects from photos while maintaining the overall structure and semantic consistency of the image.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A high-fidelity image inpainting method based on a mask-aware attention network, comprising the following steps: S1. Receive the original image to be repaired and its corresponding binary mask, and use a gated embedding layer to merge the image content and the missing geometric information into a unified feature with 32 channels. S2. Then, the three-level depth-separable mask-aware downsampling module is entered. Each level first uses depth convolution to extract features, then uses the gating signal generated by the mask to filter invalid pixels, and finally uses pointwise convolution to double the channels and add the residuals to form a pyramid-shaped feature. S3. Enter the dilated convolution attention module, run dilated convolutions with dilation rates of 1, 2, 4, and 8 in different layers, and then use dual-path weighting with channel attention and spatial attention to take into account both local texture and global structure. The result is added to the original feature residual. S4. Upsample layer by layer using PixelShuffle and splice the features with the encoder at the same scale in a skip pattern. The final image is then output as a 1×1 convolution. S5. The discriminator is jointly trained by edge-aware contrast loss and semantic contrast loss to ensure sharp edges and semantic consistency.

[0007] As an improvement to the above technical solution, the gated embedding layer consists of 3×3 image convolution and 1×1 mask convolution. The two independent convolution branches extract image texture and defect geometric information respectively. The image texture and defect geometric information extracted by each independent convolution branch are then concatenated and spatial gate weights are generated by 1×1 gated convolution. The two types of features are fused pixel by pixel and combined into a unified feature of 32 channels.

[0008] As an improvement to the above technical solution, in step S2, the depth-separable mask-aware downsampling module has three layers, with a stride of 2 for each depth convolution. The mask gate uses 1×1 depth convolution and 3×3 pointwise convolution, doubling the number of pointwise convolution channels, and the residual connections preserve information. The specific steps are as follows: S201. Feature map of the input original image to be repaired. and binary mask First, depthwise convolution (DWConv) is performed on each... and Processing is performed to obtain an intermediate representation. and : ; ; S202, will and Expand to 4 channels using 1×1 convolution, then use pointwise convolution (PConv) to... Feature extraction is performed using the expanded mask: ; ; S203, Creating a channel dimension is The all-zero tensors are merged and expanded into a mask by cyclically interleaving the channel dimensions. and feature map , obtain the fusion tensor ; when , At that time, the mask channel is used cyclically; when hour, as well as ; when hour, as well as ; in, Modulo operation is represented. express The remainder when divided by 4; S204, Through Convolutional compression fusion tensor The number of channels, and add the original feature map. The residual connections are used to obtain the final output. : .

[0009] As an improvement to the above technical solution, in step S3, the dilated convolutional attention module uses dilated convolutions with dilation rates of 1, 2, 4, and 8 in different layers, and then superimposes channel-space dual attention, followed by LayerNorm+ReLU at each layer; the specific steps are as follows: S301, For the input feature map Calculate statistics for the channel dimension, and generate two values ​​using average pooling and max pooling. The two vectors are shared. After convolution, a channel attention map is generated using the Sigmoid activation function. : ; S302, Input Features Perform multi-scale dilated convolution to obtain Then with channel attention map Element-wise multiplication yields the feature representation. : ; ; S303, to Perform global average pooling and max pooling along the channel dimension to obtain two... Feature maps are concatenated along the channel dimension to form a tensor; S304, then through Convolution reduces the number of channels to 1, generating a spatial attention map. : ; S305, to After performing multi-scale dilated convolution, it is combined with the spatial attention map. Element-wise multiplication yields the enhanced features. : ; .

[0010] As an improvement to the above technical solution, in step S4, upsampling is performed using PixelShuffle×2, and spliced ​​with features from the same-scale encoder, with the end output after 1×1 convolution + Tanh.

[0011] As an improvement to the above technical solution, the specific steps of step S5 are as follows: S501. Using the Sobel operator to compute the real graph. Single-channel edge map : ; Discriminator Extract and repair image Real pictures and mask image High-level characteristics , , : ; S502. Generate an edge attention weight map based on the edge map of the real image. And interpolate it to the discriminator feature map space size: ; Using weight graphs Weighted modulation of the feature map: ; Calculate positive sample pairs and negative sample pairs Cosine similarity: ; ; The edge-aware contrastive loss uses the InfoNCE form, optimizing the model by comparing the similarity of positive and negative sample pairs: ; S503, Calculate and generate image Real pictures and mask image The feature vectors are extracted by the discriminator and then L2 normalized: ; ; ; S504. Calculate the similarity between the anchor point (generated image feature vector) and the positive sample (real image feature vector) and the negative sample (mask image feature vector): ; ; Semantic contrastive loss ensures that the semantic similarity between the anchor point and the positive sample is at least a preset threshold higher than the semantic similarity between the anchor point and the negative sample. ; Overall contrastive loss function: The final contrastive loss function is a weighted sum of the edge-aware contrastive loss and the semantic contrastive loss: .

[0012] As an improvement to the above technical solution, in step S5, the discriminator is a variant of U-Net with a 4×4 convolution kernel, stride=2, spectral normalization + LeakyReLU(0.2), and output spatial size H / 16×W / 16.

[0013] As an improvement to the above technical solution, in step S5, the AdamW optimizer is used during the training phase, with a generator learning rate of 1e-4, a discriminator learning rate of 4e-4, a batch size of 32, a period of 40 epochs, cosine annealing to 1e-6, and gradient clipping max_norm=1.0; the training loss includes six terms: reconstruction, perception, style, adversarial, edge contrast, and semantic contrast, with weights of 1, 1, 250, 0.1, 0.1, and 0.1 respectively.

[0014] Compared with the prior art, the advantages and positive effects of this invention are: This invention presents a high-fidelity image restoration method based on a mask-aware attention network. It can run directly on any general-purpose CPU / GPU without requiring quantization or dedicated chips, and can be applied to fields such as cultural heritage restoration, facial image restoration, and photo editing. For example, in the restoration of Dunhuang murals, it can accurately locate damaged areas of the murals based on surrounding texture and color characteristics and automatically generate new content that conforms to the original style. In photo editing, it can effectively remove unwanted objects from photos while maintaining the overall structure and semantic consistency of the image.

[0015] More specifically, this is manifested in: 1. Efficient Feature Fusion and Detail Reconstruction: The DMD module solves the information loss problem in traditional downsampling. Through a channel reuse strategy, it maintains the geometric and semantic consistency of irregular masks while reducing image resolution, effectively fusing image features and mask information to provide a more accurate feature basis for subsequent restoration. DCAM innovatively combines dilated convolution and dual-path attention mechanisms, which can capture a wide range of channel dependencies and strengthen attention to discriminative spatial regions, achieving local texture restoration and global structural consistency optimization, and improving the model's ability to reconstruct details when dealing with complex irregular masks.

[0016] 2. Stable Adversarial Training and Semantic Alignment: The contrastive learning-enhanced adversarial training framework effectively mitigates the pattern collapse problem in GAN training by introducing a multi-level contrastive loss function. The edge-aware contrastive loss focuses on the fidelity of the edge structure of the restoration region, guiding the model to generate restoration results with fine edge structures; the semantic contrastive loss ensures the semantic consistency between the restoration region and the entire image, making the restoration content more reasonable at the global semantic level, thereby improving the quality and realism of image restoration.

[0017] 3. Superior repair performance and generalization ability: such as Figure 4 , 5 As shown in Figures 6 and 7, M2ANet can generate image restoration results with natural perceived quality and rich detail. It can effectively restore face images and street scene images, and performs exceptionally well in specific scenarios such as cultural heritage restoration. It effectively solves the image restoration challenges of complex textures, gradient colors, and irregularly damaged areas, such as those found in Dunhuang murals, and demonstrates good adaptability to various scenarios and datasets. Furthermore, as... Figure 8 As shown, M2ANet performs exceptionally well in object removal tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 Here is a diagram of the M2ANet model framework; Figure 2 DMD framework diagram; Figure 3 This is a DCAM framework diagram; Figure 4 Comparison images of face restoration results; Figure 5 Comparison of Dunhuang mural restoration results Figure 1 ; Figure 6 Comparison of Dunhuang mural restoration results Figure 2 ; Figure 7 Comparison images of street view restoration results; Figure 8 Image showing the effect of removing an object. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figures 1-8As shown, the high-fidelity image inpainting method based on a mask-aware attention network—M2ANet—of this invention can run directly on any general-purpose CPU / GPU without requiring quantization or dedicated chips. After receiving an RGB image and its binary mask, the system first extracts image texture and defect geometric information using two independent convolutional branches, then combines them into a unified 32-channel feature through gated fusion. Subsequently, it enters a three-level depth-separable mask-aware downsampling module: each level first extracts features using depth convolution, then filters invalid pixels using a gated signal generated by the mask, and finally doubles the channels using pointwise convolution and adds the residuals to form a pyramid-shaped feature. The pyramid feature then enters a dilated convolution attention module: dilated convolutions with dilation rates of 1, 2, 4, and 8 are run in different layers, then weighted using both channel attention and spatial attention to consider both local texture and global structure. The result is added to the original feature residual. Next, PixelShuffle upsampling is used layer by layer, and the image is spliced ​​with features from the same-scale encoder. The final image is output via a 1×1 convolution. The entire network is supervised by a U-Net discriminator, and incorporates both edge-aware contrast and semantic contrast losses to ensure sharp edges and semantic consistency. The system provides a Python API, and a single 256-image output is completed in approximately 1 second on an 8-thread CPU, maintaining a PSNR above 27dB and an FID below 30, offering advantages such as cross-platform compatibility and ease of deployment.

[0022] (II) Network Architecture like Figure 1 As shown, the generator of M2ANet adopts the classic "encoder-decoder" U-Net architecture, running at float32 precision throughout. The input receives both RGB images and binary masks simultaneously. The two are first processed by shallow convolution to extract texture and geometric information, and then fused through gating to form a unified 32-channel feature, providing prior knowledge of defect boundaries for subsequent networks.

[0023] The encoding part consists of a three-level depth-separable mask-aware downsampling module: each level first uses a 3×3 depth convolution with a stride of 2 for downsampling, then uses a gating signal generated by the mask to suppress missing pixels pixel by pixel, and then uses a 1×1 convolution to double the channels and add the residuals to form a channel pyramid of 36→72→144→288, with the spatial size being halved in turn.

[0024] At the bottleneck, a "dilated convolution attention module" is added: three parallel 1×1 dilated convolutions with different dilation rates (dilation rates 1, 2, 4, 8) expand the receptive field, and then channel attention (global statistics + MLP) and spatial attention (7×7 convolution) are superimposed with dual-path weighting to enhance the consistency between local texture and global structure.

[0025] The decoding part consists of three layers of PixelShuffle×2 upsampling. Each layer is spliced ​​with the same scale encoded features in a skip pattern. After the channels are halved, 3×3 convolution is used for thinning. Finally, a 1×1 convolution + Tanh is used to output the repaired result with the same size as the original image.

[0026] The discriminator uses a variant of U-Net and is jointly trained using adversarial loss, edge-aware contrastive loss, and semantic contrastive loss to ensure sharp edges and semantic consistency. Training is performed using AdamW for 40 epochs, and inference supports arbitrary resolution.

[0027] (III) Core Technology Module 1. Depth-Separable Mask-Aware Downsampling Module (DMD) Design Principle: To address the geometric inconsistencies and valid / invalid pixel mixing issues caused by traditional downsampling methods, the DMD module employs depthwise separable convolution to dynamically fuse image features and mask information. While reducing image resolution, it maintains geometric and semantic consistency of irregular masks through a channel reuse strategy.

[0028] Detailed operation steps: like Figure 2 As shown, for the input feature map and binary mask First, depthwise convolution (DWConv) is used to... and Processing is performed to obtain an intermediate representation. and : ; ; Will pass The convolution is expanded to 4 channels, and then pointwise convolution (PConv) is used to process each channel separately. Feature extraction is performed using the expanded mask: ; ; Create channel dimension The all-zero tensors are merged and expanded into a mask by cyclically interleaving the channel dimensions. and feature map , obtain the fusion tensor .when , When, the mask channel is used cyclically. hour, as well as ,when hour, as well as

[0029] here, Modulo operation, i.e. express The remainder when divided by 4; Finally, through Convolutional compression fusion tensor The number of channels, and add the original feature map. The residual connections are used to obtain the final output. : .

[0030] 2. Dilated Convolutional Attention Module (DCAM) Design principles: such as Figure 3 As shown, DCAM is divided into a channel attention module and a spatial attention module. Based on the extended CBAM architecture, DCAM introduces dilated convolution into the channel attention and spatial attention mechanisms to capture a wider range of channel dependencies and enhance attention to discriminative spatial regions, thereby achieving local texture restoration and global structural consistency optimization.

[0031] Detailed operation steps: Channel attention module: For the input feature map Calculate statistics for the channel dimension, and generate two values ​​using average pooling and max pooling. The two vectors are shared. After convolution, a channel attention map is generated using the Sigmoid activation function. : ; For input features Perform multi-scale dilated convolution to obtain Then with channel attention map Element-wise multiplication yields the feature representation. : ; ; Spatial attention module: right Perform global average pooling and max pooling along the channel dimension to obtain two... Feature maps are concatenated along the channel dimension to form a The tensor. Then through Convolution reduces the number of channels to 1, generating a spatial attention map. : ; right After performing multi-scale dilated convolution, it is combined with the spatial attention map. Element-wise multiplication yields the enhanced features. : ; .

[0032] 3. A contrastive learning-enhanced adversarial training framework Design principle: To solve the problems of mode collapse and gradient instability in GAN training, this scheme introduces a multi-level contrastive loss function in the discriminator. By constructing positive sample pairs of "repaired image-real image" and negative sample pairs of "repaired image-mask image", the distribution of generated images in the U-Net feature space is constrained, making the repaired image closer to the real image.

[0033] Detailed operation steps: Edge-aware contrast loss: Calculate the real image using the Sobel operator. Single-channel edge map : ; Discriminator Extract and repair image Real pictures and mask image High-level characteristics , , : ; Generate an edge attention weight map based on the edge map of the real image. And interpolate it to the discriminator feature map space size: ; Using weight graphs Weighted modulation of the feature map: ; Calculate positive sample pairs and negative sample pairs Cosine similarity: ; ; The edge-aware contrastive loss uses the InfoNCE form, optimizing the model by comparing the similarity of positive and negative sample pairs: ; Semantic contrast loss: Calculate and generate image Real pictures and mask image The feature vectors are extracted by the discriminator and then L2 normalized: ; ; ; Calculate the similarity between the anchor point (generated image feature vector) and the positive sample (real image feature vector) and the negative sample (mask image feature vector): ; ; Semantic contrastive loss ensures that the semantic similarity between the anchor point and the positive sample is at least a preset threshold higher than the semantic similarity between the anchor point and the negative sample. ; Overall contrastive loss function: The final contrastive loss function is a weighted sum of the edge-aware contrastive loss and the semantic contrastive loss: .

[0034] (iv) Model training and optimization 1. Dataset and Data Preprocessing Experiments were conducted on multiple public datasets, including ParisStreetView, CelebA-HQ, and the Dunhuang Mogao Grottoes mural dataset. Irregular masks were used to simulate missing regions in the images, and the datasets were divided into six levels (1%-10%, 10-20%, 20%-30%, 30%-40%, 40%-50%, 50%-60) based on the proportion of defects to construct a standardized evaluation environment.

[0035] 2. Experimental Setup Run on a high-performance computing system using the PyTorch deep learning framework. Configure the number of DCAM modules and the dilation rate of dilated convolutions. Use the Adam optimizer and set the learning rate and momentum parameters.

[0036] 3. Loss Function The total loss function of M2ANet includes reconstruction loss, perceptual loss, style loss, adversarial loss, and contrastive loss: Reconstruction loss: Calculate the output image and real picture Pixel-level L1 differences: ; Perceptual loss: Using feature maps from the VGG-19 network, the high-level semantic differences between the restored image and the original image are calculated. ; Style loss: Employing the Gram matrix of the VGG network to reduce texture artifacts in the generated images. ; Adversarial loss: Improve the perceptual quality of images by normalizing the PatchGAN discriminator through spectral normalization. ; Total loss function The above loss functions are then summed using weighted methods. .

[0037] 4. Training Process Phase 1 Data Preparation and Enhancement First, images are randomly selected from public datasets (ParisStreetView, CelebA-HQ, Dunhuang murals) and irregular masks are used. The masks undergo synchronous geometric transformation. All data are uniformly normalized to [-1,1] before being fed into the network to ensure that the training samples fully cover real-world application scenarios in terms of distribution, scale, and color.

[0038] Phase Two: Network Initialization and Hardware Configuration The PyTorch 2.1 framework was used, maintaining a float32 weight throughout the entire process. Both generator and discriminator weights were initialized using KaimingNormal, with biases set to zero. Training was performed on a 24GB RTX-4090 GPU, with mixed precision enabled (fp16 calculation, fp32 main weights). The batch size per GPU was set to 8, gradient accumulation was performed in 4 steps, resulting in an effective batch size of 32. Data loading used an 8-thread Dataloader with pin_memory=True to avoid CPU-GPU copy bottlenecks.

[0039] Third-stage optimizer and learning rate scheduling The generator and discriminator use the AdamW optimizer: β1=0.9, β2=0.999, weight_decay=1e-4, eps=1e-8. The learning rate strategy employs "warm-up + cosine annealing": during the 5000 iterary warm-up, the learning rate linearly increases from 0 to 1e-4 (generator) / 4e-4 (discriminator), then smoothly decreases to 1e-6 using a cosine function. A checkpoint is saved every 10000 iterary, and the EMA shadow weights (α=0.9999) are updated simultaneously. During the inference phase, the EMA weights are directly called to prevent performance fluctuations caused by training oscillations.

[0040] Fourth-stage loss combination and gradient management During training, six losses are computed in parallel: reconstruction, perception, style, adversarial, edge contrast, and semantic contrast, with weights of 1, 1, 250, 0.1, 0.1, and 0.1, respectively. Before backpropagation after summing all losses, gradient pruning is performed on the gradients of the generator and discriminator respectively before feeding them into the optimizer, effectively preventing gradient explosion. The generator is updated once for every update of the discriminator, maintaining the classic adversarial rhythm of GANs.

[0041] Phase 5 Monitoring and Early Shutdown Training logs are recorded every 100 steps, with real-time visualization of PSNR, SSIM, and LPIPS metrics. The validation set consists of 200 images not used in training, evaluated every epoch. If the FID stops decreasing for 5 consecutive epochs, early stopping is triggered immediately, preserving the optimal checkpoint. The entire training cycle takes approximately 36-40 hours to complete 40 epochs (256×256 resolution).

[0042] Phase 6 Inference Deployment and Post-processing After training, export the .pth weights. During the inference phase, turn off gradients and enable EMA weights. The final output is then processed by LAB color consistency correction and UnsharpMask sharpening, and output as PNG or JPEG.

[0043] (V) Model Reasoning and Application 1. Reasoning Steps During the testing phase, the damaged image and its corresponding mask are input into the generator. The generator extracts features through the embedding layer, performs feature encoding and decoding through multiple Transformer cores, and gradually restores image details using the DMD and DCAM modules, finally outputting the repaired image.

[0044] 2. Application Expansion This method can be applied to fields such as cultural heritage restoration, facial image restoration, and photo editing. For example, in the restoration of Dunhuang murals, it can accurately locate damaged areas of the murals based on the surrounding texture and color characteristics and automatically generate new content that conforms to the original style. In photo editing, it can effectively remove unwanted objects from photos while maintaining the overall structure and semantic consistency of the image.

[0045] (vi) Advantages of this invention: 1. Efficient Feature Fusion and Detail Reconstruction: The DMD module solves the information loss problem in traditional downsampling. Through a channel reuse strategy, it maintains the geometric and semantic consistency of irregular masks while reducing image resolution, effectively fusing image features and mask information to provide a more accurate feature basis for subsequent restoration. DCAM innovatively combines dilated convolution and dual-path attention mechanisms, which can capture a wide range of channel dependencies and strengthen attention to discriminative spatial regions, achieving local texture restoration and global structural consistency optimization, and improving the model's ability to reconstruct details when dealing with complex irregular masks.

[0046] 2. Stable Adversarial Training and Semantic Alignment: The contrastive learning-enhanced adversarial training framework effectively mitigates the pattern collapse problem in GAN training by introducing a multi-level contrastive loss function. The edge-aware contrastive loss focuses on the fidelity of the edge structure of the restoration region, guiding the model to generate restoration results with fine edge structures; the semantic contrastive loss ensures the semantic consistency between the restoration region and the entire image, making the restoration content more reasonable at the global semantic level, thereby improving the quality and realism of image restoration.

[0047] 3. Superior repair performance and generalization ability: such as Figure 4 , 5 As shown in Figures 6 and 7, M2ANet can generate image restoration results with natural perceived quality and rich detail. It can effectively restore face images and street scene images, and performs exceptionally well in specific scenarios such as cultural heritage restoration. It effectively solves the image restoration challenges of complex textures, gradient colors, and irregularly damaged areas, such as those found in Dunhuang murals, and demonstrates good adaptability to various scenarios and datasets. Furthermore, as... Figure 8 As shown, M2ANet performs exceptionally well in object removal tasks.

[0048] Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by all other embodiments obtained by those skilled in the art without creative effort should be included within the protection scope of the present invention.

Claims

1. A high-fidelity image inpainting method based on a mask-aware attention network, characterized in that: The steps of this method are: S1. Receive the original image to be repaired and its corresponding binary mask, and use a gated embedding layer to merge the image content and the missing geometric information into a unified feature with 32 channels. S2. Then, the three-level depth-separable mask-aware downsampling module is entered. Each level first uses depth convolution to extract features, then uses the gating signal generated by the mask to filter invalid pixels, and finally uses pointwise convolution to double the channels and add the residuals to form a pyramid-shaped feature. S3. Enter the dilated convolution attention module, run dilated convolutions with dilation rates of 1, 2, 4, and 8 in different layers, and then use dual-path weighting with channel attention and spatial attention to take into account both local texture and global structure. The result is added to the original feature residual. S4. Upsample layer by layer using PixelShuffle and splice the features with the encoder at the same scale in a skip pattern. The final image is then output as a 1×1 convolution. S5. The discriminator is jointly trained by edge-aware contrast and semantic contrast loss to ensure sharp edges and semantic consistency. The specific steps of step S5 are as follows: S501. Using the Sobel operator to compute the real graph. Single-channel edge map : ； Discriminator Extract and repair image Real pictures and mask image High-level characteristics , , : ； S502. Generate an edge attention weight map based on the edge map of the real image. And interpolate it to the discriminator feature map space size: ； Using weight graphs Weighted modulation of the feature map: ； Calculate positive sample pairs and negative sample pairs Cosine similarity: ； ； The edge-aware contrastive loss uses the InfoNCE form, optimizing the model by comparing the similarity of positive and negative sample pairs: ； S503, Calculate and generate image Real pictures and mask image The feature vectors are extracted by the discriminator and then L2 normalized: ； ； ； S504. Calculate the similarity between the anchor point and the positive and negative samples: ； ； Semantic contrastive loss ensures that the semantic similarity between the anchor point and the positive sample is at least a preset threshold higher than the semantic similarity between the anchor point and the negative sample. ； Overall contrastive loss function: The final contrastive loss function is a weighted sum of the edge-aware contrastive loss and the semantic contrastive loss: 。 2. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 1, characterized in that: The gated embedding layer consists of 3×3 image convolutions and 1×1 mask convolutions. The two independent convolutional branches extract image texture and defect geometric information, respectively. The image texture and defect geometric information extracted by each independent convolutional branch are then concatenated and spatially gated weights are generated through 1×1 gated convolutions. The two types of features are fused pixel by pixel. Combine the two into a unified 32-channel feature.

3. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 1, characterized in that: In step S2, the depth-separable mask-aware downsampling module has three layers, with a stride of 2 for each depth convolution. Mask gating uses 1×1 depth convolution and 3×3 pointwise convolution, doubling the number of channels in the pointwise convolution, and preserving information through residual connections. The specific steps are as follows: S201. Feature map of the input original image to be repaired. and binary mask First, depthwise convolution is performed on each... and Processing is performed to obtain an intermediate representation. and : ； ； S202, will and Expand to 4 channels using 1×1 convolution, then use pointwise convolution to... Feature extraction is performed using the expanded mask: ； ； S203, Creating a channel dimension is The all-zero tensors are merged and expanded into a mask by cyclically interleaving the channel dimensions. and feature map , obtain the fusion tensor ; when , At that time, the mask channel is used cyclically; when hour, as well as ; when hour, as well as ; in, Modulo operation is represented. express The remainder when divided by 4; S204, Through Convolutional compression fusion tensor The number of channels, and add the original feature map. The residual connections are used to obtain the final output. : .

4. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 1, characterized in that: In step S3, the dilated convolutional attention module uses dilated convolutions with dilation rates of 1, 2, 4, and 8 in different layers, followed by channel-space dual attention, and each layer is then connected to LayerNorm+ReLU; the specific steps are as follows: S301, For the input feature map Calculate statistics for the channel dimension, and generate two values ​​using average pooling and max pooling. The two vectors are shared. After convolution, a channel attention map is generated using the Sigmoid activation function. : ； S302, Input Features Perform multi-scale dilated convolution to obtain Then with channel attention map Element-wise multiplication yields the feature representation. : ； ； S303, to Perform global average pooling and max pooling along the channel dimension to obtain two... Feature maps are concatenated along the channel dimension to form a tensor; S304, then through Convolution reduces the number of channels to 1, generating a spatial attention map. : ； S305, to After performing multi-scale dilated convolution, it is combined with the spatial attention map. Element-wise multiplication yields the enhanced features. : ； 。 5. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 1, characterized in that: In step S4, upsampling is performed using PixelShuffle×2 and spliced ​​with features from the encoder at the same scale. The end is output after 1×1 convolution + Tanh.

6. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 1, characterized in that: In step S5, the discriminator is a variant of U-Net with a 4×4 convolution kernel, stride=2, spectral normalization + LeakyReLU(0.2), and output spatial size H / 16×W / 16.

7. The high-fidelity image inpainting method based on a mask-aware attention network according to claim 6, characterized in that: In step S5, the AdamW optimizer is used during the training phase, with a generator learning rate of 1e-4, a discriminator learning rate of 4e-4, a batch size of 32, a period of 40 epochs, cosine annealing to 1e-6, and gradient clipping max_norm=1.

0. The training loss includes six terms: reconstruction, perception, style, adversarial, edge contrast, and semantic contrast, with weights of 1, 1, 250, 0.1, 0.1, and 0.1, respectively.

Images