A fine-grained image inpainting network based on feature enhancement strategy

Abstract

The application provides a fine-grained image repairing network based on a feature enhancement strategy, including the following steps: sending an original incomplete image to be repaired into a coarse-grained global repairing network to obtain a global coarse-grained image; sending the global coarse-grained image into a fine-grained local repairing network to obtain a local fine-grained image; sending the local fine-grained image into a fine-grained attention supervision network to obtain an image weighted by fine-grained feature importance; sending the image weighted by fine-grained feature importance into a fine-grained global repairing network to obtain a global fine-grained image, thereby completing the repairing of the original incomplete image; the coarse-grained global repairing network in the first stage is used for completing the general semantic repairing of the image; the fine-grained local repairing network in the second stage is used for reconstructing the texture and details of the image; the fine-grained global repairing network in the third stage is used for finely optimizing the structure and semantics of the whole image; and the image repairing capacity of the image repairing network is improved.

Description

Technical Field

[0001] This invention relates to the technical field of image inpainting networks, and more specifically to a fine-grained image inpainting network based on a feature enhancement strategy. Background Technology

[0002] Image restoration is the process of using various computer algorithms and image processing techniques to reasonably infer and reconstruct damaged areas based on known information from undamaged areas of an image, making them as close as possible to the appearance and information of the original image. Image restoration technology has important applications in many fields, including image editing, image super-resolution, photo restoration, removal of occlusions from images, restoration of images damaged during transmission and storage, and cultural relic preservation. Therefore, the goal of image restoration should be to make the restored image as visually reasonable in terms of structure, texture, and coherence as possible. Traditional image restoration methods, which utilize low-level features such as local relationships between pixels, edge information, and texture statistics, still cannot achieve semantic restoration even after gradually balancing the effects of structural and texture restoration. In recent years, with the continuous advancement of computing hardware, deep learning has made significant breakthroughs in the field of computer vision, resulting in a large number of new models and technologies, which has led to a booming development of deep learning-based image restoration methods. With the introduction of methods such as context encoders (CE), convolutional neural networks (CNN), and generative adversarial networks (GAN), image restoration technology has made remarkable progress. Deep learning-based image inpainting methods, trained on large-scale datasets, automatically learn high-level features and complex semantic information of images through deep convolutional neural networks or generative adversarial networks, and can generate semantically consistent and visually plausible inpainting results. However, many deep learning-based image inpainting methods still suffer from problems such as overall or local structural errors and blurred details when repairing complex scenes with large or irregular defect areas. Summary of the Invention

[0003] To address the problems in existing technologies, this invention provides a fine-grained image inpainting network based on a feature enhancement strategy, aiming to improve the image inpainting capabilities of the network.

[0004] A fine-grained image inpainting network based on a feature enhancement strategy includes the following steps:

[0005] Step 1: Transfer the original damaged image I to be repaired in The image is fed into a coarse-grained global insulation network, and a global coarse-grained image I1 is obtained.

[0006] Step 2: The global coarse-grained image I1 and the discrimination result of the global coarse-grained image I1 after passing through the discriminator are fed into the fine-grained local repair network to obtain the local fine-grained image I2.

[0007] Step 3: Feed the local fine-grained image I2 into the fine-grained attention supervision network to obtain an image weighted by the importance of fine-grained features.

[0008] Step 4: Weight the image based on the importance of fine-grained features The image is fed into a fine-grained global insulation network to obtain a global fine-grained image I3, thus completing the insulation of the original incomplete image I. in Repair.

[0009] Furthermore, both the coarse-grained global inpainting network and the fine-grained global inpainting network adopt a CNN-based encoder-decoder architecture. The encoder contains eight downsampling layers and is used to extract and compress features from the input image, transforming the image information into a higher-level abstract feature representation. The decoder contains eight upsampling layers and is used to remap the learned abstract feature representation back to the space of the input image, restoring it into an output image of the same size as the input image.

[0010] Furthermore, a feature enhancement network is set between the encoder and the decoder. The feature enhancement network is used to connect the feature maps in the first four downsampling layers of the encoder with the feature maps in the corresponding layers of the decoder.

[0011] Furthermore, the feature enhancement network includes:

[0012] The input feature map F is obtained in the downsampling layer corresponding to the encoder. x Obtain feature map F″ containing deep, fine-grained information. x and residual feature map F y ;

[0013] F′ x =Φ n (F x (17)

[0014] F″ x =Conv(F x ||F′ x (18)

[0015] F y =Conv(F x (19)

[0016] Where, Φ n (·) represents a set of linear transformation operations, || represents a channel-level concatenation operation, and Conv(·) represents a convolution operation that compresses the feature map.

[0017] The feature map F″ containing deep, fine-grained information x With residual feature map F yThe features are concatenated along the channel dimension and then compressed using convolution to obtain the output feature map E. x :

[0018] E x =Conv(F y ||F″ x (20)

[0019] Furthermore, the fine-grained local repair network adopts a CNN-based encoder-decoder architecture and includes two downsampling layers, two FFC residual blocks, and two upsampling layers. The FFC residual blocks pass the input features layer by layer through several cascaded fully connected layers and perform residual connections, finally outputting the result.

[0020] Furthermore, fine-grained attention supervision networks include:

[0021] A 1x1 convolution operation is performed on the local fine-grained image I2 to generate the reconstructed image. For reconstructed images With real image I gt Calculate the loss using the real image I gt Supervised reconstruction images The output; secondly, the reconstructed image. Perform a series of convolutional operations, then feed the data into an attention network CA that combines spatial and channel attention to obtain a fine-grained attention feature map I. att Specifically, it is expressed as:

[0022]

[0023] Perform a 1×1 convolution on the local fine-grained image I2, and then concatenate it with the fine-grained attention feature map I. att Pixel-by-pixel multiplication yields the attention-guided calibration feature map. This calibration feature map is then added to the local fine-grained image I2 to obtain the final output of the fine-grained attention-supervised network, which is the image weighted by the importance of fine-grained features. Specifically, it is expressed as follows:

[0024]

[0025] Further steps include: first, taking the image... First, it is merged with the original incomplete image. The missing regions in the original incomplete image are used to replace the corresponding regions in the global coarse-grained image. Then, it is combined with the mask image M and fed into the global fine-grained inpainting network to obtain the global fine-grained image.

[0026] Further, the reconstruction loss of the global coarse-grained repair network. for:

[0027]

[0028]

[0029]

[0030] in, Indicates the reconstruction loss in the background area. Represents the reconstruction loss of the region to be repaired, ⊙ represents element-wise multiplication, and I represents the reconstruction loss of the region to be repaired. gt M represents the real image, and η represents the binary mask matrix; η is a hyperparameter that controls the contribution of each loss term.

[0031] Adversarial loss of global coarse-grained repair networks Adversarial loss L of the discriminator D Represented as:

[0032]

[0033]

[0034]

[0035] in, It is to use real image I gt Fusion with the global coarse-grained image I2, to obtain the real image I gt The valid values ​​of the corresponding background regions in the image are replaced with the corresponding regions in the global coarse-grained image I2; This represents the expected value distribution of the fused image. This indicates the discriminator's judgment result on the fused image; D(I) represents the expected value distribution of the real image. gt The ) represents the discriminator's judgment result on the real image;

[0036] Total loss of global coarse-grained repair network Set η = 6, μ = 0.1, where μ is the balance factor.

[0037] Further, the reconstruction loss of the local fine-grained repair network. for:

[0038]

[0039]

[0040]

[0041] in, Indicates the reconstruction loss in the background area. Represents the reconstruction loss of the region to be repaired, ⊙ represents element-wise multiplication, and I represents the reconstruction loss of the region to be repaired. gt M represents the real image, and M represents the binary mask matrix of the masked image;

[0042] Perceptual loss of local fine-grained repair networks for:

[0043]

[0044] Where, α i Let i represent the activation function feature map from the VGG-16 network, where i takes values ​​in the range {1,2,3,4,5}, and E represents the expectation of the real image.

[0045] Style loss in local fine-grained repair networks for:

[0046]

[0047] Where, β i It is the feature map of the activation function α i A gamma matrix is ​​obtained through construction;

[0048] Finally, the total loss L2 of the local fine-grained repair network is:

[0049]

[0050] Where: η mask η per η style Hyperparameters contributing to the control of various loss terms.

[0051] The beneficial effects of this invention are as follows: the first-stage coarse-grained global inpainting network completes the general semantic restoration of the image; the second-stage fine-grained local inpainting network focuses on reconstructing the texture and details of the image; considering the semantic coherence of the overall image, the third-stage fine-grained global inpainting network finely optimizes the structure and semantics of the overall image; thereby improving the image restoration capability of the image restoration network. Attached Figure Description

[0052] Figure 1 This is a flowchart of the present invention;

[0053] Figure 2 This is a network framework diagram of the present invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings. Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The directional terms such as left, center, right, top, and bottom in the embodiments of the present invention are only relative concepts or referenced to the normal use state of the product, and should not be considered restrictive.

[0055] A fine-grained image inpainting network based on feature enhancement strategies, such as Figure 1 As shown, it includes the following steps:

[0056] Step 1: Transfer the original damaged image I to be repaired in The image is fed into a coarse-grained global inpainting network, resulting in a global coarse-grained image I1; where the real image I1 is obtained from the dataset... gt Adding it to the mask image M simulates the original incomplete image I to be repaired. in In the mask image M, the missing area is marked as 1, and the background area is marked as 0;

[0057] Step 2: After simple processing, the global coarse-grained image I1 is fed into the fine-grained local inpainting network along with the discrimination result of the global coarse-grained image I1 after passing through the discriminator, resulting in a local fine-grained image I2. The local fine-grained image I2 has relatively correct semantic structure and detailed texture, making attention calculation more stable at this point. The fine-grained local inpainting network processes the input image I1 using a sliding window strategy before performing image inpainting. The sliding window strategy process is as follows: the input global coarse-grained image I1 is subdivided into multiple small regions. By sliding a fixed-size window across the entire image, each local small region is independently inpainted, ensuring that the inpainting process is not affected by other regions. This helps the network to process details more accurately when inpainting local structures and textures, while reducing dependence on the overall image context.

[0058] The process of simply processing the global coarse-grained image I1 is as follows: First, the global coarse-grained image I1 is compared with the original incomplete image I... in The process involves merging regions from the original incomplete image and replacing the corresponding regions in the global coarse-grained image. This replacement is then combined with a mask image M as input to a local fine-grained inpainting network. The purpose of this replacement is to preserve the original incomplete image I. in Information on the undamaged portions of the image is included, and the restored image is ensured to remain consistent with the original damaged image during the synthesis process. inConsistency; this makes the repaired image more natural, thereby improving the quality and accuracy of the repair effect; combined with the mask image M, it is actually to connect the replaced image with the mask image M in the channel dimension. The mask image M can indicate the part of the image that needs to be repaired. Connecting the image with the mask image M can help the network obtain the original information of the image to be repaired and the information of the area to be processed at the same time, so that the network can better understand the semantic information of the input data and learn and repair more accurately;

[0059] The input to the discriminator is a global coarse-grained image I1 and a real image I. gt The output is a two-dimensional feature map. The information in these two-dimensional feature maps can be used to calculate the loss function of the discriminator. The loss function of the discriminator can reflect the difference between the generated restored image and the real image. The discriminator will try to distinguish the generated restored image from the ground truth image. Furthermore, the loss function of the discriminator can be backpropagated to the coarse-grained global restoration network. By optimizing the network parameters, a more realistic restored image is generated to reduce the possibility of it being distinguished by the discriminator.

[0060] Step 3: Feed the local fine-grained image I2 into the fine-grained attention supervision network to obtain an image weighted by the importance of fine-grained features. Furthermore, the fine-grained attention supervision network, under the supervision of real images, reweights local features, uses the local fine-grained images to repair the network output to generate an attention map, and uses this attention map to refine the output of the previous stage before feeding it into the next stage of the network.

[0061] Step 4: Weight the image based on the importance of fine-grained features The image is fed into a fine-grained global insulation network to obtain a global fine-grained image I3, thus completing the insulation of the original incomplete image I. in Repair.

[0062] Among them, combined Figure 2As shown, both the coarse-grained and fine-grained global inpainting networks employ a CNN-based encoder-decoder architecture. The encoder contains eight downsampling layers for feature extraction and compression of the input image, transforming image information into a higher-level abstract feature representation. The decoder contains eight upsampling layers to remap the learned abstract feature representation back into the space of the input image, restoring an output image of the same size as the input image. To address the insufficient feature extraction capability of the network, a feature enhancement network is placed between the encoder and decoder. This network connects the feature maps from the first four downsampling layers in the encoder with the feature maps from the corresponding layers in the decoder, helping to better fuse coarse and fine-grained features. This compensates for the feature information loss problem during progressive upsampling / downsampling in either the coarse-grained or fine-grained global inpainting network. Instead of skip connections, it utilizes the similarity of features across channels, regenerating a larger number of features through simple linear transformations to participate in skip connections. This not only deepens the network hierarchy but also reduces the differences between encoder-decoder features, achieving better fusion of coarse / fine-grained semantics at different levels. To ensure the network obtains deep, fine-grained information while avoiding the loss of coarse-grained information in shallow features, the feature enhancement network includes obtaining the input feature map F in the downsampling layer corresponding to the encoder. x Then, a series of linear operations are performed on it to obtain a set of related feature maps F′. x Then, the feature maps before and after the transformation are concatenated along the channel dimension and compressed using a 1*1 product operation; simultaneously, the input feature map F is directly processed on the other side. x A 1x1 convolution operation is performed to obtain feature maps F″ containing deep, fine-grained information. x and residual feature map F y ;

[0063] F′ x =Φ n (F x (33)

[0064] F″ x =Conv(F x ||F′ x (34)

[0065] F y =Conv(F x (35)

[0066] Where, Φ n (·) represents a set of linear transformation operations, || represents a channel-level concatenation operation, and Conv(·) represents a convolution operation that compresses the feature map.

[0067] The feature map F″ containing deep, fine-grained information xWith residual feature map F y The network connects the features along the channel dimension, then compresses them using convolution to obtain the output feature map E (processed by the feature enhancement network). x :

[0068] E x =Conv(F y ||F″ x (36)

[0069] Formulas (1)-(4) are used to improve the feature level of encoder information, reduce the difference between encoder and decoder features, and enable better fusion of coarse / fine granular features at different levels, thereby enhancing the feature extraction capability of the network.

[0070] The fine-grained local inpainting network adopts a CNN-based encoder-decoder architecture and includes two downsampling layers, two FFC residual blocks, and two upsampling layers. The FFC residual blocks pass the input features layer by layer through several cascaded fully connected layers and perform residual connections, finally outputting the result. The FFC residual blocks can capture global information and complex features in the image and allow more interaction between features, thereby enhancing the network's expressive power.

[0071] To enhance the overall constraint of image inpainting and to perform fine-grained feature adjustments and selections, thereby improving the image inpainting effect, a fine-grained attention-supervised network includes: performing a 1*1 convolution operation on the local fine-grained image I2 to generate the reconstructed image. For reconstructed images With real image I gt Calculate the loss, the real image I gt Obtained from the dataset, using real images I gt Supervised reconstruction images The output; secondly, the reconstructed image A set of convolutional operations is performed, including a 3×3 convolution, batch normalization, and a ReLU non-linear activation function. The purpose of this step is to emphasize fine-grained feature information. After this set of convolutional operations, the data is fed into an attention network CA that combines spatial attention and channel attention to obtain a fine-grained attention feature map I. att Specifically, it is expressed as:

[0072]

[0073] Perform a 1×1 convolution on the local fine-grained image I2, and then concatenate it with the fine-grained attention feature map I. attPixel-by-pixel multiplication yields the attention-guided calibration feature map. This calibration feature map is then added to the local fine-grained image I2 to obtain the final output of the fine-grained attention-supervised network, which is the image weighted by the importance of fine-grained features. Specifically, it is expressed as follows:

[0074]

[0075] First, the image First, it is merged with the original incomplete image. The missing regions in the original incomplete image are used to replace the corresponding regions in the global coarse-grained image. Then, it is combined with the mask image M and fed into the global fine-grained inpainting network to obtain the global fine-grained image.

[0076] To improve the image quality of the global coarse-grained image, pixel-level reconstruction loss is used. and adversarial loss L D To optimize the parameters of the global coarse-grained repair network; the reconstruction loss of the global coarse-grained repair network. for:

[0077]

[0078]

[0079]

[0080] in, Indicates the reconstruction loss in the background area. Represents the reconstruction loss of the region to be repaired, ⊙ represents element-wise multiplication, and I represents the reconstruction loss of the region to be repaired. gt M represents the real image, and η represents the binary mask matrix; η is a hyperparameter that controls the contribution of each loss term.

[0081] This invention employs the least squares method to quantify the adversarial loss, and globally coarse-grained repairs the adversarial loss of the network. Adversarial loss L of the discriminator D Represented as:

[0082]

[0083]

[0084]

[0085] in, It is to use real image I gt Fusion with the global coarse-grained image I2, to obtain the real image I gt The valid values ​​of the corresponding background regions in the image are replaced with the corresponding regions in the global coarse-grained image I2; This represents the expected value distribution of the fused image. This indicates the discriminator's judgment result on the fused image; D(I) represents the expected value distribution of the real image. gt The ) represents the discriminator's judgment result on the real image;

[0086] Total loss of global coarse-grained repair network Set η = 6, μ = 0.1, where μ is a balancing factor, similar to η in formula (5), representing the weights corresponding to different losses, used to balance the impact of different losses on the entire loss function. The total loss of the global coarse-grained inpainting network is an indicator that measures the difference between the output of the inpainting network and the real image, directly reflecting the network's effect on repairing incomplete images. Through reconstruction loss, the global coarse-grained inpainting network can predict the lost or damaged image parts as accurately as possible and generate repair results that are as close as possible to the real image. The adversarial loss and reconstruction loss in formulas 5-10 will be backpropagated from the output layer to every part of the global coarse-grained inpainting network through the backpropagation algorithm in order to update the network parameters.

[0087] To improve the image quality of local fine-grained images, this invention uses reconstruction loss. Perceived loss and style loss To optimize the parameters of the local fine-grained insulation network; the reconstruction loss of the local fine-grained insulation network. for:

[0088]

[0089]

[0090]

[0091] in, Indicates the reconstruction loss in the background area. Represents the reconstruction loss of the region to be repaired, ⊙ represents element-wise multiplication, and I represents the reconstruction loss of the region to be repaired. gt M represents the real image, and M represents the binary mask matrix of the masked image;

[0092] Perceptual loss can capture high-level semantics and optimize the network structure of local fine-grained inpainting networks. Perceptual loss is implemented using a pre-trained model of the VGG16 network, calculating the distance between the feature space of the generated image and the ground truth. This perceptual loss is used in local fine-grained inpainting networks. for:

[0093]

[0094] Where, α iLet i represent the activation function feature map from the VGG-16 network, which is a pre-trained version on ImageNet. The value of i ranges from {1,2,3,4,5}, and E represents the expectation of the real image.

[0095] Style loss is used to reduce artifacts such as checkerboard patterns, helping the model better recover the texture and structure of the image; style loss in local fine-grained insulation networks. for:

[0096]

[0097] Where, β i It is the feature map of the activation function α i A gamma matrix is ​​obtained through construction;

[0098] Finally, the total loss L2 of the local fine-grained repair network is:

[0099]

[0100] Where: η mask η per η style To control the hyperparameters contributed by each loss term, the total loss L2 of the local fine-grained inpainting network is calculated using the backpropagation algorithm, and the gradients of the model parameters are calculated. The parameters in the local fine-grained image inpainting network are then updated using optimization algorithms such as gradient descent.

[0101] Methods for calculating the total loss L3 of a global fine-grained inpainting network and the reconstruction loss of a global fine-grained inpainting network. Similarly, you only need to Replace I1 with I3 in the corresponding position in the formula, that is;

[0102]

[0103]

[0104]

[0105] in, Indicates the reconstruction loss in the background area. This indicates the reconstruction loss in the area to be repaired;

[0106] Calculate the total loss function I of the three-stage network all That is, the sum of the total loss of each of the three-stage networks and the loss of the discriminator, i.e.:

[0107] L all =L1+L2+L3+L D

[0108] The core function of loss is to measure the difference between the model's predictions and the actual values. The loss function calculates the gradient of the network parameters and applies gradient-related algorithms (backpropagation algorithm) to continuously adjust the network parameters to reduce prediction errors, thereby driving the model to learn in the right direction.

[0109] Compared with the prior art, the present invention:

[0110] From the perspective of network design, the problems in existing technologies are mainly caused by the following factors: First, the network's feature extraction capability is insufficient. Many deep learning-based methods will result in information loss when acquiring deep features, which is not conducive to the restoration of image texture details. Second, the network's constraint capability is insufficient, resulting in image restoration results with unreasonable structure.

[0111] To address the issue of insufficient feature extraction capabilities in networks, this invention proposes a skip connection network based on a feature enhancement strategy to better fuse coarse and fine-grained semantic features. This compensates for information loss during progressive upsampling / downsampling. Unlike previous U-Net architectures that use simple skip connections to fuse low-level features from the encoder and high-level features from the decoder, this invention leverages the similarity of features across channels to regenerate a greater number of features through simple linear transformations to participate in skip connections. This approach not only deepens the network layers but also reduces the differences between encoder-decoder features, achieving better fusion of coarse and fine-grained semantics at different levels. This ensures the network obtains deep, fine-grained information while avoiding the loss of coarse-grained information in shallow features. Simultaneously, to enhance the network's constraint capabilities, this invention proposes a novel fine-grained attention-supervised network, which differs from many image inpainting methods based on multi-stage networks. Traditional methods primarily utilize attention in feature extraction or fusion stages, rarely focusing on image transfer between stages. This invention proposes introducing a fine-grained attention supervision network between the local and global fine-grained image inpainting networks. The image obtained through the local fine-grained image inpainting network (i.e., the local fine-grained image) possesses relatively accurate semantic structure and detailed texture, making attention computation more stable. Furthermore, under the supervision of the real image, the fine-grained attention supervision network reweights local features, uses the output of the local fine-grained image inpainting network to generate an attention map, and uses this attention map to refine the output of the previous stage before feeding it into the next stage network. This network performs more refined feature adjustment and selection, sending only the useful features that have undergone feature importance weighting in the fine-grained attention supervision network into the global fine-grained image inpainting network, thereby further improving image inpainting capabilities.

[0112] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fine-grained image inpainting network based on a feature enhancement strategy, characterized in that: Includes the following steps: Step 1: Transfer the original damaged image I to be repaired in The image is fed into a coarse-grained global insulation network, and a global coarse-grained image I1 is obtained. Step 2: The global coarse-grained image I1 and the discrimination result of the global coarse-grained image I1 after passing through the discriminator are fed into the fine-grained local repair network to obtain the local fine-grained image I2. Step 3: Feed the local fine-grained image I2 into the fine-grained attention supervision network to obtain an image weighted by the importance of fine-grained features. Step 4: Weight the image based on the importance of fine-grained features The image is fed into a fine-grained global insulation network to obtain a global fine-grained image I3, thus completing the insulation of the original incomplete image I. in Repair.

2. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: Both the coarse-grained global inpainting network and the fine-grained global inpainting network adopt a CNN-based encoder-decoder architecture. The encoder contains eight downsampling layers and is used to extract and compress features from the input image, transforming the image information into a higher-level abstract feature representation. The decoder contains eight upsampling layers and is used to remap the learned abstract feature representations back into the space of the input image, restoring an output image of the same size as the input image.

3. The fine-grained image inpainting network based on feature enhancement strategy according to claim 2, characterized in that: A feature enhancement network is set between the encoder and the decoder. The feature enhancement network is used to connect the feature maps in the first four downsampling layers of the encoder with the feature maps in the corresponding layers of the decoder.

4. The fine-grained image inpainting network based on feature enhancement strategy according to claim 3, characterized in that: Feature enhancement networks include: The input feature map F is obtained in the downsampling layer corresponding to the encoder. x Obtain feature map F″ containing deep, fine-grained information. x and residual feature map F y ; F′ x =Φ n (F x ) (1) F″ x =Conv(F x ||F′ x ) (2) F y =Conv(F x ) (3) Where, Φ n (·) represents a set of linear transformation operations, || represents a channel-level concatenation operation, and Conv(·) represents a convolution operation that compresses the feature map. The feature map F″ containing deep, fine-grained information x With residual feature map F y The features are concatenated along the channel dimension and then compressed using convolution to obtain the output feature map E. x : E x =Conv(F y ||F″ x )。 (4) 5. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: The fine-grained local inpainting network adopts a CNN-based encoder-decoder architecture and includes two downsampling layers, two FFC residual blocks, and two upsampling layers. The FFC residual blocks pass the input features layer by layer through several cascaded fully connected layers and perform residual connections, finally outputting the result.

6. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: Fine-grained attention supervision networks include: A 1x1 convolution operation is performed on the local fine-grained image I2 to generate the reconstructed image. For reconstructed images With real image I gt Calculate the loss using the real image I gt Supervised reconstruction images The output; secondly, the reconstructed image. Perform a series of convolutional operations, then feed the data into an attention network CA that combines spatial and channel attention to obtain a fine-grained attention feature map I. att Specifically, it is expressed as: Perform a 1×1 convolution on the local fine-grained image I2, and then concatenate it with the fine-grained attention feature map I. att Pixel-by-pixel multiplication yields the attention-guided calibration feature map. This calibration feature map is then added to the local fine-grained image I2 to obtain the final output of the fine-grained attention-supervised network, which is the image weighted by the importance of fine-grained features. Specifically, it is expressed as follows:

7. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: First, the image First, it is merged with the original incomplete image. The missing regions in the original incomplete image are used to replace the corresponding regions in the global coarse-grained image. Then, it is combined with the mask image M and fed into the global fine-grained inpainting network to obtain the global fine-grained image.

8. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: Reconstruction loss of global coarse-grained repair network for: in, Indicates the reconstruction loss in the background area. This represents the reconstruction loss of the area to be repaired, and ⊙ represents element-wise multiplication. M represents the real image, and η represents the binary mask matrix; η is a hyperparameter that controls the contribution of each loss term. Adversarial loss of global coarse-grained repair networks Adversarial loss L of the discriminator D Represented as: in, It is to use real image I gt Fusion with the global coarse-grained image I2, to obtain the real image I gt The valid values ​​of the corresponding background regions in the image are replaced with the corresponding regions in the global coarse-grained image I2; This represents the expected value distribution of the fused image. This indicates the discriminator's judgment result on the fused image; D(I) represents the expected value distribution of the real image. gt The ) represents the discriminator's judgment result on the real image; Total loss of global coarse-grained repair network Set η = 6, μ = 0.1, where μ is the balance factor.

9. The fine-grained image inpainting network based on feature enhancement strategy according to claim 1, characterized in that: Reconstruction loss of local fine-grained repair networks for: in, Indicates the reconstruction loss in the background area. Represents the reconstruction loss of the region to be repaired, ⊙ represents element-wise multiplication, and I represents the reconstruction loss of the region to be repaired. gt M represents the real image, and M represents the binary mask matrix of the masked image; Perceptual loss of local fine-grained repair networks for: Where, α i Let i represent the activation function feature map from the VGG-16 network, where i takes values ​​in the range {1,2,3,4,5}, and E represents the expectation of the real image. Style loss in local fine-grained repair networks for: Where, β i It is the feature map of the activation function α i A gamma matrix is ​​obtained through construction; Finally, the total loss L2 of the local fine-grained repair network is: Where: η mask η per η style Hyperparameters contributing to the control of various loss terms.

Images