An image inpainting method based on a DU-GAN network

Abstract

The application discloses a kind of based on DU-GAN network's image restoration method, comprising:1: in generator, join mask position coding to accelerate the convergence speed of network and the accuracy of filling content;2: join attention mechanism in generator network to make the texture of image more clear;3: increase adversarial loss, further constrain the content of generated image, improve the quality of repaired image.The application can be repaired when the texture and structure information of damaged image are relatively complex, make full use of the texture and structure information contained in damaged image, and use the two in combination, so that the quality of repaired image is higher, and the visual effect is more natural.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image inpainting, and particularly relates to an image inpainting method based on a DU-GAN network. BACKGROUND

[0002] In the field of traditional image inpainting, representative methods include image inpainting technology based on partial differential equations, image inpainting technology based on texture synthesis, and image inpainting technology based on sparse representation.

[0003] The repair technology based on partial differential equations was first proposed by Bertalmio et al. in 2000 and applied to the field of image inpainting. The technology uses the boundary information of the region to be repaired to determine the diffusion information and diffusion direction, and gradually diffuses from the region boundary to the boundary to achieve image inpainting. The technology achieves good results in small area filling, but has poor repair effect on damaged images with complex texture and structure information.

[0004] In 2003, Drori et al. initiated the texture image repair, which uses texture blocks as a whole to avoid the randomness of pixel filling. In 2004, Criminisi proposed a priority function for the boundary of the region to be repaired, and then determined the best matching block through global search and directly copied and filled it. This method has block matching errors, and when the priority calculation is unreasonable, the sample resource is insufficient, and the like, it often cannot obtain a satisfactory repair result.

[0005] The core idea of the image inpainting model based on sparse representation is that the damaged region and the intact region of an image have the same sparse characteristics, so the information of the intact region can be sparsely represented to obtain sparse coefficients, and then the image signal can be reconstructed through a reconstruction algorithm to restore the complete image. This method effectively avoids the situation of single texture block dissimilarity, and uses multiple texture blocks to fit similar filling blocks. However, when facing the situation of sample loss, the image cannot be repaired.

[0006] Deep learning uses deep neural network modeling, has stronger feature learning and expression ability compared with traditional learning algorithms, and can capture more deep features, and has achieved good repair results in the field of image inpainting. The deep learning can be roughly divided into image inpainting technology based on GAN (Generative Adversarial Network) and image inpainting technology based on coding and decoding.

[0007] The main idea of the image inpainting technology based on the generative adversarial network is to use the GAN generated image for inpainting, JiTero Karras and others integrate the image style content into the GAN network to solve the problem of speckle in the generated image. However, this may cause the image to be blurred, and the discriminator structure is too simple, which cannot effectively supervise the overall network. Yingchen Yu and others use a network structure based on wavelet repair to improve the image inpainting quality. However, the texture details after repair are not perfect. Tong Zhou and others proposed a GAN network with double spatial attention modules and multiple discriminators to better repair face images. This method may not work well when repairing images with large missing areas.

[0008] The encoder-based image inpainting network structure is widely used in image inpainting. Chenjie Cao and others proposed a multi-scale network for image inpainting, which uses the edges and lines of the image as priori to improve the network's ability to grasp the whole image, thereby improving the image inpainting quality. However, since the line and edge effects are similar, the repair effect is generally poor for images with severe missing. Yuqian Zhou believes that local texture and semantic information are not enough to repair the image, and proposes a repair network based on the source image, which extracts feature points according to the homography of the source image and clusters them, and then obtains the repaired image through a color space converter and a feature fusion module. However, it does not fully utilize the texture and structure information of the image, resulting in ghosting in the repair result. Xiefan Guo proposes a dual-stream network structure to extract the structure and texture features of the image respectively and combine them, but for images with large missing areas, the repair result often appears blurred. SUMMARY

[0009] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes an image inpainting method based on a DU-GAN network, so as to fully utilize the texture and structure information contained in the damaged image and combine them when repairing the image with complex texture and structure information, thereby improving the image inpainting quality and restoring the value and visual effect of the image.

[0010] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0011] The image inpainting method based on the DU-GAN network of the present application has the following steps:

[0012] Step 1, obtain the image data set and perform preprocessing:

[0013] Get the image dataset and crop it into original images of size M×M, then add a mask of size M×M to the cropped original images, and get the preprocessed image dataset denoted as B={b1,b2,...,b i ,...,b N}, where b i represents the i-th image; i = 1, 2, ..., N, where N is the total number of images in the image dataset;

[0014] The image dataset B is detected using the Canny operator to obtain the image edge dataset H = {h1,h2,...,h i ,...,h N}, where h i represents the i-th edge image;

[0015] Step 2: Construct the DU-GAN network, including the generator module G and the discriminator module A;

[0016] Step 2.1, the generator module G respectively generates the i-th image b i and the i-th edge image h i Processing is performed to obtain the texture features of the repaired decoded image and structural characteristics

[0017] Step 2.2: The repaired texture features and structural characteristics Fusion is performed at a certain ratio ɑ to obtain a generated image s with a dimension of M×M i ;

[0018] Step 2.3, construct a discriminator module A with a three-branch structure, including: the repaired image identification branch O, the repaired image grayscale image identification branch P and the repaired image edge image identification branch L; each branch contains ly convolution layers, where the convolution kernel size of the convolution layer is l y ×l y , the stride is v; the last convolution layer uses the sigmoid function for activation, and the other convolution layers use the LeakyReLU activation function;

[0019] Will generate image s i After being input into the image identification branch O and processed by the convolution layer, the convolution image s is obtained. io ;

[0020] Use the cvtColor function to generate image s i Convert to grayscale image s′ i , will s i and s′ iThe grayscale image s is input into the image identification branch P and processed by the convolution layer to obtain the grayscale image s ip ;

[0021] Use Canny operator to detect and generate image s i The edge graph s″ i , will generate image s i and edge graph s″ i After inputting the edge map identification branch L and being processed by the convolution layer, the edge image s is obtained. il ;

[0022] Step 2.4: The s obtained by the image identification branch O, the image grayscale image identification branch P and the edge image identification branch L module of the repaired image io 、s ip and s il Parallel together and identify with the original image:

[0023] Step 2.4.1. Use formula (1) to construct the generative adversarial loss:

[0024]

[0025] In formula (1), represents the expected value from the 1st image to the Nth image in the image dataset B, L s (G) represents the loss of the generator module G, L s (A) represents the loss of the discriminator module A;

[0026] Step 2.4.2: Use formula (2) to construct the global objective function L all :

[0027] L all =L s (G)+L s (A) (2)

[0028] Step 2.4.3: Based on the image dataset B and the image edge dataset H, the DU-GAN network is trained using the gradient descent method, and the global objective function L is calculated. all , until the global objective function L all Until convergence, the optimal image restoration model is obtained, which is used to repair any damaged image and output the restored image.

[0029] The image restoration method based on the DU-GAN network of the present invention is also characterized in that the generator module G is a dual-stream AU-Net network, each AU-Net network includes: a position encoder, an encoder, an attention module and a decoder; wherein the encoder includes m layers of convolutional layers; the decoder includes m layers of deconvolutional layers;

[0030] The jump connection is added between the second layer convolution layer of the encoder and the m-1 layer deconvolution layer of the decoder, and the jump connection is added between the m layer convolution layer of the encoder and the first layer deconvolution layer of the decoder in turn; and the padding operation is arranged after each convolution layer and each deconvolution layer; the step of all convolution layers is set to 2c, and the step of deconvolution layer is set to c;

[0031] Step 2.1.1, the position encoder in an AU-Net network uses the sine positioning algorithm to perform position encoding on the i-th image b i , to obtain position information P i and direction information P b,i,w of the mask in the i-th image b b,i,r , so as to obtain mask position encoding P b,i ;

[0032] The position encoder in another AU-Net network uses the sine positioning algorithm to perform position encoding on the i-th edge image h i , to obtain position information P i and direction information P h,i,w of the mask in the i-th edge image h h,i,r , so as to obtain mask position encoding P h,i ;

[0033] Step 2.1.2, the image features are extracted using the encoder;

[0034] b i and the mask position encoding P b,i are input into the encoder of the corresponding AU-Net network, and after the convolution operation of the m layer convolution layer, the structural feature f i of the i-th image b b,i is obtained;

[0035] b i , h i and the mask position encoding P h,i are input into the encoder of another AU-Net network, and after the convolution operation of the m layer convolution layer, the texture feature f i of the i-th image b h,i is obtained;

[0036] Step 2.1.3, the image texture features are reconstructed using the attention module;

[0037] The feature map f b,iThe attention module of one AU-Net network calculates the cosine similarity between the feature pixel values in the s repair blocks, so as to obtain the attention scores of the s feature pixel values according to the s cosine similarities, and multiply each attention score with the corresponding repair block, so as to obtain s new patches, and the texture feature f b,i is reconstructed to obtain the reconstructed texture feature

[0038] The attention module of another AU-Net network calculates the cosine similarity between the feature pixel values in the s repair blocks, so as to obtain the attention scores of the s feature pixel values according to the s cosine similarities, and multiply each attention score with the corresponding repair block, so as to obtain s new patches, and the texture feature f h,i is reconstructed to obtain the reconstructed texture feature

[0039] Step 2.1.4, reconstructing the image feature by using the decoder;

[0040] The decoder of one AU-Net network decodes the reconstructed texture feature to obtain the texture feature of the repaired decoded image

[0041] The decoder of another AU-Net network decodes the structure feature to obtain the structure feature of the repaired decoded image

[0042] The electronic device of the present application comprises a memory and a processor, and the feature is that the memory is used to store a program supporting the processor to execute the image repairing method, and the processor is configured to execute the program stored in the memory.

[0043] The computer readable storage medium of the present application stores a computer program, and the feature is that the computer program is executed by the processor to execute the steps of the image repairing method.

[0044] Compared with the prior art, the present application has the following advantages:

[0045] 1. The present application adds position encoding to the U-Net network, provides the absolute spatial position information of the mask to the network through the position encoding module, and obtains a 4-channel vector through 4 different kernels, which can represent the nearest direction of the mask position and the non-mask position. Its value depends on which kernel can first complete the coverage of the mask area, and the mask direction is obtained, which provides the direction information of the mask for the network, overcomes the problem that the zero padding in the traditional convolutional neural network only provides the absolute spatial position information of the mask for the network without providing the direction information, and thus improves the repair quality of the network for the image.

[0046] 2、The application adds an attention module to the U-Net network, calculates the cosine similarity of the feature map, then obtains an attention score, and combines the corresponding repair block, thereby completing the reconstruction of the image features, overcoming the difficulty that the texture and structure information of the image is not utilized, and thereby improving the visual coherence of the image repair;

[0047] 3、The application couples two AU-Net networks, couples the AU-Net networks with position encoding and attention modules, and enables the decoding structure of the AU-Net network to decode the texture features extracted by the other AU-Net network as auxiliary decoding when decoding the structure features of the image, and enables the decoding structure of the AU-Net network to decode the structure features extracted by the other AU-Net network as auxiliary decoding when decoding the texture features of the image, thereby overcoming the difficulty that the model cannot effectively utilize the texture and structure features of the image, and making the repaired image more natural in visual effect;

[0048] 4、The application discriminates the generated image, obtains a gray image of the generated image by using the CvtColor of Opencv, detects the generated image by using the Canny operator to obtain an edge image of the generated image, and discriminates the generated image, the gray image and the edge image together with the original image, thereby overcoming the difficulty that the discrimination of a single image cannot supervise the multiple feature information of the image, and improving the supervision of the discriminator module on the overall network.

[0049] 5、The application adds the LSGAN loss to the network, generates a gradient by punishing the sample according to the distance of the sample and the decision boundary based on the LSGAN loss based on the least square loss, thereby overcoming the difficulty that the conventional adversarial loss uses the minimax loss, and the gradient feedback of the pseudo sample is basically absent, and it is difficult to provide effective supervision for the repair of the image, thereby strengthening the visual effect of the generated image generated by the generator, and improving the stability of the network training. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The figure is the overall network flowchart of the image repair of the application;

[0051] Figure 2 The figure is the direction information schematic diagram of the position encoding of the application. DETAILED DESCRIPTION

[0052] In this embodiment, the image repair method based on the DU-GAN network can solve the problem that the texture features and structure features of the image cannot be combined and utilized in the process of repairing the image, and the image repair blurring problem often occurs when a large area of missing region is repaired. Figure 1 As shown in the figure, the method comprises the following steps:

[0053] Step 1: Obtain image dataset and preprocess it:

[0054] Get the image dataset and crop it into original images of size M×M, M=256, then add a mask of size M×M to the cropped original images, and get the preprocessed image dataset denoted as B={b1,b2,...,b i ,...,b N}, where b i represents the i-th image; i = 1, 2, ..., N, where N is the total number of images in the image dataset;

[0055] The image dataset B is detected using the Canny operator to obtain the image edge dataset H = {h1,h2,...,h i ,...,h N}, where h i represents the i-th edge image;

[0056] Step 2: Construct the DU-GAN network, including the generator module G and the discriminator module A;

[0057] Step 2.1. The generator module G is a two-stream AU-Net network. Each AU-Net network includes: a position encoder, an encoder, an attention module, and a decoder. The encoder includes m convolutional layers, and the decoder includes m deconvolutional layers. A skip connection is added between the second convolutional layer of the encoder and the m-1th deconvolutional layer of the decoder. Similarly, a skip connection is added between the mth convolutional layer of the encoder and the first deconvolutional layer of the decoder. Padding operations are set after each convolutional layer and each deconvolution layer. The step size of all convolutional layers is set to 2c, and the step size of the deconvolution layer is set to c. Where m = 7 and c = 1.

[0058] Step 2.1.1: A position encoder in an AU-Net network uses a sinusoidal positioning algorithm to locate the i-th image b i Perform position encoding to obtain the mask in the i-th image b i The location information P b,i,w and direction information P b,i,r , thus obtaining the mask position code P b,i ;

[0059] For the position information P of the mask b,i,w To obtain it, the following steps are used: for a mask with a width and height of 256×256, use 0 and 1 to represent the masked area and the unmasked area respectively, to obtain the i-th image b iTo obtain the spatial position information of the mask in the image, we need to first use a 3×3 kernel with all 1s to calculate the mask distance of each position in the mask area. Then use sinusoidal position coding to clip and encode the mask distance to obtain the position information P b,i,w . For the direction information P of the mask b,i,r , you can use 4 different kernels to get a 4-channel vector, which can represent the nearest direction between the mask position and the non-mask position. Its value depends on which kernel can first complete the coverage of the mask area, and get the mask direction information P b,i,r .like Figure 2 shown.

[0060] Another position encoder in the AU-Net network uses the sinusoidal positioning algorithm to locate the i-th edge image h i Perform position encoding and get the mask in the i-th edge image h i The location information P h,i,w and direction information P h,i,r , thus obtaining the mask position code P h,i ;

[0061] For the position information P of the mask h,i,w To obtain it, the following steps are used: for a mask with a width and height of 256×256, 0 and 1 are used to represent the masked area and the unmasked area respectively, to obtain the i-th image h i To obtain the spatial position information of the mask in the image, we need to first use a 3×3 kernel with all 1s to calculate the mask distance of each position in the mask area. Then use sinusoidal position coding to clip and encode the mask distance to obtain the position information P h,i,w . For the direction information P of the mask h,i,r , you can use 4 different kernels to get a 4-channel vector, which can represent the nearest direction between the mask position and the non-mask position. Its value depends on which kernel can first complete the coverage of the mask area, and get the mask direction information P h,i,r .

[0062] Step 2.1.2: Use the encoder to extract image features.

[0063] b i and mask position encoding P b,i Input into the encoder of the corresponding AU-Net network, and after the convolution operation of m layers of convolutional layers, the i-th image b is obtained. i The structural characteristics of f b,i ;

[0064] b i 、h i and mask position encoding P h,iThe texture feature f i of the i-th image b h,i is input into an encoder of another AU-Net network and is subjected to convolution operation of m convolution layers to obtain the texture feature f b,i of the i-th image b b,i .

[0065] Step 2.1.3, reconstructing the texture feature of the image by using the attention module;

[0066] The feature map f b,i is input into the attention module of one AU-Net network, and the attention module calculates the cosine similarity between the feature pixel values of s repair blocks with the same size in the feature map f b,i , so as to obtain the attention scores of the s feature pixel values according to the s cosine similarities, combine each attention score with the corresponding repair block, so as to obtain s new patches, and reconstruct the texture feature f b,i by using the s new patches to obtain the reconstructed texture feature f In this embodiment, the size of the repair block is 3x3.

[0067] When the image feature is subjected to convolution coding, if each local information is processed, the calculation amount of the entire network will be too large. Therefore, the attention module is added after the coding of the AU-Net to solve this problem. When the texture information of the image is synthesized, the attention module can be used for reconstruction of the texture feature of the image by collecting the texture feature of the image.

[0068] In the entire attention module, for the input feature f b,i , the cosine similarity between each pair of feature pixel values is first calculated, as shown in formula (3).

[0069]

[0070] Where (x, y) and (x', y') are two relative position coordinates.

[0071] Then, the attention score is calculated according to each column patch, after obtaining the attention score of each patch in the feature map, each score is combined with the corresponding patch, and finally the new patch is obtained, and in this way, the texture feature of the image is reconstructed, as shown in formula (4).

[0072]

[0073] The attention module of the other AU-Net network processes f h,i in the same way, so as to obtain the reconstructed structure feature f

[0074] Step 2.1.4, reconstructing the image feature by using the decoder;

[0075] The decoder of an AU-Net network adopts a coupled structure to reconstruct the texture feature of the texture feature of the repaired decoded image is decoded to obtain the texture feature of the repaired decoded image

[0076] The decoder of another AU-Net network decodes the structure feature of the structure feature of the repaired decoded image is also decoded to obtain the structure feature of the repaired decoded image

[0077] Step 2.2, the repaired texture feature and the structure feature are fused according to a certain proportion a, so as to obtain a generated image s with a dimension of MxM i ;

[0078] Step 2.3, a discriminator module A with a three-branch structure is constructed, including a repaired image discrimination branch O, a repaired image grayscale discrimination branch P and an edge map discrimination branch L of the repaired image; each branch contains ly layers of convolution layers, wherein the convolution kernel size of the convolution layer is l y x l y , and the stride is v; the last convolution layer uses a sigmoid function for activation, and the other convolution layers use a LeakyReLU activation function;

[0079] wherein ly=3, l y x l y =3x3, and v=2.

[0080] The generated image s i is input into the image discrimination branch O and processed by the convolution layer to obtain a convolution image s io ;

[0081] The generated image s i is converted into a grayscale image s′ i by using a cvtcolor function; s i and s′ i are input into the image grayscale discrimination branch P and processed by the convolution layer to obtain a grayscale image s ip ;

[0082] The edge map s″ i of the generated image s i is detected by using a Canny operator; the image s i and the edge map s″ i are input into the edge map discrimination branch L and processed by the convolution layer to obtain an edge map image s il ;

[0083] Step 2.5, s io , ip and s il parallel together with the original image for identification:

[0084] Step 2.5.1, construct the generative adversarial loss using formula (1), the meaning of the adversarial loss is that it can enhance the visual effect of the generated image, the conventional adversarial loss is to use the minimax loss, which has basically no gradient feedback to the pseudo sample, and it is difficult to provide effective supervision for image repair. Therefore, the LSGAN loss based on the least square loss is added to the network, which not only can enhance the supervision effect of image repair, but also the LSGAN loss based on the least square loss can punish the sample according to the distance of the sample and the decision boundary, so as to generate gradient, which can improve the stability of network training;

[0085]

[0086] In formula (1), represents the expected value from the first image to the Nth image in the image data set B, L s (G) represents the loss of the generator module G, L s (A) represents the loss of the discriminator module A;

[0087] Step 2.5.2, construct the global objective function L all :

[0088] L all = L s (G) + L s (A) (2)

[0089] Step 2.5.3, based on the image data set B and the image edge data set H, the gradient descent method is used to train the DU-GAN network, and the global objective function L all is calculated until the global objective function L all converges, so as to obtain the optimal image repair model, which is used for repairing any damaged image and outputting the repaired image.

[0090] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0091] In this embodiment, a computer readable storage medium stores a computer program on the computer readable storage medium, and the computer program is executed by the processor to perform the steps of the above method.

Claims

1. A method for image inpainting based on a DU-GAN network, characterized in that, The method comprises the following steps: Step 1, obtaining an image data set and preprocessing: An image dataset is obtained and cropped into original images with a size of respectively, and a mask with a size of is added to the cropped original images to obtain a preprocessed image dataset denoted as , where represents the th image; , is the total number of images in the image dataset. The image dataset B is detected by a Canny operator to obtain an image edge dataset wherein, denotes the the edge image; Step 2, constructing a DU-GAN network, comprising: a generator module G and a discriminator module A; wherein the generator module G is a double-flow AU-Net network, each AU-Net network comprising: a position encoder, an encoder, an attention module and a decoder; wherein the encoder comprises m layers of convolutional layers; the decoder comprises m layers of deconvolutional layers; A skip connection is added between the 2nd layer of convolutional layers of the encoder and the (m-1)th layer of deconvolutional layers of the decoder, and a skip connection is added between the mth layer of convolutional layers of the encoder and the 1st layer of deconvolutional layers of the decoder in turn; and a padding operation is arranged after each layer of convolutional layers and each layer of deconvolutional layers; the step length of all convolutional layers is set to 2c, and the step length of deconvolutional layers is set to c; Step 2.1, the generator module G processes the first and the second image respectively to obtain the texture feature and the structure feature of the repaired decoded image. Step 2.2, the generator module G generates the repaired decoded image according to the texture feature and the structure feature. ​ Step 2.2, texture features after repair and structure features are fused with a certain proportion a, so as to obtain a generated image with dimensions of ; Step 2.3, constructing discriminator module A containing three branches, including: repaired image identification branch O, repaired image gray image identification branch P and repaired image edge image identification branch L; each branch contains Layer convolution layer, wherein the convolution kernel size of the convolution layer is , and the stride is ; the last convolution layer uses a sigmoid function for activation, and other convolution layers use a LeakyReLU activation function; generate an image After inputting into the image discrimination branch O and processing through a convolution layer, a convolution image is obtained ; The generated image is converted into a grayscale image using a cvtcolor function The grayscale image is converted into a gray image The gray image is converted into a grayscale image The grayscale image is converted into a gray image The grayscale image is input into the grayscale image identification branch P and processed by a convolution layer to obtain a grayscale image ; Use Canny operator to detect and generate images Edge graph , will generate the image and edge graph After inputting the edge map identification branch L and being processed by the convolution layer, the edge image is obtained. ; Step 2.

4. The image discrimination branch O, the image grayscale discrimination branch P and the edge map discrimination branch L module of the repaired image are obtained , and parallel together, and discriminated with the original image: Step 2.4.1, constructing a generative adversarial loss using formula (1): (1) In formula (1), denotes the expected value from the first image to the Nth image in the image data set denotes the expected value from the first image to the Nth image in the image data set denotes the loss of the generator module G, denotes the loss of the discriminator module A; Step 2.4.2, constructing the global objective function using formula (2) : (2) Step 2.4.3, training the DU-GAN network based on the image dataset B and the image edge dataset , and calculating the global objective function , until the global objective function converges, thereby obtaining an optimal image inpainting model for inpainting any damaged image and outputting the inpainted image.

2. The image inpainting method based on the DU-GAN network according to claim 1, characterized in that, The step 2.1 comprises: Step 2.1.1: A position encoder in the AU-Net network uses the sinusoidal positioning algorithm to images Perform position encoding and get the mask in the first images Location information in and direction information , thus obtaining the mask position encoding ; Another AU-Net network position encoder uses a sinusoidal positioning algorithm to position the first Zoomed edge image Position encoding is performed to obtain the position information of the mask in the first Zoomed edge image and direction information , so as to obtain mask position encoding ;​ Step 2.1.2, extracting image features using the encoder; Will and mask position encoding Input into the encoder of the corresponding AU-Net network, and after the convolution operation of m layers of convolution layers, the first Zhang image Structural features ; Will 、 and mask position encoding Input into the encoder of another AU-Net network and after the convolution operation of m layers of convolutional layers, the first images Texture features ; Step 2.1.3, reconstructing image texture features using the attention module; The feature map The image is divided into s repair blocks of the same size. The attention module of an AU-Net network calculates the cosine similarity between the feature pixel values ​​in the s repair blocks, and then obtains the attention scores of the s feature pixel values ​​according to the s cosine similarities. Each attention score is multiplied by the corresponding repair block to obtain s new patches. The s new patches are used to analyze the texture features. Reconstruct and obtain the reconstructed texture features ; Another AU-Net network's attention module pair The same processing is performed to obtain the reconstructed structural features ; Step 2.1.4, reconstructing image features using the decoder; The decoder of an AU-Net network adopts a coupled structure to reconstruct the texture feature is decoded to obtain the texture feature of the repaired decoding image ; the decoder of the other AU-Net network Also decoded, resulting in a structure feature of the repaired decoded image .

3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the image inpainting method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the image inpainting method of claim 1 or 2.

Images