An image inpainting method based on a dual-gated convolution-based inter-encoding and decoding network

Abstract

The application discloses an image repairing method based on a mutual encoding-decoding network of double-gated convolution, and comprises the following steps: S1, a network model is built, including a repairing network and a discriminator network, and a BG-MED network is constructed as the repairing network; S2, a data set is selected; S3, a damaged picture is input into the BG-MED network model, repaired, and a repaired picture is output; S4, the total loss value of the discriminator is calculated, an Adam optimization algorithm is used to update the parameters of the global discriminator and the local discriminator; S5, the loss value of the repairing network is calculated, the Adam optimization algorithm is used to update the BG-MED model parameters; S6, steps S3-S5 are repeated for multiple times, and the network model value is set by using the trained model parameters; and S7, the BG-MED model is used for actual image repairing. According to the application, the mutual encoding-decoding network model based on double-gated convolution is proposed in view of the defects of the gated convolution, and the image repairing effect is excellent.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image inpainting method based on a dual-gated convolutional mutual encoding and decoding network. Background Technology

[0002] With the advent of the big data era and the rapid advancement of computer hardware, the concept of deep learning has once again come into focus. Researchers have successfully introduced deep learning models into computer vision tasks. The emergence of Convolutional Neural Networks (CNNs) has enabled computers to better learn features from images. Inspired by this, deep learning-based image inpainting methods have emerged. These methods train deep models to acquire high-level semantic information of images and learn image structure and texture to repair large areas of damaged images. These methods address the shortcomings of traditional image inpainting and achieve excellent restoration results. The emergence of Generative Adversarial Networks (GANs) has made it possible for models to learn probability distributions.

[0003] Currently, there are two main types of convolution used for image irregularity restoration: partial convolution and gated convolution. Partial convolution assumes that features from occluded areas should not be extracted. It only allows effective pixels within the receptive field to participate in the convolution operation, using a binary mask to determine which pixels are effective, i.e., which pixels are not occluded. The mask value is only 0 or 1, where 1 indicates occlusion and 0 indicates no occlusion. Partial convolution simply and crudely divides the region into occluded and unoccluded areas, but in reality, there are intermediate regions, i.e., semi-occluded areas after initial restoration. The mask value for these intermediate regions should be between 0 and 1. Furthermore, all channels of each feature layer share the same mask, which limits flexibility. Different channels should have different masks; for example, channel 1 might be occluded at a certain location, but channel 2 might not be occluded at the same location. Gated convolution addresses the shortcomings of partial convolution. It learns gate values ​​from the image / features, and the gate value is determined by M. l The superscript 'l' stands for 'learning', indicating online learning, i.e., M. l This is represented by the gating value obtained through network learning. The gating value represents the weight of image occlusion, and its value is between 0 and 1. For example, This represents the degree of occlusion of an image or feature at position (i, j) (i.e., row i, column j). A drawback of gated convolution is that M... l The accuracy is insufficient, or rather, the network has difficulty learning accurate M. l . Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and propose an image inpainting method based on a dual-gated convolutional mutual encoding and decoding network.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An image inpainting method based on a dual-gated convolutional mutual encoding / decoding network includes the following steps:

[0007] S1. Build the network model, including the repair network and the discriminator network. Construct a mutual encoding and decoding network based on dual-gated convolution as the repair network. The discriminator network adopts a global discriminator and a local discriminator.

[0008] S2. Select a dataset, select multiple training images, and use the RTV smoothing algorithm in MATLAB to obtain the structure diagram of each training image. Use a random mask set during training.

[0009] S3. Use a mask on the original image I. gt By occluding the image, the damaged image I is obtained. broken , will I broken As input to the BG-MED network, the image is repaired by the BG-MED network, and the output is the repaired image I. out ;

[0010] S4. Calculate the total loss value L of the discriminator. dis The Adam optimization algorithm is used to update the parameters of the global discriminator and the local discriminator.

[0011] S5. Calculate the loss value L of the repair network. total According to L total The Adam optimization algorithm is used to update the parameters of the BG-MED model;

[0012] S6. Repeat steps S3-S5 multiple times to set the BG-MED network model values ​​using the trained model parameters.

[0013] S7. Apply the model to actual image restoration. Use a mask image representing the damaged area of ​​the damaged image to cover the area to be restored, obtaining the occluded image I. broken Using this as input to the BG-MED model, I broken The BG-MED model outputs the restored image I. out .

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0015] 1. The method of this invention is superior to most current image restoration techniques in terms of qualitative indicators (PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity)); in terms of visual restoration results, it is superior to most image restoration techniques, especially in the restoration of object contours, which is very consistent with object coherence and semantic consistency.

[0016] 2. To address the shortcomings of current convolutional methods used for image irregularity repair, this invention proposes dual-gated convolution, which adaptively fuses a binary mask and a gating value learned by the network to obtain the final gating value describing the image / features. Dual-gated convolution combines the original binary mask, improving the recognition accuracy of the gating value and solving the problem of insufficient accuracy of the gating value extracted by gated convolution. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention;

[0018] Figure 2 This is a schematic diagram of the repair network structure in an embodiment of the present invention;

[0019] Figure 3 This is a comparison diagram of the method of the present invention with other image restoration techniques. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0021] Example

[0022] like Figure 1 As shown, this invention provides an image inpainting method based on a dual-gated convolutional mutual encoding / decoding network, comprising the following steps:

[0023] S1. Build the network model, including the repair network and the discriminator network. Construct a mutual encoding and decoding network (BG-MED) based on dual-gated convolution as the repair network. The discriminator network adopts a global discriminator and a local discriminator.

[0024] The repair network consists of three parts: an encoder, a repair module, and a decoder.

[0025] like Figure 2 As shown, the specific network repair steps are as follows:

[0026] The encoder uses five dual-gated convolutions on the damaged image to obtain five image features of different depths and sizes. The features extracted by the encoder are respectively derived by F... en1 F en2 F en3 F en4 F en5This indicates that the subscript 'en' represents the feature extracted by the encoder, and the following number represents the layer number;

[0027] The first three features extracted by the encoder are considered as shallow features. Adaptive spatial feature fusion is used to fuse these three features to obtain texture features, which are then expressed as F. te The subscript te refers to texture, and the steps of adaptive spatial feature fusion are as follows:

[0028] Image features F en1 F en2 F en3 Adjust the structural size to match F en3 The same, where for F en1 Using two convolutions, the structure is adjusted to obtain the adjusted features, expressed as F. fusion_1 This indicates that the two convolutions are conv(64, 128, 3, 2, 1) and conv(128, 256, 3, 2, 1), respectively; for F... en2 We use a convolution to restructure the feature, obtaining the adjusted feature using F. fusion_2 This indicates that the convolution is conv(128, 256, 3, 2, 1); for F... en3 Use a convolution to further extract F en3 The features in the middle are used to obtain the adjusted features, whose structure remains unchanged, and F is used. fusion_3 This indicates that the convolution is conv(256, 256, 3, 1, 1); in this embodiment, conv() is defined as a convolution operation, specifically conv(in, out, kernel_size, stride, dilation), where in represents the number of input channels, out represents the number of output channels, kernel_size represents the number of convolution kernels, stride represents the stride of the convolution, and dilation represents the dilation factor, i.e., the spatial distance between convolution kernels;

[0029] Use three convolutions to apply F respectively fusion_1 F fusion_2 F fusion_3 Perform a channel reduction operation to reduce the number of channels in the feature to 16, resulting in three features after channel reduction: and These three convolutions have the same structure, all being conv(256, 16, 1, 1, 1);

[0030] and The combined features result in a 48-channel feature set. Then, a convolutional layer (conv(48, 3, 1, 1, 1)) is used to reduce the number of channels, resulting in a 3-channel feature. To indicate, in In the first channel feature, λ α The second channel feature is λ β The third channel feature is λ γ ;

[0031] For λ α , λ β and λ γ Using the softmax function, we obtain the three coefficients α, β, and γ, as shown in the following formula:

[0032]

[0033]

[0034]

[0035] Where the subscripts i and j represent the value of the feature in row i and column j;

[0036] Finally, using the three obtained coefficients α, β, and γ, the three features are fused to obtain the texture feature F. te The formula is as follows:

[0037] F te =α⊙F fusion_1 +β⊙F fusion_2 +γ⊙F fusion_3

[0038] Where ⊙ represents the dot product operation;

[0039] The last three features extracted by the encoder are considered as deep features. Adaptive spatial feature fusion is used to fuse these three features to obtain the structural feature F. st The subscript 'st' stands for 'structure', indicating the structure. The steps for adaptive spatial feature fusion are as follows:

[0040] Image features F en3 F en4 F en5 Adjust the structural size to match F en3 The same applies, where a convolution conv(256, 256, 3, 1, 1) is used to further extract F. en3 The characteristics in it, its structure remains unchanged, use F fusion_4 Indicates; for F en4 We use nearest neighbor upsampling, which enlarges the scale of the features, and then perform convolution (conv(512, 256, 3, 1, 1)) to adjust the structure, resulting in the adjusted feature F. fusion_5 ; For F en5Next-nearest neighbor upsampling is performed, followed by convolution (conv(512, 256, 3, 1, 1)). Then, next-nearest neighbor upsampling is performed again, followed by convolution (conv(256, 256, 3, 1, 1)). This process is then structurally adjusted to obtain the adjusted feature F. fusion_6 ;

[0041] Use three convolutions with kernel size 1 to apply F respectively fusion_4 F fusion_5 F fusion_6 Perform a channel reduction operation to reduce the number of channels in the feature to 16, resulting in three features after channel reduction: and These three convolutions have the same structure, all being conv(256, 16, 1, 1, 1);

[0042] and The combined features result in a 48-channel feature set. Then, a convolutional layer (conv(48, 3, 1, 1, 1)) is used to reduce the number of channels, resulting in a 3-channel feature. To indicate, in In the first channel feature, λ α The second channel feature is λ β The third channel feature is λ γ ;

[0043] For λ α , λ β and λ γ Using the softmax function, we obtain the three coefficients α, β, and γ, as shown in the following formula:

[0044]

[0045]

[0046]

[0047] Where the subscripts i and j represent the value of the feature in row i and column j;

[0048] Finally, using the three obtained coefficients α, β, and γ, the three features are fused to obtain the texture feature F. st The formula is as follows:

[0049] F st =α⊙F fusion_4 +β⊙F fusion_5 +γ⊙F fusion_6

[0050] Here, ⊙ represents the dot product operation.

[0051] In this embodiment, the repair module has two repair branches, which respectively repair structural feature F. st and texture features F te The repair process involves two branches, each consisting of five dense multi-scale fusion modules. The dense multi-scale fusion steps are as follows:

[0052] The current layer's feature F is further reduced by performing a channel reduction operation using the convolution conv(256, 64, 3, 1, 1), resulting in a 64-channel feature F. 64 ;

[0053] F was obtained by applying dilated convolutions with dilation factors of 1, 2, 4, and 8 respectively. 64 Convolution is performed to obtain features from four different receptive fields: F d1 F d2 F d4 F d8 The subscript d represents the feature after dilated convolution, and the number following it represents the size of the dilation factor. The four convolutions are conv(64, 64, 3, 1, 1), conv(64, 64, 3, 1, 2), conv(64, 64, 3, 1, 4), and conv(64, 64, 3, 1, 8).

[0054] To supplement features from low receptive fields into features from high receptive fields, the following steps are performed:

[0055] F′ d2 =F d1 +F d2

[0056] F′ d4 =F d1 +F d2 +F d4

[0057] F′ d8 =F d1 +F d2 +F d4 +F d8

[0058] We obtained three additional high receptive field features: F′ d2 F′ d4 F′ d8 ;

[0059] The features of four different receptive fields F d1 F′ d2 F′ d4 F′ d8 The features F are obtained by splicing them together. combine F combineThe number of channels is 256, and finally a convolution conv(256, 256, 3, 1, 1) is used to obtain the feature F′ of the next layer;

[0060] Structural feature F st and texture features F te After repairing the branch, the repaired structural feature F is obtained. ost and the repaired texture feature F ote F ost and F ote The data is then stitched and merged, and each channel is reweighted to obtain the repaired feature F. out The channel reweighting steps are as follows:

[0061] The global information is compressed, and the features of the current layer are globally pooled to obtain the aggregated information z, as shown in the following formula:

[0062]

[0063] Among them, z c This refers to the average value of the features of the c-th channel of the aggregated information z, which is the result of performing global average pooling on the c-th channel feature F of the current layer in the spatial dimension H×W; f c (i, j) represents the value of the current layer feature F in the i-th row and j-th column of channel c, representing the aggregated information z of each channel. c When pieced together, this becomes the aggregated information z;

[0064] Using the aggregated information z obtained in the previous step, the channel dependencies are captured, as shown in the following formula:

[0065] S=σ(Conv2⊙δ(Conv1⊙z))

[0066] Where σ represents the sigmoid activation function, δ represents the ReLU activation function, and Conv1 and Conv2 are two different convolutions, with Conv1 being conv(512, 16, 1, 1, 1) and Conv2 being conv(16, 512, 1, 1, 1).

[0067] Perform a dot product operation between S and the current layer feature F to obtain the channel-reweighted feature F. out The formula is as follows:

[0068] F out =S⊙F

[0069] F out Convert them into features F of the corresponding encoder feature size respectively. out1 F out2 F out3 F out4 Fout5 Then, the gating value F for each encoder feature is obtained through convolution and activation functions. out The gating values ​​based on the encoder features are supplemented accordingly, as follows:

[0070] F en1 =σ(conv(F) en1 ))⊙F out1 +F en1

[0071] Where σ represents the sigmoid activation function, and conv represents the convolution operation; for F out2 F out3 F out4 F out5 Perform similar operations until all encoder features are replenished accordingly, i.e.:

[0072] F en2 =σ(conv(F) en2 ))⊙F out2 +F en2

[0073] F en3 =σ(conv(F) en3 ))⊙F out3 +F en3

[0074] F en4 =σ(conv(F) en4 ))⊙F out4 +F en4

[0075] F en5 =σ(conv(F) en5 ))⊙F out5 +F en5 .

[0076] In this embodiment, the decoder features and the corresponding encoder features are connected in skip-layer concatenation, and then transposed convolutions are performed sequentially to finally obtain the repaired image I. out The specific steps are as follows:

[0077] For F en5 Perform a transposed convolution tran_conv(512, 512, 3, 2, 1) to obtain the first layer decoder features F. de1 In this embodiment, tran_conv() is defined as the transpose convolution operation, specifically represented as tran_conv(in, out, kernel_size, stride, dilation).

[0078] F de1With F en4 The concatenation is performed, followed by a transposed convolution trar_conv(1024, 256, 3, 2, 1) to obtain the features F of the second-layer decoder. de2 ;

[0079] With F en3 The concatenation is performed, followed by a transposed convolution tran_conv(512, 128, 3, 2, 1) to obtain the features F of the third-layer decoder. de3 ;

[0080] F de3 With F en2 The concatenation is performed, followed by a transposed convolution tran_conv(256, 64, 3, 2, 1) to obtain the features F of the fourth-layer decoder. de4 ;

[0081] F de4 With F en1 The images are then stitched together, followed by a transposed convolution tran_conv(128, 3, 3, 2, 1) to obtain the repaired image I. out ;

[0082] Here, the subscript 'de' indicates that it is a feature in the decoder, and the numbers following it indicate the order in which the decoder decodes the features.

[0083] In this embodiment, the global discriminator and the local discriminator have the same structure, both consisting of 5 convolutional layers, namely:

[0084] The convolution kernel is 3, the stride is 2, the number of input channels is 3, the number of output channels is 64, the dilation factor is 1, and the activation function is LeakyReLU (0.2).

[0085] The convolution kernel is 3, the stride is 2, the number of input channels is 64, the number of output channels is 128, the dilation factor is 1, and the activation function is LeakyReLU (0.2).

[0086] The convolution kernel is 3, the stride is 2, the number of input channels is 128, the number of output channels is 256, the dilation factor is 1, and the activation function is LeakyReLU (0.2).

[0087] The convolution kernel is 3, the stride is 2, the number of input channels is 256, the number of output channels is 512, the dilation factor is 1, and the activation function is LeakyReLU(0.2).

[0088] Convolution kernel 3, convolution stride 2, number of input channels 512, number of output channels 1, dilation factor 1, activation function Sigmoid();

[0089] LeakyReLU(0.2) represents the LeakyReLU activation function, with its slope set to 0.2 in the negative half-interval. Sigmoid() represents the sigmoid activation function, used to control the value between 0 and 1. Their formulas are as follows:

[0090] LeakyReLU(x) = max(0.2x, x)

[0091]

[0092] Where x represents the input value;

[0093] The final discriminator outputs a value between 0 and 1.

[0094] S2. Select a dataset, select multiple training images, and use the RTV smoothing algorithm in MATLAB to obtain the structure diagram of each training image. Use a random mask set during training.

[0095] S3. Use a mask on the original image I. gt By occluding the image, the damaged image I is obtained. broken , will I broken As input to the BG-MED network, the image is repaired by the BG-MED network, and the output is the repaired image I. out Specifically:

[0096] According to the original image I gt and repair Figure I out To calculate the loss value L of the global discriminator dis_g Then crop the original image I gt and repair Figure I out Take a 64*64 square area from the center of each image, and extract a partial image of each. Then, use I... lout and I lgt To indicate, according to I lout and I lgt To calculate the global discriminator loss value L dis_g The loss value L of the local discriminator dis_l Finally, the total loss value L of the discriminator is calculated. dis The formula is:

[0097]

[0098]

[0099] L dis =L dis_g +L dis_l

[0100] in, σ represents the sigmoid activation function, C g (·) represents the global discriminator, (I gt I out () represents the original image and the restored image. Indicates to I gt Calculate the average value. Indicates to I out Calculate the average value;

[0101] σ represents the sigmoid activation function, C l (·) denotes a local discriminant, (I lgt I lout () represents the original image and the repaired image. Indicates to I lgt Calculate the average value. Indicates to I lout Calculate the average value.

[0102] The Adam optimization algorithm is used to update the parameters of the global and local discriminators, specifically as follows:

[0103] Adam uses momentum and an adaptive learning rate to accelerate convergence. The final epoch value T is set as follows:

[0104] Step 1: Update t, i.e., t = t + 1, where t represents the number of update steps, and the initial value of t is 0;

[0105] Step 2: Calculate the loss function L adv_d (Θ) represents the gradient of parameter Θ, and the gradient is expressed by g. t express;

[0106] Step 3: Calculate the first moment of the gradient, denoted by m. The first moment is the average of past gradients and the current gradient, obtained from the following formula:

[0107]

[0108] Where β1 represents the first-order moment decay coefficient, Let β1 be raised to the power of t, where t represents the number of update steps, and m represents the number of updates. t-1 It is the first moment from the previous step, and the initial value of m is 0, that is, m0 is 0;

[0109] Step 4: Calculate the second moment of the gradient, denoted by v. The second moment is the average of the squares of past gradients and the square of the current gradient. The formula is as follows:

[0110]

[0111] Where β2 represents the two-moment decay coefficient. Let β be raised to the power of t, where t represents the number of update steps, and v t-1 It is the first moment from the previous step, and the initial value of v is 0, that is, v0 is 0;

[0112] Step 5: Correct the first moment m. The specific calculation formula is as follows:

[0113]

[0114] in, This represents the corrected mt;

[0115] Step 6: Correct the second moment v. The specific calculation formula is as follows:

[0116]

[0117] in, Indicates the corrected v t ;

[0118] Step 7: Update parameter Θ, and update the formula as follows:

[0119]

[0120] Where, Θ t The parameter Θ represents the current step number. t-1 The parameter represents the previous step number, and α is the learning rate, used to control the step size. α is set to 0.0002, and ∈ takes a value of 10. -8 ;

[0121] Step 8: Repeat steps 1 to 7 until step number t is updated to the final round value T.

[0122] S4. Calculate the total loss value L of the discriminator. dis The Adam optimization algorithm is used to update the parameters of the global and local discriminators; specifically, this includes:

[0123] S41, According to the original diagram I gt and image repair I out Calculate the reconstruction loss L re The formula is:

[0124] L re =||I out -I gt ||1

[0125] Where ||1 represents L1 normal form, which means the sum of absolute values;

[0126] S42. Repaired texture features F in the network model te and structural features F stTo perform the supervision, the original image and the structure image are first adjusted to a size of 64*64 to obtain a small texture image I. lte and small structure diagram I lst Then calculate the small texture map I. lte With F te The reconstruction loss between L is obtained rte Small structure diagram I lst With F st The reconstruction loss between them is obtained by L rst The formula is:

[0127] L rst =||F st -I st ||1

[0128] L rte =||F te -I te ||1

[0129] S43, According to the original diagram I gt and image repair I out And the trained VGG-16 first 4 layers network model, calculate the perceptual loss L prec The formula is:

[0130]

[0131] Where, Φ i N represents the i-th layer of the VGG-16 network. i This represents the number of parameters in the i-th layer of the network. Represents the mathematical expectation;

[0132] S44, According to the original diagram I gt and image repair I out Calculate style loss L style The formula is:

[0133]

[0134] Among them, In this context, G represents the Gram matrix, the superscript Φ indicates vgg-16, and the subscript j indicates the layer number of vgg-16. This represents the Gram matrix of x after passing through the j-th layer of vgg-16; It represents the expected value of a mathematical expression, which is the sum of the results from each level.

[0135] S45, According to the original diagram I gt and repair Figure I out and the original partial image I lout and repair of local image I lgt To calculate the adversarial loss L of the repair networkadv .

[0136] Combat loss L adv The calculation is as follows:

[0137] According to the original image I gt and repair Figure I out Calculate the global adversarial loss value L adv_g According to the original partial diagram I lout and repair of local image I lgt To calculate the local adversarial loss value L adv_l Finally, calculate the total adversarial loss value L. adv The formula is as follows:

[0138]

[0139]

[0140] L adv =L adv_g +L adv_l

[0141] in, σ represents the sigmoid activation function, C g (·) represents the global discriminator, (I gt I out () represents the original image and the restored image. Indicates to I gt Calculate the average value. Indicates to I out Calculate the average value;

[0142] σ represents the sigmoid activation function, C l (·) denotes a local discriminant, (I lgt I lout () represents the original image and the repaired image. Indicates to I lgt Calculate the average value. Indicates to I lout Calculate the average value.

[0143] S5. Calculate the loss value L of the repair network. total According to L total The Adam optimization algorithm is used to update the parameters of the BG-MED model;

[0144] The loss value L of repairing the network total The calculation formula is:

[0145] L total =250*L style +0.2*Lprec +0.2*L adv +L rst +L rte +L re .

[0146] S6. Repeat steps S3-S5 multiple times to set the BG-MED network model values ​​using the trained model parameters.

[0147] S7. Apply the model to actual image restoration. Use a mask image representing the damaged area of ​​the damaged image to cover the area to be restored, obtaining the occluded image I. broken Using this as input to the BG-MED model, I broken The BG-MED model outputs the restored image I. out .

[0148] The method of this invention outperforms most current image inpainting techniques in qualitative metrics (PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity)). Test data on the celebA human dataset is shown in Table 1 below.

[0149]

[0150] Table 1

[0151] like Figure 3 The image shown is a comparison of the restoration effects of the method of the present invention with other image restoration techniques.

[0152] It should also be noted that, in this specification, terms such as "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0153] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image inpainting method based on a dual-gated convolution-based cross- codec network, characterized in that, Comprise the following steps: S1, build a network model, including repairing network and discriminator network, build a mutual encoding and decoding network based on double gate convolution as the repairing network, and the discriminator network adopts global discriminator and local discriminator; the repairing network comprises three parts: an encoder, a repairing module and a decoder; The repairing network is specifically as follows: The encoder uses 5 double-gated convolutions for the broken picture to obtain 5 image features with different depths and sizes, and the features extracted by the encoder are represented as F en1 , F en2 , F en3 , F en4 , F en5 , where the subscript en represents the features extracted by the encoder, and the following number represents the layer number; The first three features F en1 , F en2 and F en3 extracted by the encoder are regarded as shallow features, and the first three features are fused using adaptive spatial feature fusion to obtain a texture feature, denoted by F te , where the subscript te represents texture. The steps of adaptive spatial feature fusion are as follows: The image feature F en1 is adjusted in structure to be the same as F en2 , wherein two convolutions are used on F en3 , the adjusted feature is denoted as F en3 , the two convolutions are respectively conv(64, 128, 3, 2, 1) and conv(128, 256, 3, 2, 1); one convolution is used on F en1 , the adjusted feature is denoted as F fusion_1 , the convolution is conv(128, 256, 3, 2, 1); one convolution is used on F en2 , the adjusted feature is denoted as F fusion_2 , the convolution is conv(128, 256, 3, 2, 1); one convolution is used on F en3 , the adjusted feature is denoted as F en3 , the convolution is conv(256, 256, 3, 1, 1); the structure of the adjusted feature is unchanged, and is denoted as F fusion_3 ; conv() is defined as a convolution operation, and is specifically denoted as conv(in, out, kernel_size, stride, dilation), wherein in represents the number of input channels, out represents the number of output channels, kernel_size represents the number of convolution kernels, stride represents the step length of convolution, and dilation represents a dilation factor, i.e., the spatial distance between convolution kernels. F fusion_1 , F fusion_2 , F fusion_3 are respectively reduced to 16 channels by using three same convolutions, respectively obtaining three features after channel reduction: and The three same convolutions are conv(256, 16, 1, 1, 1). Concatenate to obtain a 48-channel feature and Spliced to obtain a 48-channel feature Then use the convolution conv(48, 3, 1, 1, 1) to reduce the channel operation to obtain a 3-channel feature, represented as In , the first channel feature is λ α , the second channel feature is λ β , and the third channel feature is λ γ ; For λ α , λ β , and λ γ , three coefficients α, β, γ are obtained using a softmax function, as follows: Wherein, the subscript i and j represent the value of the feature on the i row j column; Finally, the three features are fused by using the three coefficients α, β, γ obtained, and the texture feature F is finally obtained te , and the formula is as follows: F te = a o F fusion_1 + b o F fusion_2 + g o F fusion_3 Wherein, ⊙ represents the dot product operation; The last three features F en3 , F en4 , F en5 are regarded as deep features, and the last three features are fused using adaptive spatial feature fusion to obtain a structure feature F st , where the subscript st represents structure. S2, select a data set, select multiple training pictures, and obtain the structure diagram of each training picture using the RTV smoothing algorithm in matlab, and the mask set during training uses a random mask; S3, using mask to the original picture I gt occlusion, get the damaged picture I broken , I broken as the input of the BG-MED network, through the repair of the BG-MED network, output the repaired picture I out ; S4, calculate the total loss value L of the discriminator dis using the Adam optimization algorithm, update the parameters of the global discriminator and the local discriminator; S5, calculate the loss value L of the repair network total , according to L total , update the BG-MED model parameters using the Adam optimization algorithm; S6, repeat steps S3-S5 multiple times, and set the BG-MED network model value by using the trained model parameters; S7, the model is used for actual image repair, using the mask picture representing the damaged area of the damaged picture, covering the to-be-repaired area of the damaged picture, to obtain the occluded picture I broken , as the input of the BG-MED model, I broken After the calculation output of the BG-MED model, the repaired picture I is obtained out .

2. The image inpainting method of claim 1, wherein, The steps of adaptive spatial feature fusion are as follows: Image features F en3 F en4 F en5 Adjust the structural size to match F en3 The same applies, where a convolution conv(256, 256, 3, 1, 1) is used to further extract F. en3 The characteristics in it, its structure remains unchanged, use F fusion_4 Indicates; for F en4 We use nearest neighbor upsampling, which enlarges the scale of the features, and then perform convolution (conv(512,256,3,1,1)) to adjust the structure, resulting in the adjusted feature F. fusion_5 ; For F en5 Next-nearest neighbor upsampling is performed, followed by convolution (conv(512,256,3,1,1)). Then, next-nearest neighbor upsampling is performed again, followed by convolution (conv(256,256,3,1,1)). This process is then structurally adjusted to obtain the adjusted feature F. fusion_6 ; F fusion_4 , F fusion_5 , F fusion_6 are respectively reduced to 16 channels by using three same convolutions, respectively obtaining three features after channel reduction: and The three same convolutions are specifically conv(256, 16, 1, 1, 1). Will and The combined features result in a 48-channel feature set. Then, a convolutional layer (conv(48,3,1,1,1)) is used to reduce the number of channels, resulting in a 3-channel feature. To indicate, in In the first channel feature, λ α The second channel feature is λ β The third channel feature is λ γ ; For λ α , λ β , and λ γ , the softmax function is used to obtain three coefficients α, β, γ, as follows: Wherein, the subscript i and j represent the value of the feature on the i row j column; Finally, the three features are fused by using the three coefficients α, β, γ obtained, and the texture feature F is finally obtained st , and the formula is as follows: F st = a o F fusion_4 + b o F fusion_5 + g o F fusion_6 Wherein, ⊙ represents the dot product operation.

3. The image inpainting method of claim 2, wherein, The repairing module has two repairing branches, respectively repairing the structural feature F st and the texture feature F te , and both of the two repairing branches are composed of 5 dense multi-scale fusion modules, and the dense multi-scale fusion step is as follows: The current layer feature F is subjected to a down-channel operation using a convolution conv(256, 64, 3, 1, 1) to obtain a 64-channel feature F 64 ; F 64 is convolved using dilated convolution with dilation factor of 1, 2, 4, 8 respectively to obtain four features with different receptive fields: F d1 , F d2 , F d4 , F d8 , wherein subscript d represents the feature after using dilated convolution, and the number connected after represents the size of the dilation factor, and the four convolutions are conv(64, 64, 3, 1, 1), conv(64, 64, 3, 1, 2), conv(64, 64, 3, 1, 4), and conv(64, 64, 3, 1, 8) respectively. The features of the low receptive field are supplemented into the features of the high receptive field, and the operation is as follows: F' d2 = F d1 + F d2 F′ d4 = F d1 + F d2 + F d4 F′ d8 = F d1 + F d2 + F d4 + F d8 Three supplemental high-receptive-field features are obtained: F' d2 , F' d4 , F' d8 ; The features of four different receptive fields F d1 F′ d2 F′ d4 F′ d8 The features F are obtained by splicing them together. combine F combine The number of channels is 256, and finally a convolution conv(256,256,3,1,1) is used to obtain the feature F′ of the next layer; Structural feature F st and texture feature F te After the repair branch, the repaired structural feature F is obtained ost and the repaired texture feature F ote , F ost and F ote Splicing fusion is performed, and each channel is reweighted to obtain the repaired feature F out The channel reweighting step is as follows: The global information is compressed, and the global pooling of the current layer feature is obtained to obtain the aggregation information z, and the formula is as follows: wherein z c denotes the average value of the feature of the cth channel of the aggregated information z, which is the result of performing global average pooling on the cth channel feature F of the current layer in the spatial dimension HxW; f c (i,j) denotes the value of the current layer feature F in the ith row and jth column in the cth channel, and the aggregated information z c of each channel is spliced together, i.e., the aggregated information z; The obtained aggregation information z is used to capture the dependence relationship of the channel, and the formula is as follows: S=σ(Conv2⊙δ(Conv1⊙z)) Wherein, σ represents a sigmoid activation function, δ represents a relu activation function, Conv1 and Conv2 are two different convolutions, Conv1 is conv(512, 16, 1, 1, 1), and Conv2 is conv(16, 512, 1, 1, 1); S is point multiplied with the current layer feature F to obtain the channel reweighted feature F out , the formula is as follows: F out = S O F F out Convert them into features F of the corresponding encoder feature size respectively. out1 F out2 F out3 F out4 F out5 Then, the gating value F for each encoder feature is obtained through convolution and activation functions. out The gating values ​​based on the encoder features are supplemented accordingly, as follows: F en1 = σ(conv(F en1 )) ⊙ F out1 + F en1 wherein σ denotes a sigmoid activation function, and conv represents a convolution operation; for F out2 , F out3 , F out4 , F out5 Similar operations are performed until all the encoder features are supplemented with the corresponding ones.

4. The image inpainting method of claim 3, wherein, The decoder features are skip-connected with the corresponding encoder features, and then transposed convolution is performed in turn, and finally the repaired picture I is obtained out The specific steps are as follows: F en5 tran_conv(512, 512, 3, 2, 1) to get the first layer decoder feature F de1 ; tran_conv() is defined as a transpose convolution operation, which is specifically represented as tran_conv(in, out, kernel_size, stride, dilation); F de1 With F en4 Concatenation is performed, and then transpose convolution tran_conv(1024, 256, 3, 2, 1) is performed to obtain the second layer decoder feature F de2 ; With F en3 The third layer decoder feature F is obtained by performing concatenation and then performing transpose convolution tran_conv(512, 128, 3, 2, 1) de3 ; F de3 With F en2 Concatenation is performed, and then transpose convolution tran_conv(256, 64, 3, 2, 1) is performed to obtain the fourth layer decoder feature F de4 ; F de4 With F en1 is concatenated, and transpose convolution tran_conv(128, 3, 3, 2, 1) is performed, and finally the repaired picture I out is obtained. Wherein, the subscript de represents that it is a feature in the decoder, and the numbers connected behind represent the decoding order in the decoder.

5. The image inpainting method of claim 1, wherein, The global discriminator and the local discriminator have the same structure and are composed of five convolutional layers, which are respectively: Convolution kernel 3, convolution step 2, input channel number 3, output channel number 64, expansion factor 1, and activation function LeakyReLU(0.2); Convolution kernel 3, convolution step 2, input channel number 64, output channel number 128, expansion factor 1, and activation function LeakyReLU(0.2); Convolution kernel 3, convolution step 2, input channel number 128, output channel number 256, expansion factor 1, and activation function LeakyReLU(0.2); Convolution kernel 3, convolution step 2, input channel number 256, output channel number 512, expansion factor 1, and activation function LeakyReLU(0.2); Convolution kernel 3, convolution step 2, input channel number 512, output channel number 1, expansion factor 1, and activation function Sigmoid(); LeakyReLU(0.2) represents a LeakyReLU activation function, the slope of the negative half interval is set to 0.2, Sigmoid() represents a sigmoid activation function, which is used to control the value to 0 to 1, and the formulas are as follows: LeakyReLU(x)=max(0.2x,x) Wherein, x represents the input value; The final discriminator outputs a value between 0 and 1.

6. The image inpainting method of claim 1, wherein, Step S3, total loss value L of discriminator is determined dis Specifically: According to the original image I gt and repair Figure I out To calculate the loss value L of the global discriminator dis_g Then crop the original image I gt and repair Figure I out Take a 64*64 square area from the center of each image, and extract a partial image of each. Then, use I... lout and I lgt To indicate, according to I lout and I lgt To calculate the global discriminator loss value L dis_g The loss value L of the local discriminator dis_l Finally, the total loss value L of the discriminator is calculated. dis The formula is: L dis = L dis_g + L dis_l wherein, σ denotes an activation function, C g denotes a global discriminator, (I gt denotes an original image and a repaired image, out denotes computing the mean value of I gt denotes computing the mean value of I out ​​​ σ denotes an activation function, C l (·) denotes a local discriminator, (I lgt ,I lout ) denotes a local original image and a local inpainted image, denotes computing the average value of I lgt denotes computing the average value of I denotes computing the average value of I lout denotes computing the average value of I 7. The image inpainting method of claim 1, wherein the method further comprises: In step S4, the parameters of the global discriminator and the local discriminator are updated using the Adam optimization algorithm, specifically: Adam uses momentum and adaptive learning rate to accelerate the convergence speed, sets the final round value t, and the steps are as follows: Step one, update t, that is, t=t+1, t represents the number of updates, and the initial value of t is 0; Step two, compute the loss function L adv_d (Θ) the gradient of the parameters Θ, the gradient is denoted by g t ; Step three, calculate the first moment of the gradient, the first moment is represented by m, and the first moment is the average of the past gradient and the current gradient. The formula is as follows: wherein β1represents a first-moment decay coefficient, represents t power of β1, t represents a step number of updating, m t-1 is the first-moment of the previous step, and the initial value of m is 0, i.e., m0is 0; Step four, calculate the second moment of the gradient, the second moment is represented by v, and the second moment is the average of the past gradient square and the current gradient square. The formula is as follows: wherein β2represents a second moment decay coefficient, represents t power of β2, t represents a step number of updating, v t-1 is a first moment of the previous step, and the initial value of v is 0, i.e., v0is 0; Step five, correct the first moment m, the specific calculation formula is: wherein represents the corrected m t ; Step six, correct the second moment v, the specific calculation formula is: wherein represents the corrected v t ; Step seven, update the parameters Θ, the update formula is as follows: wherein, Θ t represents a parameter of the current step number, Θ t-1 represents a parameter of the last step number, a is a learning rate, used to control the step size, a is set to 0.0002, and ∈ takes a value of 10 -8 ; Step eight, repeat steps one to seven until the step number t is updated to the final round value T.

8. The image inpainting method of claim 1, wherein, Step S4 specifically includes: S41、According to the original picture I gt and repair pictures I out , calculate the reconstruction loss L re , the formula is: L re =||I out -I gt ||1 Where ‖‖1 represents the L1 norm, and represents the sum of absolute values; S42, the texture feature F repaired in the network model te and the structure feature F st Supervision is carried out, first, the original drawing and the structure drawing are adjusted to 64*64 size, and small texture drawing I lte and small structure drawing I lst are obtained, then the reconstruction loss between small texture drawing I lte and F te is obtained as L rte , the reconstruction loss between small structure drawing I lst and F st is obtained as L rst , and the formula is: L rst =‖F st -I st ‖1 L rte =‖F te -I te ‖1 S43, according to the original picture I gt and repair pictures I out , and the trained vgg-16 front 4 layer network model, calculate the perceptual loss L prec , the formula is: wherein Φ i represents the i-th layer of the vgg-16 network, N i represents the number of parameters of the i-th layer network, denotes the mathematical expectation; S44, according to the original picture I gt and repair pictures I out Calculate the style loss L style , the formula is: wherein, in G represents a Gram matrix, the superscript Φ indicates vgg-16, the subscript j indicates the layer number of vgg-16, so represents the Gram matrix of x passing through the jth layer of vgg-16; represents the mathematical expectation, i.e. the sum of the results of each layer. S45, according to the original image I gt and the repaired image I out , and the original partial image I lout and the repaired partial image I lgt to calculate the adversarial loss L of the repaired network adv .

9. The image inpainting method of claim 8, wherein, The adversarial loss L adv The calculation of Ladv is specifically: According to the original image I gt and the repaired image I out The global adversarial loss value L adv_g is calculated according to the original local image I lout and the repaired local image I lgt The local adversarial loss value L adv_l is calculated, and finally the total adversarial loss value L adv is calculated, and the formula is as follows: L adv = L adv_g + L adv_l wherein, σ denotes an activation function, C g denotes a global discriminator, (I gt denotes a global discriminator, (I out denotes an original image and a repaired image, denotes a global discriminator, (I gt denotes a global discriminator, (I denotes a global discriminator, (I out denotes a global discriminator, (I σ denotes an activation function, C l (·) denotes a local discriminator, (I lgt ,I lout ) denotes a local original image and a local inpainted image, denotes computing the mean value of I lgt denotes computing the mean value of I lout .​ 10. The image inpainting method of claim 9, wherein, In step S5, the loss value L of the repair network is calculated total The calculation formula is: L total = 250 * L style + 0.2 * L prec + 0.2 * L adv + L rst + L rte + L re .

Images