Image inpainting model based on parallel adaptive guidance network and method thereof

Abstract

The application provides an image restoration model based on a parallel adaptive guide network and a method thereof, comprising a feature extraction module, a mutual adaptive guide module, a multi-scale perception residual module, a context joint attention module and a reconstruction module; the feature extraction module is used for extracting features of a damaged image; the mutual adaptive guide module comprises a plurality of guide filters; the multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection and a convolution attention group; the context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism. The structural features of parallel branches and complete features containing textures are guided by each other through a skip connection and a guide filter, prior knowledge is provided for the reconstruction process of the other party, and the restoration capability of the network for features is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image inpainting model based on parallel adaptive guided network and a method thereof. BACKGROUND

[0002] Image inpainting is an important and challenging task in the field of image processing, which aims to predict the appropriate pixels of the missing region. In recent years, it has become a research hotspot in the field of image processing and computer vision. Image inpainting methods are mainly divided into diffusion-based methods, matching-based methods and learning-based methods. Learning-based methods are obviously superior to the other two methods in terms of operation speed and reconstruction quality, and have obtained more attention from academia and industry. SUMMARY

[0003] The purpose of the present application is to provide an image inpainting model based on parallel adaptive guided network and a method thereof, which can improve the quality of image inpainting.

[0004] In a first aspect, the present application provides an image inpainting model based on parallel adaptive guided network, comprising a feature extraction module, a mutual adaptive guided module, a multi-scale perception residual module, a context joint attention module and a reconstruction module; the feature extraction module is used to extract features from the input damaged image; the mutual adaptive guided module includes several guided filter groups; the multi-scale perception residual module includes a multi-scale module, an intra-layer residual connection and a convolution attention group; the context joint attention module includes a same-scale attention mechanism and a cross-scale attention mechanism. The mutual adaptive guided module transmits information up and down in the parallel network through guided filters and skip connections, and the complete features containing structure and texture. The multi-scale perception residual module extracts features of different scales on a finer layer through the multi-scale module and the intra-layer residual connection, and then calibrates the weights of the features through the convolution attention group, so as to strengthen the useful feature information. The context joint attention module synthesizes better quality texture through the same-scale attention mechanism and the cross-scale attention mechanism. The reconstruction module is used for upsampling and feature reconstruction, and outputs the final repaired image.

[0005] Further, the mutual adaptive guided module includes two convolution layers with different kernel sizes and a Sigmoid layer; the 1x1 convolution in the convolution layer is used to integrate information and compress the channels of the input feature map, and the 3x3 convolution and the Sigmoid function are used to generate an attention map; then the attention map is used to recalibrate the feature map, and the weight of the error feature is reduced before the feature map is transmitted to another branch.

[0006] Further, the multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection and a convolution attention group; the multi-scale module is composed of multiple branches in parallel, each branch is composed of a convolution layer with different kernel sizes and a dilated convolution layer with different dilation rates, so that each branch has a different perception domain; the convolution attention group comprises a channel attention module and a spatial attention module; the intra-layer residual connection adds the input features and the features output by the upper branch of the multi-scale module to obtain the input features of the lower branch of the multi-scale module.

[0007] Further, the context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism; the same-scale attention mechanism is used to analyze the corresponding intra-scale attention weight according to the degree feature; and the cross-scale attention mechanism is used to analyze the corresponding cross-scale attention weight according to the degree feature.

[0008] Further, the same-scale attention mechanism comprises a 'convolution layer-RELU layer' structure, a Sigmoid layer and a transposed convolution layer; a background block of the picture is extracted as a convolution kernel of the 'convolution layer-RELU layer'; the 'convolution layer-RELU layer' structure performs convolution operation on the background block and the foreground region to calculate the similarity between the foreground block and the same-scale background block; the Sigmoid layer scales the value to the range of [0, 1] to obtain the same-scale attention score of the same-scale background block and the foreground block; and the transposed convolution layer uses the attention score as input and the extracted background block as a transposed convolution kernel to reconstruct the foreground region.

[0009] Further, the cross-scale attention mechanism comprises a down-sampling layer, a 'convolution layer-RELU layer' structure, a Sigmoid layer and a transposed convolution layer; the down-sampling layer is used to reduce the resolution of the input feature map to obtain a down-sampled feature map g; then a background block of the down-sampled feature map g is extracted as a convolution kernel of the 'convolution layer-RELU layer'; the 'convolution layer-RELU layer' structure performs convolution operation on the extracted background block and the foreground region to calculate the similarity between the foreground block and the cross-scale background block; the Sigmoid layer scales the value to the range of [0, 1] to obtain the cross-scale attention score of the cross-scale background block and the foreground block; and the transposed convolution layer uses the attention score as input and the background block re-extracted from the original feature map as a transposed convolution kernel to reconstruct the foreground region.

[0010] In a second aspect, the present application provides an image repairing method based on a parallel adaptive guide network, applied to the image repairing and reconstructing model, comprising:

[0011] S1. establishing a training set according to an image degradation model to obtain a structure graph Is of N images and an image mask Im corresponding to the N real images I; wherein N is an integer greater than 1;

[0012] S2. The structure map, real image and image mask are divided into two groups and input to the feature extraction modules of two parallel networks (texture repair network and structure repair network) to extract the features of the image;

[0013] S3. The features of the image extracted by the two parallel networks are input to the multi-scale perception residual module to extract deep features;

[0014] S4. The deep features of the structure repair network are input to the context joint attention module to obtain refined features with more complete structure and clearer texture;

[0015] S5. The features of the feature extraction modules and reconstruction modules of the two parallel networks are transmitted between the two parallel networks through the mutual adaptive guidance module to help the repair of the other party;

[0016] S6. The refined features and deep features are divided and input to the reconstruction modules of the two parallel networks, and the deconvolution is completed to complete the upsampling processing, and the finally repaired image is reconstructed;

[0017] S7. The image repair model is optimized by a loss function, and the loss function includes a deep supervision perception perception loss function, an L1 loss function, a generative adversarial loss function, a perception loss and a style loss, and the expression is:

[0018]

[0019]

[0020] L s adv = E[log(1-D S (G S (I in ,S in ,M)))] + E[log(D S (S gt ))]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] In the formula, I gt denotes a real image, S gt denotes a real structure diagram, Φ i denotes the i-th activation layer of the selected VGG19 network, F i denotes a predicted feature map, I in denotes a damaged input image, S in denotes a structure diagram of a damaged image, M denotes a mask of an input image, G S denotes a structure reconstruction branch, D S denotes a discriminator,

[0028] I re denotes a repaired image, G t denotes a texture repair branch, denotes a matrix of C*C, I denotes a current iteration number, I max denotes a maximum iteration number, wherein the parameter are 4, 1 and 0.001 respectively, wherein the parameters are 1, 5, 0.01, 180 and 1 respectively.

[0029] Further, the multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection and a convolution attention group, and the S3 comprises:

[0030] S31. The features of the image are processed through the multi-scale module to obtain multi-scale features that fuse feature information of different scales;

[0031] S32. The input features are added to the features output by the upper branch of the multi-scale module through the intra-layer residual connection to obtain the input features of the lower branch of the multi-scale module.

[0032] S33. The output features of the multi-scale module are adjusted in weight through the channel attention and spatial attention in the convolution attention group, and important features are emphasized.

[0033] Further, the context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism, and the S4 comprises:

[0034] S41. The scale-in attention weight of the multi-feature is analyzed through the same-scale attention mechanism;

[0035] S42. The cross-scale attention weight of the feature is analyzed through the cross-scale attention mechanism;

[0036] Further, the mutual self-adaptive guidance module comprises two convolution layers with different kernel sizes and a Sigmoid layer, and the S5 comprises:

[0037] The 1*1 convolution in the convolution layer is used for integrating information and compressing the channel of the input feature map, and the 3*3 convolution and the Sigmoid function are used for generating the attention map; then the attention map is used for recalibrating the feature map, and the weight of the error feature is reduced before the feature map is transmitted to another branch.

[0038] The application has the beneficial effects that: the image repairing model constructed by the application makes the structural and texture features of different scales guide each other through the connection of the guide filter, promotes repairing through two parallel networks, improves the repairing capability of the network to the features, and further improves the quality of image repairing BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments of the application will be briefly introduced as follows, and it should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0040] Figure 1 A topological structure diagram of an image repairing model provided by the embodiment of the application is shown in the figure.

[0041] Figure 2 A topological structure diagram of a mutual self-adaptive guide module provided by the embodiment of the application is shown in the figure.

[0042] Figure 3 A topological structure diagram of a multi-scale perception residual module provided by the embodiment of the application is shown in the figure.

[0043] Figure 4 A topological structure diagram of a convolution attention group provided by the embodiment of the application is shown in the figure.

[0044] Figure 5 A topological structure diagram of a context joint attention module provided by the embodiment of the application is shown in the figure.

[0045] Figure 6 A flowchart of an image repairing method provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application.

[0047] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings, and once an item is defined in one drawing, it is not necessary to further define and explain it in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0048] Please refer to Figure 1 , Figure 1 A topological structure schematic diagram of an image inpainting model provided by an embodiment of the present application.

[0049] It is found by the applicant that many existing image inpainting methods tend to use structure prior to guide the inpainting of image content to obtain higher evaluation indexes, but most of the inpainting models based on structure guidance adopt a two-stage series structure, and the non-end-to-end training mode thereof not only has complex training, but also is easily affected by unreasonable content generated in the first stage. Therefore, when inpainting a larger missing region, structure distortion, texture artifacts, color difference, and non-real problems are prone to occur. Therefore, an image inpainting model based on a parallel adaptive guidance network is provided in an embodiment of the present application to solve the above problems.

[0050] In an embodiment, the image inpainting model provided by the present application comprises a feature extraction module, a mutual adaptive guidance module, a multi-scale perception module, a context joint attention module, and a reconstruction module; the feature extraction module is used to extract features from an input damaged image; the mutual adaptive guidance module comprises a plurality of guidance filters; the multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection, and a convolution attention group; the context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism. The mutual adaptive guidance module transmits information up and down in the parallel network through guidance filters and skip connections of structure and complete features containing texture; the multi-scale perception residual module extracts features of different scales on a finer layer through the multi-scale module and the intra-layer residual connection, and then calibrates the weights of the features through the convolution attention group, so as to strengthen useful feature information. The context joint attention module synthesizes structure with higher continuity and more realistic texture through the cross-scale attention mechanism and the same-scale intra-attention mechanism. The reconstruction module is used for upsampling and feature reconstruction, and outputs a final inpainted image.

[0051] As Figure 1As shown, in the above implementation process, the shallow feature extraction module includes an n×n convolutional layer (n×nCONV); the mutual adaptive guidance module includes a 1×1 convolutional layer (1×1CONV) and a 3×3 convolutional layer (3×3CONV) to extract features from different network branches; the multi-scale perceptual residual module includes a multi-scale perceptual network, an inner residual adder, and a convolutional attention group; the context joint attention module includes a cross-scale attention generation sub-network and an intra-scale attention generation sub-network; the reconstruction module includes three residual convolutional layers, three 5×5 convolutional layers (5×5CONV), and an upsampling layer with a scaling factor of 2, used to upsample and reconstruct the features output by the multi-scale perceptual module, and output the final repaired image.

[0052] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the topology of the mutually adaptive guidance module provided in an embodiment of the present invention.

[0053] In one implementation, the mutually adaptive guidance module includes a 1×1 convolutional layer (1×1CONV) and a 3×3 convolutional layer (3×3CONV).

[0054] like Figure 2 As shown, in the above implementation process, each mutually adaptive guidance module extracts features from one network branch and passes them to another. Specifically: the input feature X1 passes through a 1×1 convolutional layer and a 3×3 convolutional layer, then through a Sigmoid layer to obtain attention weights W. The input X1 is multiplied by the weights to obtain a new feature X2, which is then added to X1. This process can be represented as:

[0055] x1 = F(x)

[0056] x2 = F1(x1)

[0057] x³ = x·s(x²)

[0058] z = x + x3

[0059] In the formula, F and F1 represent the function of the n×n convolutional layer, z represents the output feature, s represents the function of the Sigmoid layer, and "·" represents pixel-wise multiplication.

[0060] Please refer to Figure 3 and Figure 4 , Figure 3 A schematic diagram of the topology of the multi-scale sensing residual module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the topology of the convolutional attention group provided in an embodiment of the present invention.

[0061] In an embodiment, the multi-scale perception residual module comprises a multi-scale module, a CONV-ReLU layer connected one-to-one with each convolutional layer of the multi-scale perception network, a convolutional attention group, and an intra-layer residual connection; the CONV-ReLU layer is a 1*1 convolutional layer with ReLU activation function and an output channel of 1; the multi-scale module is composed of multiple branches in parallel, each branch is composed of a convolutional layer with different kernel sizes and a dilated convolutional layer with different dilation rates, so each branch has a different perception domain; the convolutional attention group adjusts the multi-scale features according to the spatial attention weight and the channel attention weight and performs feature fusion; the intra-layer residual adder adds the input features and the output features of the convolutional attention group to obtain the corresponding fused features.

[0062] As shown in the above implementation process, the multi-scale module is composed of multiple branches in parallel, each branch is composed of a convolutional layer with different kernel sizes and a dilated convolutional layer with different dilation rates, so each branch has a different perception domain; then the outputs of each branch are spliced along the channel; finally, the “CONV-ReLu” layer is input into the convolutional attention group. Figure 3

[0063] Specifically, the multi-scale perception residual module uses a multi-scale module to extract multi-scale features, and then the outputs of each branch are spliced along the channel into a feature M1 as the input of the convolutional attention group, and the expression is:

[0064] M1=C(f1+f2+f3+f4)

[0065] In the formula, f1, f2, f3, and f4 are four different scale features, and C represents the Concat layer.

[0066] As shown in the above implementation process, the multi-scale module is composed of multiple branches in parallel, each branch is composed of a convolutional layer with different kernel sizes and a dilated convolutional layer with different dilation rates, so each branch has a different perception domain; then the outputs of each branch are spliced along the channel; finally, the “CONV-ReLu” layer is input into the convolutional attention group. Figure 4

[0067] x a =F2(P​​a (M1)

[0068] x m = F3 (P m (M1)

[0069]

[0070] x out = x c · [F4 (c (P m (x c )+ p a (x c )))]

[0071] In the formula, F2, F3 represent the function functions of the convolutional layers of 2 layers 1X1 with the activation function of ReLU, F4 represents the function function of the convolutional layer of 1 layer 5x5, P a , P m respectively represent the average pooling, the maximum pooling function functions, represents element-wise addition, x out represents the output of the convolutional attention group.

[0072] Please refer to Figure 5 , Figure 5 for a topological structure diagram of a context joint attention module provided by an embodiment of the present application.

[0073] As Figure 5As shown, in one embodiment, the context joint attention module includes a same-scale attention mechanism and a cross-scale attention mechanism. The same-scale attention mechanism includes a "convolutional layer-ReLU layer" structure, a Sigmoid layer, and a transposed convolutional layer. First, the background block of the image is extracted as the convolution kernel of the "convolutional layer-ReLU layer". The "convolutional layer-ReLU layer" structure performs a convolution operation between the background block and the foreground region to calculate the similarity between the foreground block and the same-scale background block. The Sigmoid layer scales the value to the range of [0, 1] to obtain the same-scale attention score of the same-scale background block and the foreground block. The transposed convolutional layer uses the attention score as input and the extracted background block as the transposed convolutional kernel to reconstruct the foreground region. The cross-scale attention mechanism includes a downsampling layer, a "convolutional layer-ReLU layer" structure, a Sigmoid layer, and a transposed convolutional layer. The downsampling layer reduces the resolution of the input feature map to obtain a downsampled feature map g. Then, the background block of the downsampled feature map g is extracted as the convolutional kernel of the "convolutional layer-ReLU layer." The "convolutional layer-ReLU layer" structure convolves the extracted background block with the foreground region to calculate the similarity between the foreground block and the cross-scale background block. The Sigmoid layer scales the value to the range [0, 1] to obtain the cross-scale attention score between the cross-scale background block and the foreground block. The transposed convolutional layer uses the attention score as input and the re-extracted background block from the original feature map as the transposed convolutional kernel to reconstruct the foreground region.

[0074] Specifically, the outputs F of the cross-scale attention generation subnetwork and the intra-scale attention generation subnetwork are obtained. cross F in Then, use the difference between these two outputs as R. attn As input to the residual convolution, we calculate their residual features and apply them to the output F of the convolution attention group. attn With input F in Establishing skip connections between them enhances the network's authentication capabilities; this process can be represented as:

[0075] R attn =F in -F cross

[0076] F attn =ResConv(R attn )+F in

[0077] F out =F attn +F input

[0078] In the formula, F cross F is used to generate output features for the cross-scale attention subnetwork.in F denotes the output feature of the scale-in attention generation subnetwork, input denotes the input feature of the context joint attention module, ResConv denotes a residual convolution function, and F denotes the output feature of the context joint attention module. out denotes the output feature of the context joint attention module,

[0079] In an embodiment, the present application provides an image inpainting method based on a parallel adaptive guidance network, which is applied to the image inpainting reconstruction model described above, and the specific content is as follows.

[0080] S1. A training set is established according to an image degradation model, and a structure graph Is of N images and an image mask Im corresponding to the N real images I are obtained; wherein N is an integer greater than 1.

[0081] S2. The structure graph, real image and image mask are divided into two groups and input to the feature extraction modules of two parallel networks (texture repair network and structure repair network) to extract the features of the images.

[0082] S3. The features of the images extracted by the two parallel networks are input to a multi-scale perception residual module to extract deep features.

[0083] S4. The deep features of the structure repair network are input to a context joint attention module to obtain refined features with more complete structure and clearer texture.

[0084] S5. The features of the feature extraction modules and reconstruction modules of the two parallel networks are transmitted between the two parallel networks through the mutual adaptive guidance module to promote the repair of the images.

[0085] S6. The structure features and complete features extracted by the structure repair network and the texture rich network are respectively input to the reconstruction modules of the two parallel networks, deconvolution is performed to complete the up-sampling processing, and the finally repaired image is reconstructed.

[0086] S7. The image repair model is optimized by a loss function, and the loss function includes a deep supervision perception loss function, an L1 loss function, a generative adversarial loss function, a perception loss and a style loss, and the expression is:

[0087]

[0088]

[0089] L s adv = E[log(1-D S (G S (I in ,S inM) + E[log(D S (S gt ))

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] where I gt denotes the real image, S gt denotes the real structure map, F i denotes the i-th activation layer of the selected VGG19 network, G i denotes the predicted feature map, I in denotes the damaged input image, S in denotes the structure map of the damaged image, M denotes the mask of the input image, G S denotes the structure reconstruction branch, D S denotes the discriminator,

[0097] I re denotes the repaired image, G t denotes the texture repair branch, denotes a matrix of C*C, I denotes the current iteration number, I max denotes the maximum iteration number, wherein the parameter are 4, 1, 0.001 respectively, wherein the parameters are 1, 5, 0.01, 180, 1 respectively.

[0098] Further, the multi-scale perception residual module comprises a multi-scale perception network, an inner-layer residual adder and a convolution attention group, and the S3 comprises:

[0099] S31. The features of the image are processed by the multi-scale perception network to obtain multi-scale features that fuse feature information of different scales;

[0100] S32. The multi-scale features are analyzed by the convolution attention group to obtain spatial attention weights and channel attention weights, and the multi-scale features are adjusted and fused according to the spatial attention weights and the channel attention weights;

[0101] S33. Adding the input features to the features after the convolutional attention group through the inner layer adder to obtain corresponding deep features.

[0102] Further, the context joint attention module includes a same-scale attention mechanism and a cross-scale attention mechanism, and the S4 includes:

[0103] S41. Analyzing the scale-in attention weight of the multiple features through the same-scale attention mechanism;

[0104] S42. Analyzing the cross-scale attention weight of the features through the cross-scale attention mechanism;

[0105] Further, the mutual self-adaptive guidance module includes two convolutional layers with different kernel sizes and a Sigmoid layer, and the S5 includes:

[0106] The 1x1 convolution in the convolutional layer is used for integrating information and compressing the channel of the input feature map, and the 3x3 convolution and the Sigmoid function are used for generating an attention map; then the attention map is used for recalibrating the feature map, and the weight of the error feature is reduced before the feature map is transmitted to another branch.

[0107] Specifically, the embodiment of the present application uses Place2, CelebA-HQ and Paris as the training set and test set, and uses a rolling guidance filter to process the real image to obtain the structure map of the image, wherein the parameter σs is 3 and the parameter σr is 0.05.

[0108] After the training set is constructed, the training and testing of the model can be performed on the pytorch framework. Each time, 8 image blocks of 256x256 are randomly input. The Adam gradient descent method is used to realize the optimization of the network parameters, wherein the parameter setting of the Adam optimizer is β1=0, β2=0.999, and the learning rate is 1x10e -4 The peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are used to evaluate the performance of the model.

[0109] The present application uses three data sets of Place2, CelebA-HQ and Paris to test the performance of the model, and five representative image restoration methods are selected for comparison with the experimental results of the present application, and the experimental results are shown in Table 1, wherein PAGN is the method proposed in the present application, and the five representative image restoration methods include:

[0110] EC: The method proposed by Nazeri et al., reference "K. Nazeri, E. Ng, T. Joseph, F. Qureshi, M. Ebrahimi. EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [J], in: Proceedings of the 2019 ICCVW, 2019."

[0111] CSA: The method proposed by Liu et al., reference "H. Liu, B. Jiang, Y. Xiao, Coherent Semantic Attention for Image Inpainting, in: Proceedings of the 2019 ICCV, 2019, pp. 4169-4178."

[0112] RFR: The method proposed by Li et al., reference "J. Li, N. Wang, L. Zhang, B. Du and D. Tao (2020). Recurrent Feature Reasoning for Image Inpainting, in: Proceedings of the 2020 CVPR, 2020, pp. 7757-7765."

[0113] SF: The method proposed by Ren et al., reference "Y. Ren, X. Yu, R. Zhang, Structure Flow: Image Inpainting via Structure-aware Appearance Flow, in: Proceedings of the 2019 ICCV, 2019, pp. 181-190."

[0114] MEDEF: The method proposed by Liu et al., reference "H. Liu, B. Jiang, Y. Song, W. Huang, and C. Yang, Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations, in: Proceedings of the 2020 ECCV, 2020, pp. 725-741."

[0115] As can be seen from Table 1 (the optimal values are in black and bold), in most cases, the PSNR and SSIM of the present application are the highest, and the repair effect is significantly better than some representative image repair methods at present.

[0116] Table 1 Comparison of average PSNR and SSIM values on 2 test sets

[0117] Table 1

[0118]

[0119] To sum up, the embodiment of the present application provides an image repair model and method based on a parallel adaptive guide network, which makes different scale structure and texture features guide each other through the connection of the guide filter to promote repair, thereby improving the network's repair capability for features, and further improving the quality of image repair.

[0120] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. An image inpainting model based on a parallel adaptive guidance network, characterized in that, The parallel network is a structure repairing network and a texture rich network; The method comprises a feature extraction module, a mutual adaptive guiding module, a multi-scale perception residual module, a context joint attention module and a reconstruction module; The feature extraction module is used for extracting features from an input damaged image; The mutual adaptive guiding module comprises a plurality of guiding filters; The multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection and a convolution attention group; The context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism; The mutual adaptive guiding module transmits information in the parallel network through guiding filters and skip connections of complete features containing structure and texture; The multi-scale perception residual module extracts features of different scales on a finer layer through the multi-scale module and the intra-layer residual connection, and then calibrates the features through the convolution attention group to strengthen useful feature information; The context joint attention module synthesizes structure with higher continuity and texture with higher authenticity through the cross-scale attention mechanism and the same-scale intra-attention mechanism. The reconstruction module is used for upsampling and feature reconstruction, and outputs a final repaired image.

2. The image inpainting model based on parallel adaptive guidance network according to claim 1, wherein, The mutual adaptive guiding module comprises two convolution layers with different kernel sizes and a Sigmoid layer; The 1x1 convolution in the convolution layer is used for integrating information and compressing the channels of the input feature map, and the 3x3 convolution and the Sigmoid function are used for generating an attention map; Then the attention map is used for recalibrating the feature map, and the weight of the error feature is reduced before the feature map is transmitted to another branch.

3. The image inpainting model based on parallel adaptive guidance network according to claim 1, wherein, The multi-scale perception residual module comprises a multi-scale module, an intra-layer residual connection and a convolution attention group; The multi-scale module is composed of a plurality of branches in parallel, each branch is composed of a convolution layer with different convolution kernel sizes and a dilated convolution layer with different dilated rates, and each branch has a different perception domain; The convolution attention group comprises a channel attention module and a spatial attention module; The intra-layer residual connection adds the input feature and the feature output by the upper branch of the multi-scale module to obtain the input feature of the lower branch of the multi-scale module.

4. The image inpainting model based on parallel adaptive guidance network according to claim 1, wherein, The multi-scale perception residual module comprises a multi-scale module, a CONV-ReLU layer connected one by one with each convolution layer of the multi-scale perception network, a convolution attention group and an intra-layer residual connection; The CONV-ReLU layer is a 1x1 convolution layer with a ReLU activation function with an output channel of 1; The multi-scale module is composed of a plurality of branches in parallel, each branch is composed of a convolution layer with different convolution kernel sizes and a dilated convolution layer with different dilated rates, and each branch has a different perception domain; The convolution attention group adjusts the multi-scale features according to the spatial attention weight and the channel attention weight and fuses the features; The intra-layer residual connection adds the input feature and the output feature of the convolution attention group to obtain the corresponding fused feature.

5. The image inpainting model based on parallel adaptive steering network according to claim 1, wherein, The context joint attention module comprises a same-scale attention mechanism and a cross-scale attention mechanism; The same-scale attention mechanism analyzes the intra-scale attention weight according to the same-scale feature. The cross-scale attention mechanism is used to analyze the corresponding cross-scale attention weight according to the cross-scale feature.

6. The image inpainting model based on parallel adaptive guidance network according to claim 5, wherein, The same-scale attention mechanism includes a "convolutional layer-rectified linear unit (ReLU) layer" structure, a Sigmoid layer and a transposed convolutional layer; a background block of the picture is extracted as a convolution kernel of the "convolutional layer-rectified linear unit (ReLU) layer"; the "convolutional layer-rectified linear unit (ReLU) layer" structure performs convolution operation on the background block and the foreground region to calculate the similarity between the foreground block and the same-scale background block; The Sigmoid layer scales the value to the range of [0, 1] to obtain the same-scale attention score of the same-scale background block and the foreground block; The transposed convolutional layer uses the attention score as input and the extracted background block as a transposed convolution kernel to reconstruct the foreground region. The cross-scale attention mechanism includes a down-sampling layer, a "convolutional layer-rectified linear unit (ReLU) layer" structure, a Sigmoid layer and a transposed convolutional layer; the "down-sampling layer" is used to reduce the resolution of the input feature map to obtain a down-sampled feature map g; then a background block of the down-sampled feature map g is extracted as a convolution kernel of the "convolutional layer-rectified linear unit (ReLU) layer"; the "convolutional layer-rectified linear unit (ReLU) layer" structure performs convolution on the extracted background block and the foreground region to calculate the similarity between the foreground block and the cross-scale background block; The Sigmoid layer scales the value to the range of [0, 1] to obtain the cross-scale attention score of the cross-scale background block and the foreground block; The transposed convolutional layer uses the attention score as input and the background block re-extracted from the original feature map as a transposed convolution kernel to reconstruct the foreground region.

7. A method for image inpainting based on parallel adaptive guidance network, applied to the image inpainting model of any one of claims 1-6, characterized in that, The method comprises the following steps: S1. A training set is established according to an image degradation model to obtain a structure diagram Is of N images and an image mask Im corresponding to the N real images I; wherein N is an integer greater than 1; S2. The structure diagram, the real image and the image mask are divided into two groups and input to feature extraction modules of two parallel networks to extract features of the image; the two parallel networks are a structure repair network and a texture enrichment network; S3. The features of the image extracted by the two parallel networks are input to a multi-scale perception module to extract deep features; S4. The deep features of the structure repair network are input to a context joint attention module to obtain refined features with more complete structure and clearer texture; S5. The features of the feature extraction modules of the two parallel networks and the reconstruction modules are transmitted between the two parallel networks through a mutual adaptive guidance module to promote the repair of the image; S6. The refined features and the deep features are divided and input to reconstruction modules of the two parallel networks to perform deconvolution to complete up-sampling processing and reconstruct the finally repaired image; S7. The image repair model is optimized through a loss function; the loss function uses the L1 error between the repaired image and the corresponding real image.

8. The image inpainting method based on parallel adaptive steering network according to claim 7, characterized in that, The multi-scale perception residual module includes a multi-scale module, an intra-layer residual connection and a convolutional attention group; S3 includes: S31. The features of the image are processed through the multi-scale module to obtain multi-scale features that fuse feature information of different scales; S32. Adjusting the multi-scale features according to the spatial attention weights and the channel attention weights and performing feature fusion by the convolution attention group; S33. Adding the input features and the features after the convolution attention group by the inner layer adder to obtain the corresponding deep features.

9. The image inpainting method based on parallel adaptive steering network according to claim 7, characterized in that, The context joint attention module includes the same scale attention mechanism and the cross-scale attention mechanism; the S4 includes: S41. Analyzing the scale-in attention weights of the multi-features by the same scale attention mechanism; S42. Analyzing the cross-scale attention weights of the features by the cross-scale attention mechanism.

10. The image inpainting method based on parallel adaptive steering network according to claim 7, characterized in that, The mutual self-adaptive guidance module includes two convolution layers with different kernel sizes and a Sigmoid layer, and the S5 includes: The 1×1 convolution in the convolution layer is used for integrating information and compressing the channels of the input feature map, the 3×3 convolution and the Sigmoid function are used for generating the attention map; then the attention map is used for recalibrating the feature map, and the weight of the error feature is reduced before the feature map is transmitted to another branch.

Images