An image inpainting method of structure and texture mutual promotion

Abstract

The present application belongs to the field of natural image processing, and particularly relates to a structure and texture mutual promotion image repairing method. The present application follows the basic framework of a generative adversarial network, and comprises two parts of a double-branch parallel coupling generator and a double-branch discriminator. The double-branch parallel coupling generator utilizes two branches to respectively reconstruct a structure and synthesize a texture, in the process, the two branches frequently interact in both directions, so that the structure and the texture acquire complementary information from each other, form a mode of structure guiding texture synthesis and texture promoting structure reconstruction, and meanwhile, the frequent interaction of the two branches realizes mutual perception of the structure and the texture, and accurately ensures the consistency between the two. The double-branch discriminator discriminates the generated image of the generator from the structure and the texture, contains two processes of structure and texture adversarial, and supervises the generator to generate an image with reasonable structure and texture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of natural image processing, and particularly relates to an image inpainting method with mutual promotion of structure and texture. BACKGROUND

[0002] Image inpainting aims to predict the unknown region in the generated image according to the known region in the damaged image, so as to synthesize a visually reasonable high-quality image. Image inpainting is widely used in image editing, old photo restoration, target removal and other tasks. However, generating reasonable image content is extremely challenging.

[0003] At present, image inpainting methods mainly include traditional methods and deep learning-based methods. Traditional methods mainly include diffusion-based methods and small block-based methods. The idea of diffusion-based methods is to derive an iterative algorithm through a variational problem, so as to diffuse the boundary pixels of the known region to the unknown region. The small block-based method aims to find the most relevant small block in the known region to the unknown region, so as to fill the missing region in the form of copying the small block. However, these traditional methods can only mechanically reason at the image level and cannot mine high-level semantic features of the image, and the effect of filling a larger missing region is not ideal.

[0004] In recent years, with the development and application of deep learning and neural networks, deep learning-based image inpainting methods have made up for the shortcomings of traditional methods and achieved better performance. Among them, convolutional neural networks and generative adversarial networks are widely used in image inpainting tasks. Pathak et al. proposed the first deep learning-based inpainting method, designed a context encoder, adopted an encoder-decoder structure, embedded a channel fully connected layer to improve the network prediction ability, and applied adversarial training to the network to make the image clearer. Since then, many researchers have improved the context encoder. Some researchers use convolutional neural networks as a tool to seek better feature representation and feature reasoning methods. Iizuka et al. capture more distant context information by stacking multiple layers of dilated convolution. Wang et al. extract multi-scale features of the image using different size convolution kernels. Zeng et al. use a context conversion module to collect more diverse context information and realize adaptive filtering of context information. Some researchers apply the ideas of traditional methods to deep methods. For example, Yu et al. constructed a two-stage framework consisting of coarse inpainting and fine inpainting, and introduced a context attention mechanism to calculate the cosine similarity between feature blocks, aggregate the most relevant feature blocks in the known region to the unknown region, and realize feature restoration. Subsequently, Liu et al. proposed a semantic coherent attention mechanism to solve the problem of semantic incoherence of the context attention mechanism, making the local details of the image smoother. However, the above methods lack the guidance of structural prior knowledge, resulting in incorrect structures of the generated images.

[0005] In recent years, some researchers have explored reasonable structural prior tools. In 2019, Nazeri et al. proposed EdgeConnect, which first introduced edges into the image inpainting task. The main idea is: in the first stage, the damaged edges are repaired, and the repaired edges are used as structural priors to guide the repair of image texture details in the second stage. In the same year, Ren et al. proposed StructureFlow, which uses a smooth structure image that preserves edges as a structural prior and applies it to a two-stage model. However, these two-stage architectures have drawbacks. It is very difficult to obtain reasonable structural priors in the first stage, and unreasonable structural priors will adversely affect the subsequent image inpainting results. Subsequently, a series of novel one-stage models were proposed to model the structure and texture of images at the feature map level. Li et al. proposed a visual structure progressive reconstruction network that uses an encoder-decoder structure to reconstruct edges while repairing images. In 2020, Liu et al. proposed a structure and texture mutual encoder network that aims to use low-level features with high resolution to synthesize image textures and high-level features with low resolution to reconstruct image structures. However, these methods share an encoding-decoding network for structure and texture, which can cause confusion and mixing of image structure and texture. In 2021, some researchers proposed a dual-encoder-single-decoder network that uses two encoders to encode structure and texture information, respectively, and fuses them in a shared decoder. However, the above structure-texture modeling methods have a common problem. The structure reconstruction and texture synthesis processes are relatively separate, and they cannot obtain complementary information, which is not conducive to maintaining the consistency of structure and texture.

[0006] The existing methods have the following disadvantages: first, traditional image inpainting methods cannot capture deep semantic features of images; second, methods based on feature representation and feature reasoning cannot obtain correct structural priors for guidance; third, in existing structure-texture modeling methods, the relationship between structure and texture is not fully considered, and the two are relatively separate, making it difficult to obtain guiding information from each other, which is not conducive to maintaining the consistency of structure and texture.

[0007] Therefore, the idea of the present application is that there should be information exchange and mutual influence between structure reconstruction and texture synthesis to ensure consistency between the two, while achieving the effect of mutual promotion of structure and texture. SUMMARY

[0008] The purpose of the present application is to provide a structure and texture mutual promotion image restoration method. This method follows the basic framework of the generative adversarial network, including two parts of double-branch parallel coupled generator and double-branch discriminator. The double-branch parallel coupled generator utilizes two branches to respectively reconstruct the structure and synthesize the texture, in this process, the two branches frequently interact in both directions, so that the structure and the texture obtain complementary information from each other, forming a pattern of structure guiding texture synthesis and texture promoting structure reconstruction, at the same time, the frequent interaction of the two realizes the mutual perception of structure and texture, accurately ensuring the consistency between them. The double-branch discriminator respectively discriminates the true and false of the image generated by the generator from the structure and the texture, contains two processes of structure and texture confrontation, and supervises the generator to generate images with reasonable structure and texture.

[0009] In order to achieve the above purpose, the present application provides the following technical scheme:

[0010] A structure and texture mutual promotion image restoration method, comprising the following steps:

[0011] Step a, building the network framework of double-branch parallel coupled generator:

[0012] The double-branch parallel coupled generator takes edge as structure representation and image as texture representation to model the structure and texture of the image; the framework contains parallel structure branch and texture branch; the structure branch takes damaged edge E in , damaged gray image Y in and binary mask M as input, aiming to reconstruct the damaged structure, wherein in the binary mask, 0 represents the undamaged area and 1 represents the damaged area; the texture branch takes damaged image I in and binary mask M as input, aiming to synthesize texture details for the damaged area; at the same time, two feature interaction mechanisms, structure-texture interaction unit STIU and spatial domain and channel domain based interaction module SCIM, are embedded between the two branches, to realize the bidirectional interaction of structure features and texture features, so that they can obtain complementary information from each other to enhance their own features;

[0013] The double-branch parallel coupled generator includes four parts: 1, structure-texture interaction encoder STIE; 2, structure-texture long-distance interaction STDI; 3, structure-texture interaction decoder STID; 4, structure-texture fusion STF;

[0014] The structure-texture interaction encoder STIE includes a structure encoder SE, a texture encoder TE, and three structure-texture interaction units STIU; the structure encoder SE and the texture encoder TE both include three convolutional layers, the structure encoder SE extracts structure features, the texture encoder TE extracts texture features, and the structure features and the texture features are bidirectionally interacted and complementary information is extracted through the structure-texture interaction unit STIU; the steps of the structure-texture interaction encoder STIE are represented by the following formula:

[0015]

[0016] In formula 1, i = 1, 2, 3; is the structure feature output by the i-th convolutional layer in SE; T i TE is the texture feature output by the i-th convolutional layer in TE; and are the i-th convolutional layers in SE and TE, respectively; and are both enhanced structure features; and are both enhanced texture features; is the i-th structure-texture interaction unit in STIE; “||” means a channel dimension concatenation operation; according to the above method, and are calculated

[0017] The structure-texture long-distance interaction STDI includes a structure bottleneck layer SB, a texture bottleneck layer TB, and four SCIMs; the structure bottleneck layer SB and the texture bottleneck layer TB both consist of four residual dilated convolution modules RDCB, which can search for context information using different dilated convolution layers to capture the context information of structure and texture, respectively; the interaction modules SCIM based on the spatial domain and the channel domain pay attention to the captured structure features and texture features in the spatial domain and the channel domain, respectively, to realize the long-distance interaction of structure-texture; the mode of the structure-texture long-distance interaction STDI is represented in the following form:

[0018]

[0019] In formula 2, j = 1, 2, 3, 4; is the long-distance structure feature output by the j-th RDCB in SB; is the long-distance texture feature output by the j-th RDCB in TB; and are the j-th residual dilated convolution modules in SB and TB, respectively; and are enhanced long-distance structural features; and are enhanced long-distance texture features; j (·,·) is the jth spatial domain and channel domain based interaction module; In this way, the following is calculated and Both are then input to the structure-texture interaction decoder described below;

[0020] The structure-texture interaction decoder STID includes a structure decoder SD, a texture decoder TD, and three STIU; the structure decoder SD and the texture decoder TD both include two convolution layers and two deconvolution layers, respectively recovering the structure and the texture; the structure-texture interaction unit STIU is used to interact the structure and the texture; specifically, the first convolution layer in the structure decoder SD and the texture decoder TD is used to integrate and to obtain and The expression of the subsequent process is as follows:

[0021]

[0022] In formula 3, k = 1, 2, 3; and are enhanced structural features; and are enhanced texture features;

[0023] is the kth structure-texture interaction unit in STID; is the structure feature output by the kth convolution layer in SD; is the structure feature output by the k+1th convolution layer in SD; is the texture feature output by the kth convolution layer in TD; is the texture feature output by the k+1th convolution layer in TD; and is the k+1th convolution layer in SD and TD; in this way, the recovered structure feature and the texture feature

[0024] At the same time, a structure feature pyramid and a texture feature pyramid are constructed in the structure-texture interaction decoder STID;

[0025] In this way, the recovered structure feature and the texture feature After that, a structure-texture interaction unit is embedded to fuse the two, denoted as the following form:

[0026]

[0027] In formula 4, F1 and F2 are two fused features; STIU STF (·, ·) is a structure-texture interaction unit; is the recovered structure feature; is the recovered texture feature;

[0028] Finally, the two fused features are concatenated along the channel dimension and input to the output layer to generate the final predicted image I out .

[0029] Step b, design of structure-texture bidirectional interaction module:

[0030] As seen from step a, two structure-texture bidirectional interaction mechanisms are embedded between the structure branch and the texture branch, which are structure-texture interaction unit STIU and spatial and channel domain based interaction module SCIM, respectively; both structure-texture interaction unit STIU and spatial and channel domain based interaction module SCIM follow the following form:

[0031]

[0032] In formula 5, and are the structure feature and the texture feature after interaction enhancement, respectively; S and T are the structure feature and the texture feature, respectively; f() and g() are learnable weights, which control the trade-off between structure (texture) information and texture (structure) information; represents element-wise addition; ⊙ represents element-wise multiplication;

[0033] In structure-texture interaction unit STIU, the expression forms of f(·) and g(·) are as follows:

[0034]

[0035] In formula 6, W f and W g are gate weights; σ is a Sigmoid function; Conv 3×3 (·) is a convolution layer with a convolution kernel size of 3; S and T are the structure feature and the texture feature, respectively;

[0036] In spatial and channel domain based interaction module SCIM, f(·) and g(·) are realized through spatial attention and channel attention, denoted as the following form:

[0037]

[0038] In formula 7, P f and P g are the gating weights for spatial attention; Q f and Q g are the gating weights for channel attention; σ is the Sigmoid function; Conv 5×5 (·) is a convolutional layer with a kernel size of 5; Avgpool(·) and Maxpool(·) respectively refer to the average pooling operation and the maximum pooling operation; MLP(·) is a multi-layer perceptron; S and T are the structure feature and the texture feature, respectively;

[0039] Step c, constructing a dual-branch discriminator:

[0040] The dual-branch discriminator aims to distinguish the generated image I out from the real image I, and includes a parallel structure discriminator and a texture discriminator, which respectively distinguish the structure and the texture of the image; wherein, the structure discriminator and the texture discriminator both contain 5 convolutional layers, and are connected with a Sigmoid function at the end;

[0041] The texture discriminator takes the generated image I out or the real image I as input, and can output the texture scores t fake and t true of the two; for the structure discriminator, the generated edge E out of the generated image I out is first extracted by an edge detector, which is used as the structure representation of the generated image; the structure discriminator takes the generated edge E out or the real edge E as input, and outputs the structure scores s fake and s true of the two; wherein, the edge detector is realized by a residual network module; the above process is expressed as:

[0042] t fake = D s (E out ), t true = D s (E), s fake = D t (I out ), s true = D t (I). Formula 8

[0043] In formula 8, t fake is the score of the texture of the generated image by the texture discriminator; t true is the score of the texture of the real image by the texture discriminator; s fake is the score of the structure of the generated image by the structure discriminator; strue is the score of the real image structure for the structure discriminator; D s and D t are the structure discriminator and the texture discriminator, respectively; I out is the generated image; I is the real image; E out is the generated edge; E is the real edge;

[0044] Subsequently, the texture and structure scores t fake and s fake are spliced in the channel dimension to obtain the total score x1, and the texture and structure scores t true and s true are spliced in the channel dimension to obtain the total score x2, and the two scores will be used for subsequent calculation of the adversarial loss;

[0045] Step d, training of the network:

[0046] The two structure-texture interaction mechanisms designed in step b are embedded into the double-branch parallel coupled generator in step a to realize bidirectional interaction between the structure features and the texture features, so that the structure and the texture can promote and contribute to each other, so that the double-branch parallel coupled generator can generate an image with reasonable structure and texture; the double-branch discriminator in step c scores and discriminates the true and false of the image generated by the double-branch parallel coupled generator, supervises the generation quality of the generator, and further stimulates the generation potential of the generator.

[0047] First, the adversarial loss is calculated to update the network parameters of the double-branch discriminator; then, the network parameters of the double-branch discriminator are frozen, and the L1 reconstruction loss, the perception loss, the style loss, the adversarial loss and the pyramid loss are calculated to update the network parameters of the double-branch parallel coupled generator; according to this training method, the discrimination ability of the discriminator is continuously improved, and the generation ability of the generator is also continuously improved, so that more reasonable images are generated.

[0048] In step d, the adversarial loss involves the generative adversarial process of the generator and the discriminator; the process is represented as:

[0049]

[0050] In formula 9, log() is a logarithmic function; x1 and x2 are the scores of the discriminator for the generated image and the real image, respectively; I out is the generated image; I is the real image; E out is the generated edge; E is the real edge; and the same as the idea of the generative adversarial network, through the above formula, the effect of the generator and the discriminator in the adversarial generation is realized.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] The application proposes a novel double-branch parallel coupling generator, which uses two branches to respectively perform structure reconstruction and texture synthesis, and at the same time, the structure branch and the texture branch communicate with each other and guide each other, effectively ensuring the consistency of structure and texture.

[0053] The application adopts a novel double-branch discriminator to respectively discriminate and score the structure and texture of an image, so as to supervise the generation process of the generator. According to this generation and confrontation process, the generator can generate an image with more reasonable structure and clearer texture.

[0054] In summary, in the double-branch parallel coupling generator, the frequent bidirectional interaction of structure and texture can realize mutual perception of the two, accurately ensure the consistency of structure and texture, and promote the mode of structure guiding texture and texture guiding structure. In the double-branch discriminator, the structure and texture of the generated image are respectively discriminated and scored, which improves the comprehensiveness of the discrimination process. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1a For the damaged image;

[0056] Figure 1b For the repaired image;

[0057] Figure 1c For the real image;

[0058] Figure 1d For the structure edge of the generated image;

[0059] Figure 2a For the schematic diagram of the structure pyramid reconstruction process of the image;

[0060] Figure 2b For the schematic diagram of the texture pyramid reconstruction process of the image;

[0061] Figure 3 For the flowchart of the double-branch parallel coupling generator of the application;

[0062] Figure 4 For the flowchart of the double-branch discriminator of the application;

[0063] Figure 5 For the schematic diagram of the structure-texture interaction unit STIU embedded in the application;

[0064] Figure 6 For the schematic diagram of the spatial domain and channel domain based interaction module SCIM embedded in the application;

[0065] Figure 7 For the specific implementation step diagram of the application. DETAILED DESCRIPTION

[0066] The specific embodiments of the present application are further described below with reference to the accompanying drawings.

[0067] The present application takes the structure and texture of an image as two important features of the image, and establishes a link between the two in a proper manner, thereby promoting the pattern of structure guiding texture and texture promoting structure.

[0068] The basic idea of the present application is:

[0069] Firstly, a dual-branch parallel coupled generator is built. The parallel structure branch and the texture branch are used to recover the structure and texture of the damaged image respectively. In the process of recovering the structure and texture, the two branches interact with each other, so that the structure information and the texture information influence and guide each other, thereby constructing the pattern of structure guiding texture synthesis and texture promoting structure reconstruction. The frequent interaction between the structure and the texture can also ensure the consistency between the structure and the texture.

[0070] Secondly, in order to complete the above interaction between the structure and the texture, a structure-texture bidirectional interaction module is designed and embedded between the structure branch and the texture branch, so that the auxiliary information in the structure feature can adaptively guide the texture feature synthesis, and the guiding information in the texture feature can also adaptively promote the structure feature reconstruction.

[0071] Thirdly, a dual-branch discriminator is designed to distinguish between true and false generated images. The dual-branch discriminator adopts two branches, namely a structure discriminator and a texture discriminator, which can evaluate the structure and texture of the generated image and supervise the structure and texture generation of the generator.

[0072] Finally, the weighted sum of L1 reconstruction loss, perceptual loss, style loss, adversarial loss and pyramid loss is used as the total loss function, and the Adam optimization algorithm is used to train and optimize the model. Different models are obtained according to different training sets.

[0073] An image restoration method based on mutual promotion of structure and texture, comprising the following steps:

[0074] Step a, building a network framework of a dual-branch parallel coupled generator:

[0075] As shown in Figure 3 , the dual-branch parallel coupled generator takes edges as structure representation and images as texture representation to model the structure and texture of an image. Specifically, the framework includes a parallel structure branch and a texture branch. The structure branch takes the damaged edge E in , the damaged grayscale image Y in and the binary mask M (0 represents the undamaged area and 1 represents the damaged area) as input, aiming to reconstruct the damaged structure; the texture branch takes the damaged image I inand binary mask M as input, aiming to synthesize texture details for the damaged region. Meanwhile, two feature interaction mechanisms, i.e., structure-texture interaction unit (STIU) and spatial-channel interaction module (SCIM), are embedded between the two branches to realize the bidirectional interaction between structure and texture features, so that they can obtain complementary information from each other to enhance their own features.

[0076] The dual-branch parallel coupled generator includes four parts: 1) structure-texture interactive encoder (STIE); 2) structure-texture distant interaction (STDI); 3) structure-texture interactive decoder (STID); 4) structure-texture fusion (STF).

[0077] In Figure 3 The diagram of the dual-branch parallel coupled generator is shown in FIG. 1. The operation mode is described in detail as follows:

[0078] 1) Structure-texture interactive encoder (STIE):

[0079] The encoder can extract the feature representation of the image. The structure-texture interactive encoder STIE includes a structure encoder (SE), a texture encoder (TE), and three structure-texture interaction units STIU. Among them, the structure encoder SE and the texture encoder TE both contain three convolutional layers. The structure encoder SE is responsible for extracting structure features, and the texture encoder TE is responsible for extracting texture features. The structure features and the texture features are bidirectionally interacted through the structure-texture interaction unit STIU to extract complementary information. The steps of the structure-texture interactive encoder STIE are represented as follows:

[0080]

[0081] In formula 1, i = 1, 2, 3; is the structure feature output by the i-th convolutional layer in SE; T i TE is the texture feature output by the i-th convolutional layer in TE; and are the i-th layer convolutional layers in SE and TE, respectively; and are the enhanced structural features; and are the enhanced texture features; is the i-th structure-texture interaction unit in STIE. In particular, “||” denotes the concatenation operation in the channel dimension. In this way, and are the enhanced long-range structural features;

[0082] 2) Structure-texture long-range interaction (STDI):

[0083] The purpose of structure-texture long-range interaction is to collect long-range context features by increasing the receptive field of the network, and then synthesize context features with semantic information. Structure-texture long-range interaction STDI includes a structure bottleneck layer (SB), a texture bottleneck layer (TB), and four SCIMs. The structure bottleneck layer SB and the texture bottleneck layer TB are both composed of four residual dilated convolutional blocks (RDCB). Through the residual dilated convolutional block RDCB, different dilated convolutional layers can be used to search for context information, and the context information of structure and texture can be captured, respectively. The spatial and channel domain interaction module SCIM can focus on the captured structure features and texture features in the spatial domain and the channel domain, respectively, to realize the long-range interaction of structure-texture. The mode of structure-texture long-range interaction STDI is represented in the following form:

[0084]

[0085] In formula 2, j = 1, 2, 3, 4; is the long-range structure feature output by the j-th RDCB in SB; is the long-range texture feature output by the j-th RDCB in TB; and are the j-th residual dilated convolutional block in SB and TB, respectively; and are the enhanced long-range structure features; and are the enhanced long-range texture features; SCIM j (·, ·) is the j-th spatial and channel domain interaction module. In particular, In this way, and are then input to the structure-texture interaction decoder.

[0086] 3) Structure-texture interaction decoder (STID):

[0087] The role of the decoder in the image inpainting process is to restore the damaged features and map the features back to the image level. The structure-texture interaction decoder STID includes a structure decoder (SD), a texture decoder (TD), and three STIUs. Both SD and TD contain two layers of convolutional layers and two layers of deconvolutional layers, which are used to restore the structure and texture, respectively. The STIUs are used to interact the structure and texture. Specifically, the first layer of convolutional layers in the structure decoder SD and the texture decoder TD are used to integrate the structure and texture, respectively, to obtain the first layer of deconvolutional layers in the SD and the TD, respectively. and and T1 TD , and the expression of the subsequent process is as follows:

[0088]

[0089] In formula 3, k = 1, 2, 3; and are enhanced structure features; and are enhanced texture features; is the kth structure-texture interaction unit in the STID; is the structure feature output by the kth convolutional layer in the SD; is the structure feature output by the k+1th convolutional layer in the SD; is the texture feature output by the kth convolutional layer in the TD; is the texture feature output by the k+1th convolutional layer in the TD; and are the k+1th convolutional layers in the SD and the TD. In the above manner, the restored structure features and the texture features

[0090] Meanwhile, a structure feature pyramid and a texture feature pyramid are constructed in the structure-texture interaction decoder STID. In Figure 2a and Figure 2b , the gradual restoration of the structure and the texture in the decoding process is shown.

[0091] 4) Structure-texture fusion (STF):

[0092] In the structure-texture fusion STF, the restored structure features​ and texture features After that, a structure-texture interaction unit is embedded to fuse the two, denoted as the following form:

[0093]

[0094] In formula 4, F1 and F2 are two fused features; STIU STF (·, ·) is a structure-texture interaction unit; is the recovered structure feature; is the recovered texture feature.

[0095] Finally, the two fused features are concatenated along the channel dimension and input to the output layer to generate the final predicted image I out .

[0096] The above is the operation process of the dual-branch parallel coupling generator. Through the process of structure-texture interaction, the mutual guidance and mutual promotion of structure and texture are realized. Figures 1a-1d The effect of the generator generating images is shown.

[0097] Step b, design of structure-texture bidirectional interaction module:

[0098] As can be seen from step a, two structure-texture bidirectional interaction mechanisms are embedded between the structure branch and the texture branch, which are structure-texture interaction unit (STIU) and spatial and channel domain-based interaction module (SCIM), respectively. In Figure 5 and Figure 6 The diagrams of the two interaction mechanisms are shown. The structure-texture interaction unit STIU and the spatial and channel domain-based interaction module SCIM both follow the following form:

[0099]

[0100] In formula 5, and are the structure and texture features after interaction enhancement; S and T are the structure and texture features, respectively; f() and g() are learnable weights that control the trade-off between structure (texture) information and texture (structure) information. represents element-wise addition; ⊙ represents element-wise multiplication.

[0101] As shown in Figure 5 , in the structure-texture interaction unit STIU, the expressions of f() and g() are as follows:

[0102]

[0103] In formula 6, W f and Wg σ represents the gating weights; σ is the Sigmoid function; Conv 3×3 () represents a convolutional layer with a kernel size of 3; S and T represent structural features and texture features, respectively.

[0104] like Figure 6 As shown, in the SCIM interaction module based on spatial and channel domains, f() and g() are both implemented through spatial and channel concerns, expressed in the following form:

[0105]

[0106] In Formula 7, P f and P g Gating weights for spatial focus; Q f and Q g The channel's gate weights are the factors of interest; σ is the Sigmoid function; Conv 5×5 () is a convolutional layer with a kernel size of 5; Avgpool() and Maxpool() refer to average pooling and max pooling operations, respectively; MLP() is a multilayer perceptron; S and T are structural features and texture features, respectively.

[0107] Step c: Construct a two-branch discriminator:

[0108] The dual-branch discriminator is designed to generate image I out To distinguish the image from the real image I, parallel structure and texture discriminators are used, which determine whether the image's structure and texture are real or fake. Both the structure and texture discriminators contain 5 convolutional layers and are followed by a sigmoid function. Figure 4 The diagram shows the discriminator.

[0109] Texture discriminator to generate image I out Alternatively, taking a real image I as input, it can output a texture score t for both. fake and t true For the structure discriminator, the first step is to extract and generate image I using an edge detector. out The generation edge E out This serves as the structural representation of the generated image. The structure discriminator generates edge E. out Alternatively, take the true edge E as input and output the structure score s of both. fake and s true The edge detector is implemented using a residual network module. The above process can be expressed as follows:

[0110] t fake =D s (E out ),t true =Ds (E),s fake =D t (I out ),s true =D t (I). Formula 8

[0111] In formula 8, t fake is a score of a texture discriminator on a generated image texture; t true is a score of the texture discriminator on a real image texture; s fake is a score of a structure discriminator on a generated image structure; s true is a score of the structure discriminator on a real image structure; D s and D t are the structure discriminator and the texture discriminator, respectively; I out is the generated image; I is the real image; E out is a generated edge; and E is a real edge.

[0112] Subsequently, the generated image texture and structure scores t fake and s fake are spliced in the channel dimension to obtain a total score x1, and the real image texture and structure scores t true and s true are spliced in the channel dimension to obtain a total score x2, and the two scores will be used for subsequent calculation of the adversarial loss.

[0113] Step d, training of the network:

[0114] The two structure-texture interaction mechanisms designed in step b are embedded into the double-branch parallel coupled generator in step a to realize bidirectional interaction between the structure features and the texture features, so that the structure and the texture can promote and contribute to each other, so that the double-branch parallel coupled generator can generate an image with reasonable structure and texture. The double-branch discriminator in step c scores and discriminates the true and false of the image generated by the double-branch parallel coupled generator, supervises the generation quality of the generator, and further stimulates the generation potential of the generator.

[0115] The present application is a deep neural network model, which is trained end-to-end with a large amount of training data, so that the model is continuously optimized until the error between the generated image and the real image reaches an acceptable range, and the optimal network model is saved. Specifically, the weighted sum of L1 reconstruction loss, perceptual loss, style loss, adversarial loss and pyramid loss is used as the total loss function, and the network parameters are optimized by the Adam optimization algorithm, so that a more reasonable structure and texture are repaired for the damaged image.

[0116] wherein the adversarial loss involves a generative adversarial process of the generator and the discriminator. For the generator, it hopes to generate an image Iout The gap between the structure and texture of the generated image I and the structure and texture of the real image I is as small as possible, while the mutual promotion of the structure and texture of the image improves the quality of the image. For the discriminator, it hopes to distinguish the generated image I from the real image I from the perspective of structure and texture. The above process can be represented as the following min-max problem: out

[0117]

[0118] In formula 9, log() is a logarithmic function; x1 and x2 are the scores of the discriminator for the generated image and the real image, respectively; I out is the generated image; I is the real image; E out is the generated edge; E is the real edge. The same as the idea of the generative adversarial network, through the above formula, the effect of the generator and the discriminator against generation can be realized.

[0119] The specific operation steps are as follows: first, the adversarial loss is calculated, and the network parameters of the dual-branch discriminator are updated; then, the network parameters of the dual-branch discriminator are frozen, the L1 reconstruction loss, the perception loss, the style loss, the adversarial loss and the pyramid loss are calculated, and the network parameters of the dual-branch parallel coupled generator are updated. According to this training method, the discrimination ability of the discriminator is continuously improved, and the generation ability of the generator is also continuously improved, so that more reasonable images are generated.

[0120] Embodiment

[0121] As shown in Figure 7 , a structure and texture mutual promotion image restoration method comprises the following steps:

[0122] 1. Data preparation.

[0123] The damaged-undamaged image is constructed by using the binary mask, and the damaged-undamaged edge is obtained by the edge extraction algorithm. These data will directly participate in the subsequent training process as the training set.

[0124] 2. Build a dual-branch parallel coupled generator.

[0125] First, two parallel encoding-decoding networks are constructed, wherein one branch restores the damaged edge, i.e. the structure reconstruction process; the other branch restores the damaged image, i.e. the texture synthesis process. The network settings of the two branches are the same, both composed of an encoder, a bottleneck layer and a decoder. The encoder is composed of 3 convolutional layers. The bottleneck layer is composed of 4 residual dilated convolution modules (RDCB). The decoder is composed of 2 deconvolutional layers and 2 convolutional layers. In addition, the output layer is composed of one convolutional layer.

[0126] ​The specific network settings of the generator are shown in Table 1. Among them, Conv, Diconv and Deconv are convolutional layer, dilated convolutional layer and deconvolutional layer respectively. BN is batch normalization, and ReLU and LReLU are both activation functions, which are linear rectifier function and linear rectifier function with leakage respectively. k represents the size of the convolution kernel; s and u are the down-sampling and up-sampling factors respectively; p is the zero padding number; and r is the hole rate.

[0127] Table 1 Specific network settings of the generator

[0128]

[0129] Secondly, the structure-texture interaction unit (STIU) is used in the structure-texture interaction encoder, the structure-texture interaction decoder and the structure-texture fusion, and the spatial domain and channel domain interaction module (SCIM) is used in the structure-texture long-distance interaction. The embedding modes of the two modules are shown in Figure 3 The two modules can complete the interaction between the structure reconstruction and the texture synthesis, realize the mutual promotion of structure information and texture information, and achieve the required structure information and texture information. The schematic diagrams of the two modules are shown in Figure 5 and Figure 6 The specific algorithm and network settings are shown in step b above.

[0130] 3) Construction of a double-branch discriminator.

[0131] The double-branch discriminator accepts the image generated by the generator and the real image as input, and aims to distinguish the two. It distinguishes true and false by judging the structure and texture of the image. For the discrimination of structure, the image is first input into a residual module for edge extraction. The residual module can be trained to have the ability of edge extraction, which can be regarded as an edge extractor. Then, the extracted edge is input into the structure discriminator for judgment and scoring. For the discrimination of texture, the image is directly input into the texture discriminator to evaluate the credibility of the image texture. Finally, the evaluation results of structure and texture are integrated to serve as the total basis for true and false judgment. The network structure of the structure discriminator and the texture discriminator is the same, and the specific network settings of the discriminator are shown in Table 2.

[0132] Table 2 Specific network settings of the discriminator

[0133]

[0134] 4) Construction of the generative adversarial process.

[0135] The generated image can be obtained by the structure-texture mutual promotion mode of the dual-branch coupled generator. Then, the generated image and the real image are input into the dual-branch discriminator to be judged from the aspects of structure and texture. Specifically, we design corresponding labels. If the image is true, the score is 1, and if the image is false, the score is 0. For the generator, the image generated by the generator is expected to be closer to 1 after being scored by the discriminator. For the discriminator, the score of the real image is expected to be close to 1, and the score of the generated image is expected to be close to 0. In this process, the generator and the discriminator are mutually opposed, and the network parameters are updated constantly. The generation ability of the generator is constantly improved under the supervision of the discriminator, and the discrimination ability of the discriminator is also constantly improved with the upgrading of the generator.

[0136] 5) Model training. The above model is optimized and trained by the L1 reconstruction loss, the perception loss, the style loss, the adversarial loss and the pyramid loss as constraints, and the generator and the discriminator are optimized by the Adam optimization algorithm. The learning rate of the generator and the discriminator is 1x10 -4 and 1x10 -5 , respectively, and the batch size is 4. The model is trained according to the above training process until the loss of the model no longer decreases, and then the optimal model is saved for subsequent testing.

[0137] 6) Model testing. The trained network can accept any damaged image (such as Figure 1a ) as input, and can output a generated image with reasonable structure and clear texture (such as Figure 1b ).

Claims

1. A structure and texture mutually promoting image inpainting method, characterized by: The method comprises the following steps: Step a, building a network framework of a double-branch parallel coupling generator: The double-branch parallel coupled generator takes edges as structure representation and images as texture representation to model the structure and texture of images; the framework contains parallel structure branch and texture branch; the structure branch takes damaged edges E in , damaged gray image Y in and binary mask M as input, aiming to reconstruct the damaged structure, wherein 0 represents undamaged area and 1 represents damaged area in the binary mask; the texture branch takes damaged image I in and binary mask M as input, aiming to synthesize texture details for damaged areas; meanwhile, two feature interaction mechanisms, namely structure-texture interaction unit STIU and spatial domain and channel domain based interaction module SCIM, are embedded between the two branches to realize bidirectional interaction of structure features and texture features, so that they can obtain complementary information from each other to enhance their own features; The double-branch parallel coupling generator comprises four parts: 1, an encoder of structure-texture interaction STIE; 2, structure-texture distant interaction STDI; 3, a decoder of structure-texture interaction STID; and 4, structure-texture fusion STF; The encoder of structure-texture interaction STIE comprises a structure encoder SE, a texture encoder TE and three structure-texture interaction units STIU; the structure encoder SE and the texture encoder TE both comprise three convolution layers, the structure encoder SE extracts structure features, the texture encoder TE extracts texture features, and the structure features and the texture features are bidirectionally interacted through the structure-texture interaction units STIU to extract complementary information; the steps of the encoder of structure-texture interaction STIE are represented by the following formula: In formula 1, i = 1, 2, 3; is the structural feature of the output of the i-th convolutional layer in SE; T i TE is the texture feature of the output of the i-th convolutional layer in TE; and are the i-th convolutional layer in SE and TE, respectively; and are the enhanced structural features; and are the enhanced texture features; is the i-th structure-texture interaction unit in STIE; "||" means the concatenation operation in the channel dimension; according to the above manner, and and are output to the structure-texture long-distance interaction described below; The structure-texture distant interaction STDI comprises a structure bottleneck layer SB, a texture bottleneck layer TB and four SCIMs; the structure bottleneck layer SB and the texture bottleneck layer TB both comprise four residual dilated convolution modules RDCB, the residual dilated convolution modules RDCB can search context information by using different dilated convolution layers to capture the context information of the structure and the texture respectively; the interaction modules SCIM based on the spatial domain and the channel domain pay attention to the captured structure features and texture features on the spatial domain and the channel domain respectively, and realize the distant interaction of the structure and the texture; the mode of the structure-texture distant interaction STDI is represented in the following form: In Equation 2, j = 1, 2, 3, 4; is the long-range structure feature of the jth RDCB output in SB; is the long-range texture feature of the jth RDCB output in TB; and are the jth residual hole convolution module in SB and TB, respectively; and are both enhanced long-range structure features; and are both enhanced long-range texture features; SCIM j (,) is the jth spatial and channel interaction module; In this way, the following are calculated and Both are then input to the structure-texture interaction decoder described below; The decoder of structure-texture interaction STID comprises a structure decoder SD, a texture decoder TD and three STIUs; the structure decoder SD and the texture decoder TD both comprise two convolution layers and two deconvolution layers, which are used to recover the structure and the texture respectively; the structure-texture interaction unit STIU is used to interact the structure and the texture; specifically, the first convolution layers in the structure decoder SD and the texture decoder TD are used to integrate the structure and the texture respectively, and the second convolution layers in the structure decoder SD and the texture decoder TD are used to recover the structure and the texture respectively. and to obtain and T1 TD respectively; the expression of the subsequent process is as follows: In formula 3, k = 1, 2, 3; and are enhanced structural features; and are enhanced texture features; is the kth structure-texture interaction unit in the STID; is the kth convolutional layer output structural feature in the SD; is the k+1th convolutional layer output structural feature in the SD; is the kth convolutional layer output texture feature in the TD; is the k+1th convolutional layer output texture feature in the TD; and is the k+1th convolutional layer in the SD and TD; the recovered structural features and texture features Meanwhile, structure feature pyramids and texture feature pyramids are constructed in the decoder of structure-texture interaction STID; After deriving the recovered structural features and textural features , a structure-texture interaction unit is embedded to fuse both of them, denoted as follows: In Equation 4, F1 and F2 are two fused features; STIU STF (·, ) is a structure-texture interaction unit; is the recovered structure feature; is the recovered texture feature; Finally, the two fused features are concatenated along the channel dimension and input to the output layer to generate the final predicted image I out ; Step b, design of a structure-texture bidirectional interaction module: As seen from step a, two structure-texture bidirectional interaction mechanisms are embedded between the structure branch and the texture branch, which are the structure-texture interaction unit STIU and the interaction module SCIM based on the spatial domain and the channel domain; the structure-texture interaction unit STIU and the interaction module SCIM based on the spatial domain and the channel domain both follow the following form: In formula 5, and are the structure feature and the texture feature after interaction enhancement respectively; S and T are the structure feature and the texture feature respectively; f() and g() are learnable weights, which control the trade-off between the structure information and the texture information, or the trade-off between the texture information and the structure information; represents element-wise addition; ⊙ represents element-wise multiplication; In the structure-texture interaction unit STIU, the expressions of f() and g() are as follows: In Equation 6, W f and W g are the gating weights; σ is the Sigmoid function; Conv 3×3 () is a convolutional layer with a kernel size of 3; S and T are the structure and texture features, respectively. In the interaction module SCIM based on the spatial domain and the channel domain, f() and g() are realized by spatial attention and channel attention, and are represented in the following form: In Equation 7, P f and P g are the gating weights for spatial attention; Q f and Q g are the gating weights for channel attention; σ is the Sigmoid function; Conv 5×5 () is a convolutional layer with kernel size 5; Avgpool() and Maxpool() refer to the average pooling operation and the max pooling operation, respectively; MLP() is a multi-layer perceptron; S and T are the structure feature and the texture feature, respectively. Step c, construction of a double-branch discriminator: The dual-branch discriminator aims to distinguish the generated image I out and the real image I, including a parallel structure discriminator and a texture discriminator, which respectively distinguish the structure and texture of the image; wherein the structure discriminator and the texture discriminator both contain 5 convolution layers and are connected with a Sigmoid function at the end. The texture discriminator takes the generated image I out or the real image I as input, and can output the texture scores t fake and t true of the two; for the structure discriminator, the generated edges E out of the generated image I out are first extracted by an edge detector, which is used as the structure representation of the generated image; the structure discriminator takes the generated edges E out or the real edges E as input, and outputs the structure scores s fake and s true of the two; wherein the edge detector is implemented by a residual network module; the above process is expressed as: t fake = D s (E out ), t true = D s (E), s fake = D t (I out ), s true = D t (I). Equation 8 In Equation 8, t fake is the score of the texture discriminator pair on the generated image texture; t true is the score of the texture discriminator pair on the real image texture; s fake is the score of the structure discriminator pair on the generated image structure; s true is the score of the structure discriminator pair on the real image structure; D s and D t are the structure discriminator and the texture discriminator, respectively; I out is the generated image; I is the real image; E out is the generated edge; E is the real edge; Subsequently, generate image texture and structure scores t fake and s fake Concatenate along the channel dimension to obtain total score x1, real image texture and structure score t true and s true Concatenate along the channel dimension to obtain total score x2, the two scores will be used for subsequent adversarial loss calculation; Step d, training of the network: The two structure-texture interaction mechanisms designed in step b are embedded into the double-branch parallel coupling generator in step a, realizing the bidirectional interaction between the structure features and the texture features, so that the structure and the texture can promote and contribute to each other, so that the double-branch parallel coupling generator can generate images with reasonable structure and texture; the double-branch discriminator in step c scores and discriminates the true and false of the images generated by the double-branch parallel coupling generator, supervises the generation quality of the generator, and further stimulates the generation potential of the generator; Firstly, the adversarial loss is calculated, and the network parameters of the dual-branch discriminator are updated; then, the network parameters of the dual-branch discriminator are frozen, the L1 reconstruction loss, the perception loss, the style loss, the adversarial loss and the pyramid loss are calculated, and the network parameters of the dual-branch parallel coupled generator are updated; according to this training mode, the discrimination ability of the discriminator is continuously improved, and the generation ability of the generator is also continuously improved, so as to generate more reasonable images.

2. The structure and texture co-faceted image inpainting method of claim 1, wherein: In step d, the adversarial loss involves the generative adversarial process of the generator and the discriminator; the process is represented as: In formula 9, log(·) is a logarithmic function; x1 and x2 are scores of the discriminator on the generated image and the real image respectively; I out is the generated image; I is the real image; E out is the generated edge; E is the real edge; and the same as the idea of the generative adversarial network, through the above formula, the effect of the generator and the discriminator in the adversarial generation is realized.

Images