An incremental image restoration method based on wireframe and edge structure

Abstract

The present invention relates to an incremental image restoration method based on wireframe and edge structure, comprising the following steps: acquiring a scene data image; constructing a mask layer adapted to downstream tasks for model training; constructing and training a structure recovery model; constructing and training a wireframe structure upsampling network; if the resolution of the mask image is greater than 256×256, upsampling the 256×256 restored wireframe and edge structure to the same resolution as the mask image using the structure upsampling network; inputting the restored wireframe and edge information into a structural feature encoder to obtain structural features; obtaining mask position codes based on the mask layer; constructing and training an image restoration network; and performing image restoration after model training. Compared with existing technologies, this invention offers advantages such as improved image restoration results and greater adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an incremental image restoration method based on wireframe and edge structure. Background Art

[0002] Image inpainting aims to fill in missing areas in damaged images, ensuring the authenticity and rationality of texture details in the restored image areas. It has been studied for many years and is extremely useful for many real-world applications such as photography, object removal, and image editing.

[0003] As a hot topic in computer vision research in recent years, deep learning-based image inpainting algorithms have seen significant progress. Many of these efforts focus on improving texture details in the inpainted areas through model refinement. For example, the paper "Resolution-robust large mask inpainting with fourier convolutions" proposes an image inpainting algorithm based on fast Fourier convolution, effectively improving irregular masking and high-resolution image inpainting. The paper "Learning pyramid-context encoder network for high-quality image inpainting" proposes a multi-scale attention model, further enhancing texture quality in image inpainting. However, these models fail to account for missing structural information. To address this missing structural information, the paper "Edgeconnect: Generative image inpainting with adversarial edge learning" proposes a two-stage inpainting system that first restores edge information in the missing area and then performs edge-based inpainting and coloring. The paper "Learning a sketchtensor space for image inpainting of man-made scenes" utilizes both edge and wireframe information to assist in image inpainting and coloring. However, the edge information in the aforementioned paper "Edgeconnect: Generative Image Inpainting with Adversarial Edge Learning" is often based on gradient operations, which has some drawbacks. For example, straight edges are often discontinuous, and the reliance on gradients can lose structural information in color-insensitive areas. The paper "Learning a Sketch Tensor Space for Image Inpainting of Man-Made Scenes" uses adversarial training to restore structure, but the restoration effect is poor. Furthermore, the entire network is trained from scratch to integrate structural information, which is very expensive and cannot handle high-resolution image restoration. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an incremental image restoration method based on wireframe and edge structure with good image restoration effect and strong adaptability.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An incremental image restoration method based on wireframe and edge structure, the image restoration method comprising:

[0007] Step 1: Get the scene data image;

[0008] Step 2: Construct a mask layer suitable for downstream tasks to train the model;

[0009] Step 3: Build a structure recovery model and train it;

[0010] Step 4: Build a wireframe upsampling network and train it;

[0011] Step 5: If the resolution of the mask image is greater than 256×256, use the structure upsampling network to upsample the 256×256 repaired wireframe and edge structure to the same resolution as the mask image;

[0012] Step 6: Input the repaired wireframe and edge information into the structural feature encoder to obtain structural features;

[0013] Step 7: Get the mask position code according to the mask layer;

[0014] Step 8: Build an image restoration network and train it;

[0015] Step 9: After model training is completed, perform inference on tasks such as image restoration.

[0016] Preferably, the step 2 is specifically as follows:

[0017] First, we construct a mask layer by introducing the semantic segmentation label layer of the COCO dataset;

[0018] Secondly, simulate the human graffiti habit to iteratively apply the brush at random angles to build graffiti layers;

[0019] Finally, a random combination of semantic segmentation mask layer and scribble layer is used.

[0020] Preferably, the structural recovery model is specifically:

[0021] The structure restoration model TSR consists of three convolutional downsampling layers, eight Transformer blocks, and three convolutional upsampling layers. The Transformer block consists of an axial attention module combined with a normal attention module.

[0022] The axial attention module is specifically:

[0023]

[0024]

[0025]

[0026]

[0027] E=FFN(LN(I+R+C))+I+R+C

[0028] Among them, X is the input feature Output after layer normalization; x ri ,x rj ,x ci ,x cj are the c-dimensional eigenvectors of the i-th row, j-th row, i-th column, and j-th column of X; W rq ,W rk ,W cq ,W ck All are trainable parameters; It is a trainable relative position encoding; The elements of the row and column axial attention score matrices are the i-th row and j-th column respectively; the attention score is softmax normalized and multiplied by the corresponding eigenvector to obtain the output of the attention module. The final output of the axial attention module is E, where LN and FFN are layer normalization and feedforward neural network layers respectively;

[0029] Inputting the output of the axial attention into the ordinary attention module can obtain the output F of a Transformer block. The calculation process of the ordinary attention module is:

[0030]

[0031]

[0032] F=FFN(LN(SA))+SA

[0033] Finally, the binary cross entropy loss function is used:

[0034]

[0035]

[0036] in, Extract the unmasked image for the canny edge extractor to obtain the edge layer; Extract the wireframe layer from the unmasked image for the wireframe extraction model;

[0037] The recovered structural information is:

[0038]

[0039] Among them, Il To mask the wireframe layer; I e is the mask edge layer; M is the mask map; I m To mask the image.

[0040] Preferably, the training method of the structure recovery model is:

[0041] The wireframe layer is obtained by pre-training the wireframe extraction model, the edge layer is extracted using the canny edge extractor, and the mask wireframe layer I is used to extract the edge layer. l , Mask edge layer I e , mask layer M and mask image I m As the input of the structure recovery model TSR, the structure recovery model is trained.

[0042] Preferably, the step 4 is specifically as follows:

[0043] The structure upsampling network is trained using real wireframe information with a resolution of 256×256 and a resolution of 512×512. The forward process of the network is:

[0044]

[0045]

[0046] in, is the real wireframe information with a resolution of 256×256; Resize is bilinear interpolation; CNN consists of four convolutional layers. The network outputs wireframe information with a resolution of 512×512;

[0047] The loss function is the binary cross entropy loss function:

[0048]

[0049] in, Represents the real wireframe information with a resolution of 512×512.

[0050] Preferably, the step 6 is specifically as follows:

[0051] The repaired wireframe, edge information and mask layer are input into the structural feature encoder SFE to obtain the feature map of the structural feature:

[0052]

[0053] Among them, S k ,k∈{0,1,2,3} are four feature maps with resolutions from high to low;

[0054] The forward process of the structural feature encoder SFE is:

[0055] X i+1 =ReLU(BN(GC(X i ))),i=0,1,2,3

[0056] X i+1 =DilatedResnetBlock(X i ),i=4,5,6

[0057] X i+1 =ReLU(BN(GC(X i ))),i=7,8,9

[0058] S0,S1,S2,S3=X7,X8,X9,X 10

[0059] in, ReLU represents the ReLU activation function; BN represents batch normalization; DilatedResnetBlock represents the residual block using dilated convolution; GC represents gated convolution, and the forward process of gated convolution is:

[0060] F=Conv f (X),G=Conv g (X)

[0061] GateConv(X)=σ(G)·F

[0062] Among them, X is the input feature; σ is the sigmoid activation function.

[0063] Preferably, the step 7 is specifically as follows:

[0064] First, a 3×3 convolution kernel with all elements set to 1 is used to obtain the distance between each pixel in the masked area and the non-masked area. Then, the distance is mapped to a d-dimensional sinusoidal position code P. dis :

[0065]

[0066]

[0067] Among them, D max =128,d=64;

[0068] Four 3×3 convolution kernels representing different directions are used to obtain the direction of each pixel in the masked area from the nearest non-masked area. This tensor is projected into the positional encoding:

[0069]

[0070] Among them, W dir is a learnable embedded parameter;

[0071] P dis and P dir Add up to get the final mask position code P

[0072] P=P dis +P dir .

[0073] Preferably, the step 8 is specifically as follows:

[0074] The mask position encoding and structural features are injected into the image restoration network for image restoration training. The forward process of injecting structural features and mask position encoding is as follows:

[0075] X1=ReLU(BN(Conv2d(X0)))

[0076] X2 = ReLU(BN(Conv2d(X1+α×P+α0×S0)))

[0077] X i+1 =ReLU(BN(Conv2d(X i +α i-1 ×S i-1 ))),i=2,3

[0078] X5=FFCBlock(X4+α3×S3)

[0079] X i+1 =ReLU(BN(Conv2d(X i ))),i=7,8,9

[0080]

[0081] Among them, α, α0, α1, α2, and α3 are all trainable parameters initialized to 0; is the image restored by the network; Conv2d is the two-dimensional convolution operator; FFCBlock is the Fourier residual block;

[0082] The forward process of the fast Fourier convolution block FFC:

[0083] X l ,X g =Split(X)

[0084] X' g =ReLU(BN(Conv2d(X g )))

[0085] X' g '=X' g+InvFFT2d(ReLU(BN(Conv2d(FFT2d(X' g )))))

[0086] X″′ g =Conv2d(X″ g )+Conv2d(X l )

[0087] X o =ReLU(BN(Concat(Conv2d(X),X″′ g )))

[0088] Where X is the input feature; Split represents the operation of splitting the tensor along the channel dimension; FFT2d represents the two-dimensional fast Fourier transform; InvFFT2d represents the two-dimensional inverse fast Fourier transform; Concat represents the operation of concatenating tensors along the channel dimension; X o is the final output feature;

[0089] The forwarding process of FFCBlock is as follows:

[0090] X′=X+FFC(FFC(X)).

[0091] More preferably, the loss function of the image restoration network includes:

[0092] L1 loss function:

[0093]

[0094] Among them, ⊙ represents element-by-element multiplication;

[0095] Adversarial loss function:

[0096]

[0097]

[0098]

[0099]

[0100] Where D represents the discriminator network; is the gradient penalty loss;

[0101] Feature matching loss function:

[0102]

[0103] Among them, D (i) is the i-th layer of the discriminator network D; N iis the total number of neurons in layer i;

[0104] Perceptual loss function:

[0105]

[0106] Finally, the loss function of the structural repair model is:

[0107]

[0108] Among them, λ L1 ,λ adv ,λ fm and λ hrf are the weights of L1 loss function, adversarial loss function, feature matching loss function and perceptual loss function respectively.

[0109] Preferably, the step 9 is specifically as follows:

[0110] Input the damaged image to be repaired I m and M, using the wireframe extraction model to obtain its wireframe I l , use the canny edge extractor to extract its edge I e ; will I m ,M,I l and I e The image is scaled to a resolution of 256×256 and input into the structure restoration model to obtain the restored structure information; then the edge structure and wireframe structure are upsampled to the initial resolution using the structure upsampling network, and then the structure feature encoder is used to obtain the structure features and the mask of the input mask, which are then input into the image restoration network G to obtain the final restored image.

[0111] Compared with the prior art, the present invention has the following beneficial effects:

[0112] 1. Good image restoration effect: The incremental image restoration method in the present invention proposes a structural information model based on the improved Transformer block, which greatly improves the ability of structural recovery. On this basis, the present invention also proposes a structural feature extraction network and an incremental training method for injecting structural information features, which successfully injects structural information into the subsequent image system and improves the accuracy of the restored image.

[0113] 2. Strong adaptability: The incremental image restoration method in this invention proposes mask position encoding of the masked area to solve the image restoration of large masked areas and a structural upsampling network to adapt to the restoration of high-resolution images; it achieves very good image restoration effects in various scenes and resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1Schematic diagram of the process of the incremental image restoration method of the present invention;

[0115] Figure 2 Schematic diagram of the specific structure of the mask position coding in an embodiment of the present invention. DETAILED DESCRIPTION

[0116] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0117] An incremental image restoration method based on wireframe and edge structure, the process is as follows Figure 1 Shown, including:

[0118] Step 1: Get the scene data image;

[0119] Step 2: Construct a mask layer suitable for downstream tasks to train the model;

[0120] Step 3: Build a structure recovery model and train it;

[0121] Step 4: Build a wireframe upsampling network and train it;

[0122] Step 5: If the resolution of the mask image is greater than 256×256, use the structure upsampling network to upsample the 256×256 repaired wireframe and edge structure to the same resolution as the mask image;

[0123] Step 6: Input the repaired wireframe and edge information into the structural feature encoder to obtain structural features;

[0124] Step 7: Get the mask position code according to the mask layer;

[0125] Step 8: Build an image restoration network and train it;

[0126] Step 9: After model training is completed, perform inference on tasks such as image restoration.

[0127] The following describes each step in detail:

[0128] Step 1: This example uses the Places2 training set to train the structure recovery model, upsampling network, structural feature encoder, and image inpainting network. The Places2 dataset contains 1.8 million training images and a 36,500 test images. An Indoor dataset was also collected. The Indoor training set consists of 5,000 images from the ShanghaiTech dataset and 15,055 images from the NYUDepthV2 dataset, for a total of 20,055 training images. The Indoor test set consists of 1,000 images, of which 462 and 538 images are from the ShanghaiTech and NYUDepthV2 datasets, respectively. The Indoor dataset is also used for training and testing the structure recovery model, upsampling network, structural feature encoder, and image inpainting network. Furthermore, testing was performed on the MatterPort3D dataset, which contains 1,965 images with a resolution of 1024×1024.

[0129] Step 2: This embodiment uses two methods to construct a mask layer that conforms to real human graffiti. First, by introducing the semantic segmentation label layer of the COCO dataset, 91,707 mask layers are constructed. These mask layers are all composed of semantic layers of real objects. In addition, a program is used to simulate human graffiti habits. Specifically, the brush is iteratively applied at random angles, and the number of random times is set from 2 to 5. In addition, the semantic segmentation mask layer and the graffiti layer are randomly combined to increase the difficulty of network learning. Semantic segmentation, graffiti, and mixed mask layers are used, accounting for 40%, 40%, and 20% respectively.

[0130] Step 3: Use the pre-trained wireframe extraction model to obtain the wireframe layer, use the canny edge extractor to extract the edge layer, and convert the mask wireframe layer I with a resolution of 256×256 into a mask layer. l , Mask edge layer I e , mask layer M and mask image I m As the input of the structure recovery model TSR, the recovered structural information can be obtained as follows:

[0131]

[0132] The structure recovery model TSR in this embodiment consists of three convolutional downsampling layers, eight Transformer blocks, and three convolutional upsampling layers. The Transformer block consists of an axial attention module combined with a normal attention module.

[0133] The axial attention module is specifically:

[0134]

[0135]

[0136]

[0137]

[0138] E=FFN(LN(I+R+C))+I+R+C

[0139] Among them, X is the input feature Output after layer normalization; x ri ,x rj ,x ci ,x cj are the c-dimensional eigenvectors of the i-th row, j-th row, i-th column, and j-th column of X; W rq ,W rk ,W cq ,W ck All are trainable parameters; It is a trainable relative position encoding; The elements of the row and column axial attention score matrices are the i-th row and j-th column respectively; the attention score is softmax normalized and multiplied by the corresponding eigenvector to obtain the output of the attention module. The final output of the axial attention module is E, where LN and FFN are layer normalization and feedforward neural network layers respectively;

[0140] Inputting the output of the axial attention into the ordinary attention module can obtain the output F of a Transformer block. The calculation process of the ordinary attention module is:

[0141]

[0142]

[0143] F=FFN(LN(SA))+SA

[0144] Finally, the binary cross entropy loss function is used:

[0145]

[0146]

[0147] in, Extract the unmasked image for the canny edge extractor to obtain the edge layer; Extract the wireframe layer from the unmasked image for the wireframe extraction model;

[0148] Step 4: When upsampling the model, the structure upsampling network is trained using real wireframe information with a resolution of 256×256 and a resolution of 512×512. The forward process of the network is:

[0149]

[0150]

[0151] in, is the real wireframe information with a resolution of 256×256; Resize is bilinear interpolation; CNN consists of four convolutional layers. The network outputs wireframe information with a resolution of 512×512;

[0152] The loss function is the binary cross entropy loss function:

[0153]

[0154] in, Represents the real wireframe information with a resolution of 512×512.

[0155] During the testing phase, the upsampling network can simultaneously upsample the wireframe structure and edge structure until the desired resolution is reached. Specifically, the following process is repeated multiple times:

[0156] I' e =sigmoid[γ(I' e +β)]

[0157] I' l =sigmoid[γ(I' l +β)]

[0158] Where, set γ=β=2; I' e and I' l The initial resolution is h×w. After repeating the above process q times, the resolution is 2 q h×2 q The structural information of w I' e and I' l , and finally bilinear interpolation is used to scale the structural information to the target resolution.

[0159] Step 5: During the training and testing process, if the resolution of the given target mask image is greater than 256×256, the structure upsampling network obtained in step 4 is used to upsample the 256×256 restored structure obtained by the structure recovery module in step 3 to the same resolution as the mask image.

[0160] Step 6: Input the repaired wireframe, edge information and mask layer into the structural feature encoder SFE to obtain the feature map of the structural feature:

[0161]

[0162] Among them, S k ,k∈{0,1,2,3} are four feature maps with resolutions from high to low;

[0163] The forward process of the structural feature encoder SFE is:

[0164] X i+1 =ReLU(BN(GC(X i ))),i=0,1,2,3

[0165] X i+1 =DilatedResnetBlock(X i ),i=4,5,6

[0166] X i+1 =ReLU(BN(GC(X i ))),i=7,8,9

[0167] S0,S1,S2,S3=X7,X8,X9,X 10

[0168] in, ReLU represents the ReLU activation function; BN represents batch normalization; DilatedResnetBlock represents the residual block using dilated convolution; GC represents gated convolution, and the forward process of gated convolution is:

[0169] F=Conv f (X),G=Conv g (X)

[0170] GateConv(X)=σ(G)·F

[0171] Among them, X is the input feature; σ is the sigmoid activation function.

[0172] Step 7: Figure 2 As shown, first use a 3×3 convolution kernel with all elements set to 1 to obtain the distance between each pixel in the masked area and the non-masked area, and then obtain the distance mapped to a d-dimensional sinusoidal position code P dis :

[0173]

[0174]

[0175] Among them, D max =128,d=64;

[0176] Four 3×3 convolution kernels representing different directions are used to obtain the direction of each pixel in the masked area from the nearest non-masked area. This tensor is projected into the positional encoding:

[0177]

[0178] Among them, W dir is a learnable embedded parameter;

[0179] P dis and P dir Add up to get the final mask position code P

[0180] P=P dis +P dir .

[0181] Step 8: Inject the mask position code and structural features into the image restoration network for image restoration training. The forward process of injecting structural features and mask position code is as follows:

[0182] X1=ReLU(BN(Conv2d(X0)))

[0183] X2 = ReLU(BN(Conv2d(X1+α×P+α0×S0)))

[0184] X i+1 =ReLU(BN(Conv2d(X i +α i-1 ×S i-1 ))),i=2,3

[0185] X5=FFCBlock(X4+α3×S3)

[0186] X i+1 =ReLU(BN(Conv2d(X i ))),i=7,8,9

[0187]

[0188] Among them, α, α0, α1, α2, and α3 are all trainable parameters initialized to 0; is the image restored by the network; Conv2d is the two-dimensional convolution operator; FFCBlock is the Fourier residual block;

[0189] The forward process of the fast Fourier convolution block FFC:

[0190] X l ,Xg =Split(X)

[0191] X' g =ReLU(BN(Conv2d(X g )))

[0192] X″′ g =X' g +InvFFT2d(ReLU(BN(Conv2d(FFT2d(X' g )))))

[0193] X″′ g =Conv2d(X″ g )+Conv2d(X l )

[0194] X o =ReLU(BN(Concat(Conv2d(X),X″′ g )))

[0195] Where X is the input feature; Split represents the operation of splitting the tensor along the channel dimension; FFT2d represents the two-dimensional fast Fourier transform; InvFFT2d represents the two-dimensional inverse fast Fourier transform; Concat represents the operation of concatenating tensors along the channel dimension; X o is the final output feature;

[0196] The forwarding process of FFCBlock is as follows:

[0197] X′=X+FFC(FFC(X)).

[0198] The loss functions of the image restoration network include:

[0199] L1 loss function:

[0200]

[0201] Among them, ⊙ represents element-by-element multiplication;

[0202] Adversarial loss function:

[0203]

[0204]

[0205]

[0206]

[0207] Where D represents the discriminator network; is the gradient penalty loss;

[0208] Feature matching loss function:

[0209]

[0210] Among them, D (i) is the i-th layer of the discriminator network D; N i is the total number of neurons in layer i;

[0211] Perceptual loss function:

[0212]

[0213] Finally, the loss function of the structural repair model is:

[0214]

[0215] Among them, λ L1 ,λ adv ,λ fm and λ hrf They are the weights of L1 loss function, adversarial loss function, feature matching loss function and perceptual loss function respectively. In this embodiment, λ L1 =10,λ adv =10,λ fm =100,λ hrf =30.

[0216] Step 9: Input the damaged image to be repaired I m and M, using the wireframe extraction model to obtain its wireframe I l , use the canny edge extractor to extract its edge I e ; will I m ,M,I l and I e The image is scaled to a resolution of 256×256 and input into the structure restoration model to obtain the restored structure information; then the edge structure and wireframe structure are upsampled to the initial resolution using the structure upsampling network, and then the structure feature encoder is used to obtain the structure features and the mask of the input mask, which are then input into the image restoration network G to obtain the final restored image.

[0217] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An incremental image restoration method based on wireframe and edge structure, characterized in that: The image restoration method comprises: Step 1: Get the scene data image; Step 2: Construct a mask layer suitable for downstream tasks to train the model; Step 3: Build a structure recovery model and train it; Step 4: Build a wireframe upsampling network and train it; Step 5: If the resolution of the mask image is greater than 256×256, use the structure upsampling network to upsample the 256×256 repaired wireframe and edge structure to the same resolution as the mask image; Step 6: Input the repaired wireframe and edge information into the structural feature encoder to obtain structural features; Step 7: Get the mask position code according to the mask layer; Step 8: Build an image restoration network and train it; Step 9: After model training is completed, perform image restoration; The step 8 is specifically as follows: The mask position encoding and structural features are injected into the image restoration network for image restoration training. The forward process of injecting structural features and mask position encoding is as follows: X1=ReLU(BN(Conv2d(X0))) X2 = ReLU(BN(Conv2d(X1+α×P+α0×S0))) X i+1 =ReLU(BN(Conv2d(X i +α i-1 ×S i-1 ))), i=2,3 X5=FFCBlock(X4+α3×S3) X i+1 =ReLU(BN(Conv2d(X i ))),i=7,8,9 Among them, α, α0, α1, α2, and α3 are all trainable parameters initialized to 0; is the image restored by the network; Conv2d is the two-dimensional convolution operator; FFCBlock is the Fourier residual block; The forward process of the fast Fourier convolution block FFC: X l ,X g =Split(X) X″′ g =Conv2d(X″ g )+Conv2d(X l ) X o =ReLU(BN(Concat(Conv2d(X),X″′ g ))) Where X is the input feature; Split represents the operation of splitting the tensor along the channel dimension; FFT2d represents the two-dimensional fast Fourier transform; InvFFT2d represents the two-dimensional inverse fast Fourier transform; Concat represents the operation of concatenating tensors along the channel dimension; X o is the final output feature; The forwarding process of FFCBlock is as follows: X′=X+FFC(FFC(X)); The loss function of the image restoration network includes: L1 loss function: Among them, ⊙ represents element-by-element multiplication; Adversarial loss function: Where D represents the discriminator network; is the gradient penalty loss; Feature matching loss function: Among them, D (i) is the i-th layer of the discriminator network D; N i is the total number of neurons in layer i; Perceptual loss function: Finally, the loss function of the structural repair model is: Among them, λ L1 ,λ adv ,λ fm and λ hrf are the weights of L1 loss function, adversarial loss function, feature matching loss function and perceptual loss function respectively.

2. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The step 2 is specifically as follows: First, we construct a mask layer by introducing the semantic segmentation label layer of the COCO dataset; Secondly, simulate the human graffiti habit to iteratively apply the brush at random angles to build graffiti layers; Finally, a random combination of semantic segmentation mask layer and scribble layer is used.

3. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The structural recovery model is specifically: The structure restoration model TSR consists of three convolutional downsampling layers, eight Transformer blocks, and three convolutional upsampling layers. The Transformer block consists of an axial attention module combined with a normal attention module. The axial attention module is specifically: E=FFN(LN(I+R+C))+I+R+C Among them, X is the input feature Output after layer normalization; x ri ,x rj ,x ci ,x cj are the c-dimensional eigenvectors of the i-th row, j-th row, i-th column, and j-th column of X; W rq ,W rk ,W cq ,W ck All are trainable parameters; It is a trainable relative position encoding; The elements of the row and column axial attention score matrices are the i-th row and j-th column respectively; the attention score is softmax normalized and multiplied by the corresponding eigenvector to obtain the output of the attention module. The final output of the axial attention module is E, where LN and FFN are layer normalization and feedforward neural network layers respectively; Inputting the output of the axial attention into the ordinary attention module can obtain the output F of a Transformer block. The calculation process of the ordinary attention module is: F=FFN(LN(SA))+SA Finally, the binary cross entropy loss function is used: in, Extract the unmasked image for the canny edge extractor to obtain the edge layer; Extract the wireframe layer from the unmasked image for the wireframe extraction model; The recovered structural information is: Among them, I l To mask the wireframe layer; I e is the mask edge layer; M is the mask map; I m To mask the image.

4. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The training method of the structure recovery model is: The wireframe layer is obtained by pre-training the wireframe extraction model, the edge layer is extracted using the canny edge extractor, and the mask wireframe layer I is used to extract the edge layer. l , Mask edge layer I e , mask layer M and mask image I m As the input of the structure recovery model TSR, the structure recovery model is trained.

5. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The step 4 is specifically as follows: The structure upsampling network is trained using real wireframe information with a resolution of 256×256 and a resolution of 512×512. The forward process of the network is: in, is the real wireframe information with a resolution of 256×256; Resize is bilinear interpolation; CNN consists of four convolutional layers. The network outputs wireframe information with a resolution of 512×512; The loss function is the binary cross entropy loss function: in, Represents the real wireframe information with a resolution of 512×512.

6. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The step 6 is specifically as follows: The repaired wireframe, edge information and mask layer are input into the structural feature encoder SFE to obtain the feature map of the structural feature: Among them, S k ,k∈{0,1,2,3} are four feature maps with resolutions from high to low; The forward process of the structural feature encoder SFE is: X i+1 =ReLU(BN(GC(X i ))),i=0,1,2,3 X i+1 =DilatedResnetBlock(X i ),i=4,5,6 X i+1 =ReLU(BN(GC(X i ))),i=7,8,9 <h2 style=";text-align:left;direction:ltr">S0,S1,S2,S3=X7,X8,X9,X<h2 style=";text-align:left;direction:ltr"> 10 in, ReLU represents the ReLU activation function; BN represents batch normalization; DilatedResnetBlock represents the residual block using dilated convolution; GC represents gated convolution, and the forward process of gated convolution is: F=Conv f (X),G=Conv g (X) GateConv(X)=σ(G)·F Among them, X is the input feature; σ is the sigmoid activation function.

7. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The step 7 is specifically as follows: First, a 3×3 convolution kernel with all elements set to 1 is used to obtain the distance between each pixel in the masked area and the non-masked area. Then, the distance is mapped to a d-dimensional sinusoidal position code P. dis : Among them, D max =128,d=64; Four 3×3 convolution kernels representing different directions are used to obtain the direction of each pixel in the masked area from the nearest non-masked area. This tensor is projected into the positional encoding: Among them, W dir is a learnable embedded parameter; P dis and P dir Add up to get the final mask position code P P=P dis +P dir 。 8. The incremental image restoration method based on wireframe and edge structure according to claim 1, characterized in that: The step 9 is specifically as follows: Input the damaged image to be repaired I m and M, using the wireframe extraction model to obtain its wireframe I l , use the canny edge extractor to extract its edge I e ; will I m ,M,I l and I e The image is scaled to a resolution of 256×256 and input into the structure restoration model to obtain the restored structure information; then the edge structure and wireframe structure are upsampled to the initial resolution using the structure upsampling network, and then the structure feature encoder is used to obtain the structure features and the mask of the input mask, which are then input into the image restoration network G to obtain the final restored image.

Images