Self-priori guided image restoration method and system

Abstract

The invention belongs to the technical field of image processing, and relates to a self-priori-guided image restoration method and system, and the method comprises the following steps: S1, constructing a damaged image data set; step S2, a step of network construction; step S3, a step of network training and optimization; step S4, a network performance test step; according to the technical scheme of the invention, aiming at the problems of content repetition, color distortion and the like which are easy to occur when a traditional image restoration method is used for processing large-area defect of a continuous region or reference content is insufficient, a two-stage image restoration method consisting of a semantic prior estimation network and a semantic prior guide restoration network is adopted; and residual semantic information in the damaged image is fully mined and utilized, so that stronger robustness is realized while the visual naturalness and detail accuracy of a restoration result are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a self-prior-guided image inpainting method and system. BACKGROUND

[0002] Digital images are important information carriers and are widely used in news, medicine, monitoring and other fields. However, digital images may be damaged by human factors or have missing continuous regions due to damaged storage, or specified regions in the image need to be edited for subjective human purposes, such as removing target objects or adding new objects.

[0003] Traditional image inpainting methods can be divided into diffusion-based methods and sample-based methods, which perform well in the case of small-area proportion loss or in the case of sufficient reference content for missing content. However, they are prone to problems such as repeated content and color distortion in the case of large-area loss or insufficient reference content. Deep learning-based image inpainting algorithms have made significant progress. However, in the prior art, prior-guided image inpainting methods usually rely on pixel-level prior knowledge and lack object-level semantics. The method of introducing semantic segmentation still has the problem of mapping the pixel values of the segmentation results to specific class labels, which has low robustness. This is an insufficient aspect of the prior art.

[0004] Therefore, it is necessary to provide a self-prior-guided image inpainting method and system to solve the above-mentioned defects in the prior art. SUMMARY

[0005] The present application aims to provide a self-prior-guided image inpainting method and system to solve the above-mentioned technical problems.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solution: A self-prior-guided image inpainting method, comprising the following steps: Step S1, constructing a training data set, a verification data set and a test data set for training a semantic prior estimation network and a semantic prior guided inpainting network.

[0007] Step S2, constructing a semantic prior estimation network and a semantic prior guided inpainting network; the semantic prior estimation network comprises a feature extraction network and a mask classification segmentation network, and the semantic prior guided inpainting network comprises a cross-domain cross-attention module and an aggregated context transformation module.

[0008] Step S3, the semantic prior estimation network and the semantic prior guided inpainting network are trained by using the training data set, and an initial image inpainting network is obtained; in the training process, the initial image inpainting network is adjusted and optimized in parameters by using the verification data set, and an optimized final image inpainting network is obtained.

[0009] Step S4, the test data set is inpainted by using the optimized final image inpainting network, and the inpainted complete image is output as the inpainting result; the inpainting result is quantitatively evaluated and compared.

[0010] As preferred, the step S1 specifically comprises: The public image data set is selected, the public image data set comprises a training data part, a verification data part and a test data part, wherein the images in the training data part are generated into damaged images by using a random damaged area generation method, and a specific size of cutting operation is performed, and the images in the training data part after the above processing are used as the training data set; the images in the verification data part are generated into damaged images by using a random damaged area generation method, and a specific size of cutting operation is performed, and the images in the verification data part after the above processing are used as the verification data set; the images in the test data part are cut into a specific size, and the cut images are used as the test data set; The test data set is generated into damaged images by using a random damaged area generation method, and is grouped according to the area proportion of the damaged area of the image, and the number of images in each group is consistent with the number of images in the test data set; the grouped data is saved as a fixed damaged area data set; In the training process using the above data set, no undamaged image is involved.

[0011] As preferred, the semantic prior estimation network in the step S2, wherein the feature extraction network and the mask classification and segmentation network specifically comprise: The feature extraction network adopts an encoder-decoder structure, comprising a basic calculation module, a down-sampling and an up-sampling; the basic calculation module comprises a full connection feedforward network layer and a gate unit, wherein the gate unit is composed of a two-dimensional convolution and a Sigmoid activation function, and is used for filtering invalid features, and the full connection feedforward network layer is composed of layer normalization, two-dimensional data linear transformation and Gelu activation function; the down-sampling adopts two-dimensional convolution, so that the spatial size of the output feature map becomes half of the input; the up-sampling is pixel rearrangement up-sampling, so that the spatial size of the output feature map becomes twice of the input.

[0012] The mask classification segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework, which is composed of a linear classifier, a multi-layer perceptron and a mask generator. The linear classifier is responsible for generating the prediction probability of the class, the multi-layer perceptron is responsible for converting the input feature into a mask embedding vector, and the mask generator is responsible for calculating the dot product similarity between each mask embedding vector and the feature output by the feature extraction network, and generating the corresponding binary mask prediction. Finally, the combination of the class prediction probability and the binary mask prediction is output as the object-level semantic prior information.

[0013] Preferably, the cross-domain cross-attention module specifically comprises: The cross-domain cross-attention module inputs the object-level semantic prior information output by the semantic prior estimation network and the damaged image feature obtained by downsampling the image data of the training data set. The object-level semantic prior information is downsampled by one two-dimensional convolution and layer normalization to obtain semantic prior features, and the semantic prior features are convolved by Haar wavelets to obtain four groups of frequency domain semantic prior features. The damaged image feature is convolved by one two-dimensional convolution to obtain four groups of spatial domain damaged image features. The four groups of frequency domain semantic prior features and the four groups of spatial domain damaged image features are fused by four groups of cross-attention modules to output results, and then the fused data features are obtained by inverse Haar wavelet convolution. The fused data features are connected in residual connection with the damaged image feature through a learnable scaling parameter to obtain the damaged image feature guided and reinforced by the object-level semantic prior information.

[0014] Preferably, the aggregated context transformation module specifically comprises: The aggregated context transformation module adopts the AOT block structure in the AOT-GAN algorithm, and fuses multi-scale context information based on the multi-stage form of "segmentation-transformation-aggregation" to enhance the semantic reasoning ability of the model for large missing areas. The aggregated context transformation module divides the standard convolution kernel into multiple parallel sub-convolution kernels in the segmentation stage; in the transformation stage, each sub-convolution kernel performs a dilated convolution operation on the input feature with different dilation rates, thereby realizing the coverage of different receptive fields; in the aggregation stage, the features output by all sub-convolution kernels are spliced, and cross-channel fusion is performed through convolution operation to generate residual features of aggregated multi-scale context, and a gated residual connection is introduced to weight and fuse the features input into the aggregated context transformation module and the residual features, thereby improving the feature fusion effect for missing areas and known areas.

[0015] Preferably, the step S3 of training the semantic prior estimation network and the semantic prior guided repair network using the training data set to obtain the initial image repair network comprises: Step S31, the step of extracting the prior: The data of the training data set is input into a semantic prior estimation network to obtain object-level semantic prior information, wherein the semantic prior estimation network generates multi-scale features of an image through a feature extraction network, and the multi-scale features are input into a mask classification segmentation network to generate the object-level semantic prior information.

[0016] Step S32, a step of repairing from the prior: The data of the training data set is down-sampled through three two-dimensional convolutions to obtain damaged image features, and the damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into a semantic prior guided repair network; wherein the semantic prior guided repair network processes the damaged image features and the object-level semantic prior information through a cross-domain cross-attention module, and then processes the data through an aggregated context transformation module; the cross-domain cross-attention module and the aggregated context transformation module process the feature data in series and are stacked 8 times to obtain deep fusion features, and the deep fusion features are up-sampled through three pixel rearrangements to obtain a repaired image and output the repaired image.

[0017] Preferably, in the step S3, during the training process, the initial image repair network is adjusted and optimized in parameters by using a verification data set to obtain an optimized final image repair network, and the step specifically includes: In each training iteration, the output result of the semantic prior guided repair network is taken as a global optimization target, and the semantic prior estimation network and the semantic prior guided repair network are trained and optimized by using an Adam optimizer; After each training iteration, the performance of the current network is evaluated by using a verification data set, and a composite training loss function is used, which is a weighted combination of an average absolute error loss function , a perceptual loss function , a style loss function , and an adversarial loss function The total loss value obtained by verifying the verification data set is calculated as follows:

[0018] wherein, is the weight of each loss function in the composite training loss function; The expression of the average absolute error loss function is as follows:

[0019] wherein, N is the total number of pixels, is the output image of the repair network, is a real complete image.

[0020] The perceptual loss function ​The expression of is as follows:

[0021] wherein, represents the feature map of the l-th layer in the network, is the weight coefficient of the corresponding layer.

[0022] Style loss function The expression of is as follows:

[0023] wherein, represents the Gram matrix of the feature map.

[0024] Adversarial loss function The expression of is as follows:

[0025] wherein, represents the discriminator network used for the adversarial loss function.

[0026] When the total loss value is stable, the final image inpainting network weight is saved and the training is completed.

[0027] Preferably, the step S4 specifically comprises: The final image inpainting network is tested using a test data set and a fixed damaged area data set, and the PSNR (peak signal-to-noise ratio) index, the SSIM (structural similarity) index, the LPIPS (learned perceptual image patch similarity) index, the MAE (mean absolute error) index and the FID (Fréchet inception distance) index are calculated. The results of the same test set are compared with the results of the same test set of the same period advanced image inpainting comparison method, and when the evaluation index of the method is better than or equal to the same period advanced image inpainting comparison method, it is determined that the image inpainting network test evaluation is qualified, and can be directly used for image inpainting.

[0028] In addition, the present application also provides an image inpainting system guided by priori, comprising: A unit for constructing a damaged image data set, wherein: A unit for constructing a training data set, a verification data set and a test data set for training a semantic prior estimation network and a semantic prior guided inpainting network.

[0029] A network construction unit, wherein: The semantic prior estimation network and the semantic prior guided inpainting network are constructed; the semantic prior estimation network comprises a feature extraction network and a mask classification and segmentation network, and the semantic prior guided inpainting network comprises a cross-domain cross-attention module and an aggregated context transformation module.

[0030] a unit for network training and optimization, in which: the semantic prior estimation network and the semantic prior guided inpainting network are trained using a training data set to obtain an initial image inpainting network; during the training process, the initial image inpainting network is adjusted and optimized in parameters using a verification data set to obtain a final image inpainting network after optimization.

[0031] a unit for network performance testing, in which: the final image inpainting network after optimization is used to perform image inpainting on a test data set and output the inpainted complete image as the inpainting result; the inpainting result is quantitatively evaluated and compared.

[0032] As a preferred embodiment, the damaged image data set construction unit specifically comprises: selecting a public image data set, the public image data set including a training data part, a verification data part and a test data part, wherein the images in the training data part are generated into damaged images using a random damaged area generation method and are subjected to a specific size cropping operation, and the images in the training data part after the above processing are used as the training data set; the images in the verification data part are generated into damaged images using a random damaged area generation method and are subjected to a specific size cropping operation, and the images in the verification data part after the above processing are used as the verification data set; the images in the test data part are subjected to a specific size cropping operation, and the cropped images are used as the test data set; the test data set is generated into damaged images using a random damaged area generation method, and is grouped according to the area proportion of the damaged area of the image, and the number of images in each group is consistent with the number of images in the test data set; the grouped data is saved as a fixed damaged area data set; During the training process using the above data set, no undamaged image is involved.

[0033] As a preferred embodiment, the semantic prior estimation network in the network construction unit, wherein the feature extraction network and the mask classification and segmentation network specifically comprise: the feature extraction network adopts an encoder-decoder structure, including a basic calculation module, downsampling and upsampling; the basic calculation module includes a fully connected feedforward network layer and a gating unit, wherein the gating unit is composed of a two-dimensional convolution and a Sigmoid activation function, and is used to filter invalid features, and the fully connected feedforward network layer is composed of layer normalization, two-dimensional data linear transformation and Gelu activation function; the downsampling adopts a two-dimensional convolution, so that the spatial size of the output feature map becomes half of the input; the upsampling is pixel rearrangement upsampling, so that the spatial size of the output feature map becomes twice the input.

[0034] The mask classification segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework, which is composed of a linear classifier, a multi-layer perceptron and a mask generator. The input of the mask classification segmentation network is the multi-scale features obtained by the feature extraction network. First, the linear classifier generates probability predictions of N categories, and at the same time, the multi-layer perceptron in the segmentation module converts the same input features into N mask embedding vectors. Then, the dot product similarity between each mask embedding vector and the high-resolution pixel embedding feature map output by the feature extraction network is calculated to generate N corresponding binary mask predictions. The combination of the category probability prediction and the binary mask prediction is output as the object-level semantic prior information.

[0035] Preferably, the cross-domain cross-attention module in the network construction unit specifically comprises: The cross-domain cross-attention module inputs the object-level semantic prior information output by the semantic prior estimation network and the damaged image features obtained by down-sampling the training data set image data. The object-level semantic prior information is down-sampled by one two-dimensional convolution and layer normalization to obtain semantic prior features, and the semantic prior features are convolved by Haar wavelets to obtain four groups of frequency domain semantic prior features. The damaged image features are convolved by one two-dimensional convolution to obtain four groups of spatial domain damaged image features. The four groups of frequency domain semantic prior features and the four groups of spatial domain damaged image features are fused by four groups of cross-attention modules, and the output results are obtained. After reverse Haar wavelet convolution, the fused data features are obtained. The fused data features are connected in residual connection with the damaged image features through a learnable scaling parameter to obtain the damaged image features guided and strengthened by the object-level semantic prior information.

[0036] Preferably, the aggregation context transformation module in the network construction unit specifically comprises: The aggregation context transformation module adopts the AOT block structure in the AOT-GAN algorithm, and fuses multi-scale context information based on the multi-stage form of "segmentation-transformation-aggregation" to enhance the semantic reasoning ability of the model for large missing areas. The aggregation context transformation module divides the standard convolution kernel into multiple parallel sub-convolution kernels in the segmentation stage; in the transformation stage, each sub-convolution kernel performs a dilated convolution operation on the input features with different dilation rates, thereby realizing the coverage of different receptive fields; in the aggregation stage, the features output by all sub-convolution kernels are spliced, and cross-channel fusion is performed through convolution operation to generate residual features of aggregated multi-scale context, and a gated residual connection is introduced to weight and fuse the features input into the aggregation context transformation module and the residual features, thereby improving the feature fusion effect for missing areas and known areas.

[0037] Preferably, the network training and optimization unit uses a training dataset to train the semantic prior estimation network and the semantic prior guided insulation network to obtain an initial image insulation network, which includes: a self-prior extraction sub-unit and a self-prior guided insulation sub-unit. The self-prior extracted sub-units include: The training dataset is input into the semantic prior estimation network to obtain object-level semantic prior information. The semantic prior estimation network generates multi-scale features of the image through a feature extraction network, and the multi-scale features are input into a mask classification and segmentation network to generate object-level semantic prior information.

[0038] The self-a priori guided repair subunit includes: The training dataset is downsampled through three 2D convolutions to obtain damaged image features. These damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into the semantic prior-guided repair network. The semantic prior-guided repair network fuses the damaged image features with the object-level semantic prior information through a cross-domain cross-attention module, and then processes the data through an aggregation context transformation module. The cross-domain cross-attention module and the aggregation context transformation module process the feature data sequentially in a series and are stacked eight times to obtain deep fusion features. The deep fusion features are upsampled by rearranging three pixels to obtain the repaired image, which is then output.

[0039] Preferably, in the network training and optimization unit, during the training process, a validation dataset is used to adjust and optimize the parameters of the initial image restoration network to obtain the optimized final image restoration network, specifically including: In each training iteration, the output of the semantic prior guided repair network is used as the global optimization objective, and the Adam optimizer is used to train and optimize the semantic prior estimation network and the semantic prior guided repair network. After each training iteration, the current network performance is evaluated using a validation dataset, employing a loss function based on the mean absolute error. Perceptual loss function Style loss function Adversarial loss function Weighted composite training loss function Calculate the total loss value obtained from validation on the validation dataset. Once the total loss value stabilizes, save the final image inpainting network weights and complete the training.

[0040] Preferably, the network performance testing unit specifically includes: The final image repairing network is tested using a test data set and a fixed damaged area data set, and the PSNR (peak signal-to-noise ratio) index, the SSIM (structural similarity) index, the LPIPS (learning perceptual image block similarity) index, the MAE (mean absolute error) index and the FID (Frechet inception distance) index are calculated, and the results of the same period advanced image repairing comparison method on the same test set are compared, when the evaluation index of the method is better than or equal to the same period advanced image repairing comparison method, that is, the image repairing network test evaluation is qualified, and can be directly used for image repairing.

[0041] The application has the advantages that the object-level semantic prior is effectively utilized, the cross-domain cross-attention module is deeply fused with image features, so that when facing an image with large-area missing or insufficient reference content, image content that is semantically reasonable and conforms to the logic of the real world can be generated, and the problems of content repetition and color distortion caused by traditional methods are effectively avoided; the robustness and fineness of feature fusion are improved, the rigid fusion mode of simply mapping the semantic segmentation result to a fixed class label is abandoned, the cross-domain cross-attention module utilizes the frequency domain information of the object-level semantic prior information, avoids the defect of directly regarding the pixel value of the segmentation mask as a class label, significantly improves the visual naturalness and detail accuracy of the repairing result, and has high robustness; in addition, the two-stage image repairing method composed of the semantic prior estimation network and the semantic prior guided repairing network effectively utilizes the residual semantic information in the damaged image, so that the repairing result conforms to the reasonable object contour and conforms to the human eye perception.

[0042] It can be seen that, compared with the prior art, the application has outstanding substantial features and significant progress, and the beneficial effects of the implementation are also obvious. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0044] Figure 1 is a flowchart of the image repairing method guided by the prior art provided by the application.

[0045] Figure 2 is a principle block diagram of the image repairing system guided by the prior art provided by the application.

[0046] Figure 3This is an overall structural block diagram of a self-a priori guided image restoration method provided by the present invention.

[0047] Figure 4 This is a schematic diagram of the semantic prior estimation network provided by the present invention.

[0048] Figure 5 This is a schematic diagram of the basic computational module of the encoder-decoder structure in the semantic prior estimation network provided by this invention.

[0049] Figure 6 This is a schematic diagram of the cross-domain cross-attention module of the semantic prior guided repair network provided by the present invention.

[0050] Figure 7 This is a comparison image of the visual effects provided by this invention and existing image restoration algorithms under the same missing conditions.

[0051] Among them, 1-unit for constructing damaged image dataset, 2-unit for network construction, 3-unit for network training and optimization, and 4-unit for network performance testing. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following implementation methods.

[0053] Example 1: like Figure 1 As shown, this embodiment provides a self-guided image inpainting method, which includes the following steps: Step S1: Construct the training dataset, validation dataset, and test dataset for training the semantic prior estimation network and the semantic prior guided repair network.

[0054] Step S1 specifically includes: The ADE20K and COCO2017 public image datasets are selected, sRGB image data of the image ADE20K Train and COCO2017 Train in the training data part is generated into damaged images by using a random damaged area generation method, and is cropped into 256*256 size, the image in the training data part after the above processing is used as the training dataset; the sRGB image data of the image ADE20K Validation and COCO2017 Validation in the verification data part is generated into damaged images by using a random damaged area generation method, and is cropped into 256*256 size, the image in the verification data part after the above processing is used as the verification dataset; the sRGB image data of the image ADE20K Test and COCO2017 Test in the test data part is taken in the order of the original image data, and the first 10% of the sRGB image data is cropped into 256*256 size as the test dataset; The test dataset is generated into damaged images by using a random damaged area generation method, and is divided into four groups according to the area proportion of the damaged area of the image, 0% to 20%, 20% to 40%, 40% to 60%, and 60% to 80%, and the number of each group is completely consistent with the number of images in the test dataset; the grouped data is saved as a fixed damaged area dataset; In the training process using the above dataset, no undamaged image is involved.

[0055] Step S2, a semantic prior estimation network and a semantic prior guided repair network are constructed, and the overall architecture block diagram is as shown in Figure 3 The semantic prior estimation network includes a feature extraction network and a mask classification segmentation network, and the structure diagram is as shown in Figure 4 The semantic prior guided repair network includes a cross-domain cross-attention module and an aggregated context transformation module.

[0056] The feature extraction network and the mask classification segmentation network in the semantic prior estimation network in the step S2, specifically include: The feature extraction network adopts an encoder-decoder structure, including a basic calculation module, downsampling and upsampling; the basic calculation module includes a full connection feedforward network layer and a gating unit, wherein the gating unit is composed of a two-dimensional convolution and a Sigmoid activation function, and is used for filtering invalid features; the full connection feedforward network layer is composed of layer normalization, two-dimensional data linear transformation and Gelu activation function; the downsampling adopts two-dimensional convolution, so that the spatial size of the output feature map becomes half of the input; the upsampling is pixel rearrangement upsampling, so that the spatial size of the output feature map becomes twice of the input.

[0057] As shown in Figure 5As shown, the input features of the basic calculation module are first subjected to a gated convolution, that is, the combination of convolution and sigmoid branch and single convolution branch convolution point multiplication, and the convolution and sigmoid branch is used to distinguish the weight values of the effective pixel area and the invalid pixel area, and the effective pixel is closer to 1 and the invalid pixel is closer to 0. The convolution point multiplication obtains the effective processing result of the initial input feature by multiplying the output features of the single convolution branch with the weight values output by the convolution and sigmoid branch. The layer normalization is used to normalize the effective processing result. After normalization, the features are mapped to 4 times the channel number through convolution, and then mapped back to the original channel number through Gelu activation function and convolution. This method can better preserve potential features while filtering out useless features. Finally, the output features of the basic calculation module are obtained by adding a learnable coefficient and the residual of the initial input feature.

[0058] In the encoder-decoder structure, the encoder path includes three stages, each stage being composed of multiple basic calculation modules and down-sampling operations, and the decoder path includes four stages, each stage being composed of multiple basic calculation modules and up-sampling operations. The three stages in the encoder path are respectively a first stage of 3 basic calculation modules and down-sampling operations, a second stage of 3 basic calculation modules and down-sampling operations, and a third stage of 9 basic calculation modules and down-sampling operations; the four stages in the decoder path are respectively a first stage of 6 basic calculation modules and up-sampling operations, a second stage of 9 basic calculation modules and up-sampling operations, a third stage of 3 basic calculation modules and up-sampling operations, and a fourth stage of 3 basic calculation modules and up-sampling operations. The multi-scale features output by the feature extraction network are composed of the outputs of the last basic calculation module in each stage of the decoder path of the feature extraction network.

[0059] The mask classification segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework and is composed of a linear classifier, a multi-layer perceptron and a mask generator. The linear classifier is responsible for generating the prediction probability of the class, the multi-layer perceptron is responsible for converting the input features into mask embedding vectors, and the mask generator is responsible for calculating the dot product similarity between each mask embedding vector and the features output by the feature extraction network and generating the corresponding binary mask prediction. Finally, the combination of the class prediction probability and the binary mask prediction is output as the object-level semantic prior information.

[0060] The cross-domain cross-attention module specifically includes: The cross-domain cross-attention module is as shown in Figure 6As shown, the input is the object-level semantic prior information output by the semantic prior estimation network and the damaged image features obtained by downsampling the image data of the training data set. The object-level semantic prior information is processed by one-dimensional convolution downsampling and layer normalization to obtain semantic prior features, denoted as Then, the semantic prior features are subjected to Haar wavelet convolution to obtain four groups of frequency domain semantic prior features :

[0061] wherein, is the four filter kernels of the Haar wavelet transform, represent high-pass filters in three different directions, i.e., horizontal, vertical and diagonal directions, respectively; represents a low-pass filter, which has an effect similar to average pooling. The kernel definitions of the high-pass filter and the low-pass filter are as follows:

[0062] The formula expression of performing Haar wavelet convolution on the semantic prior features can obtain The expression of the four sub-band components in the frequency domain is i.e., the four groups of frequency domain semantic prior features, which have a learnable channel scaling mechanism in the wavelet convolution to align the feature shape and enhance the expression ability of the effective feature channel while maintaining the frequency domain unchanged. Then, the four groups of frequency domain semantic prior features are subjected to cross-domain cross-attention calculation with the damaged image features representing the spatial domain information, because has four groups of features, the 256-dimensional damaged image features need to be subjected to a channel expansion mapping convolution operation to obtain four groups of 256-dimensional spatial domain damaged image features denoted as:

[0063] The four groups of frequency domain semantic prior features and the four groups of spatial domain damaged image features are subjected to data fusion by four cross-attention modules to complete the cross-domain interaction of the spatial domain information and the frequency domain information, obtaining :

[0064] wherein, , , is a learnable parameter matrix, for attention scaling, for the mixed information sequence obtained through the cross-domain cross-attention calculation.

[0065] for The reverse Haar wavelet convolution is performed again to obtain the fused data features, denoted as:

[0066] wherein, is the fused data feature, i.e., the prior guided image feature.

[0067] the most fused data feature Through a learnable scaling parameter a and the original damaged image feature Residual connection is performed to obtain the object-level semantic prior information, which is used to guide the enhanced damaged image feature, and the process is denoted as:

[0068] wherein, is the output feature.

[0069] The aggregation context transformation module specifically comprises: The aggregation context transformation module adopts the AOT block structure in the AOT-GAN algorithm, and fuses multi-scale context information in a multi-stage form based on "segmentation-transformation-aggregation" to enhance the semantic reasoning ability of the model to large-scale missing regions. In the segmentation stage, the standard convolution kernel is segmented into multiple parallel sub-convolution kernels; in the transformation stage, each sub-convolution kernel performs a dilated convolution operation on the input feature with different dilation rates, thereby realizing the coverage of different receptive fields; in the aggregation stage, the features output by all sub-convolution kernels are spliced, and cross-channel fusion is performed through convolution operation to generate residual features of aggregated multi-scale contexts, and a gated residual connection is introduced to weight fuse the features input into the aggregation context transformation module and the residual features, further improving the feature fusion effect of the missing regions and the known regions.

[0070] Step S3: training the semantic prior estimation network and the semantic prior guided repair network using the training data set to obtain an initial image repair network; in the training process, the initial image repair network is parameter adjusted and optimized using a verification data set to obtain an optimized final image repair network.

[0071] The step S3 of training the semantic prior estimation network and the semantic prior guided repair network using the training data set to obtain an initial image repair network comprises: Step S31: from the step of prior extraction: The training dataset is input into the semantic prior estimation network to obtain object-level semantic prior information. The semantic prior estimation network generates multi-scale features of the image through a feature extraction network, and the multi-scale features are input into a mask classification and segmentation network to generate object-level semantic prior information.

[0072] Step S32, the steps for self-prioritized boot repair: The training dataset is downsampled through three 2D convolutions to obtain damaged image features. The damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into the semantic prior guided repair network. The semantic prior guided repair network fuses the damaged image features with the object-level semantic prior information through a cross-domain cross attention module, and then processes them through an aggregation context transformation module. The cross-domain cross attention module and the aggregation context transformation module process the feature data sequentially in a series and are stacked eight times to obtain deep fusion features. The deep fusion features are upsampled by rearranging three pixels to obtain the repaired image and output it.

[0073] In step S3, during the training process, the initial image inpainting network is adjusted and optimized using a validation dataset to obtain the optimized final image inpainting network. This specifically includes: In each training iteration, the output of the semantic prior guided repair network is used as the global optimization objective, and the Adam optimizer is used to train and optimize the semantic prior estimation network and the semantic prior guided repair network. After each training iteration, the network performance is evaluated using a validation dataset, employing the mean absolute error loss function. Perceptual loss function Style loss function Adversarial loss function Weighted composite training loss function Calculate the total loss obtained from validation on the validation dataset:

[0074] Once the total loss value stabilizes, save the final image inpainting network weights and complete the training.

[0075] Step S4 involves a quantitative evaluation and comparison of the optimized image inpainting network using a test dataset.

[0076] Step S4 specifically includes: The final image restoration network is tested using the test data set and the fixed damaged area data set, and the PSNR (peak signal-to-noise ratio) index, the SSIM (structural similarity) index, the LPIPS (learning perceptual image block similarity) index, the MAE (mean absolute error) index and the FID (Fréchet inception distance) index are calculated.

[0077] Deepfillv2, BOWN, AOT-GAN, CMT, and SpaFormer are tested using the same test data set, and the corresponding evaluation indicators are calculated, and the image visualization comparison is as shown in Figure 7 As shown in Table 1, the PSNR index, the SSIM index, the LPIPS index, the MAE index and the FID index of the priori-guided image restoration method and the existing image restoration algorithm on the top 10% of the COCO2017 Test data set are compared.

[0078] Table 1

[0079] Step S5, using the image restoration network that passes the test evaluation to restore the image, generate and output the restored complete image.

[0080] Embodiment 2: As shown in Figure 2 The priori-guided image restoration system provided by the embodiment comprises: A damaged image data set construction unit 1, in which: A training data set, a validation data set and a test data set for training a semantic prior estimation network and a semantic prior guided restoration network are constructed.

[0081] The damaged image data set construction unit 1 specifically comprises: A public image data set is selected, which includes a training data part, a validation data part and a test data part, wherein the images in the training data part are generated into damaged images by a random damaged area generation method, and a specific size cropping operation is performed, and the images in the training data part after the above processing are used as the training data set; the images in the validation data part are generated into damaged images by a random damaged area generation method, and a specific size cropping operation is performed, and the images in the validation data part after the above processing are used as the validation data set; the images in the test data part are subjected to a specific size cropping operation, and the cropped images are used as the test data set; The test data set is generated into damaged images by a random damaged area generation method, and is grouped according to the damaged area area ratio of the images, and the number of images in each group is consistent with the number of images in the test data set; the grouped data is saved as a fixed damaged area data set; During the training process using the above dataset, no unimpaired images are involved.

[0082] The network construction unit 2, in which: The semantic prior estimation network and the semantic prior guided inpainting network are constructed; the semantic prior estimation network comprises a feature extraction network and a mask classification segmentation network, and the semantic prior guided inpainting network comprises a cross-domain cross-attention module and an aggregated context transformation module.

[0083] The semantic prior estimation network in the network construction unit 2, wherein the feature extraction network and the mask classification segmentation network specifically comprise: The feature extraction network adopts an encoder-decoder structure, comprising a basic calculation module, downsampling and upsampling; the basic calculation module comprises a full-connection feedforward network layer and a gating unit, wherein the gating unit is composed of a two-dimensional convolution and a Sigmoid activation function, and is used for filtering invalid features; the full-connection feedforward network layer is composed of layer normalization, two-dimensional data linear transformation and Gelu activation function; the downsampling adopts two-dimensional convolution, so that the spatial size of the output feature map becomes half of the input; the upsampling is pixel rearrangement upsampling, so that the spatial size of the output feature map becomes twice of the input.

[0084] The mask classification segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework, and is composed of a linear classifier, a multilayer perceptron and a mask generator. The input of the mask classification segmentation network is the multi-scale features obtained by the feature extraction network. First, the linear classifier is used to generate probability predictions of N categories, and at the same time, the multilayer perceptron in the segmentation module is used to convert the same input features into N mask embedding vectors. Then, the dot product similarity between each mask embedding vector and the high-resolution pixel embedding feature map output by the feature extraction network is calculated to generate N corresponding binary mask predictions. The combination of the category probability prediction and the binary mask prediction is output as the object-level semantic prior information.

[0085] The cross-domain cross-attention module in the network construction unit 2 specifically comprises: The cross-domain cross-attention module takes as input the object-level semantic prior information output by the semantic prior estimation network and the down-sampled damaged image features. The object-level semantic prior information is down-sampled by one two-dimensional convolution and layer normalization to obtain semantic prior features, and the semantic prior features are then convolved by Haar wavelets to obtain four groups of frequency domain semantic prior features. The damaged image features are convolved by one two-dimensional convolution to obtain four groups of spatial domain damaged image features. The four groups of frequency domain semantic prior features and the four groups of spatial domain damaged image features are fused by four cross-attention modules, and the output is then inversely Haar wavelet-convolved to obtain the fused data features. The fused data features are connected in residual connection with the damaged image features through a learnable scaling parameter to obtain the damaged image features guided and reinforced by the object-level semantic prior information.

[0086] The aggregation context transformation module in the network construction unit 2 specifically includes: The AOT block structure in the AOT-GAN algorithm is adopted to fuse multi-scale context information based on the "segmentation-transformation-aggregation" strategy to enhance the semantic reasoning ability of the model for large-scale missing regions. The aggregation context transformation module divides the standard convolution kernel into multiple parallel sub-convolution kernels in the segmentation stage; in the transformation stage, each sub-convolution kernel performs a dilated convolution operation on the input features with different dilation rates, thereby covering different receptive fields; in the aggregation stage, the features output by all sub-convolution kernels are spliced and fused across channels through convolution operation to generate residual features of aggregated multi-scale context, and a gated residual connection is introduced to weight and fuse the features input into the aggregation context transformation module and the residual features, thereby improving the feature fusion effect for missing regions and known regions.

[0087] The network training and optimization unit 3, in which: The semantic prior estimation network and the semantic prior guided repair network are trained using a training data set to obtain an initial image repair network; during the training process, the initial image repair network is parameter-adjusted and optimized using a validation data set to obtain an optimized final image repair network.

[0088] The network training and optimization unit 3, in which: The self-prior extraction sub-unit includes: The data of the training data set is input into the semantic prior estimation network to obtain object-level semantic prior information, wherein the semantic prior estimation network generates multi-scale features of the image through a feature extraction network, and the multi-scale features are input into a mask classification segmentation network to generate object-level semantic prior information.

[0089] The self-a priori guided repair subunit includes: The training dataset is downsampled through three 2D convolutions to obtain damaged image features. The damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into the semantic prior guided repair network. The semantic prior guided repair network fuses the damaged image features with the object-level semantic prior information through a cross-domain cross attention module, and then processes them through an aggregation context transformation module. The cross-domain cross attention module and the aggregation context transformation module process the feature data sequentially in a series and are stacked eight times to obtain deep fusion features. The deep fusion features are upsampled by rearranging three pixels to obtain the repaired image and output it.

[0090] In the network training and optimization unit 3, during the training process, the initial image inpainting network is adjusted and optimized using a validation dataset to obtain the optimized final image inpainting network, specifically including: In each training iteration, the output of the semantic prior guided repair network is used as the global optimization objective, and the Adam optimizer is used to train and optimize the semantic prior estimation network and the semantic prior guided repair network. After each training iteration, the current network performance is evaluated using a validation dataset, employing a loss function based on the mean absolute error. Perceptual loss function Style loss function Adversarial loss function Weighted composite training loss function Calculate the total loss value obtained from validation on the validation dataset. Once the total loss value stabilizes, save the final image inpainting network weights and complete the training.

[0091] Unit 4 of the network performance test includes: The optimized final image inpainting network is used to inpaint images on the test dataset and outputs the inpainted complete image as the inpainting result; the inpainting results are quantitatively evaluated and compared.

[0092] The network performance testing unit 4 specifically includes: The final image repairing network is tested using a test data set and a fixed damaged region data set, and the PSNR (peak signal-to-noise ratio) index, SSIM (structural similarity) index, LPIPS (learning perceptual image block similarity) index, MAE (mean absolute error) index and FID (Frechet inception distance) index are calculated, and the results of the same period advanced image repairing comparison method on the same test set are compared, when the evaluation index of the method is better than or equal to the same period advanced image repairing comparison method, that is, the image repairing network test evaluation is qualified, and can be directly used for image repairing.

[0093] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the method disclosed in the embodiments, since it corresponds to the system disclosed in the embodiments, the description is relatively simple, and the relevant parts are described in the method part.

[0094] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0095] In several embodiments provided by the present application, it should be understood that the disclosed system, system and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, system or unit, and can be electrical, mechanical or other forms.

[0096] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0097] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present separately, or two or more modules can be integrated in one unit.

[0098] Similarly, each processing unit in each embodiment of the present application can be integrated in one function module, or each processing unit can be physically present separately, or two or more processing units can be integrated in one function module.

[0099] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it needs to be pointed out that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0101] The above disclosure is only the preferred embodiments of the present application, but the present application is not limited thereto, any non-creative changes that can be thought of by those skilled in the art, and several improvements and refinements made without departing from the principles of the present application, should fall within the scope of protection of the present application.

Claims

1. A self-guided image inpainting method, characterized in that, Includes the following steps: Step S1: Construct training datasets, validation datasets, and test datasets for training the semantic prior estimation network and the semantic prior guided repair network; Step S2: Construct a semantic prior estimation network and a semantic prior guided repair network; The semantic prior estimation network includes a feature extraction network and a mask classification and segmentation network, while the semantic prior guided repair network includes a cross-domain attention module and an aggregation context transformation module. Step S3: Train the semantic prior estimation network and the semantic prior guided insulation network using the training dataset to obtain the initial image insulation network; During training, the initial image inpainting network was adjusted and optimized using a validation dataset to obtain the optimized final image inpainting network. Step S4: Use the optimized final image inpainting network to inpaint the test dataset and output the inpainted complete image as the inpainting result; The repair results were quantitatively evaluated and compared.

2. The self-guided image inpainting method according to claim 1, characterized in that, The semantic prior estimation network in step S2, including the feature extraction network and the mask classification and segmentation network, specifically includes: The feature extraction network adopts an encoder-decoder structure, including a basic computation module, downsampling, and upsampling. The basic computation module includes a fully connected feedforward network layer and a gating unit. The gating unit consists of a 2D convolution and a sigmoid activation function. The fully connected feedforward network layer consists of layer normalization, 2D data linear transformation, and a Gelu activation function. Downsampling uses 2D convolution to make the spatial size of the output feature map half that of the input. Upsampling is pixel rearrangement upsampling, which makes the spatial size of the output feature map twice that of the input. The mask classification and segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework, which consists of a linear classifier, a multilayer perceptron, and a mask generator to generate and output object-level semantic prior information.

3. The self-guided image inpainting method according to claim 1, characterized in that, The cross-domain attention module specifically includes: The input to the cross-domain attention module is the object-level semantic prior information output by the semantic prior estimation network and the damaged image features obtained by downsampling the training dataset image data; The semantic prior information of the object level is subjected to a two-dimensional convolution downsampling and layer normalization to obtain semantic prior features. For semantic prior features Perform Haar wavelet convolution to obtain frequency domain semantic prior features. : in, The four filters that form the core of the Haar wavelet transform are... These represent high-pass filters in three different directions: horizontal, vertical, and diagonal. Represents a low-pass filter; The damaged image features are processed by convolution to obtain four sets of spatial domain damaged image features. : in, For spatially damaged image features, This is a set of four spatial domain damaged image features obtained after convolution operation with channel extension mapping. Four sets of frequency domain semantic prior features and four sets of spatial domain damaged image features Data fusion was performed using four sets of cross-attention modules to obtain... : in, , , For a learnable parameter matrix, For scaling of attention, This is a mixed information sequence obtained through cross-domain cross-attention computation; right Perform inverse Haar wavelet convolution to obtain the fused data features : in, The fused data features are the prior guided image features. Fusion data features By performing a residual connection between the original damaged image features and a learnable scaling parameter α, the enhanced damaged image features guided by object-level semantic prior information are obtained.

4. The self-guided image inpainting method according to claim 1, characterized in that, The aggregation context transformation module specifically includes: The context transformation module integrates multi-scale contextual feature information based on a multi-stage process of "segmentation-transformation-aggregation," specifically as follows: The standard convolutional kernel is divided into multiple parallel sub-convolutional kernels; Each sub-convolutional kernel performs a dilated convolution operation on the initial features of the input aggregation context transformation module with a different dilation rate to cover different receptive fields; The features output by all sub-convolutional kernels are concatenated and fused across channels using 1×1 convolution to generate residual features that aggregate multi-scale context. We use a spatially adaptive gating graph generated by convolution and sigmoid function of the initial features to perform weighted fusion of the initial features and the residual features of the aggregated multi-scale context.

5. The self-guided image inpainting method according to claim 1, characterized in that, Step S3, which involves training the semantic prior estimation network and the semantic prior guided inpainting network using a training dataset to obtain the initial image inpainting network, includes: Step S31, the self-prior extraction step: The training dataset is input into the semantic prior estimation network to obtain object-level semantic prior information. The semantic prior estimation network generates multi-scale features of the image through the feature extraction network, and the multi-scale features are input into the mask classification and segmentation network to generate object-level semantic prior information. Step S32, the steps for self-prioritized boot repair: The data from the training dataset is downsampled through three two-dimensional convolutions to obtain damaged image features. The damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into the semantic prior-guided repair network. The semantic prior guided repair network integrates the damaged image features with the object-level semantic prior information through a cross-domain cross-attention module, then processes the data through an aggregation context transformation module, and finally performs 3-pixel rearrangement upsampling to obtain the repaired image and output it.

6. A self-guided image inpainting system, characterized in that, include: A cell constructed from a damaged image dataset, in which: Construct training, validation, and test datasets for training the semantic prior estimation network and the semantic prior guided repair network; The unit that constructs the network, in which: Construct a semantic prior estimation network and a semantic prior guided repair network; the semantic prior estimation network includes a feature extraction network and a mask classification and segmentation network, and the semantic prior guided repair network includes a cross-domain attention module and an aggregation context transformation module. The unit for network training and optimization, in which: The semantic prior estimation network and the semantic prior guided insulation network were trained using the training dataset to obtain the initial image insulation network; During training, the initial image inpainting network was adjusted and optimized using a validation dataset to obtain the optimized final image inpainting network. The network performance testing unit, in which: The optimized final image inpainting network is used to inpaint images on the test dataset and outputs the inpainted complete image as the inpainting result. The repair results were quantitatively evaluated and compared.

7. The self-guided image restoration system according to claim 6, characterized in that, The semantic prior estimation network in the aforementioned network construction unit, including the feature extraction network and the mask classification and segmentation network, specifically includes: The feature extraction network adopts an encoder-decoder structure, including downsampling, upsampling, and basic computation modules. The basic computation module includes a fully connected feedforward network layer and gating units. The gating units consist of two-dimensional convolutions and a sigmoid activation function, while the fully connected feedforward network layer consists of layer normalization, two-dimensional linear data transformation, and a Gelu activation function. Downsampling uses two-dimensional convolutions to make the spatial size of the output feature map half that of the input. Upsampling is pixel rearrangement upsampling, which makes the spatial size of the output feature map twice that of the input. The mask classification and segmentation network adopts the segmentation module structure in the MaskFormer algorithm framework, which consists of a linear classifier, a multilayer perceptron, and a mask generator to generate and output object-level semantic prior information.

8. The self-guided image restoration system according to claim 6, characterized in that, The cross-domain attention module in the network construction unit specifically includes: The input to the cross-domain attention module is the object-level semantic prior information output by the semantic prior estimation network and the damaged image features obtained by downsampling the training dataset image data; The semantic prior information of the object level is subjected to a two-dimensional convolution downsampling and layer normalization to obtain semantic prior features. For semantic prior features Perform Haar wavelet convolution to obtain frequency domain semantic prior features. : in, The four filters that form the core of the Haar wavelet transform are... These represent high-pass filters in three different directions: horizontal, vertical, and diagonal. Represents a low-pass filter; The damaged image features are processed by convolution to obtain four sets of spatial domain damaged image features. : in, For spatially damaged image features, This is a set of four spatial domain damaged image features obtained after convolution operation with channel extension mapping. Four sets of frequency domain semantic prior features and four sets of spatial domain damaged image features Data fusion was performed using four sets of cross-attention modules to obtain... : in, , , For a learnable parameter matrix, For scaling of attention, This is a mixed information sequence obtained through cross-domain cross-attention computation; right Perform inverse Haar wavelet convolution to obtain the fused data features : in, The fused data features are the prior guided image features. Fusion data features By performing a residual connection between the damaged image features and a learnable scaling parameter α, the enhanced damaged image features guided by object-level semantic prior information are obtained.

9. A self-guided image restoration system according to claim 6, characterized in that, The aggregation context transformation module in the network construction unit specifically includes: The context transformation module integrates multi-scale contextual feature information based on a multi-stage process of "segmentation-transformation-aggregation," specifically as follows: The standard convolutional kernel is divided into multiple parallel sub-convolutional kernels; Each sub-convolutional kernel performs a dilated convolution operation on the initial features of the input aggregation context transformation module with a different dilation rate to cover different receptive fields; The features output by all sub-convolutional kernels are concatenated and fused across channels using 1×1 convolution to generate residual features that aggregate multi-scale context. We use a spatially adaptive gating graph generated by convolution and sigmoid function of the initial features to perform weighted fusion of the initial features and the residual features of the aggregated multi-scale context.

10. A self-guided image restoration system according to claim 6, characterized in that, The network training and optimization unit uses a training dataset to train the semantic prior estimation network and the semantic prior guided inpainting network to obtain an initial image inpainting network, which includes: a self-prior extraction sub-unit and a self-prior guided inpainting sub-unit. The self-prior extracted sub-units include: The training dataset is input into the semantic prior estimation network to obtain object-level semantic prior information. The semantic prior estimation network generates multi-scale features of the image through the feature extraction network, and the multi-scale features are input into the mask classification and segmentation network to generate object-level semantic prior information. The self-a priori guided repair subunit includes: The data from the training dataset is downsampled through three two-dimensional convolutions to obtain damaged image features. The damaged image features and the object-level semantic prior information output by the semantic prior estimation network are jointly input into the semantic prior-guided repair network. The semantic prior guided repair network integrates the damaged image features with the object-level semantic prior information through a cross-domain cross-attention module, then processes the data through an aggregation context transformation module, and finally performs 3-pixel rearrangement upsampling to obtain the repaired image and output it.

Images