Image restoration method based on diffusion mode and prior feature generation and related device

Abstract

The invention relates to the technical field of image enhancement or restoration, in particular to an image restoration method based on diffusion mode and prior feature generation and a related device. In order to solve the technical problem of poor image restoration quality in the prior art, real data is utilized to train a priori feature restoration image model composed of a priori feature extraction network, a U-net-based mask combination restoration network and a mask restoration network; then, constructing a priori feature repairing and enhancing image model based on a training priori feature extraction network, an image noise adding network, an image denoising network, a priori feature estimation network and a training mask repairing network; a training image noise adding network, a training image denoising network, a training prior feature estimation network and a training mask repairing network are extracted to form an image repairing model, the model is used for processing a to-be-repaired image, and efficient and high-quality image repairing is carried out on a specific area.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement or restoration, and specifically to an image inpainting method and related device generated based on a diffusion pattern and prior features. Background Art

[0002] Image inpainting is the process of filling in missing regions in a digital image. The goal of image inpainting is to maintain semantically reasonable and visually realistic content while being consistent with the rest of the image when filling in the missing regions. Therefore, image inpainting technology has received extensive attention.

[0003] Recently, diffusion models have performed well in image synthesis and inpainting tasks. However, directly applying diffusion models to image inpainting tasks faces two major challenges: on the one hand, diffusion models usually require a large number of iterations, and their high computational cost limits practical applications; on the other hand, the generation process based on the image synthesis paradigm may lead to inconsistencies with known regions, thus affecting the quality of image inpainting. Summary of the Invention

[0004] The purpose of the present invention is to provide an image inpainting method and related device generated based on a diffusion pattern and prior features.

[0005] The technical solution of the present invention is as follows: An image inpainting method generated based on a diffusion pattern and prior features, including the following operations: The image to be inpainted is processed by an image inpainting model to obtain an image inpainting result; the image inpainting model includes a training image noise addition network, a training image denoising network, a training prior feature estimation network, and a training mask inpainting network in the training prior feature inpainting and enhancement image model; The training prior feature inpainting and enhancement image model is obtained by training the prior feature inpainting and enhancement image model using a training data set; the prior feature inpainting and enhancement image model includes: a training prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a training mask inpainting network; The training prior feature extraction network and the training mask inpainting network are obtained based on the training prior feature inpainting image model; The training prior feature inpainting image model is obtained by training the prior feature inpainting image model using a training data set; the prior feature inpainting image model includes: a prior feature extraction network, a mask inpainting combined network based on U-net, and a mask inpainting network; The training data set is formed by a number of real standard images and corresponding mask images.

[0006] During the process of training the prior feature restoration and enhancement image model, the real standard image and the corresponding mask image are processed by the prior feature extraction network and the image noise addition network to obtain a noisy image; the mask image is processed by the prior feature estimation network to obtain a prior estimation feature map; the prior estimation feature map and the noisy image are processed by the image denoising network and the training mask restoration network until the enhancement image loss value is less than the enhancement image loss threshold, and the training ends.

[0007] During the process of the prior feature extraction network, after splicing the real standard image and the corresponding mask image, a spliced image is obtained; the spatial information of the spliced image is rearranged to the preset channel dimension, and the rearranged features are subjected to a splicing operation to obtain a channel dimension enhanced map; the channel dimension enhanced map is successively subjected to convolution processing, LReLU activation function processing, residual block processing, average pooling processing, linear processing, and LReLU activation function processing to obtain a prior feature map for performing the operation processed by the mask restoration combined network based on U-net.

[0008] The operation processed by the mask combination restoration network based on U-net is specifically as follows: the mask image and the prior feature map are processed by the mask restoration network to obtain a first mask restoration feature map; the first mask restoration feature map and the prior feature map are processed by the mask restoration network to obtain a second mask restoration feature map; the second mask restoration feature map and the prior feature map are processed by the mask restoration network to obtain a third mask restoration feature map; the third mask restoration feature map and the prior feature map are processed by the mask restoration network to obtain a fourth mask restoration feature map; after the fourth mask restoration feature map and the third mask restoration feature map are spliced, they are processed by the mask restoration network together with the prior feature map to obtain a first mask combination restoration feature map; after the first mask combination restoration feature map and the second mask restoration feature map are spliced, they are processed by the mask restoration network together with the prior feature map to obtain a second mask combination restoration feature map; after the second mask combination restoration feature map and the first mask restoration feature map are spliced, they are processed by the mask restoration network together with the prior feature map to obtain an initial restoration map for performing the operation processed by the mask restoration network.

[0009] During the process of the mask restoration network, the prior feature map is injected into the initial restoration map to obtain an initial restoration prior feature injection map; the initial restoration prior feature injection map is subjected to global and local attention fusion processing and feature aggregation processing for several times to obtain an attention aggregation feature map; the attention aggregation feature map is subjected to convolution processing and then subjected to residual connection processing with the mask image to obtain a prior feature restoration map; the initial restoration map is the output of the mask combination restoration network based on U-net.

[0010] The operation of global and local attention fusion processing is as follows: The initial repair injection prior feature map undergoes global-local self-attention processing and global region-guided attention processing to obtain an attention fusion feature map, which is used to perform the operation of feature aggregation processing; The operation of global-local self-attention processing is as follows: The initial repair injection prior feature map is divided into horizontal windows and vertical windows, and after undergoing multi-head attention processing of non-overlapping sub-windows respectively, all the outputs are concatenated in the channel dimension to obtain an initial self-attention map; The initial attention map and the initial repair injection prior feature map undergo residual connection processing to obtain a self-attention feature map, which is used to perform the operation of global region-guided attention processing.

[0011] The operation of global region-guided attention processing is as follows: Inject the prior feature map into the self-attention feature map to obtain a self-attention injected prior feature map; Recursively use a single depthwise separable convolution to process the self-attention injected prior feature map to obtain a rough fusion map; The rough fusion map undergoes depthwise separable convolution and pixel-level convolution processing to obtain a representative feature map; Based on the key feature matrix and value feature matrix of the representative feature map, an attention matrix is obtained; Based on the query matrix of the attention matrix and the self-attention injected prior feature map, attention cross-processing is performed to obtain a cross-attention feature map; After the cross-attention feature map undergoes reshaping processing, it undergoes residual connection processing with the self-attention feature map to obtain an attention fusion feature map.

[0012] An image restoration system based on diffusion mode and prior feature generation, which is used to implement the above-mentioned image restoration method based on diffusion mode and prior feature generation, includes: A training prior feature restoration image model generation module, which is used to use a training data set to train a prior feature restoration image model to obtain a trained prior feature restoration image model; The prior feature restoration image model includes: a prior feature extraction network, a mask restoration combined network based on U-net, and a mask restoration network; The training data set is formed by a number of real standard images and corresponding mask images; A training prior feature restoration enhanced image model generation module, which is used to use a training data set to train a prior feature restoration enhanced image model to obtain a trained prior feature restoration enhanced image model; The prior feature restoration enhanced image model includes: a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask restoration network; The trained prior feature extraction network and the trained mask restoration network are obtained based on the trained prior feature restoration image model; An image restoration result generation module, which is used to process the image to be restored through an image restoration model to obtain an image restoration result; The image restoration model includes the trained image noise addition network, the trained image denoising network, the trained prior feature estimation network, and the trained mask restoration network in the trained prior feature restoration enhanced image model.

[0013] An image inpainting device generated based on a diffusion pattern and prior features, including a processor and a memory. When the processor executes the computer program stored in the memory, the above-mentioned image inpainting method generated based on a diffusion pattern and prior features is implemented.

[0014] A computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, the above-mentioned image inpainting method generated based on a diffusion pattern and prior features is implemented.

[0015] The beneficial effects of the present invention are as follows: The image inpainting method provided by the present invention, which is generated based on a diffusion pattern and prior features, first uses real data to train a prior feature inpainting image model. The prior feature extraction network can extract effective features. The mask combination inpainting network and the mask inpainting network based on U-net can use the mask information to specifically inpaint the image, improving the accuracy and effect of image inpainting to better restore the missing or damaged parts of the image. Then, a prior feature inpainting enhanced image model is trained using a training data set. The prior feature inpainting enhanced image model includes a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask inpainting network, which can refine the details of image inpainting and enhance the effect of image inpainting. Through the cooperation of multiple networks, with the help of prior feature-related networks and mask inpainting networks, as well as image noise addition and denoising operations, the image is processed more precisely, improving the quality of image inpainting. Finally, a trained image noise addition network, a trained image denoising network, a trained prior feature estimation network, and a trained mask inpainting network are obtained from the trained prior feature inpainting enhanced image model to form an image inpainting model. During the process of using this model to process the image to be inpainted, the trained noise addition and denoising networks process and optimize the image, the trained prior feature estimation network provides prior information, and the trained mask inpainting network repairs specific regions, thus efficiently and high-qualityly completing image inpainting. Applied in the field of image inpainting, it can improve the pertinence, effect, and efficiency of image inpainting. Description of the Drawings

[0016] By reading the detailed description of the preferred embodiments below, the solutions and advantages of the present application will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.

[0017] In the drawings: Figure 1 For the embodiment, it is the image inpainting effect diagram of the method of this embodiment. Detailed Embodiments

[0018] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings.

[0019] This embodiment provides an image inpainting method based on diffusion mode and prior features, including the following operations: The image to be inpainted is processed by an image inpainting model to obtain an image inpainting result; the image inpainting model includes a training image noise adding network, a training image denoising network, a training prior feature estimation network, and a training mask inpainting network in the training prior feature inpainting enhanced image model. The training prior feature inpainting enhanced image model is obtained by training the prior feature inpainting enhanced image model using a training data set; the prior feature inpainting enhanced image model includes: a training prior feature extraction network, an image noise adding network, an image denoising network, a prior feature estimation network, and a training mask inpainting network. The training prior feature extraction network and the training mask inpainting network are obtained based on the training prior feature inpainting image model. The training prior feature inpainting image model is obtained by training the prior feature inpainting image model using a training data set; the prior feature inpainting image model includes: a prior feature extraction network, a mask inpainting combined network based on U-net, and a mask inpainting network. The training data set is formed by a number of real standard images and corresponding mask images.

[0020] The specific operation process of a specific embodiment is as follows.

[0021] S1. Obtain a number of real standard images and corresponding mask images to form a training data set; use the training data set to train the prior feature inpainting image model to obtain the training prior feature inpainting image model; the prior feature inpainting image model includes: a prior feature extraction network, a mask combined repair network based on U-net, and a mask inpainting network.

[0022] Using real data to train the prior feature inpainting image model, the prior feature extraction network can extract effective features, and the mask combined repair network based on U-net and the mask inpainting network can use the mask information to specifically inpaint the image, improving the accuracy and effect of image inpainting to better restore the missing or damaged parts of the image.

[0023] Obtain a number of real standard images and corresponding mask images. Each real standard image and the corresponding mask image form a set of images, and all the image sets form the training data set, which is used as the data set for training the neural network.

[0024] Among them, the mask image is obtained by merging the real standard image and the corresponding mask graph; the mask graph is composed of pixels with values of 1 and 0, where 0 represents the masked area (usually the part to be processed or ignored), and 1 represents the non-masked area (effective area or original image).

[0025] Using the training dataset, train the prior feature image restoration model to obtain the trained prior feature image restoration model. The prior feature image restoration model includes: a prior feature extraction network, a mask combination restoration network based on U-net, and a mask restoration network.

[0026] During the process of training the prior feature image restoration model, the real standard image and the corresponding mask image are processed by the prior feature extraction network, which completely retains the most important detail parts in the image and can save computing resources, obtaining the prior feature map; the prior feature map and the mask image are processed by the mask combination restoration network based on U-net. Guided by the prior feature map, the fusion of local and global information is guided, enhancing the model's ability to perceive the missing area, and realizing image restoration of the mask image to obtain the initial restoration map; the initial restoration map and the prior feature map are processed by the mask restoration network until the first training loss value is less than the first image loss threshold, and the training ends, obtaining the prior feature restoration map.

[0027] The loss function during training is as follows: , is the first training loss value for training the prior feature image restoration model, is the real standard image, is the prior feature restoration map, is the normalization process.

[0028] The specific operations of the above prior feature extraction network processing are as follows: After splicing the real standard image and the corresponding mask image, a spliced image is obtained; the spatial information of the spliced image is rearranged to the preset channel dimension to enhance the height of the channel dimension, and the rearranged features are spliced to obtain a channel dimension enhanced map; the channel dimension enhanced map is successively processed by convolution, LReLU activation function, residual block, average pooling, linear processing, and LReLU activation function to obtain the prior feature map, which is used to perform the operations processed by the mask restoration combination network based on U-net.

[0029] The operations processed by the above-mentioned U-net-based mask combination repair network are specifically as follows: The mask image and the prior feature map are processed by the mask repair network to obtain the first mask repair feature map; the first mask repair feature map and the prior feature map are processed by the mask repair network to obtain the second mask repair feature map; the second mask repair feature map and the prior feature map are processed by the mask repair network to obtain the third mask repair feature map; the third mask repair feature map and the prior feature map are processed by the mask repair network to obtain the fourth mask repair feature map; after the fourth mask repair feature map and the third mask repair feature map are concatenated, they are processed by the mask repair network together with the prior feature map to obtain the first mask combination repair feature map; after the first mask combination repair feature map and the second mask repair feature map are concatenated, they are processed by the mask repair network together with the prior feature map to obtain the second mask combination repair feature map; after the second mask combination repair feature map and the first mask repair feature map are concatenated, they are processed by the mask repair network together with the prior feature map to obtain the initial repair map.

[0030] Taking the processing of the initial repair map and the prior feature map by the mask repair network as an example, the operations of the mask repair network are specifically as follows.

[0031] Step 1: Inject the prior feature map into the initial repair map to obtain the initial repair map injected with the prior feature map. The operation of obtaining the initial repair map injected with the prior feature map can be achieved through the following formula: , is the initial repair map injected with the prior feature map, is the prior feature map, is the mask image, or the mask repair feature map, or the mask combination repair concatenated feature map, ⊙ is element-wise multiplication, is layer normalization processing, , are the first linear weight and the second linear weight respectively.

[0032] Step 2: The initial repair map injected with the prior feature map undergoes several global and local attention fusion processes and feature aggregation processes, which can not only capture local features but also focus on global context information, thereby promoting more information to flow into the deep network to obtain the attention aggregation feature map.

[0033] Taking the first global and local attention fusion process and feature aggregation process as an example, the specific operation details are as follows.

[0034] The operation of the global and local attention fusion process is: The initial repair map injected with the prior feature map undergoes global-local self-attention processing and global region-guided attention processing to obtain the attention fusion feature map, which is used to perform the operation of the feature aggregation process.

[0035] Among them, the operation of global-local self-attention processing is as follows: The initial repair injection prior feature map is divided into horizontal windows and vertical windows. After being processed by multi-head attention of non-overlapping sub-windows respectively, all the outputs are concatenated in the channel dimension to obtain the initial self-attention map; The initial attention map and the initial repair injection prior feature map are processed by residual connection to obtain the self-attention feature map, which is used to perform the operation of global region-guided attention processing.

[0036] The operation of global region-guided attention processing is as follows.

[0037] Step a: Inject the prior feature map into the self-attention feature map (the injection process is similar to the operation of obtaining the initial repair injection prior feature map above) to obtain the self-attention injection prior feature map; Recursively use a single depthwise separable convolution to process the self-attention injection prior feature map to compress the feature space size and obtain a rough fusion map.

[0038] Among them, the number of recursions is obtained based on the size of the self-attention injection prior feature map and the convolution stride of the depthwise separable convolution processing; The number of recursions is , is the convolution stride of the depthwise separable convolution, is a constant, is the height of the self-attention injection prior feature map.

[0039] Step b: The rough fusion map is processed by a depthwise separable convolution with a convolution stride of and a pixel-wise convolution with a convolution stride of (which can be achieved by pointwise convolution) for feature refinement and channel scaling to obtain a representative feature map.

[0040] Step c: Based on the key feature matrix and value feature matrix of the representative feature map, obtain the attention matrix; Based on the attention matrix and the query matrix of the self-attention injection prior feature map, perform attention cross-processing to obtain the cross-attention feature map; After the cross-attention feature map is reshaped to reach the preset size, it is processed by residual connection with the self-attention feature map to obtain the attention fusion feature map, which is used to perform the operation of feature aggregation processing.

[0041] The operation of attention cross-processing can be realized by the following formula: , , is the attention matrix, and the sum of the elements in each row of the attention matrix is 1, 、 are the transposes of the key feature matrix and value feature matrix of the representative feature map respectively, is the adjustment factor, is the query matrix for injecting prior feature maps into self-attention, is the fusion projection matrix, is the cross-attention feature map.

[0042] The operation of feature aggregation processing is as follows: the attention fusion feature map undergoes channel feature aggregation processing to obtain an attention channel aggregation map; the attention channel aggregation map undergoes spatial neighborhood pixel feature aggregation processing to obtain an attention pixel aggregation map; the attention fusion feature map undergoes gating mechanism processing to enhance information encoding and obtain a gated aggregation map; after the gated aggregation map and the attention pixel aggregation map are element-wise multiplied, they are subjected to residual connection with the attention fusion feature map to obtain an attention aggregation feature map.

[0043] Step 3: After the attention aggregation feature map is subjected to convolution processing with a convolution stride of it is subjected to residual connection processing with the initial repair map to obtain a prior feature repair map.

[0044] S2. Using the training dataset, train the prior feature repair and enhancement image model to obtain a trained prior feature repair and enhancement image model; the prior feature repair and enhancement image model includes: a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask repair network; the trained prior feature extraction network and the trained mask repair network are obtained based on the trained prior feature repair image model.

[0045] Using the training dataset to train the prior feature repair and enhancement image model, the prior feature repair and enhancement image model includes a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask repair network, which can refine the details of image repair, enhance the image repair effect, and through multi-network cooperation, with the help of prior feature-related networks and mask repair networks, as well as operations such as image noise addition and denoising, more accurately process images and improve the quality of image repair.

[0046] Using the training dataset, train the prior feature repair and enhancement image model to obtain a trained prior feature repair and enhancement image model. The prior feature repair and enhancement image model includes: a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask repair network.

[0047] Among them, the trained prior feature extraction network and the trained mask repair network are obtained based on the trained prior feature repair image model in S1.

[0048] During the process of training the prior feature restoration and enhancement image model, the real standard image and the corresponding mask image are processed by the prior feature extraction network and the image noise addition network to obtain a noisy image; the mask image is processed by the prior feature estimation network to obtain a prior estimation feature map; the prior estimation feature map and the noisy image are processed by the image denoising network and the training mask restoration network until the enhancement image loss value is less than the enhancement image loss threshold, at which point the training ends and a prior feature restoration and enhancement map is obtained.

[0049] The enhancement image loss value is obtained through the following formula: , where is the enhancement image loss value, is the mask image,

[0050] The operation of obtaining the above prior estimation feature map specifically is as follows: The mask image is subjected to spatial information rearrangement to the preset channel dimension for processing, the channel dimension height is enhanced, and the rearranged features are subjected to a splicing operation to obtain a mask channel dimension enhanced map; the mask channel dimension enhanced map is successively subjected to convolution processing, LReLU activation function processing, residual block processing, average pooling processing, linear processing, and LReLU activation function processing to obtain the prior estimation feature map.

[0051] During the processing of the image noise addition network, it follows that: , , is the conditional probability distribution of the entire noise addition process from the initial latent variable to , is the original data sample, is the latent variable at the t th step of the noise addition process, is the latent variable at the t -1th step of the noise addition process, is the conditional Gaussian distribution of the noise addition process, is the noise variance, is the covariance structure of the noise.

[0052] During the processing of the image denoising network, it follows that: , , is the conditional probability distribution at the t th step of the denoising process, is the cumulative noise attenuation factor, is at the tThe latent variable of the step, is the original data sample, is the latent variable of the t -1 step, is the Gaussian noise mean, ( ) is T the noise attenuation coefficient at time is the amount of noise removed.

[0053] S3. Obtain the training image noise addition network, training image denoising network, training prior feature estimation network, and training mask repair network in the training prior feature repair and enhancement image model to form an image repair model; the image to be repaired is processed by the image repair model to obtain an image repair result.

[0054] Obtain the training image noise addition network, training image denoising network, training prior feature estimation network, and training mask repair network from the training prior feature repair and enhancement image model to form an image repair model. During the process of processing the image to be repaired with this model, through the collaborative action of each network, the noise addition and denoising networks process and optimize the image, the prior feature estimation network provides prior information, and the mask repair network repairs specific regions, so as to complete image repair efficiently and with high quality.

[0055] Obtain the training image noise addition network, training image denoising network, training prior feature estimation network, and training mask repair network in the training prior feature repair and enhancement image model obtained in S2 to form an image repair model; the image to be repaired is processed by the image repair model to obtain an image repair result.

[0056] During the process of processing the image to be repaired by the image repair model, the mask image of the image to be repaired is processed by the training prior feature estimation network to obtain the prior feature to be processed; the prior feature to be processed and the noise feature map in the training image noise addition network (the image noise addition network can directly extract the noise feature map according to needs after training) are processed by the training image denoising network and the training mask repair network to obtain the image repair result, see the image repair effect diagram in Figure 1 for details.

[0057] This embodiment also provides an image repair system based on diffusion mode and prior feature generation for implementing the above-mentioned image repair method based on diffusion mode and prior feature generation, including: The training prior feature inpainting image model generation module is used to train the prior feature inpainting image model by using the training data set, and obtain the trained prior feature inpainting image model; the prior feature inpainting image model includes: a prior feature extraction network, a mask inpainting combination network based on U-net, and a mask inpainting network; the training data set is formed by a number of real standard images and corresponding mask images; The training prior feature inpainting enhanced image model generation module is used to train the prior feature inpainting enhanced image model by using the training data set, and obtain the trained prior feature inpainting enhanced image model; the prior feature inpainting enhanced image model includes: a trained prior feature extraction network, an image noise adding network, an image denoising network, a prior feature estimation network, and a trained mask inpainting network; the trained prior feature extraction network and the trained mask inpainting network are obtained based on the trained prior feature inpainting image model; The image inpainting result generation module is used to process the image to be inpainted through the image inpainting model to obtain an image inpainting result; the image inpainting model includes the trained image noise adding network, the trained image denoising network, the trained prior feature estimation network, and the trained mask inpainting network in the trained prior feature inpainting enhanced image model.

[0058] This embodiment also provides an image inpainting device based on diffusion mode and prior feature generation, including a processor and a memory. When the processor executes the computer program stored in the memory, the above image inpainting method based on diffusion mode and prior feature generation is implemented.

[0059] This embodiment also provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, the above image inpainting method based on diffusion mode and prior feature generation is implemented.

[0060] An image restoration method based on diffusion mode and prior features provided by this embodiment first uses real data to train a prior feature restoration image model. The prior feature extraction network can extract effective features. The mask combination restoration network and the mask restoration network based on U-net can use the mask information to specifically restore the image, improving the accuracy and effect of image restoration to better restore the missing or damaged parts of the image. Then, a prior feature restoration enhanced image model is trained using a training data set. The prior feature restoration enhanced image model includes a trained prior feature extraction network, an image noise addition network, an image denoising network, a prior feature estimation network, and a trained mask restoration network, which can refine the details of image restoration and enhance the image restoration effect. Through the cooperation of multiple networks, with the help of prior feature related networks and mask restoration networks, as well as image noise addition and denoising operations, the image is processed more precisely to improve the quality of image restoration. Finally, an image restoration model is obtained from the trained prior feature restoration enhanced image model, which consists of a trained image noise addition network, a trained image denoising network, a trained prior feature estimation network, and a trained mask restoration network. During the process of using this model to process the image to be restored, the trained noise addition and denoising networks process and optimize the image, the trained prior feature estimation network provides prior information, and the trained mask restoration network repairs specific regions, thus efficiently and high-qualityly completing image restoration. Applied in the field of image restoration, it can improve the pertinence, image restoration effect, and image restoration efficiency of image restoration.

Claims

1. An image restoration method based on diffusion pattern and prior feature generation, characterized in that: The following operations are included: The image to be repaired is processed by the image repair model to obtain the image repair result; the image repair model includes a training image denoising network, a training image denoising network, a training prior feature estimation network and a training mask repair network in the training prior feature repair and enhancement image model; The prior feature restoration and enhancement image model is trained by using a training data set to train the prior feature restoration and enhancement image model; The prior feature restoration and enhancement image model includes: training a priori feature extraction network, image denoising network, image denoising network, prior feature estimation network and training mask restoration network; The training prior feature extraction network and the training mask restoration network are obtained based on the training prior feature restoration image model; The prior feature restoration image model is trained by using a training data set to train the prior feature restoration image model; the prior feature restoration image model includes: a prior feature extraction network, a mask restoration combination network based on U-net, and a mask restoration network; The training dataset is formed by several real standard images and corresponding mask images.

2. The image restoration method based on diffusion pattern and prior feature generation according to claim 1, characterized in that: In the process of training the prior feature restoration and enhancement image model, the real standard image and the corresponding mask image are processed by the trained prior feature extraction network and the image denoising network to obtain a noisy image; The mask image is processed by the prior feature estimation network to obtain the prior estimated feature map; the prior estimated feature map and the noise image are processed by the image denoising network and the training mask restoration network until the enhanced image loss value is less than the enhanced image loss threshold, and the training ends.

3. The image restoration method based on diffusion pattern and prior feature generation according to claim 1, characterized in that: During the processing of the prior feature extraction network, the real standard image and the corresponding mask image are spliced ​​to obtain a spliced ​​image; the spatial information of the spliced ​​image is rearranged to the preset channel dimension, and the rearranged features are spliced ​​to obtain a channel dimension enhanced map; the channel dimension enhanced map is sequentially processed by convolution, LReLU activation function, residual block, average pooling, linear processing and LReLU activation function to obtain a prior feature map, which is used to perform operations processed by the U-net-based mask repair combination network.

4. The image restoration method based on diffusion pattern and prior feature generation according to claim 1 or 3, characterized in that: The operations of the U-net-based mask combination repair network processing are as follows: The mask image and the prior feature map are processed by the mask restoration network to obtain a first mask restoration feature map; The first mask repair feature map and the prior feature map are processed by the mask repair network to obtain a second mask repair feature map; The second mask repair feature map and the prior feature map are processed by the mask repair network to obtain a third mask repair feature map; The third mask repair feature map and the prior feature map are processed by the mask repair network to obtain a fourth mask repair feature map; The fourth mask restoration feature map and the third mask restoration feature map are concatenated and processed with the prior feature map by a mask restoration network to obtain a first mask combination restoration feature map; The first mask combination repair feature map and the second mask repair feature map are concatenated and processed with the prior feature map by a mask repair network to obtain a second mask combination repair feature map; The second mask combination repair feature map and the first mask repair feature map are spliced ​​and then processed with the prior feature map by the mask repair network to obtain an initial repair map for performing mask repair network processing operations.

5. The image restoration method based on diffusion pattern and prior feature generation according to claim 1, characterized in that: During the mask repair network processing, The prior feature map is injected into the initial repair map to obtain the initial repair injection prior feature map; the initial repair injection prior feature map is subjected to several global and local attention fusion processes and feature aggregation processes to obtain an attention aggregation feature map; The attention aggregation feature map is processed by convolution and then residually connected with the mask image to obtain the prior feature restoration map; The initial inpainting map is the output of the mask combination inpainting network based on U-net.

6. The image restoration method based on diffusion pattern and prior feature generation according to claim 5, characterized in that: The operation of global and local attention fusion processing is: The initial repair injection prior feature map is processed by global local self-attention and global regional guided attention to obtain an attention fusion feature map for performing feature aggregation processing; The operation of global local self-attention processing is as follows: the initial repair injection prior feature map is divided into horizontal windows and vertical windows, and after multi-head attention processing of non-overlapping sub-windows, all outputs are spliced ​​in the channel dimension to obtain the initial self-attention map; the initial attention map and the initial repair injection prior feature map are processed by residual connection to obtain a self-attention feature map, which is used to perform the global area guided attention processing operation.

7. The image restoration method based on diffusion pattern and prior feature generation according to claim 6, characterized in that: The operation of global region guided attention processing is: Inject the prior feature map into the self-attention feature map to obtain the self-attention injected prior feature map; recursively use a single depth-wise separable convolution to process the self-attention injected prior feature map to obtain a coarse fusion map; The coarse fusion image is processed by depth-wise separable convolution and pixel-level convolution to obtain a representative feature map; Based on the key feature matrix and the value feature matrix of the representative feature map, an attention matrix is ​​obtained; Based on the attention matrix and the query matrix of the self-attention injection prior feature map, attention cross processing is performed to obtain a cross attention feature map; After the cross-attention feature map is reshaped, it is residually connected with the self-attention feature map to obtain the attention fusion feature map.

8. An image restoration system based on diffusion pattern and prior feature generation, used to implement the image restoration method based on diffusion pattern and prior feature generation according to claim 1, characterized in that: include: A training prior feature restoration image model generation module is used to train a prior feature restoration image model using a training data set to obtain a training prior feature restoration image model; The prior feature restoration image model includes: a prior feature extraction network, a U-net-based mask restoration combination network and a mask restoration network; the training data set is formed by several real standard images and corresponding mask images; The training prior feature restoration and enhancement image model generation module is used to use the training data set to train the prior feature restoration and enhancement image model to obtain the training prior feature restoration and enhancement image model; the prior feature restoration and enhancement image model includes: a training prior feature extraction network, an image denoising network, an image denoising network, a prior feature estimation network and a training mask restoration network; the training prior feature extraction network and the training mask restoration network are obtained based on the training prior feature restoration image model; The image restoration result generation module is used to process the image to be restored through the image restoration model to obtain the image restoration result; the image restoration model includes a training image denoising network, a training image denoising network, a training prior feature estimation network and a training mask restoration network in the training prior feature restoration and enhancement image model.

9. An image restoration device based on diffusion pattern and prior feature generation, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the image restoration method based on diffusion pattern and prior feature generation as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the image restoration method based on diffusion pattern and prior feature generation as described in any one of claims 1 to 7 is implemented.

Images