An image restoration method and system based on asymmetric diffusion process and structural texture consistency of diffusion model

Abstract

The present invention discloses an image restoration method and system based on an asymmetric diffusion process and structural texture consistency based on a diffusion model, which relates to the field of image processing. The method comprises the following steps: obtaining an RGB image to be restored with masking, and marking masked and unmasked areas using a binary mask; performing asymmetric diffusion processing based on the RGB image of the unmasked area to obtain a texture image and a structural image containing noise at each time step; extracting a grayscale image from the RGB image to be restored, and superimposing the grayscale image with the texture image containing Gaussian noise at the last time step as the denoising initial image for the denoising process; reconstructing a texture feature map using a pre-trained structure-guided denoising network to obtain a reconstructed image; and splicing the masked area of ​​the reconstructed image with the unmasked area of ​​the RGB image to be restored to obtain the final restored image. The present invention solves the problem of structural texture inconsistency caused by large semantic differences between masked and unmasked areas of an image, thereby improving the quality of image restoration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to an image restoration method and system based on an asymmetric diffusion process and structural texture consistency of a diffusion model. Background Art

[0002] In the field of image restoration, restoring masked areas using the semantic information of unmasked areas requires the masked area to generate reasonable semantics and maintain semantic consistency with the unmasked area. Traditional image restoration methods, such as diffusion-based and patch-based methods, may be effective when processing small masked areas or simple images, but they have limitations when dealing with irregular or complex masks, making it difficult to generate reasonable semantics and ensuring semantic consistency. While convolutional neural networks (CNNs) have improved image restoration results to some extent, their encoding focuses primarily on local semantic information around the masked area and tends to ignore global information in the unmasked area, resulting in blurred areas far from the mask boundary. The introduction of the self-attention mechanism enhances the global correlation between the masked and unmasked areas, but it also reduces the semantic correlation between different patches in the masked area.

[0003] The emergence of denoising diffusion probability models (DDPMs) has brought new breakthroughs to image restoration, achieving significant progress in semantic generation and pattern convergence. However, existing image restoration methods based on DDPMs still have problems. For example, methods such as Repaint rely heavily on resampling strategies, and when resampling strategies are not adopted, it is difficult to achieve semantic consistency between masked and unmasked areas. StrDiffusion attempts to use semantically sparse structures to solve the semantic difference problem. However, due to the sparse nature of the structure, the texture denoising speed of the unmasked area is faster than the structure denoising speed of the masked area, resulting in differences in the correlation between structure and texture, and the semantic quality of the restoration result is poor.

[0004] Therefore, how to solve the problem of large semantic differences and inconsistent structural textures between masked and unmasked areas of an image and improve the quality of image restoration is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In view of this, the present invention provides an image restoration method and system based on an asymmetric diffusion process and structural texture consistency of a diffusion model, which solves the above problems.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention discloses an image restoration method based on an asymmetric diffusion process and structural texture consistency of a diffusion model, and the specific steps are as follows:

[0008] Get the RGB image to be repaired with masking, and use a binary mask to mark the masked and unmasked areas;

[0009] Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step;

[0010] Extracting a grayscale image from the RGB image to be repaired, and superimposing the grayscale image with the texture image containing Gaussian noise at the last time step as a denoising initial image for the denoising process;

[0011] Use the pre-trained structure to guide the denoising network, perform texture feature mapping reconstruction, and obtain the reconstructed image;

[0012] The masked area of ​​the reconstructed image is spliced ​​with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image.

[0013] Furthermore, the asymmetric diffusion processing is specifically as follows: during the diffusion process, noise of different intensities is applied to the texture map and the structure for noise addition, so that more noise is added to the structure map than the texture map, and by gradually adding noise, the texture map and the structure map at each time step are obtained.

[0014] Furthermore, noise is added through the mean reversion stochastic differential formula, which is:

[0015] dy=θ t (μ y -y)dt+η t dw;

[0016] dx=δ t (μ x -x)dt+η t dw;

[0017] Among them, dy represents the small change of the current texture state y, μ y Represents the texture semantic parameters of the unmasked area; dx represents the small change in the current texture state x, μ x Represents the texture semantic parameters of the unmasked area; θ t , δ t Respectively represent the positive parameters of the mean regression speed in the texture map and structure map changing with time step t; η t represents the positive parameter of the diffusion random volatility changing with time step t; dw represents the standard Wiener process, which introduces randomness into the entire differentiation process.

[0018] Furthermore, θ t and δ t The specific calculation formula is:

[0019]

[0020] Among them, f(t) is a function of time step t, specifically:

[0021]

[0022] Where T is the total number of time steps, s = 0.008;

[0023] By introducing the adjustment parameter ω, δ t Replaced by δ t,ω , thereby adjusting the structural diffusion speed and achieving asymmetric diffusion; δ t,ω The formula is: Where ω<1.

[0024] Furthermore, the structure guides the denoising network to be a CNN convolutional network, and the mapping function of each layer in the network is:

[0025]

[0026] Where t represents the time step, x′ t-1 Represents the structural diagram of the t-1 time step; conv γ 、conv β Respectively represent x′ t-1 Perform scaled convolution and bias value convolution; represents the texture feature map at the t-th time step of the k-th layer of the CNN network, Represents the reconstructed texture feature mapping result; They represent the statistical mean and variance of different pixel channels in the height and width directions at the t-th time step of the k-th layer of the CNN network, respectively. The specific formula is:

[0027]

[0028] Among them, (h k ,w k ,c k ) indicates that the input feature map of the k-th layer of the CNN network is at height h k 、Width w k and channel c k The eigenvalue on h k =1,2,...,H k ,w k =1,2,...,W k , c k =1,2,...,C k , H k 、W k 、C k are the feature map height, width, and total number of channels, respectively.

[0029] Furthermore, the structure guides the denoising network to adopt a multi-scale spatial adaptive normalization strategy.

[0030] The present invention also discloses an image restoration system based on an asymmetric diffusion process and structural texture consistency of a diffusion model, comprising:

[0031] Image marking module: obtains the RGB image to be repaired with masking, and uses binary masks to mark the masked and unmasked areas;

[0032] Asymmetric diffusion module: Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step;

[0033] Overlay module: extracting a grayscale image from the RGB image to be repaired, and overlaying the grayscale image with the texture image containing Gaussian noise at the last time step as the denoising initial image in the denoising process;

[0034] Denoising and reconstruction module: uses the pre-trained structure to guide the denoising network, reconstructs the texture feature map, and obtains the reconstructed image;

[0035] Stitching and repair module: stitching the masked area of ​​the reconstructed image with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image.

[0036] Through the above technical solutions, it can be seen that compared with the prior art, the present invention discloses an image restoration method and system based on an asymmetric diffusion process and structural texture consistency of a diffusion model. Through the asymmetric diffusion process and structural semantic enhancement strategy, the semantic difference between the masked area and the unmasked area is effectively reduced, making the restored image more semantically coherent and reasonable; in the denoising process, consistent structural texture correlation is achieved between the masked area and the unmasked area, avoiding the correlation difference problem caused by inconsistent structural and texture denoising speeds, and improving the quality of the restored image; extracting a grayscale image from the texture RGB image and superimposing it with the texture image containing Gaussian noise at the last time step as the denoising initial image of the denoising process, and combining it with the optimization training of the structure-guided denoising network, the structural semantics is enhanced, so that the restoration result has higher semantic quality and is more consistent with the characteristics of the real image. The present invention solves the problem of large semantic differences and structural texture inconsistency in the masked and unmasked areas of the image, and improves the quality of image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 Schematic diagram of the overall process of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] The embodiment of the present invention discloses an image restoration method based on an asymmetric diffusion process and structural texture consistency of a diffusion model. Figure 1 The specific steps are as follows:

[0041] Get the RGB image to be repaired with masking, and use a binary mask to mark the masked and unmasked areas;

[0042] Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step;

[0043] Extract a grayscale image from the RGB image to be repaired, and superimpose the grayscale image with the texture image containing Gaussian noise at the last time step as the denoising initial image in the denoising process;

[0044] Use the pre-trained structure to guide the denoising network, perform texture feature mapping reconstruction, and obtain the reconstructed image;

[0045] The masked area of ​​the reconstructed image is spliced ​​with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image.

[0046] In a specific embodiment, the asymmetric diffusion processing is specifically as follows: during the diffusion process, noise of different intensities is applied to the texture map and the structure map respectively to add noise, so that more noise is added to the structure map than the texture map, and by gradually adding noise, the texture map and structure map at each time step are obtained.

[0047] In a specific embodiment, noise is added by using a mean reversion stochastic differential formula, which is:

[0048] dy=θt (μ y -y)dt+η t dw;

[0049] dx=δ t (μ x -x)dt+η t dw;

[0050] Among them, dy represents the small change of the current texture state y, μ y Represents the texture semantic parameters of the unmasked area; dx represents the small change in the current texture state x, μ x Represents the texture semantic parameters of the unmasked area; θ t , δ t Respectively represent the positive parameters of the mean regression speed in the texture map and structure map changing with time step t; η t represents the positive parameter of the diffusion random volatility changing with time step t; dw represents the standard Wiener process, which introduces randomness into the entire differentiation process.

[0051] In a specific embodiment, θ t and δ t The specific calculation formula is:

[0052]

[0053] Among them, f(t) is a function of time step t, specifically:

[0054]

[0055] Where T is the total number of time steps, s = 0.008;

[0056] By introducing the adjustment parameter ω, δ t Replaced by δ t,ω , thereby adjusting the structural diffusion speed and achieving asymmetric diffusion; δ t,ω The formula is: Where ω<1.

[0057] Specifically, the image restoration method of the present invention mainly includes a diffusion section and a noise reduction section. There are two key bottlenecks in the denoising process: the weakness of structural semantics, and the inconsistent structural texture correlation between masked and non-masked areas in the early and late stages. In order to solve the above problems, a noise addition method that applies asymmetric noise intensity to the structure and texture is adopted in the diffusion process to ensure that the denoising rhythm of the two can be consistent during the denoising process. In the diffusion section, different noise intensities are used in the structure and texture diffusion process to obtain the structure map and texture map. By introducing adjustment parameters, the diffusion speed of the structure map and texture map is adjusted. This strategy is called an asymmetric diffusion strategy. Where ω is less than 1, so that in the asymmetric diffusion process, more noise is added to the structure.

[0058] In a specific embodiment, the structure-guided denoising network is a CNN convolutional network, and the mapping function of each layer in the network is:

[0059]

[0060] Where t represents the time step, x′ t-1 Represents the structural diagram of the t-1 time step; conv γ 、conv β Respectively represent x′ t-1 Perform scaled convolution and bias value convolution; represents the texture feature map at the t-th time step of the k-th layer of the CNN network, Represents the reconstructed texture feature mapping result; They represent the statistical mean and variance of different pixel channels in the height and width directions at the t-th time step of the k-th layer of the CNN network, respectively. The specific formula is:

[0061]

[0062]

[0063] Among them, (h k ,w k ,c k ) indicates that the input feature map of the k-th layer of the CNN network is at height h k 、Width w k and channel c k The eigenvalue on h k =1,2,...,H k ,w k =1,2,...,W k , c k =1,2,...,C k , H k 、W k 、Ck are the feature map height, width, and total number of channels, respectively.

[0064] Specifically, the goal of the structure-guided denoising process is to train a structure-guided denoising network, using a fixed noise variance to predict noise. The structure-guided denoising network is pre-trained as an auxiliary tool to obtain the structure after denoising. By adding simulated texture and structural noise to a known image, the noisy texture map, structure map, and original image are used as a sample for training. This allows the convolutional network to learn the mapping relationship between texture features and noise mapping of the original image at each layer. Each layer can also reconstruct the mapped feature map based on the structure map, resulting in a reconstructed image with consistent texture and structure.

[0065] In a specific embodiment, the structure-guided denoising network adopts a multi-scale spatial adaptive normalization strategy.

[0066] The embodiment of the present invention further discloses an image restoration system based on an asymmetric diffusion process and structural texture consistency of a diffusion model, comprising:

[0067] Image marking module: obtains the RGB image to be repaired with masking, and uses binary masks to mark the masked and unmasked areas;

[0068] Asymmetric diffusion module: Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step;

[0069] Overlay module: extracts a grayscale image from the RGB image to be repaired, and overlays the grayscale image with the texture image containing Gaussian noise at the last time step as the denoising initial image for the denoising process;

[0070] Denoising and reconstruction module: uses the pre-trained structure to guide the denoising network, reconstructs the texture feature map, and obtains the reconstructed image;

[0071] Stitching and repair module: stitching the masked area of ​​the reconstructed image with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0073] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image restoration method based on asymmetric diffusion process and structural texture consistency of diffusion model, characterized by: The specific steps are as follows: Get the RGB image to be repaired with masking, and use a binary mask to mark the masked and unmasked areas; Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step; Extracting a grayscale image from the RGB image to be repaired, and superimposing the grayscale image with the texture image containing Gaussian noise at the last time step as a denoising initial image for the denoising process; Use the pre-trained structure to guide the denoising network, perform texture feature mapping reconstruction, and obtain the reconstructed image; The masked area of ​​the reconstructed image is spliced ​​with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image; The asymmetric diffusion processing specifically includes: applying noise of different intensities to the texture map and the structure map respectively during the diffusion process to add noise, so that the structure map has more noise than the texture map, and obtaining the texture map and the structure map at each time step by gradually adding noise; Noise is added through the mean reversion stochastic differential formula, the formula is: dy=θ t (m y -y)dt+η t dw; dx=δ t (m x -x)dt+η t dw; Among them, dy represents the small change of the current texture state y, μ y Represents the texture semantic parameters of the unmasked area; dx represents the small change in the current texture state x, μ x Represents the texture semantic parameters of the unmasked area; θ t , δ t Respectively represent the positive parameters of the mean regression speed in the texture map and structure map changing with time step t; η t represents the positive parameter of the diffusion random volatility changing with time step t; dw represents the standard Wiener process, which introduces randomness into the entire differentiation process; θ t and δ t The specific calculation formula is: Among them, f(t) is a function of time step t, specifically: Where T is the total number of time steps, s = 0.008; By introducing the adjustment parameter ω, δ t Replaced by δ t,ω , thereby adjusting the structural diffusion speed and achieving asymmetric diffusion; δ t,ω The formula is: Where ω<1.

2. The image restoration method based on asymmetric diffusion process and structural texture consistency of a diffusion model according to claim 1 is characterized in that: The structure guides the denoising network to be a CNN convolutional network, and the mapping function of each layer in the network is: Where t represents the time step, x′ t-1 Represents the structural diagram of the t-1 time step; conv γ 、conv β Respectively represent x′ t-1 Perform scaled convolution and bias value convolution; represents the texture feature map at the t-th time step of the k-th layer of the CNN network, Represents the reconstructed texture feature mapping result; They represent the statistical mean and variance of different pixel channels in the height and width directions at the t-th time step of the k-th layer of the CNN network, respectively. The specific formula is: Among them, (h k ,w k ,c k ) indicates that the input feature map of the k-th layer of the CNN network is at height h k 、Width w k and channel c k The eigenvalue on h k =1,2,...,H k ,w k =1,2,...,W k , c k =1,2,...,C k , H k 、W k 、C k are the feature map height, width, and total number of channels, respectively.

3. The image restoration method based on asymmetric diffusion process and structural texture consistency of a diffusion model according to claim 1 is characterized in that: The structure guides the denoising network to adopt a multi-scale spatial adaptive normalization strategy.

4. An image restoration system based on an asymmetric diffusion process and structural texture consistency of a diffusion model, using an image restoration method based on an asymmetric diffusion process and structural texture consistency of a diffusion model according to any one of claims 1 to 3, characterized in that: include: Image marking module: obtains the RGB image to be repaired with masking, and uses binary masks to mark the masked and unmasked areas; Asymmetric diffusion module: Based on the RGB image of the unmasked area, asymmetric diffusion processing is performed to obtain the texture image and structure image containing noise at each time step; Overlay module: extracting a grayscale image from the RGB image to be repaired, and overlaying the grayscale image with the texture image containing Gaussian noise at the last time step as the denoising initial image in the denoising process; Denoising and reconstruction module: uses the pre-trained structure to guide the denoising network, reconstructs the texture feature map, and obtains the reconstructed image; Stitching and repair module: stitching the masked area of ​​the reconstructed image with the unmasked area of ​​the RGB image to be repaired to obtain the final repaired image.

Images