Secondary reconstruction image generation method based on conditional diffusion model

Abstract

The invention relates to the technical field of image generation, and discloses a secondary image reconstruction generation method based on a conditional diffusion model, which comprises the following steps of: 1, acquiring a data set, and classifying images of the data set into a target image y, an original image x and a test image z; 2, training a reconstructed subnet based on the target image y; 3, performing secondary reconstruction on the test image z; according to the method, on the basis of the conditional diffusion model, the problem of low diversity caused by low-temperature sampling is solved by embedding the target image and fusing the style loss function in the training process and cooperating with a secondary reconstruction method, so that a high-quality image similar to a target style is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a secondary reconstruction image generation method based on a conditional diffusion model. Background Art

[0002] Image generation technology is a hot research topic in artificial intelligence, with diffusion models attracting significant attention for their ability to generate high-quality and diverse images. While traditional diffusion models can generate images, they lack directionality and cannot be customized to meet user needs. To address this issue, conditional diffusion models have emerged. These models incorporate conditional inputs (such as text descriptions and category labels) into the diffusion process, guiding the image toward a direction that meets the conditions, thereby achieving directed generation. In conditional diffusion models, the inverse generation process relies not only on noisy images but also incorporates conditional inputs to ensure that the generated image meets the conditions. In text-to-image generation tasks, conditional diffusion models can generate images corresponding to a given text description. This capability holds broad application prospects in areas such as image creation, game design, and virtual reality. Despite significant progress in image generation, conditional diffusion models still face challenges, such as improving the speed and efficiency of sample generation and better handling complex image generation tasks. Therefore, developing more efficient and accurate image sample generation methods based on conditional diffusion models is crucial for advancing image generation technology.

[0003] The current generation methods based on the conditional diffusion model include: Invention Patent Application Number: 202411174111.4, Name: An image denoising method based on the conditional diffusion model, which uses edge information corresponding to texture to guide the diffusion model to perform generation tasks. Due to the limitations of edge information, the diversity of generation is greatly limited, resulting in low-temperature sampling; Invention Patent Application Number: 202411005676.X, Name: A robustness optimization method for the conditional diffusion model, which uses conditional and unconditional neural networks for joint training, but the added target image will make the model tend to the target image, and it is impossible to balance diversity and high quality; Invention Patent Application Number: 202411023647.6, Name: An image stylization method based on the diffusion model, which uses DDIM and implicit diffusion model to coordinate text guidance to generate images, but the time and space costs are high; in addition, the model will also be affected by factors such as the original noise and brightness in the image, further reducing the generation effect.

[0004] Therefore, it is necessary to develop a new generation method that can overcome the low diversity problem caused by low-temperature sampling and consume less resources to generate high-quality images. Summary of the Invention

[0005] To overcome the deficiencies in the prior art, the present invention aims to provide a secondary reconstruction image generation method based on a conditional diffusion model, which does not require a classifier and can generate high-quality images by guiding the diffusion model according to the style of the target image.

[0006] The present invention comprises the steps of:

[0007] A method for generating a secondary reconstructed image based on a conditional diffusion model comprises the following steps:

[0008] Step 1: Get the dataset and classify it into target image y, original image x, and test image z;

[0009] Step 2: Train the reconstruction subnetwork based on the target image y;

[0010] Step 2.1: Obtain the feature extraction network wide_Resnet50, resize all image pixels to 256×256, set the base channel to 64, set the number of attention heads to 4, and use the Adam optimizer for optimization with an initial learning rate of 6e-4;

[0011] Step 2.2: Extract image x and target image y from the training set, and perturb image x with noise. The variance table predefined by the noise addition is β1<β2<…<β t , where β t ∈(0,1), the total time step is T, and step 2.2 is performed once at each time step t to obtain a new noise ε t , then add noise to the image x according to formula (1) to obtain the noise image x t ;

[0012]

[0013] Step 2.3: Construct the implicit classifier gradient according to formula (2):

[0014]

[0015] Among them, p(y|x t ) represents the sample x t The probability of belonging to the target image y, p(x t |y) means that under the condition of target image y, sample x t The generation probability, p(x t ) represents the sample x under global conditions t The generation probability of θ The implicit classifier gradient is expressed as formula (3):

[0016]

[0017] in, represents the cumulative noise attenuation coefficient during the diffusion process, ε θ (x t ,t,y) represents the noise prediction model for x under category y t Noise prediction, ε θ (x t ,t) represents the unconditional t Noise prediction;

[0018] Step 2.4: Embed the implicit classifier gradient into the noisy prediction, as shown in formula (4):

[0019]

[0020] Among them, w represents the guidance strength, which controls the influence of the classifier guidance. represents the noise prediction value after adding the implicit classifier guidance. The final noise prediction is a weighted linear combination of the conditional noise prediction and the unconditional noise prediction;

[0021] Step 2.5: Transform the style loss function L style Perform weighted fusion with the noise prediction function according to formula (5);

[0022] Loss=η t *L style +(1-η t )*L noise (5)

[0023] Among them, Loss represents the overall loss function, L style Represents the style loss function, specifically the square difference between the Gram matrix of the feature vector of sample x in the L layer and the Gram matrix of the target image y in the same layer, L noise represents the square difference between the predicted noise and the actual noise, η t is the weight, which decreases as t increases, specifically as formula (6):

[0024]

[0025] Step 2.6: Train the noise prediction network and repeat steps 2.1 to 2.5 until the overall loss function converges.

[0026] Step 3: Perform secondary reconstruction on the test image z. The specific steps are as follows:

[0027] Step 3.1: Input the test image z into the trained reconstruction network, reset the denoising step size T, set α = 1-β, and randomly use a target style image y to perform the first reconstruction according to formula (7);

[0028]

[0029] Among them, z t represents the image after noise disturbance, represents the reconstructed image at time step t, Represents the image y after pseudo noise processing, the purpose is to ensure that Have the same signal-to-noise ratio;

[0030] Step 3.2: The guide image y will always have a certain gap with the image z in terms of features. After a complete back propagation, a result similar to y will be obtained. Therefore, a critical point is selected to end the reconstruction. The critical point is calculated according to formula (8);

[0031]

[0032] Step 3.3: The result after the first reconstruction is Perform the second reconstruction according to formula (9);

[0033]

[0034] After the secondary reconstruction is completed, the result image x0 is obtained.

[0035] The design idea of ​​the present invention is as follows: first, an implicit classifier is constructed and embedded into the noise prediction, and the conditional diffusion model is used for training. Then, the trained conditional diffusion model is used to reconstruct the original image according to the target image to generate a new image with the target style.

[0036] By adopting the above technology, compared with the existing technology, the beneficial effects of the present invention are as follows: based on the conditional diffusion model, the present invention overcomes the low diversity problem caused by low-temperature sampling by embedding the target image and fusing the style loss function during the training process, and coordinates the secondary reconstruction method, thereby generating high-quality images with similar target style. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Original images selected for the examples;

[0038] Figure 2 Target map selected for the embodiment;

[0039] Figure 3 Test images selected for the embodiment;

[0040] Figure 4 Noise graph selected for the embodiment;

[0041] Figure 5 Result plots generated for the described method. DETAILED DESCRIPTION

[0042] The specific implementation of the secondary reconstruction image generation method based on the conditional diffusion model of the present invention is described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The secondary reconstruction image generation method based on the conditional diffusion model of the present invention specifically includes the following steps:

[0044] Step 1: Use the public texture dataset DTD and classify it into the original image x as Figure 1 As shown, the target image y is Figure 2 As shown and the test image z is Figure 3 As shown;

[0045] Step 2: Train the reconstruction subnetwork based on the target image y;

[0046] Step 2.1: Obtain the feature extraction network wide_Resnet50, resize all image pixels to 256×256, set the base channel to 64, set the number of attention heads to 4, and use the Adam optimizer for optimization with an initial learning rate of 6e-4;

[0047] Step 2.2: Extract the original image x and the target image y from the training set, and use noise to disturb the image x (add noise). The variance table predefined by the noise addition is β1<β2<…<β t , where β t = 0.01, the total time step T is set to 500, and the operation of step 2.2 is performed once at each time step t to obtain a new noise ε t , then add noise to the image x according to formula (1) to obtain the noise image x t , the noise map sample is Figure 4 ;

[0048]

[0049] Step 2.3: Construct the implicit classifier gradient according to formula (2):

[0050]

[0051] Among them, p(y|x t ) means that in sample x t The probability of belonging to the target image y, p(x t |y) means that under the condition of target image y, sample x t The generation probability, p(x t ) represents the sample x under global conditions t The generation probability of θ The implicit classifier gradient is expressed as formula (2):

[0052]

[0053] in, represents the cumulative noise attenuation coefficient during the diffusion process, ε θ (x t ,t,y) represents the noise prediction model for x under category y t Noise prediction, ε θ (x t ,t) represents the unconditional t Noise prediction;

[0054] Step 2.4: Embed the implicit classifier gradient into the noisy prediction, as shown in formula (4):

[0055]

[0056] Where w is the guidance strength, which controls the influence of the classifier guidance. To add the noise prediction value after the implicit classifier is guided, the final noise prediction is a weighted linear combination of the conditional noise prediction and the unconditional noise prediction;

[0057] Step 2.5: Transform the style loss function L style Perform weighted fusion with the noise prediction function according to formula (5);

[0058] Loss=η t *L style +(1-η t )*L noise (5)

[0059] Among them, Loss is the overall loss function, L style is the style loss function, specifically the square difference between the Gram matrix of the feature vector of sample x in the L layer and the Gram matrix of the target image y in the same layer, L noise is the square difference between the predicted noise and the actual noise, η t is the weight, which decreases as t increases, specifically as formula (6):

[0060]

[0061] Step 2.6: Train the noise prediction network and repeat steps 2.1 to 2.5 until the overall loss function converges.

[0062] Step 3: Select the test image z for secondary reconstruction. The specific steps are as follows:

[0063] Step 3.1: Input the test image z into the trained reconstruction network, reset the denoising step size T to 1000, set α = 1-β, and randomly use a target style image y to perform the first reconstruction according to formula (7);

[0064]

[0065] Among them, the target image y is as follows Figure 2 As shown, z t is the image disturbed by noise, is the image reconstructed at time step t, is the image y after pseudo noise processing, the purpose is to ensure With the same signal-to-noise ratio, V as the bootstrap level parameter is set to 0.2 in this instance;

[0066] Step 3.2: The guide image y will always have a certain gap with the image z in terms of features. After a complete back-propagation, a result similar to y may be obtained. Therefore, we select a critical point to end this reconstruction. The critical point is calculated according to formula (8);

[0067]

[0068] Step 3.3: The result after the first reconstruction is Perform the second reconstruction according to formula (9);

[0069]

[0070] After the secondary reconstruction, the result image x0 is obtained. The result image is as follows Figure 5 The generation performance of the model on the Place365 and DTD datasets is shown in Table 1:

[0071] Table 1 Quantitative generation performance and comparison of this scheme

[0072]

[0073]

[0074] Among them, ↓ represents the lower the better, ↑ represents the higher the better, and "-" means that the method has no module for measuring the generation speed. Quantitative experiments show that this method reaches the SOTA level in the SIFID index of the two datasets, and the single image generation quality is higher than that of other models. It achieves the best LPIPS index on the DTD dataset, has outstanding performance in local details and perceptual quality, and has the best generation speed.

[0075] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A secondary image reconstruction generation method based on a conditional diffusion model, characterized in that: The steps include: Step 1: Obtain the public dataset DTD and classify the dataset images into target images y, original images x, and test images z; Step 2: Train the reconstruction subnetwork based on the target image y; Step 3: Perform secondary reconstruction on the test image z.

2. The secondary image reconstruction generation method based on the conditional diffusion model according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 2.1: Get the feature extraction network, adjust the image pixels, set the base channel, set the number of attention heads, select the corresponding optimizer for optimization, and set the initial learning rate; Step 2.2: Extract the original image x and the target image y from the dataset, and add noise to the image x. The variance table predefined for the noise addition is β1<β2<…<β t , where β t ∈(0,1), the total time step is T, and the operation of step 2.2 is performed once at each time step t to obtain a new noise εt. Then, according to formula (1), the image x is denoised to obtain the noise image x t ; Step 2.3: Construct the implicit classifier gradient according to formula (2): Among them, p(y|x t ) represents the sample x t The probability of belonging to the target image y, p(x t |y) means that under the condition of target image y, sample x t The generation probability, p(x t ) represents the sample x under global conditions t The generation probability of θ The implicit classifier gradient is expressed as formula (3): in, represents the cumulative noise attenuation coefficient during the diffusion process, ε θ (x t ,t,y) represents the noise prediction model for x under category y t Noise prediction, ε θ (x t ,t) represents the unconditional t Noise prediction; Step 2.4: Embed the implicit classifier gradient into the noisy prediction, as shown in formula (4): Among them, w represents the guidance strength, which controls the influence of the classifier guidance. represents the noise prediction value after adding the implicit classifier guidance. The final noise prediction is a weighted linear combination of the conditional noise prediction and the unconditional noise prediction; Step 2.5: Transform the style loss function L style Perform weighted fusion with the noise prediction function according to formula (5); Loss=η t *L style +(1-th t )*L noise (5) Among them, Loss represents the overall loss function, L style Represents the style loss function, specifically the sample x t The square difference between the Gram matrix of the feature vector at layer L and the Gram matrix of the target image y at the same layer, L noise represents the square difference between the predicted noise and the actual noise, η t is the weight, which decreases as t increases, specifically as formula (6): Step 2.6: Train the noise prediction network and repeat steps 2.1 to 2.5 until the overall loss function converges.

3. The secondary image reconstruction generation method based on the conditional diffusion model according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Input the test image z into the trained reconstruction subnet, reset the denoising step size T, set α = 1-β, and randomly use a target image y to perform the first reconstruction according to formula (7); Among them, z t represents the image after noise disturbance, represents the reconstructed image at time step t, Represents the target image y after pseudo noise processing, the purpose is to ensure Have the same signal-to-noise ratio; Step 3.2: Select a critical point to end the first reconstruction. The critical point is calculated according to formula (8); in, are all forward process parameters of the diffusion model, which are obtained by summing the probability formula; V is the parameter that controls the guidance level; Step 3.3: The result after the first reconstruction is Perform the second reconstruction according to formula (9); After the secondary reconstruction is completed, the result image x0 is obtained; in, is the updated predicted image z t Noise on ε(z t ) is the image z predicted by the model for the tth time t The noise on This is part of the boot item replaced for the second reconstruction. Since the first reconstruction went through s steps, in order to ensure the integrity of the reconstruction process, the original t is replaced with ts and y is replaced with This ensures that the generated image can retain the global features of the original image and reduce feature loss caused by style changes.

4. The secondary image reconstruction generation method based on the conditional diffusion model according to claim 2, characterized in that: In step 2.1: the feature extraction network is wide_Resnet50, all image pixels are resized to 256×256, the basic channel is set to 64, the number of attention heads is set to 4, and the Adam optimizer is used for optimization with an initial learning rate of 6e-4.

Images