A small sample image generation method and system based on a diffusion model

Abstract

The application discloses a kind of based on diffusion model small sample image generation method and system, wherein method includes the following steps: obtaining target image, establishing auxiliary data set;Noise is added to auxiliary data set multiple times to obtain noise image satisfying isotropic Gaussian distribution;Noise image is denoised inference, to gradually remove the noise of intermediate hidden code with original image as training target training model;According to the pixel mean square error between original image and generated image and frequency domain information loss adjustment model parameter, complex frequency is highlighted learning by the dynamic weighting of frequency domain loss;With the sample of the few new data set as reference image and respectively extract its semantic and structural features to assist denoising model to generate the same kind of image of this sample image.The application generates small sample image using diffusion model, generates image from noise, improves the diversity of generated image, and ensures the high quality of generated image by additional information guidance to generation process.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a method and system for generating few-sample images based on a diffusion model. Background Technology

[0002] Deep learning is currently a cutting-edge solution in the field of computer vision, but a large amount of high-quality training data is fundamental to its success in solving computer vision problems. In many professional fields, collecting image datasets is an extremely time-consuming and expensive process, and the lack of sufficient image data is a prominent issue.

[0003] To alleviate the limitations imposed by the extremely limited amount of data on the development of deep learning, data augmentation can be used to expand the dataset. Among these methods, generative models are more realistic and reliable in generating images of the same category. Few-sample image generation refers to the task of generating realistic and diverse new images for a given category when only a very small number of images are available.

[0004] Most existing few-shot image generation methods are based on Generative Adversarial Networks (GANs), typically employing fusion-based strategies at the image or feature level to generate new images. MatchingGAN matches random vectors with a given real image and maps the fused features to the new image; F2GAN further improves MatchingGAN through fusion and padding paradigms, but its generation results are still unsatisfactory; LofGAN proposes local fusion of image features at the feature level, while adding local reconstruction loss to further improve the quality of the generated images, but it struggles to perform reasonable feature fusion for slightly complex images, easily generating meaningless images; WaveGAN adds low-frequency and high-frequency skip connections to the generator, providing more perceptual information and achieving good generation quality, but it is prone to aliasing artifacts. All of these methods are based on GANs, and when the amount of data is limited, they all suffer from problems such as difficulty in adjusting training parameters and insufficient diversity of generated samples. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for generating small-sample images based on a diffusion model, so as to generate high-quality, diverse and similar images from a small number of samples.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A few-sample image generation method based on a diffusion model includes the following steps:

[0008] Acquire the target image and create an auxiliary dataset;

[0009] Noise was added to the auxiliary dataset multiple times to obtain a noisy image that satisfies an isotropic Gaussian distribution;

[0010] Denoising inference is performed on noisy images, and the diffusion model is trained using the original image as the training target to gradually remove noise from the intermediate hidden code.

[0011] The model parameters are adjusted based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information. Complex frequencies are learned by dynamically weighting the frequency domain loss.

[0012] Using a small number of samples from a new dataset as reference images, and extracting their semantic and structural features, we can assist the denoising model in generating similar images to the sample images of the new dataset.

[0013] Furthermore, the acquisition of the target image and the establishment of the auxiliary dataset include:

[0014] Images of the target are captured using image acquisition equipment and used as supplementary dataset images.

[0015] The auxiliary dataset is expanded by performing rotation, brightness adjustment, and contrast adjustment operations on the images in the auxiliary dataset.

[0016] Furthermore, the process of adding noise multiple times to the auxiliary dataset to obtain a noisy image that satisfies an isotropic Gaussian distribution is expressed as follows:

[0017]

[0018] Where X0 is the original image, X t This is the image after adding noise, where t is the time step and β is the value. t Let be the variance used in the t-th noise addition. via β t α was calculated t =1-β t ,

[0019] Furthermore, the denoising inference of the noisy image, using the original image as the training target to train the model to gradually remove noise in the intermediate hidden code, includes:

[0020] The noise reduction process is expressed as follows:

[0021]

[0022] Where X t This is the image with added noise, X t-1 It is X tThe intermediate result obtained after one denoising step is t, which is the time step. During the denoising process, the dimension of the intermediate result remains unchanged, and the original image is used as the target for denoising the noisy image.

[0023] Furthermore, the model parameters are adjusted based on the pixel mean square error between the original and generated images and the loss of frequency domain information. Complex frequencies are learned primarily through dynamic weighting of the frequency domain loss, including:

[0024] The expression for mean square error is:

[0025]

[0026] Where X0 is the original image, X θ (X t (t) is the image predicted by the model, where t is the time step and β is the value of β. t Let be the variance used in the t-th noise addition. and Through β t The calculation is as follows: α t =1-β t ,

[0027] Furthermore, the expression for frequency domain information loss is:

[0028]

[0029] The image size is M×N, and X0(u, v) is the spatial frequency value of the original image in the spectral coordinate system. θ (u, v) represents the spatial frequency value of the predicted image in the spectral coordinate system, and w(u, v) is the spectral weight matrix of the image. The value of w(u, v) is dynamically determined based on the current loss of each frequency during training. Low weights are assigned to simple frequencies, and high weights to complex frequencies, thus shifting the training focus in the frequency domain towards complex frequencies. The overall objective function L is L0. mse With L fre sum.

[0030] Furthermore, the image is sampled from the conditional distribution p(X0|C), and the sampling process is represented as follows:

[0031] p θ (X0|C)=∫p θ (X 0：T |C)dX 1：T

[0032]

[0033] Where C represents the sampling condition.

[0034] Furthermore, the step of using a small number of samples from a new dataset as reference images and extracting their semantic and structural features to assist the denoising model in generating similar images of those sample images includes:

[0035] After adding noise to the reference image for t-1 steps, upsampling and downsampling are performed by φ. C Acquire perceptual information C from the reference image ref Through C ref Guided image X t Denoising generated image X t-1 The process generates an image related to the content of the reference image;

[0036] After adding noise to the reference image for t-1 steps, a high-pass filter φ is applied. s Obtain structural information S from the reference image ref Through S ref Guided image X t Denoising generated image X t-1 The process generates an image with a structure similar to that of a reference image.

[0037] With C ref and S ref As sampling condition C.

[0038] Furthermore, the new dataset refers to a set of image samples that did not appear in the training set. During the process of sampling images by the diffusion model, samples from the new dataset are selected sequentially as reference images for the denoising process, and the information contained in the reference images is used as a conditional constraint for denoising.

[0039] The system for implementing the few-sample image generation method based on a diffusion model includes:

[0040] The first module is used to acquire the target image and establish an auxiliary dataset.

[0041] The second module is used to add noise to the auxiliary dataset multiple times to obtain a noisy image that satisfies an isotropic Gaussian distribution;

[0042] The third module is used to perform denoising inference on noisy images, using the original image as the training target to train the model and gradually remove the noise in the intermediate hidden code;

[0043] The fourth module is used to adjust the model parameters based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information. It focuses on learning complex frequencies through dynamic weighting of frequency domain loss.

[0044] The fifth module uses a small number of samples from a new dataset as reference images and extracts their semantic and structural features to assist the denoising model in generating similar images to the sample images.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention learns a parameterized Markov chain during the model training stage, avoiding the mode collapse problem that may be caused by adversarial training. It is more stable and the parameters are easier to adjust than generative adversarial networks. The generation process uses noise as the initial sample to gradually remove noise. The content and structure of the generated image are adjusted by the low-frequency and high-frequency information of the reference image, respectively, ensuring the high quality and diversity of the generated image. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the implementation of a few-sample image generation method based on a diffusion model in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the overall structure of a few-sample image generation method based on a diffusion model in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of spatial domain constraints and frequency domain constraints in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Example 1

[0052] like Figure 1 and Figure 2 As shown in the figure, this invention discloses a few-sample image generation method based on a diffusion model, including the following steps:

[0053] S1. Obtain the target image and establish an auxiliary dataset.

[0054] Images are captured by an image acquisition device (such as a camera) to image a small number of targets (such as defective industrial parts). The resulting images are first augmented using traditional data augmentation methods such as rotation and brightness adjustment, which do not change the image category label. Then, the images are sorted by category and used as an auxiliary dataset.

[0055] S2. Add noise to the auxiliary dataset multiple times to obtain a noisy image that satisfies an isotropic Gaussian distribution.

[0056] Using an auxiliary dataset as the training set, all images in the training set must undergo the forward pass of the diffusion model. The images in the auxiliary dataset are data points X0, X0~q(X) sampled from the real data distribution. During the forward pass of the diffusion model, Gaussian noise is added to the sampled data T times consecutively, resulting in a series of noise samples X1, X2…X… T X T The image is the final result after adding noise T times, where the noise step size is determined by the variance. Control. The expression for the noise-adding process is:

[0057]

[0058] Where X0 is the original image, X t This is the image after adding noise, where t is the time step and β is the value. t Let be the variance used in the t-th noise addition. Through β t The calculation is as follows: α t =1-β t ,

[0059] X t It can be represented as a linear combination of X0 and the noise ∈ added at each time step:

[0060]

[0061] During the noise addition process, as t increases, X t It gradually loses its discernible features and becomes increasingly close to pure noise. As T→∞, X T It approximates an isotropic Gaussian distribution.

[0062] S3. Denoising the noisy image and using the original image as the training target to train the model to gradually remove the noise in the intermediate hidden code.

[0063] Denoising inference aims to extract noise from a noisy image X. T Reconstruct the original image X0. Directly predict the inverse process q(X) of the diffusion process. t-1 |X t Learning the parameterized Gaussian transform p is difficult, therefore the diffusion model learns the parameterized Gaussian transform p. θ (X t-1 |X t The denoising process and the noise-adding process have the same functional form, which can be expressed as:

[0064]

[0065] μ θ Represented as X tThe denoising process, which combines the predicted values ​​of the original image with the linear combination of the predicted values, is as follows:

[0066]

[0067] Where X θ X represents a Unet network with the same input and output dimensions. θ Used to predict the generation result of each step in the reverse diffusion process; z ~ N(0, I) indicates that there is randomness in each generation step, which is beneficial to the diversification of generated images.

[0068] S4. Adjust the model parameters based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information, and focus on learning complex frequencies through dynamic weighting of frequency domain loss.

[0069] See Figure 3 During the training of the diffusion model, the parameters of the diffusion model are optimized by spatial and frequency domain constraints between the original real image and the image generated by the diffusion model. The mean square error L between the original image and the predicted image... mse Limit the pixel loss in the spatial domain of the generated image, while utilizing the frequency domain information difference L between the original image and the predicted image. fre Supplementing the image space with frequency information allows the model to simultaneously focus on the generation effects in both the spatial and frequency domains.

[0070] The expression for mean square error is:

[0071]

[0072] Where X0 is the original image, X θ (X t (t) is the image predicted by the model, where t is the time step and β is the value of β. t Let be the variance used in the t-th noise addition. and Through β t The calculation is as follows: α t =1-β t ,

[0073] The frequency domain information error expression is:

[0074]

[0075] The image size is M×N, and X0(u, v) is the spatial frequency value of the original image in the spectral coordinate system. θ(u, v) represents the spatial frequency value of the predicted image in the spectral coordinate system, where w(u, v) is the spectral weight matrix of the image. The value of w(u, v) is dynamically determined based on the current loss of each frequency during training. Low weights are assigned to simple frequencies, and high weights are assigned to complex frequencies, thus shifting the training focus in the frequency domain towards complex frequencies.

[0076] The overall objective function L is L mse With L fre sum.

[0077] At this point, the training process of the diffusion model is complete. During the training phase, the diffusion model learns a parameterized Markov chain, avoiding the mode collapse problem that may occur during adversarial training in generative adversarial models. Compared with generative adversarial networks, the training process is more stable and the parameters are easier to adjust.

[0078] S5. Using a small number of samples from a new dataset as reference images, extract their semantic and structural features to assist the denoising model in generating similar images to the sample images.

[0079] The new dataset refers to a collection of image samples that did not appear in the training set, and the number of samples in the new dataset can be less than a hundred. During the sampling process of the diffusion model, samples from the new dataset are sequentially selected as reference images for the denoising process, and the information contained in the reference images is used as a conditional constraint for denoising.

[0080] To generate images that share high-level semantic and structural features with a given reference image, images can be sampled from the conditional distribution p(X0|C). The sampling process is represented as follows:

[0081] p θ (X0|C)=∫p θ (X 0：T |C)dX 1：T

[0082]

[0083] Where C represents the sampling condition.

[0084] After adding noise to the reference image for t-1 steps, upsampling and downsampling are performed by φ. C Acquire perceptual information C from the reference image ref Through C ref Guided image X t Denoising generated image X t-1 The process generates an image that is related to the content of the reference image.

[0085] After adding noise to the reference image for t-1 steps, a high-pass filter φ is applied. s Obtain structural information S from the reference image ref Through Sref Guided image X t Denoising generated image X t-1 The process generates an image with a structure similar to that of a reference image.

[0086] C ref The scope of application can be determined by the range parameter. C Adjustments are made, with the effective range being the denoising range. C ~T steps, the larger the range of effect, the more C ref The stronger the constraint effect, the more similar the content, color scheme, and other information of the generated image will be to the reference image.

[0087] S ref The scope of application can be determined by the range parameter. s Adjustments are made, with the effective range being the denoising range. s ~T steps, the larger the range of action, S ref The stronger the constraint effect, the more similar the overall structure, target contour and other information of the generated image are to the reference image.

[0088] In C ref With S ref Within its scope, each transformation p in the sampling process θ (X t-1 |X t Both C and C are affected by condition C, and C is determined by C. ref and S ref Together they form a whole; if the number of denoising steps exceeds C... ref or S ref If one of the terms is within a certain range, then C equals C. ref or S ref The sampling process is flexible and editable, through C ref and S ref Adjustments can be made to modify the quality and diversity of the generated images.

[0089] Example 2

[0090] This embodiment provides a system for implementing the aforementioned few-sample image generation method based on a diffusion model, comprising:

[0091] The first module is used to acquire the target image and establish an auxiliary dataset.

[0092] The second module is used to add noise to the auxiliary dataset multiple times to obtain a noisy image that satisfies an isotropic Gaussian distribution;

[0093] The third module is used to perform denoising inference on noisy images, using the original image as the training target to train the model and gradually remove the noise in the intermediate hidden code;

[0094] The fourth module is used to adjust the model parameters based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information. It focuses on learning complex frequencies through dynamic weighting of frequency domain loss.

[0095] The fifth module is used to extract semantic and structural features from a small number of samples from a new dataset as reference images to assist the denoising model in generating similar images to the sample images.

[0096] Example 3

[0097] This embodiment provides an apparatus for implementing the aforementioned diffusion model-based few-sample image generation system, comprising:

[0098] At least one processor;

[0099] At least one memory for storing at least one program;

[0100] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0101] This embodiment provides a few-sample image generation device based on a diffusion model, which can execute a few-sample image generation method based on a diffusion model provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0102] This embodiment also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0103] This embodiment also provides a storage medium storing an executable method embodiment of the present invention. Figure 1 The instructions or program shown are for a small sample image generation method based on a diffusion model. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0104] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the implementation flowchart. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0105] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

Claims

1. A few-sample image generation method based on a diffusion model, characterized in that, Includes the following steps: Acquire the target image and create an auxiliary dataset; Noise is added multiple times to the auxiliary dataset to obtain a noisy image that follows an isotropic Gaussian distribution; the expression for the noise addition process is: in This is the original image. This is the image after noise has been added. It is the time step. For the first The variance used in the subsequent noise addition. pass The calculation yielded: , ; Denoising inference is performed on noisy images. The original image is used as the training target to train a diffusion model to progressively remove noise from the intermediate hidden codes, including: The noise reduction process is expressed as follows: in This is the image after noise has been added. yes The intermediate result obtained after one round of noise reduction. It is the time step. During the denoising process, the dimension of the intermediate result remains unchanged, and the original image is used as the target for denoising the noisy image. The model parameters are adjusted based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information. Complex frequencies are learned by dynamically weighting the frequency domain loss. Using a small number of samples from a new dataset as reference images and extracting their semantic and structural features to assist the denoising model in generating sample images for the new dataset, similar images are used, including: Add noise to the reference image Upsampling and downsampling are performed after each step. Acquire perceptual information from reference image ,pass Guided images Denoising generated image The process generates an image related to the content of the reference image; Add noise to the reference image High-pass filtering is performed after the step. Obtain structural information from the reference image ,pass Guided images Denoising generated image The process generates an image with a structure similar to that of a reference image; by and As sampling conditions From the conditional distribution The sampling process for the mid-sampled image is represented as follows: in These are the sampling conditions.

2. The method for generating few-sample images based on a diffusion model according to claim 1, characterized in that, The acquisition of the target image and the establishment of the auxiliary dataset include: Images of the target are captured using image acquisition equipment and used as supplementary dataset images. The auxiliary dataset is expanded by performing rotation, brightness adjustment, and contrast adjustment operations on the images in the auxiliary dataset.

3. The method for generating few-sample images based on a diffusion model according to claim 1, characterized in that, The model parameters are adjusted based on the pixel mean square error between the original image and the generated image, as well as the loss of frequency domain information. Complex frequencies are learned primarily through dynamic weighting of the frequency domain loss. The mean square error expression is as follows: in This is the original image. It is the image predicted by the model. It is the time step. For the first The variance used in the subsequent noise addition. and It is possible The calculation is as follows: , .

4. The method for generating few-sample images based on a diffusion model according to claim 1, characterized in that, The model parameters are adjusted based on the pixel mean square error between the original image and the generated image, as well as the frequency domain information loss. Complex frequencies are learned primarily through dynamic weighting of the frequency domain loss. The expression for the frequency domain information loss is as follows: The image size is , This represents the spatial frequency value of the original image in the spectral coordinate system. To predict the spatial frequency values ​​of an image in a spectral coordinate system, The image's spectral weight matrix is... The value is dynamically determined based on the current loss at each frequency during training. Lower weights are assigned to simple frequencies, and higher weights to complex frequencies, thus shifting the training focus in the frequency domain towards more complex frequencies. The overall objective function... for and sum.

5. The method for generating few-sample images based on a diffusion model according to claim 1, characterized in that, The new dataset refers to a collection of image samples that did not appear in the training set. During the sampling process of the diffusion model, samples from the new dataset are selected sequentially as reference images for the denoising process, and the information contained in the reference images is used as a conditional constraint for denoising.

6. A system for implementing the few-sample image generation method based on a diffusion model as described in claim 1, characterized in that, include: The first module is used to acquire the target image and establish an auxiliary dataset. The second module is used to add noise to the auxiliary dataset multiple times to obtain a noisy image that satisfies an isotropic Gaussian distribution; The third module is used to perform denoising inference on noisy images, using the original image as the training target to train the model and gradually remove the noise in the intermediate hidden code; The fourth module is used to adjust the model parameters based on the pixel mean square error between the original image and the generated image and the loss of frequency domain information. It focuses on learning complex frequencies through dynamic weighting of frequency domain loss. The fifth module uses a small number of samples from a new dataset as reference images and extracts their semantic and structural features to assist the denoising model in generating similar images to the sample images.

Images