A small sample image generation method and system based on feature difference dynamic guidance

Abstract

The application discloses a kind of based on feature difference dynamic guiding small sample image generation method and system.The method includes the following steps: obtaining image data and its category label, establishes training set;Diffusion model is constructed, in diffusion model, the image after adding noise to input reference image constantly t steps is obtained, equivalent to a isotropic Gaussian noise, then the image after adding noise to t steps is iteratively executed multiple single-step denoising restoration and obtains predicted image;With the feature difference of predicted image and reference image, loss function is constructed, and the data in training set is used to train diffusion model to obtain trained diffusion model;With unknown category image data that does not appear in training set as reference image, input trained diffusion model, and generate to obtain small sample image.The application is beneficial to improve the diversity of generated image, and the visual quality of image is improved by fusing the information of reference image.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a method and system for generating few-sample images based on feature difference dynamic guidance. Background Technology

[0002] The widespread application of deep learning has made daily life more convenient and efficient, greatly improving people's work efficiency. However, a remaining problem in applying deep learning to real-world scenarios is the need for models to receive large amounts of training data. To alleviate the limitations imposed by limited data on deep learning applications, the few-shot image generation task has been proposed. Few-shot image generation refers to the task of synthesizing diverse new samples of a target category using a limited set of data samples through a deep learning model. Few-shot image generation can be seen as a data augmentation method, expanding existing datasets and assisting in the application and promotion of deep learning models in real-life and production fields.

[0003] Existing few-shot image generation methods are mainly divided into three categories: optimization-based methods, transformation-based methods, and fusion-based methods. The vast majority of these methods are based on Generative Adversarial Networks (GANs), and in recent years, diffusion-based methods have also been proposed. FIGR pioneered the use of optimization-based meta-learning Reptile in the training of GAN models to generate few-shot images, but the computational cost of training the model is very high. DAGAN generates novel samples by injecting random noise into the representation of a single input image, but the generated images lack diversity. WaveGAN (Frequency-aware GAN for High-Fidelity Few-shot Image Generation) decomposes the encoded features into multiple frequency components, feeds high-frequency components to the decoder through high-frequency skip connections to alleviate the difficulty of the generator synthesizing fine details, and uses low-frequency skip connections to preserve perceptible basic information, but it is prone to aliasing artifacts. FSDM uses ViT to aggregate image patch information in units of image sets. It uses cross-attention to fuse image patch information and latent variables to adapt to the generation process conditioned on a set of images of a given category. However, the generated images have a strong sense of blur and semantic convergence, resulting in poor generation quality. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for generating small sample images based on feature difference dynamic guidance, so as to generate similar images with high visual quality and rich diversity for a small number of samples.

[0005] This invention studies the few-shot image generation task based on the diffusion model. Compared with GAN, the diffusion model has a definite training objective and better distribution coverage, and has great potential in the few-shot image generation task.

[0006] This invention fully utilizes information from the reference image and rationally guides the generation process of the diffusion model. The generation process provides the latent variables with partial contextual information about the unknown class of the image, assisting the diffusion model in controlling the details of the generated image. By reducing the distance between the generated image predicted by the latent variables and the reference image in the feature space, the noise distribution removed in each step is adjusted. The generated image has greater flexibility, which is beneficial for obtaining diverse and high-quality generated images, while avoiding the aliasing artifact problem common in GAN-based models.

[0007] The objective of this invention is achieved by at least one of the following technical solutions.

[0008] A few-shot image generation method based on feature difference dynamic guidance includes the following steps:

[0009] S1. Obtain image data and its category labels, and build a training set;

[0010] S2. Construct a diffusion model. In the diffusion model, noise is continuously added to the input reference image to obtain noise. t The image after the step is essentially an isotropic Gaussian noise, and then further noise is added. t The image after each step is iteratively denoised and restored multiple times to obtain the predicted image;

[0011] S3. Construct a loss function based on the feature difference between the predicted image and the reference image, and train the diffusion model using the data in the training set to obtain the trained diffusion model.

[0012] S4. Use the image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

[0013] Furthermore, in step S1, the target object is imaged using an image acquisition device (e.g., a high-definition camera), and the image is placed into a folder of the corresponding category. The folder name is used as the image label to complete the establishment of the training set.

[0014] Furthermore, in step S2, in the diffusion model, noise is continuously added to the input known image, as expressed by:

[0015]

[0016] Where X0 is the input reference image, X t Let α be a latent variable, representing the image after t steps of noise addition, where t = 1, 2, 3, ..., T, and T is the total number of noise addition steps.t =1-β t ,β t Let be the variance used during the t-th noise addition process. ∈~N(0,1), where ∈ is Gaussian noise and has the same dimension as the original image X0.

[0017] Furthermore, in step S2, the existing deep neural network U-Net (U-Net: Convolutional Networks for Biomedical Image Segmentation) is used to predict the noise removed in the single step, and the latent variable X is... t The expression for single-step denoising is:

[0018]

[0019] Among them, latent variable X t-1 It is a latent variable X t The result of a single denoising operation, ∈ θ (X t δ(t) represents the noise distribution predicted by the deep neural network Unet. t For latent variable X t The standard deviation of Z is given by Z ~ N(0, I).

[0020] Furthermore, in each iteration of the diffusion model's denoising step, the latent variables are fused with the contextual information of the input reference image, specifically including the following steps:

[0021] S2.1, Latent variable X t The latent variable X′ is obtained after single-step denoising. t-1 ;

[0022] S2.2, Noise is added to the reference image Y in step t-1 to obtain Y t-1 ;

[0023] S2.3 The fusion expression for latent variables and contextual information from the reference image is as follows:

[0024] X″ t-1 =M·Y t-1 +(1-M)·X′ t-1

[0025] Where M is the mask, X″ t-1 This is the result of fusing latent variables with a reference image.

[0026] Furthermore, in the diffusion model, through the latent variable X t Iterative denoising yields the current latent variable X t Predicted image after denoising The expression is as follows:

[0027]

[0028] Furthermore, in step S3, during the training process of the diffusion model, for a single reference image in the training set of the input diffusion model, the predicted output image is calculated. The distance D between the feature representation of the reference image and the reference image feature The expression is as follows:

[0029]

[0030] in, Representative predicted image The feature representation of the k-th layer after N layers of feature extractors, k = 1, 2, ..., N. D represents the feature representation of the reference image at the k-th layer after passing through the feature extractor. k Representative predicted image The distance between the reference image and the feature representation output by the k-th layer of the feature extractor; D feature It is the sum of the distances between the feature representations output by each layer of the N-layer feature extractor.

[0031] Furthermore, during the training process of the diffusion model, the constructed loss function is to minimize the predicted image. The sum of distances D between the feature representations output by each layer of the N-layer feature extractor and the reference image. feature .

[0032] The difference between the feature representations output by each layer of the feature extractor and the reference image varies. If the difference between the feature representations of a certain layer is the largest, then D... feature The feature difference in this layer will dominate, and the noise distribution predicted by Unet will be adjusted mainly by reducing the feature difference in this layer.

[0033] Furthermore, the feature extractor for the N layers includes ResNet or VGG.

[0034] A few-sample image generation system based on a diffusion model, comprising:

[0035] The first module is used to acquire image data and its category labels, and to build a training set.

[0036] The second module is used to construct the diffusion model, in which noise is continuously added to the input reference image to obtain the noise. t The image after the step is essentially an isotropic Gaussian noise, and then further noise is added. t The image after each step is iteratively denoised and restored multiple times to obtain the predicted image;

[0037] The third module is used to construct a loss function based on the feature difference between the predicted image and the reference image, and to train the diffusion model using the data in the training set to obtain the trained diffusion model.

[0038] The fourth module is used to take image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] This invention studies the few-shot image generation task based on the diffusion model. Compared with GAN, the diffusion model has a definite training objective and better distribution coverage, and has great potential in the few-shot image generation task.

[0041] This invention fully utilizes the information of the reference image and reasonably guides the generation process of the diffusion model. The generation process provides the latent variables with some contextual information of the unknown category image, which helps the diffusion model control the details of the generated image. By reducing the distance between the generated image predicted by the latent variables and the reference image in the feature space, the noise distribution removed in each step is adjusted. The generated image has greater flexibility, which is conducive to obtaining diverse and high-quality generated images, while avoiding the aliasing artifact problem common in GAN-based models. Attached Figure Description

[0042] The technical solutions are illustrated in the following description with specific figures to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and similar extended embodiments made by those skilled in the art without inventive effort are all within the scope of protection of this invention.

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the implementation of a few-sample image generation method based on dynamic guidance of feature difference in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the overall structure of a small sample image generation method based on dynamic guidance of feature difference in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram illustrating the fusion of latent variables and contextual information of a reference image in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram illustrating the calculation of the feature distance between the latent variable prediction image and the reference image in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions are illustrated in the following description with specific figures to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and similar extended embodiments made by those skilled in the art without inventive effort are all within the scope of protection of this invention.

[0049] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0050] It should be understood that although terms such as "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are used only to distinguish similar objects and are not necessarily used to indicate the order or sequence of features described in one or more embodiments of this specification. Furthermore, the terms "possessing," "comprising," and similar expressions are intended to indicate coverage of a non-exclusive scope. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to the detailed list but may include inherent content related to those steps or modules that is not listed.

[0051] Example:

[0052] A few-sample image generation method based on feature difference dynamic guidance, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0053] S1. Obtain image data and its category labels, and build a training set;

[0054] In one embodiment, the target object is imaged using an image acquisition device (e.g., a high-definition camera), and the images are placed into folders of the corresponding categories. The folder names are used as image labels to complete the establishment of the training set.

[0055] S2. Construct a diffusion model. In the diffusion model, noise is continuously added to the input reference image to obtain a noisy image after t steps, which is equivalent to an isotropic Gaussian noise. Then, the noisy image after t steps is iteratively denoised and restored multiple times to obtain the predicted image.

[0056] In the diffusion model, noise is continuously added to the input known image, expressed as:

[0057]

[0058] Where X0 is the input reference image, X t Let α be a latent variable, representing the image after t steps of noise addition, where t = 1, 2, 3, ..., T, and T is the total number of noise addition steps. t =1-β t ,β t Let be the variance used during the t-th noise addition process. ∈~N(0,1), where ∈ is Gaussian noise and has the same dimension as the original image X0.

[0059] Data sample X0 gradually loses its discernible features as the step size t increases. Finally, as T→∞, X... T It is equivalent to an isotropic Gaussian distribution.

[0060] In one embodiment, the existing deep neural network U-Net (U-Net: Convolutional Networks for Biomedical Image Segmentation) is used to predict the noise removed in a single step, such as... Figure 3 As shown, for the latent variable X t The expression for single-step denoising is:

[0061]

[0062] Among them, latent variable X t-1 It is a latent variable X t The result of a single denoising operation, ∈ θ (X t δ(t) represents the noise distribution predicted by the deep neural network Unet. t For latent variable X t The standard deviation of Z is given by Z ~ N(0, I).

[0063] In each iteration of the diffusion model's denoising step, the latent variables are fused with contextual information from the input reference image, specifically including the following steps:

[0064] S2.1, Latent variable X t The latent variable X′ is obtained after single-step denoising. t-1 ;

[0065] S2.2, Noise is added to the reference image Y in step t-1 to obtain Y t-1 ;

[0066] S2.3 The fusion expression for latent variables and contextual information from the reference image is as follows:

[0067] X″ t-1 =M·Y t-1 +(1-M)·X′ t-1

[0068] Where M is the mask, X″ t-1 This is the result of fusing latent variables with a reference image.

[0069] In the diffusion model, through the latent variable X t Iterative denoising yields the current latent variable X t Predicted image after denoising The expression is as follows:

[0070]

[0071] S3. In one embodiment, such as Figure 4 As shown, a loss function is constructed using the feature difference between the predicted image and the reference image, and the diffusion model is trained using the data in the training set to obtain the trained diffusion model.

[0072] During the training process of the diffusion model, for a single reference image input into the training set of the diffusion model, the predicted output image is calculated. The distance D between the feature representation of the reference image and the reference image feature The expression is as follows:

[0073]

[0074] in, Representative predicted image The feature representation of the k-th layer after N layers of feature extractors, k = 1, 2, ..., N. D represents the feature representation of the reference image at the k-th layer after passing through the feature extractor. k Representative predicted image The distance between the reference image and the feature representation output by the k-th layer of the feature extractor; D feature It is the sum of the distances between the feature representations output by each layer of the N-layer feature extractor.

[0075] During the training of the diffusion model, the loss function constructed is to minimize the predicted image. The sum of distances between the feature representations output by each layer of the N-layer feature extractor and the reference image. D feature .

[0076] The difference between the feature representations output by each layer of the feature extractor and the reference image varies. If the difference between the feature representations of a certain layer is the largest, then D... feature The feature difference in this layer will dominate, and the noise distribution predicted by Unet will be adjusted mainly by reducing the feature difference in this layer.

[0077] In one embodiment, the feature extractor for the N layers is ResNet.

[0078] In one embodiment, the feature extractor for the Nth layer is VGG.

[0079] S4. Use the image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

[0080] In one embodiment, a few-sample image generation system based on a diffusion model includes:

[0081] The first module is used to acquire image data and its category labels, and to build a training set.

[0082] The second module is used to construct the diffusion model, in which noise is continuously added to the input reference image to obtain the noise. t The image after the step is essentially an isotropic Gaussian noise, and then further noise is added. t The image after each step is iteratively denoised and restored multiple times to obtain the predicted image;

[0083] The third module is used to construct a loss function based on the feature difference between the predicted image and the reference image, and to train the diffusion model using the data in the training set to obtain the trained diffusion model.

[0084] The fourth module is used to take image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

[0085] The preferred embodiments disclosed above are merely illustrative of the invention and its core concepts. Those skilled in the art will recognize that variations may occur in specific application scenarios and implementation based on the ideas of this invention; therefore, this specification should not be construed as limiting the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating few-sample images based on dynamic guidance of feature differences, characterized in that, Includes the following steps: S1. Obtain image data and its category labels, and build a training set; S2. Construct a diffusion model. In the diffusion model, noise is continuously added to the input reference image to obtain a noisy image after t steps, which is equivalent to an isotropic Gaussian noise. Then, the noisy image after t steps is iteratively denoised and restored multiple times to obtain the predicted image. S3. Construct a loss function using the feature difference between the predicted image and the reference image, and train the diffusion model using the data in the training set to obtain the trained diffusion model; during the training process of the diffusion model, for a single reference image input into the training set of the diffusion model, calculate the output predicted image. The distance D between the feature representation of the reference image and the reference image feature The expression is as follows: in, Representative predicted image The feature representation of the l-th layer after N layers of feature extractors, k = 1, 2, ..., N. D represents the feature representation of the reference image at layer l after passing through the feature extractor. k Representative predicted image The distance between the reference image and the feature representation output by the l-th layer of the feature extractor; D feature This is the sum of the distances between the feature representations output by each layer of the N-layer feature extractor; During the training of the diffusion model, the loss function constructed is to minimize the predicted image. The sum of distances D between the feature representations output by each layer of the N-layer feature extractor and the reference image. feature ; S4. Use the image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

2. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 1, characterized in that, In step S1, the target object is imaged using an image acquisition device, and the image is placed into a folder of the corresponding category. The folder name is used as the image label to complete the establishment of the training set.

3. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 1, characterized in that, In step S2, in the diffusion model, noise is continuously added to the input known image, as expressed by: Where X0 is the input reference image, X t Let α be a latent variable, representing the image after t steps of noise addition, where t = 1, 2, 3, ..., T, and T is the total number of noise addition steps. t =1-β t ,β t Let be the variance used during the t-th noise addition process. ∈~N(0,1), where ∈ is Gaussian noise and has the same dimension as the original image X0.

4. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 3, characterized in that, In step S2, the existing deep neural network Unet is used to predict the noise removed in the single step, and the latent variable X is... t The expression for single-step denoising is: Among them, latent variable X t-1 It is a latent variable X t The result of a single denoising operation, ∈ θ (X t δ(t) represents the noise distribution predicted by the deep neural network Unet. t For latent variable X t The standard deviation of Z is given by Z ~ N(0, I).

5. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 4, characterized in that, In each iteration of the diffusion model's denoising step, the latent variables are fused with contextual information from the input reference image, specifically including the following steps: S2.1, Latent variable X t The latent variable X′ is obtained after single-step denoising. t-1 ; S2.2, Noise is added to the reference image Y in step t-1 to obtain Y t-1 ; S2.3 The fusion expression for latent variables and contextual information from the reference image is as follows: X″ t-1 =M·Y t-1 +(1-M)·X′ t-1 Where M is the mask, X″ t-1 This is the result of fusing latent variables with a reference image.

6. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 5, characterized in that, In the diffusion model, through the latent variable X t Iterative denoising yields the current latent variable X t Predicted image after denoising The expression is as follows:

7. The method for generating few-sample images based on dynamic guidance of feature difference according to claim 6, characterized in that, The feature extractor for the N layers includes ResNet or VGG.

8. A few-shot image generation system based on a diffusion model, used to implement the few-shot image generation method based on feature difference dynamic guidance as described in any one of claims 1 to 7, characterized in that, include: The first module is used to acquire image data and its category labels, and to build a training set. The second module is used to construct the diffusion model. In the diffusion model, noise is continuously added to the input reference image to obtain the image after t steps of noise addition, which is equivalent to an isotropic Gaussian noise. Then, the image after t steps of noise addition is iteratively denoised and restored multiple times to obtain the predicted image. The third module is used to construct a loss function based on the feature difference between the predicted image and the reference image, and to train the diffusion model using the data in the training set to obtain the trained diffusion model. The fourth module is used to take image data of unknown categories that do not appear in the training set as reference images, input them into the trained diffusion model, and generate small sample images.

Images