Data augmentation method based on diffusion model

Abstract

The invention belongs to the technical field of data enhancement, and particularly relates to a data augmentation method based on a diffusion model, which comprises an image inversion module, a latent variable fusion module, a latent variable augmentation module and a synthetic image generation module. Performing forward diffusion on the input seed image and the background image through a pre-trained diffusion model to obtain potential representation of the seed image and the background image; s2, in the latent variable fusion module, fusing the latent representations of the seed image and the background image in proportion by using a latent variable fusion technology, and generating a fused latent variable representation; s3, in the latent variable augmentation module, in the inverse diffusion process, augmentation methods corresponding to the prompt texts injected and input are represented for the fused latent variables, and the diversity of the image is enhanced; s4, in the composite image generation module, the perturbed latent variable representation is restored into a composite image through an inverse diffusion process, and the method is suitable for wider scenes.

Description

Technical Field

[0001] This invention belongs to the field of data augmentation technology, and in particular relates to a data augmentation method based on a diffusion model. Background Technology

[0002] Human motion recognition generally falls into two categories: those based on optical cameras and those based on wireless sensors. For the human body, the swinging arms, legs, and torso, as their movements change over time, produce unique micro-Doppler features that can be used to distinguish between humans and animals, and to differentiate between different activities such as walking, running, or crawling. Therefore, this has been a research hotspot and cutting-edge topic in human target detection in recent years. Existing technologies utilize deep learning methods to process micro-Doppler radar spectrograms, which can significantly improve the accuracy of human motion recognition and classification. However, training deep learning algorithms requires a large quantity and diversity of datasets. But radar image sample acquisition is costly, and the amount of data is usually limited, thus often restricting the training of deep learning models. In recent years, generative models, represented by generative adversarial networks (GANs), have developed rapidly. Using generative models to process existing small-sample radar spectrograms can achieve data augmentation. To generate more diverse images without being affected by mode collapse, and to effectively address the problems of high model training difficulty and poor image diversity in spectrogram augmentation algorithms, existing technologies employ diffusion models to convert prior data distributions into random noise, and then use a series of Markov chains to gradually remove prediction noise. For example, a micro-Doppler radar spectrogram augmentation method based on a diffusion model (application number CN117491994A) involves: S1, obtaining labeled micro-Doppler radar spectrogram training set samples, preprocessing the obtained samples, and constructing a dataset; S2, establishing the forward diffusion process of the diffusion model: selecting an image x0 from the dataset with a true data distribution q(x), and gradually adding Gaussian noise ε with variance βt to xt. 1. Obtain a new noisy image sample xt, and calculate q(xt|xt) 1) S3. Use the Unet neural network εθ(xt,t) to predict noise, and calculate the loss 1OSs and gradient between the network's predicted noise and the target noise; S4. Establish the reverse diffusion process of the diffusion model, and calculate q(xt|xt) according to Bayes' theorem. 1) Mean and variance; S5, Train a classifier pφ(y|xt,t) on the noisy image sample xt and use gradient to guide the sampling process of the diffusion model; S6, Evaluate the micro-Doppler radar spectrogram samples generated by the diffusion model.

[0003] However, the computational costs of existing methods are high, and they do not involve the fusion and synthesis of multimodal data (such as images, text, or other signals), which limits their application in complex, multi-source information scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a data augmentation method based on a diffusion model, which aims to solve the technical problem that existing computational methods have limitations in complex and multi-source information scenarios, and to achieve a wider range of application scenarios.

[0005] To achieve the above objectives, embodiments of the present invention provide a data augmentation method based on a diffusion model, comprising an image inversion module, a latent variable fusion module, a latent variable augmentation module, and a synthetic image generation module, wherein: S1: In the image inversion module, the input seed image and background image are forward diffused through a pre-trained diffusion model to obtain their latent representation; S2: In the latent variable fusion module, the latent representations of the seed image and the background image are fused proportionally using latent variable fusion technology to generate the fused latent variable representation; S3: In the latent variable augmentation module, during the inverse diffusion process, the augmentation method corresponding to the input prompt text is injected into the fused latent variable representation to enhance the diversity of the image; S4: In the synthetic image generation module, the disturbed latent variable representation is restored to a synthetic image through a reverse diffusion process.

[0006] Preferably, in step S1, the latent representation of the seed image captures the shape and appearance features of the target subject, and the latent representation of the background image contains different background information.

[0007] Preferably, in step S2, the latent variable fusion technique balances the contributions of the seed image and the background image in the final synthesized image by controlling parameters, thereby generating the latent variable representation with mixed background and lighting conditions.

[0008] Preferably, in step S3, the reverse diffusion process generates images with different background images and augmentation conditions by perturbing the feature representation of the latent variables.

[0009] Preferably, in step S3, during the reverse diffusion process, different types of noise are dynamically injected or feature perturbations are performed based on the input prompt text to generate diverse images that meet specific conditions.

[0010] Preferably, in step S4, the synthesized image retains the core features of the seed image.

[0011] Preferably, in step S4, the synthesized image has different images and augmentation conditions.

[0012] The data augmentation method based on a diffusion model provided in this invention has at least one of the following technical effects: 1. By proportionally mixing the latent representations of the seed image and the background image, the generated synthetic image retains both the core features of the seed image (such as the shape and texture of the bottle) and the diverse information of the background image. This fusion method allows for more flexible control over the combination of image features, thereby generating high-quality synthetic images with different backgrounds and conditional augmentations, providing the model with richer and more diverse training samples; 2. The latent variable augmentation process is guided by introducing conditional cues (such as text descriptions). During the backdiffusion process, different types of noise or feature perturbations are dynamically injected based on the input cues to generate diverse images that meet specific conditions (such as specific lighting, background, or object position). This conditional latent variable augmentation method significantly improves the diversity and specificity of the generated images, making data augmentation more flexible and accurate, and better adaptable to various complex application scenarios. 3. This application directly utilizes a pre-trained diffusion model to generate diverse synthetic images without additional training steps, significantly reducing computational costs and time overhead. By using a pre-trained model for image inversion and latent variable perturbation, it not only improves the efficiency of data augmentation but also ensures the quality and diversity of generated images, providing a better solution for tasks with limited data. Attached Figure Description To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a sequence diagram of the module devices provided in an embodiment of the present invention.

[0014] Figure 2 This is a flowchart of a synthetic image generation method provided in an embodiment of the present invention. Detailed Implementation

[0015] The embodiments of the present invention are described in detail below, and examples of these embodiments are provided in the appendix. Figures 1-2As shown, the same or similar reference numerals throughout denote the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the invention, and should not be construed as limiting the invention.

[0016] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0017] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0018] In the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention according to the specific circumstances.

[0019] In one embodiment of the present invention, such as Figures 1-2 As shown, a data augmentation method based on a diffusion model is provided, including an image inversion module, a latent variable fusion module, a latent variable augmentation module, and a synthetic image generation module. It directly utilizes a pre-trained diffusion model and, during the inference stage, efficiently generates synthetic images that retain the core features of the target subject while possessing rich background and conditional variations through techniques such as image inversion, latent variable fusion, and augmentation. This expands the diversity and richness of the training dataset, generating synthetic images with diverse backgrounds and conditional augmentations. Specifically, the method includes the following steps: Step S1, in the image inversion module: The "diffusion model" of deep generation, which is pre-trained, learns the data distribution by simulating the process of adding noise (forward diffusion) to the data at multiple time steps and gradually reconstructing the data from the noise (reverse diffusion). The input seed image and background image are processed by "forward diffusion" according to the predefined forward process of the diffusion model. Gaussian noise is gradually added to the original image at multiple time steps until the image is approximately transformed into pure noise. The "latent representation" of the noise latent variable or feature vector obtained at a specific time step or through a specific encoder after the forward diffusion process is generated. The specific steps are as follows: Input images of bottles from different brands as "seed images," and input a set of different scene images (such as images containing elements like belts, wooden tabletops, and outdoor environments) as "background images." The main content of these background images is usually different from that of the seed images. Each seed image and background image is input into a pre-trained diffusion model. Through its built-in "image inversion," a real image is transformed into its latent variable space through the model's forward process or a specific encoder. This yields a latent variable representation that can be approximately reconstructed through the inverse process, allowing us to obtain their representation in the model's latent space. This process requires no additional training and relies entirely on the model's pre-trained knowledge. The latent representation of the seed image mainly captures the main features of the bottle, such as its shape, appearance, and trademark, while the latent representation of the background image encodes different background textures, lighting conditions, and color style information.

[0020] Step S2, in the latent variable fusion module: The "latent variable fusion technique" is used to combine the representations of two or more latent variables through mathematical operations (such as linear interpolation, weighted averaging, feature channel mixing, etc.) to generate a new latent variable that integrates the features from their respective sources.

[0021] The latent representation of the seed image obtained in step S1 is mixed with the latent representation of the background image according to the requirements to generate a new latent variable representation after fusion. The fusion operation can be achieved by setting an adjustable parameter (usually between 0 and 1) "weight coefficient" to control the contribution ratio of the latent variables of the seed image and the latent variables of the background image during the fusion process. By adjusting this coefficient, the significance of the target subject features and background information in the final synthesized image can be balanced. The fused "synthetic latent variable representation" contains both the subject features from the seed image and the background information from the background image. As the starting point of the reverse diffusion process, it not only retains the core visual features of the subject, but also introduces a new background context, laying the foundation for generating an image with a mixed background.

[0022] Step S3, in the latent variable augmentation module: After the reverse diffusion process begins, to improve the diversity of the generated images, especially by introducing variations that meet specific conditions, this step implements "conditional augmentation": based on external input conditions (such as text prompts), the image is guided to change in a direction that satisfies those conditions during the generation process. For example, changing the lighting angle, weather conditions, or the orientation of objects, etc., the specific method is as follows: The input description is a "cue text" used to guide the generation direction of the diffusion model, such as "under soft natural light," "with shadows," or "blurred background." At each or key step of the inverse diffusion process, "noise perturbation" is injected into the fused latent variable representation. During inverse diffusion denoising, a small amount of controllable Gaussian noise or a tiny random transformation of the feature space is intentionally added to the currently estimated latent variables to break determinism, explore neighboring variants in the generation space, or "conditional perturbation" is performed on the feature representation based on the cue text using the conditional mechanism in the pre-trained diffusion model. This injects the information from the text cue into the latent variable update process, making the generation process biased towards the features described in the text. For example, based on the cue "side lighting," the latent variable update is biased towards generating image features with side lighting effects. Through this controllable perturbation, it is ensured that the final generated image not only varies in the background but also exhibits sufficient diversity in lighting, detail texture, and viewpoint, thereby expanding the image variation space and making the generation results no longer limited to simple interpolation but capable of producing broader and more unexpected visual variants, enhancing the coverage of the dataset.

[0023] Step S4, in the image synthesis generation module: The latent variable representation after perturbation in step S3 is input into the core generation stage of the pre-trained diffusion model, the "reverse diffusion process." The model progressively denoises and reconstructs the latent variables. After the complete reverse diffusion process, the final synthetic image is obtained. It retains the identifiable core features of the target subject in the original seed image, while also possessing new background and augmentation conditions that conform to text prompts. The generated synthetic image is of high quality and has clear features. It maintains consistency with the seed image in terms of the subject and has rich background and condition variations, which can be directly used to expand the training dataset for related visual tasks.

[0024] Compared with traditional data augmentation techniques (such as rotation, cropping, and color dithering), this application generates images with richer semantic layers and more novel content through latent variable-level fusion and conditional augmentation. It breaks through the limitations of traditional methods that can only perform low-level geometric and pixel transformations. By introducing a conditional augmentation mechanism guided by prompt text, it achieves precise and flexible control over the attributes of the generated image, significantly improving the targeting and diversity of the augmentation. It can better adapt to complex and ever-changing practical application needs. The entire process is based entirely on pre-trained model inference, without any additional model training or fine-tuning, greatly reducing computational costs and time overhead. It provides an efficient, plug-and-play, high-quality data augmentation solution.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data augmentation method based on diffusion model, characterized in that: It includes an image inversion module, a latent variable fusion module, a latent variable augmentation module, and a synthetic image generation module, wherein: S1: In the image inversion module, the input seed image and background image are forward diffused through a pre-trained diffusion model to obtain their latent representation; S2: In the latent variable fusion module, the latent representations of the seed image and the background image are fused proportionally using latent variable fusion technology to generate the fused latent variable representation; S3: In the latent variable augmentation module, during the inverse diffusion process, the augmentation method corresponding to the input prompt text is injected into the fused latent variable representation to enhance the diversity of the image; S4: In the synthetic image generation module, the disturbed latent variable representation is restored to a synthetic image through a reverse diffusion process.

2. The data augmentation method based on a diffusion model according to claim 1, characterized in that: In step S1, the latent representation of the seed image captures the shape and appearance features of the target subject, and the latent representation of the background image contains different background information.

3. The data augmentation method based on a diffusion model according to claim 1, characterized in that: In step S2, the latent variable fusion technique balances the contributions of the seed image and the background image in the final synthesized image by controlling parameters, thereby generating the latent variable representation with mixed background and lighting conditions.

4. The data augmentation method based on a diffusion model according to claim 1, characterized in that: In step S3, the inverse diffusion process generates images with different background images and augmentation conditions by perturbing the feature representation of the latent variables.

5. The data augmentation method based on a diffusion model according to claim 4, characterized in that: In step S3, during the reverse diffusion process, different types of noise are dynamically injected or feature perturbations are performed based on the input prompt text to generate diverse images that meet specific conditions.

6. The data augmentation method based on a diffusion model according to claim 1, characterized in that: In step S4, the synthesized image retains the core features of the seed image.

7. The data augmentation method based on a diffusion model according to claim 6, characterized in that: In step S4, the synthesized image has different images and augmentation conditions.

Images