A diversity image synthesis method based on retrieval-enhanced diffusion
By constructing an image synthesis method with a dynamic retrieval pool and an anti-attention module, the problem of insufficient similarity and diversity of image synthesis samples in existing technologies is solved, realizing the generation of diverse images without fine-tuning and improving the generalization ability and image quality of downstream models.
Patent Information
- Application Number
- CN202610430693.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-02
- Publication Date
- 2026-07-03
AI Technical Summary
Existing image synthesis methods generate samples that are highly similar to the training data, lack diversity evaluation methods, are difficult to expand the data distribution range, require fine-tuning for specific scenarios, have high training costs, and have limited transferability.
A retrieval-enhanced diffusion-based diversity image synthesis method is adopted. By constructing a dynamic retrieval pool and an anti-attention module, the generated image is actively guided to explore the uncovered areas of the feature space during the inference stage. A pre-trained feature extractor and an approximate nearest neighbor algorithm are used for efficient retrieval. An anti-attention module is constructed to exclude similar features and generate diverse images.
It significantly improves the diversity of synthetic datasets, enhances the generalization ability and robustness of downstream models, and verifies the diversity and practical value of generated images through experiments using the diversity FID index and end-to-end verification.
Smart Images
Figure CN122335525A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and deep learning technology, and in particular relates to a method for synthesizing diverse images based on retrieval-enhanced diffusion. Background Technology
[0002] Image synthesis is one of the core applications of generative artificial intelligence. By generating supplementary training samples through algorithms, it can effectively alleviate two major challenges in training discriminative models: overfitting on small datasets and the high cost of acquiring high-quality real data. In scenarios where data is scarce or difficult to obtain, synthesized images provide a practical solution for expanding data coverage.
[0003] Since its inception, the Denoising Diffusion Probabilistic Model (DDPM) has become the mainstream paradigm in image generation. By learning to gradually reverse the noise addition process, the diffusion model can generate high-quality, high-resolution images. Building on this, the Latent Diffusion Model (LDM) transfers the diffusion process to a low-dimensional latent space to improve computational efficiency, and the classifier-guided mechanism further enables conditional generation without the need for an external classifier. In addition, ControlNet, by introducing a zero-initialization control layer, supports fine-grained control of the text-image diffusion backbone using various conditions such as edge maps, depth maps, and segmentation masks, greatly improving the controllability of the generation process. In autonomous driving scenarios, methods such as MagicDrive combine the latent diffusion model with 3D geometric control information (BEV road maps, 3D bounding boxes, text descriptions, etc.) to achieve high-quality street scene synthesis with consistent multi-viewpoints through a cross-camera attention mechanism.
[0004] However, the above methods all take visual fidelity as the main optimization target, and have the following shortcomings: First, existing image synthesis methods generate samples that are highly similar to the training data in style and pixel distribution, tending to reproduce existing data patterns and contributing very limited new information to downstream model training. Studies have shown that when retrieving nearest neighbors of real data from generated images, most synthetic samples exhibit extremely high structural similarity to real samples, making it difficult for classifiers trained with synthetic data to obtain broader feature representation capabilities.
[0005] Second, existing methods generally lack systematic evaluation tools for the diversity of synthetic data, making it difficult to quantify the extent to which the generated dataset covers the distribution of real data, resulting in the inability to effectively measure and optimize the actual gain of synthetic data.
[0006] Third, existing retrieval-enhanced generation (RAG) methods in the field of image generation, such as KNN-Diffusion, ImageRAG, and RealRAG, mainly guide the generated image to move closer to the reference sample by the retrieval sample. This is an "attractive" retrieval enhancement, which further strengthens the similarity between the synthetic data and the existing data. This fundamentally contradicts the goal of improving diversity and cannot effectively fill the gaps in the real data distribution, such as areas with abnormal lighting, occlusion, and rare subcategories.
[0007] Fourth, some existing methods require specific fine-tuning of the diffusion model for specific scenarios, resulting in high training costs and limited transferability, making it difficult to flexibly reuse different pre-trained base models.
[0008] Therefore, there is an urgent need to design an image synthesis technology that does not require model fine-tuning and prioritizes diversity. While ensuring the visual quality of the synthesized image, it should actively guide the generation process to explore the uncovered areas of the real data distribution and continuously expand the feature space distribution range of the synthesized dataset, thereby effectively improving the generalization ability and robustness of the downstream model. Summary of the Invention
[0009] To address the issues of insufficient diversity and high similarity between synthesized and real data in existing image synthesis methods, this invention provides a diversity image synthesis method based on retrieval-enhanced diffusion. This method does not require fine-tuning of the diffusion model and actively guides the generated image to explore uncovered areas of the feature space during the inference stage in a plug-and-play manner, thereby significantly improving the diversity of the synthesized dataset and the generalization ability of the downstream model.
[0010] A method for synthesizing diverse images based on retrieval-enhanced diffusion includes the following steps: (1) Construct a dynamic retrieval pool containing real images and generated images, and use a pre-trained feature extractor to extract features from all images in the retrieval pool, embed them, and store them; (2) For the current image to be generated, a specific denoising time step is determined according to the preset strategy, and the Top-K reference samples most similar to the current generation state are retrieved from the retrieval pool using the approximate nearest neighbor algorithm; (3) Based on the retrieved reference samples, an anti-attention module is constructed after the self-attention layer of the generative model to make the currently generated features actively move away from the retrieved reference sample features, so as to exclude similar features and promote the diversity of generation. (4) The features after anti-attention adjustment are fed into the subsequent denoising steps of the diffusion model to complete the complete reverse diffusion process and output a synthetic image with the maximum difference from the existing data distribution.
[0011] In step (1), the update strategy of the dynamic retrieval pool is as follows: each time a new batch of images is generated, it is added to the retrieval pool and the feature index is updated accordingly; as the accumulated number of generated images in the retrieval pool increases, the subsequent generated images are subject to the exclusion constraint from a wider feature space, thereby continuously guiding the newly generated samples to explore the uncovered feature regions.
[0012] In step (1), the pre-trained feature extractor is a visual language model based on contrastive learning, a visual encoder based on self-supervised learning, or the encoder of the generative model itself.
[0013] The feature representations of each reference sample in the retrieval pool are pre-computed and stored offline before the synthesis begins to avoid the overhead of repeated computation during the synthesis process.
[0014] The feature representations of each reference sample in the retrieval pool are stored aligned to the denoising time step. The feature representation stored at each time step is in the same semantic space as the intermediate state of the generative model at the corresponding time step; at a specific denoising time step... When performing a retrieval, the intermediate state features of the current generated state at that time step are used as the query, and only those features from the same time step in the retrieval pool are compared. The corresponding feature representations are compared for similarity to ensure semantic alignment between the retrieved features and the current generated state.
[0015] In step (2), the approximate nearest neighbor algorithm uses an efficient similarity retrieval algorithm to achieve retrieval with sublinear time complexity.
[0016] The selection strategy for specific denoising time steps is as follows: in the early stage of the denoising process, retrieval and anti-attention operations are triggered at a higher frequency to guide the model to explore diverse solution spaces at the global structural level; in the later stage of the denoising process, the triggering frequency is reduced to minimize the negative impact on local details and the quality of the generated image.
[0017] In step (3), the calculation process of the anti-attention module is as follows: With the query matrix of the current self-attention layer For probes; For the retrieved first Feature representation of each reference sample By using the key projection matrix that shares weights with the current self-attention layer Sum projection matrix The corresponding bond matrices were obtained respectively. Sum matrix ; For each reference sample, calculate the repulsion attention weight and repulsion vector separately. Then, calculate the average of the repulsion vectors of the Top-K reference samples to obtain the final repulsion vector. and the final repulsion vector From the original self-attention output Subtracting from the middle yields the adjusted feature representation. ,in, This is the repulsion strength hyperparameter.
[0018] In step (3), the anti-attention module is inserted after one or more self-attention layers of the generated model, coexisting with the original self-attention layer rather than replacing it.
[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. To address the issue of existing methods producing synthetic data that is highly similar to real data, this invention actively rejects generated features from approaching similar samples through an anti-attention mechanism. While ensuring the visual quality of the synthesized image (the standard FID index is comparable to the baseline method), it guarantees the diversity of the synthesized data from a mechanism perspective. Experimental results show that the diversity FID index is significantly improved compared to the baseline method.
[0020] 2. To address the lack of quantitative evaluation methods for diversity in existing approaches, this invention proposes a multi-dimensional evaluation system centered on diversity FID, which can systematically measure the coverage expansion of synthetic datasets relative to the real distribution. Simultaneously, this invention uses the enhancement of downstream 3D object detection model training effects with synthetic data as end-to-end validation. Experimental results show that the detection model trained using the synthetic data of this invention outperforms the model trained using baseline methods with synthetic data in both mAP and NDS metrics. These two evaluation systems validate the effectiveness of this invention in improving the practical value of synthetic data.
[0021] 3. To address the problem of data homogenization in existing RAG methods, this invention reverses the retrieval signal from "attraction" to "repulsion," applying the RAG paradigm to diversity-driven image generation for the first time and establishing a new paradigm of diversity-first generation.
[0022] 4. To address the issue of existing methods relying on fine-tuning, this invention adopts a training-independent design, running as a plug-and-play module during the inference phase without requiring modification of any parameters of the pre-trained diffusion model, and can flexibly adapt to different diffusion model bases. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart of a diversity image synthesis method based on retrieval-enhanced diffusion, according to an embodiment of the present invention.
[0025] Figure 2 This is a schematic diagram illustrating the construction of the dynamic retrieval pool and the approximate nearest neighbor retrieval process based on locality-sensitive hashing in an embodiment of the present invention.
[0026] Figure 3 This is a detailed structural diagram of the anti-attention module in an embodiment of the present invention. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0029] like Figure 1 As shown, this invention provides a diversity image synthesis method based on retrieval-enhanced diffusion. The overall framework comprises four core components: a dynamic retrieval pool module, an approximate nearest neighbor retrieval module, an anti-attention module, and a diffusion denoising module. This invention requires no parameter fine-tuning of the basic diffusion model and guides the generation process towards diversity in a plug-and-play manner during the inference phase.
[0030] Step S1: Construct a dynamic retrieval pool.
[0031] like Figure 2 As shown, an initial retrieval pool is constructed before the synthesis begins. Its initial elements are a set of real images. ,in Using a pre-trained visual encoder Semantic features are extracted from all images in the retrieval pool, embedded, and stored offline to avoid redundant computation during the synthesis process. To ensure that the retrieved features are in the same semantic space as the current denoising state, the method of extracting image semantic features needs to remain consistent throughout the process.
[0032] This embodiment uses the DDIM Inversion method to perform denoising time step inversion on each image in the search pool, extracting each time step. Corresponding latent variables This serves as the stored feature for that time step, forming a feature index aligned to the time step. At the denoising time step that triggers the anti-attention operation... During retrieval, the latent variables of the image to be generated at that time step are used. As a query, at the same time step in the retrieval pool The corresponding latent variables are compared for similarity to ensure that the retrieved features are semantically aligned with the current generated state.
[0033] During the generation process, each new image is generated Add it to the search pool: ← The feature index is updated accordingly to continuously expand the feature space coverage of the retrieval pool.
[0034] Step S2: Use the approximate nearest neighbor algorithm for fast retrieval.
[0035] like Figure 2 As shown, at a specific denoising time step determined according to preset hyperparameters... The latent variables of the image to be generated at this time step As a query feature, an approximate nearest neighbor search is performed, retrieving the Top-K samples with the highest cosine similarity from the search pool feature index.
[0036] To achieve efficient retrieval with sublinear time complexity, an Angular Locality Sensitive Hash (AngularLSH) algorithm is employed. This algorithm is used for feature... Calculate its binary hash code: , in It is a random projection matrix. It is a random vector offset. The above operation maps query features to their corresponding hash buckets, calculates the cosine similarity precisely only within the candidate set of the same hash bucket, and returns the Top-K samples with the highest similarity. .
[0037] The specific noise reduction time step The selection strategy is as follows: in the early stage of the denoising process (when the time step is large), the retrieval and anti-attention operations are triggered at a higher frequency to guide the model to explore diverse solution spaces at the global structure level; in the later stage of the denoising process (when the time step is small), the triggering frequency is reduced to reduce interference with local details and image quality.
[0038] Step S3: Construct the anti-attention module.
[0039] like Figure 3 As shown, based on the retrieved Top-K reference samples, an anti-attention module is constructed after the first self-attention layer of the diffusion model UNet architecture to correct the original self-attention output.
[0040] (S3-1) The query matrix of the current self-attention layer is: , where n is the length of the feature sequence and d is the feature dimension; (S3-2) For the retrieved first A reference sample is taken at the current time step. Post-encoded features As a feature representation, it is obtained through the key projection matrix that shares weights with the current self-attention layer. Sum projection matrix We obtained the following respectively: ; (S3-3) Calculate the repulsion attention weight and repulsion vector for each reference sample. Average the repulsion vectors of the Top-K reference samples to obtain the final repulsion vector: , ; (S3-4) Apply rejection correction to the original self-attention output to obtain the adjusted feature representation: in, The exclusion intensity hyperparameter is used to adjust the balance between diversity and image quality. In this embodiment, it is taken as... The repulsion operation causes the generated features to actively move away from the retrieved reference sample features, guiding the denoising process to explore regions in the feature space that are not covered by existing images.
[0041] Step S4: Complete the noise reduction generation.
[0042] Features adjusted by counterattention Used for subsequent computations in UNet to complete the prediction for the current denoising time step. In non-triggered time steps, skip S2-S3 and directly execute the standard denoising steps.
[0043] This invention is based on the DDIM sampling framework and performs a total of 20 denoising steps, with the classifier free guided coefficient (CFG) set to 2.0. The complete backdiffusion process is iteratively executed for each conditional input using the above steps, ultimately outputting a synthetic image that maximizes the difference from the existing data distribution.
[0044] To verify the effectiveness of this invention, street scene image synthesis for autonomous driving was used as the verification scenario, and a comparative experiment was conducted using MagicDrive as the basic diffusion model. The input conditions were 3D geometric control information (BEV road map, 3D bounding box, and scene text description) from the nuScenes dataset, and the output was a multi-view consistent street scene image from six camera perspectives. The initial retrieval pool contained DDIM inversion features from approximately 28,000 images across 700 scenes in the nuScenes training set, pre-computed and stored at 20 denoising time steps. The Angular LSH algorithm was used, returning [data] for each retrieval. The most similar sample. Anti-attention repulsion strength. In the denoising step, select time steps such as {949, 899, 849, 799, 749, 649, 549, 450, 300, 200, 100} to trigger retrieval and anti-attention operations, while skipping other steps to ensure the quality of the generated image.
[0045] As shown in Table 1, compared with the baseline method MagicDrive, the method of the present invention improves the diversity FID index by 160% while ensuring image quality.
[0046] Table 1 This embodiment also includes an experiment to train the BEVFusion 3D object detection model using synthetic data augmentation. On the nuScenes validation set, both the mAP and NDS metrics are superior to the model trained using baseline synthetic data, verifying the effectiveness of this invention in improving the diversity and practical value of synthetic data.
[0047] Table 2 The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for synthesizing diverse images based on retrieval-enhanced diffusion, characterized in that, Includes the following steps: (1) Construct a dynamic retrieval pool containing real images and generated images, and use a pre-trained feature extractor to extract features from all images in the retrieval pool, embed them, and store them; (2) For the current image to be generated, a specific denoising time step is determined according to the preset strategy, and the Top-K reference samples most similar to the current generation state are retrieved from the retrieval pool using the approximate nearest neighbor algorithm; (3) Based on the retrieved reference samples, an anti-attention module is constructed after the self-attention layer of the generative model to make the currently generated features actively move away from the retrieved reference sample features, so as to exclude similar features and promote the diversity of generation. (4) The features after anti-attention adjustment are fed into the subsequent denoising steps of the diffusion model to complete the complete reverse diffusion process and output a synthetic image with the maximum difference from the existing data distribution.
2. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, In step (1), the update strategy of the dynamic retrieval pool is as follows: each time a new batch of images is generated, it is added to the retrieval pool and the feature index is updated accordingly; as the accumulated number of generated images in the retrieval pool increases, the subsequent generated images are subject to the exclusion constraint from a wider feature space, thereby continuously guiding the newly generated samples to explore the uncovered feature regions.
3. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, In step (1), the pre-trained feature extractor is a visual language model based on contrastive learning, a visual encoder based on self-supervised learning, or the encoder of the generative model itself.
4. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, The feature representations of each reference sample in the retrieval pool are stored aligned to the denoising time step, and the feature representation stored at each time step is in the same semantic space as the intermediate state of the generative model at the corresponding time step.
5. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 4, characterized in that, At a specific denoising time step When performing a retrieval, the intermediate state features of the current generated state at that time step are used as the query, and only those features from the same time step in the retrieval pool are compared. The corresponding feature representations are compared for similarity to ensure semantic alignment between the retrieved features and the current generated state.
6. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, In step (2), the selection strategy for specific denoising time steps is as follows: in the early stage of the denoising process, the retrieval and anti-attention operations are triggered at a higher frequency to guide the model to explore diverse solution spaces at the global structure level; in the later stage of the denoising process, the triggering frequency is reduced to reduce the negative impact on local details and the quality of the generated image.
7. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, In step (3), the calculation process of the anti-attention module is as follows: With the query matrix of the current self-attention layer For probes; For the retrieved first Feature representation of each reference sample By using the key projection matrix that shares weights with the current self-attention layer Sum projection matrix The corresponding bond matrices were obtained respectively. Sum matrix ; For each reference sample, calculate the repulsion attention weight and repulsion vector separately. Then, calculate the average of the repulsion vectors of the Top-K reference samples to obtain the final repulsion vector. and the final repulsion vector From the original self-attention output Subtracting from the middle yields the adjusted feature representation. ,in, This is the repulsion strength hyperparameter.
8. The method for synthesizing diverse images based on retrieval-enhanced diffusion according to claim 1, characterized in that, In step (3), the anti-attention module is inserted after one or more self-attention layers of the generated model, coexisting with the original self-attention layer rather than replacing it.