Long tail image generation method for automatic driving system based on heterogeneous data fusion

Abstract

The present application belongs to the technical field of automatic driving test, and in particular to a long-tail image generation method for an automatic driving system based on heterogeneous data fusion. The method comprises the following steps: S1, extracting data input information for heterogeneous data fusion; S2, selecting a pre-trained diffusion model as a heterogeneous data fusion backbone network, disassembling the inference diffusion process thereof, and constructing a first-stage inference pipeline; S3, performing refinement and enhancement processing on the intermediate fusion features of the diffusion model; S4, developing a second-stage inference pipeline; and S5, screening and deduplicating long-tail image samples, and establishing a high-quality long-tail image database. The present application can establish a long-tail image database, which can be used for efficient and high-coverage test requirements of an automatic driving perception system, and can accelerate the industrialization landing of an automatic driving vehicle.

Description

A method for generating long-tailed images in autonomous driving systems based on heterogeneous data fusion Technical Field

[0001] This invention belongs to the field of autonomous driving testing technology, specifically a method for generating long-tail images for autonomous driving systems based on heterogeneous data fusion. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous vehicles have made significant progress in many core technologies and their ability to cope with complex scenarios has been greatly enhanced. However, the driving scenarios in the real world are endless, and long-tail scenarios with low probability but high risk always limit the safe operation of autonomous vehicles. As a key part of autonomous vehicles to perceive the external environment, the importance of the perception system is self-evident. Utilizing long-tail images to test the autonomous driving perception system and continuously improve the detection capability of the perception system is a key problem that urgently needs to be solved at this stage.

[0003] However, the number of long-tail images extracted from natural driving data is scarce, making it difficult to meet the high-efficiency, high-coverage testing requirements of autonomous driving perception systems. Therefore, there is an urgent need to explore long-tail image generation methods to build a comprehensive and abundant long-tail image database. Data-driven methods can learn the data features of existing long-tail images and generate image data with feature distributions similar to the original images. However, the scarcity of long-tail images makes such methods difficult to apply. Heterogeneous data fusion, which does not rely on original long-tail image input, is a highly replaceable solution for data-driven methods. However, existing heterogeneous data fusion schemes generate long-tail images with low realism and their applicability still need improvement. Furthermore, the data annotation process for model training under this method is very cumbersome, and model training is very time-consuming, limiting the flexibility and convenience of the method's application and making it difficult to efficiently generate large amounts of long-tail image data for testing tasks of autonomous driving perception systems. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention provides a method for generating long-tail images for autonomous driving systems based on heterogeneous data fusion. The method involves extracting a set of natural driving images to be fused and a set of injected objects, and constructing heterogeneous data fusion prompts. A pre-trained diffusion model is selected as the backbone network for heterogeneous data fusion, and its inference diffusion process is deconstructed. Based on structural transfer, an inference pipeline with a weighted attention mechanism in the first stage is constructed. Based on the first-stage inference pipeline, the intermediate fusion features of the diffusion model are refined and enhanced. After feature enhancement, a second-stage inference pipeline is developed to realize the heterogeneous data fusion process. Long-tail image samples are screened and deduplicated to establish a high-quality long-tail image database.

[0005] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0006] This invention provides a method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion, comprising the following steps:

[0007] S1. Extract data input information for heterogeneous data fusion;

[0008] Using publicly available and collected natural driving data as input, images are extracted to construct a set of natural driving images to be fused. The set of injected objects is extracted through mask segmentation method, and heterogeneous data fusion prompt words are constructed.

[0009] S2. Select a pre-trained diffusion model as the backbone network for heterogeneous data fusion, decompose its inference diffusion process, and construct the first-stage inference pipeline.

[0010] A pre-trained diffusion model is selected as the backbone network to encode heterogeneous input data information. The model inference process is decomposed, and a first-stage inference pipeline containing a weighted attention mechanism is constructed to obtain the intermediate fusion features of the diffusion model.

[0011] S3. Refine and enhance the intermediate fusion features of the diffusion model;

[0012] The attention response is extracted from the injected object token to obtain the injected object attention map. Under the spatial constraints of the injected bounding box, the coarse mask of the injected object is obtained and input into the segmentation model for refinement to obtain the precise injected mask.

[0013] S4. Develop the second-stage reasoning pipeline;

[0014] Develop the second-stage inference pipeline to perform feature fusion in the latent space, that is, to fuse the source latent variables and the target latent variables in the latent space according to the exact mask;

[0015] S5. Filter and deduplicate long-tail image samples to establish a high-quality long-tail image database.

[0016] Long-tail image samples are filtered and deduplicated, and labels including category, bounding box, precise mask and source scene information are added to the generated images to establish a high-quality long-tail image database.

[0017] Furthermore, the specific method of S1 is as follows:

[0018] S11. Construct a set of natural driving images to be fused based on natural driving data;

[0019] Using publicly available natural driving data (nuScenes) and collected natural driving data as input, source scene images are extracted from them. For each frame of image, a sharpness filter is performed, and the source scene image is evaluated using Laplacian variance. The clarity is as shown in equation (1):

[0020] (1)

[0021] In the formula, Source image; For the Laplace operator; This is the variance operator; A sharpness score is given; source scene images that are below a certain threshold. Then it will be removed from the source scene image. After performing sharpness filtering, the resolution and numerical range of the images are uniformly retained, as shown in equation (2):

[0022] (2)

[0023] In the formula, To scale to the target resolution ( , ); The height of the target image; The width of the target image; This is a normalization operation; it records scene information. Ultimately with The resulting set of natural driving images to be fused is constructed by merging them. As shown in equation (3):

[0024] (3)

[0025] In the formula, For the set of natural driving images to be fused The number of samples in the sample; For sample index;

[0026] S12. Extract the set of injection objects based on mask segmentation technology;

[0027] Construct an injection object set based on natural driving data and external instance databases. Each object in the collection contains source images. Object mask and object category object mask Extraction is performed using detection techniques and segmentation models, as shown in equation (4):

[0028] (4)

[0029] In the formula, For the segmentation model, the Segment Anything Model is adopted. Input image; For the bounding box of the object; To output the mask; The vertical pixel count of the mask; The number of horizontal pixels in the mask; the mask is used to extract the foreground of the object to construct the source image. ; For element-wise multiplication; merge object categories Construct a collection of injection objects As shown in equation (5):

[0030] (5)

[0031] In the formula, The number of objects to be injected; Index for object samples;

[0032] S13, Construction of heterogeneous data fusion prompt words;

[0033] For each source image Object category and the injection region bounding box Build integrated prompts The prompt words contain object semantics, spatial relationships and realism constraints. The prompt words are generated using a structured template, as shown in Equation (6):

[0034] (6)

[0035] In the formula, For text template functions; For the bounding box of the injected region The generated relation description; This is scene information.

[0036] Furthermore, the specific method of S2 is as follows:

[0037] S21. Select a pre-trained diffusion model as the backbone network to encode heterogeneous input data information;

[0038] The FLUX pre-trained diffusion model was selected as the backbone network and its parameters were frozen. The model input received five sources of information, namely the set of natural driving images to be fused. Extracted from Text prompts Initial noise of the target Injection region boundary box With the collection of injected objects Extracted from The image input is encoded into the diffusion latent space to obtain noise-free latent variables, as shown in equation (7):

[0039] (7)

[0040] In the formula, For image encoders, a VAE encoder is used; Source scene sample The noiseless latent variables; For injecting object samples Noise-free latent variables; text prompts Injection region boundary box Object appearance information Encode them as tokens, as shown in equation (8):

[0041] (8)

[0042] In the formula, , , These are a text encoder, an injected region bounding box encoder, and an object encoder, each outputting a text-encoded token sequence. , region-coded token sequence Object encoding token sequence The initial noise of the target satisfies a standard Gaussian distribution, as shown in equation (9):

[0043] (9)

[0044] In the formula, This represents the total diffusion time step; This represents a Gaussian distribution with a mean of 0 and a variance equal to the identity matrix. Noise latent variables as the starting point for generation;

[0045] S22. Perform structural transfer from the source image to the target generated image;

[0046] Will Global structure injection of target noise For latent variables Add noise to time step As shown in equation (10):

[0047] (10)

[0048] In the formula, For the source sample at time step Noisy latent variables; This is the cumulative coefficient for noise dispatching; It is Gaussian noise; and The target starting point after structural migration is obtained through fusion, as shown in Equation (11):

[0049] (11)

[0050] In the formula, Potential variables representing the target starting point after structural transfer; Injecting intensity into the structure, the larger the value, the closer the target starting point is to the source image structure;

[0051] S23. Construct a weighted attention mechanism in the first-stage inference pipeline;

[0052] The diffusion reasoning process is broken down into two stages, the first... The first time step is the first stage of the inference pipeline, used for multi-source information injection and feature fusion formation; subsequently... The second-stage inference pipeline, defined by a time step, is used for detail convergence and latent space fusion. An attention mechanism is modeled within the first-stage inference pipeline, generating queries based on target latent variable features. The five source conditions respectively generate key-value pairs ( , ), ( , ), ( , ), ( , ), ( , And introduce different attention weight coefficients. Then we get equations (12) and (13):

[0053] (12)

[0054] (13)

[0055] In the formula, , , , , These are the weight coefficients for the text prompt, the original sample, the target noise, the bounding box region, and the injected object, respectively. For text suggestion keys; The value for the text suggestion; The original sample key; These are the original sample values; For target noise keys; The target noise value; For bounding box region keys; Values ​​for the bounding box region; For injecting object keys; The value to be injected is used; the attention calculation for the first stage of the inference pipeline is shown in equation (14):

[0056] (14)

[0057] In the formula, For attention channel dimension; This is the normalization function; These are the fused contextual features used to guide the subsequent denoising and update process.

[0058] Furthermore, the specific method of S3 is as follows:

[0059] S31. Generate a coarse mask for the injected object under the spatial constraints of the injection bounding box. ;

[0060] Generate coarse masks based on the attention response of object tokens. , used to represent the approximate region of object injection, extracts the attention of object tokens pointing to image patches from the attention matrix and aggregates them to obtain the object heatmap, as shown in Equation (15):

[0061] (15)

[0062] In the formula, The number of attention layers; For the number of attention heads; For the first Layer Attention weights of the head; Injecting attention heatmaps into objects; adding bounding boxes Convert to same-scale soft mask Spatial constraints are applied to the heatmap, and a coarse mask is obtained by thresholding, as shown in Equation (16):

[0063] (16)

[0064] In the formula, For the reason The generated soft mask; For threshold; For indicator functions; This is the resulting coarse binary mask;

[0065] S32. Using a segmentation model to process the coarse mask. Perform feature enhancement to generate a precise injection mask. ;

[0066] Coarse mask The input segmentation model SAM is refined; the target latent variables at the current time step are... Decoding yields the intermediate image ; For decoders; For the diffusion time step; With ( , The segmentation hint input model SAM constructed by the ) yields an accurate mask, as shown in Equation (17):

[0067] (17)

[0068] In the formula, For segmentation, the SAM model is adopted; This is a hint constructed from a coarse mask and a bounding box; For precise injection masking.

[0069] Furthermore, the specific method of S4 is as follows:

[0070] S41. Fuse features in the latent space;

[0071] Selecting fusion step t in the second-stage inference pipeline b The source latent variables and the target latent variables are fused in the latent space using a precise mask to preserve background details and the generated result of the injected object region to the maximum extent, as shown in Equation (18):

[0072] (18)

[0073] In the formula, To integrate time steps; For target latent variables; For source latent variables; This is element-wise multiplication; For the latent variables after fusion; Continue performing the remaining denoising process to obtain And decode to obtain the output image, the decoding process is , For the final latent variable; To generate an image.

[0074] Furthermore, the specific method of S5 is as follows:

[0075] S51. Filter and deduplicate long-tail image samples;

[0076] The batch-generated image samples are filtered to ensure the consistency of the image background and the diversity of the images. Specifically, a background consistency measurement method is defined, which calculates consistency only in non-injected regions, as shown in Equation (19):

[0077] (19)

[0078] In the formula, For source scene samples; To generate an image; It is an L1 norm; The worse the consistency between the generated image and the source scene samples, the more it exceeds a certain threshold. Then, these are removed, and the image samples are further deduplicated to extract the feature vector of the output image. If the two samples satisfy Then it is determined to be a duplicate sample and removed, where, For feature extraction networks; Cosine similarity; This is the deduplication threshold;

[0079] S52. Add tags to the images, including category, weather, and rarity, to create a long-tail image database;

[0080] Create structured label entries for each preserved image sample, including the injected object category. Injection region boundary box Precise mask and source scene information The final long-tail image database is shown in equation (20):

[0081] (20)

[0082] In the formula, This is the final long-tail image database; For long-tail image indexing; The number of image samples in the database; each image sample has an image category. Injection region boundary box Precise mask and source scene information .

[0083] The beneficial effects of this invention are as follows:

[0084] This invention proposes a long-tail image generation method for autonomous driving systems based on heterogeneous data fusion. It develops a two-stage inference pipeline for the inference process of a publicly available pre-trained diffusion model and integrates feature refinement and enhancement processing. This method achieves efficient and high-quality generation of long-tail images without requiring large-scale model training. The method involves extracting a set of natural driving images to be fused and a set of injected objects, and constructing heterogeneous data fusion prompts. A pre-trained diffusion model is selected as the backbone network for heterogeneous data fusion, and its inference diffusion process is deconstructed. Based on structural transfer, a first-stage inference pipeline containing a weighted attention mechanism is constructed. Based on the first-stage inference pipeline, the intermediate fusion features of the diffusion model are refined and enhanced. After feature enhancement, a second-stage inference pipeline is developed to realize the heterogeneous data fusion process. Long-tail image samples are screened and deduplicated to establish a high-quality long-tail image database. The results of this invention can establish a long-tail image database to meet the needs of efficient and high-coverage testing of autonomous driving perception systems, accelerating the industrialization of autonomous vehicles. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 is a flowchart of the present invention;

[0087] Figure 2 is a schematic diagram of the inference process of the pre-trained diffusion model;

[0088] Figure 3 is a schematic diagram of the long-tail image generation result 1;

[0089] Figure 4 is a schematic diagram of the long-tail image generation result 2. Detailed Implementation

[0090] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0091] Example 1

[0092] Referring to Figure 1, this invention provides a method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion, comprising the following steps:

[0093] S1. Extract data input information for heterogeneous data fusion;

[0094] Using publicly available and collected natural driving data as input, high-quality images are extracted to construct a set of natural driving images to be fused. Based on the natural driving data and an external instance database, a set of injection objects is extracted using a masking segmentation method, and corresponding heterogeneous data fusion prompts are constructed, as detailed below:

[0095] S11. Construct a set of natural driving images to be fused based on natural driving data;

[0096] Using publicly available natural driving data (nuScenes) and collected natural driving data as input, source scene images are extracted from them. For each frame of the image, a sharpness screening is performed. This invention uses Laplacian variance to evaluate the source scene image. The clarity is as shown in equation (1):

[0097] (1)

[0098] In the formula, Source image; For the Laplace operator; This is the variance operator; A sharpness score is given; source scene images that are below a certain threshold. Then it will be removed from the source scene image. After performing sharpness filtering, the resolution and numerical range of the images are uniformly retained, as shown in equation (2):

[0099] (2)

[0100] In the formula, To scale to the target resolution ( , ); The height of the target image; The width of the target image; To facilitate the normalization process, this invention records scene information to aid in subsequent prompt generation and image labeling. (Weather, lighting, road type, etc.), ultimately with The resulting set of natural driving images to be fused is constructed by merging them. As shown in equation (3):

[0101] (3)

[0102] In the formula, For the set of natural driving images to be fused The number of samples in the sample; For sample index;

[0103] S12. Extract the set of injection objects based on mask segmentation technology;

[0104] Construct an injection object set based on natural driving data and external instance databases. Each object in the collection contains source images. Object mask and object category object mask Extraction is performed using detection techniques and segmentation models, as shown in equation (4):

[0105] (4)

[0106] In the formula, For the segmentation model, the Segment Anything Model is adopted. Input image; For the bounding box of the object; To output the mask; The vertical pixel count of the mask; The number of horizontal pixels in the mask; the mask is used to extract the foreground of the object to construct the source image. ; For element-wise multiplication; merge object categories Construct a collection of injection objects As shown in equation (5):

[0107] (5)

[0108] In the formula, The number of objects to be injected; Index for object samples;

[0109] S13, Construction of heterogeneous data fusion prompt words;

[0110] For each source image Object category and the injection region bounding box Build integrated prompts The prompt words contain object semantics, spatial relationships and realism constraints. This invention uses a structured template to generate prompt words, as shown in equation (6):

[0111] (6)

[0112] In the formula, For text template functions; For the bounding box of the injected region The generated relational descriptions, such as positional relationships on the street, to the left of a car, etc. This is scene information.

[0113] S2. Select a pre-trained diffusion model as the backbone network for heterogeneous data fusion, decompose its inference diffusion process, and construct the first-stage inference pipeline.

[0114] Referring to Figure 2, a pre-trained diffusion model suitable for heterogeneous data fusion tasks is selected as the backbone network to encode heterogeneous input data information. The model's inference process is then decomposed. Based on structural transfer from the source image to the target generated image, a first-stage inference pipeline containing a weighted attention mechanism is constructed to obtain the intermediate fusion features of the diffusion model. The specific method is as follows:

[0115] S21. Select a pre-trained diffusion model as the backbone network to encode heterogeneous input data information;

[0116] This invention selects the FLUX pre-trained diffusion model as the backbone network and freezes its parameters. The model input receives five sources of information, which are the sets of natural driving images to be fused. Extracted from Text prompts Initial noise of the target Injection region boundary box With the collection of injected objects Extracted from The image input is encoded into the diffusion latent space to obtain noise-free latent variables, as shown in equation (7):

[0117] (7)

[0118] In the formula, For image encoders, a VAE encoder is used; Source scene sample The noiseless latent variables; For injecting object samples The noise-free latent variables; this invention will use text prompts Injection region boundary box Object appearance information Encode them as tokens, as shown in equation (8):

[0119] (8)

[0120] In the formula, , , These are a text encoder, an injected region bounding box encoder, and an object encoder, each outputting a text-encoded token sequence. , region-coded token sequence Object encoding token sequence The initial noise of the target satisfies a standard Gaussian distribution, as shown in equation (9):

[0121] (9)

[0122] In the formula, This represents the total diffusion time step; This represents a Gaussian distribution with a mean of 0 and a variance equal to the identity matrix. Noise latent variables as the starting point for generation;

[0123] S22. Perform structural transfer from the source image to the target generated image;

[0124] To reduce structural drift caused by diffusion randomness, structural migration is performed in the early stages of the diffusion process. Global structure injection of target noise To preserve the structural features of the source image to the greatest extent possible, latent variables... Add noise to time step As shown in equation (10):

[0125] (10)

[0126] In the formula, For the source sample at time step Noisy latent variables; This is the cumulative coefficient for noise dispatching; It is Gaussian noise; and The target starting point after structural migration is obtained through fusion, as shown in Equation (11):

[0127] (11)

[0128] In the formula, Potential variables representing the target starting point after structural transfer; Injecting intensity into the structure, the larger the value, the closer the target starting point is to the source image structure;

[0129] S23. Construct a weighted attention mechanism in the first-stage inference pipeline;

[0130] This invention breaks down the diffusion reasoning process into two stages, the first stage... The first time step is the first stage of the inference pipeline, used for multi-source information injection and feature fusion formation; subsequently... The second-stage inference pipeline, defined by a time step, is used for detail convergence and latent space fusion. An attention mechanism is modeled within the first-stage inference pipeline, generating queries based on target latent variable features. The five source conditions respectively generate key-value pairs ( , ), ( , ), ( , ), ( , ), ( , And introduce different attention weight coefficients. Then we get equations (12) and (13):

[0131] (12)

[0132] (13)

[0133] In the formula, , , , , These are the weight coefficients for the text prompt, the original sample, the target noise, the bounding box region, and the injected object, respectively. For text suggestion keys; The value for the text suggestion; The original sample key; These are the original sample values; For target noise keys; The target noise value; For bounding box region keys; Values ​​for the bounding box region; For injecting object keys; The value to be injected is used; the attention calculation for the first stage of the inference pipeline is shown in equation (14):

[0134] (14)

[0135] In the formula, For attention channel dimension; This is the normalization function; These are the fused contextual features used to guide the subsequent denoising and update process.

[0136] S3. Refine and enhance the intermediate fusion features of the diffusion model;

[0137] Referring to Figure 2, based on the intermediate fusion features of the diffusion model obtained from the first-stage inference pipeline, attention responses are extracted from the injected object token to obtain the injected object attention map. Under the spatial constraints of the injected bounding box, a coarse mask of the injected object is obtained and then input into the segmentation model for refinement to obtain the precise injection mask. The specific method is as follows:

[0138] S31. Generate a coarse mask for the injected object under the spatial constraints of the injection bounding box. ;

[0139] This invention generates a coarse mask based on the attention response of an object token. , used to represent the approximate region of object injection. Specifically, the attention of object tokens pointing to image patches is extracted from the attention matrix and aggregated to obtain the object heatmap, as shown in Equation (15):

[0140] (15)

[0141] In the formula, The number of attention layers; For the number of attention heads; For the first Layer Attention weights of the head; Injecting attention heatmaps into objects; adding bounding boxes Convert to same-scale soft mask Spatial constraints are applied to the heatmap, and a coarse mask is obtained by thresholding, as shown in Equation (16):

[0142] (16)

[0143] In the formula, For the reason The generated soft mask; For threshold; For indicator functions; This is the resulting coarse binary mask;

[0144] S32. Using a segmentation model to process the coarse mask. Perform feature enhancement to generate a precise injection mask. ;

[0145] To obtain the precise boundaries of the injected object, this invention uses a coarse mask. The input segmentation model SAM is refined; the target latent variables at the current time step are... Decoding yields the intermediate image ; For decoders; For the diffusion time step; With ( , The segmentation hint input model SAM constructed by the ) yields an accurate mask, as shown in Equation (17):

[0146] (17)

[0147] In the formula, For segmentation, the SAM model is adopted; This is a hint constructed from a coarse mask and a bounding box; For precise injection masking.

[0148] S4. Develop the second-stage reasoning pipeline;

[0149] The second-stage inference pipeline is developed, which performs feature fusion in the latent space. This involves fusing the source latent variables and the target latent variables in the latent space using a precise mask to maximize the preservation of background details and the generated results of the injected object region. The specific method is as follows:

[0150] S41. Fuse features in the latent space;

[0151] This invention selects a fusion step in the second-stage inference pipeline. The source latent variables and the target latent variables are fused in the latent space using a precise mask to preserve background details and the generated result of the injected object region to the maximum extent, as shown in Equation (18):

[0152] (18)

[0153] In the formula, To integrate time steps; For target latent variables; For source latent variables; This is element-wise multiplication; For the latent variables after fusion; Continue performing the remaining denoising process to obtain And decode to obtain the output image, the decoding process is , As the final latent variable, To generate an image.

[0154] S5. Filter and deduplicate long-tail image samples to establish a high-quality long-tail image database.

[0155] After generating long-tailed images in batches, the long-tailed image samples are filtered and deduplicated to ensure the consistency of the image background and the diversity of the image samples. Labels, including category, bounding box, precise mask, and source scene information, are added to the generated images to establish a high-quality long-tailed image database. The specific method is as follows:

[0156] S51. Filter and deduplicate long-tail image samples;

[0157] This invention filters batch-generated image samples to ensure the consistency of image backgrounds and the diversity of images. Specifically, it defines a background consistency measurement method that calculates consistency only in non-injected regions, as shown in equation (19):

[0158] (19)

[0159] In the formula, For source scene samples; To generate an image; It is an L1 norm; The worse the consistency between the generated image and the source scene samples, the more it exceeds a certain threshold. Then, these are removed, and the image samples are further deduplicated to extract the feature vector of the output image. If the two samples satisfy Then it is determined to be a duplicate sample and removed, where, For feature extraction networks; Cosine similarity; This is the deduplication threshold;

[0160] S52. Add tags to the images, including category, weather, and rarity, to create a long-tail image database;

[0161] This invention establishes structured label entries for each preserved image sample, including the injected object category. Injection region boundary box Precise mask and source scene information (Weather, lighting) to ultimately establish a high-quality long-tail image database, as shown in equation (20):

[0162] (20)

[0163] In the formula, Database refers to the final long-tail image database; For long-tail image indexing; The number of image samples in the database; each image sample has an image category. Injection region boundary box Precise mask and source scene information It can be applied to the testing process of autonomous driving perception systems to verify their performance robustness.

[0164] Example 2

[0165] This invention selects the FLUX pre-trained diffusion model as the backbone network and performs a long-tail image generation task according to the developed two-stage inference pipeline. In Figures 3 and 4, special objects such as dogs and horses, which are rare in normal driving conditions, are injected into the area in front of the vehicle. As can be seen from Figures 3 and 4, the generated images have highly realistic natural driving data features. The image connection near the injected object area is natural and there are no obvious unreasonable visual features, which proves the effectiveness of the method proposed in this invention.

[0166] In summary, the proposed method for generating long-tail images for autonomous driving systems based on heterogeneous data fusion develops a two-stage inference pipeline for the inference process of publicly available pre-trained diffusion models and integrates feature refinement and enhancement processing. This enables efficient and high-quality generation of long-tail images without requiring large-scale model training. The method involves extracting the set of natural driving images to be fused and the set of injected objects, and constructing heterogeneous data fusion prompts. A pre-trained diffusion model is selected as the backbone network for heterogeneous data fusion, and its inference diffusion process is deconstructed. Based on structural transfer, a first-stage inference pipeline containing a weighted attention mechanism is constructed. Based on the first-stage inference pipeline, the intermediate fusion features of the diffusion model are refined and enhanced. After feature enhancement, a second-stage inference pipeline is developed to realize the heterogeneous data fusion process. Long-tail image samples are screened and deduplicated to establish a high-quality long-tail image database. The results of this invention can establish a long-tail image database to meet the high-efficiency, high-coverage testing needs of autonomous driving perception systems, accelerating the industrialization of autonomous vehicles.

[0167] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating long-tail images in an autonomous driving system based on heterogeneous data fusion, characterized in that, Includes the following steps: S1. Extract data input information for heterogeneous data fusion; Using publicly available and collected natural driving data as input, images are extracted to construct a set of natural driving images to be fused. The set of injected objects is extracted through mask segmentation method, and heterogeneous data fusion prompt words are constructed. S2. Select a pre-trained diffusion model as the backbone network for heterogeneous data fusion, deconstruct its inference diffusion process, and construct the first-stage inference pipeline; select a pre-trained diffusion model as the backbone network, encode heterogeneous input data information, deconstruct the model inference process, construct the first-stage inference pipeline containing a weighted attention mechanism, and obtain the intermediate fusion features of the diffusion model. S3. Refine and enhance the intermediate fusion features of the diffusion model; extract the attention response from the injected object token to obtain the injected object attention map, obtain the coarse mask of the injected object under the spatial constraints of the injected bounding box, and input it into the segmentation model for refinement to obtain the precise injected mask. S4. Develop the second-stage inference pipeline; develop the second-stage inference pipeline to perform feature fusion in the latent space, that is, to fuse the source latent variables and the target latent variables in the latent space according to the precise mask; S5. Filter and deduplicate long-tail image samples to establish a high-quality long-tail image database; filter and deduplicate long-tail image samples, and add labels including category, bounding box, precise mask and source scene information to the generated images to establish a high-quality long-tail image database; the specific method of S1 is as follows: S11. Construct a set of natural driving images to be fused based on natural driving data; take the publicly available natural driving data nuScenes and the collected natural driving data as input, and extract the source scene images from them. For each frame of image, a sharpness filter is performed, and the source scene image is evaluated using Laplacian variance. The clarity is shown in equation (1): (1) In the formula, Source image; For the Laplace operator; This is the variance operator; Rate the clarity; Source scene images below the threshold Then it will be removed from the source scene image. After performing sharpness filtering, the resolution and numerical range of the images are uniformly retained, as shown in equation (2): (2) In the formula, To scale to the target resolution ( , )； The height of the target image; The width of the target image; For normalization operations; record scene information. Ultimately with The resulting set of natural driving images to be fused is constructed by merging them. As shown in equation (3): (3) In the formula, For the set of natural driving images to be fused The number of samples in the sample; For sample indexing; S12, extract the injection object set based on mask segmentation technology; construct the injection object set based on natural driving data and external instance database. Each object in the collection contains source images. Object mask and object category object mask Extraction is performed using detection techniques and segmentation models, as shown in equation (4): (4) In the formula, For the segmentation model, the Segment Anything Model is adopted. Input image; For the bounding box of the object; To output the mask; The vertical pixel count of the mask; The number of horizontal pixels in the mask; the mask is used to extract the foreground of the object to construct the source image. ； For element-wise multiplication; merge object categories Construct a collection of injection objects As shown in equation (5): (5) In the formula, The number of objects to be injected; Indexing of object samples; S13, construction of heterogeneous data fusion prompts; for each source image Object category and the injection region bounding box Build integrated prompts The prompt words contain object semantics, spatial relationships and realism constraints. The prompt words are generated using a structured template, as shown in Equation (6): (6) In the formula, For text template functions; For the bounding box of the injected region The generated relation description; This is scene information.

2. The method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion according to claim 1, characterized in that, The specific method of S2 is as follows: S21, Select a pre-trained diffusion model as the backbone network to encode heterogeneous input data information; Select a FLUX pre-trained diffusion model as the backbone network and freeze the parameters; The model input receives five sources of information, which are the set of natural driving images to be fused. Extracted from Text prompts Initial noise of the target Injection region boundary box With the collection of injected objects Extracted from The image input is encoded into the diffusion latent space to obtain noise-free latent variables, as shown in equation (7): In equation (7), For image encoders, a VAE encoder is used; Source scene sample The noiseless latent variables; For injecting object samples Noise-free latent variables; text prompts Injection region boundary box Object appearance information Encode them as tokens, as shown in equation (8): (8) In the formula, 、 、 These are a text encoder, an injected region bounding box encoder, and an object encoder, each outputting a text-encoded token sequence. , region-coded token sequence Object encoding token sequence The initial noise of the target satisfies a standard Gaussian distribution, as shown in equation (9): (9) In the formula, This represents the total diffusion time step; This represents a Gaussian distribution with a mean of 0 and a variance equal to the identity matrix. S22: Perform structural transfer from the source image to the target generated image, using latent noise variables as the starting point for generation. Will Global structure injection of target noise For latent variables Add noise to time step T, as shown in equation (10): In formula (10), For the source sample, the noisy latent variable at time step T; This is the cumulative coefficient for noise dispatching; It is Gaussian noise; and The target starting point after structural migration is obtained through fusion, as shown in Equation (11): In formula (11), Potential variables representing the target starting point after structural transfer; Injecting strength into the structure, a larger value indicates that the target starting point is closer to the source image structure; S23, constructing a weighted attention mechanism in the first-stage inference pipeline; decomposing the diffusion inference process into two stages, the first K time steps constitute the first-stage inference pipeline, used for multi-source information injection and fusion feature formation; the last T−K time steps constitute the second-stage inference pipeline, used for detail convergence and latent space fusion, modeling an attention mechanism in the first-stage inference pipeline, generating queries based on target latent variable features. The five source conditions respectively generate key-value pairs ( , )、( , )、( , )、( , )、( , And introduce different attention weight coefficients. Then we get equations (12) and (13): (12) In equation (13), 、 、 、 、 These are the weight coefficients for the text prompt, the original sample, the target noise, the bounding box region, and the injected object, respectively. For text suggestion keys; The value for the text suggestion; The original sample key; These are the original sample values; For target noise keys; The target noise value; For bounding box region keys; Values ​​for the bounding box region; For injecting object keys; The value to be injected is used; the attention calculation for the first stage of the inference pipeline is shown in equation (14): In formula (14), For attention channel dimension; This is the normalization function; These are the fused contextual features used to guide the subsequent denoising and update process.

3. The method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion according to claim 1, characterized in that, The specific method of S3 is as follows: S31, Generate a coarse mask of the injected object under the spatial constraints of the injection bounding box. Generate coarse masks based on the attention response of object tokens. , used to represent the approximate region of object injection, extracts the attention of object tokens pointing to image patches from the attention matrix and aggregates them to obtain the object heatmap, as shown in Equation (15): In equation (15), The number of attention layers; For the number of attention heads; Let h be the attention weight of the h-th head in the l-th layer; Injecting attention heatmaps into objects; adding bounding boxes Convert to same-scale soft mask Spatial constraints are applied to the heatmap, and a coarse mask is obtained by thresholding, as shown in Equation (16): In formula (16), For the reason The generated soft mask; For the threshold; For indicator functions; S32. Obtain the binary coarse mask; S33. Use the segmentation model to process the coarse mask. Perform feature enhancement to generate a precise injection mask. ; Coarse mask The input segmentation model SAM is refined; the target latent variables at the current time step are... Decoding yields the intermediate image ； The decoder is t; the diffusion time step is t. Will Yoyu , The segmentation hint input model SAM constructed by the ) yields an accurate mask, as shown in Equation (17): In equation (17), For segmentation, the SAM model is adopted; This is a hint constructed from a coarse mask and a bounding box; For precise injection masking.

4. The method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion according to claim 3, characterized in that, The specific method of S4 is as follows: S41, fusing features in the latent space; In the second-stage inference pipeline, a fusion step tb is selected to fuse the source latent variables and the target latent variables in the latent space using an exact mask, so as to preserve the background details and the generation results of the injected object region to the maximum extent, as shown in Equation (18): (18) In the formula, tb is the fusion time step; For target latent variables; For source latent variables; This is element-wise multiplication; For the latent variables after fusion; Continue performing the remaining denoising process to obtain And decode to obtain the output image, the decoding process is ， For the final latent variable; To generate an image.

5. The method for generating long-tail images for an autonomous driving system based on heterogeneous data fusion according to claim 1, characterized in that, The specific method of S5 is as follows: S51, screen and deduplicate long-tail image samples; screen batch-generated image samples to ensure the consistency of image background and image diversity. Specifically, define a background consistency measurement method, and calculate consistency only in non-injection areas, as shown in Equation (19): In equation (19), For source scene samples; To generate an image; It is an L1 norm; To indicate that the consistency between the generated image and the source scene samples is worse, exceeding a threshold d... bg Then, these are removed, and the image samples are further deduplicated to extract the feature vector of the output image. If the two samples satisfy Then it is determined to be a duplicate sample and removed, where, For feature extraction networks; Cosine similarity; S52: Add labels including category, weather, and rarity to the images to build a long-tailed image database; create a structured label entry for each retained image sample, including the injected object category. Injection region boundary box Precise mask and source scene information The final long-tail image database is shown in equation (20): In equation (20), This represents the final long-tail image database; n is the long-tail image index. The number of image samples in the database; each image sample has an image category. Injection region boundary box Precise mask and source scene information 。

Images