Edge scene generation method for autonomous driving perception system based on diffusion model

Abstract

The present application belongs to the technical field of automatic driving test, and particularly relates to an automatic driving perception system edge scene generation method based on a diffusion model, a natural driving scene image set and an edge element set are constructed, and key prior information such as a road mask and depth is extracted; a text controllable diffusion model is trained and adapted using the natural driving scene set to learn the natural driving scene distribution; a multi-label discriminant sub-model is trained based on the edge element set to output calibrated element confidence; in the sampling stage, text semantics and elements are fused, and regionalization injection is carried out under structural prior constraints to guide the sampling process; geometric and physical consistency verification and automatic screening form a closed loop to obtain high-fidelity edge scene image data. The present application can realize efficient testing of the performance of an automatic driving perception system, quickly excavate functional defects of the system, and accelerate the industrialization of automatic driving cars.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving testing technology, and specifically relates to a method for generating edge scenes in an autonomous driving perception system based on a diffusion model. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous vehicles have made significant progress in many core technologies and their ability to cope with complex scenarios has been greatly enhanced. However, the driving scenarios in the real world are endless, and long-tail edge scenarios with low probability but high risk always limit the safe operation of autonomous vehicles. As a key component for autonomous vehicles to perceive the external environment, the importance of the perception system is self-evident. Testing the autonomous driving perception system using edge scenarios and continuously improving the detection capability of the perception system is a key problem that urgently needs to be solved at this stage.

[0003] However, edge scenes have significant long-tail characteristics. The number of edge scene images collected and extracted in the real world is scarce and the cost is high, which is difficult to meet the data volume requirements of autonomous driving perception system testing tasks. There is an urgent need to expand the number of edge scene images. Data-driven architectures such as generative adversarial networks and variational autoencoders are the mainstream methods for obtaining a large number of edge scene images. They generate edge scene image data similar to the input samples by learning the high-dimensional distribution characteristics of edge scene data. However, edge scenes are not easy to extract and are scarce, which leads to a lack of learnable edge scene data for such methods, limiting the efficient application of such methods.

[0004] Heterogeneous data fusion is a method to avoid strong dependence on the amount of input data for edge scenes. However, there is domain offset between heterogeneous data sources, and the edge scene images generated by existing heterogeneous data fusion methods are not very realistic and have limited applicability. This invention aims to provide a new method for generating edge scenes in autonomous driving perception systems for use in testing tasks of autonomous driving perception systems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for generating edge scenes in an autonomous driving perception system based on a diffusion model, comprising the following steps:

[0006] Step 1: Based on publicly available natural driving datasets, obtain diverse natural driving scene images, clean the images, and construct a natural driving scene image set; collect edge features from public network platforms and construct an edge feature set; extract key prior information from the natural driving scene image set and the edge feature set.

[0007] Step 2: Construct the basic architecture of the text-controlled diffusion model and establish diffusion model prompt word templates; design loss functions for different training stages of the model and learn the data distribution of natural driving scene images;

[0008] Step 3: Train a multi-label discriminant sub-model and its corresponding loss function based on the edge feature set; calibrate the confidence of scene features, map the calibration probability to intensity, set a threshold as a trigger boundary, and control whether edge features are injected and the injection intensity during the model sampling stage.

[0009] Step 4: The semantic information and element information of the text are fused to obtain the noise prediction result of the multi-condition branch weighted combination. At the same time, the prior constraints of scene structure are incorporated, and the mask is regionalized and mixed in the latent space to realize the guided sampling process.

[0010] Step 5: Design evaluation metrics for the consistency of sample scale and the correctness of occlusion relationships, assess the quality of generated scene images and perform combined weighted scoring, calculate sample feature similarity, deduplicate samples, and obtain high-fidelity edge scene data based on weighted scores.

[0011] Furthermore, step one specifically includes:

[0012] 11) Construct a set of natural driving scene images;

[0013] Based on publicly available natural driving datasets, diverse natural driving scene images are obtained from them. After obtaining the natural driving scene images, low-quality, repetitive, and distorted images are cleaned and the color space is unified to construct a natural driving scene image set.

[0014] 12) Construct a set of edge elements;

[0015] Collect edge elements from public online platforms, including one or more of the following: night, rain, fog, strong glare, dogs, pedestrians, and camels. Edge elements of the same type should have different forms to construct a rich set of edge elements.

[0016] 13) Extract key prior information about the scene from the set of natural driving scene images and the set of edge features;

[0017] Semantic segmentation algorithms are used to perform semantic segmentation on natural driving scene images to obtain semantic image I. S And use this to obtain the road drivable area mask M road Background weather area mask M weather The depth map D1 corresponding to the original image is obtained using a monocular depth estimation network, and (I S M road M weather Combining D1 and D2 represents key prior information of the same natural driving image;

[0018] Semantic segmentation algorithms are used to perform semantic segmentation on edge features, thereby obtaining the semantic contour C of the edge features. S The feature depth D2 is obtained using a monocular depth estimation network, and then combined to obtain (C S ,D2), serving as key prior information for subsequent heterogeneous data fusion edge elements.

[0019] Furthermore, step two specifically includes:

[0020] 21) Construct the basic architecture of a text-controlled diffusion model;

[0021] Natural driving scene images are compressed into low-dimensional codes using a variational autoencoder. x0 is the input image. For variational autoencoders, z0 represents the latent coding representation, and a Markov process with progressively increasing noise is applied to the latent space, as shown in equation (1):

[0022] (1)

[0023] In the formula, Represents the probability distribution, z t Let I represent the noisy latent coding at time step t, I represent the unit covariance matrix, N represent the Gaussian distribution, and α represent the noisy latent coding at time step t. t α represents the signal retention rate at time step t. t =1-β t , The noise intensity is represented by T, and the total step size of the noise addition process is represented by T. This represents the cumulative signal retention rate up to time step t. α s The retention coefficient represents the original signal at the s-th time step;

[0024] In the denoising process, the gated variable g is used. Q Semantic contours C of control edge features S Combined with feature depth D2, Q=concat(C S Whether D2) participates in the denoising process is shown in equation (2):

[0025] (2)

[0026] In the formula, This is the predicted noise value. This is a backbone denoising network that relies solely on text and structural priors, where y represents the conditional control vector, P represents the structural prior, and P = concat(I S M road M weather ,D1),I S For semantic images, M roadFor the road drivable area mask, M weather D1 is the background weather area mask, and D1 is the depth map corresponding to the original image. For edge feature residual adapters, the gate variable g Q The value can be either 0 or 1; it is set to 0 during training and to 1 when generating the fused edge feature samples. , These are the parameters for their respective generation processes;

[0027] 22) Construct text-controlled diffusion model prompt word templates;

[0028] To establish stable and controllable text conditions for the diffusion model, "road, weather, scene, and traffic participants" are combined according to a template and encoded into a condition vector, which serves as the condition control vector y of the backbone denoising network, as shown in equations (3) and (4):

[0029] (3)

[0030] (4)

[0031] In the formula, Representative prompt word template function, s road s weather s scenario s participant Let e(y) represent text fragments related to roads, weather, scenes, and traffic participants, respectively, and let e(y) be the fused text embedding. Represents a text encoder, λ u s represents the weighting coefficients for different text segments. u These represent prompts for different text segments.

[0032] 23) Design training objectives for a text-controlled diffusion model;

[0033] The model training objective is split into two stages: the backbone and the adapter. During the backbone stage, when learning the distribution of natural driving scene images, only (y, P) is used for training. As shown in equation (5), when edge features are needed in the adapter stage, g will be... Q The value is 1, for training. To learn the minimum incremental changes, the learned natural driving scene image distribution is not disrupted when injecting edge feature combination Q constraints, as shown in Equation (6):

[0034] (5)

[0035] (6)

[0036] In the formula, L backbone L adapterThese represent the loss functions for the backbone stage and the adapter stage, respectively. Represents the expectation computation operator, β KL The adjustment coefficient representing the loss function of the main stage. Represents the KL divergence. Represents the encoder posterior, λ spar Represents the sparse regularization coefficients of the adapter phase. This represents Gaussian noise.

[0037] Furthermore, step three specifically includes:

[0038] 31) Construct the basic architecture of the multi-label discriminant sub-model and the corresponding loss function;

[0039] A multi-label discriminant sub-model is constructed to predict the existence probability of edge elements. The multi-label discriminant sub-model adopts an architecture of visual backbone plus multi-label classification head, as shown in equations (7) and (8):

[0040] (7)

[0041] (8)

[0042] In the formula, k is the image-level feature vector, and f w This forms the backbone of the ViT model, with x0 representing the input image. Represents the uncalibrated probability of each element. Let be the uncalibrated probability of the k-th scene element, where K is the number of element categories, and W and b are the classification header parameters, respectively. For the Sigmoid function;

[0043] The loss function of the multi-label discriminant sub-model is shown in equation (9):

[0044] (9)

[0045] In the formula, L clf The loss function representing the multi-label discriminant sub-model, y k Let α be the binary truth value of the k-th scene element. k Here, γ represents the category weights, and γ is the Focal index.

[0046] 32) Calibrate the confidence level of scene elements;

[0047] A posteriori calibration is performed using class-by-class affine temperatureization, and the calibration quality is measured by the expected calibration error. The calibration probability is mapped to the guiding intensity at sampling time, as shown in equations (10) and (11).

[0048] (10)

[0049] (11)

[0050] In the formula, Represents the calibrated probability, a k b k These are the calibration parameters, For the Sigmoid function and the logit function u1 represents the independent variable of the logit function, ECE is the expected calibration error, and B m is the confidence bucket, M is the number of samples, and acc and conf represent the in-bucket accuracy and average confidence, respectively.

[0051] The calibrated probability is mapped to intensity, and a threshold is set as the trigger boundary. The statistical data adaptively determines whether edge features are injected and the injection intensity, as shown in Equation (12):

[0052] (12)

[0053] In the formula, s k Let τ be the guiding intensity of the k-th type of edge element. k As the trigger threshold, The softplus function is the intensity scaling factor. u2 represents the argument of the softplus function.

[0054] Furthermore, step four specifically includes:

[0055] 41) Combine semantic information and element information in the text;

[0056] The noise predictions of "unconditional", "text and scene prior conditions" and "text conditions, scene prior conditions and element conditions" are weighted and combined to obtain an extended classifier-free guidance that takes into account both global semantic alignment and local element injection, as shown in Equation (13):

[0057] (13)

[0058] In the formula, Noise prediction representing a weighted combination, This represents unconditional output. This indicates that the control condition at the corresponding position is empty, z t Represents the noisy latent coding at time step t. This is the predicted noise value. This represents the output of the text and the prior conditions of the scene. The output represents textual conditions, scene prior conditions, and feature conditions. P and Q represent structural priors and marginal feature combinations, respectively. text The text guidance coefficient. Represents the set of edge elements that trigger the event. This represents adding edge feature vectors to the text prompt vector, e k Feature vectors representing edge elements;

[0059] 42) Implement regionalized injection under structural prior constraints to guide the sampling process;

[0060] Based on depth prior information, perspective transformation and scale alignment are performed on the elements, and their rasterization is used to inject pixel-level masks. Downsampling is then used to obtain latent space soft masks. Noise prediction in step 41) The "Text and Scene Prior Conditions" branch and the "Text Conditions, Scene Prior Conditions, and Feature Conditions" branch are regionalized and mixed in the latent space according to the mask to obtain noise for reverse updating. As shown in equation (14):

[0061] (14)

[0062] In the formula, η represents the injection intensity coefficient. This represents noise in the "text and scene prior conditions" branch. The noise representing the branches of "text conditions, scene prior conditions, and element conditions" is updated in reverse as shown in Equation (15):

[0063] (15)

[0064] In the formula, z t z t-1 These represent the noisy latent codes at time steps t and t-1, respectively. , These represent the cumulative signal retention rates up to time steps t and t-1, respectively.

[0065] Furthermore, step five specifically includes:

[0066] 51) Verify the scale consistency and occlusion relationship correctness of the generated samples;

[0067] The scale consistency and occlusion relationship correctness of the generated samples are evaluated by depth estimation and boundary comparison, as shown in equations (16) and (17):

[0068] (16)

[0069] (17)

[0070] In the formula, A occ These represent the accuracy rates for scale consistency error and occlusion relationship, respectively. The subscript q represents the depth estimate of the generated image. obj q bg These represent the target pixel and the background pixel, respectively, and Ω represents the injection area. The mean represents the average depth calculated from the original image and the injected features; the mean operator is used to represent the average depth. The set of sampling points representing the semantic contour boundary. Represents tolerance;

[0071] 52) Screen the generated samples;

[0072] A weighted comprehensive score is calculated based on multiple indicators, and a deduplication strategy is designed to enhance the diversity of generated samples, thereby constructing a high-fidelity and wide-coverage edge scene image set. The comprehensive weighted score is shown in Equation (18):

[0073] (18)

[0074] In the formula, S total The weighted score is represented by w1 and w2, which represent the weighting coefficients of different scoring items. The sample deduplication strategy is shown in equation (19):

[0075] (19)

[0076] In the formula, d ij This represents the feature space distance between sample i and sample j. Represents the perceptual feature extractor, and dist represents the sample feature space distance calculation function. , Representing the i-th and j-th generated samples respectively, when d ij When the value is less than the set threshold, it indicates that there are many duplicate features between the two samples, and only S in the sample is retained. total The highest number of samples.

[0077] The beneficial effects of this invention are as follows:

[0078] This invention proposes a diffusion model-based edge scene generation method for autonomous driving perception systems. It constructs a set of natural driving scene images and an edge element set, extracting key prior information such as road masks and depth. The method trains and adapts a text-controlled diffusion model using the natural driving scene set to learn the distribution of natural driving scenes. A multi-label discriminant sub-model is trained based on the edge element set, outputting calibrated element confidence scores. During the sampling phase, text semantics and elements are fused, and regional injection is performed under structural prior constraints to guide the sampling process. A closed loop is formed through geometric and physical consistency verification and automatic filtering to obtain high-fidelity edge scene image data. The edge scene images generated by this invention have higher realism and stronger applicability; they can achieve efficient testing of the performance of autonomous driving perception systems, quickly uncover system functional defects, and accelerate the industrialization of autonomous vehicles. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0080] Figure 2 This is a schematic diagram of the edge scene image sample selection process of the present invention;

[0081] Figure 3 This is a schematic diagram of the generated results for an edge scene image. Detailed Implementation

[0082] Example 1

[0083] See Figure 1 As shown in the figure, this embodiment provides a method for generating edge scenes in an autonomous driving perception system based on a diffusion model, which includes the following steps:

[0084] Step 1: Construct a set of natural driving scene images and edge feature sets, and extract key information;

[0085] Based on publicly available natural driving datasets, diverse natural driving scene images are obtained, cleaned, and a natural driving scene image set is constructed. Edge elements are collected from public network platforms, an edge element set is constructed, and key prior information from both is extracted.

[0086] Specifically as follows:

[0087] 11) Construct a set of natural driving scene images;

[0088] This invention is based on the nuScenes and CODA natural driving datasets, in which diverse natural driving scene images are obtained, including various road types such as urban, suburban, and highway, and different times of day, dusk, and night with different traffic densities. After obtaining the natural driving scene images, low-quality, repetitive, and distorted images are cleaned and the color space is unified to construct a natural driving scene image set.

[0089] 12) Construct a set of edge elements;

[0090] Since edge elements have a low probability of appearing in natural driving datasets and are difficult to obtain, this invention collects edge elements from public network platforms, including one or more of the following: night, rainy day, foggy day, strong glare, dogs, pedestrians, and camels. Edge elements of the same type should have different forms, such as different rainfall intensities, dogs with different movement postures, and pedestrians with different clothing colors, in order to construct a rich set of edge elements.

[0091] 13) Extract key prior information about the scene from the set of natural driving scene images and the set of edge features;

[0092] The semantic segmentation algorithm DeepLabV3Plus is used to perform semantic segmentation on natural driving scene images to obtain semantic image I. S And use this to obtain the road drivable area mask M road Background weather area mask M weather The depth map D1 corresponding to the original image is obtained using a monocular depth estimation network, and (I S M road M weather Combining D1 and D2 represents key prior information of the same natural driving image;

[0093] For the edge feature set, this invention also uses the DeepLabV3Plus algorithm to perform semantic segmentation on the edge features, thereby obtaining the semantic contour C of the edge features. S The feature depth D2 is obtained using a monocular depth estimation network, and the two are combined to obtain (C S ,D2), serving as key prior information for subsequent heterogeneous data fusion edge elements.

[0094] Step 2: Construct a text-controlled diffusion model to learn the data distribution of natural driving scene images;

[0095] Construct the basic architecture of the diffusion model, and on this basis, establish the diffusion model prompt word template to improve the model's controllable text generation capability. Also, design loss functions for different training stages of the model to maximize the model's learning ability for different data sources.

[0096] Specifically as follows:

[0097] 21) Construct the basic architecture of a text-controlled diffusion model;

[0098] This invention performs diffusion and denoising processes in the latent space to reduce computational costs and improve model stability. Specifically, it compresses natural driving scene images into low-dimensional codes using a variational autoencoder. x0 is the input image. For variational autoencoders, z0 represents the latent coding representation, and then a Markov process with progressively increasing noise is applied to the latent space, as shown in equation (1):

[0099] (1)

[0100] In the formula, Represents the probability distribution, z t Let I represent the noisy latent coding at time step t, I represent the unit covariance matrix, N represent the Gaussian distribution, and α represent the noisy latent coding at time step t. t α represents the signal retention rate at time step t. t =1-β t , The noise intensity is represented by T, and the total step size of the noise addition process is represented by T. This represents the cumulative signal retention rate up to time step t;

[0101] In the denoising process, the gated variable g is used. Q Semantic contours C of control edge features S Combined with feature depth D2, Q=concat(C S Whether D2) participates in the denoising process, in this step, g Q Setting it to 0, the edge feature combination Q does not affect the modeling of the natural driving scene image distribution in this step, as shown in equation (2):

[0102] (2)

[0103] In the formula, This is a backbone denoising network that relies solely on the text y and the structural prior P, where y represents the conditional control vector. For edge feature residual adapters, the gate variable g Q The value can be either 0 or 1; it is set to 0 during training and to 1 when generating the fused edge feature samples. , These are the parameters for their respective generation processes, P=concat(I S M road M weather ,D1),I S For semantic images, M road For the road drivable area mask, M weatherD1 is the background weather area mask, and D1 is the depth map corresponding to the original image. This is the predicted noise value;

[0104] 22) Construct text-controlled diffusion model prompt word templates;

[0105] This invention establishes stable and controllable text conditions for the diffusion model, combining and encoding "road, weather, scene, and traffic participants" into a condition vector according to a template, which serves as the condition control vector y of the backbone denoising network, as shown in equations (3) and (4):

[0106] (3)

[0107] (4)

[0108] In the formula, Representative prompt word template function, s road s weather s scenario s participant Text fragments representing roads, weather, scenes, and traffic participants, respectively. Represents a text encoder, λ u Here, represents the weighting coefficients for different text fragments, e(y) represents the fused text embedding, and s represents the weighting coefficients for different text fragments. u These represent prompts for different text segments.

[0109] 23) Design training objectives for a text-controlled diffusion model;

[0110] This invention splits the model training objective into two stages: the backbone and the adapter. In the backbone stage, when learning the distribution of natural driving scene images, only (y, P) is used for training. As shown in equation (5), when edge features are needed in the adapter stage, g will be... Q The value is 1, for training. To learn the minimum incremental changes, the learned natural driving scene image distribution is not disrupted when injecting edge feature combination Q constraints, as shown in Equation (6):

[0111] (5)

[0112] (6)

[0113] In the formula, L backbone L adapter These represent the loss functions for the backbone stage and the adapter stage, respectively, β KL The adjustment coefficient representing the loss function of the main stage. Represents the KL divergence. Represents the expectation computation operator, Represents the encoder posterior, λspar Represents the sparse regularization coefficients of the adapter phase. This represents Gaussian noise.

[0114] Step 3: Train the multi-label discriminant sub-model and calibrate the confidence scores of scene elements;

[0115] A multi-label discriminant sub-model and its corresponding loss function are trained based on the edge element set, which facilitates the calculation of trigger intensity during the sampling stage. At the same time, in order to make the edge element probabilities comparable, the confidence of scene elements is calibrated, the calibration probability is mapped to intensity, and a threshold is set as the trigger boundary to control whether edge elements are injected and the injection intensity during the model sampling stage.

[0116] Specifically as follows:

[0117] 31) Construct the basic architecture of the multi-label discriminant sub-model and the corresponding loss function;

[0118] This invention constructs a multi-label discriminant sub-model to predict the existence probability of edge elements, which facilitates the calculation of trigger intensity during the sampling stage. The multi-label discriminant sub-model adopts an architecture of visual backbone plus multi-label classification head, which can be trained end-to-end and is also easy to align with the text space, as shown in equations (7) and (8):

[0119] (7)

[0120] (8)

[0121] In the formula, f w This forms the backbone of the ViT model, where k is the image-level feature vector, and W and b are the classification head parameters. For the Sigmoid function, The uncalibrated probability of each element is represented by K, where K is the number of element categories and x0 is the input image.

[0122] To address the challenges of long-tailed sample distribution and enhance the detection capability of rare edge features, this invention designs a loss function for a multi-label discriminant sub-model as shown in equation (9):

[0123] (9)

[0124] In the formula, L clf The loss function representing the multi-label discriminant sub-model, y k Let α be the binary truth value of the k-th scene element. k For category weights, Let γ be the uncalibrated probability of the k-th scene element, and γ be the Focal exponent.

[0125] 32) Calibrate the confidence level of scene elements;

[0126] To ensure the comparability of edge feature probabilities, this invention employs class-by-class affine temperatureization for posterior calibration and uses the expected calibration error to measure calibration quality, mapping the calibration probability to the guiding intensity at sampling time, as shown in equations (10) and (11):

[0127] (10)

[0128] (11)

[0129] In the formula, Represents the calibrated probability, a k b k These are the calibration parameters, For the Sigmoid function and the logit function u1 represents the independent variable of the logit function, ECE is the expected calibration error, and B m is the confidence bucket, M is the number of samples, and acc and conf represent the in-bucket accuracy and average confidence, respectively.

[0130] The calibrated probability is mapped to intensity, and a threshold is set as the trigger boundary. The statistical data adaptively determines whether edge features are injected and the injection intensity, as shown in Equation (12):

[0131] (12)

[0132] In the formula, s k Let τ be the guiding intensity of the k-th type of edge element. k As the trigger threshold, The softplus function is the intensity scaling factor. u2 represents the argument of the softplus function.

[0133] Step 4: Integrate textual semantic information and element information to achieve regionalized injection under structural prior constraints, thereby guiding the sampling process;

[0134] By fusing textual semantic information and element information, a noise prediction result of multi-condition branch weighted combination is obtained. At the same time, prior constraints of scene structure are incorporated, and the mask is regionalized and mixed in the latent space to guide the sampling process.

[0135] Specifically as follows:

[0136] 41) Combine semantic information and element information in the text;

[0137] This invention weights and combines the noise predictions of "unconditional", "text and scene prior conditions", and "text conditions, scene prior conditions and element conditions" to obtain an extended classifier-free guidance that takes into account both global semantic alignment and local element injection, as shown in Equation (13):

[0138] (13)

[0139] In the formula, Noise prediction representing a weighted combination, This represents unconditional output. This indicates that the control condition at the corresponding position is empty, z t Represents the noisy latent coding at time step t. This represents the output of the text and the prior conditions of the scene. The output represents textual conditions, scene prior conditions, and feature conditions. P and Q represent structural priors and marginal feature combinations, respectively. text The text guidance coefficient. Represents the set of edge elements that trigger the event. This represents adding edge feature vectors to the text prompt vector, e k Feature vectors representing edge elements;

[0140] 42) Implement regionalized injection under structural prior constraints to guide the sampling process;

[0141] This invention performs perspective transformation and scale alignment on elements based on depth prior information, and injects them into a pixel-level raster mask, then downsamples to obtain a latent space soft mask. This leads to step 41) noise prediction. The "Text and Scene Prior Conditions" branch and the "Text Conditions, Scene Prior Conditions, and Feature Conditions" branch are regionalized and mixed in the latent space according to the mask to obtain noise for reverse update. As shown in equation (14):

[0142] (14)

[0143] In the formula, η represents the injection intensity coefficient. This represents noise in the "text and scene prior conditions" branch. The noise representing the branches of "text conditions, scene prior conditions, and element conditions" is updated in reverse as shown in Equation (15):

[0144] (15)

[0145] In the formula, z t z t-1 These represent the noisy latent codes at time steps t and t-1, respectively. , These represent the cumulative signal retention rates up to time steps t and t-1, respectively.

[0146] Step 5: Perform scale consistency and occlusion relationship verification and automatic filtering on the sampled scene images to obtain high-fidelity edge scene data;

[0147] The design evaluates the consistency of sample scale and the correctness of occlusion relationships, assesses the quality of generated samples and performs combined weighted scoring, calculates sample feature similarity, removes duplicate samples, and obtains high-fidelity edge scene data based on the weighted scores.

[0148] Specifically as follows:

[0149] 51) Verify the scale consistency and occlusion relationship correctness of the generated samples;

[0150] The generated results of model downsampling need to have strong scale consistency and correct occlusion relationships. This invention evaluates the scale consistency and occlusion relationship correctness of the generated samples by depth estimation and boundary comparison, as shown in equations (16) and (17):

[0151] (16)

[0152] (17)

[0153] In the formula, A occ These represent the accuracy rates for scale consistency error and occlusion relationship, respectively. The subscript q represents the depth estimate of the generated image. obj q bg These represent the target pixel and the background pixel, respectively, and Ω represents the injection area. The mean represents the average depth calculated from the original image and the injected features; the mean operator is used to represent the average depth. The set of sampling points representing the semantic contour boundary. Represents tolerance;

[0154] 52) Screen the generated samples;

[0155] This invention calculates a weighted comprehensive score based on multiple indicators and designs a deduplication strategy to enhance the diversity of generated samples, thereby constructing a high-fidelity and wide-coverage edge scene image set. The comprehensive weighted score is shown in Equation (18):

[0156] (18)

[0157] In the formula, S totalThe weighted score is represented by w1 and w2, which represent the weighting coefficients of different scoring items. The sample deduplication strategy is shown in equation (19):

[0158] (19)

[0159] In the formula, d ij This represents the feature space distance between sample i and sample j. Represents the perceptual feature extractor, and dist represents the feature space distance calculation function for samples. , Representing the i-th and j-th generated samples respectively, when d ij When the value is less than the set threshold, it indicates that there are many duplicate features between the two samples, and only S in the sample is retained. total The highest number of samples.

[0160] Example 2

[0161] In this embodiment, 5000 scene images are selected from a public dataset to form a natural driving scene image set, and 1000 edge element images are selected from a public network platform to form an edge element set. Based on these two, the diffusion model architecture constructed in embodiment 1 is trained on a 2*NVIDIA L20 graphics card. After training, 2000 edge scene images are generated. Finally, according to the comprehensive weighted scoring shown in equation (18), 1000 high-quality edge scene images are selected according to the score.

[0162] To verify the effectiveness of the proposed method, this embodiment extracts 1000 images from the nuScenes dataset and the CODA dataset respectively. The YOLOv8 algorithm is used to identify the 1000 images from each dataset, including those from nuScenes, CODA, and the image generated by this invention. The success rate of edge feature detection is used as an evaluation metric to measure the impact of images from different sources on the autonomous driving perception system. The detection success rates are shown in Table 1. The edge scene image generation results are available in [reference needed]. Figure 3 Pedestrians dressed in black appearing in the dark pose a significant challenge to autonomous driving perception algorithms.

[0163] Table 1. Success rate of YOLOv8 algorithm under different data sources

[0164] Data source Detection success rate 1000 images in dataset 1 0.982 1000 images in dataset 2 0.721 1000 images generated by the method of this invention 0.524

[0165] In Table 1, Dataset 1 is the nuScenes dataset and Dataset 2 is the CODA dataset. As can be seen from Table 1, the edge scene generation method for autonomous driving perception system based on diffusion model constructed in this invention can fuse heterogeneous scene data and generate high-quality edge scene images that are more challenging for autonomous driving perception systems without relying heavily on a large number of edge scene image inputs. This significantly reduces the detection success rate of the perception system, fully explores the functional limitations of perception algorithms, and verifies the effectiveness of the method proposed in this invention.

[0166] In summary, the proposed method for generating edge scenes in autonomous driving perception systems based on a diffusion model constructs a set of natural driving scene images and an edge element set, and extracts key prior information such as road masks and depth. It trains and adapts a text-controlled diffusion model using the natural driving scene set to learn the distribution of natural driving scenes; trains a multi-label discriminant sub-model based on the edge element set, and outputs calibrated element confidence scores; integrates text semantics and elements during the sampling phase, and performs regional injection under structural prior constraints to guide the sampling process; and forms a closed loop through geometric and physical consistency verification and automatic filtering to obtain high-fidelity edge scene image data. The results of this invention enable efficient testing of the performance of autonomous driving perception systems, rapid identification of system functional defects, and accelerated industrialization of autonomous vehicles.

[0167] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims.

Claims

1. A method for generating edge scenes in an autonomous driving perception system based on a diffusion model, characterized in that: Includes the following steps: Step 1: Based on publicly available natural driving datasets, obtain diverse natural driving scene images, clean the images, and construct a natural driving scene image set. Collect marginal elements from public network platforms and construct a set of marginal elements; Extract key prior information from the set of natural driving scene images and the set of edge features to obtain key prior information of the images and key prior information of the edge features; Step 2: Construct the basic framework of the text-controlled diffusion model and establish diffusion model prompt word templates; The loss function is designed to correspond to different training stages of the model, and the data distribution of natural driving scene images is learned. Step 3: Train a multi-label discriminant sub-model and its corresponding loss function based on the edge feature set; The confidence level of scene elements is calibrated, the calibration probability is mapped to the intensity, and a threshold is set as the trigger boundary to control whether edge elements are injected and the injection intensity of the text controllable diffusion model during the sampling stage. Step 4: Based on the text-controlled diffusion model obtained in Step 2, the text semantic information and element information are fused to obtain the noise prediction result of multi-condition branch weighted combination. At the same time, the depth prior information in the key prior information of the image is incorporated, and the mask is regionalized and mixed in the latent space to realize the guided sampling process. Step 5: Design evaluation metrics for the consistency of sample scale and the correctness of occlusion relationships, assess the quality of generated scene images and perform combined weighted scoring, calculate sample feature similarity, deduplicate samples, and obtain high-fidelity edge scene data based on weighted scores.

2. The method for generating edge scenes in an autonomous driving perception system based on a diffusion model according to claim 1, characterized in that: Step one specifically includes: 11) Construct a set of natural driving scene images; Based on publicly available natural driving datasets, diverse natural driving scene images are obtained from them. After obtaining the natural driving scene images, low-quality, repetitive, and distorted images are cleaned and the color space is unified to construct a natural driving scene image set. 12) Construct a set of edge elements; Collect edge elements from public online platforms, including one or more of the following: night, rain, fog, strong glare, dogs, pedestrians, and camels. Edge elements of the same type should have different forms to construct a rich set of edge elements. 13) Extract key prior information about the scene from the set of natural driving scene images and the set of edge features; Semantic segmentation algorithms are used to perform semantic segmentation on natural driving scene images to obtain semantic image I. S And use this to obtain the road drivable area mask M road Background weather area mask M weather The depth map D1 corresponding to the original image is obtained using a monocular depth estimation network, and (I S M road M weather Combining D1 and D2 represents key prior information of the same natural driving image; Semantic segmentation algorithms are used to perform semantic segmentation on edge features, thereby obtaining the semantic contour C of the edge features. S The feature depth D2 is obtained using a monocular depth estimation network, and then combined to obtain (C S ,D2), serving as key prior information for subsequent heterogeneous data fusion edge elements.

3. The method for generating edge scenes in an autonomous driving perception system based on a diffusion model according to claim 1, characterized in that: Step two specifically includes: 21) Construct the basic architecture of a text-controlled diffusion model; Natural driving scene images are compressed into low-dimensional codes using a variational autoencoder. x0 is the input image. For variational autoencoders, z0 represents the latent coding representation, and a Markov process with progressively increasing noise is applied to the latent space, as shown in equation (1): (1) In the formula, Represents the probability distribution, z t Let I represent the noisy latent coding at time step t, I represent the unit covariance matrix, N represent the Gaussian distribution, and α represent the noisy latent coding at time step t. t α represents the signal retention rate at time step t. t =1-β t , The noise intensity is represented by T, and the total step size of the noise addition process is represented by T. This represents the cumulative signal retention rate up to time step t. α s The retention coefficient represents the original signal at the s-th time step; In the denoising process, the gated variable g is used. Q Semantic contours C of control edge features S The combination of feature depth D2 and Q = concat(C) S Whether D2) participates in the denoising process is shown in equation (2): (2) In the formula, This is the predicted noise value. This is a backbone denoising network that relies solely on text and structural priors. y represents the conditional control vector, P represents key prior information about the image, and Q represents key prior information about edge features. P = concat(I S M road M weather ,D1),I S For semantic images, M road For the road drivable area mask, M weather D1 is the background weather area mask, and D1 is the depth map corresponding to the original image. For edge feature residual adapters, the gate variable g Q The value can be either 0 or 1; it is set to 0 during training and to 1 when generating the fused edge feature samples. , These are the parameters for their respective generation processes; 22) Construct text-controlled diffusion model prompt word templates; To establish stable and controllable text conditions for the diffusion model, "road, weather, scene, and traffic participants" are combined according to a template and encoded into a condition vector, which serves as the condition control vector y of the backbone denoising network, as shown in equations (3) and (4): (3) (4) In the formula, Representative prompt word template function, s road s weather s scenario s participant Let e(y) represent text fragments related to roads, weather, scenes, and traffic participants, respectively, and let e(y) be the fused text embedding. Represents a text encoder, λ u s represents the weighting coefficients for different text segments. u These represent prompts for different text segments. 23) Design training objectives for a text-controlled diffusion model; The model training objective is split into two stages: the backbone and the adapter. During the backbone stage, when learning the distribution of natural driving scene images, only (y, P) is used for training. As shown in equation (5), when edge features are needed in the adapter stage, g will be... Q The value is 1, for training. To learn the minimum incremental changes, the learned natural driving scene image distribution is not disrupted when injecting edge feature combination Q constraints, as shown in Equation (6): (5) (6) In the formula, L backbone L adapter These represent the loss functions for the backbone stage and the adapter stage, respectively. Represents the expectation computation operator, β KL The adjustment coefficient representing the loss function of the main stage. Represents the KL divergence. Represents the encoder posterior, λ spar Represents the sparse regularization coefficients of the adapter phase. This represents Gaussian noise.

4. The method for generating edge scenes in an autonomous driving perception system based on a diffusion model according to claim 1, characterized in that: Step three specifically includes: 31) Construct the basic architecture of the multi-label discriminant sub-model and the corresponding loss function; A multi-label discriminant sub-model is constructed to predict the existence probability of edge elements. The multi-label discriminant sub-model adopts an architecture of visual backbone plus multi-label classification head, as shown in equations (7) and (8): (7) (8) In the formula, k is the image-level feature vector, and f w This forms the backbone of the ViT model, with x0 representing the input image. Represents the uncalibrated probability of each element. Let be the uncalibrated probability of the k-th scene element, where K is the number of element categories, and W and b are the classification header parameters, respectively. For the Sigmoid function; The loss function of the multi-label discriminant sub-model is shown in equation (9): (9) In the formula, L clf The loss function representing the multi-label discriminant sub-model, y k Let α be the binary truth value of the k-th scene element. k Here, γ represents the category weights, and γ is the Focal index. 32) Calibrate the confidence level of scene elements; A posteriori calibration is performed using class-by-class affine temperatureization, and the calibration quality is measured by the expected calibration error. The calibration probability is mapped to the guiding intensity at sampling time, as shown in equations (10) and (11). (10) (11) In the formula, Represents the calibrated probability, a k b k These are the calibration parameters, For the Sigmoid function and the logit function u1 represents the independent variable of the logit function, ECE is the expected calibration error, and B m is the confidence bucket, M is the number of samples, and acc and conf represent the in-bucket accuracy and average confidence, respectively. The calibrated probability is mapped to intensity, and a threshold is set as the trigger boundary. The statistical data adaptively determines whether edge features are injected and the injection intensity, as shown in Equation (12): (12) In the formula, s k Let τ be the guiding intensity of the k-th type of edge element. k As the trigger threshold, The softplus function is the intensity scaling factor. u2 represents the argument of the softplus function.

5. The method for generating edge scenes in an autonomous driving perception system based on a diffusion model according to claim 1, characterized in that: Step four specifically includes: 41) Combine semantic information and element information in the text; The noise predictions of "unconditional", "text and scene prior conditions" and "text conditions, scene prior conditions and element conditions" are weighted and combined to obtain an extended classifier-free guidance that takes into account both global semantic alignment and local element injection, as shown in Equation (13): (13) In the formula, Noise prediction representing a weighted combination, This represents unconditional output. This indicates that the control condition at the corresponding position is empty, z t Represents the noisy latent coding at time step t. This is the predicted noise value. This represents the output of the text and the prior conditions of the scene. The output represents textual conditions, scene prior conditions, and feature conditions. P and Q represent key prior information of the image and key prior information of edge features, respectively. text The text guidance coefficient. Represents the set of edge elements that trigger the event. This represents adding edge feature vectors to the text prompt vector, e k Feature vectors representing edge elements; 42) Implement regionalized injection under structural prior constraints to guide the sampling process; Based on depth prior information, perspective transformation and scale alignment are performed on the elements, and their rasterization is used to inject pixel-level masks. Downsampling is then used to obtain latent space soft masks. Noise prediction in step 41) The "Text and Scene Prior Conditions" branch and the "Text Conditions, Scene Prior Conditions, and Feature Conditions" branch are regionalized and mixed in the latent space according to the mask to obtain noise for reverse updating. As shown in equation (14): (14) In the formula, η represents the injection intensity coefficient. This represents noise in the "text and scene prior conditions" branch. The noise representing the branches of "text conditions, scene prior conditions, and element conditions" is updated in reverse as shown in Equation (15): (15) In the formula, z t z t-1 These represent the noisy latent codes at time steps t and t-1, respectively. , These represent the cumulative signal retention rates up to time steps t and t-1, respectively.

6. The method for generating edge scenes in an autonomous driving perception system based on a diffusion model according to claim 1, characterized in that: Step five specifically includes: 51) Verify the scale consistency and occlusion relationship correctness of the generated samples; The scale consistency and occlusion relationship correctness of the generated samples are evaluated by depth estimation and boundary comparison, as shown in equations (16) and (17): (16) (17) In the formula, A occ These represent the accuracy rates for scale consistency error and occlusion relationship, respectively. The subscript q represents the depth estimate of the generated image. obj q bg These represent the target pixel and the background pixel, respectively, and Ω represents the injection area. The mean represents the average depth calculated from the original image and the injected features; the mean operator is used to represent the average depth. The set of sampling points representing the semantic contour boundary. Represents tolerance; 52) Screen the generated samples; A weighted comprehensive score is calculated based on multiple indicators, and a deduplication strategy is designed to enhance the diversity of generated samples, thereby constructing a high-fidelity and wide-coverage edge scene image set. The comprehensive weighted score is shown in Equation (18): (18) In the formula, S total The weighted score is represented by w1 and w2, which represent the weighting coefficients of different scoring items. The sample deduplication strategy is shown in equation (19): (19) In the formula, d ij This represents the feature space distance between sample i and sample j. Represents the perceptual feature extractor, and dist represents the feature space distance calculation function for samples. , Representing the i-th and j-th generated samples respectively, when d ij When the value is less than the set threshold, it indicates that there are many duplicate features between the two samples, and only S in the sample is retained. total The highest number of samples.

Images