An intelligent vehicle extreme test scenario generation method and system based on deep learning

Abstract

The present invention discloses a method for generating extreme test scenarios for intelligent vehicles based on deep learning, including the following steps: building a hybrid adversarial generation network scenario; data collection and preprocessing; dynamic adversarial training; simulating extreme tests: an extreme test scenario generation system for intelligent vehicles based on deep learning, including a hybrid adversarial generation network model, a data input module, a preprocessing module, an edge cloud module, and an interaction module. The present invention adopts a deeply coupled model of an adversarial generation network and a variational autoencoder, and the dual-channel adversarial mechanism enables the generated scenarios to have relatively high risk controllability, and conducts dynamic adversarial training; enhances and annotates real multi-source data sets, and through a point cloud missing area feature completion algorithm, the integrity of the point cloud data is relatively high. The introduction of a social force model in dynamic obstacle modeling effectively simulates the entry of dynamic risks; the multi-modal risk assessment network improves the assessment accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle testing, and specifically to a method and system for generating extreme test scenarios of intelligent vehicles based on deep learning. Background Art

[0002] An intelligent vehicle is a product of the combination of the latest scientific and technological achievements such as electronic computers and modern automobile industry.

[0003] An intelligent vehicle is a comprehensive system integrating functions such as environmental perception, planning and decision-making, and multi-level assisted driving. It centrally applies technologies such as computers, modern sensors, information fusion, communication, artificial intelligence, and automatic control, and is a typical high-tech complex. The research on intelligent vehicles mainly focuses on improving the safety and comfort of automobiles, as well as providing an excellent human-vehicle interaction interface.

[0004] An intelligent vehicle is a vehicle that adds advanced sensors (such as radars, cameras, etc.), controllers, actuators and other devices to a general vehicle. Through the in-vehicle environmental perception system and information terminal, it realizes information exchange with people, vehicles, roads, etc., enables the vehicle to have the ability of intelligent environmental perception, can automatically analyze the safety and dangerous states of vehicle driving, and enables the vehicle to reach the destination according to people's wishes, and finally realizes the purpose of replacing people to operate.

[0005] Generally speaking, an intelligent vehicle is a new generation of vehicle that is equipped with advanced sensing systems, decision-making systems, and execution systems, uses new technologies such as information communication, Internet, big data, cloud computing, and artificial intelligence, has partial or fully autonomous driving functions, and gradually transforms from a simple transportation tool to an intelligent mobile space.

[0006] Intelligent vehicle technology is different from the generally mentioned autonomous driving technology. It refers to the autonomous driving of vehicles realized by using a variety of sensors and intelligent highway technologies.

[0007] In the Chinese invention patent with the application publication number CN115935642A, an automatic generation method and system for extreme test scenarios of intelligent vehicles based on accident information are disclosed. The method includes: obtaining the investigation information text of actual traffic accidents, first performing preprocessing, and then using natural language processing technology to extract the static elements of the actual traffic accident scenarios, including scene road network information, weather environment information, scene start and end point information, and collision basic information; initializing the to-be-generated scenario based on the extracted static elements of the actual traffic accident scenario, and using the deep deterministic gradient strategy algorithm to search for the dynamic parameter combination set of the to-be-generated scenario; taking the extracted static elements of the actual traffic accident scenario as the static elements of the to-be-generated scenario, and combining them with the searched dynamic parameter combination set to generate an extreme test scenario of an intelligent vehicle traffic accident similar to the actual traffic accident. The present invention supports the construction of an extreme test scenario library for intelligent vehicles and accelerates the implementation of its testing.

[0008] However, the above technologies mainly focus on the research of extracting existing accident extreme scenarios, emphasizing the clustering extraction of key type scenarios. Some scenario difference information will be ignored, and the automation degree of the final extreme scenario construction process is not high. Moreover, the process of training for extreme test scenarios is relatively scarce, which is likely to cause problems such as resource scarcity of extreme scenarios, low efficiency of extreme scenario generation, and incomplete coverage.

[0009] Therefore, the present invention provides an intelligent vehicle extreme test scenario generation method and system based on deep learning. Summary of the Invention

[0010] (I) Technical Problems to be Solved

[0011] Aiming at the deficiencies of the existing technologies, the present invention provides an intelligent vehicle extreme test scenario generation method and system based on deep learning. By fusing the generative adversarial network and the variational autoencoder, a hybrid adversarial network model is generated. Moreover, the collected real accident multi-source data set is enhanced and labeled to improve the integrity of the multi-source data set, generating diverse scenarios that are physically reasonable and risk-intensive, covering the long-tail distribution, thereby solving the technical problems described in the background art.

[0012] (II) Technical Solutions

[0013] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0014] An intelligent vehicle extreme test scenario generation method based on deep learning, comprising the following steps:

[0015] Construct a hybrid adversarial generation network scenario: fuse the generative adversarial network and the variational autoencoder to generate a hybrid adversarial generation network model;

[0016] Data collection and preprocessing: collect a real multi-source data set, and enhance and label the multi-source data set to generate a dense data set;

[0017] Dynamic adversarial training: input the generated dense data set into the hybrid adversarial generation network model for training, and generate high-risk extreme scenarios through a dual-channel adversarial mechanism for dynamic adversarial training;

[0018] Simulate extreme testing: input the various data of the intelligent vehicle into the trained adversarial generation network model for testing in an extreme environment.

[0019] Furthermore, the dynamic adversarial training further includes the following steps:

[0020] Multi-modal Dynamic Risk Assessment: Use a two-stream network to separately process the physical constraints and logical constraints of the scenario, then introduce an attention mechanism to weight and fuse risk factors, and then perform risk interpretability analysis to generate quantifiable risk indicators. Finally, input the quantifiable risk indicators into the hybrid adversarial generation network model to promote the generation of high-risk scenarios;

[0021] Evolutionary Algorithm Optimization and Simulation Verification: Perform multi-objective optimization on extreme scenarios, then verify through a digital twin closed-loop, and input the verification results into the hybrid adversarial generation network model.

[0022] Furthermore, the hybrid adversarial generation network model performs probability modeling of the latent space through the variational autoencoder;

[0023] Suppose, input a dense dataset , output the latent distribution parameters ; Sample to obtain the latent variable ;

[0024] Control the regularization strength of the latent space through the KL divergence constraint;

[0025] ,

[0026] Among them, ;

[0027] Among them, represents the divergence function of the space, represents the regularization coefficient, , represents the conditional distribution defined by the encoder, given the input dense dataset , the posterior distribution of the latent variable , represents the standard normal distribution: the mean is 0 and the covariance matrix is the identity matrix , represents an index to measure the difference between two probability distributions, represents the mean vector output by the encoder network, represents the standard deviation vector output by the encoder network.

[0028] Furthermore, the hybrid adversarial generation network model includes a generator, a discriminator, and a risk assessment module;

[0029] The generator is used to input the latent variable , and output the generated scenario ;

[0030] The discriminator is used to separately evaluate: physical rationality and risk level ;

[0031] The calculation of the adversarial loss function is as follows:

[0032] ,

[0033] The total loss function is a multi-objective weighted sum:

[0034] ;

[0035] Among them, represents the total loss function of the generative adversarial network, represents the calculated expected value, represents the discrimination probability of the discriminator for the dense data set of, represents that the generator outputs a generated scenario according to the latent variable, and the latent variable is the noise vector, represents the discrimination probability of the discriminator for the generated sample of, represents the dense data set score item, represents the generated data score item, and respectively represent the GAN adversarial loss weight, the KL divergence loss weight, and the risk loss weight, and the initialized fixed weights are set to , represents the total loss function, represents the divergence function of the space, represents the risk adversarial loss function.

[0036] Furthermore, the steps of the data acquisition and preprocessing are as follows:

[0037] Multi-source data acquisition: Extract accident segments from the real road test database; The acquisition parameters include: sensor data, environmental parameters, and driving behavior;

[0038] Preprocess the data: That is, filter the acquired data, detect the abnormal data in the data, eliminate and replace the abnormal data, and then normalize the data to reduce the dimension of the data;

[0039] Data augmentation and annotation: Perform 3D reconstruction and physical parameter interpolation on the accident segments to generate a dense data set, and the steps are as follows;

[0040] Extract features from the accident segments, and generate a feature map through the extracted features;

[0041] Then perform point cloud calculation on the accident segments, and predict the point cloud distribution of the missing areas;

[0042] First, divide the accident segment into regions, and complete it through the feature map near the missing region. First, normalize the feature map so that uniform reference points are generated in the missing region, and then generate predicted point cloud data points through the mean or variance of the feature map;

[0043] Environmental variable expansion: Randomly change weather parameters in the reconstructed scene; Add dynamic obstacles to enhance and annotate the data.

[0044] Furthermore, the feature map of the missing region is completed as follows:

[0045] Point cloud containing missing region , the missing region is marked as , and the local feature map of each point ;

[0046] Local feature aggregation:

[0047] For each point , within a radius of Aggregate the K-nearest neighbor features in the neighborhood:

[0048] ,

[0049] Among them, is expressed as the neighborhood point index of is expressed as the neighborhood point of is expressed as a learnable feature transformation function, is expressed as max pooling to retain significant features, is expressed as the local feature map of the missing region, is expressed as the local feature map of the neighborhood point of is expressed as the operation function of a multi-layer perceptron;

[0050] For the local feature map of the missing region Generate predicted point cloud data points by combining the mean or variance.

[0051] Furthermore, the calculation of adding dynamic obstacles is as follows:

[0052] Dynamic obstacle modeling:

[0053] ,

[0054] Among them, is expressed as the dynamic obstacle model, is expressed as the driving force of the dynamic obstacle model, is expressed as the social repulsive force of the dynamic obstacle model, Random perturbations represented as dynamic obstacle models.

[0055] Furthermore, the specific steps of the multi-modal dynamic risk assessment are as follows:

[0056] Input the feature matrix of the scene generated by the hybrid adversarial generation network model into the multi-modal dynamic risk assessment network, which contains information on physical and logical flows, and output the comprehensive risk value ;

[0057] Physical flow processing: Use 3D convolutional layers to extract spatio-temporal features and check for acceleration changes beyond the safe range;

[0058] Logical flow processing: Represent traffic participants as graph nodes and predict the relative motion and positional relationships between vehicles and other traffic participants;

[0059] Multi-head attention mechanism: 4 heads of attention are calculated in parallel, each focusing on different modal information, calculating attention weights, and generating the comprehensive risk value ;

[0060] ,

[0061] Among them, = 64 is the key vector dimension, and 4 heads of attention are calculated in parallel. It is represented as the calculated value of the query key value, which is the comprehensive risk value , is represented as the query vector, the feature vector of the input sequence, representing the information of a certain position in the current sequence. is represented as the key vector, another feature vector associated with the query vector, used to measure the correlation or matching degree between the query vector and itself. is represented as the value vector, another feature vector associated with the query vector and the key vector, usually carrying important information. is represented as the scaling factor, which is used to prevent the dot product of the query vector and the key vector from being too large in the attention calculation. is represented as the function that converts the attention weights into a probability distribution to ensure that the sum of the weights of all attention heads is 1. is represented as the attention weights. is the product of two matrices, and the result is a matrix, which is normalized by the scaling factor ;

[0062] The steps of the risk interpretability analysis are as follows:

[0063] Input the comprehensive risk value , set the high-risk scenario = 0.85, and output the interpretability report;

[0064] Counterfactual reasoning: Through backpropagation of gradients, analyze the model based on the input comprehensive risk value Draw a high-risk conclusion, calculate feature importance, and identify key risk sources;

[0065] Risk heatmap generation: Mark the high-risk area in the feature space and display the distribution of risk factors with a heatmap;

[0066] ,

[0067] Among them, represents the high-risk area, respectively represent the feature values in the input comprehensive risk value in, represents the integral variable, controlling the degree of perturbation, represents the gradient of the risk function with respect to the feature .

[0068] An intelligent vehicle extreme test scenario generation system based on deep learning, including a hybrid adversarial generation network model, a data input module, a preprocessing module, an edge cloud module, and an interaction module;

[0069] The hybrid adversarial generation network model fuses the adversarial generation network and the variational autoencoder, and then performs dynamic adversarial training to generate extreme test scenarios and conduct extreme tests on intelligent vehicles;

[0070] The data input module is used to input a multi-source data set into the hybrid adversarial generation network model for dynamic adversarial training to simulate extreme test scenarios; and to input various data of the intelligent vehicle to conduct extreme scenario test simulations;

[0071] The preprocessing module is used to perform noise reduction, anomaly removal and replacement, and normalization on the multi-source data set, and to enhance and label the multi-source data set to generate a dense data set;

[0072] The edge cloud module is used to generate and train the hybrid adversarial generation network model based on the multi-source data set, and to conduct extreme scenario test simulations after inputting various data of the intelligent vehicle, and to generate the results of extreme scenario tests, and transmit them to the interaction module through the Internet of Things or a 5G network;

[0073] The interaction module is used to display the scenario of the extreme scenario test simulation of the intelligent vehicle, and to achieve control and adjustment, set according to requirements, and receive and display the results of the extreme scenario test.

[0074] Furthermore, the calculation process of the noise reduction is as follows:

[0075] For multi-source data sets , median filtering is performed using windows of different scales, and then the filtering results of different scales are weighted and fused;

[0076] ,

[0077] Among them, represents the output data after filtering; represents the number of scales; represents the scale weight, which is used to perform weighted averaging on the filtering results of different scales; represents the result of median filtering on the data at scale , represents the scale corresponding neighborhood window;

[0078] The calculation of the abnormal value rejection and replacement is as follows:

[0079] By calculating the difference between the data and the mean divided by the standard deviation, the Z-score of the data is obtained;

[0080] ,

[0081] Among them, is the value of the multi-source data set, is the mean of the multi-source data set, is the standard deviation of the multi-source data set, points whose values are greater than 3 or less than -3 are considered outliers, and the outliers are removed;

[0082] The vacancies after outlier removal are filled with the mean, and the calculation of the mean is as follows:

[0083] ,

[0084] Among them, is the value of the non-missing data, is the number of non-missing data;

[0085] The calculation of the normalization process is as follows:

[0086] ,

[0087] Among them, is the minimum value in the multi-source data set, is the maximum value in the multi-source data set, represents the data after normalization, and the multi-source data set is scaled to the range of [0, 1].

[0088] (III) Beneficial effects

[0089] The present invention provides a method and system for generating extreme test scenarios for intelligent vehicles based on deep learning, having the following beneficial effects:

[0090] When the present invention is used, a deep coupling model is adopted with a generative adversarial network and a variational autoencoder. Through the latent space probability modeling of the variational autoencoder, a regularized distribution of scene elements is achieved. Combining the adversarial generation ability of the generative adversarial network, extremely rapid generation of extreme scenarios is realized. At the same time, the diversity of extreme scenarios can also be improved, and the dual-channel adversarial mechanism enables a relatively high risk controllability of the generated scenarios. Moreover, extremely high-risk scenarios are generated for dynamic adversarial training;

[0091] Furthermore, real multi-source data sets are enhanced and labeled, and through a point cloud missing area feature completion algorithm, a local feature aggregation and prediction generation mechanism is used to make the point cloud data have a relatively high integrity. The introduction of a social force model in dynamic obstacle modeling can effectively generate extremely high-risk scenarios and can effectively simulate the entry of dynamic risks, improving the authenticity of extremely high-risk scenarios;

[0092] And a multi-modal risk assessment network is constructed. Through the dual-stream processing of physical flow and logical flow, combined with a four-head attention mechanism, dynamic weighted fusion of risk factors is achieved, which can effectively reduce the false alarm rate and improve the assessment accuracy. The risk heat map generation module realizes the visual positioning of key risk sources;

[0093] Digital twin closed-loop verification realizes the closed-loop iteration of scenario generation and verification through an evolutionary algorithm, improving the test efficiency. Combining with the three-dimensional visualization interface of the interaction module, it supports real-time parameter adjustment and risk traceability analysis, greatly improving the test design efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a schematic flow chart of the steps of a method for generating extreme test scenarios for intelligent vehicles based on deep learning according to the present invention;

[0095] Figure 2 It is a schematic system structure diagram of a system for generating extreme test scenarios for intelligent vehicles based on deep learning according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0096] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0097] Please refer toFigure 1 , the present invention provides a method for generating extreme test scenarios for intelligent vehicles based on deep learning, including the following steps:

[0098] Construct a hybrid adversarial generation network scenario: Integrate the adversarial generation network and the variational autoencoder to generate a hybrid adversarial generation network model;

[0099] Data collection and preprocessing: Collect real multi-source datasets, and enhance and annotate the multi-source datasets to generate dense datasets;

[0100] Dynamic adversarial training: Input the generated dense datasets into the hybrid adversarial generation network model for training, and generate high-risk extreme scenarios through a dual-channel adversarial mechanism for dynamic adversarial training;

[0101] Simulate extreme testing: Input the various data of the intelligent vehicle into the trained adversarial generation network model for testing in an extreme environment.

[0102] In this embodiment, preferably, the dynamic adversarial training further includes the following steps:

[0103] Multi-modal dynamic risk assessment: Use a two-stream network to separately process the physical constraints and logical constraints of the scenario, then introduce an attention mechanism to weightedly fuse risk factors, and then perform risk interpretability analysis to generate quantifiable risk indicators. Finally, input the quantifiable risk indicators into the hybrid adversarial generation network model to promote the generation of high-risk scenarios;

[0104] Evolutionary algorithm optimization and simulation verification: Perform multi-objective optimization on the extreme scenarios, then verify through digital twin closed-loop, and input the verification results into the hybrid adversarial generation network model;

[0105] It should be noted that the two-stream network architecture realizes multi-dimensional feature decoupling analysis through the parallel processing of the physical constraint stream and the logical constraint stream. The physical stream detects hard indicators such as the acceleration mutation rate, and the logical stream predicts the vehicle-pedestrian interaction trajectory, improving the risk assessment dimension by 2.3 times compared with the single-stream network; the weighted fusion of the four-head attention mechanism improves the recognition accuracy of key risk factors; and the risk interpretability analysis locates the key risk sources through counterfactual reasoning, guiding the GAN generator to focus on high-risk features, improving the generation efficiency of high-risk scenarios. The vehicle collision probability in the generated scenarios reaches 0.87, and the heat map visualization system enables engineers to quickly locate potential risk points.

[0106] In this embodiment, preferably, the hybrid adversarial generation network model performs probability modeling of the latent space through the variational autoencoder;

[0107] Suppose, input the dense dataset , output the latent distribution parameters ; Sample the latent variables ;

[0108] Control the regularization strength of the latent space through KL divergence constraints;

[0109] ,

[0110] wherein, ;

[0111] wherein, is expressed as the divergence function of the space, is expressed as the regularization coefficient, , is expressed as the conditional distribution defined by the encoder, given the input dense dataset , the latent variable posterior distribution of, is expressed as the standard normal distribution: mean is 0 and covariance matrix is the identity matrix , is expressed as an index to measure the difference between two probability distributions, is expressed as the mean vector output by the encoder network, is expressed as the standard deviation vector output by the encoder network;

[0112] It should be noted that through the KL divergence constraint, the latent variable obeys the standard normal distribution, realizing decoupled feature expression, improving the independence between the latent space dimensions, supporting the characteristics of the single-variable adjustment scenario; enhancing the generation controllability, increasing the coverage rate of the variable sampling space; and suppressing overfitting, enabling the dynamic adjustment of the regularization coefficient, making the generation scenario diverse and efficient; the parameterized design of the mean and standard deviation output by the encoder network realizes the bidirectional mapping from the dense dataset to the latent space, reducing the reconstruction error, sampling the high-risk latent areas specifically, and increasing the generation probability of the collision scenario.

[0113] In this embodiment, preferably, the hybrid adversarial generation network model includes a generator, a discriminator, and a risk assessment module;

[0114] The generator is used to input the latent variable and output the generated scenario ;

[0115] The discriminator is used to evaluate respectively: physical rationality and risk level ;

[0116] The calculation of the adversarial loss function is as follows:

[0117] ,

[0118] The total loss function is the weighted sum of multiple objectives:

[0119] ;

[0120] where represents the total loss function of the generative adversarial network, represents the calculated expected value, represents the discrimination probability of the discriminator for the dense dataset ; represents that the generator outputs a generated scenario according to the latent variable, and the latent variable is the noise vector, represents the discrimination probability of the discriminator for the generated sample ; represents the dense dataset score term, represents the generated data score term, and represent the GAN adversarial loss weight, KL divergence loss weight, and risk loss weight respectively, and the initialized fixed weights are set to , represents the total loss function, represents the divergence function of the space, represents the risk adversarial loss function;

[0121] It should be noted that the dual-channel discrimination evaluation system detects hard indicators through a 3D convolutional network, reducing the misjudgment rate. The adversarial loss weight is adaptively adjusted with the number of training rounds, and the KL divergence weight implements temperature coefficient annealing, reducing the occurrence rate of mode collapse; and the risk-aware generation mechanism increases the generation probability of high-risk scenarios. The risk loss function design makes the high-risk value of the generated scenario account for a relatively high proportion. The weight strategy of collaborative optimization of the multi-objective dynamic balance loss function reduces the gradient conflict rate and improves the training convergence speed. And the KL divergence regularization constrains the latent space distribution, improving the physical rationality of the generated scenario;

[0122] Risk adversarial loss function The calculation formula is as follows:

[0123] ,

[0124] where represents the output probability of the pre-trained risk assessment model, and the risk value ∈ [0, 1]; the negative sign indicates that the risk value needs to be maximized, forcing the generator to generate high-risk scenarios.

[0125] Dynamic weight adjustment

[0126] Strategy: Dynamically adjust based on reinforcement learning (PPO algorithm) ;

[0127] Reward function design:

[0128] ,

[0129] Among them, = 0.6, = 0.4 is used to balance risk and diversity, and the λ value is updated every 100 iterations. The learning rate = 0.001, represents the risk assessment value of the generated scenario, represents the diversity assessment value of the generated scenario, is the high-risk value.

[0130] In this embodiment, preferably, the steps of data collection and preprocessing are as follows:

[0131] Multi-source data collection: Extract accident fragments from the real road test database; the collected parameters include: sensor data, environmental parameters, and driving behavior;

[0132] Preprocess the data: That is, filter the collected data, detect abnormal data in the data, eliminate and replace the abnormal data, and then normalize the data to reduce the dimension of the data;

[0133] Data augmentation and annotation: Perform three-dimensional reconstruction and physical parameter interpolation on the accident fragments to generate a dense data set. The steps are as follows;

[0134] Extract features from the accident fragments and generate a feature map through the extracted features;

[0135] Then perform point cloud calculation on the accident fragments and predict the point cloud distribution in the missing area;

[0136] First, divide the accident fragments into regions, and complete them through the feature maps near the missing regions. First, normalize the feature maps so that the missing regions generate uniform reference points, and then generate predicted point cloud data points through the mean or variance of the feature maps;

[0137] Environmental variable expansion: Randomly change the weather parameters in the reconstructed scene; add dynamic obstacles to enhance and annotate the data;

[0138] It should be noted that local feature aggregation realizes the completion of the missing area, generates sudden obstacles through the social force model, improves the coverage rate of dangerous scenarios, supports random combinations of weather parameters, makes the generated scenarios diverse, automatically extracts features, generates feature maps, and makes the reconstruction efficiency of accident fragments relatively high.

[0139] In this embodiment, preferably, the feature map of the missing area is completed:

[0140] Point cloud with missing regions , where the missing regions are marked as , and the local feature map of each point ;

[0141] Local feature aggregation:

[0142] For each point , within a radius neighborhood, aggregate the K-nearest neighbor features:

[0143] ,

[0144] where, is denoted as the neighborhood point index of is denoted as the neighborhood points of is denoted as a learnable feature transformation function, is denoted as max pooling to retain significant features, is denoted as the local feature map of the missing region, is denoted as the local feature map of the neighborhood points of is denoted as the operation function of the multi-layer perceptron;

[0145] For the local feature map of the missing region combine the mean or variance to generate the predicted point cloud data points;

[0146] It should be noted that in the missing region, for each point, its local feature information is extracted to form a feature map, and the feature map can include various features such as the coordinates, normal vectors, and distances of the points; for each point in the missing region, a radius neighborhood is defined, and the K-nearest neighbor features of the neighborhood points are collected. By aggregating these features, the local geometry and shape information around the neighborhood points can be captured; through feature aggregation and generation prediction, combining statistical methods and deep learning models, the problem of point cloud data completion is effectively solved, and multi-modal features can be captured to generate reasonable completion data.

[0147] In this embodiment, preferably, the calculation of adding dynamic obstacles is as follows:

[0148] Dynamic obstacle modeling:

[0149] ,

[0150] where, is denoted as the dynamic obstacle model, is denoted as the driving force of the dynamic obstacle model, The social repulsive force represented as a dynamic obstacle model The random perturbation represented as a dynamic obstacle model;

[0151] It should be noted that the social repulsive force and random perturbation simulate the uncertainty and diversity in complex traffic scenarios, generating more realistic obstacle behaviors. The driving force predicts the future behavior of obstacles through speed and acceleration, enhancing the forwardness and prediction ability of the model.

[0152] In this embodiment, preferably, the specific steps of the multi-modal dynamic risk assessment are as follows:

[0153] Input the feature matrix of the scene generated by the hybrid adversarial generation network model into the multi-modal dynamic risk assessment network, which contains information on physical and logical flows, and output the comprehensive risk value ;

[0154] Physical flow processing: Use a 3D convolutional layer to extract spatio-temporal features and check for acceleration changes beyond the safe range;

[0155] Logical flow processing: Represent traffic participants as graph nodes and predict the relative motion and positional relationships between vehicles and other traffic participants;

[0156] Multi-head attention mechanism: 4 heads of attention are calculated in parallel, each focusing on different modal information, calculating attention weights, and generating the comprehensive risk value ;

[0157] ,

[0158] Among them, =64 is the dimension of the key vector, and 4 heads of attention are calculated in parallel, Represents the calculated value of the query key value, which is the comprehensive risk value , Represents the query vector, the feature vector of the input sequence, indicating the information at a certain position of the current sequence, Represents the key vector, another feature vector associated with the query vector, used to measure the correlation or matching degree between the query vector and itself, Represents the value vector, another feature vector associated with the query vector and the key vector, usually carrying important information, Represents the scaling factor, which is used to prevent the dot product of the query vector and the key vector from being too large in the attention calculation, Represents the function that converts the attention weights into a probability distribution to ensure that the sum of the weights of all attention heads is 1, Represents the attention weight, Is the product of two matrices, and the result is a matrix. The scaling factor Normalizes it;

[0159] The risk interpretability analysis steps are as follows:

[0160] Input the comprehensive risk value Set the high-risk scenario = 0.85, and output the interpretability report;

[0161] Counterfactual reasoning: Through gradient backpropagation, analyze that the model reaches a high-risk conclusion based on the input comprehensive risk value Calculate the feature importance and find the key risk sources;

[0162] Risk heatmap generation: Mark the high-risk area in the feature space and display the distribution of risk factors with a heatmap;

[0163] ,

[0164] Among them, represents the high-risk area, respectively represent the feature values in the input comprehensive risk value in, represents the integral variable to control the perturbation degree, represents the gradient of the risk function with respect to the feature ;

[0165] It should be noted that the multi-modal dynamic risk assessment generates comprehensive and multi-dimensional risk assessment results through the processing of physical flow and logical flow, combined with the multi-head attention mechanism. The dynamic obstacle modeling simulates the behavioral logic of dynamic obstacles through parameters such as physical attributes, social repulsive forces, frontal driving forces, and random perturbations to ensure the authenticity and flexibility of the modeling. The risk interpretability analysis further provides interpretability support for high-risk scenarios through counterfactual reasoning and heatmap generation, helping to optimize the model and scenario configuration; it not only improves the model's ability to capture dynamic obstacles and risk factors, but also ensures the reliability and safety of the system in complex traffic scenarios through multi-modal fusion and interpretability analysis, and has high advantages and practical value.

[0166] Reference Figure 2 , an intelligent vehicle extreme test scenario generation system based on deep learning, including a hybrid adversarial generation network model, a data input module, a preprocessing module, an edge cloud module, and an interaction module;

[0167] The hybrid adversarial generation network model fuses the adversarial generation network and the variational autoencoder, and then conducts dynamic adversarial training to generate extreme test scenarios and perform extreme tests on intelligent vehicles;

[0168] The data input module is used to input a multi-source data set into the hybrid adversarial generation network model for dynamic adversarial training to simulate extreme test scenarios; and to input various data of the intelligent vehicle for extreme scenario test simulation.

[0169] The preprocessing module is used to perform noise reduction, anomaly rejection and replacement, and normalization on the multi-source data set, and to enhance and annotate the multi-source data set to generate a dense data set.

[0170] The edge cloud module is used to generate and train a hybrid adversarial generation network model based on the multi-source data set, and to perform extreme scenario test simulation after inputting various data of the intelligent vehicle, and to generate the results of extreme scenario tests, and transmit them to the interaction module through the Internet of Things or 5G network.

[0171] The interaction module is used to display the scenarios of extreme scenario test simulation of the intelligent vehicle, and to implement control and adjustment, set according to requirements, and receive and display the results of extreme scenario tests.

[0172] In this embodiment, preferably, the calculation process of the noise reduction is as follows:

[0173] For the multi-source data set , median filtering is performed using windows of different scales, and then the filtering results of different scales are weighted and fused.

[0174] ,

[0175] Among them, represents the output data after filtering; represents the number of scales; represents the scale of the weight, which is used to perform weighted averaging on the filtering results of different scales; represents the result of median filtering on the data at scale , represents the scale corresponding neighborhood window;

[0176] The calculation of the anomaly rejection and replacement is as follows:

[0177] By calculating the difference between the data and the mean divided by the standard deviation, the Z-score of the data is obtained.

[0178] ,

[0179] Among them, is the value of the multi-source data set, is the mean of the multi-source data set, is the standard deviation of the multi-source data set, Points with values greater than 3 or less than -3 are considered outliers and the outliers are removed;

[0180] The vacant positions after outlier removal are filled by the mean value, and the calculation of the mean value is as follows:

[0181] ,

[0182] where, is the value of non-missing data, is the number of non-missing data;

[0183] The calculation of the normalization process is as follows:

[0184] ,

[0185] where, is the minimum value in the multi-source dataset, is the maximum value in the multi-source dataset, represents the data after normalization, and the multi-source dataset is scaled to the range of [0, 1];

[0186] It should be noted that the combination of median filtering and weighted fusion can effectively remove noise and retain the effective components of the signal. Windows of different scales capture noises of different frequencies, improving the robustness of noise reduction. Through the weighted average of filtering results of different scales, the noise reduction effects of different scales are balanced, avoiding the deficiencies of single filtering. Z-score standardization makes outliers stand out significantly, facilitating rapid removal. Mean value filling is simple and effective, avoiding complex interpolation methods, suitable for large-scale data processing. Removing outliers and filling vacant positions significantly improve data quality, providing more complete data for subsequent analysis. Scaling the data to the range of [0, 1] prevents the problem of gradient explosion or disappearance, stabilizing the training process. The normalized data makes the comparison of different features more intuitive, facilitating model training and optimization; The combination of noise reduction, outlier removal and normalization addresses data noise, outliers and scale problems, improving data quality.

[0187] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVD ), or semiconductor media. The semiconductor media can be a solid-state drive.

[0188] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0189] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0190] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only for some logical function divisions, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0191] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, can also exist physically separately for each unit, or two or more units can be integrated in one unit.

[0193] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. And the aforementioned storage medium includes: U disks, mobile hard disks, read-only memories ( read - only memory , ROM ), random access memories ( random access memory , RAM ), magnetic disks or optical disks and other media that can store program codes.

[0194] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0195] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, and all should be covered by the protection scope of the present application.

Claims

1. A method for generating extreme test scenarios for smart cars based on deep learning, characterized in that: The following steps are involved: Build a hybrid generative adversarial network scenario: Use the generative adversarial network and variational autoencoder to fuse and generate a hybrid generative adversarial network model; Data collection and preprocessing: Collect real multi-source data sets, enhance and annotate them, and generate dense data sets; Dynamic adversarial training: The generated dense data set is input into the hybrid adversarial generative network model for training. High-risk extreme scenarios are generated through a dual-channel adversarial mechanism for dynamic adversarial training. Simulate extreme tests: Input various data of smart cars into the trained adversarial generative network model to conduct tests in extreme environments; The dynamic adversarial training also includes the following steps: Multimodal dynamic risk assessment: A dual-stream network is used to process the physical and logical constraints of the scene respectively. The risk factors are then weighted and integrated by introducing an attention mechanism. Then, risk interpretability analysis is performed to generate quantifiable risk indicators. Finally, the quantifiable risk indicators are input into the hybrid adversarial generative network model to promote the generation of high-risk scenarios. Evolutionary algorithm optimization and simulation verification: Multi-objective optimization is performed on extreme scenarios, and then the results are verified through a digital twin closed loop, and the verification results are input into the hybrid adversarial generative network model; The dual-stream network architecture realizes multi-dimensional feature decoupling analysis through parallel processing of physical constraint stream and logical constraint stream. The physical stream detects the hard index of acceleration mutation rate, and the logical stream predicts the vehicle-pedestrian interaction trajectory. The weighted fusion of the four-head attention mechanism locates the key risk source through counterfactual reasoning. The hybrid generative adversarial network model performs probabilistic modeling of the latent space through the variational autoencoder; Assume that the input data set is dense , output potential distribution parameters ; Sampling to obtain latent variables ; Control the regularization strength of the latent space through KL divergence constraints; ， in, ; in, Expressed as the divergence function of the space, is represented as the regularization coefficient, , Represents the conditional distribution defined by the encoder, given an input dense dataset , latent variables The posterior distribution of Represented as a standard normal distribution: the mean is 0 and the covariance matrix is ​​the identity matrix , It is expressed as an indicator to measure the difference between two probability distributions. Represented as the mean vector output by the encoder network, Denoted as the standard deviation vector of the encoder network output.

2. The method for generating extreme test scenarios for smart cars based on deep learning according to claim 1, characterized in that: The hybrid adversarial generative network model includes a generator, a discriminator and a risk assessment module; Generator is used to input latent variables , output generation scene ; The discriminator is used to evaluate: physical plausibility and risk level ; The adversarial loss function is calculated as follows: ， The total loss function is a weighted sum of multiple objectives: ； in, Expressed as the total loss function of the generative adversarial network, Expressed as the expected value of the calculation, Represented as the discriminator for dense data sets The probability of discrimination, It is represented as the generator outputting the generated scene according to the latent variable, and the latent variable is the noise vector. Represented as the discriminator to generate samples The probability of discrimination, Represented as a dense dataset score term, is represented as the generated data score term, and They are respectively represented as GAN adversarial loss weight, KL divergence loss weight and risk loss weight, and the initial fixed weights are set to , Expressed as the total loss function, Expressed as the divergence function of the space, It is expressed as risk adversarial loss function.

3. The method for generating extreme test scenarios for smart cars based on deep learning according to claim 1, characterized in that: The steps of data collection and preprocessing are as follows: Multi-source data collection: extract accident clips from the real road test database; collection parameters include: sensor data, environmental parameters and driving behavior; Preprocess the data: filter the collected data, detect abnormal data in the data, remove and replace the abnormal data, and then normalize the data to reduce the dimension of the data; Data enhancement and annotation: Perform 3D reconstruction and physical parameter interpolation on the accident fragment to generate a dense data set. The steps are as follows; Extract features from the accident fragments and generate feature maps based on the extracted features; Then, point cloud computing of the accident fragment is realized, and the point cloud distribution of the missing area is predicted; First, the accident fragment is divided into regions, and the missing regions are supplemented by feature maps near the missing regions. The feature maps are first normalized to generate uniform reference points in the missing regions, and then the predicted point cloud data points are generated by the mean or variance of the feature maps. Environmental variable expansion: Randomly change weather parameters in the reconstructed scene; add dynamic obstacles to enhance and annotate the data.

4. The method for generating extreme test scenarios for smart cars based on deep learning according to claim 3 is characterized in that: The characteristic map of the missing area is completed: Point cloud with missing areas , the missing regions are marked as , local feature map of each point ; Local feature aggregation: For each point , in radius Aggregate K nearest neighbor features within the neighborhood: ， in, Expressed as The neighborhood point index of Expressed as The neighboring points of Expressed as a learnable feature transformation function, Indicates that the maximum pooling retains significant features, Represented as the local feature map of the missing area, Expressed as The local feature map of the neighborhood points, It is expressed as the operation function of a multi-layer perceptron; Local feature maps of missing areas Combine the mean or variance to generate predicted point cloud data points.

5. The method for generating extreme test scenarios for smart cars based on deep learning according to claim 3 is characterized in that: The calculation of adding dynamic obstacles is as follows: Dynamic Obstacle Modeling: ， in, Represented as a dynamic obstacle model, Expressed as the additional driving force of the dynamic obstacle model, The social repulsion force expressed as a dynamic obstacle model, is represented as a random perturbation modulo the dynamic obstacle.

6. The method for generating extreme test scenarios for smart cars based on deep learning according to claim 1, characterized in that: The specific steps of the multimodal dynamic risk assessment are as follows: The feature matrix of the scenario generated by the hybrid adversarial generative network model is input into the multimodal dynamic risk assessment network, which contains information on physical and logical flows, and outputs a comprehensive risk value. ; Physical flow processing: Use 3D convolutional layers to extract spatiotemporal features and check whether there are acceleration changes beyond the safe range; Logical flow processing: traffic participants are represented as graph nodes, and the relative movement and position relationship between the vehicle and other traffic participants are predicted; Multi-head attention mechanism: 4 attention heads perform parallel calculations, focusing on different modal information, calculating attention weights, and generating comprehensive risk values ; ， in, =64 is the key vector dimension, 4 attention heads are calculated in parallel, Represents the calculated value of the query key value, which is the comprehensive risk value , Represented as the query vector, the feature vector of the input sequence, represents the information of a certain position in the current sequence, Represented as a key vector, another feature vector associated with the query vector, which is used to measure the relevance or matching degree of the query vector to itself. Represented as a value vector, another feature vector associated with the query vector and the key vector carries important information, Represented as a scaling factor, used in attention calculations to prevent the dot product of the query vector and the key vector from being too large. Expressed as a function, it converts the attention weights into a probability distribution, ensuring that the sum of the weights of all attention heads is 1. Denoted as attention weight, is the product of two matrices, the result is a matrix, the scaling factor Normalize it; The steps of risk interpretability analysis are as follows: The comprehensive risk value Enter and set high-risk scenarios =0.85, output interpretability report; Counterfactual reasoning: Through gradient back propagation, the analysis model is based on the input comprehensive risk value Draw high-risk conclusions, calculate feature importance, and identify key risk sources; Risk heat map generation: mark high-risk areas in the feature space and use heat maps to show the distribution of risk factors; ， in, Indicates a high-risk area. Respectively represented as input comprehensive risk value The eigenvalues ​​in Expressed as an integral variable, it controls the degree of disturbance, Expressed as risk function versus feature gradient.

7. A system for generating extreme test scenarios for smart cars based on deep learning, used to execute the method according to any one of claims 1 to 6, characterized in that: It includes a hybrid adversarial generative network model, a data input module, a preprocessing module, an edge cloud module and an interaction module; The hybrid generative adversarial network model uses a generative adversarial network and a variational autoencoder for fusion, and then performs dynamic adversarial training to generate extreme test scenarios and perform extreme tests on smart vehicles; The data input module is used to input multi-source data sets into the hybrid adversarial generative network model, perform dynamic adversarial training, and simulate extreme test scenarios; and realize the input of various data of the smart car to perform extreme scenario test simulation; The preprocessing module is used to perform noise reduction, anomaly elimination, replacement and normalization on the multi-source data set, and enhance and annotate the multi-source data set to generate a dense data set; The edge cloud module is used to generate and train a hybrid adversarial generative network model based on a multi-source data set, and is used to perform extreme scenario test simulations after inputting various data of the smart car, and to generate results of the extreme scenario tests, and transmit them to the interactive module through the Internet of Things or the 5G network; The interactive module is used to display the scenes simulated by the extreme scenario test of the smart car, and to realize control adjustment, set according to requirements, and receive and display the results of the extreme scenario test.

8. The system for generating extreme test scenarios for smart cars based on deep learning according to claim 7, characterized in that: The calculation process of the noise reduction is as follows: For multi-source datasets , use windows of different scales to perform median filtering, and then perform weighted fusion on the filtering results of different scales; ， in, Represents the output data after filtering; Indicates the number of scales; Representation scale The weights are used to perform weighted averaging of filtering results of different scales; Indicated in scale The result of median filtering the data is as follows: Representative scale The corresponding neighborhood window; The calculation of the abnormal elimination replacement is as follows: The Z-score of the data is obtained by calculating the difference between the data and the mean and dividing it by the standard deviation; ， in, is the value of the multi-source dataset, is the mean of the multi-source dataset, is the standard deviation of the multi-source dataset, Points with values ​​greater than 3 or less than -3 are considered outliers and are removed; The vacancies removed by outliers are filled by the mean, and the mean is calculated as follows: ， in, is the value of non-missing data, is the number of non-missing data; The calculation of the normalization process is as follows: ， in, is the minimum value in the multi-source dataset, is the maximum value in the multi-source dataset, Represented as normalized data, the multi-source datasets are scaled to the range of [0, 1].

Images