Autonomous driving corner case dynamic generation and algorithm optimization system and method based on spatiotemporal data

Abstract

The application relates to the technical field of intelligent networked vehicles and automatic driving, in particular to an automatic driving Corner Case dynamic generation and algorithm optimization system and method based on space-time data, which comprises vehicle-end data monitoring, cloud-end processing, scene generation, simulation testing and optimization; in the vehicle-end data monitoring, abnormal markers are triggered in real time according to the perception missing detection rate, prediction trajectory error and control delay of the automatic driving system; in the cloud-end processing, high-value scenes are screened based on KL divergence and mutual information entropy; in the scene generation, a physical constraint loss term is introduced into a generative adversarial network to constrain the physical motion parameters of the moving targets in the generated scenes to conform to preset physical laws, so that optimal scenes are obtained; the simulation testing and optimization layer positions algorithm short boards according to a test report, generates optimization suggestions and dynamically adjusts an acquisition strategy; through a data feedback module, the newly-added high-value scenes are fed back to the cloud end to update a dynamic scene library, and a complete closed loop of "data mining-scene generation-algorithm testing-strategy adjustment-data feedback" is constructed.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicles and autonomous driving technology, specifically to an autonomous driving corner case dynamic generation and algorithm optimization system and method based on spatiotemporal data. It is applicable to the robustness improvement and efficient iteration of autonomous driving algorithms and can adapt to the R&D needs of different levels of autonomous driving systems. Background Technology

[0002] The safety and reliability of autonomous driving algorithms heavily rely on Corner Case data. This type of data encompasses complex weather conditions, such as heavy rain and blizzards; sudden traffic incidents, such as pedestrians crossing the road and vehicles cutting in; and unconventional road environments, such as signal interruptions in tunnels and extreme scenarios like high-altitude low temperatures. This data is crucial for exposing shortcomings in algorithmic perception, such as missed detections and decision-making delays. However, existing technologies suffer from three core pain points:

[0003] 1. Low efficiency and high cost in acquiring corner cases: Corner cases are inherently characterized by low probability and high dispersion. Traditional methods rely on large-scale road testing plus manual annotation: fewer than 50 valid cases are collected per vehicle per year, requiring hundreds of test vehicles to cover different areas, resulting in extremely high manpower and time costs; at the same time, manual annotation is affected by subjective experience, with an accuracy rate of less than 70% in identifying "false anomalies caused by sensor noise," such as camera lens dirt being misjudged as obstacles, and invalid data accounting for more than 30%.

[0004] 2. Lack of realism and specificity in scene generation: Existing data augmentation solutions, such as simple image flipping and point cloud rotation, can only expand the amount of data and cannot generate complex scenes such as "snowy weather + unprotected left turn + cyclist crossing"; some generative models, such as traditional GANs, do not consider traffic physics, such as pedestrian walking speed ≤1.8m / s and vehicle braking distance ≥v² / (2μg); resulting in problems such as pedestrian running speed of 5m / s and vehicle right-angle turns in the generated scenes that do not conform to real working conditions and cannot be used for algorithm testing.

[0005] 3. Disconnection between algorithm optimization and data mining, lack of closed loop: The existing solution has not established a linkage mechanism of "data mining - scenario generation - algorithm testing - data collection strategy adjustment". The shortcomings of the algorithm change dynamically in different R&D stages, such as the perception optimization period and the decision iteration period, but the case generation is still executed according to fixed rules. This results in "repeated generation of scenarios that have been solved by the algorithm, and insufficient coverage of unsolved shortcomings", resulting in low iteration efficiency and extending the deployment cycle of autonomous driving technology to more than 5 years.

[0006] While there have been attempts within the industry to expand the scope of road testing and generate static scenes, none have broken through the triple limitations of "efficiency, realism, and closed loop." Therefore, there is an urgent need for a dynamic, efficient, and accurate Corner Case mining and algorithm optimization system. Summary of the Invention

[0007] In view of this, the present invention proposes a dynamic generation and algorithm optimization system and method for autonomous driving Corner Case based on spatiotemporal data; to solve the technical problems in the existing autonomous driving Corner Case mining and generation process, such as difficulty in covering long-tail scenes due to reliance on a large number of road tests, high data screening costs, difficulty in data sharing under privacy protection, and insufficient realism of simulation scenes.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] An autonomous driving corner case dynamic generation and algorithm optimization system based on spatiotemporal data includes a vehicle-side data monitoring layer, a cloud processing layer, a scene generation layer, and a simulation testing and optimization layer;

[0010] The vehicle-side data monitoring layer includes a multi-sensor acquisition module, a consistency verification module, and an anomaly marking and uploading module. The multi-sensor acquisition module synchronously acquires sensor data through LiDAR, cameras, millimeter-wave radar, and inertial measurement units, and synchronously acquires the perception results, predicted trajectories, and control command sequences output by the autonomous driving system as algorithm decision data. The consistency verification module is used to perform cross-sensor target matching and dynamic feature stability verification on the acquired data to identify and eliminate invalid data. The anomaly marking and uploading module marks the time period as an abnormal scene based on the perception miss rate, predicted trajectory error, and control delay of the autonomous driving system. When any of these values ​​exceeds a preset threshold, the module encapsulates the synchronous data before and after the preset time period, encrypts it, and uploads it to the cloud processing layer as a candidate scene.

[0011] The cloud processing layer includes a dual-index screening module, a feature dimensionality reduction module, and a federated aggregation node. The dual-index screening module receives candidate scenes and simultaneously filters them based on scene scarcity and algorithm understanding to obtain high-value scenes. Scene scarcity is measured by KL divergence to determine the difference in feature distribution between the candidate scene and existing scenes in the dynamic scene library, while algorithm understanding is measured by mutual information entropy to determine the correlation strength between sensor data and algorithm decisions. The feature dimensionality reduction module performs feature dimensionality reduction on the scenes filtered by the dual-index screening module, and stores them in the dynamic scene library after building an index according to a preset classification system. The federated aggregation node performs horizontal federated learning with multiple automakers' clouds. It only receives encrypted scene distribution statistics and algorithm failure mode percentage statistics uploaded by each automaker, and performs dynamic weighted aggregation based on the distribution of long-tail scenes provided by each automaker to generate a global scene distribution report. The report is then distributed to each automaker's cloud to guide their data collection strategies.

[0012] The scene generation layer includes a large-model semantic understanding module, a generative adversarial network (GAN) module, and a multi-objective optimization module. The large-model semantic understanding module acquires sensor data and algorithm decisions corresponding to high-value scenes output by the dual-index screening module, converts them into natural language descriptions, extracts environment, road conditions, target behavior, and algorithm failure points as core elements, and generates derived elements as semantic constraints through reasoning. The GAN module generates initial derived scene data in batches based on the core elements and semantic constraints by introducing a physical constraint loss term. The physical constraint loss term is used to constrain the physical motion parameters of moving targets in the generated scene to conform to preset physical laws. The multi-objective optimization module filters the initial derived scene data using the NSGA-II algorithm, with scene realism and algorithm challenge as optimization objectives, to obtain the optimal simulation scene.

[0013] The simulation testing and optimization layer includes a simulation testing module, an optimization adjustment module, and a data feedback module. The simulation testing module is configured to obtain the autonomous driving algorithm to be optimized and the optimal simulation scenario output by the scenario generation layer. Based on the optimal simulation scenario, the autonomous driving algorithm to be optimized is tested, the algorithm performance indicators are recorded, and a test report is generated according to the preset "threshold benchmarking - process decomposition - root cause attribution" rules to locate the algorithm's shortcomings.

[0014] The optimization and adjustment module generates algorithm optimization suggestions based on the test report, according to the perception layer, prediction layer, and decision control layer, and outputs them to the algorithm development end. At the same time, it dynamically adjusts the collection strategy of mass-produced vehicles. The data feedback module is used to obtain the data corresponding to the high-value scenarios newly collected by mass-produced vehicles according to the adjusted collection strategy, and feeds it back to the cloud processing layer to perform dual-index screening. After the screening is passed, it is updated to the dynamic scenario library.

[0015] Furthermore, in the vehicle-side data monitoring layer:

[0016] The cross-sensor target matching uses the RANSAC algorithm to align the target contours of the lidar point cloud and the camera image. In a single frame of data, when the cross-union ratio is less than 0.5, it is determined to be sensor failure data and is excluded.

[0017] The dynamic feature stability verification is performed by fusing the target velocity information from the millimeter-wave radar and the inertial measurement unit using Kalman filtering. When the velocity fluctuation is greater than 2 m / s, it is marked as dynamic noise and not included in the candidates.

[0018] The preset thresholds include a detection miss rate greater than 10%, a predicted trajectory error greater than 0.5m, and a control delay greater than 0.5s;

[0019] The preset time before and after the abnormal moment is 5 minutes before and after, and the compressed size does not exceed 100MB. It is uploaded in encrypted form via 4G or 5G network.

[0020] Furthermore, in the cloud processing layer:

[0021] The dual-index screening module determines candidate scenarios as high-value scenarios when the KL divergence is greater than or equal to 0.6 and the mutual information entropy is less than or equal to 0.3.

[0022] The feature dimensionality reduction processing module uses the t-SNE algorithm to reduce the 256-dimensional features to 32-dimensional features, and builds an index according to weather-road-event classification, with a query latency of no more than 100ms;

[0023] The federated aggregation nodes use the FedAvg algorithm for aggregation. Long-tail scenarios are those with a global share of less than 5%. During dynamic weighted aggregation, the aggregation weight of the provider is increased to 3 times. Encryption uses the AES-256 encryption algorithm.

[0024] Furthermore, in the scene generation layer:

[0025] The generative adversarial network is an improved Wasserstein generative adversarial network, constructed by introducing a physical constraint loss term into the traditional Wasserstein generative adversarial network. Its total loss function is:

[0026] L total =L Wasserstein +λL physics ;

[0027] Among them, L Wasserstein Wasserstein distance loss is used to fit the true data distribution; L physics The physical constraint loss is used to constrain pedestrian walking speeds to no more than 1.8 m / s and vehicle braking distances to no less than [a certain value]. v is the vehicle speed, μ is the friction coefficient, g is the gravitational acceleration; λ > 0 is the balance coefficient, used to balance the distribution fitting accuracy and physical compliance; the generative adversarial network is trained using FP16 mixed-precision training;

[0028] The multi-objective optimization algorithm is the NSGA-II non-dominated sorting genetic algorithm. It quantifies the authenticity score by measuring the similarity between the generated scenario and the real road test scenario, and quantifies the challenge of the algorithm by perceiving the false negative rate and decision delay. The optimal simulation scenario obtained by screening satisfies the requirement that the authenticity score is not less than 80 points and the algorithm failure probability is not less than 70%.

[0029] Furthermore, in the simulation testing and optimization layer:

[0030] The simulation platform is a high-fidelity simulation platform based on Prescan or Simulink, containing no less than 100 road models and no less than 30 weather models;

[0031] The algorithm performance metrics include algorithm pass rate, perception accuracy, and decision latency.

[0032] The dynamic adjustment strategy for mass-produced vehicles includes: setting the scenario as a high priority and increasing the acquisition frequency by 3 times when the algorithm pass rate for a certain scenario is below 80%; and adjusting sensor parameters according to regional characteristics, including adjusting the LiDAR acquisition frequency from 10Hz to 15Hz and the camera exposure time from 1 / 500s to 1 / 300s in snowy areas.

[0033] A method for dynamic generation and algorithm optimization of autonomous driving corner cases based on spatiotemporal data includes the following steps:

[0034] S1. Vehicle-side data monitoring: Sensor data is simultaneously collected through LiDAR, cameras, millimeter-wave radar, and inertial measurement units, and the perception results, predicted trajectories, and control command sequences output by the autonomous driving system are also collected as algorithm decision data. Cross-sensor target matching and dynamic feature stability verification are performed on the collected data to identify and eliminate invalid data. Based on the perception miss rate, predicted trajectory error, and control delay of the autonomous driving system, when any of these values ​​exceeds a preset threshold, the time period is marked as an abnormal scene, and the synchronous data before and after the preset time period is encapsulated and encrypted and uploaded to the cloud as a candidate scene.

[0035] S2. Cloud Processing Steps: Receive candidate scenes and simultaneously filter them based on scene scarcity and algorithm understanding to obtain high-value scenes. Scene scarcity is measured by KL divergence to determine the difference in feature distribution between the candidate scene and existing scenes in the dynamic scene library, while algorithm understanding is measured by mutual information entropy to determine the correlation strength between sensor data and algorithm decisions. The selected high-value scenes undergo feature dimensionality reduction and are indexed according to a preset classification system before being stored in the dynamic scene library. Horizontal federated learning is performed through federated aggregation nodes and multiple automakers' clouds. Encrypted scene distribution statistics and algorithm failure mode percentage statistics uploaded by each automaker are received. Dynamic weighted aggregation is performed based on the distribution of long-tail scenes provided by each automaker to generate a global scene distribution report, which is then distributed to each automaker's cloud to guide their data collection strategies.

[0036] S3. Scene Generation: Acquire sensor data and algorithm decisions corresponding to high-value scenes selected through dual-index screening, call a large model to convert them into natural language descriptions, extract environment, road conditions, target behavior, and algorithm failure points as core elements, and generate derived elements as semantic constraints through reasoning; based on the core elements and semantic constraints, use a generative adversarial network with a physical constraint loss term to generate initial derived scene data in batches; the physical constraint loss term is used to constrain the physical motion parameters of moving targets in the generated scene to conform to preset physical laws; the initial derived scene data is screened through a multi-objective optimization algorithm, with scene realism and algorithm challenge as optimization objectives, to obtain the optimal simulation scene;

[0037] S4. Simulation Testing and Optimization: Obtain the autonomous driving algorithm to be optimized and the optimal simulation scenario obtained in step S3. Test the autonomous driving algorithm to be optimized based on the optimal simulation scenario, record the algorithm performance indicators, and generate a test report to locate algorithm shortcomings. Generate algorithm optimization suggestions based on the test report and output them to the algorithm development end. At the same time, dynamically adjust the data collection strategy of mass production vehicles. Obtain the data corresponding to the high-value scenarios newly collected by mass production vehicles according to the adjusted data collection strategy, and feed it back to the cloud to perform dual-indicator screening. After the screening is passed, return to the cloud processing steps to update the dynamic scenario library, forming a closed loop.

[0038] Compared with the prior art, this application has the following beneficial effects:

[0039] 1. Significantly improves corner case acquisition efficiency and reduces collection costs. At the vehicle-side data monitoring layer, this application triggers real-time anomaly marking based on three algorithm failure indicators: perception miss rate, predicted trajectory error, and control delay of the autonomous driving system. This transforms the traditional passive collection mode into an active collection mode driven by algorithm failure, eliminating false anomaly data at the source. During cloud processing, a dual-indicator screening module simultaneously filters high-value scenarios based on scenario scarcity (KL divergence) and algorithm understanding (mutual information entropy), ensuring that the scenarios included in the database possess both scarcity and challenge. Compared to existing technologies that rely on manual annotation and have an invalid data ratio exceeding 30%, this application significantly improves the effective case output rate per unit mileage and reduces collection costs.

[0040] 2. Achieving realistic composite scene generation that conforms to physical laws. This application's scene generation uses a large-model semantic understanding module to extract core elements of high-value scenes and infer derived elements as semantic constraints, overcoming the technical deficiency of existing solutions in generating composite scenes. By introducing a generative adversarial network with a physical constraint loss term, the physical motion parameters of moving targets in the generated scene are constrained to conform to preset physical laws, avoiding the generation of invalid scenes such as "pedestrian running at a speed of 5 m / s" that do not conform to real-world conditions. Furthermore, a multi-objective optimization module filters scenes based on both scene realism and algorithmic challenge, ensuring that the optimal simulation scene achieves the best balance between physical realism and algorithmic challenge.

[0041] 3. Establish a closed-loop linkage mechanism between data mining and algorithm optimization. This application uses a simulation testing module to generate test reports to pinpoint algorithm shortcomings, an optimization and adjustment module to generate algorithm optimization suggestions and dynamically adjust the data collection strategy for mass-produced vehicles, and a data feedback module to feed newly collected high-value scenarios back to the cloud to update the dynamic scenario library, thus constructing a complete closed loop of "data mining—scenario generation—algorithm testing—data collection strategy adjustment—new data feedback". Compared to existing technologies where the data collection strategy is fixed and cannot respond to dynamic algorithm shortcomings, this application enables the data collection strategy to adaptively adjust according to changes in algorithm shortcomings, avoiding the inefficient iteration of "repeatedly generating solutions that have been solved while providing insufficient coverage for unsolved problems", and significantly compressing the algorithm optimization cycle.

[0042] 4. Achieving long-tail data reuse in the industry through federated learning. This application conducts horizontal federated learning with multiple automakers' cloud platforms through federated aggregation nodes. It only receives encrypted scene distribution statistics and algorithm failure mode percentage statistics, generating a global scene distribution report and distributing it to each automaker without disclosing the original data. This guides each automaker's data collection strategy to focus on common industry shortcomings, avoiding repeated data collection of already fully covered scenarios and reducing the overall cost of acquiring CornerCase at the industry level.

[0043] In summary, this application systematically solves the problems of existing technologies in three dimensions—corner case mining efficiency, scene generation realism, and algorithm optimization closed loop—by constructing a full-chain closed-loop technical solution, significantly improving the iteration efficiency and reliability of autonomous driving algorithms. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the workflow of the autonomous driving Corner Case dynamic generation and algorithm optimization system based on spatiotemporal data in this embodiment.

[0045] Figure 2 This is a detailed diagram of the technical roadmap for the autonomous driving Corner Case dynamic generation and algorithm optimization system in this embodiment. Detailed Implementation

[0046] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0047] like Figure 1 As shown, this invention provides an autonomous driving corner case dynamic generation and algorithm optimization system based on spatiotemporal data, comprising a vehicle-side data monitoring layer, a cloud processing layer, a scene generation layer, and a simulation testing and optimization layer. Through the collaborative work of these layers, it solves the problems of existing technologies in three dimensions: corner case mining efficiency, scene generation realism, and algorithm optimization closed loop. Each layer is described in detail below:

[0048] The vehicle-side data monitoring layer includes a multi-sensor acquisition module, a consistency verification module, and an anomaly marking and uploading module. Among them:

[0049] The multi-sensor acquisition module synchronously collects data from multiple sensors and algorithm decision-making data. The multi-sensor data includes data from the vehicle-mounted 16-line LiDAR, 8-megapixel camera, 4D millimeter-wave radar, and high-precision IMU. Data synchronization is achieved through timestamp alignment, with an alignment error ≤1ms. Specifically: LiDAR data: resolution 1024×1024, point cloud density 200 points / ㎡, containing four-dimensional information (x, y, z, RCS); Camera data: resolution 1920×1080, frame rate 30fps, containing RGB color and texture information; Millimeter-wave radar data: sampling rate 10Hz, containing four-dimensional information (x, y, z, v), where v is radial velocity. Algorithm decision-making data includes the perception results, predicted trajectory, and control command sequence output by the autonomous driving system. For example, if the predicted trajectory deviates after the system detects a pedestrian, it may cause a braking delay of 0.8 seconds.

[0050] The consistency verification module is used to perform cross-sensor target matching and dynamic feature stability verification on the collected data to identify and exclude invalid data. In this embodiment, cross-sensor target matching uses the RANSAC algorithm to align the target contours of LiDAR point clouds and camera images, such as vehicle edges and pedestrian contours. When the intersection-union ratio (IU) is less than 0.5, it is judged as sensor invalid data, such as camera lens contamination, and is excluded from the initial screening. Dynamic feature stability verification fuses the target velocity information of millimeter-wave radar and IMU through Kalman filtering. When the velocity fluctuation is greater than 2 meters per second, it exceeds the range of normal traffic scenes and is marked as dynamic noise, and is not included in the candidate scene.

[0051] The anomaly marking and uploading module, based on the valid data filtered by the consistency verification module and combined with the operating status of the autonomous driving system, performs anomaly judgment and data uploading, providing candidate scenarios for the cloud processing layer. The specific operation is as follows: Based on the perception miss rate, prediction trajectory error, and control delay of the autonomous driving system, when any of the following occurs, the perception miss rate > 10%, the prediction trajectory error > 0.5 meters, or the control delay > 0.5 seconds, the time period is marked as an anomaly scenario. The synchronization data of the preceding and following 5 minutes is encapsulated, compressed to no more than 100MB, and uploaded to the cloud processing layer via 4G or 5G encryption as a candidate scenario.

[0052] The cloud processing layer receives candidate scenarios uploaded by the vehicle-side data monitoring layer and is configured to perform dual-indicator screening and federated collaborative data mining. The cloud processing layer includes a dual-indicator screening module, a feature dimensionality reduction module, and a federated aggregation node. Specifically:

[0053] The dual-indicator screening module receives candidate scenes and simultaneously filters them based on scene scarcity and algorithm understanding to obtain high-value scenes. Scene scarcity is measured using KL divergence to determine the difference in feature distribution between the candidate scene and existing scenes in the dynamic scene library; algorithm understanding is measured using mutual information entropy to determine the correlation strength between sensor data and algorithm decisions. In this embodiment, KL divergence is used to calculate the difference in feature distribution between the candidate case and the existing 100,000-level case library. A difference ≥ 0.6 indicates high scene scarcity, proceeding to the next stage. Mutual information entropy is used to calculate correlation; a correlation ≤ 0.3 indicates that the algorithm does not understand the scene, such as location drift caused by tunnel signal interruption, and is thus determined to be a high-value scene.

[0054] The feature dimensionality reduction module performs feature dimensionality reduction on the scenes selected by the dual-index screening module, and then builds an index according to a preset classification system before storing it in the dynamic scene library. In this embodiment, the t-SNE algorithm is used to reduce the 256-dimensional features (sensor statistical features + algorithm decision features) of high-value scenes to 32 dimensions, which are then stored in the MySQL dynamic Case library and indexed according to the "weather-road-event" classification to ensure that the query latency is ≤100ms.

[0055] A federated averaging algorithm is used to build a horizontal federated learning privacy-preserving architecture between the federated aggregation node and the cloud of multiple car manufacturers. Each car manufacturer, as a client, only completes feature extraction and initial scene screening locally, without uploading raw data to the cloud. Instead, it uploads scene distribution statistics and algorithm failure mode percentage statistics after AES-256 encryption, such as rainstorm scenarios accounting for 15% and perception missed detections accounting for 60%.

[0056] This mechanism introduces a dynamic contribution weighting strategy with fixed weights in a stepped, segmented manner. The global proportion of a scene is the sole core criterion, and the weighting function is designed as a discontinuous stepped, segmented function: the aggregated weight of regular scenes is the base weight, assigned a value of 1; for niche, long-tail scenes with a global proportion <5%, such as high-altitude, low-temperature scenes, their aggregated weight is increased to three times the base weight; for regular scenes with a global proportion ≥5%, the base weight remains unchanged. This weighting strategy has preconditions for effectiveness, only applying to high-value scenes selected through cloud-based dual-indicator screening. Invalid and low-value scenes are not included in the weighted statistics, and no other subjective evaluation correction factors are introduced, ensuring rule determinism and reproducibility. Furthermore, this weighting strategy only assigns fixed coefficients to the statistics uploaded by the client, without modifying the core loss function, parameter aggregation rules, or iterative update logic of the federated average algorithm, and without changing the algorithm's original convergence boundaries and convergence conditions.

[0057] The above incentive mechanism can effectively enhance automakers' enthusiasm for sharing high-value niche data, increasing the coverage of long-tail cases by 60%. The federated aggregation node ultimately generates a global scenario distribution report and distributes it to the cloud of each automaker, guiding them to supplement the collection of scarce scenarios in a targeted manner, such as guiding automakers in western China to focus on collecting high-altitude tunnel scenarios.

[0058] The scene generation layer includes a large model semantic understanding module, a generative adversarial network module, and a multi-objective optimization module. Among them:

[0059] The large-scale model semantic understanding module receives high-value scenarios output by the dual-index screening module and performs semantic analysis on them, extracting core elements such as environment, road conditions, target behavior, and algorithm failure points. Constraints are then generated based on these core elements. Specifically, the module performs the following operations: Case semantic transformation: Sensor data and algorithm decisions from high-value cases are transformed into natural language descriptions, such as heavy rain, unprotected left-turn intersections, pedestrians crossing at 1.2 meters per second, and algorithm braking delay of 0.8 seconds. Derivative scenario association: Based on the large-scale model's reasoning capabilities, derivative elements are associated from the core elements. For example, heavy rain combined with pedestrians crossing can be associated with snowy weather combined with cyclists crossing, and foggy weather combined with children entering the road, serving as semantic constraints to avoid generating meaningless scenarios.

[0060] The Generative Adversarial Network (GAN) module generates initial derived scene data in batches based on core elements and semantic constraints by introducing a physical constraint loss term; at the same time, it ensures the realism of the scene through model optimization.

[0061] In terms of model design: The generative adversarial network in this embodiment is an improved Wasserstein generative adversarial network, constructed by introducing a physical constraint loss term on the basis of the traditional Wasserstein generative adversarial network. Its total loss function is:

[0062] L total =L Wasserstein +λL physics ;

[0063] Among them, L Wasserstein Wasserstein distance loss is used to fit the true data distribution; L physics The physical constraint loss is used to constrain pedestrian walking speeds to no more than 1.8 m / s and vehicle braking distances to no less than [a certain value]. Where v is the vehicle speed, μ is the friction coefficient, and g is the gravitational acceleration; λ > 0 is the balance coefficient, used to balance the accuracy of the distribution fitting with physical compliance. A specific example of calculating physical constraint loss is as follows:

[0064] ①If the instantaneous walking speed v of the pedestrian is extracted from the generated scene ped =2.2m / s, violating the upper limit constraint on walking speed, then L ped =max(0,2.2−1.8)2=0.16; If the pedestrian's walking speed v ped =1.5m / s, which meets the constraint, then L ped =0.

[0065] ② If the generated scene is a snow-covered road surface (μ=0.2), the extracted initial braking speed v of the vehicle is... veh =16.67m / s (corresponding to 60km / h), the calculated theoretical minimum braking distance is approximately 70.9m. If the generated actual braking distance dbrake = 50m, the constraint is violated, then L veh =max(0,70.9−50)2≈436.81; if the generated actual braking distance d brake =75m, which satisfies the constraint, then L veh =0.

[0066] In terms of generation efficiency and quality optimization: FP16 precision training is adopted, which reduces the time to generate one scene (Case) from 5 seconds to 1.2 seconds, and the batch generation efficiency reaches 100 scenes / hour; inputting core features such as rainstorm + unprotected left turn + pedestrian crossing can generate 20 initial derived scenes.

[0067] The multi-objective optimization module employs a multi-objective optimization algorithm to filter the initial derived scenario data output by the adversarial network module, balancing scenario realism with algorithmic challenge, and providing the optimal scenario for simulation testing. The specific selection criteria are as follows:

[0068] Authenticity objective: Calculate the similarity between the generated scene and the real road test scene. Use cosine similarity to calculate the matching degree of LiDAR point cloud density distribution and histogram intersection to calculate the matching degree of camera image RGB histogram. The weighted average of the two is used to obtain the authenticity score, which is required to be >80 points.

[0069] Challenging objective: Import the scenario into the simplified algorithm testing module, and statistically analyze the perception false negative rate and decision delay. When the perception false negative rate is greater than 10% or the decision delay is greater than 0.5 seconds, the algorithm is deemed to have failed in that scenario. The proportion of failed scenarios to the total number of scenarios is the algorithm failure probability. In this embodiment, the failure probability is set to ≥70% to filter out extremely rare scenarios or scenarios that the algorithm cannot handle at all.

[0070] Screening Results: This embodiment aims to maximize both the authenticity score and the challenge score. It uses the non-dominated sorting genetic algorithm II for optimization, with a population size of 100 and 500 iterations. The Pareto front is obtained through non-dominated sorting and crowding calculation. The top eight optimal scenarios are selected according to crowding, such as snowfall of 5 mm per hour, unprotected left turns, and cyclists crossing at a speed of 5 meters per second. The final scenario has an authenticity score of ≥82 and a challenge score of ≥78.

[0071] The simulation testing and optimization layer includes a simulation testing module, an optimization adjustment module, and a data feedback module. These three modules work together to complete the "test-optimize-feedback" process, wherein:

[0072] The simulation testing module is configured to acquire the autonomous driving algorithm to be optimized, as well as the optimal simulation scenario output by the scenario generation layer. Based on the optimal simulation scenario, the autonomous driving algorithm to be optimized is tested, its performance metrics are recorded, and a test report is generated to pinpoint algorithm weaknesses. Specifically:

[0073] Import 500 test cases from the dynamic Case library, including original and derived scenarios, into the simulation platform by type;

[0074] Deploy the algorithm to be optimized and record metrics such as algorithm pass rate, perception accuracy, and decision delay. For example, the initial pass rate for a scenario involving snow, unprotected left turns, and cyclists crossing the road was 62%.

[0075] Generate test reports to pinpoint algorithm shortcomings, such as snow noise in LiDAR point clouds during snowy weather causing missed detections.

[0076] This embodiment, based on the results of the weakness identification, clarifies and quantifies optimization rules to form a closed-loop management system. The specific optimization rules are as follows:

[0077] ① Triggering rules: When the pass rate of a single scenario algorithm is lower than 80%, or any core indicator exceeds the preset failure judgment threshold, the optimization process of the corresponding module will be started.

[0078] ② Layered execution: Targeted optimization is carried out for the shortcomings of different layers. For the shortcomings of the perception layer, point cloud denoising and target detection algorithms are optimized first. For the shortcomings of the prediction layer, trajectory prediction model is optimized. For the shortcomings of the decision control layer, risk assessment and braking triggering logic are optimized.

[0079] ③ Validation criteria: The optimized algorithm must meet the requirement that the core indicators regress to within the failure threshold in the corresponding scenario, and the pass rate for a single scenario must be improved to over 90%. The optimization effect needs to be finally verified through full-scale scenario retesting.

[0080] The optimization and adjustment module generates algorithm optimization suggestions based on the test report produced by the simulation testing module, and adjusts the data acquisition strategy of the algorithm to be optimized in order to improve algorithm performance and enrich the scene library. The specific operations are as follows:

[0081] Regarding algorithm optimization: In this embodiment, the rule for generating algorithm optimization suggestions is as follows: Algorithm weaknesses identified in the test report through a three-level logic of "threshold benchmarking - process breakdown - root cause attribution" are matched, and corresponding optimization directions are generated according to the module to which the weakness belongs. Specifically: weaknesses in the perception layer correspond to optimization suggestions for point cloud denoising and target detection algorithms; weaknesses in the trajectory prediction layer correspond to optimization suggestions for trajectory prediction models; and weaknesses in the decision control layer correspond to optimization suggestions for risk assessment and vehicle control logic.

[0082] Taking the algorithm to be optimized as an example: The preprocessing stage of the improved LiDAR point cloud segmentation algorithm is enhanced by adding a snowflake noise filtering module. This module first filters out noisy point clouds such as snowflakes, water splashes, and raindrops, then inputs the clean point cloud into the subsequent segmentation network to avoid target misses caused by noise interference. This improves the perception accuracy from 78% to 92%. A dynamic risk assessment model is introduced, triggering avoidance 0.3 seconds in advance when the speed exceeds 50 km / h, reducing the decision delay from 0.65 seconds to 0.4 seconds.

[0083] The dynamic adjustment rules for the mass production vehicle data collection strategy are as follows: take the scenario type with the highest percentage of algorithm failures in the test report as the core targeted data collection target, clarify the core elements, regions, cycles and vehicle-side initial screening rules of the collected scenarios, and increase the collection weight of algorithm failure scenarios and long-tail scarce scenarios with a global percentage of <5%.

[0084] At the execution level of mass-produced vehicle data acquisition strategy adjustments: when the algorithm pass rate for a certain scenario increases by 50% or more, that scenario is designated as a high-priority acquisition target, and the acquisition frequency is tripled. For example, the algorithm pass rate for snowy weather and unprotected left turns increased from 62% to 79%, an increase of 17%, meeting the high-priority criteria. Simultaneously, acquisition parameters are adapted to regional characteristics: for example, in the snowy regions of Northeast China, the LiDAR acquisition frequency of mass-produced vehicles is increased from 10Hz to 15Hz, and the camera exposure time is adjusted from 1 / 500s to 1 / 300s to ensure clear capture of detailed actions such as pedestrians crossing and vehicles cutting in under low-light and snowy conditions.

[0085] The data feedback module receives collection strategy adjustment instructions from the optimization and adjustment module, acquires high-value scenario data collected from mass-produced vehicles according to the adjusted strategy (e.g., 120 scenario data points collected within three months), and transmits it back to the cloud processing layer. After data cleaning, scenario structured analysis, and feature extraction, the transmitted data is filtered based on both scenario scarcity and algorithm comprehension. High-value scenarios that pass the filter are updated to the dynamic scenario library, while those that fail are removed. The updated dynamic scenario library is then re-input into the scenario generation layer and simulation testing and optimization layer for a new round of scenario generation, algorithm testing and optimization, and collection strategy iteration, ultimately forming a closed-loop process encompassing scenario collection, cloud filtering, scenario generation, algorithm optimization, strategy adjustment, and data transmission.

[0086] Example 1:

[0087] To further clarify the complete implementation process of this system, combined with Figure 2 The technical approach, taking the discovery and algorithm optimization of the "snowy weather + unprotected left turn + cyclist crossing" case as an example, involves the following specific implementation steps:

[0088] S1: Vehicle-side data monitoring:

[0089] Environment and Equipment: Winter snow in a city in Northeast China, snowfall 5mm / h, unprotected left-turn intersection, vehicle-side edge computing unit is NVIDIA Jetson AGX Orin, 64GB of video memory;

[0090] Data Processing: The multi-sensor acquisition module simultaneously acquires LiDAR point cloud clusters, camera target images, and millimeter-wave radar velocity data. The coordinates of the LiDAR point cloud clusters are x=12.3m, y=-2.8m, and z=1.6m, and the millimeter-wave radar velocity data is 5m / s. The consistency verification module verifies the acquired data. The IOU value between the LiDAR and camera target contours is 0.8, and the velocity fluctuation is 0.5m / s, which is considered valid data. The anomaly marking and uploading module detects a false negative rate of 18% due to the algorithm. Since this value is greater than 10%, the scene is marked as a candidate case. The data is compressed to 85MB, encrypted, and then uploaded to the cloud processing layer.

[0091] S2: Cloud processing:

[0092] Dual-indicator calculation: The dual-indicator screening module calculates the candidate cases uploaded by the vehicle. The KL divergence between the candidate case and the existing database is 0.72, which is greater than or equal to 0.6, and the mutual information entropy is 0.25, which is less than or equal to 0.3. The candidate case is judged as a high-value case.

[0093] Federated Collaboration: The federated aggregation node receives statistics uploaded by each automaker. Automaker A is an automaker in Northeast China, and its uploads account for 3% of the cases. Automaker B is an automaker in Northwest China, and its uploads account for 8% of the cases. After aggregation, the global proportion of this case is 5.5%, which is marked as a long-tail scenario. At the same time, a global scenario distribution report is generated to guide Automaker A to focus on collecting data on this type of scenario.

[0094] S3: Corresponding scene generation:

[0095] Semantic understanding: The semantic understanding module of the large model parses the high-value case as "snowy day, unprotected left turn, cyclist crossing at 5m / s, algorithm missed detection", extracts core features and generates derived constraints;

[0096] WGAN generation: The generative adversarial network module takes the core features as input and generates 20 initial scenes, including combinations of snowfall of 3mm / h, 5mm / h, and 8mm / h and cycling speed of 3m / s, 5m / s, and 7m / s.

[0097] NSGA-II Screening: The multi-objective optimization module uses the NSGA-II algorithm to screen out 8 optimal scenarios, such as a scenario with snowfall of 5mm / h and cycling speed of 5m / s. This scenario has a realism score of 85 and a challenge score of 79, and is finally output to the simulation testing and optimization layer.

[0098] S4. Simulation Testing and Optimization:

[0099] Simulation Testing: The simulation testing module imports the optimal scenario into the Prescan / Simulink simulation platform, deploys the algorithm to be optimized for testing, and measures the algorithm's pass rate at 62%, the perception false negative rate at 22%, and the decision delay at 0.65s. A test report is generated and the algorithm's shortcomings are identified.

[0100] Algorithm optimization: Based on the test report, the optimization and adjustment module was optimized in a targeted manner. A snow noise filtering module was added to the perception layer, reducing the false negative rate to 8%. The decision layer triggered avoidance 0.3 seconds in advance through a dynamic risk assessment model, reducing the delay to 0.4 seconds. At the same time, the mass production vehicle acquisition strategy was adjusted, increasing the acquisition frequency of such scenarios in the Northeast region by 3 times, and setting the LiDAR frequency to 15Hz.

[0101] Closed-loop feedback: The data feedback module collects 35 similar cases added to the mass-produced vehicle within 3 months and sends them back to the cloud processing layer to update the dynamic scenario library, thus completing the closed loop;

[0102] Closed-loop effect: The optimized algorithm has increased the pass rate to 88%, shortened the iteration cycle from 6 months to 3.6 months, and improved efficiency by 40%.

Claims

1. A dynamic generation and algorithm optimization system for autonomous driving corner cases based on spatiotemporal data, comprising a vehicle-side data monitoring layer, a cloud processing layer, a scene generation layer, and a simulation testing and optimization layer; characterized in that: The vehicle-side data monitoring layer includes a multi-sensor acquisition module, a consistency verification module, and an anomaly marking and uploading module. The multi-sensor acquisition module synchronously acquires sensor data through lidar, camera, millimeter-wave radar, and inertial measurement unit, and synchronously acquires the perception results, predicted trajectory, and control command sequence output by the autonomous driving system as algorithm decision data. The consistency verification module is used to perform cross-sensor target matching and dynamic feature stability verification on the collected data in order to identify and exclude invalid data; the anomaly marking and uploading module marks the time period as an abnormal scene when any of the values ​​of perception missed detection rate, prediction trajectory error and control delay of the autonomous driving system exceeds the preset threshold, encapsulates the synchronous data before and after the preset time period and encrypts and uploads it to the cloud processing layer as a candidate scene. The cloud processing layer includes a dual-index screening module, a feature dimensionality reduction module, and a federated aggregation node. The dual-index screening module receives candidate scenes and simultaneously filters them based on scene scarcity and algorithm understanding to obtain high-value scenes. Scene scarcity is measured by KL divergence to determine the difference in feature distribution between the candidate scene and existing scenes in the dynamic scene library, while algorithm understanding is measured by mutual information entropy to determine the correlation strength between sensor data and algorithm decisions. The feature dimensionality reduction module performs feature dimensionality reduction on the scenes filtered by the dual-index screening module, and stores them in the dynamic scene library after building an index according to a preset classification system. The federated aggregation node performs horizontal federated learning with multiple automakers' clouds. It only receives encrypted scene distribution statistics and algorithm failure mode percentage statistics uploaded by each automaker, and performs dynamic weighted aggregation based on the distribution of long-tail scenes provided by each automaker to generate a global scene distribution report. The report is then distributed to each automaker's cloud to guide their data collection strategies. The scene generation layer includes a large-scale model semantic understanding module, a generative adversarial network module, and a multi-objective optimization module. The large-scale model semantic understanding module acquires sensor data and algorithm decisions corresponding to high-value scenes output by the dual-index screening module, converts them into natural language descriptions, extracts environment, road conditions, target behavior, and algorithm failure points as core elements, and generates derived elements as semantic constraints through reasoning. Based on the core elements and semantic constraints, the generative adversarial network module generates initial derived scene data in batches by introducing a physical constraint loss term. The physical constraint loss term is used to constrain the physical motion parameters of moving targets in the generated scene to conform to preset physical laws. The multi-objective optimization module filters the initial derived scene data through a multi-objective optimization algorithm, with scene realism and algorithm challenge as optimization objectives, to obtain the optimal simulation scene. The simulation testing and optimization layer includes a simulation testing module, an optimization adjustment module, and a data feedback module. The simulation testing module is configured to acquire the autonomous driving algorithm to be optimized, acquire the optimal simulation scenario output by the scenario generation layer, test the autonomous driving algorithm to be optimized based on the optimal simulation scenario, record the algorithm performance indicators, and generate a test report to locate the algorithm's shortcomings. The optimization and adjustment module generates algorithm optimization suggestions based on the test report and outputs them to the algorithm development end, while dynamically adjusting the collection strategy of mass-produced vehicles; the data feedback module is used to obtain data corresponding to the high-value scenarios newly collected by mass-produced vehicles according to the adjusted collection strategy, and feeds it back to the cloud processing layer to update the dynamic scenario library.

2. The optimization system of claim 1, wherein, In the vehicle-side data monitoring layer: The cross-sensor target matching uses the RANSAC algorithm to align the target contours of the lidar point cloud and the camera image. In a single frame of data, when the cross-union ratio is less than 0.5, it is determined to be sensor failure data and is excluded. The dynamic feature stability verification is performed by fusing the target velocity information from the millimeter-wave radar and the inertial measurement unit using Kalman filtering. When the velocity fluctuation is greater than 2 m / s, it is marked as dynamic noise and not included in the candidates. The preset thresholds include a detection miss rate greater than 10%, a predicted trajectory error greater than 0.5m, and a control delay greater than 0.5s; The preset time before and after the abnormal moment is 5 minutes before and after, and the compressed size does not exceed 100MB. It is uploaded in encrypted form via 4G or 5G network.

3. The optimization system of claim 1, wherein, In the cloud processing layer: The dual-index screening module determines candidate scenarios as high-value scenarios when the KL divergence is greater than or equal to 0.6 and the mutual information entropy is less than or equal to 0.

3. The feature dimensionality reduction processing module uses the t-SNE algorithm to reduce the 256-dimensional features to 32-dimensional features, and builds an index according to weather-road-event classification, with a query latency of no more than 100ms; The federated aggregation nodes use the FedAvg algorithm for aggregation. Long-tail scenarios are those with a global share of less than 5%. During dynamic weighted aggregation, the aggregation weight of the provider is increased to 3 times. Encryption uses the AES-256 encryption algorithm.

4. The optimization system of claim 1, wherein, In the scene generation layer: The generative adversarial network is an improved Wasserstein generative adversarial network, constructed by introducing a physical constraint loss term into the traditional Wasserstein generative adversarial network. Its total loss function is: THE total =L Wasserstein +λL physics ; Among them, L Wasserstein Wasserstein distance loss is used to fit the true data distribution; L physics The physical constraint loss is used to constrain pedestrian walking speeds to no more than 1.8 m / s and vehicle braking distances to no less than [a certain value]. v is the vehicle speed, μ is the friction coefficient, g is the gravitational acceleration; λ > 0 is the balance coefficient, used to balance the distribution fitting accuracy and physical compliance; the generative adversarial network is trained using FP16 mixed-precision training; The multi-objective optimization algorithm is the NSGA-II non-dominated sorting genetic algorithm. It quantifies the authenticity score by measuring the similarity between the generated scenario and the real road test scenario, and quantifies the challenge of the algorithm by perceiving the false negative rate and decision delay. The optimal simulation scenario obtained by screening satisfies the requirement that the authenticity score is not less than 80 points and the algorithm failure probability is not less than 70%.

5. The optimization system of claim 1, wherein, In the simulation testing and optimization layer: The simulation platform is a high-fidelity simulation platform based on Prescan or Simulink, containing no less than 100 road models and no less than 30 weather models; The algorithm performance metrics include algorithm pass rate, perception accuracy, and decision latency. The dynamic adjustment strategy for mass-produced vehicles includes: when the algorithm pass rate for a certain type of scenario increases by no less than 15%, setting that type of scenario as a high priority and increasing the acquisition frequency by 3 times; and adjusting sensor parameters according to regional characteristics, including adjusting the LiDAR acquisition frequency from 10Hz to 15Hz and the camera exposure time from 1 / 500s to 1 / 300s in snowy areas.

6. A method for dynamic generation and algorithm optimization of autonomous driving corner cases based on spatiotemporal data, comprising the following steps: S1. Vehicle-side data monitoring: Sensor data is simultaneously collected through LiDAR, cameras, millimeter-wave radar, and inertial measurement units, and the perception results, predicted trajectories, and control command sequences output by the autonomous driving system are also collected as algorithm decision data. Cross-sensor target matching and dynamic feature stability verification are performed on the collected data to identify and eliminate invalid data. Based on the perception miss rate, predicted trajectory error, and control delay of the autonomous driving system, when any of these values ​​exceeds a preset threshold, the time period is marked as an abnormal scene, and the synchronous data before and after the preset time period is encapsulated and encrypted and uploaded to the cloud as a candidate scene. S2. Cloud Processing Steps: Receive candidate scenes and simultaneously filter them based on scene scarcity and algorithm understanding to obtain high-value scenes. Scene scarcity is measured by KL divergence to determine the difference in feature distribution between the candidate scene and existing scenes in the dynamic scene library, while algorithm understanding is measured by mutual information entropy to determine the correlation strength between sensor data and algorithm decisions. The selected high-value scenes undergo feature dimensionality reduction and are indexed according to a preset classification system before being stored in the dynamic scene library. Horizontal federated learning is performed through federated aggregation nodes and multiple automakers' clouds. Encrypted scene distribution statistics and algorithm failure mode percentage statistics uploaded by each automaker are received. Dynamic weighted aggregation is performed based on the distribution of long-tail scenes provided by each automaker to generate a global scene distribution report, which is then distributed to each automaker's cloud to guide their data collection strategies. S3. Scene Generation: Acquire sensor data and algorithm decisions corresponding to high-value scenes selected through dual-index screening, call a large model to convert them into natural language descriptions, extract environment, road conditions, target behavior, and algorithm failure points as core elements, and generate derived elements as semantic constraints through reasoning; based on the core elements and semantic constraints, use a generative adversarial network with a physical constraint loss term to generate initial derived scene data in batches; the physical constraint loss term is used to constrain the physical motion parameters of moving targets in the generated scene to conform to preset physical laws; the initial derived scene data is screened through a multi-objective optimization algorithm, with scene realism and algorithm challenge as optimization objectives, to obtain the optimal simulation scene; S4. Simulation Testing and Optimization: Obtain the autonomous driving algorithm to be optimized and the optimal simulation scenario obtained in step S3. Test the autonomous driving algorithm to be optimized based on the optimal simulation scenario, record the algorithm performance indicators, and generate a test report to locate algorithm shortcomings. Generate algorithm optimization suggestions based on the test report and output them to the algorithm development end. At the same time, dynamically adjust the data collection strategy of mass production vehicles. Obtain the data corresponding to the high-value scenarios newly collected by mass production vehicles according to the adjusted data collection strategy, and feed it back to the cloud. Return to the cloud processing steps to form a closed loop.

Images