Neural network model training method based on virtual reality and simulation and related equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a neural network model training method based on virtual reality and simulation and related equipment. The method comprises the steps of generating a three-dimensional visual virtual reality scene based on a target real scene and constructing a dynamic physical simulation environment; deploying the neural network model to a test vehicle digital twin execution model for reasoning, and outputting a reasoning result; identifying a to-be-optimized target test scene type of the neural network model based on a reasoning result; dynamically adjusting scene parameters of the dynamic physical simulation environment to simulate a target test scene type, and generating a high-fidelity synthetic data set; and performing iterative training on the neural network model by mixing the high-fidelity synthetic data set and the real data set to form a closed-loop optimization mechanism until the generalization ability of the neural network model in the complex scene reaches a preset threshold. Based on the method, the training efficiency and generalization ability of the neural network model in a complex scene can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a neural network model training method and related devices based on virtual reality and simulation. Background Art

[0002] In recent years, with the wide application of deep learning technology in fields such as autonomous driving and intelligent robots, the training and verification of neural network models have become increasingly dependent on high-quality data sets. However, traditional training methods are severely limited by the acquisition efficiency and cost of real-scene data. In the field of autonomous driving, real-road testing faces problems such as difficulties in simulating extreme weather, high costs in reproducing high-risk scenarios, and ethical restrictions, resulting in a serious shortage of data sets for key scenarios (such as pedestrian obstacle avoidance in heavy rain environments and low-light signal light recognition at night), directly affecting the generalization ability and safety of the model.

[0003] Although existing simulation technologies can generate some training data through virtual environments, their limitations are significant: on the one hand, most simulation platforms only support single-module testing (such as vehicle dynamics or sensor simulation), and it is difficult to build a high-fidelity virtual scene with multi-modal interaction, resulting in insufficient semantic consistency between synthetic data and real scenes; on the other hand, the simulation environment is separated from the model training process, lacking a dynamic data generation and feedback mechanism based on test results, and unable to optimize the data distribution directionally for model defects. In addition, the compatibility of existing cross-platform deployment technologies (such as model format conversion) is insufficient, resulting in low inference efficiency of models in virtual test environments and difficulty in real-time verifying decision-making logics in complex scenarios.

[0004] In summary, the existing technologies are limited by the high cost of real data acquisition, insufficient fidelity of simulation scenarios and fragmentation of multiple modules, and lack of a dynamic closed-loop data generation mechanism based on model test results, resulting in the generalization ability and training efficiency of neural network models in complex scenarios being difficult to meet the actual application requirements. Summary of the Invention

[0005] A neural network model training method and related devices based on virtual reality and simulation provided by an embodiment of the present invention at least solve the problem in related technologies that due to being limited by the high cost of real data acquisition, insufficient fidelity of simulation scenarios and fragmentation of multiple modules, and lack of a dynamic closed-loop data generation mechanism based on model test results, the generalization ability and training efficiency of neural network models in complex scenarios are difficult to meet the actual application requirements.

[0006] According to the first aspect of the embodiment of the present invention, a neural network model training method based on virtual reality and simulation is provided, including: Based on the spatial data, physical properties, and environmental information of the target real scene, a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements is generated through parametric three-dimensional modeling methods; In the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of a test vehicle based on vehicle dynamics parameters, environmental light intensity, color temperature, and rain, snow, fog weather effects dynamically adjusted through a particle system and a skybox component; Deploy a neural network model trained based on real datasets to the digital twin of the test vehicle, and perform model inference according to the real-time data collected by the virtual camera from the dynamic physical simulation environment, and output the inference result; Based on the precision, recall rate, and virtual test collision frequency of the inference result, identify the performance defects of the neural network model, and determine the target test scene type to be optimized according to the performance defects; Dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic dataset corresponding to the target test scene type; Mix the high-fidelity synthetic dataset and the real dataset to perform iterative training on the neural network model, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenes reaches a preset threshold.

[0007] According to the second aspect of the embodiments of the present invention, there is provided a neural network model training device based on virtual reality and simulation, including: A generation module, configured to generate a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements through parametric three-dimensional modeling methods based on the spatial data, physical properties, and environmental information of the target real scene; A construction module, configured to construct a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of a test vehicle based on vehicle dynamics parameters, environmental light intensity, color temperature, and rain, snow, fog weather effects dynamically adjusted through a particle system and a skybox component; An inference module, configured to deploy a neural network model trained based on real datasets to the digital twin of the test vehicle, perform model inference according to the real-time data collected by the virtual camera from the dynamic physical simulation environment, and output the inference result; An identification module, configured to identify performance defects of the neural network model based on the precision rate, recall rate, and virtual test collision frequency of the inference result, and determine the type of target test scenario to be optimized according to the performance defects; The generation module is further configured to dynamically adjust the scenario parameters of the dynamic physical simulation environment to simulate the type of target test scenario, and generate a high-fidelity synthetic data set corresponding to the type of target test scenario; An iterative training module, configured to perform iterative training on the neural network model by mixing the high-fidelity synthetic data set and the real data set, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0008] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor, and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to execute the method according to the first aspect.

[0009] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect.

[0010] Advantageous effects of the embodiments of the present invention: The method for training a neural network model based on virtual reality and simulation provided by the embodiments of the present invention can improve the training efficiency and generalization ability of the neural network model in complex scenarios by integrating virtual scene modeling, dynamic physical simulation, and synthetic data generation technologies. First, a high-fidelity virtual scene is quickly generated based on the parametric modeling method, which can reproduce diverse real environmental elements (such as roads, buildings, dynamic traffic flows, etc.) at low cost, solving the problem of insufficient model coverage scenarios caused by relying on limited real data in traditional training methods. Second, by simulating real vehicle dynamics characteristics, multi-modal sensor data (such as camera images, etc.), and variable environmental conditions (such as extreme weather, day-night light changes) through a dynamic physical simulation environment, the model can be exposed to complex interactions close to real scenarios in virtual tests, accurately identifying its performance defects (such as missed detections in object detection, misjudgments of traffic lights, etc.). Further, a targeted synthetic data set is dynamically generated based on the performance defects, and it is mixed with real data for iterative training to form a closed-loop mechanism of "virtual test - defect diagnosis - data supplement - model optimization", significantly reducing the problem of insufficient model robustness caused by the scarcity of special scenario data. Finally, through the closed-loop optimization of virtual-real mapping, the method significantly reduces the cost and safety risks of model training and testing in high-risk fields such as autonomous driving, while ensuring the decision reliability of the model in complex environments.

[0011] Details of one or more embodiments of the present invention are set forth in the following drawings and description to make other features, objects, and advantages of the present invention more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative efforts.

[0013] Figure 1 It is a flowchart of a neural network model training method based on virtual reality and simulation provided for an embodiment of the present invention.

[0014] Figure 2 It is a schematic diagram of the system architecture of a neural network model training system based on virtual reality and simulation provided for an embodiment of the present invention.

[0015] Figure 3 It is a schematic diagram of the system architecture of an autonomous driving virtual test provided for an embodiment of the present invention.

[0016] Figure 4 It is a schematic diagram of a human-machine interaction interface provided for an embodiment of the present invention.

[0017] Figure 5 It is a schematic diagram of a behavior tree structure provided for an embodiment of the present invention.

[0018] Figure 6 It is a flowchart of a lane line detection and maintenance method in a virtual reality autonomous driving test system provided for an embodiment of the present invention.

[0019] Figure 7 It is a schematic diagram of the structure of a neural network model training device based on virtual reality and simulation provided for an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following will describe the embodiments of the present embodiment in more detail with reference to the drawings. Although some embodiments of the present embodiment are shown in the drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not used to limit the protection scope of the present embodiment.

[0021] As the requirements for the perception and decision-making capabilities of autonomous driving technology in complex scenarios become increasingly stringent, the traditional model training method that relies on real-road testing has become difficult to meet the high-precision and high-safety requirements. Although existing virtual simulation technologies can partially replace real testing, their defects such as low scene modeling efficiency, insufficient physical interaction fidelity, and fragmentation of model deployment and data closed-loop lead to the difficulty of eliminating the semantic deviation between simulation data and real scenarios, and there are still significant shortcomings in the performance verification of models under extreme conditions. For example, in rainy, snowy, or low-light night scenarios, due to limited real data collection, the model is prone to fatal defects such as missed target detection and misjudgment of traffic lights, and traditional simulation platforms cannot dynamically generate adaptive data for such defects to optimize model parameters.

[0022] To solve the above problems, the embodiments of the present invention provide a neural network model training method based on virtual reality and simulation. By parametric modeling, a high-fidelity virtual scene is quickly constructed, and combined with cross-platform model deployment technology, real-time inference and defect diagnosis in virtual and real environments are realized, and targeted generation and data-driven model iterative training are performed based on the test results. The method provided by the embodiments of the present invention aims to break through the technical bottlenecks of scene construction efficiency, physical interaction authenticity, and data closed-loop feedback ability, and provide a systematic solution for the efficient training and safety verification of autonomous driving models.

[0023] Figure 1 It is a flowchart of a neural network model training method based on virtual reality and simulation provided by the embodiments of the present invention. As Figure 1 shown, the method includes the following steps.

[0024] Step S101, based on the spatial data, physical attributes, and environmental information of the target real scene, generate a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements through parametric three-dimensional modeling methods.

[0025] Step S102, in the three-dimensional visual virtual reality scene, construct a dynamic physical simulation environment based on real physical characteristics. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, environmental light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through particle systems and skybox components.

[0026] Step S103, deploy the neural network model trained based on the real dataset to the digital twin of the test vehicle, perform model inference according to the real-time data generated by the virtual camera collecting the dynamic physical simulation environment, and output the inference result.

[0027] Step S104, based on the precision, recall, and virtual test collision frequency of the inference result, identify the performance defects of the neural network model, and determine the type of target test scene to be optimized according to the performance defects.

[0028] Step S105: Dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic dataset corresponding to the target test scene type.

[0029] Step S106: Iteratively train the neural network model by mixing the high-fidelity synthetic dataset with the real dataset, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0030] First, based on the spatial data, physical attributes, and environmental information of the target real scene, generate a three-dimensional visual virtual reality scene containing buildings, roads, and dynamic traffic elements through parametric three-dimensional modeling.

[0031] In this embodiment, the spatial data may include a high-definition plane map covering the target real scene, a traffic road network topology structure, and road width measurement data; the physical attributes may include building surface material reflection parameters, road friction coefficient thresholds, and vehicle dynamics parameters such as steering, torque, braking, and suspension parameters; and the environmental information may include day-night light gradual change parameters, rain-snow-fog weather particle densities, and traffic participant behavior rules.

[0032] Using the above spatial data, physical attributes, and environmental information, the road generation logic, building density, and building style parameters can be dynamically adjusted through predefined rule files to batch generate a three-dimensional digital city model containing two-way four-lane roads, green vegetation, and multi-type building complexes.

[0033] In the three-dimensional engine, material reflection parameters can be configured for the building surface, a friction coefficient threshold can be set for the road surface, and torque, braking, and suspension parameters can be configured for the digital twin of the test vehicle. At the same time, the day-night light change can be simulated through the skybox component, the rain-snow-fog weather special effects can be dynamically generated using the particle system, and the path planning and obstacle avoidance decisions of virtual vehicles and virtual pedestrians can be driven based on traffic participant behavior rules to form a three-dimensional visual virtual reality scene containing dynamic traffic elements.

[0034] In this embodiment, by generating a three-dimensional visual virtual reality scene through parametric three-dimensional modeling, a high-fidelity virtual environment can be quickly and automatically constructed based on real scene data, significantly reducing the manual modeling cost and time investment, improving the scene construction efficiency, and reducing the construction cost of the virtual test platform.

[0035] Next, in the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. Among them, the dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, ambient light intensity, color temperature, and rain, snow, fog weather effects dynamically adjusted through a particle system and a skybox component.

[0036] By defining signal light configuration rules and switching timings, reverse synchronous switching of oncoming lane signal lights is simulated, and the signal lights at the same intersection are controlled to cycle through the passing states according to a preset period. Configure torque output curves, braking response delays, and suspension stiffness thresholds for the digital twin of the test vehicle, and simulate the physical contact and friction effects between the vehicle and the road through the wheel collider component.

[0037] At the same time, based on ambient light and weather parameters, the intensity and color temperature of directional light are dynamically adjusted to simulate the gradual change between day and night, generate the particle density and movement trajectories of rain, snow, and fog weather, and link with the threshold of the road surface friction coefficient to realize the simulation of the impact of weather on vehicle dynamics. In addition, path planning algorithms, obstacle avoidance decision-making logics, and random behavior parameters are set for virtual vehicles and virtual pedestrians, and their environmental perception and dynamic interactions are simulated through ray detection and physical collision mechanisms to form a dynamic physical simulation environment containing variable traffic flow densities.

[0038] Subsequently, the neural network model trained based on the real dataset is deployed to the digital twin of the test vehicle, and model inference is performed according to the real-time data generated by the virtual camera capturing the dynamic physical simulation environment, and the inference result is output. In this embodiment, when collecting the real-time data generated by the dynamic physical simulation environment, collaborative collection can also be performed through the virtual camera and the radar component.

[0039] In this embodiment, the neural network model can be converted into an Open Neural Network Exchange format file, and the file is loaded through a three-dimensional engine to create a localized inference engine. The virtual camera can capture the image data of traffic participants in the dynamic physical simulation environment in real time, and the radar component can collect radar data within the fan-shaped detection area, including obstacle distance and azimuth data. Among them, the image data captured by the virtual camera can be used for model inference in the virtual environment, and the obstacle distance and azimuth data collected by the radar component can assist the model in making inferences.

[0040] After resetting the resolution and channel order of the above image data and radar data, input them into the neural network model, asynchronously execute the model inference process, and output the original inference data including the position of the target detection box, the classification of the signal light state, and the lane line detection result. Based on the above original inference data, the correct recognition number and missed detection number of target detection can be counted to calculate the precision and recall rate, and the number of collision events in the three-dimensional visualization virtual reality scene can be recorded synchronously to generate the collision frequency. Superimpose the precision, recall rate, and collision frequency on the virtual camera screen, render the inference result in real time through the three-dimensional visualization interface, and mark the performance indicators corresponding to the inference result and the corresponding dynamic physical simulation environment parameters in the human-computer interaction interface.

[0041] Based on the model inference result, the performance defects of the neural network model can be identified, and the type of target test scenario to be optimized can be determined according to the performance defects.

[0042] In this embodiment, when the detection precision rate of the inference result is lower than the preset precision rate threshold, the recall rate is lower than the preset recall rate threshold, or the virtual test collision frequency is higher than the preset collision frequency threshold, it is determined that the neural network model has performance defects. According to the type and distribution law of the performance defects, match the corresponding combination of scenario parameters in the dynamic physical simulation environment, and determine at least one type of scenario among rainy, snowy, foggy weather, low-light conditions at night, or dense traffic flow scenarios as the target test scenario type to be optimized.

[0043] To further optimize the model performance, the scenario parameters of the dynamic physical simulation environment can be dynamically adjusted to simulate the target test scenario type, and a high-fidelity synthetic dataset corresponding to the target test scenario type can be generated.

[0044] In this embodiment, the traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment can be adjusted in real time through the human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environments in the target test scenario type.

[0045] Based on the adjusted dynamic physical simulation environment, the virtual camera component and radar component can be driven to synchronously collect real-time image data and radar point cloud data including target detection box annotations, signal light state classifications, and lane line annotations, and record the driving trajectory, collision events, and model inference results of the digital twin of the test vehicle. Perform multi-modal data fusion and standardized annotation processing on the collected data to generate a high-fidelity synthetic dataset that matches the target test scenario type, whose annotation format, data dimension, and storage structure are compatible with the real dataset, and includes multi-dimensional labels of environmental parameters, vehicle behavior, and model decisions.

[0046] Finally, the high-fidelity synthetic dataset and the real dataset are mixed to iteratively train the neural network model, and the optimized neural network model is redeployed to the digital twin of the test vehicle to form a closed-loop optimization mechanism.

[0047] In this embodiment, the high-fidelity synthetic dataset and the real dataset can be mixed according to a preset ratio to generate a mixed training set. Based on the mixed training set, the neural network model is iteratively trained for multiple rounds. After each round of training is completed, the optimized model is redeployed to the digital twin of the test vehicle, and virtual tests are performed through the dynamic physical simulation environment, and the precision, recall, and virtual test collision frequency of the neural network model inference are recorded. If these indicators do not reach the preset threshold, the scene parameters of the dynamic physical simulation environment are dynamically adjusted according to the performance defects of the current neural network model, a new high-fidelity synthetic dataset is generated and the mixed training set is updated. Repeat the steps of iterative training, deployment testing, and scene parameter adjustment until the generalization ability of the neural network model in complex scenarios such as rainy and snowy weather, night lighting, and dense traffic flow reaches the preset threshold, thus forming a closed-loop optimization mechanism for the linkage of virtual and real data.

[0048] Through the above steps, the embodiment of the present invention can efficiently construct an integrated environment that integrates virtual reality scene modeling, dynamic physical simulation, neural network model deployment and inference, supports the visualization of the real-time inference results of the model and the generation of high-fidelity synthetic datasets, forms a closed-loop optimization mechanism for model training with virtual-real mapping, effectively improves the generalization ability of the neural network model in complex scenarios, reduces the model training and testing costs and improves the training efficiency, and provides strong support for the research and application in fields such as autonomous driving.

[0049] Figure 2 It is a schematic diagram of the system architecture for training a neural network model based on virtual reality and simulation provided by an embodiment of the present invention.

[0050] As Figure 2 shown, each part of the system closely cooperates to jointly build a complete and efficient test system.

[0051] In this embodiment, the real road test module includes an actual vehicle, sensors, dynamics, and control systems, which constitute the physical basis of autonomous driving; the autonomous driving stack (algorithms and decision-making control) is the core logic, carrying the functions of algorithm operation and decision-making. The two jointly provide a real-scene reference and data support for virtual tests to verify the accuracy of virtual tests.

[0052] In the virtual simulation system, first, through the City Engine tool, a virtual test scenario is generated based on a real high-definition map to construct a highly restored real environment. Then, the vehicle, sensors, dynamics, and control systems are simulated and modeled to achieve precise mapping, laying a foundation for testing. Then, an Open Neural Network Exchange (ONNX) model is deployed in the virtual scenario to perform autonomous driving tests, simulating the real driving process and detecting the model performance.

[0053] The test workflow module integrates multiple functions under user settings: the artificial intelligence model deployment component implants the model into the virtual environment; the virtual environment interaction control component adjusts the scenario parameters to simulate complex working conditions; the lane line detection component real-time identifies lane lines; the virtual navigation component plans paths; the acceleration test component improves efficiency; the synthetic dataset collection and annotation component collects and processes multi-source data for training.

[0054] The neural network model module contains multiple key models: the image recognition and classification model is used for signal light recognition; the object detection model detects traffic participants. The autonomous driving decision-making module makes reasonable decisions based on the outputs of the former two, which is one of the core parts of virtual testing.

[0055] The test data recording module collects data such as model inference results and collision events, feeds them back to the neural network, forms a data closed-loop, and optimizes the training process. Through cyclic iteration, the autonomous driving virtual test and visualization based on the fusion of virtual reality and multi-source perception are realized, the generalization ability and reliability of the model are improved, an efficient verification tool is provided for R & D, the problems of limited traditional test scenarios and insufficient data are solved, and the model iteration and safety verification are accelerated.

[0056] Figure 3 This is a schematic diagram of the system architecture for an autonomous driving virtual test provided by an embodiment of the present invention.

[0057] As Figure 3 shown, this system architecture combines two core parts of "virtual reality and simulation" and "autonomous driving virtual test process", and realizes efficient connection and integration through ONNX, finely simulating the operation logic of autonomous driving in complex environments.

[0058] Virtual reality and simulation construct the basic framework of virtual testing. The virtual environment includes elements such as urban roads, traffic lights, vehicles, pedestrians, weather, and lighting, restoring the diversity and complexity of real traffic scenarios. The urban roads plan driving paths, the traffic lights simulate traffic rules, the vehicles and pedestrians move according to preset rules, and the changes in weather and lighting create a diverse environment to test the adaptability of the autonomous driving system under different conditions.

[0059] As the perceptive antenna of the system, the sensor module includes a camera that provides visual images for target recognition, a radar that detects the distance, azimuth, and speed of obstacles, a navigation system that provides positioning and route planning, and an IMU that measures the vehicle's acceleration and angular velocity to support attitude estimation and control. In addition, ultrasonic sensors can supplement the short-range environmental information, and the 3D map combined with high-definition data constructs an accurate geographical scene to provide detailed coordinates and terrain information for the vehicle, realizing multi-source data acquisition.

[0060] The virtual test process of autonomous driving deeply processes and applies data. The fusion perception module analyzes sensor data to achieve target recognition and detection, including lane line recognition, traffic signal classification, and vehicle and pedestrian detection, and evaluates the collision risk. This module efficiently integrates the perception function to ensure consistent and efficient data processing and provides accurate information for subsequent decision-making.

[0061] The virtual test vehicle makes intelligent decisions based on the perception information. The prediction and inference function identifies the states of traffic participants and formulates strategies in advance; the behavior decision-making function makes reasonable driving behaviors such as accelerating, decelerating, turning, and stopping according to the prediction results and driving rules; the route planning combines navigation and real-time information to plan the test route.

[0062] ONNX enables efficient model deployment, ensures smooth interaction in the process from data acquisition to decision-making, accurately simulates real scenarios, comprehensively tests and verifies algorithms, improves the performance and safety of the autonomous driving system in complex environments, and promotes the development of technology towards a more reliable and intelligent direction.

[0063] In the embodiment of the present invention, the neural network model training method based on virtual reality and simulation further refines the construction process of the dynamic physical simulation environment to ensure that the generated virtual scene is not only highly realistic visually but also highly consistent with the real scene in physical characteristics. The following are the specific implementation steps of this method: First, based on the traffic signal state switching logic of the target real scene, the signal lamp configuration rules and switching timings can be defined to simulate the reverse synchronous switching of the oncoming lane signal lamps and control the signal lamps at the same intersection to cycle and change the passing state according to a preset period.

[0064] In this embodiment, a signal lamp control algorithm based on real traffic rules is adopted to simulate the signal lamp state switching. By analyzing the actual scene, the signal lamp configuration and switching timings are defined: the signal lamps in the oncoming lane cycle through red, yellow, and green with adjustable durations; the signal lamps at the same intersection change according to a cycle to ensure smooth and safe traffic. For example, the straight and left-turn signal lamps at an intersection are independent and staggered to avoid conflicts, achieving a real coordination effect and providing a realistic environment for virtual testing.

[0065] Then, based on the vehicle dynamics parameters of the target real scenario, the torque output curve, braking response delay, and suspension stiffness threshold can be configured for the test vehicle digital twin, and the physical contact and friction effects between the vehicle and the road can be simulated through the wheel collider component.

[0066] In this embodiment, in terms of vehicle dynamics simulation, the dynamics parameters of the test vehicle digital twin are configured, including the torque output curve, braking response delay, and suspension stiffness threshold. The torque curve is set based on engine data to ensure reasonable torque output at different speeds; the braking delay simulates the time difference from the driver's instruction to actual deceleration; the suspension stiffness is set according to physical characteristics to reflect the reaction under different road conditions. Through the wheel collider component, the contact and friction between the vehicle and the road, such as turning lateral force and bumps, can be simulated. The precise configuration of these parameters makes the virtual vehicle driving behavior close to the real one, providing a reliable physical basis for neural network training.

[0067] In an embodiment of the present invention, based on the wheel collider and physics engine of the Unity engine, the core characteristics of the vehicle, such as torque, braking, steering, suspension, and friction, are simulated. The environmental perception is realized by simulating the visual and radar sensors through the camera and ray detection, and the virtual navigation is realized by combining the A* algorithm and the navigation mesh, enabling the test vehicle to have the characteristics of a real vehicle. The specific steps are as follows: First, import the vehicle 3D model into Unity, add the wheel collider component, and set the weight, torque, braking, friction, suspension, wheel hub, and steering parameters to simulate the dynamic behavior. By adjusting the suspension stiffness and friction coefficient, the driving dynamics under different road conditions are simulated, and the collider range is set according to the shape to ensure the authenticity of the physical collision effect.

[0068] Next, add four camera components, front, rear, left, and right, to the vehicle to simulate the front-view camera and rearview mirrors, and adjust the resolution, field of view angle, and frame rate to make the image data consistent with the real one. By inputting the included angle , ray length and the number of rays within the range , the simulation of the radar detection range , detection distance and detection density is realized, and its functional relationship is shown in the following formula (1): (1); For example, set the included angle θ to 90 degrees, the ray length l to 150 meters, and the number of rays n to 32 to simulate the detection ability of the radar in the fan-shaped area.

[0069] Finally, rasterize the virtual road network into a navigation grid and mark the roads as reachable areas. Randomly generate the initial positions of vehicles in each round of testing, specify the target points by the user, and the vehicles plan paths through the A* algorithm and simulate autonomous driving by combining dynamic parameters. Record the trajectories, collisions, and sensor data during the testing for model training.

[0070] Through the above implementation steps, the embodiments of the present invention can efficiently simulate the dynamic behavior and perception ability of test vehicles in a virtual scenario, providing a highly realistic virtual test platform for the training of neural network models.

[0071] Secondly, based on the environmental lighting and weather parameters of the target real scenario, dynamically adjust the intensity and color temperature of the parallel light through the skybox component to simulate the gradual change between day and night, use the particle system to generate the particle density and movement trajectories of rain, snow, and fog weather, and link with the friction coefficient threshold of the road surface to realize the simulation of the impact of weather on vehicle dynamics.

[0072] In this embodiment, the weather effect simulation is realized through the skybox and the particle system. The skybox dynamically adjusts the intensity and color temperature of the parallel light to simulate the change between day and night; the particle system generates rain, snow, and fog, and adjusts the particle density, movement trajectories, and characteristics according to the weather parameters. The particle effect is linked with the road surface friction coefficient to simulate the impact of different weather on vehicles, such as braking and skidding in rainy days, and reducing speed due to reduced visibility in foggy days.

[0073] Finally, based on the behavior rules of traffic participants in the target real scenario, set path planning algorithms, obstacle avoidance decision-making logics, and random behavior parameters for virtual vehicles and virtual pedestrians, and simulate the environmental perception and dynamic interaction of virtual vehicles and virtual pedestrians through ray detection and physical collision mechanisms to form a dynamic physical simulation environment including variable traffic flow densities.

[0074] In this embodiment, the behavior rules of traffic participants are based on real traffic observations and analyses, combined with surveillance videos, accident cases, regulations, and psychological theories to design the path planning and obstacle avoidance logics of virtual vehicles and pedestrians. Vehicles obey traffic lights, keep a safe distance, and avoid pedestrians; simulate radar to sense obstacles through ray detection and make decisions based on distance and speed, and decide to drive or stop at intersections according to traffic light rules. Pedestrians also follow behaviors such as avoiding obstacles and crossing the street randomly based on the ray monitoring mechanism. These rules, combined with ray detection and physical collision mechanisms, make the virtual traffic more realistic and diverse.

[0075] Through the above implementation steps, the embodiments of the present invention can construct a highly realistic dynamic physical simulation environment, providing a reliable and flexible virtual platform for the training and testing of neural network models. This virtual environment can not only simulate various complex scenarios in the real world but also be customized according to test requirements, providing strong support for improving the generalization ability and reliability of neural network models.

[0076] In the embodiments of the present invention, the deployment and inference process of the neural network model in the virtual test vehicle is further refined to ensure that the model can operate efficiently in the virtual environment and provide accurate inference results. The following are the specific implementation steps of the method: First, convert the neural network model into an Open Neural Network Exchange (ONNX) format file, and load the ONNX format file through a 3D engine to create a local inference engine.

[0077] In this embodiment, the neural network model trained with real data is converted into the ONNX format to achieve cross-platform operation through a serialized structure and weights. When converting, it is necessary to adapt the input and output layers to ensure consistency with the virtual environment data format, such as converting images to the RGB format. After being converted to ONNX, it can be flexibly deployed locally without front-end and back-end communication, facilitating the loading of models with different structures and different decision-making tasks, improving adaptability and interaction efficiency, and enabling the model to be directly used in the real scenarios of edge devices after virtual testing, enhancing generality and convenience. Subsequently, load the ONNX file through a 3D engine and create a local inference engine. In Unity, it is necessary to import the ONNX file as a resource and initialize the AI inference engine, including allocating memory, configuring computing resources (such as GPU acceleration), and setting hyperparameters (such as batch size, frequency, etc.).

[0078] Then, use a virtual camera to capture the image data of traffic participants in the dynamic physical simulation environment in real time, and collect radar data within the fan-shaped detection area through a radar component. After resetting the resolution and channel order of the image data and radar data, input them into the neural network model. The radar data includes obstacle distance and azimuth data.

[0079] In terms of data collection, the virtual camera can capture the images of traffic participants in the dynamic simulation environment in real time, and its parameters (such as resolution, frame rate, viewing angle) need to be adjusted according to the model requirements. For example, if the model requires 1280*720 pixels, the camera output should be set to the corresponding size. The position and angle of the camera also need to be set according to the situation of the test vehicle to truly reflect the visual perception range. The radar component is responsible for collecting data such as the distance, azimuth, and speed of obstacles within the fan-shaped area, which is achieved through simulated ray detection. Parameters such as the number of rays, length, and angle need to match the actual radar. After collection, coordinate transformation, filtering, and fusion are required to ensure the accuracy of the data.

[0080] In specific implementation, to improve the safety of the neural network in complex environments, machine vision algorithms can be used to obtain surrounding information in real time, especially the traffic conditions ahead. Combining the digital twin camera images and radar data, accurately identify pedestrians, vehicles, obstacles, traffic lights, and lane lines. The vision module focuses on achieving object detection, traffic light recognition, and lane line detection, among which object detection and traffic light recognition are completed by the trained neural network.

[0081] In practical applications, more than 4,650 traffic images covering different time periods, weather conditions, and locations were collected through mobile phone shooting, online image screening, and public datasets, including 650 traffic signal images and 4,000 pedestrian and vehicle images, to improve the generalization ability of the model. Subsequently, the traffic signals were classified and the pedestrians and vehicles were labeled to generate YOLO format labels to ensure the accuracy and consistency of the data. The AlexNet and YOLOv7 models were built using Pytorch to train the traffic signal recognition and object detection models respectively, and the hyperparameters were optimized to improve the recognition accuracy and efficiency. To enhance the cross-platform deployment ability, the models trained with Pytorch were converted to the ONNX format, and the preprocessing was optimized and the structure was adjusted to facilitate direct integration into the virtual test scenario, improving the performance and facilitating model replacement.

[0082] Through the above steps, the embodiments of the present invention successfully deployed a computer vision model, providing a reliable decision-making basis for autonomous driving in virtual scenarios and effectively improving the performance of the model in complex traffic environments.

[0083] In a specific embodiment of the present invention, the local deployment and inference of the ONNX model are realized through the Unity Sentis engine, and the image preprocessing and postprocessing are completed in combination with OpenCV. The specific steps are as follows: First, the ONNX model file is specified as the Unity resource path, loaded, and an inference engine is created to ensure the efficient operation of the model in Unity. Next, each frame of the virtual camera image is written to the renderer and converted into Texture data, and the resolution (640*640 pixels in this example) and channel order (such as converting BGR to RGB) are adjusted to match the model input requirements. Subsequently, the Texture is converted into a Tensor and input into the model, and the asynchronous inference method is adopted to feedback the results in a timely manner, including object detection boxes, traffic signal states, and lane line information. Finally, OpenCV is used to visualize the results, such as labeling the objects with different colors and displaying the traffic signal states, providing intuitive feedback to the testers and guiding the driving behavior of the vehicle in the virtual scenario.

[0084] This method efficiently realizes the deployment and inference of the ONNX model, ensuring the real-time performance and accuracy in the virtual environment, and providing a reliable tool for the evaluation of autonomous driving models.

[0085] The resolution and channel order of the image data are reset and then input into the neural network model. This process involves normalizing the data to eliminate the dimensional differences between different sensor data. For example, the pixel values of the image data usually need to be normalized to the range of [0, 1]. In addition, the channel order of the image data (such as RGB or BGR) needs to be adjusted according to the input requirements of the model to ensure that the model can correctly parse the input data.

[0086] After that, asynchronously execute the model inference process, and output the original inference data including the positions of the target detection boxes, the classification of the traffic signal states, and the results of the lane line detection.

[0087] In this embodiment, the model inference process is executed asynchronously to ensure that the inference results can be timely fed back to the control system of the virtual test vehicle. During each frame of the inference process, the model outputs information such as the positions of the target detection boxes, the classification results of the traffic signal states, and the results of the lane line detection. These original inference data need to undergo post-processing, including steps such as confidence threshold filtering of the classification results, non-maximum suppression (NMS) to eliminate redundant detection boxes, and fitting optimization of the lane lines, etc., to improve the accuracy and reliability of the inference results.

[0088] Based on the original inference data, the system can count the number of correctly recognized targets and the number of missed detections in the target detection to calculate the precision and recall rate. The precision rate reflects the proportion of correct targets among the detected targets, while the recall rate reflects the proportion of real targets that the model can detect. These metrics are calculated as follows: The precision rate refers to the proportion of the number of correctly recognized objects among all the recognized objects. Its specific calculation method is: divide the number of correctly recognized objects by the sum of the number of correctly recognized objects and the number of misrecognized objects, that is, the precision rate is equal to the number of correctly recognized objects divided by (the number of correctly recognized objects plus the number of misrecognized objects).

[0089] The recall rate refers to the proportion of the number of successfully and correctly recognized objects among all the actually existing objects that need to be recognized. Its specific calculation method is: divide the number of correctly recognized objects by the sum of the number of correctly recognized objects and the number of missed detections, that is, the recall rate is equal to the number of correctly recognized objects divided by (the number of correctly recognized objects plus the number of missed detections).

[0090] Meanwhile, the system can record the collision frequency by monitoring the collision events between the virtual test vehicle and the obstacles in the environment. The detection of collision events can be achieved through the collision detection mechanism of the physical engine. Whenever a collision occurs between the virtual vehicle and an obstacle, the system will record information such as the time, position, and speed of the collision.

[0091] Finally, the precision rate, recall rate, and collision frequency can be overlaid on the virtual camera image, and the inference results can be rendered in real time through a 3D visualization interface, and the performance metrics corresponding to the inference results and the corresponding dynamic physical simulation environment parameters can be marked on the human-machine interaction interface.

[0092] In this embodiment, the system superimposes the precision rate, recall rate, and collision frequency on the virtual camera screen and displays them in real time through a three-dimensional visualization interface. The target detection boxes are marked with different colors (for example, green for vehicles and red for pedestrians), the signal light status is displayed in text or icons, and the lane lines are drawn as dotted or solid lines. At the same time, performance indicators and environmental parameters, such as weather, traffic density, and signal light status, are marked on the human-computer interaction interface to help users comprehensively understand the model performance.

[0093] The above steps can ensure the efficient deployment and inference of the neural network in the virtual environment, providing a reliable evaluation tool for scenarios such as autonomous driving. This method improves the test efficiency and accuracy and also provides a flexible and scalable test platform for researchers to support the evaluation and optimization of multiple models.

[0094] In the embodiment of the present invention, the neural network model training method based on virtual reality and simulation further details how to identify the performance defects of the neural network model and determine the types of target test scenarios to be optimized based on these defects. The following are the specific implementation steps of this method: First, when the detection precision rate of the inference result is lower than the preset precision rate threshold, the recall rate is lower than the preset recall rate threshold, or the virtual test collision frequency is higher than the preset collision frequency threshold, it is determined that the neural network model has performance defects.

[0095] In the virtual test, the system continuously monitors and records the inference results of the neural network model, including the precision rate, recall rate, and collision frequency of target detection. The precision rate reflects the accuracy of the model in identifying targets, the recall rate represents the proportion of real targets detected, and the collision frequency reflects the safety of the model in complex environments. These indicators are calculated by the real-time data acquisition and analysis module and compared with the preset thresholds.

[0096] When the precision rate of the model inference result is lower than the threshold, it means that the model has a high false alarm rate for some targets (such as pedestrians and vehicles). For example, the precision rate of pedestrian detection in rainy days is only 75%, lower than the threshold of 90%, indicating insufficient detection ability. If the recall rate is lower than the threshold, it means that the missed detection is serious. For example, the recall rate of vehicles at night is 60%, lower than 85%, indicating insufficient coverage. If the collision frequency is too high, it means that the model makes incorrect decisions, which may be related to insufficient accuracy of lane line detection, traffic signal recognition, or too close target distance.

[0097] Based on the model performance analysis results, the system can identify the possible defects of the model related to specific environments (such as rain, snow, and night) or scenarios (such as high-density traffic and complex intersections). For example, the poor pedestrian detection in rainy days may be due to image noise interference; the low recall rate under low light at night may be due to the weak ability of the model to recognize low-contrast targets.

[0098] Then, based on the types and distribution laws of performance defects, match the corresponding combination of scene parameters in the dynamic physical simulation environment, and determine at least one type among the rain, snow, fog weather, low-light conditions at night, or dense traffic flow scenarios as the target test scene type to be optimized.

[0099] To optimize the model performance, the system can match the scene parameters in the dynamic physical simulation according to the types and distribution of performance defects. For example, if the model performs poorly in rainy weather, the system sets the rain and snow weather as the optimization target and adjusts parameters such as raindrop density and road surface reflectivity; if the light is insufficient at night, it reduces the light and adjusts the brightness of street lights; if the traffic flow is dense, it increases the number and complexity of virtual vehicles. In this way, the system can dynamically generate a high-fidelity synthetic dataset related to performance defects, including environmental information (such as weather, light, traffic density) and detailed annotations of the target position, type, and status, ensuring data compatibility and usability. For example, the rainy day dataset annotates raindrop density, the positions of pedestrians and vehicles, and the status of traffic lights, and the format is consistent with real data. This method can effectively identify model defects, determine the optimization scenarios, improve the training efficiency, enhance the robustness and reliability of the model in complex scenarios, and provide technical support for autonomous driving.

[0100] In the embodiment of the present invention, the neural network model training method based on virtual reality and simulation further details how to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate a specific target test scene type and generate the corresponding high-fidelity synthetic dataset. The following are the specific implementation steps of this method: First, the traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment can be adjusted in real time through the human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environments in the target test scene type.

[0101] In this embodiment, to generate a high-fidelity synthetic dataset corresponding to the target test scene, it is necessary to adjust the parameters in the dynamic physical simulation environment in real time through the human-computer interaction interface, including traffic flow density, signal light logic, rain and snow concentration, and light intensity. For example, in an extreme weather scene, it is necessary to increase the rain and snow concentration and reduce the light; in a high-density traffic scene, it is necessary to increase the number and complexity of participants and adjust the signal light logic to simulate congestion.

[0102] Then, based on the adjusted dynamic physical simulation environment, drive the virtual camera component and the radar component to synchronously collect real-time image data and radar point cloud data including target detection box annotations, signal light state classification, and lane line annotations, and record the driving trajectory, collision events, and model inference results of the digital twin of the test vehicle.

[0103] In this embodiment, the virtual camera and the radar synchronously collect images and point cloud data including target detection frames, signal light states, and lane line annotations. The camera captures visual information at a preset frequency and resolution, and the radar obtains the distance, azimuth, and speed of obstacles through ray detection. The data is stored together with the trajectory, collision events, and inference results of the vehicle digital twin for subsequent processing.

[0104] Finally, multi-modal data fusion and standardized annotation processing are performed on the real-time image data, radar point cloud data, driving trajectory, collision events, and model inference results to generate a high-fidelity synthetic dataset that matches the target test scenario type. Among them, the annotation format, data dimension, and storage structure of the high-fidelity synthetic dataset are compatible with the real dataset, and it contains multi-dimensional labels of environmental parameters, vehicle behavior, and model decisions.

[0105] In this embodiment, multi-modal data fusion and standardized annotation are performed on real-time images, radar point clouds, driving trajectories, collision events, and model inference results. Multi-modal fusion aligns different data on the time axis to provide a consistent scene description; standardized annotation includes target box position, signal light state, lane line fitting, etc., to ensure a unified data format.

[0106] The generated high-fidelity synthetic dataset contains rich environmental information and detailed annotations, such as weather, lighting, traffic density, as well as target position, type, and status. The format is consistent with the real data, ensuring compatibility. For example, in a rainy day scenario, each image is annotated with raindrop density, pedestrian and vehicle positions, and signal light states, and contains multi-dimensional labels of environmental parameters, vehicle behavior, and model decisions, providing comprehensive support for model training.

[0107] To ensure data quality, the system verifies and calibrates the data. Verification checks the integrity and consistency of the data to ensure accurate annotation; calibration performs normalization processing to eliminate the dimensional differences of sensors, facilitating neural network parsing.

[0108] Through the above steps, this embodiment can dynamically adjust the physical simulation parameters, generate a high-fidelity dataset for the corresponding test scenario, improve training diversity, optimize the performance of the model in specific scenarios, and enhance the generalization and reliability of the model.

[0109] In the embodiment of the present invention, the neural network model training method based on virtual reality and simulation further details how to perform iterative training by mixing the high-fidelity synthetic dataset and the real dataset, and forms a closed-loop optimization mechanism to improve the generalization ability of the neural network model in complex scenarios. The following are the specific implementation steps of this method: First, mix the high-fidelity synthetic dataset and the real dataset according to a preset ratio to generate a mixed training set.

[0110] In this embodiment, the generated high-fidelity synthetic dataset can be mixed with the real dataset according to a preset ratio to generate a mixed training set. In practical applications, the mixing ratio can be adjusted according to the current performance of the model and training requirements. For example, if the performance of the model is poor in a specific scenario, the ratio of the synthetic dataset corresponding to this scenario can be increased to strengthen the model's learning of this scenario. The generation process of the mixed training set needs to ensure the consistency and diversity of the data, that is, the data is consistent in terms of format, annotation method, etc., and at the same time covers data samples under different scenarios and conditions.

[0111] Next, the neural network model is trained iteratively for multiple rounds based on the mixed training set. Among them, after each round of training is completed, the optimized neural network model is redeployed to the digital twin of the test vehicle, and virtual tests are performed through the dynamic physical simulation environment, and the precision, recall rate of the neural network model inference and the virtual test collision frequency are recorded.

[0112] In this embodiment, when the neural network model is trained iteratively for multiple rounds based on the mixed training set, before the start of each round of training, the training set can be randomly shuffled to avoid bias in the model due to the data order. During the training process, appropriate optimization algorithms (such as Adam, SGD, etc.) and loss functions (such as cross-entropy loss, mean square error, etc.) can be used, and the model parameters are adjusted according to the difference between the output of the model and the labeled data. The hyperparameters (such as learning rate, batch size, etc.) during the training process need to be dynamically adjusted according to the convergence situation and performance of the model.

[0113] After each round of training is completed, the optimized neural network model is redeployed to the digital twin of the test vehicle. The redeployment process involves converting the updated model weights and structure files into a format compatible with the virtual test environment (such as ONNX), and loading them into the test vehicle through a 3D engine. The redeployed model needs to perform virtual tests in the dynamic physical simulation environment to evaluate its performance improvement.

[0114] During the virtual test process, the system can record performance metrics such as the precision, recall rate of the neural network model inference and the virtual test collision frequency. These metrics are compared with preset performance thresholds to determine whether the model meets the expected performance standards. If the precision, recall rate and collision frequency all reach the preset thresholds, it is considered that the performance of the model in the current scenario meets the requirements, and the training process can be ended. Otherwise, it is necessary to further analyze the performance defects of the model and dynamically adjust the scenario parameters of the dynamic physical simulation environment accordingly.

[0115] Dynamically adjusting the scene parameters of a dynamic physical simulation environment aims to generate more challenging test scenarios to expose potential defects in the model. For example, if the collision frequency of the model is high in a rainy day scenario, the raindrop density can be further increased, the light intensity can be decreased, or the traffic flow can be increased to simulate more extreme weather and traffic conditions. The adjusted scene parameters need to match the type and distribution law of the performance defects to ensure that the generated test scenarios can specifically optimize the model performance.

[0116] In addition, if the precision, recall, and virtual test collision frequency do not reach the preset thresholds, the scene parameters of the dynamic physical simulation environment are dynamically adjusted according to the performance defects of the current neural network model to generate a new high-fidelity synthetic dataset and update the mixed training set.

[0117] In this embodiment, based on the adjusted dynamic physical simulation environment, the data collection and synthetic dataset generation processes are re-executed. The newly generated synthetic dataset contains more complex and diverse scene information and is used to update the mixed training set. In this way, the diversity and difficulty of the training data are continuously enriched, promoting the model to improve its performance in a wider range of scenarios.

[0118] Finally, the steps of iterative training, deployment testing, and scene parameter adjustment are repeated until the generalization ability of the neural network model in complex scenarios such as rainy and snowy weather, night lighting, and dense traffic flow reaches the preset threshold, forming a closed-loop optimization mechanism for the linkage of virtual and real data.

[0119] In this embodiment, by repeating the above steps of iterative training, deployment testing, and scene parameter adjustment until the generalization ability of the neural network model in complex scenarios such as rainy and snowy weather, night lighting, and dense traffic flow reaches the preset threshold. This process forms a closed-loop optimization mechanism for the linkage of virtual and real data. By continuously supplementing the real dataset with the synthetic dataset and guiding the model optimization with the virtual test results, the continuous improvement of the model performance is ultimately achieved.

[0120] Through the above implementation steps, the embodiment of the present invention can effectively utilize the high-fidelity synthetic dataset to optimize the training process of the neural network model, form a closed-loop optimization mechanism, and significantly improve the generalization ability and reliability of the model in complex scenarios. This training method based on virtual reality and simulation provides an efficient, flexible, and low-cost solution for the development of neural network models in fields such as autonomous driving.

[0121] In one embodiment of the present invention, the system meets the requirements of autonomous driving virtual testing, supports users to efficiently collect synthetic data in a virtual scenario, supplements the deficiency of real data, and improves the model training effect. By simulating complex scenarios, users can dynamically adjust environmental parameters (such as traffic, lighting, weather, etc.) to generate synthetic data highly similar to real scenarios, which contains rich environmental information and detailed annotations, such as object detection boxes, signal light states, lane line positions, etc., effectively improving data diversity and coverage.

[0122] To verify the effectiveness of the synthetic data, a model experiment was conducted. Taking signal light recognition as an example, after training with a mixed dataset (250 real + 500 synthetic), the model accuracy was 92.9%, and the average correct rate in 100 rounds of virtual testing was 92.3%; while the model trained with all real data (650 pieces) had an accuracy of 93.2%, and the average correct rate in 100 rounds of virtual testing was 92.6%. It can be seen that the synthetic dataset can effectively supplement the real dataset and improve the model generalization ability.

[0123] To further verify the object detection model, under the training of a mixed dataset (1000 real + 1000 synthetic), the precision was 85% - 86%, the recall rate was 80% - 85%, and the correct rate of virtual testing was 92.5%; while when only using real data (4000 pieces), the precision was 83% - 85%, the recall rate was 80% - 82%, and the correct rate of virtual testing was 90.3%. The mixed data was only 2000 pieces, and the effect was equivalent to double the real data, indicating that synthetic data can improve the training efficiency and performance when real samples are limited.

[0124] In summary, this embodiment proves the effectiveness of synthetic data in neural network training. The model trained with mixed data still has high generalization ability in real scenarios, and the virtual test results accurately reflect the training effect, which is persuasive. This method reduces the data collection and model training costs, providing support for the efficient training of autonomous driving models.

[0125] In a specific embodiment of the present invention, to improve the flexibility and adaptability of testing, the system provides two virtual testing modes: fully automatic testing and manual control testing. In the fully automatic testing mode, users do not need to intervene. Once the test is started, the test vehicle will execute the autonomous driving task according to the driving route pre-calculated by the system. During this process, the vehicle will automatically adjust the driving speed, direction, and obstacle avoidance strategy according to the real-time perceived environmental information and the preset decision logic. The test will continue until the vehicle successfully reaches the specified target location, or collides with an unavoidable obstacle, and is forced to stop driving and end the current test round. This mode is particularly suitable for batch testing and automated evaluation of model performance, and can efficiently collect a large amount of test data for subsequent model optimization and performance analysis.

[0126] In contrast, the manual control test mode gives users more control. In this mode, the user directly drives the test vehicle, while the system focuses on providing an intuitive user interface that gives real-time feedback on the vehicle's decision-making information, such as object detection results, signal light status, lane line positions, and potential collision risks. However, the system does not actually participate in the vehicle control process, and all driving operations are completed by the user themselves. This mode is mainly used for detailed testing in specific scenarios, or for training users' understanding and operation capabilities of the autonomous driving system. It also provides an opportunity for researchers to observe and analyze the performance of the model under human intervention.

[0127] To achieve seamless switching between these two test modes, the system is designed with a flexible control architecture. In the full-automatic test mode, the system's autonomous driving algorithm takes over all vehicle control, while the user interface is mainly used to display the vehicle's driving status and system decision-making information. In the manual control test mode, the user interface provides rich control options, including simulated controls for the steering wheel, accelerator, and brakes, as well as real-time environmental perception data and decision-making suggestions. This design ensures that users can flexibly switch between different test requirements, thus maximizing the efficiency and effectiveness of testing.

[0128] In addition, the system is equipped with a detailed logging module to record all key events and data during the test. Whether it is a full-automatic test or a manual control test, the system records the vehicle's driving trajectory, sensor data, decision-making information, and any collision events that occur. These data are not only used for performance evaluation of the current test but also serve as important inputs for subsequent model iterative training to help further optimize the performance and robustness of the neural network model.

[0129] By providing these two complementary test modes, the present invention can meet the diverse needs in different test scenarios. Whether it is large-scale automated testing or refined manual evaluation, it can be effectively supported. This flexibility not only improves the testing efficiency but also enhances the reliability and comprehensiveness of the test results, providing solid support for the research and verification of autonomous driving technology.

[0130] In a specific embodiment of the present invention, a flexible and feature-rich human-machine interaction interface is provided for users to support the virtual testing and optimization of the autonomous driving model. This interface allows users to freely select single-model testing (such as signal light recognition, object detection, lane line detection, etc.) or multi-model fusion testing to meet different testing requirements. Users can intuitively adjust environmental parameters such as traffic flow, light intensity, and weather conditions (such as rain, snow, and fog) through the interface to simulate testing conditions in various complex scenarios. In addition, the interface also provides a variety of perspective switching functions, and users can select front camera, rear camera, left camera, right camera, or bird's-eye view perspectives according to their needs to comprehensively observe the behavior of the test vehicle and its interaction with the environment.

[0131] Figure 4 Schematic diagram of a human-machine interaction interface provided by an embodiment of the present invention.

[0132] As Figure 4 shown, the test interface displays the vehicle status, perception data, etc. in real time, and shows indicators such as recognition accuracy, number of collisions, and trajectory deviation through charts. The signal light recognition rate is displayed as a percentage, pedestrians and vehicles are detected and marked with different colors, and the lane lines are presented as dotted or solid lines. The system also records detailed logs, including timestamps, environmental parameters, model inference results, and collision information. To enhance the experience, the interface provides a data visualization and analysis function, supports performance comparison, scenario performance analysis, and result export, and helps optimize the model. Through intuitive display and detailed recording, users can quickly locate problems, adjust parameters, and form a closed-loop optimization mechanism. This interface is flexible and powerful, improves testing efficiency and accuracy, provides an easy-to-use platform for researchers, supports the evaluation of multiple neural network models, and helps the development of autonomous driving technology.

[0133] In a specific embodiment of the present invention, by constructing a high-fidelity virtual test environment, combining dynamic scenario parameter adjustment with virtual-real data hybrid training, a closed-loop optimization mechanism is formed, thereby improving the generalization ability and decision reliability of the neural network model in complex traffic scenarios. The following is a detailed description in combination with specific embodiments: In the specific implementation process, first complete the preparation work of the virtual test system, that is, first construct the driving decision logic of the digital twin of the test vehicle and initialize the environmental parameters. By combining and configuring parameters such as time, number of people flow, and weather, simulate the environment required for testing, such as setting the time as "19:00 - 21:00", traffic flow as "high", and rainy and snowy weather, to construct a complex scenario with low visibility and dense traffic flow.

[0134] The autonomous driving decision-making module of the digital twin of the test vehicle controls the vehicle by fusing the neural network output and the simulation radar data. The decision logic is based on the behavior tree structure (see Figure 5Implement dynamic control of driving behaviors such as vehicle deceleration and following based on information such as traffic lights, obstacles, and vehicle distance to simulate real driving decisions.

[0135] Lane line detection ensures the stability of autonomous driving. In this embodiment, Canny edge detection combined with Hough transform is used to identify lane lines, and the results are displayed in the visualization interface. After successful detection, the endpoint coordinates of the lane lines are obtained to fit the center line, and the steering is adjusted according to the angle between the vehicle driving direction and the center line. When the threshold is exceeded, correction is performed to achieve lane keeping. When the vehicle approaches the center line, the steering angle is optimized by calculating the vanishing point offset to improve the trajectory stability.

[0136] Specifically, in a two-dimensional coordinate system, the intersection point of the extension lines of the left and right lane lines is defined as the vanishing point , and the midpoint of the lower endpoints of the lane lines is . By calculating the offset, the steering angle is finely adjusted, and the adjustment amplitude is negatively correlated with the vehicle speed to ensure safety. The calculation formula for the vanishing point coordinates is as follows: (2); Among them, , are the slopes of the left and right lane lines respectively, and , are the coordinates of any point on the left and right lane lines respectively.

[0137] If the lane lines cannot be detected, the lane keeping system pauses, and the digital twin of the test vehicle will perform steering according to the virtual navigation path, such as passing through intersections or right-angle bends.

[0138] To illustrate the process of lane line recognition and measurement in the virtual reality autonomous driving test system, including image acquisition, preprocessing, edge detection, lane offset calculation, and visualization, and to provide a basis for subsequent control, it will be described in detail below in combination with the attached drawings and examples.

[0139] Figure 6 This is a flowchart of a method for lane line detection and lane offset calculation in a virtual reality autonomous driving test system provided by an embodiment of the present invention. As Figure 6 shown, the method includes the following steps.

[0140] 1. Check whether the camera is successfully opened: When the system starts, the status of the virtual camera is first detected. If the camera initialization fails (such as insufficient permissions or not enabled), the program continuously polls and detects until the status is normal. After successful detection, the next stage of the data acquisition process is triggered. In this embodiment, if the check result is negative, the loop continues to check. If it is positive, go to the next step.

[0141] 2. Capture camera data frame by frame in the Update() function: The Update() function is called in each frame rendering cycle to capture the RGB image data of the scene in front of the current vehicle through the virtual camera component. In this embodiment, the image resolution is adapted to 650×360 pixels.

[0142] 3. Process the frame in the ProcessFrame() function: This function converts the Texture2D texture image into a Mat image and starts the lane line detection pipeline. The detection process includes the following sub-steps: 3.1 Convert the frame to a grayscale image: Use the cvtColor() function in OpenCV to convert the color image to a grayscale image, eliminating color interference and reducing the subsequent computational complexity.

[0143] 3.2 Apply Gaussian blur: Perform Gaussian filtering with a 3×3 convolution kernel on the grayscale image through the GaussianBlur() function to smooth out noise interference and improve the stability of edge detection.

[0144] 3.3 Apply Canny edge detection: Set double thresholds (e.g., a low threshold of 50 and a high threshold of 150), and use the Canny() function to extract the edge contours of the image, preferentially retaining the lane line features.

[0145] 3.4 Apply Canny edge detection again: Adjust the thresholds again (e.g., a low threshold of 30 and a high threshold of 100) for edge enhancement to compensate for the weak edges missed in the first detection.

[0146] 4. Define and apply the region of interest (ROI): Set a trapezoidal mask region to focus on the ground area in front of the vehicle (e.g., the ground area within the range of 3 to 10 meters in front of the vehicle), excluding irrelevant background interferences such as the sky and buildings.

[0147] 5. Detect lines using the Hough transform: Call the HoughLinesP() function to perform the probabilistic Hough transform. In this embodiment, the parameter configuration is as follows: Resolution 1 pixel (indicating the detection of straight lines), Resolution / 180 radians, voting threshold 20, minimum line segment length 20 pixels, line segment interval 30 pixels. Extract the lane line candidate set.

[0148] 6. Filter and fit the lane lines: Select the set of line segment points with a slope greater than 0.4, and divide the left and right lane lines according to the positive and negative of the slope.

[0149] 7. Draw the detected lines on the frame: Map the line segment endpoint coordinates output by the Hough transform to the original image. In this embodiment, draw them superimposed with green lines (BGR value [0, 255, 0]) to visualize the lane line detection results.

[0150] 8. Convert Mat to Texture2D for display in Unity: Use Unity's OpenCV plug-in to convert the processed Mat image data to Texture2D format to adapt to the Unity rendering pipeline requirements.

[0151] 9. Lane offset calculation and lane keeping. In this embodiment, the following operations are performed: Extract the coordinates of the left and right lane line endpoints , , calculate the midpoint ; Calculate the vanishing point according to formula (2) ,in , To fit the slope of the straight line; calculate the lateral offset , combined with the current vehicle speed (unit: ) Dynamically adjust the steering angle ,in is the sensitivity coefficient (default );like Exceeding the threshold (In this embodiment =0.3m), triggering the direction correction mechanism and outputting the steering control command to the vehicle dynamics model.

[0152] 10. Record the offset and direction correction log information: Value, steering angle The corresponding timestamp is written into the log file for subsequent virtual test analysis.

[0153] 11. Display the Texture2D texture image frame by frame in the Unity renderer: Render the generated Texture2D to the Raw Image component of the Unity scene, and display the driving perspective image including the lane line markings in real time in the 3D visualization interface.

[0154] 12. End: Complete a single lane line detection cycle and wait for the next frame to trigger a new round of processing.

[0155] This embodiment achieves highly robust lane line recognition and dynamic trajectory correction through edge detection and geometric modeling of virtual-real fusion, verifying the effectiveness of virtual reality scenes in the closed-loop test of autonomous driving "perception-decision-making". Based on the above-mentioned virtual reality and simulation-based neural network model training method provided by the embodiment of the present invention, Figure 7As shown in the figure, an embodiment of the present invention further provides a neural network model training device based on virtual reality and simulation, including: a generation module 701, a construction module 702, an inference module 703, an identification module 704, a generation module 701, and an iterative training module 705.

[0156] The generation module 701 is configured to generate a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements through a parametric three-dimensional modeling method based on the spatial data, physical attributes, and environmental information of the target real scene. The construction module 702 is configured to construct a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of a test vehicle based on vehicle dynamics parameters, environmental light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and a skybox component. The inference module 703 is configured to deploy a neural network model trained based on a real data set to the digital twin of the test vehicle, perform model inference according to the real-time data generated by the virtual camera collecting the dynamic physical simulation environment, and output an inference result. The identification module 704 is configured to identify the performance defects of the neural network model based on the precision rate, recall rate, and virtual test collision frequency of the inference result, and determine the target test scene type to be optimized according to the performance defects. The generation module 701 is further configured to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic data set corresponding to the target test scene type. The iterative training module 705 is configured to perform iterative training on the neural network model by mixing the high-fidelity synthetic data set and the real data set, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenes reaches a preset threshold.

[0157] An embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The above-mentioned memory stores a computer program capable of being executed by the above-mentioned at least one processor, and the above-mentioned computer program, when executed by the above-mentioned at least one processor, is used to cause the electronic device to execute the method of the embodiment of the present invention.

[0158] An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the above-mentioned computer program, when executed by a processor of a computer, is used to cause the above-mentioned computer to execute the method of the embodiment of the present invention.

[0159] An embodiment of the present invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present invention.

[0160] It should be noted that in the embodiments of the present invention, "including" is an open inclusion, that is, "including but not limited to"; "based on" means "at least partially based on"; "one embodiment" means "at least one embodiment", "another embodiment" means "at least one additional embodiment", and "some embodiments" means "at least some embodiments". "One" and "plural" are illustrative modifications and are not restrictive. Unless otherwise clearly stated, they should be understood as "one or more".

[0161] The steps in the method implementation manner can be executed in a different order or in parallel, and steps can also be added or subtracted, and the protection scope is not limited by this.

[0162] "Embodiment" means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment. The word appears in different positions in the text, and does not necessarily refer to the same embodiment, nor does it exclude coexistence with other embodiments. The same or similar parts between the embodiments are referred to each other. In particular, the device, equipment, and system embodiments are basically similar to the method embodiments and are described briefly. For relevant parts, refer to the method embodiments.

[0163] The above are only several implementation manners of the present invention, which are described in detail, but should not be construed as a limitation on the patent protection scope. Those skilled in the art can make several deformations and improvements without departing from the concept of the present invention, and all belong to the protection scope. Therefore, the protection scope of the present invention is subject to the appended claims.

Claims

1. A neural network model training method based on virtual reality and simulation, characterized in that, Including: Based on the spatial data, physical properties, and environmental information of the target real scene, a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements is generated through parametric three-dimensional modeling methods; In the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of a test vehicle based on vehicle dynamics parameters, environmental light intensity, color temperature, and rain, snow, fog weather effects dynamically adjusted through a particle system and a skybox component; Deploy a neural network model trained based on real datasets to the digital twin of the test vehicle, perform model inference according to the real-time data collected by the virtual camera from the dynamic physical simulation environment, and output the inference result; Based on the precision, recall, and virtual test collision frequency of the inference result, identify the performance defects of the neural network model, and determine the target test scene type to be optimized according to the performance defects; Dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic dataset corresponding to the target test scene type; Mix the high-fidelity synthetic dataset and the real dataset to iteratively train the neural network model, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenes reaches a preset threshold.

2. The method according to claim 1, wherein The spatial data includes the high-definition plane map, traffic road network topology structure, and road width measurement data of the target real scene; the physical properties include the building surface material reflection parameters, road friction coefficient threshold, and torque, braking, and suspension parameters in vehicle dynamics parameters; the environmental information includes day-night light gradual change parameters, rain, snow, fog weather particle density, and traffic participant behavior rules; The generating of a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements through parametric three-dimensional modeling methods based on the spatial data, physical properties, and environmental information of the target real scene includes: Based on the spatial data, dynamically adjust the road generation logic, building density, and building style parameters through a predefined rule file, and batch generate a three-dimensional digital city model including two-way four-lane roads, green vegetation, and multiple types of building groups; Based on the physical properties, configure the material reflection parameters for the building surface in a three-dimensional engine, set the friction coefficient threshold for the road surface, and configure the torque, braking, and suspension parameters for the digital twin of the test vehicle; Based on the environmental information, simulate the day-night light change through a skybox component, dynamically generate rain, snow, fog weather special effects using a particle system, and drive the path planning and obstacle avoidance decisions of virtual vehicles and virtual pedestrians according to the traffic participant behavior rules to form the three-dimensional visual virtual reality scene including dynamic traffic elements.

3. The method according to claim 1, wherein The constructing of a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene includes: Based on the traffic signal state switching logic of the target real scenario, define the signal configuration rules and switching timings, simulate the reverse synchronous switching of the oncoming lane signals, and control the signals at the same intersection to cyclically change the traffic states according to a preset period; Based on the vehicle dynamics parameters of the target real scenario, configure the torque output curve, braking response delay, and suspension stiffness threshold for the digital twin of the test vehicle, and simulate the physical contact and friction effects between the vehicle and the road through the wheel collider component; Based on the ambient light and weather parameters of the target real scenario, dynamically adjust the intensity and color temperature of the directional light through the skybox component to simulate the gradual change between day and night, generate the particle density and movement trajectories of rain, snow, and fog weather using the particle system, and link with the friction coefficient threshold of the road surface to realize the simulation of the impact of weather on vehicle dynamics; Based on the behavior rules of traffic participants in the target real scenario, set the path planning algorithm, obstacle avoidance decision logic, and random behavior parameters for virtual vehicles and virtual pedestrians, and simulate the environmental perception and dynamic interaction of the virtual vehicles and virtual pedestrians through the ray detection and physical collision mechanisms to form the dynamic physical simulation environment including variable traffic flow densities; 4. The method according to claim 1, characterized in that, Deploy the neural network model trained based on the real dataset to the digital twin of the test vehicle, perform model inference according to the real-time data collected by the virtual camera from the dynamic physical simulation environment, and output the inference results, including: Convert the neural network model into an Open Neural Network Exchange (ONNX) format file, load the Open Neural Network Exchange format file through the 3D engine and create a local inference engine; Real-time capture the image data of traffic participants in the dynamic physical simulation environment through the virtual camera, and input the image data into the neural network model after resetting the resolution and channel order; Asynchronously execute the model inference process, and output the original inference data including the position of the target detection box, signal state classification, and lane line detection results; Based on the original inference data, count the correct recognition number and missed detection number of target detection to calculate the precision and recall rate, and synchronously record the number of collision events in the 3D visual virtual reality scene to generate the collision frequency; Overlay the precision rate, recall rate, and collision frequency with the virtual camera screen, render the inference results in real time through the 3D visual interface, and label the performance indicators corresponding to the inference results and the corresponding dynamic physical simulation environment parameters in the human-computer interaction interface; 5. The method according to claim 1, characterized in that, Based on the precision rate, recall rate, and virtual test collision frequency of the inference results, identify the performance defects of the neural network model, and determine the type of target test scenario to be optimized according to the performance defects, including: When the detection precision rate of the inference result is lower than the preset precision rate threshold, the recall rate is lower than the preset recall rate threshold, or the virtual test collision frequency is higher than the preset collision frequency threshold, it is determined that the neural network model has performance defects; Based on the type and distribution law of the performance defects, match the corresponding combination of scene parameters in the dynamic physical simulation environment, and determine at least one of the types of rain, snow, fog weather, low-light conditions at night, or dense traffic flow scenarios as the target test scene type to be optimized.

6. The method according to claim 1, characterized in that, Dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic dataset corresponding to the target test scene type, including: Real-time adjust the traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment through the human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environment in the target test scene type; Based on the adjusted dynamic physical simulation environment, drive the virtual camera component to synchronously collect real-time image data including target detection box annotation, signal light state classification, and lane line annotation, and record the driving trajectory, collision events, and model inference results of the digital twin of the test vehicle; Perform multi-modal data fusion and standardized annotation processing on the real-time image data, driving trajectory, collision events, and model inference results to generate a high-fidelity synthetic dataset matching the target test scene type. Among them, the annotation format, data dimension, and storage structure of the high-fidelity synthetic dataset are compatible with the real dataset, and include multi-dimensional labels of environmental parameters, vehicle behavior, and model decisions.

7. The method according to claim 6, characterized in that, Mix the high-fidelity synthetic dataset and the real dataset to iteratively train the neural network model, and redeploy the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold, including: Mix the high-fidelity synthetic dataset and the real dataset according to a preset ratio to generate a mixed training set; Based on the mixed training set, perform multiple rounds of iterative training on the neural network model. Among them, after each round of training is completed, redeploy the optimized neural network model to the digital twin of the test vehicle, perform virtual tests through the dynamic physical simulation environment, and record the precision, recall, and virtual test collision frequency of the neural network model inference; If the precision, recall, and virtual test collision frequency do not reach the preset threshold, dynamically adjust the scene parameters of the dynamic physical simulation environment according to the performance defects of the current neural network model, generate a new high-fidelity synthetic dataset, and update the mixed training set; Repeat the steps of iterative training, deployment testing, and scene parameter adjustment until the generalization ability of the neural network model in complex scenarios of rain and snow weather, night lighting, and dense traffic flow reaches a preset threshold, forming a closed-loop optimization mechanism of virtual and real data linkage.

8. A neural network model training device based on virtual reality and simulation, characterized in that, Including: A generation module for generating a three-dimensional visual virtual reality scene including buildings, roads, and dynamic traffic elements through a parametric three-dimensional modeling method based on the spatial data, physical attributes, and environmental information of the target real scene; A building module for constructing a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic signal state switching logic and traffic participant behavior rules, a digital twin of a test vehicle based on vehicle dynamics parameters, ambient light intensity, color temperature, and rain, snow, fog weather effects dynamically adjusted through a particle system and a skybox component; An inference module for deploying a neural network model trained based on real datasets to the digital twin of the test vehicle, performing model inference according to real-time data generated by a virtual camera collecting the dynamic physical simulation environment, and outputting an inference result; An identification module for identifying performance defects of the neural network model based on the precision, recall rate, and virtual test collision frequency of the inference result, and determining the type of target test scenario to be optimized according to the performance defects; The generation module is further configured to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the type of target test scenario, and generate a high-fidelity synthetic dataset corresponding to the type of target test scenario; An iterative training module for iteratively training the neural network model by mixing the high-fidelity synthetic dataset and the real dataset, and redeploying the optimized neural network model to the digital twin of the test vehicle to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

9. An electronic device, comprising: A processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to execute the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images