Neural network model training method and related equipment based on virtual reality and simulation

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a neural network model training method based on virtual reality and simulation and related equipment. The method includes: generating a three-dimensional visual virtual reality scene based on the target real scene and constructing a dynamic physical simulation environment; deploying the neural network model to the digital twin of the test vehicle to perform model reasoning and output the reasoning results; identifying the target test scene type to be optimized for the neural network model based on the reasoning results; dynamically adjusting the scene parameters of the dynamic physical simulation environment to simulate the target test scene type and generate a high-fidelity synthetic data set; mixing the high-fidelity synthetic data set with the real data set to iteratively train the neural network model to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold. Based on the above method, the training efficiency and generalization ability of the neural network model in complex scenarios can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a neural network model training method based on virtual reality and simulation and related equipment. Background Art

[0002] In recent years, with the widespread application of deep learning technology in fields such as autonomous driving and intelligent robotics, the training and validation of neural network models has become increasingly dependent on high-quality datasets. However, traditional training methods are severely limited by the efficiency and cost of collecting real-world data. In the field of autonomous driving, real-world road testing faces challenges such as the difficulty of simulating extreme weather conditions, the high cost of reproducing high-risk scenarios, and ethical constraints. This has led to a severe shortage of datasets for key scenarios (such as pedestrian avoidance in heavy rain and low-light traffic light recognition at night), directly impacting the model's generalization and safety.

[0003] While existing simulation technologies can generate some training data through virtual environments, they have significant limitations. For one thing, most simulation platforms only support single-module testing (such as vehicle dynamics or sensor simulation), making it difficult to construct high-fidelity virtual scenarios with multimodal interaction. This results in a lack of semantic consistency between the synthesized data and the real world. Furthermore, the simulation environment is disconnected from the model training process, lacking a dynamic data generation and feedback mechanism based on test results, making it impossible to optimize data distribution to address model deficiencies. Furthermore, the lack of compatibility between existing cross-platform deployment technologies (such as model format conversion) results in inefficient model reasoning in virtual testing environments, making it difficult to verify decision logic in complex scenarios in real time.

[0004] In summary, existing technologies are limited by the high cost of real data collection, insufficient fidelity of simulation scenarios and multi-module fragmentation, and the lack of a dynamic closed-loop data generation mechanism based on model test results. As a result, the generalization ability and training efficiency of neural network models in complex scenarios are difficult to meet actual application needs. Summary of the Invention

[0005] The embodiments of the present invention provide a neural network model training method and related equipment based on virtual reality and simulation, which at least solve the problem in related technologies that the generalization ability and training efficiency of neural network models in complex scenarios are difficult to meet actual application needs due to the high cost of real data acquisition, insufficient fidelity of simulation scenarios and multi-module fragmentation, and the lack of a dynamic closed-loop data generation mechanism based on model test results.

[0006] According to a first aspect of an embodiment of the present invention, a neural network model training method based on virtual reality and simulation is provided, comprising:

[0007] Based on the spatial data, physical properties and environmental information of the target real scene, a parametric 3D modeling method is used to generate a 3D visualization virtual reality scene containing buildings, roads and dynamic traffic elements;

[0008] In the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components;

[0009] Deploying a neural network model trained based on a real data set to the digital twin of the test vehicle, performing model inference based on real-time data generated by the dynamic physical simulation environment captured by a virtual camera, and outputting inference results;

[0010] Based on the precision and recall of the inference results and the virtual test collision frequency, identifying performance defects of the neural network model, and determining the target test scenario type to be optimized according to the performance defects;

[0011] Dynamically adjusting scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generating a high-fidelity synthetic data set corresponding to the target test scene type;

[0012] The neural network model is iteratively trained by mixing the high-fidelity synthetic dataset with the real dataset, and the optimized neural network model is redeployed to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0013] According to a second aspect of an embodiment of the present invention, there is provided a neural network model training device based on virtual reality and simulation, comprising:

[0014] A generation module is used to generate a 3D visualization virtual reality scene containing buildings, roads, and dynamic traffic elements through a parametric 3D modeling method based on the spatial data, physical properties, and environmental information of the target real scene;

[0015] A construction module is used to construct a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a test vehicle digital twin based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components;

[0016] An inference module is used to deploy a neural network model trained based on a real data set to the digital twin of the test vehicle, perform model inference based on real-time data generated by the dynamic physical simulation environment captured by a virtual camera, and output inference results;

[0017] an identification module for identifying performance defects of the neural network model based on the precision, recall and virtual test collision frequency of the inference results, and determining a target test scenario type to be optimized based on the performance defects;

[0018] The generation module is further configured to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic data set corresponding to the target test scene type;

[0019] An iterative training module is used to iteratively train the neural network model by mixing the high-fidelity synthetic dataset and the real dataset, and redeploy the optimized neural network model to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0020] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, and a memory storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to execute the method according to the first aspect.

[0021] According to a fourth aspect of an embodiment of the present invention, a non-transitory machine-readable medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect.

[0022] Beneficial effects of the embodiments of the present invention:

[0023] The virtual reality and simulation-based neural network model training method provided by the present invention improves the training efficiency and generalization capabilities of neural network models in complex scenarios by integrating virtual scene modeling, dynamic physical simulation, and synthetic data generation technologies. First, based on a parametric modeling approach, high-fidelity virtual scenes are rapidly generated, enabling the cost-effective reproduction of diverse real-world environmental elements (such as roads, buildings, and dynamic traffic flows). This addresses the issue of insufficient model coverage in traditional training methods due to their reliance on limited real-world data. Second, by simulating real-world vehicle dynamics, multimodal sensor data (such as camera images), and variable environmental conditions (such as extreme weather and day-night lighting variations) in a dynamic physical simulation environment, the model is exposed to complex interactions resembling those in real-world scenarios during virtual testing, enabling accurate identification of performance deficiencies (such as missed object detection and misjudgment of traffic lights). Furthermore, targeted synthetic datasets are dynamically generated based on performance deficiencies and mixed with real-world data for iterative training. This creates a closed-loop mechanism of "virtual testing-defect diagnosis-data supplementation-model optimization," significantly reducing the issue of insufficient model robustness due to the scarcity of data in specific scenarios. Ultimately, this method significantly reduces the cost and safety risks of model training and testing in high-risk areas such as autonomous driving through closed-loop optimization of virtual-reality mapping, while ensuring the reliability of the model's decision-making in complex environments.

[0024] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be derived from these drawings without inventive effort.

[0026] Figure 1 A flowchart of a neural network model training method based on virtual reality and simulation is provided in an embodiment of the present invention.

[0027] Figure 2 A schematic diagram of the architecture of a neural network model training system based on virtual reality and simulation provided by an embodiment of the present invention.

[0028] Figure 3 A schematic diagram of the system architecture for autonomous driving virtual testing provided by an embodiment of the present invention.

[0029] Figure 4 A schematic diagram of a human-computer interaction interface provided by an embodiment of the present invention.

[0030] Figure 5 A schematic diagram of a behavior tree structure provided by an embodiment of the present invention.

[0031] Figure 6 A flowchart of a lane line detection and maintenance method in a virtual reality autonomous driving test system provided by an embodiment of the present invention.

[0032] Figure 7 A structural diagram of a neural network model training device based on virtual reality and simulation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0034] As autonomous driving technology places increasingly stringent demands on complex scene perception and decision-making capabilities, traditional model training methods that rely on real-world road testing can no longer meet the high-precision and high-safety requirements. While existing virtual simulation technologies can partially replace real-world testing, they suffer from low scene modeling efficiency, insufficient fidelity of physical interactions, and a disconnect between model deployment and data closure. This makes it difficult to eliminate semantic deviations between simulated data and real-world scenarios, and model performance verification under extreme conditions still has significant shortcomings. For example, in rainy, snowy weather or low-light scenarios at night, due to limited real-world data collection, models are prone to fatal flaws such as missed target detection and misjudgment of traffic lights. Traditional simulation platforms are unable to dynamically generate adaptive data to optimize model parameters to address these flaws.

[0035] To address these issues, an embodiment of the present invention provides a neural network model training method based on virtual reality and simulation. This method rapidly constructs high-fidelity virtual scenes through parametric modeling, combines cross-platform model deployment technology to achieve real-time reasoning and defect diagnosis in both virtual and real environments, and uses test results to generate and iterate data-driven model training. The method provided by this embodiment of the present invention aims to overcome technical bottlenecks in scene construction efficiency, physical interaction authenticity, and data closed-loop feedback capabilities, providing a systematic solution for efficient training and safety verification of autonomous driving models.

[0036] Figure 1 The flowchart of a neural network model training method based on virtual reality and simulation is provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps.

[0037] Step S101 , based on the spatial data, physical properties and environmental information of the target real scene, a three-dimensional visualization virtual reality scene including buildings, roads and dynamic traffic elements is generated through a parametric three-dimensional modeling method.

[0038] Step S102: In a three-dimensional visualized virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components.

[0039] In step S103, the neural network model trained based on the real data set is deployed to the digital twin of the test vehicle, model reasoning is performed based on the real-time data generated by the dynamic physical simulation environment collected by the virtual camera, and the reasoning results are output.

[0040] Step S104: Based on the precision, recall and virtual test collision frequency of the inference results, the performance defects of the neural network model are identified, and the target test scenario type to be optimized is determined according to the performance defects.

[0041] Step S105 : dynamically adjusting the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generating a high-fidelity synthetic data set corresponding to the target test scene type.

[0042] In step S106, the neural network model is iteratively trained by mixing a high-fidelity synthetic dataset with a real dataset, and the optimized neural network model is redeployed to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0043] First, based on the spatial data, physical properties and environmental information of the target real scene, a three-dimensional visualization virtual reality scene containing buildings, roads and dynamic traffic elements is generated through parametric three-dimensional modeling methods.

[0044] In this embodiment, spatial data may include high-definition plan maps covering the real scene of the target, traffic network topology and road width measurement data; physical properties may include building surface material reflection parameters, road friction coefficient thresholds and steering, torque, braking and suspension parameters in vehicle dynamics parameters; environmental information may include day and night lighting gradient parameters, rain, snow and fog weather particle density and traffic participant behavior rules.

[0045] By utilizing the aforementioned spatial data, physical properties, and environmental information, the road generation logic, building density, and building style parameters can be dynamically adjusted through predefined rule files, allowing for batch generation of three-dimensional digital city models with four-lane bidirectional roads, green vegetation, and multiple types of buildings.

[0046] Within the 3D engine, you can configure material reflectance parameters for building surfaces, set friction thresholds for road surfaces, and configure torque, braking, and suspension parameters for the test vehicle's digital twin. Furthermore, you can simulate daytime and nighttime lighting changes using a skybox component, dynamically generate rain, snow, and fog weather effects using a particle system, and drive path planning and obstacle avoidance decisions for virtual vehicles and pedestrians based on traffic participant behavior rules, creating a 3D visual virtual reality scene that incorporates dynamic traffic elements.

[0047] In this embodiment, a three-dimensional visual virtual reality scene is generated by a parametric three-dimensional modeling method, which can quickly and automatically construct a high-fidelity virtual environment based on real scene data, greatly reducing the cost and time investment of manual modeling, improving the efficiency of scene construction, and reducing the cost of building a virtual test platform.

[0048] Next, a dynamic physics simulation environment is constructed within a 3D virtual reality visualization scene based on real-world physical characteristics. This includes traffic light state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, and dynamically adjusted ambient lighting intensity, color temperature, and rain, snow, and fog weather effects through a particle system and skybox components.

[0049] By defining signal configuration rules and switching sequences, the system simulates the synchronous switching of signals in the opposite lane and controls the traffic flow of signals at the same intersection to cycle through traffic according to a preset period. The test vehicle's digital twin is configured with torque output curves, brake response delays, and suspension stiffness thresholds, while a wheel impactor component simulates the physical contact and friction between the vehicle and the road.

[0050] Based on ambient lighting and weather parameters, the system dynamically adjusts the intensity and color temperature of parallel light to simulate the gradual transition between day and night. It also generates particle density and motion trajectories for rain, snow, and fog, and integrates this with the road surface friction coefficient threshold to simulate the impact of weather on vehicle dynamics. Furthermore, path planning algorithms, obstacle avoidance decision logic, and randomized behavior parameters are set for virtual vehicles and pedestrians. Ray detection and physical collision mechanisms are used to simulate their environmental perception and dynamic interaction, creating a dynamic physical simulation environment with variable traffic flow density.

[0051] The neural network model trained on the real dataset is then deployed to the test vehicle's digital twin. The model performs inference based on real-time data generated by the dynamic physical simulation environment, captured by the virtual camera, and outputs the inference results. In this embodiment, the virtual camera can also collaborate with the radar component to collect real-time data generated by the dynamic physical simulation environment.

[0052] In this embodiment, the neural network model can be converted into an open neural network exchange format file, loaded into the 3D engine, and a localized inference engine can be created. A virtual camera can capture real-time image data of traffic participants in a dynamic physical simulation environment, while a radar component can collect radar data within a sector-shaped detection area, including obstacle distance and orientation data. The image data collected by the virtual camera can be used for model reasoning in the virtual environment, while the obstacle distance and orientation data collected by the radar component can assist the model in reasoning.

[0053] The image and radar data are then fed into a neural network model after resetting their resolution and channel order. The model inference process is then executed asynchronously, outputting raw inference data including the target detection box location, traffic light status classification, and lane line detection results. Based on this raw inference data, the number of correct and missed target detections is counted to calculate precision and recall, while the number of collision events in the 3D visualization virtual reality scene is simultaneously recorded to generate a collision frequency. The precision, recall, and collision frequency are then overlaid on the virtual camera image, and the inference results are rendered in real time through a 3D visualization interface. The corresponding performance indicators and dynamic physical simulation environment parameters are then annotated in the human-computer interaction interface.

[0054] Based on the model inference results, the performance defects of the neural network model can be identified, and the target test scenario type to be optimized can be determined based on the performance defects.

[0055] In this embodiment, the neural network model is determined to have performance deficiencies when the inference result's detection accuracy falls below a preset accuracy threshold, the recall falls below a preset recall threshold, or the virtual test collision frequency exceeds a preset collision frequency threshold. Based on the type and distribution of the performance deficiencies, the model matches corresponding scenario parameter combinations within the dynamic physical simulation environment, selecting at least one of the following: rain, snow, and fog weather, nighttime low-light conditions, or dense traffic flow scenarios as target test scenarios for optimization.

[0056] To further optimize model performance, the scene parameters of the dynamic physical simulation environment can be dynamically adjusted to simulate the target test scene type, generating a high-fidelity synthetic dataset corresponding to the target test scene type.

[0057] In this embodiment, the traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment can be adjusted in real time through the human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environments in the target test scenario type.

[0058] Based on the adjusted dynamic physics simulation environment, the virtual camera and radar components are driven to simultaneously collect real-time image data and radar point cloud data, including target detection box annotations, signal light status classification, and lane markings. The data is also recorded for the test vehicle's digital twin, including its trajectory, collision events, and model inference results. The collected data is then subjected to multimodal data fusion and standardized annotation processing to generate a high-fidelity synthetic dataset that matches the target test scenario type. Its annotation format, data dimensions, and storage structure are compatible with real-world datasets, and it includes multi-dimensional labels for environmental parameters, vehicle behavior, and model decisions.

[0059] Finally, the neural network model is iteratively trained by mixing high-fidelity synthetic datasets with real datasets, and the optimized neural network model is redeployed to the test vehicle digital twin to form a closed-loop optimization mechanism.

[0060] In this embodiment, a high-fidelity synthetic data set and a real data set can be mixed in a preset ratio to generate a hybrid training set. The neural network model is iteratively trained for multiple rounds based on the hybrid training set. After each round of training, the optimized model is redeployed to the digital twin of the test vehicle, and virtual testing is performed through a dynamic physical simulation environment. The accuracy, recall rate and virtual test collision frequency of the neural network model reasoning are recorded. If these indicators do not reach the preset threshold, the scene parameters of the dynamic physical simulation environment are dynamically adjusted according to the performance defects of the current neural network model, and a new high-fidelity synthetic data set is generated and the hybrid training set is updated. Repeat the iterative training, deployment test and scene parameter adjustment steps until the generalization ability of the neural network model in complex scenes such as rainy and snowy weather, night lighting and dense traffic flow reaches the preset threshold, thereby forming a closed-loop optimization mechanism for the linkage of virtual and real data.

[0061] Through the above steps, the embodiments of the present invention can efficiently build an integrated environment that integrates virtual reality scene modeling, dynamic physical simulation, neural network model deployment and reasoning, support the visualization of real-time model reasoning results and the generation of high-fidelity synthetic data sets, form a closed-loop optimization mechanism for model training of virtual-reality mapping, effectively improve the generalization ability of neural network models in complex scenarios, reduce model training and testing costs and improve training efficiency, and provide strong support for research and application in fields such as autonomous driving.

[0062] Figure 2 A schematic diagram of the architecture of a neural network model training system based on virtual reality and simulation provided by an embodiment of the present invention.

[0063] like Figure 2 As shown in the figure, the various parts of the system work closely together to build a complete and efficient testing system.

[0064] In this embodiment, the real-road testing module includes the actual vehicle, sensors, dynamics, and control systems, forming the physical foundation of autonomous driving. The autonomous driving stack (algorithms and decision control) is the core logic, carrying out algorithm execution and decision-making. Together, they provide real-world scenario references and data support for virtual testing, verifying its accuracy.

[0065] In the virtual simulation system, the City Engine tool is first used to generate a virtual test scene based on a real high-definition map, creating a highly realistic environment. Next, the vehicle, sensors, dynamics, and control systems are simulated and modeled to achieve precise mapping, laying the foundation for testing. The Open Neural Network Exchange (ONNX) model is then deployed in the virtual scene to perform autonomous driving tests, simulating the real driving process and testing the model's performance.

[0066] The test workflow module integrates multiple functions under user settings: the artificial intelligence model deployment component implants the model into the virtual environment; the virtual environment interactive control component adjusts scene parameters and simulates complex working conditions; the lane line detection component identifies lane lines in real time; the virtual navigation component plans the path; the accelerated test component improves efficiency; and the synthetic data set acquisition and annotation component collects and processes multi-source data for training.

[0067] The neural network model module includes several key models: an image recognition and classification model for signal light recognition; and an object detection model for detecting traffic participants. The autonomous driving decision-making module, a core component of virtual testing, makes informed decisions based on the outputs of these two models.

[0068] The test data recording module collects model inference results, collision events, and other data, feeding it back into the neural network to form a closed data loop and optimize the training process. Through iterative cycles, this module enables virtual testing and visualization of autonomous driving based on the fusion of virtual reality and multi-source perception, improving model generalization and reliability. This provides an efficient verification tool for R&D, addressing the limitations of traditional test scenarios and insufficient data, and accelerating model iteration and safety verification.

[0069] Figure 3 A schematic diagram of the system architecture for autonomous driving virtual testing provided by an embodiment of the present invention.

[0070] like Figure 3 As shown in the figure, the system architecture combines the two core parts of "virtual reality and simulation" and "autonomous driving virtual test process", achieves efficient connection and integration through ONNX, and accurately simulates the operating logic of autonomous driving in complex environments.

[0071] Virtual reality and simulation form the foundational framework for virtual testing. The virtual environment incorporates elements such as urban roads, traffic lights, vehicles, pedestrians, weather, and lighting, recreating the diversity and complexity of real-world traffic scenarios. Urban roads plan driving routes, traffic lights simulate traffic rules, vehicles and pedestrians follow pre-set rules, and weather and lighting variations create a diverse environment, enabling testing of the autonomous driving system's adaptability under diverse conditions.

[0072] The sensor module serves as the system's sensory antennae. Cameras provide visual images for target recognition; radar detects the distance, direction, and speed of obstacles; the navigation system provides positioning and route planning; and the IMU measures vehicle acceleration and angular velocity, supporting attitude estimation and control. Ultrasonic sensors also provide additional close-range environmental information. Three-dimensional maps, combined with high-definition data, create precise geographic scenes, providing detailed coordinate and terrain information to the vehicle, enabling multi-source data collection.

[0073] The autonomous driving virtual testing process deeply processes and applies data. The fused perception module analyzes sensor data to perform object recognition and detection, including lane identification, traffic light classification, vehicle and pedestrian detection, and collision risk assessment. This module efficiently integrates perception functions, ensuring consistent and efficient data processing and providing accurate information for subsequent decision-making.

[0074] The virtual test vehicle makes intelligent decisions based on sensory information. Predictive reasoning identifies the status of traffic participants and formulates strategies in advance. Behavioral decision-making uses predictions and driving rules to make appropriate driving decisions, such as accelerating, decelerating, steering, and stopping. Route planning combines navigation with real-time information to plan test routes.

[0075] ONNX enables efficient model deployment, ensuring smooth interaction from data collection to decision-making, accurately simulating real-world scenarios, and comprehensively testing and validating algorithms. This improves the performance and safety of autonomous driving systems in complex environments, driving the technology towards a more reliable and intelligent direction.

[0076] In this embodiment of the present invention, the neural network model training method based on virtual reality and simulation further refines the process of constructing a dynamic physical simulation environment to ensure that the generated virtual scene is not only highly realistic in visual terms but also highly consistent with the real scene in terms of physical properties. The following are the specific implementation steps of this method:

[0077] First, based on the traffic light state switching logic of the target real scene, the signal light configuration rules and switching timing can be defined, the reverse synchronous switching of the traffic lights in the opposite lane can be simulated, and the traffic lights at the same intersection can be controlled to change the traffic state according to the preset cycle.

[0078] This example uses a signal light control algorithm based on real-world traffic rules to simulate signal light state transitions. By analyzing real-world scenarios, signal light configurations and switching sequences are defined: Oncoming lane lights cycle through red, yellow, and green with adjustable durations; and signals at the same intersection cycle through time to ensure smooth and safe traffic flow. For example, at an intersection, straight ahead and left turn signals are independent and staggered to avoid conflict, achieve realistic coordination, and provide a realistic environment for virtual testing.

[0079] Then, based on the vehicle dynamics parameters of the target real-world scenario, the torque output curve, brake response delay, and suspension stiffness threshold can be configured for the test vehicle digital twin, and the physical contact and friction effects between the vehicle and the road can be simulated through the wheel collider component.

[0080] In this embodiment, the dynamic parameters of the test vehicle's digital twin are configured for vehicle dynamics simulation, including the torque output curve, brake response delay, and suspension stiffness threshold. The torque curve is set based on engine data to ensure reasonable torque output at different speeds; the brake delay simulates the time difference between the driver's command and actual deceleration; and the suspension stiffness is set based on physical properties to reflect the response under different road conditions. The wheel collider assembly simulates vehicle-road contact and friction, such as cornering lateral forces and bumps. The precise configuration of these parameters ensures that the virtual vehicle's driving behavior is close to reality, providing a reliable physical foundation for neural network training.

[0081] In one embodiment of the present invention, the Unity engine's wheel collider and physics engine are used to simulate core vehicle characteristics such as torque, braking, steering, suspension, and friction. Vision and radar sensors are simulated using cameras and raycasting to achieve environmental perception. Virtual navigation is achieved by combining the A* algorithm with a navigation grid, giving the test vehicle realistic vehicle characteristics. The specific steps are as follows:

[0082] First, import the 3D vehicle model into Unity, add wheel collider components, and set weight, torque, brake, friction, suspension, wheel hub, and steering parameters to simulate dynamic behavior. Simulate driving dynamics under different road conditions by adjusting suspension stiffness and friction coefficient. Set the collider range based on the vehicle's shape to ensure realistic physical collision effects.

[0083] Next, add four camera components to the vehicle, simulate the front camera and rearview mirror, and adjust the resolution, field of view and frame rate to make the image data consistent with the real one. , ray length and the number of rays within range , achieving radar detection range , detection distance and detection density The functional relationship is shown in the following formula (1):

[0084] (1);

[0085] For example, the angle θ is set to 90 degrees, the ray length l is set to 150 meters, and the number of rays n is set to 32, thereby simulating the detection capability of the radar in a fan-shaped area.

[0086] Finally, the virtual road network is rasterized into a navigation mesh, marking the roads as reachable areas. Each test round randomly generates an initial vehicle position. The user specifies a target point, and the vehicle uses the A* algorithm to plan a path, incorporating dynamic parameters to simulate autonomous driving. During testing, trajectory, collision, and sensor data are recorded for model training.

[0087] Through the above implementation steps, the embodiment of the present invention can efficiently simulate the dynamic behavior and perception ability of the test vehicle in the virtual scene, and provide a highly realistic virtual test platform for the training of the neural network model.

[0088] Secondly, based on the ambient lighting and weather parameters of the target real scene, the intensity and color temperature of parallel light can be dynamically adjusted through the skybox component to simulate the gradual change from day to night. The particle system can be used to generate the particle density and motion trajectory of rain, snow and fog weather, and linked with the road surface friction coefficient threshold to simulate the impact of weather on vehicle dynamics.

[0089] In this example, weather simulation is achieved through a skybox and particle system. The skybox dynamically adjusts the intensity and color temperature of parallel light to simulate the change between day and night. The particle system generates rain, snow, and fog, adjusting particle density, trajectory, and characteristics based on weather parameters. The particle effects are linked to the road friction coefficient to simulate the impact of different weather conditions on vehicles, such as skidding during braking in rainy weather and reduced visibility requiring deceleration in foggy weather.

[0090] Finally, based on the behavioral rules of traffic participants in the target real scene, path planning algorithms, obstacle avoidance decision logic and random behavior parameters are set for virtual vehicles and virtual pedestrians. Through ray detection and physical collision mechanisms, the environmental perception and dynamic interaction of virtual vehicles and virtual pedestrians are simulated to form a dynamic physical simulation environment with variable traffic flow density.

[0091] In this embodiment, traffic participant behavior rules are based on real-world traffic observation and analysis, combined with surveillance video, accident cases, regulations, and psychological theory to design path planning and obstacle avoidance logic for virtual vehicles and pedestrians. Vehicles obey traffic lights, maintain safe distances, and yield to pedestrians. Radar-like detection uses raycasting to simulate obstacle perception and makes decisions based on distance and speed. At intersections, traffic signals determine whether to proceed or stop. Pedestrians also follow raycasting-based behavior, such as avoiding obstacles and randomly crossing the street. These rules, combined with raycasting and physical collision mechanisms, make virtual traffic more realistic and diverse.

[0092] Through the above implementation steps, the embodiments of the present invention can construct a highly realistic dynamic physical simulation environment, providing a reliable and flexible virtual platform for training and testing neural network models. This virtual environment not only simulates various complex scenarios in the real world but can also be customized according to testing requirements, providing strong support for improving the generalization ability and reliability of neural network models.

[0093] This embodiment of the present invention further refines the deployment and inference process of the neural network model in a virtual test vehicle, ensuring that the model can run efficiently in the virtual environment and provide accurate inference results. The following are the specific implementation steps of this method:

[0094] First, the neural network model is converted into an open neural network exchange format file, the open neural network exchange format file is loaded through the three-dimensional engine and a localized inference engine is created.

[0095] In this embodiment, the neural network model trained with real data is converted into the ONNX format, and cross-platform operation is achieved by serializing the structure and weights. During the conversion, the input and output layers need to be adapted to ensure consistency with the virtual environment data format, such as converting images to RGB format. After conversion to ONNX, it can be flexibly deployed locally without the need for front-end and back-end communication, making it easy to load models with different structures and different decision-making tasks, improving adaptability and interaction efficiency, and enabling the model to be directly used in real scenarios of edge devices after virtual testing, enhancing versatility and convenience. Subsequently, the ONNX file is loaded through the three-dimensional engine and a local inference engine is created. In Unity, the ONNX file needs to be imported as a resource and the AI ​​inference engine needs to be initialized, including allocating memory, configuring computing resources (such as GPU acceleration), and setting hyperparameters (such as batch size, frequency, etc.).

[0096] Then, a virtual camera is used to capture image data of traffic participants in a dynamic physical simulation environment in real time, and a radar component is used to collect radar data within the fan-shaped detection area. The image data and radar data are reset in resolution and channel order and then input into the neural network model. The radar data includes obstacle distance and azimuth data.

[0097] In terms of data acquisition, the virtual camera can capture images of traffic participants in a dynamic simulation environment in real time. Its parameters (such as resolution, frame rate, and viewing angle) need to be adjusted according to the model requirements. For example, if the model requires 1280*720 pixels, the camera output should be set to the corresponding size. The camera position and angle also need to be set according to the test vehicle to truly reflect the visual perception range. The radar component is responsible for collecting data such as the distance, direction, and speed of obstacles in the fan-shaped area. This is achieved through simulated ray detection. Parameters such as the number of rays, length, and angle must match the actual radar. After acquisition, coordinate transformation, filtering, and fusion are required to ensure data accuracy.

[0098] In practice, to enhance the safety of neural networks in complex environments, machine vision algorithms can be used to acquire real-time surrounding information, particularly traffic conditions ahead. This information is combined with digital twin camera footage and radar data to accurately identify pedestrians, vehicles, obstacles, traffic lights, and lane markings. The vision module focuses on object detection, traffic light recognition, and lane marking detection, with the latter two being performed by a trained neural network.

[0099] In actual applications, more than 4,650 traffic images covering different time periods, weather conditions, and locations were collected through mobile phone photography, online image screening, and public datasets, including 650 traffic light images and 4,000 pedestrian and vehicle data, to improve the model's generalization capabilities. The traffic lights were then classified and pedestrians and vehicles were annotated, generating YOLO format labels to ensure data accuracy and consistency. Pytorch was used to build the AlexNet and YOLOv7 models, respectively training the traffic light recognition and target detection models, and optimizing hyperparameters to improve recognition accuracy and efficiency. To enhance cross-platform deployment capabilities, the Pytorch-trained models were converted to the ONNX format, with optimized preprocessing and structural adjustments to facilitate direct integration into virtual test scenarios, improve performance, and facilitate model replacement.

[0100] Through the above steps, the embodiment of the present invention successfully deployed a computer vision model, provided a reliable decision-making basis for autonomous driving in virtual scenes, and effectively improved the performance of the model in complex traffic environments.

[0101] In a specific embodiment of the present invention, the local deployment and inference of the ONNX model are implemented through the Unity Sentis engine, and image pre-processing and post-processing are completed in combination with OpenCV. The specific steps are as follows:

[0102] First, the ONNX model file is specified as a Unity resource path, loaded, and an inference engine is created to ensure efficient model execution within Unity. Next, each frame from the virtual camera is written to the renderer and converted into Texture data. The resolution (640*640 pixels in this example) and channel order (e.g., BGR to RGB) are adjusted to match the model input requirements. The Texture is then converted into a Tensor and input into the model. Asynchronous inference is used to provide timely feedback, including object detection bounding boxes, traffic light status, and lane marking information. Finally, OpenCV is used to visualize the results, such as color-coding objects and displaying traffic light status. This provides intuitive feedback to the tester and helps guide the vehicle's driving behavior in the virtual scene.

[0103] This method efficiently implements the deployment and inference of ONNX models, ensures real-time performance and accuracy in virtual environments, and provides a reliable tool for autonomous driving model evaluation.

[0104] The image data is resized to its resolution and channel order before being fed into the neural network model. This process involves normalizing the data to eliminate dimensional differences between sensor data. For example, pixel values ​​in image data typically need to be normalized to the range [0, 1]. Furthermore, the channel order of the image data (e.g., RGB or BGR) needs to be adjusted based on the model's input requirements to ensure the model can correctly interpret the input data.

[0105] Afterwards, the model inference process is executed asynchronously, and the raw inference data including the target detection box position, traffic light status classification, and lane line detection results is output.

[0106] In this embodiment, the model inference process is executed asynchronously to ensure timely feedback of inference results to the virtual test vehicle's control system. During each frame of inference, the model output includes information such as the location of the object detection box, the classification result of the traffic light status, and the lane line detection results. This raw inference data undergoes post-processing, including confidence threshold filtering of the classification results, non-maximum suppression (NMS) to eliminate redundant detection boxes, and lane line fitting optimization to improve the accuracy and reliability of the inference results.

[0107] Based on the raw inference data, the system can count the number of correct object detections and the number of missed detections to calculate precision and recall. Precision reflects the proportion of correct objects detected by the model, while recall reflects the proportion of true objects that the model was able to detect. These metrics are calculated as follows:

[0108] Precision refers to the ratio of correctly identified objects to all identified objects. It is calculated by dividing the number of correctly identified objects by the sum of the number of correctly identified objects plus the number of incorrectly identified objects. In other words, precision is equal to the number of correctly identified objects divided by (the number of correctly identified objects plus the number of incorrectly identified objects).

[0109] Recall rate refers to the proportion of objects that are correctly identified among all the objects that actually need to be identified. It is calculated by dividing the number of correctly identified objects by the sum of the number of correctly identified objects plus the number of missed objects. In other words, recall rate is equal to the number of correctly identified objects divided by (the number of correctly identified objects plus the number of missed objects).

[0110] The system can also monitor collisions between the virtual test vehicle and obstacles in the environment, recording the frequency of collisions. Collision detection is achieved through the physics engine's collision detection mechanism. Whenever a virtual vehicle collides with an obstacle, the system records information such as the time, location, and speed of the collision.

[0111] Finally, the precision, recall rate, and collision frequency can be superimposed on the virtual camera image, and the inference results can be rendered in real time through a three-dimensional visualization interface. The performance indicators corresponding to the inference results and the corresponding dynamic physical simulation environment parameters can be marked in the human-computer interaction interface.

[0112] In this embodiment, the system overlays precision, recall, and collision frequency onto the virtual camera feed and displays them in real time through a 3D visualization interface. Object detection frames are colored (e.g., green for vehicles, red for pedestrians), traffic light status is displayed as text or icons, and lane lines are drawn as dashed or solid lines. Furthermore, performance indicators and environmental parameters, such as weather, traffic density, and traffic light status, are annotated on the human-computer interface to help users fully understand the model's performance.

[0113] The above steps ensure efficient deployment and inference of neural networks in a virtual environment, providing a reliable evaluation tool for scenarios such as autonomous driving. This approach improves testing efficiency and accuracy, while also providing researchers with a flexible and scalable testing platform that supports the evaluation and optimization of multiple models.

[0114] In an embodiment of the present invention, a neural network model training method based on virtual reality and simulation further refines how to identify performance deficiencies in the neural network model and, based on these deficiencies, determine the target test scenario type to be optimized. The following are the specific implementation steps of this method:

[0115] First, when the detection accuracy of the inference result is lower than the preset accuracy threshold, the recall rate is lower than the preset recall rate threshold, or the virtual test collision frequency is higher than the preset collision frequency threshold, it is determined that the neural network model has performance defects.

[0116] During virtual testing, the system continuously monitors and records the neural network model's inference results, including object detection precision, recall, and collision frequency. Precision reflects the model's accuracy in identifying objects, recall indicates the proportion of true objects detected, and collision frequency reflects the model's safety in complex environments. These metrics are calculated by the real-time data collection and analysis module and compared against preset thresholds.

[0117] When the model's inference accuracy falls below the threshold, it indicates a high false positive rate for certain targets (such as pedestrians and vehicles). For example, a pedestrian detection accuracy of only 75% in rainy weather, below the 90% threshold, indicates insufficient detection capability. A recall rate below the threshold indicates a significant number of missed detections. For example, a nighttime vehicle recall rate of 60%, below 85%, indicates insufficient coverage. A high collision frequency indicates model decision errors, potentially due to insufficient lane detection and traffic sign recognition accuracy, or close proximity to the target.

[0118] Based on the model performance analysis results, the system can identify model flaws that may be related to specific environments (such as rain, snow, and nighttime) or scenarios (such as high-density traffic and complex intersections). For example, poor pedestrian detection in the rain may be due to image noise, while low recall rates in low-light conditions at night may be due to the model's poor ability to recognize low-contrast objects.

[0119] Then, based on the type and distribution pattern of performance defects, the corresponding scene parameter combinations in the dynamic physical simulation environment are matched, and at least one type of rainy, snowy and foggy weather, low-light conditions at night or dense traffic flow scenes is determined as the target test scene type to be optimized.

[0120] To optimize model performance, the system matches scenario parameters in dynamic physics simulations based on the type and distribution of performance flaws. For example, if the model performs poorly in rainy conditions, the system sets rainy and snowy conditions as the optimization target and adjusts parameters such as raindrop density and road surface reflectivity. If nighttime illumination is insufficient, the system reduces illumination and adjusts streetlight brightness. If traffic is dense, the system increases the number and complexity of virtual vehicles. This approach dynamically generates high-fidelity synthetic datasets relevant to performance flaws, including environmental information (such as weather, lighting, and traffic density) and detailed annotations of object locations, types, and states, ensuring data compatibility and usability. For example, the rainy dataset includes annotations for raindrop density, pedestrian and vehicle locations, and traffic light states in a format consistent with real-world data. This approach effectively identifies model flaws, identifies optimization scenarios, improves training efficiency, and enhances model robustness and reliability in complex scenarios, providing technical support for autonomous driving.

[0121] In this embodiment of the present invention, the neural network model training method based on virtual reality and simulation further refines how to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate specific target test scene types and generate corresponding high-fidelity synthetic datasets. The following are the specific implementation steps of this method:

[0122] First, the traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment can be adjusted in real time through the human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environments in the target test scenario type.

[0123] In this example, to generate a high-fidelity synthetic dataset corresponding to the target test scenario, parameters within the dynamic physical simulation environment, including traffic flow density, signal logic, rain and snow concentration, and light intensity, are adjusted in real time through a human-computer interface. For example, extreme weather scenarios require increasing rain and snow concentration and reducing light intensity; high-density traffic scenarios require increasing the number and complexity of participants and adjusting signal logic to simulate congestion.

[0124] Then, based on the adjusted dynamic physical simulation environment, the virtual camera component and radar component can be driven to synchronously collect real-time image data and radar point cloud data including target detection frame annotations, signal light status classifications, and lane line annotations, and record the driving trajectory, collision events, and model inference results of the test vehicle digital twin.

[0125] In this embodiment, a virtual camera and radar simultaneously collect image and point cloud data, including target detection boxes, traffic light status, and lane markings. The camera captures visual information at a preset frequency and resolution, while the radar uses ray detection to determine obstacle distance, direction, and speed. This data is stored along with the vehicle's digital twin trajectory, collision events, and inference results for subsequent processing.

[0126] Finally, multimodal data fusion and standardized annotation processing are performed on real-time image data, radar point cloud data, driving trajectories, collision events and model inference results to generate a high-fidelity synthetic dataset that matches the target test scenario type. The annotation format, data dimension and storage structure of the high-fidelity synthetic dataset are compatible with the real dataset, and include multi-dimensional labels for environmental parameters, vehicle behavior and model decisions.

[0127] This embodiment performs multimodal data fusion and standardized annotation on real-time images, radar point clouds, driving trajectories, collision events, and model inference results. Multimodal fusion aligns different data on the timeline, providing a consistent scene description. Standardized annotation includes target box position, traffic light status, lane line fitting, and other features to ensure consistent data format.

[0128] The resulting high-fidelity synthetic dataset contains rich environmental information and detailed annotations, such as weather, lighting, traffic density, and object location, type, and status. The format is consistent with real-world data to ensure compatibility. For example, each image in a rainy scene is annotated with raindrop density, pedestrian and vehicle locations, and traffic light status. Multi-dimensional labels for environmental parameters, vehicle behavior, and model decisions are also included, providing comprehensive support for model training.

[0129] To ensure data quality, the system verifies and calibrates the data. Verification checks data integrity and consistency to ensure accurate labeling; calibration normalizes the data to eliminate sensor dimensional differences and facilitate neural network analysis.

[0130] Through the above steps, this embodiment can dynamically adjust physical simulation parameters, generate high-fidelity data sets corresponding to test scenarios, improve training diversity, optimize the performance of the model in specific scenarios, and enhance model generalization and reliability.

[0131] In this embodiment of the present invention, the neural network model training method based on virtual reality and simulation further refines how to perform iterative training by mixing high-fidelity synthetic datasets with real datasets, and forms a closed-loop optimization mechanism to improve the generalization ability of the neural network model in complex scenarios. The following are the specific implementation steps of this method:

[0132] First, the high-fidelity synthetic dataset and the real dataset are mixed in a preset ratio to generate a mixed training set.

[0133] In this embodiment, the generated high-fidelity synthetic dataset can be mixed with the real dataset in a preset ratio to generate a mixed training set. In practical applications, the mixing ratio can be adjusted according to the current performance of the model and training requirements. For example, if the model performs poorly in a specific scenario, the proportion of the synthetic dataset corresponding to the scenario can be increased to enhance the model's learning of the scenario. The generation process of the mixed training set needs to ensure the consistency and diversity of the data, that is, the data is consistent in terms of format, annotation method, etc., and covers data samples in different scenarios and under different conditions.

[0134] Next, the neural network model is iteratively trained for multiple rounds based on the mixed training set. After each round of training, the optimized neural network model is redeployed to the digital twin of the test vehicle, and virtual testing is performed in a dynamic physical simulation environment. The accuracy and recall rate of the neural network model reasoning and the frequency of virtual test collisions are recorded.

[0135] In this embodiment, when a neural network model is trained iteratively over multiple rounds based on a mixed training set, the training set can be randomly shuffled before each round of training to prevent model bias due to data order. During training, appropriate optimization algorithms (such as Adam and SGD) and loss functions (such as cross-entropy loss and mean squared error) can be used, and model parameters can be adjusted based on the difference between the model output and the labeled data. Hyperparameters during training (such as the learning rate and batch size) can be dynamically adjusted based on the model's convergence and performance.

[0136] After each round of training, the optimized neural network model is redeployed to the test vehicle's digital twin. This redeployment process involves converting the updated model weights and structure files into a format compatible with the virtual test environment (such as ONNX) and loading them into the test vehicle through a 3D engine. The redeployed model is then virtual tested in a dynamic physics simulation environment to evaluate its performance improvements.

[0137] During virtual testing, the system records performance metrics such as the neural network model's inference precision, recall, and virtual test collision frequency. These metrics are compared against preset performance thresholds to determine whether the model meets the expected performance standards. If precision, recall, and collision frequency all reach the preset thresholds, the model's performance in the current scenario is considered satisfactory, and training can be terminated. Otherwise, further analysis of the model's performance deficiencies is required, and the scenario parameters of the dynamic physical simulation environment can be dynamically adjusted accordingly.

[0138] Dynamically adjusting the scene parameters of the dynamic physics simulation environment aims to generate more challenging test scenarios to expose potential model flaws. For example, if the model experiences a high collision frequency in rainy scenarios, the raindrop density can be increased, the light intensity can be reduced, or the traffic volume can be increased to simulate more extreme weather and traffic conditions. The adjusted scene parameters need to match the type and distribution of performance flaws to ensure that the generated test scenarios can specifically optimize model performance.

[0139] In addition, if the precision, recall rate, and virtual test collision frequency do not reach the preset thresholds, the scene parameters of the dynamic physical simulation environment are dynamically adjusted according to the performance defects of the current neural network model, and a new high-fidelity synthetic dataset is generated and the hybrid training set is updated.

[0140] In this embodiment, the data collection and synthetic dataset generation process is re-executed based on the adjusted dynamic physics simulation environment. The newly generated synthetic dataset contains more complex and diverse scene information and is used to update the hybrid training set. In this way, the diversity and difficulty of the training data are continuously enriched, driving the model's performance improvement in a wider range of scenarios.

[0141] Finally, the iterative training, deployment testing, and scenario parameter adjustment steps are repeated until the generalization ability of the neural network model in complex scenarios such as rainy and snowy weather, night lighting, and dense traffic flow reaches the preset threshold, forming a closed-loop optimization mechanism that links virtual and real data.

[0142] In this example, the iterative training, deployment testing, and scenario parameter adjustment steps are repeated until the neural network model's generalization capability reaches a preset threshold in complex scenarios such as rain and snow, nighttime illumination, and dense traffic. This process forms a closed-loop optimization mechanism that integrates virtual and real data. By continuously supplementing the real dataset with synthetic datasets and guiding model optimization with virtual test results, the model's performance is ultimately continuously improved.

[0143] Through the above implementation steps, the present invention effectively utilizes high-fidelity synthetic datasets to optimize the training process of neural network models, forming a closed-loop optimization mechanism that significantly improves the model's generalization and reliability in complex scenarios. This virtual reality and simulation-based training method provides an efficient, flexible, and cost-effective solution for the development of neural network models in fields such as autonomous driving.

[0144] In one embodiment of the present invention, the system meets the needs of virtual autonomous driving testing, enabling users to efficiently collect synthetic data in virtual scenarios, supplementing real-world data and improving model training. By simulating complex scenarios, users can dynamically adjust environmental parameters (such as traffic, lighting, and weather) to generate synthetic data that closely resembles real-world scenarios. This data contains rich environmental information and detailed annotations, such as object detection boxes, traffic light status, and lane line locations, effectively increasing data diversity and coverage.

[0145] To verify the effectiveness of synthetic data, model experiments were conducted. Taking traffic light recognition as an example, after training with a mixed dataset (250 real images + 500 synthetic images), the model achieved an accuracy of 92.9%, with an average accuracy of 92.3% across 100 virtual test rounds. In contrast, a model trained entirely with real data (650 images) achieved an accuracy of 93.2%, with an average accuracy of 92.6% across 100 virtual test rounds. This demonstrates that synthetic datasets can effectively supplement real datasets and improve model generalization.

[0146] Further validation of the object detection model revealed that training with a mixed dataset (1,000 real images + 1,000 synthetic images) achieved precision of 85%-86%, recall of 80%-85%, and a virtual test accuracy of 92.5%. Using only real data (4,000 images), the model achieved precision of 83%-85%, recall of 80%-82%, and a virtual test accuracy of 90.3%. With only 2,000 images of mixed data, the model achieved the equivalent of double the real data, demonstrating that synthetic data can improve training efficiency and performance when real samples are limited.

[0147] In summary, this example demonstrates the effectiveness of synthetic data in neural network training. Models trained with hybrid data demonstrate high generalization capabilities in real-world scenarios, and the virtual test results accurately and convincingly reflect the effectiveness of training. This method reduces data acquisition and model training costs, supporting efficient training of autonomous driving models.

[0148] In a specific embodiment of the present invention, in order to improve the flexibility and adaptability of the test, the system provides two virtual test modes: fully automatic test and manually controlled test. In the fully automatic test mode, the user does not need to perform any intervention. Once the test is started, the test vehicle will perform the autonomous driving task according to the driving route pre-calculated by the system. During this process, the vehicle will automatically adjust the driving speed, direction and obstacle avoidance strategy based on the real-time perceived environmental information and preset decision logic. The test will continue until the vehicle successfully reaches the designated target location, or collides with an unavoidable obstacle, forcing it to stop and end the current test round. This mode is particularly suitable for batch testing and automated evaluation of model performance, and can efficiently collect a large amount of test data for subsequent model optimization and performance analysis.

[0149] In contrast, the manual control test mode gives users greater control. In this mode, the user directly drives the test vehicle, while the system focuses on providing an intuitive user interface and real-time feedback on the vehicle's decision-making information, such as object detection results, traffic light status, lane position, and potential collision risks. However, the system does not actually participate in the vehicle's control process; all driving operations are completed by the user. This mode is primarily used for detailed testing of specific scenarios or to train users' understanding and operational capabilities of autonomous driving systems. It also provides researchers with an opportunity to observe and analyze the model's performance under human intervention.

[0150] To enable seamless switching between these two test modes, the system has been designed with a flexible control architecture. In fully automated test mode, the system's autonomous driving algorithm assumes full control of the vehicle, while the user interface primarily displays the vehicle's driving status and system decision-making information. In manual control test mode, the user interface offers a rich set of control options, including simulated steering, accelerator, and brake controls, as well as real-time environmental perception data and decision-making recommendations. This design ensures users can flexibly switch between different testing requirements, maximizing test efficiency and effectiveness.

[0151] The system also features a detailed logging module that records all key events and data during the test. Whether conducting fully automated or manually controlled tests, the system records the vehicle's trajectory, sensor data, decision-making information, and any collision events. This data is not only used to evaluate the performance of the current test but also serves as an important input for subsequent iterative model training, helping to further optimize the performance and robustness of the neural network model.

[0152] By providing these two complementary test modes, the present invention can meet the diverse needs of different testing scenarios, effectively supporting both automated large-scale testing and refined manual evaluation. This flexibility not only improves testing efficiency but also enhances the reliability and comprehensiveness of test results, providing solid support for the development and verification of autonomous driving technology.

[0153] In one specific embodiment of the present invention, a flexible and feature-rich human-computer interaction interface is provided to support virtual testing and optimization of autonomous driving models. This interface allows users to freely select single-model testing (such as signal light recognition, object detection, lane detection, etc.) or multi-model fusion testing to meet different testing requirements. Through the interface, users can intuitively adjust environmental parameters such as traffic flow, light intensity, and weather conditions (such as rain, snow, and fog) to simulate testing conditions in various complex scenarios. Furthermore, the interface provides multiple perspective switching functions, allowing users to select front, rear, left, right, or bird's-eye view cameras as needed to comprehensively observe the test vehicle's behavior and environmental interactions.

[0154] Figure 4 A schematic diagram of a human-computer interaction interface provided by an embodiment of the present invention.

[0155] like Figure 4 As shown, the test interface displays vehicle status, perception data, etc. in real time, and displays indicators such as recognition accuracy, number of collisions, and trajectory deviation through charts. The signal light recognition rate is displayed as a percentage, pedestrian and vehicle detections are marked with different colors, and lane lines are presented as dotted or solid lines. The system also records detailed logs, including timestamps, environmental parameters, model inference results, and collision information. To enhance the experience, the interface provides data visualization and analysis functions, supports performance comparison, scenario performance analysis, and result export, and assists in model optimization. Through intuitive display and detailed records, users can quickly locate problems, adjust parameters, and form a closed-loop optimization mechanism. This flexible and powerful interface improves test efficiency and accuracy, provides researchers with an easy-to-use platform, supports the evaluation of multiple neural network models, and assists in the development of autonomous driving technology.

[0156] In a specific embodiment of the present invention, a high-fidelity virtual test environment is constructed, combined with dynamic scene parameter adjustment and mixed training of virtual and real data to form a closed-loop optimization mechanism, thereby improving the generalization ability and decision reliability of the neural network model in complex traffic scenarios. The following is a detailed description of the specific embodiment:

[0157] During implementation, the virtual test system preparations were first completed. This involved building the driving decision logic for the test vehicle's digital twin and initializing environmental parameters. By configuring parameters such as time, traffic flow, and weather, the desired test environment was simulated. For example, setting the time to "7:00 PM - 9:00 PM," traffic flow to "high," and rainy or snowy weather conditions allowed for the creation of a complex scenario with low visibility and dense traffic.

[0158] The autonomous driving decision module of the test vehicle digital twin is integrated with the neural network output and simulated radar data to control the vehicle. The decision logic is based on the behavior tree structure (see Figure 5 ) is implemented by dynamically controlling the vehicle's deceleration, following and other driving behaviors based on information such as traffic lights, obstacles, and vehicle distances, simulating real driving decisions.

[0159] Lane detection ensures the stability of autonomous driving. This implementation uses Canny edge detection combined with Hough transform to identify lane lines, with the results displayed in a visual interface. After successful detection, the coordinates of the lane line endpoints are fitted to the centerline. Steering is adjusted based on the angle between the vehicle's direction and the centerline. Correction is initiated when the angle exceeds a threshold, achieving lane keeping. As the vehicle approaches the centerline, the steering angle is optimized by calculating the vanishing point offset to improve trajectory stability.

[0160] Specifically, in a two-dimensional coordinate system, the intersection of the extended lines of the left and right lanes is defined as the vanishing point , the midpoint of the lane line end point is By calculation The steering angle is fine-tuned by the offset, and the adjustment range is negatively correlated with the vehicle speed to ensure safety. The formula for calculating the vanishing point coordinates is as follows:

[0161] (2);

[0162] in, 、 are the slopes of the left and right lane lines, 、 are the coordinates of any point on the left and right lane lines respectively.

[0163] If the lane line cannot be detected, the lane keeping system will suspend operation, and the test vehicle digital twin will execute steering according to the virtual navigation path, such as through an intersection or a right-angle turn.

[0164] To illustrate the lane recognition and measurement process in a virtual reality autonomous driving test system, including image acquisition, preprocessing, edge detection, lane offset calculation, and visualization, and to provide a basis for subsequent control, the following detailed description is provided with accompanying figures and examples.

[0165] Figure 6This is a flow chart of a method for lane line detection and lane offset calculation in a virtual reality autonomous driving test system provided by an embodiment of the present invention. Figure 6 As shown, the method includes the following steps.

[0166] 1. Checking if the camera is successfully opened: When the system boots up, it first checks the status of the virtual camera. If camera initialization fails (e.g., permissions are not granted or enabled), the program continues polling until the status is normal. If the test is successful, the next stage of the data collection process is triggered. In this embodiment, if the check result is negative, the loop continues to check. If it is positive, the next step is performed.

[0167] 2. Capture camera data frame by frame in the Update() function: The Update() function is called during each frame rendering cycle to capture RGB image data of the scene in front of the current vehicle using the virtual camera component. In this embodiment, the image resolution is adapted to 650×360 pixels.

[0168] 3. Process the frame in the ProcessFrame() function: This function converts the Texture2D texture image into a Mat image and starts the lane detection pipeline. The detection process includes the following substeps:

[0169] 3.1 Convert frames to grayscale images: Use OpenCV's cvtColor() function to convert color images to grayscale images, eliminating color interference and reducing subsequent computational complexity.

[0170] 3.2 Apply Gaussian blur: Use the GaussianBlur() function to perform a Gaussian filter with a 3×3 convolution kernel on the grayscale image to smooth out noise interference and improve edge detection stability.

[0171] 3.3 Apply Canny edge detection: Set dual thresholds (e.g., low threshold 50, high threshold 150), use the Canny() function to extract image edge contours, and prioritize lane line features.

[0172] 3.4 Reapply Canny edge detection: Adjust the threshold value a second time (e.g., low threshold 30, high threshold 100) to enhance the edge and compensate for the weak edges missed by the first detection.

[0173] 4. Define and apply a region of interest (ROI): Set a trapezoidal mask area to focus on the ground area in front of the vehicle (for example, the ground area within 3 to 10 meters in front of the vehicle), excluding irrelevant background interference such as the sky and buildings.

[0174] 5. Use Hough transform to detect lines: Call the HoughLinesP() function to perform probabilistic Hough transform. In this embodiment, the parameter configuration is: Resolution 1 pixel (indicates detection of straight lines), Resolution / 180 radians, voting threshold 20, minimum segment length 20 pixels, segment interval 30 pixels. Extract lane line candidate sets.

[0175] 6. Filter and fit lane lines: Select line segment points with a slope greater than 0.4 and divide the left and right lane lines based on the positive and negative slopes.

[0176] 7. Draw the detected lines on the frame: Map the line endpoint coordinates output by the Hough transform to the original image. In this example, a green line (BGR value [0, 255, 0]) is overlaid to visualize the lane detection results.

[0177] 8. Convert Mat to Texture2D for display in Unity: Use Unity's OpenCV plug-in to convert the processed Mat image data into Texture2D format to adapt to the requirements of Unity's rendering pipeline.

[0178] 9. Lane offset calculation and lane keeping. In this embodiment, the following operations are performed:

[0179] Extract the left and right lane line endpoint coordinates 、 , calculate the midpoint ; Calculate the vanishing point according to formula (2) ,in 、 To fit the slope of the straight line; calculate the lateral offset , combined with the current vehicle speed (unit: ) Dynamically adjust the steering angle ,in is the sensitivity coefficient (default );like Exceeding the threshold (In this embodiment =0.3m), triggering the direction correction mechanism and outputting the steering control command to the vehicle dynamics model.

[0180] 10. Record the offset and direction correction log information: Value, steering angle The corresponding timestamp is written into the log file for subsequent virtual test analysis.

[0181] 11. Display the Texture2D texture image frame by frame in the Unity renderer: Render the generated Texture2D to the Raw Image component of the Unity scene, and display the driving perspective image including lane markings in real time in the 3D visualization interface.

[0182] 12. End: Completes a single lane detection cycle and waits for the next frame to trigger a new round of processing.

[0183] This embodiment achieves highly robust lane line recognition and dynamic trajectory correction through edge detection and geometric modeling of virtual-real fusion, verifying the effectiveness of virtual reality scenes in the closed-loop test of autonomous driving "perception-decision-making". Based on the above-mentioned neural network model training method based on virtual reality and simulation provided by the embodiment of the present invention, Figure 7 As shown, an embodiment of the present invention also provides a neural network model training device based on virtual reality and simulation, including: a generation module 701, a construction module 702, an inference module 703, an identification module 704, a generation module 701 and an iterative training module 705.

[0184] A generation module 701 is used to generate a three-dimensional visual virtual reality scene including buildings, roads and dynamic traffic elements through a parametric three-dimensional modeling method based on the spatial data, physical properties and environmental information of the target real scene;

[0185] Construction module 702 is used to construct a dynamic physical simulation environment based on real physical characteristics in a three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a test vehicle digital twin based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components;

[0186] The inference module 703 is used to deploy the neural network model trained based on the real data set to the digital twin of the test vehicle, perform model inference based on the real-time data generated by the dynamic physical simulation environment collected by the virtual camera, and output the inference results;

[0187] Identification module 704, for identifying performance defects of the neural network model based on the precision and recall of the inference results and the virtual test collision frequency, and determining the target test scenario type to be optimized based on the performance defects;

[0188] The generation module 701 is further configured to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type and generate a high-fidelity synthetic data set corresponding to the target test scene type;

[0189] The iterative training module 705 is used to iteratively train the neural network model by mixing high-fidelity synthetic datasets and real datasets, and redeploy the optimized neural network model to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

[0190] An embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the method of an embodiment of the present invention.

[0191] An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method of the embodiment of the present invention.

[0192] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to perform the method of the embodiment of the present invention.

[0193] It should be noted that in the embodiments of the present invention, "including" is an open inclusion, i.e., "including but not limited to"; "based on" means "based at least in part on"; "one embodiment" means "at least one embodiment", "another embodiment" means "at least one additional embodiment", and "some embodiments" means "at least some embodiments". "One" and "a plurality of" are illustrative modifications and are not limiting. Unless expressly stated otherwise, they should be understood to mean "one or more".

[0194] The steps in the method implementation may be executed in different orders or in parallel, and steps may be added or removed, without limiting the scope of protection.

[0195] "Embodiment" means that the specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment. The appearance of this term in different places herein does not necessarily refer to the same embodiment, nor does it exclude the coexistence of other embodiments. Reference will be made to the same or similar portions of the various embodiments. In particular, since the device, equipment, and system embodiments are generally similar to the method embodiments, their descriptions will be simplified. For relevant portions, refer to the method embodiments.

[0196] The foregoing descriptions are merely a few embodiments of the present invention, and are described in detail. They should not be construed as limiting the scope of patent protection. Persons skilled in the art may make various modifications and improvements without departing from the scope of the present invention. Therefore, the scope of protection of the present invention shall be subject to the appended claims.

Claims

1. A neural network model training method based on virtual reality and simulation, characterized in that: include: Based on the spatial data, physical properties and environmental information of the target real scene, a parametric 3D modeling method is used to generate a 3D visualization virtual reality scene containing buildings, roads and dynamic traffic elements; In the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a digital twin of the test vehicle based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components; Deploying a neural network model trained based on a real dataset to the digital twin of the test vehicle, performing model inference based on real-time data generated by the dynamic physical simulation environment captured by a virtual camera, and outputting inference results, including target detection box positions, traffic light status classifications, and lane line detection results; Based on the precision, recall and virtual test collision frequency of the inference results, the performance defects of the neural network model are identified, and the target test scenario type to be optimized is determined according to the performance defects, wherein when the detection precision of the inference results is lower than a preset precision threshold, the recall is lower than a preset recall threshold, or the virtual test collision frequency is higher than a preset collision frequency threshold, it is determined that the neural network model has performance defects; based on the type and distribution pattern of the performance defects, the corresponding scene parameter combination in the dynamic physical simulation environment is matched, and at least one type of rain, snow and fog weather, low light conditions at night or dense traffic flow scenes is determined as the target test scenario type to be optimized; Dynamically adjusting scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generating a high-fidelity synthetic data set corresponding to the target test scene type; The neural network model is iteratively trained by mixing the high-fidelity synthetic dataset with the real dataset, and the optimized neural network model is redeployed to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

2. The method according to claim 1, characterized in that The spatial data includes a high-definition plan map of the target real scene, traffic network topology, and road width measurement data; the physical attributes include building surface material reflection parameters, road friction coefficient thresholds, and torque, braking, and suspension parameters in vehicle dynamics parameters; the environmental information includes day and night illumination gradient parameters, rain, snow, and fog weather particle density, and traffic participant behavior rules; The method of generating a three-dimensional visualization virtual reality scene including buildings, roads and dynamic traffic elements by using a parametric three-dimensional modeling method based on the spatial data, physical properties and environmental information of the target real scene includes: Based on the spatial data, the road generation logic, building density and building style parameters are dynamically adjusted through a predefined rule file to batch generate a three-dimensional digital city model containing two-way four-lane roads, green vegetation and multiple types of building complexes; Based on the physical properties, configuring material reflection parameters for the building surface, setting a friction coefficient threshold for the road surface, and configuring the torque, braking, and suspension parameters for the test vehicle digital twin in a three-dimensional engine; Based on the environmental information, the skybox component is used to simulate the changes in day and night lighting, the particle system is used to dynamically generate rain, snow and fog weather effects, and the path planning and obstacle avoidance decisions of virtual vehicles and virtual pedestrians are driven according to the traffic participant behavior rules to form the three-dimensional visual virtual reality scene containing dynamic traffic elements.

3. The method according to claim 1, characterized in that In the three-dimensional visual virtual reality scene, a dynamic physical simulation environment is constructed based on real physical characteristics, including: Based on the traffic light state switching logic of the target real scene, define the traffic light configuration rules and switching timing, simulate the reverse synchronous switching of the opposite lane traffic lights, and control the traffic lights at the same intersection to change the traffic state according to the preset cycle; Based on the vehicle dynamics parameters of the target real-world scenario, configure a torque output curve, brake response delay, and suspension stiffness threshold for the test vehicle digital twin, and simulate physical contact and friction effects between the vehicle and the road through a wheel impactor assembly; Based on the ambient lighting and weather parameters of the target real scene, the intensity and color temperature of the parallel light are dynamically adjusted through the skybox component to simulate the gradual change of day and night. The particle system is used to generate the particle density and motion trajectory of rain, snow and fog weather, and is linked with the road surface friction coefficient threshold to simulate the impact of weather on vehicle dynamics. Based on the traffic participant behavior rules in the target real scene, path planning algorithms, obstacle avoidance decision logic and random behavior parameters are set for virtual vehicles and virtual pedestrians. The environmental perception and dynamic interaction of the virtual vehicles and virtual pedestrians are simulated through ray detection and physical collision mechanisms to form the dynamic physical simulation environment with variable traffic flow density.

4. The method according to claim 1, wherein The neural network model trained based on the real data set is deployed to the digital twin of the test vehicle, model reasoning is performed based on real-time data generated by the dynamic physical simulation environment collected by the virtual camera, and reasoning results are output, including: Converting the neural network model into an open neural network exchange format file, loading the open neural network exchange format file through a three-dimensional engine and creating a localized inference engine; Capturing image data of traffic participants in the dynamic physical simulation environment in real time through the virtual camera, resetting the resolution and channel order of the image data and inputting them into the neural network model; Asynchronously executes the model inference process and outputs raw inference data including the location of the target detection box, traffic light status classification, and lane line detection results; Based on the original inference data, the number of correct identifications and the number of missed detections of target detection are counted to calculate the precision and recall rate, and the number of collision events in the three-dimensional visualization virtual reality scene is simultaneously recorded to generate a collision frequency; The precision rate, the recall rate, and the collision frequency are superimposed on the virtual camera image, the inference result is rendered in real time through a three-dimensional visualization interface, and the performance indicators corresponding to the inference result and the corresponding dynamic physical simulation environment parameters are marked in the human-computer interaction interface.

5. The method according to claim 1, wherein The dynamically adjusting the scene parameters of the dynamic physical simulation environment to simulate the target test scene type and generating a high-fidelity synthetic data set corresponding to the target test scene type includes: Traffic flow density, signal light state switching logic, rain and snow particle concentration, and light intensity parameters in the dynamic physical simulation environment are adjusted in real time through a human-computer interaction interface to simulate extreme weather conditions, dense traffic flow, or low-light environments in target test scenarios; Based on the adjusted dynamic physical simulation environment, the virtual camera component is driven to synchronously collect real-time image data including target detection frame annotations, signal light status classifications, and lane line annotations, and record the driving trajectory, collision events, and model inference results of the test vehicle digital twin; Multimodal data fusion and standardized annotation processing are performed on the real-time image data, driving trajectories, collision events, and model inference results to generate a high-fidelity synthetic dataset that matches the target test scenario type. The annotation format, data dimension, and storage structure of the high-fidelity synthetic dataset are compatible with the real dataset, and include multi-dimensional labels for environmental parameters, vehicle behavior, and model decisions.

6. The method according to claim 5, characterized in that The mixing of the high-fidelity synthetic dataset and the real dataset to iteratively train the neural network model, and redeploying the optimized neural network model to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization capability of the neural network model in complex scenarios reaches a preset threshold, including: Mixing the high-fidelity synthetic dataset and the real dataset in a preset ratio to generate a mixed training set; performing multiple rounds of iterative training on the neural network model based on the hybrid training set, wherein after each round of training, the optimized neural network model is redeployed to the digital twin of the test vehicle, virtual testing is performed in the dynamic physical simulation environment, and the precision and recall of the neural network model reasoning and the virtual test collision frequency are recorded; If the precision, the recall, and the virtual test collision frequency do not reach a preset threshold, dynamically adjusting the scene parameters of the dynamic physical simulation environment based on the performance defects of the current neural network model, generating a new high-fidelity synthetic data set, and updating the hybrid training set; Repeat the iterative training, deployment testing and scenario parameter adjustment steps until the generalization ability of the neural network model in complex scenarios such as rainy and snowy weather, night lighting and dense traffic flow reaches the preset threshold, forming a closed-loop optimization mechanism that links virtual and real data.

7. A neural network model training device based on virtual reality and simulation, characterized in that: include: A generation module is used to generate a 3D visualization virtual reality scene containing buildings, roads, and dynamic traffic elements through a parametric 3D modeling method based on the spatial data, physical properties, and environmental information of the target real scene; A construction module is used to construct a dynamic physical simulation environment based on real physical characteristics in the three-dimensional visual virtual reality scene. The dynamic physical simulation environment includes: traffic light state switching logic and traffic participant behavior rules, a test vehicle digital twin based on vehicle dynamics parameters, and ambient light intensity, color temperature, and rain, snow, and fog weather effects dynamically adjusted through a particle system and skybox components; An inference module is configured to deploy a neural network model trained based on a real dataset to the digital twin of the test vehicle, perform model inference based on real-time data generated by the dynamic physical simulation environment captured by a virtual camera, and output inference results, including target detection box positions, signal light status classifications, and lane line detection results; an identification module for identifying performance defects of the neural network model based on the precision, recall, and virtual test collision frequency of the inference results, and determining a target test scenario type to be optimized based on the performance defects, wherein the neural network model is determined to have performance defects when the detection precision of the inference results is lower than a preset precision threshold, the recall is lower than a preset recall threshold, or the virtual test collision frequency is higher than a preset collision frequency threshold; and matching corresponding scene parameter combinations in the dynamic physical simulation environment based on the type and distribution pattern of the performance defects, determining at least one type of rain, snow, and fog weather, nighttime low-light conditions, or dense traffic flow scenes as the target test scenario type to be optimized; The generation module is further configured to dynamically adjust the scene parameters of the dynamic physical simulation environment to simulate the target test scene type, and generate a high-fidelity synthetic data set corresponding to the target test scene type; An iterative training module is used to iteratively train the neural network model by mixing the high-fidelity synthetic dataset and the real dataset, and redeploy the optimized neural network model to the test vehicle digital twin to form a closed-loop optimization mechanism until the generalization ability of the neural network model in complex scenarios reaches a preset threshold.

8. An electronic device comprising: A processor, and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory machine-readable medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

Images