Automatic driving simulation system and method based on data closed loop and hardware-in-loop

Abstract

The invention discloses an automatic driving simulation system and method based on data closed loop and hardware-in-loop. The system comprises an automatic driving real vehicle module, a cloud platform module, a data closed loop verification module, a remote upgrading module, a simulation bridging module, a simulation scene generator module and a real vehicle verification data recording module. The automatic driving real vehicle automatically drives according to a preset test scene; the simulation scene generator module inputs simulation parameters to generate simulation scene data; judging whether the hardware-in-the-loop simulation test is normally carried out or not, if so, recording key operation data, and analyzing the data through a data closed-loop verification module; judging whether the scene operation parameters conform to design expectation or not, if yes, collecting standard scene parameters, and recording test data; and if not, optimizing the generation strategy of the simulation scene. The method has the advantages of closed-loop processing of simulation data, quick verification of a simulation algorithm, improvement of simulation scene accuracy, acceleration of iteration speed and the like, and a new way is provided for high-authenticity simulation of an automatic driving system.

Description

Technical Field

[0001] The present invention relates to the technical field of automotive autonomous driving, and more specifically, it refers to an autonomous driving simulation system and method based on data closed-loop and hardware-in-the-loop. Background Art

[0002] Simulation technology is one of the key fundamental technical links in the simulation of autonomous driving systems. The authenticity of the simulation environment, as well as the real-time and accuracy of the construction of simulation scenarios, are crucial for simulation. Whether the simulation environment can be closest to the real application environment to the greatest extent has become one of the bottleneck factors restricting the development of simulation.

[0003] Data closed-loop refers to the data generated and the use and processing of data in a complete data cycle, including links such as data collection, processing, analysis, application, and feedback. The data closed-loop has now been widely used as a key technology in the research and development and testing fields of autonomous driving. Through means such as cloud extraction, processing and mining, automatic annotation, model training, and simulation test upgrade of data, the research and development and verification process of autonomous driving systems can be rapidly improved. Hardware-in-the-loop is a simulation test technology that simulates the actual operating environment of autonomous driving vehicles by sending virtual scenario data to real vehicles to improve the efficiency of simulation testing. It has the following characteristics: high efficiency - reducing the number of real vehicle road tests, shortening the development time, and reducing costs; safety - conducting tests in a safe test environment to avoid the risks faced by actual road tests; high precision - realistically simulating the real test scenario environment to ensure the accuracy of test results.

[0004] In the existing simulation technologies for autonomous driving systems, there are generally types such as pure software simulation or controller-level hardware-in-the-loop simulation. Among them, pure software simulation cannot accurately reflect the real application environment, and although the controller-level hardware-in-the-loop simulation system improves the simulation accuracy to a certain extent, there is still a large gap from the real vehicle-level simulation system. For example, Patent CN202210833733.8 discloses an intelligent driving software / hardware-in-the-loop automated simulation test system, which focuses on separately testing software-in-the-loop and hardware-in-the-loop. Summary of the Invention

[0005] In view of the above-mentioned defects (shortcomings) of the prior art, the purpose of the present invention is to provide an autonomous driving simulation system and method based on data closed-loop and hardware-in-the-loop.

[0006] The present invention adopts the following technical solutions: An autonomous driving simulation system based on data closed-loop and hardware-in-the-loop, comprising: an autonomous driving real vehicle module, which is a test vehicle with autonomous driving function and is the basic carrier for data closed-loop processing and hardware-in-the-loop verification; a cloud platform module, which is responsible for collecting key data generated during the operation of the autonomous driving real vehicle module; a data closed-loop verification module, which summarizes, analyzes and refines the data collected by the cloud platform module, and obtains suggestions for modifying the algorithm program and simulation scenario parameters according to the data change situation. The upgraded algorithm program and scenario parameters are first injected into the autonomous driving real vehicle module to complete the program upgrade, and then the scenario parameters are injected into the simulation scenario generator module through the simulation bridging module as the parameters for its optimization feedback; a remote upgrade module, which is interconnected with the autonomous driving real vehicle module through the network to push upgrade data; a simulation bridging module, which designs different data systems of the autonomous driving real vehicle module and the simulation scenario generator module, and has functions including cross-system data format conversion, data transceiver and data parsing; a simulation scenario generator module, which injects scenario data into the autonomous driving real vehicle module through the simulation bridging module; a real vehicle verification data recording module, which records and replays the key data of the real vehicle operation under the simulation scenario, and provides data basis for the analysis of the operation state and performance improvement of the autonomous driving system.

[0007] In a preferred implementation, the above-mentioned autonomous driving functions include steer-by-wire, drive-by-wire, brake-by-wire, centimeter-level positioning, following the built-in map, and lane change and obstacle avoidance.

[0008] In a preferred implementation, the above-mentioned key data includes, but is not limited to, environmental perception data of pedestrians, vehicles, and traffic signs generated and injected by the simulation scenario generator module, vehicle state data reflecting the real-time operation of the autonomous driving real vehicle module such as speed, acceleration, and steering angle, map positioning data reflecting the position of the current vehicle module, and weather and environmental data.

[0009] In a preferred implementation, the above-mentioned simulation bridging module supports the mutual transmission of ROS and UDP format data.

[0010] In a preferred implementation, the above-mentioned scenarios include, but are not limited to, pedestrian crossing, vehicle emergency braking, vehicle cutting in, speed limit recognition, warning of approaching vehicle from side or rear, and green wave passing; the scenario data includes obstacle type, obstacle speed, obstacle moving direction, speed limit sign information, and traffic light information.

[0011] The present invention also provides an autonomous driving simulation method based on data closed-loop and hardware-in-the-loop, comprising the following steps: (1) Start relevant software and hardware systems such as the autonomous driving real vehicle; (2) The autonomous driving real vehicle automatically drives according to the preset test scenario; (3) The simulation scenario generator module inputs simulation parameters and generates simulation scenario data; (4) Determine whether the hardware-in-the-loop simulation test is proceeding normally. If so, the in-vehicle verification data module records key operation data and analyzes the data through the data closed-loop verification module; (5) Determine whether the scenario operation parameters meet the design expectations. If so, collect them as standard scenario parameters and record the test data; if not, optimize the algorithms related to the in-vehicle autonomous driving and the generation strategy of the simulation scenario.

[0012] Further, between the above step (1) and the step (2), the following steps are also included: Determine whether there are key faults affecting the system operation. If not, enter step (2); if so, determine whether the faults can be excluded through suggestions such as restarting. If so, re-enter step (1); if not, the simulation cannot be performed.

[0013] Further still, the optimized autonomous driving strategy in the above step (5) is sent to the in-vehicle autonomous driving through the remote upgrade module. The in-vehicle autonomous driving synchronizes the update of the autonomous driving strategy to the autonomous driving controller, and at the same time updates the optimized part of the simulation scenario generation strategy through the simulation bridging module.

[0014] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following advantages: 1. The autonomous driving simulation system of the present invention is based on the data closed-loop verification and hardware-in-the-loop test technologies, uses the in-vehicle autonomous driving module as the data link, and utilizes the "double-loop" mutual verification effect of the data closed-loop link and the hardware-in-the-loop link to achieve advantages such as closed-loop processing of simulation data, rapid verification of simulation algorithms, improvement of simulation scenario accuracy, and acceleration of iteration speed, providing a new approach for high-fidelity simulation of autonomous driving systems.

[0015] 2. The autonomous driving simulation method of the present invention is a process of data circulation and verification. The in-vehicle autonomous driving runs in the real road environment, and the external environment data required for constructing the scenario is generated by the scenario generator module. The two perform data interaction through the data bridging system. At the same time, the key data of the in-vehicle autonomous driving during the simulation process will be uploaded to the cloud platform for background analysis and processing. Based on the analysis and processing results, algorithm remote iteration upgrade and simulation scenario upgrade and optimization are performed; after the system upgrade, the effect of the previous upgrade is verified through hardware-in-the-loop testing, and at the same time, this test also provides data analysis basis for the next optimization and upgrade. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a block diagram of the modules of the autonomous driving simulation system of the present invention.

[0017] Figure 2 It is a schematic diagram of the double-loop link of the autonomous driving simulation system of the present invention.

[0018] Figure 3 This is the flowchart of the working process of the automatic driving simulation method of the present invention. Detailed implementation manners

[0019] The following refers to Figure 1 to illustrate the detailed implementation manners of the present invention. To fully understand the present invention, many details are described below. However, for those skilled in the art, the present invention can be implemented without these details. For well-known components, methods and processes, no further detailed description will be given below.

[0020] This embodiment provides an automatic driving simulation system based on data closed-loop and hardware-in-the-loop. Referring to Figure 1 , it includes an automatic driving real vehicle module, a cloud platform module, a data closed-loop verification module, a remote upgrade module, a simulation bridging module, a simulation scenario generator module, and a real vehicle verification data recording module.

[0021] The automatic driving real vehicle module, whose main component is a test vehicle with automatic driving function, is the basic carrier for data closed-loop processing and hardware-in-the-loop verification; the test vehicle has basic automatic driving functions such as steer-by-wire, drive-by-wire, brake-by-wire, centimeter-level positioning, following the built-in map for driving, and lane-changing to avoid obstacles.

[0022] The cloud platform module is responsible for collecting key data generated during the operation of the automatic driving real vehicle module; the key data includes but is not limited to: environmental perception data such as pedestrians, vehicles, and traffic signs generated and injected by the simulation scenario generator module, vehicle state data such as speed, acceleration, and steering angle reflecting the real-time operation of the automatic driving real vehicle module, map positioning data reflecting the position of the current vehicle module, and weather and environmental data.

[0023] The data closed-loop verification module aggregates, analyzes, and refines the data collected by the cloud platform module, obtains suggestions for modifying the algorithm program and simulation scenario parameters according to the data change situation, first injects the upgraded algorithm program and scenario parameters into the automatic driving real vehicle module to complete the program upgrade, and then injects the scenario parameters into the simulation scenario generator module through the simulation bridging module as the parameters for its optimization feedback.

[0024] The remote upgrade module is interconnected with the automatic driving real vehicle module through 4G / 5G network to push data such as program upgrade.

[0025] The simulation bridging module is designed for different data systems of the actual vehicle module for autonomous driving and the simulation scenario generator module, supports data transmission between ROS (Robot Operating System) and UDP (User Datagram Protocol) formats, and has functions such as cross-system data format conversion, data sending and receiving, and data parsing.

[0026] The simulation scenario generator module can inject scenario data into the actual vehicle module for autonomous driving through the simulation bridging module. The scenarios include but are not limited to pedestrians crossing the road, sudden braking of vehicles, vehicle cutting in, speed limit recognition, warning of vehicles coming from the side or rear, and green wave passing. The scenario data includes but is not limited to obstacle types, obstacle speeds, obstacle moving directions, speed limit sign information, traffic light information, etc.

[0027] The actual vehicle verification data recording module can record and playback the key data of the actual vehicle running under the simulation scenario, providing data basis for the analysis of the running state and performance improvement of the autonomous driving system.

[0028] Refer to Figure 2 , the main working environments of the above systems can be integrated into a data closed-loop link and a hardware-in-the-loop link (hereinafter referred to as the "dual-loop" link).

[0029] Among them, the data closed-loop link is mainly divided into key links such as the actual vehicle for autonomous driving, the enterprise cloud, data processing, algorithm training, algorithm verification, and algorithm deployment; the hardware-in-the-loop link is mainly divided into key links such as the actual vehicle for autonomous driving, simulation input, simulation scenario generation, scenario data bridging system, actual vehicle verification, and actual vehicle data acquisition and processing. The working principles of each key link are shown in Table 1 and Table 2.

[0030] Table 1 Working Principles of Key Links in the Data Closed-Loop Link Serial number Key links How it works 1 Autonomous driving real car Serving as a carrier for generating, transmitting, receiving and verifying autonomous driving data; it can transmit data to the enterprise cloud platform through the on-board 4G / 5G equipment, can receive upgrade program data issued through remote upgrade in the algorithm deployment phase, has basic autonomous driving functions, and can conduct real-vehicle autonomous driving function simulation testing and verification. 2 Enterprise Cloud As a transit link for data collection and forwarding, it can collect key operating data uploaded by autonomous driving vehicles through 4G / 5G networks, and organize the data and send it to the data processing link. 3 Data processing The massive amount of data collected by the enterprise cloud is streamlined and processed. A typical processing method involves extracting data from 10 seconds before to 5 seconds after key operational status changes, while deleting all other data. Key operational status changes include, but are not limited to, exiting autonomous driving, obstacles encroaching on the driving route, positioning failure, sensor failure, and other key system failures. The processed data is then sent to the algorithm training phase for use. 4 Algorithm training Receive key data segments provided by the data processing link, inject these data segments into the algorithm training equipment system as input data for algorithm verification, review the response methods of the autonomous driving system at key operational change moments, detect algorithm problems and improve the algorithm. 5 Algorithm Verification The improved algorithm is further verified and checked in the algorithm training equipment system. Once it is correct, the upgrade program is packaged and sent to the algorithm deployment link for remote upgrade. 6 Algorithm deployment Through a specific program interface, the upgrade package is sent to the autonomous driving vehicle for algorithm deployment. Table 2 Working Principles of Key Links in the Hardware-in-the-Loop Link Serial number Key links How it works 1 Autonomous driving real car As a physical carrier for autonomous driving hardware-in-the-loop simulation, it can complete basic autonomous driving functions such as automatic tracking, automatic obstacle avoidance, and automatic lane changing. 2 Simulation Input Simulation parameters are externally input, where the simulation parameters include but are not limited to obstacle type, obstacle latitude and longitude coordinates, obstacle movement direction, obstacle speed, etc. 3 Simulation scenario generation According to the simulation input parameters, the obstacle virtualization algorithm is used to calculate the obstacle driving trajectory data containing key information in the time domain. This data serves as the hardware-in-the-loop input information for the simulation scenario. 4 Scene data bridging system The system is designed for different data systems of the autonomous driving vehicle module and the simulation scenario generator module. The real vehicle supports ROS (Robot Operating System) topic format data, and the simulation system supports UDP (User Datagram Protocol) format data. The bridge system enables real-time mutual transmission of the above two data formats, and has functions such as data format conversion, data transmission and reception, and data analysis. 5 Actual vehicle verification The autonomous driving vehicle normally follows the predetermined map route. At this time, it receives simulated obstacle information through the bridge system, and the vehicle will make corresponding actions such as parking, avoidance, and detour according to the movement of the obstacle. 6 Real vehicle data collection and processing The data recording module records key data from the actual vehicle hardware-in-the-loop test, and through data playback, it troubleshoots program problems in the actual vehicle control algorithm and scenario generator, thereby improving and optimizing them. Refer to Figure 3 , the present invention also provides an autonomous driving simulation method based on data closed-loop and hardware in the loop, including the following steps: (1) Start relevant software and hardware systems such as the actual vehicle for autonomous driving; (2) Determine whether there are key faults affecting the system operation. If not, the actual vehicle for autonomous driving automatically drives according to the preset test scenario; if so, determine whether the fault can be eliminated through suggestions such as restarting. If it can, re-enter step (1); if not, the simulation cannot be performed.

[0031] (3) The simulation scenario generator module inputs simulation parameters to generate simulation scenario data; (4) Determine whether the hardware-in-the-loop simulation test is carried out normally. If so, the real vehicle verification data module records key operating data and analyzes the data through the data closed-loop verification module; (5) Determine whether the scenario operating parameters meet the design expectations. If so, collect them as standard scenario parameters and record test data. If not, optimize the algorithms related to the autonomous driving vehicle and optimize the generation strategy of the simulation scenario.

[0032] The following is a typical application of the present invention. Through hardware-in-the-loop simulation testing, the adaptability of the autonomous driving system's response measures to pedestrians crossing the road, the validity of the simulation scenario parameters, and the effectiveness of the data closed-loop processing are verified. The following steps are included: Step 1: Record a GPS (Global Positioning System) map for the autonomous vehicle in a closed test area. During hardware-in-the-loop simulation testing, the vehicle must be located on a straight road at least 200 meters long. A pedestrian crossing point should be located in the middle of the road. The autonomous vehicle should be debugged to ensure it can automatically follow the GPS map and perform automatic emergency braking.

[0033] Step 2: Input simulation parameters into the simulation scenario generator module. These parameters include obstacle type (pedestrian), obstacle crossing direction (movement direction is at a 90° angle to the vehicle's direction of travel, with the positive direction pointing toward the road centerline), obstacle crossing speed (1 m / s), and obstacle crossing time (when the distance to the vehicle is less than or equal to 20 meters).

[0034] Step 3. Start the simulation. Through the simulation bridge module, the autonomous driving real vehicle module and the simulation scene generator module can interact with the real vehicle speed, real vehicle moving direction, real vehicle longitude and latitude coordinates, and various parameter information of obstacles. The autonomous driving real vehicle travels along the predetermined straight section. After it starts, the speed is increased to 30 kilometers per hour and then maintained at a constant speed. When the vehicle is just less than 20 meters away from the pedestrian crossing point, the simulation scene generator module releases the virtual pedestrian target. At this time, the pedestrian moves virtually according to the preset parameters, and its current longitude and latitude coordinates can be calculated. After this data is sent to the real vehicle through the simulation bridge module, the real vehicle makes a judgment based on its built-in prediction mechanism. When it reaches the pedestrian crossing position, the distance between the vehicle and the pedestrian is less than 2 meters, and there is a risk of collision. The vehicle then performs emergency braking. Finally, the real vehicle brakes to a stop 5 meters away from the pedestrian crossing position, with a maximum deceleration of 5 meters per second. 2 .

[0035] Step 4: The key data of the above simulation process is recorded through the real vehicle verification data recording module, and the data is uploaded to the enterprise backend through the cloud platform module. The data closed-loop verification module finds two problems through the refined analysis of the data.

[0036] Problem (1): After the autonomous driving vehicle determines the risk of pedestrian crossing collision, it brakes the vehicle to a stop through an emergency braking strategy. At this time, the deceleration reaches 5 m / s² 2 , but there is still a distance of 5 meters after braking to a stop. It is judged that the deceleration setting during braking is too large and the stopping distance is too far. By adjusting the emergency stop algorithm, the distance from the vehicle to the pedestrian target after stopping can be reduced, the peak value of the braking deceleration can be decreased, the smoothness of the deceleration transition can be increased, and the comfort of the passengers can be improved. Problem (2): The virtual pedestrian release strategy needs to be improved. The existing pedestrian movement direction strategy is to maintain a 90° angle with the vehicle movement direction and point to the center line of the lane. This will cause the pedestrian's movement direction to jitter when the vehicle movement direction jitters due to unstable positioning system, and the pedestrian's movement direction will flip 180° and walk back after crossing the center line. The above phenomena are manifested in the real vehicle detection as a pedestrian whose movement direction often changes and who walks back and forth near the center line of the lane, which does not conform to the actual pedestrian crossing.

[0037] Step Five: In response to the problems found above, propose a rectification and optimization plan. Set the maximum deceleration to 3 m / s² for the above problem (1) 2 , the distance from the vehicle to the obstacle after stopping is not greater than 3 meters and not less than 2 meters, optimize the gradient deceleration strategy, and smooth the deceleration value during the entire braking process. For the above problem (2), set the pedestrian crossing direction to maintain a 90° angle with the center line of the lane and not walk back during the crossing process to form a standardized test scenario.

[0038] Send the above rectification measures to the autonomous driving vehicle through the remote upgrade module. The autonomous driving vehicle synchronizes the update of the autonomous driving strategy to the autonomous driving controller, and at the same time updates the optimized part of the strategy for the simulation scenario generation through the simulation bridging module.

[0039] Step Six: After the update, conduct the same simulation scenario test again. The test results show that the pedestrian crossing parameter values meet the expectations. The maximum deceleration during the vehicle braking process is 2.6 m / s² 2 , and the distance from the vehicle to the pedestrian target after stopping is 2.8 meters. The test results meet the expectations.

[0040] The above is only the specific implementation manner of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantive modification made to the present invention using this concept shall fall within the scope of infringement of the protection of the present invention.

Claims

1. An autonomous driving simulation system based on data closed-loop and hardware-in-the-loop, characterized in that Including: An autonomous driving real vehicle module, which is a test sample vehicle with autonomous driving function and is the basic carrier for data closed-loop processing and hardware-in-the-loop verification; A cloud platform module, which is responsible for collecting key data generated during the operation of the autonomous driving real vehicle module; A data closed-loop verification module, which summarizes, analyzes, and refines the data collected by the cloud platform module, obtains suggestions for modifying the algorithm program and simulation scenario parameters based on the data changes, first injects the upgraded algorithm program and scenario parameters into the autonomous driving real vehicle module to complete the program upgrade, and then injects the scenario parameters into the simulation scenario generator module through the simulation bridging module as the parameters for its optimization feedback; A remote upgrade module, which is interconnected with the autonomous driving real vehicle module through the network to push upgrade data; A simulation bridging module, which designs different data systems for the autonomous driving real vehicle module and the simulation scenario generator module, and has functions including cross-system data format conversion, data sending and receiving, and data parsing; A simulation scenario generator module, which injects scenario data into the autonomous driving real vehicle module through the simulation bridging module; A real vehicle verification data recording module, which records and replays the key data of the real vehicle operation under the simulation scenario, and provides data basis for the analysis of the operation state and performance improvement of the autonomous driving system.

2. The autonomous driving simulation system based on data closed-loop and hardware in the loop according to claim 1, characterized in that: The autonomous driving function includes steer-by-wire, drive-by-wire, brake-by-wire, centimeter-level positioning, following the built-in map, and lane change and obstacle avoidance.

3. The autonomous driving simulation system based on data closed-loop and hardware-in-the-loop according to claim 1, characterized in that: The key data includes environmental perception data of pedestrians, vehicles, and traffic signs generated and injected by the simulation scenario generator module, vehicle state data such as speed, acceleration, and steering angle reflecting the real-time operation of the autonomous driving real vehicle module, map positioning data reflecting the position of the current vehicle module, and weather and environmental data.

4. The autonomous driving simulation system based on data closed-loop and hardware-in-the-loop according to claim 1, wherein: The simulation bridging module supports the mutual transmission of ROS and UDP format data.

5. The autonomous driving simulation system based on data closed-loop and hardware-in-the-loop according to claim 1, characterized in that: The scenarios include pedestrian crossing, vehicle emergency stop, vehicle cut-in, speed limit recognition, warning of vehicle coming from the side or rear, and green wave passing; the scenario data includes obstacle type, obstacle speed, obstacle moving direction, speed limit sign information, and traffic light information.

6. A method for autonomous driving simulation based on data closed-loop and hardware-in-the-loop, including an autonomous driving simulation system based on data closed-loop and hardware-in-the-loop as described in claim 1, characterized in that, Including the following steps: (1) Start relevant software and hardware systems such as the autonomous driving real vehicle; (2) The autonomous driving real vehicle automatically drives according to the preset test scenario; (3) The simulation scenario generator module inputs simulation parameters and generates simulation scenario data; (4) Judge whether the hardware-in-the-loop simulation test is proceeding normally. If so, the real vehicle verification data module records the key operation data and analyzes the data through the data closed-loop verification module; (5) Judge whether the scenario operation parameters meet the design expectations. If so, collect them as standard scenario parameters and record the test data; if not, optimize the relevant algorithms of the autonomous driving real vehicle and the generation strategy of the simulation scenario.

7. The method for simulating an autonomous vehicle based on data closed-loop and hardware in the loop according to claim 6, wherein: Between the step (1) and the step (2), the following steps are also included: Judge whether there are key faults affecting the system operation. If not, enter the step (2); if so, judge whether the faults can be excluded through suggestions such as restart. If so, re-enter the step (1); if not, the simulation cannot be performed.

8. The method for simulating autonomous driving based on data closed-loop and hardware-in-the-loop according to claim 6, wherein: The optimized autonomous driving strategy in step (5) is sent to the actual autonomous driving vehicle through the remote upgrade module. The actual autonomous driving vehicle synchronizes the update of the autonomous driving strategy to the autonomous driving controller, and at the same time updates the optimized part of the strategy generated for the simulation scenario through the simulation bridging module.

Images