Automatic driving complex scene safety decision-making system based on digital twinning

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to an automatic driving complex scene safety decision-making system based on digital twinning, which comprises an environment sensing module, a data processing module, a cloud decision-making module, a digital twinning rehearsal module and a vehicle judgment and execution module, the data acquisition module is used for acquiring vehicle surrounding data and vehicle state data; the data processing module is used for processing the original data of the environment sensing module in real time to generate an obstacle information list; the cloud decision module takes security as a main target; the digital twin rehearsal module is based on a high-precision vehicle dynamics model and a safety rule base; and the vehicle judgment and execution module is used for judging whether the cloud instruction is invalid. According to the automatic driving complex scene safety decision-making system based on digital twinning, the collision probability of vehicles in a complex intersection scene can be effectively reduced, and the real-time performance and safety of automatic driving decision-making are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to an automatic driving complex scene safety decision system based on digital twinning. BACKGROUND

[0002] With the rapid development and close integration of current automobile, computer, radar, visual sensor, surveying and mapping, robot and other technologies, the advantages of automatic driving technology are becoming more and more obvious, but safety has become a key problem faced by automatic driving technology. Safety is an important factor hindering the commercialization of automatic driving cars and has been a research hotspot in recent years. The automatic driving decision planning module, as the "brain" of the automatic driving car, has a considerable influence on the safety of the automatic driving car. How to make autonomous safety decisions has been widely studied in recent years.

[0003] The current unmanned system must train and test motion planning and other decision algorithms in the corresponding physical scene, which has high cost, long cycle and is not comprehensive enough. Therefore, introducing digital twinning technology into automatic driving perfectly solves this problem. Compared with the vehicle decision system which is limited by the detection distance of the vehicle-mounted sensor and has a perception blind area, and the cloud control decision system which has the defect of insufficient decision generalization ability in complex scenes, the digital twinning technology can create digital twin models of vehicles, environments and traffic conditions for deep scene reproduction and simulation, helping the decision system of the automatic driving car to better understand complex situations and improve the accuracy and safety of the decision of the automatic driving car.

[0004] In summary, although the decision method based on deep reinforcement learning has strong scene adaptability, it still has reliability problems. In the prior art, the application number CN202211649994.0 discloses a cloud control automatic driving system based on an automatic driving special road network and a digital twinning map, which includes an automatic driving special road network planning module, a digital twinning map module, a plurality of scene cloud control systems and a plurality of edge cloud control systems. The automatic driving special road network planning module creates a safe and orderly physical driving environment for automatic driving vehicles, and the digital twinning map module constructs a real-time twinned traffic digital space and provides scheduling services. The scene cloud control system generates the best right of way, driving right and global driving trajectory reference line. The edge cloud control system provides low-delay cloud control scheduling services for all automatic driving vehicles connected to the edge cloud control system, ensuring traffic efficiency and driving safety in local scenes. However, the application only uses digital twinning technology for scheduling services of automatic driving vehicles and does not directly interact with the automatic driving vehicles. Therefore, there is an urgent need for an automatic driving complex scene safety decision system based on digital twinning to improve the safety of automatic driving car decision planning. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an automatic driving complex scene safety decision system based on digital twinning, which can provide an automatic driving complex scene system based on digital twinning, realize real-time communication of vehicle-side and cloud-side decisions, and perform safety constraint on vehicle-side decisions through a digital twinning scene to ensure the safety of automatic driving.

[0006] The technical solution adopted by the present application is:

[0007] An automatic driving complex scene safety decision system based on digital twinning is provided, which comprises an environment perception module, a data processing module, a cloud-side decision module, a digital twinning pre-play module, and a vehicle judgment and execution module,

[0008] The environment perception module is configured to perceive environment data and ego vehicle state data around a vehicle through vehicle-side sensors and road-side sensors, and send the data to the data processing module through an MQTT protocol.

[0009] The data processing module is configured to process raw data of various sensors on the vehicle side and the road side, generate a target obstacle type, length, width, height, speed, heading angle, and other information list through a multi-sensor fusion perception algorithm, and transmit the information list to the cloud-side decision module and the digital twinning pre-play module through an MQTT protocol.

[0010] The cloud-side decision module is configured to output candidate control instructions of the vehicle based on a DDQN deep reinforcement learning model based on horizontal and vertical safety rule constraints according to the target obstacle information list, and send the instructions to the digital twinning pre-play module for testing, while re-planning unsafe control instructions according to the test feedback results of the digital twinning pre-play module.

[0011] The digital twinning pre-play module is configured to construct a digital twinning automatic driving scene based on the target obstacle information list and ego vehicle state information using UE4 software, perform a t-second time window ahead deduction test on the candidate control instructions output by the cloud-side decision module, feed back unsafe candidate control instructions to the cloud-side decision module for re-planning, and send safe candidate control instructions to the vehicle judgment and execution module through a ZMQ protocol.

[0012] The vehicle judgment and execution module is configured to judge whether the cloud-side instructions are invalid, control the vehicle to execute the control instructions, establish a safety override mechanism, judge whether the cloud-side instructions are invalid, and execute the final cloud-side instructions or the control instructions determined by the RSS model based on physical rules on the vehicle side according to the judgment result.

[0013] Further, the vehicle-end sensors of the environment perception module include a 16-line mechanical laser radar and a 1080P camera, and the road-end sensors include an MMR millimeter wave radar, a 192-line solid-state laser radar, and a 1080P camera. The millimeter wave radar is used to detect the speed information of obstacles, specifically including the speed of obstacles in x and y directions. The laser radar is used to provide three-dimensional spatial information of obstacles, specifically including the size (length, width, and height) and distance of obstacles to help determine the accurate position and shape of obstacles. The camera is used to collect image information of obstacles, specifically including the type of obstacles, target tracking ID, and position x and y.

[0014] Further, the environment data of the environment perception module includes image data around the vehicle, point cloud image data of the environment around the vehicle, high-precision map data around the vehicle, and vehicle body attitude data. The vehicle body attitude data includes vehicle speed, vehicle acceleration, vehicle coordinates, and vehicle heading angle.

[0015] Further, the data processing of the data processing module is performed by real-time processing of the image data around the vehicle using the visual YOLO algorithm to generate a type and position target list, real-time processing of the environment data around the vehicle using the JPDA algorithm to generate a speed target list, and combining radar point cloud image data using a perception fusion algorithm to generate a final fusion perception list and transmitting it to the cloud decision module through a 5G network using the MQTT protocol.

[0016] Further, the deep reinforcement learning model of the cloud decision module uses the DDQN algorithm. The state space S includes vehicle state Si and pedestrian state Sj, as shown below:

[0017]

[0018] wherein the vehicle state Si includes the speed, acceleration, coordinate position, and heading angle of the vehicle, and the pedestrian state Sj includes the moving speed, moving direction, and position coordinate of the pedestrian in the environment. The deep reinforcement learning model generates optimal decision instructions based on these state space information and a pre-set reward mechanism through continuous learning and training to guide the safe driving and path planning of the vehicle in complex environments.

[0019] The action space A includes a set of all available actions of the ego vehicle, and the action space includes five discrete action instructions,

[0020] A={‘accelerate’, ‘decelerate’, ‘constant speed’, ‘left turn’, ‘right turn’}

[0021] The reward function design mainly considers comfort, efficiency, and safety to ensure safe and efficient driving of the vehicle,

[0022] Total Rewards:

[0023] Comfort rewards:

[0024] Efficiency Rewards:

[0025] Security Rewards: =

[0026] Furthermore, the digital twin pre-simulation module uses UE4 software to model and generate a digital twin autonomous driving scenario based on the environmental data and vehicle posture data around the vehicle, and uses LOD technology to perform lightweight processing on the digital twin autonomous driving scenario.

[0027] Furthermore, in the digital twin pre-simulation module, surrounding traffic participants use the LSTM algorithm to predict their trajectories based on the perception list from the data processing module. These participants then simulate their predicted future trajectories in the twin scenario. The pre-simulated trajectories are dynamically corrected based on feedback from vehicles regarding the actual surrounding environment. A deviation threshold is set; if the threshold is exceeded, the future trajectories of the traffic participants are re-predicted. The positional deviation between the pre-simulated and actual trajectories is calculated using Euclidean distance, with the following formula:

[0028]

[0029] in , The predicted coordinates of traffic participants at time t in a digital twin simulation; , The actual coordinates of traffic participants at time t, collected by vehicle-mounted / roadside sensors; .

[0030] Furthermore, the digital twin pre-simulation module performs advance simulation of the received behavioral decision candidate instructions within a time window of t seconds. The t seconds are determined based on the actual vehicle speed, basic braking distance, minimum simulation time, and total system latency of the autonomous vehicle, as shown in the following formula:

[0031]

[0032] In the formula This indicates the minimum rehearsal time for advanced simulation of digital twin driving scenarios. Indicates the vehicle's actual speed. Indicates the vehicle's basic braking distance. This represents the total system latency.

[0033] Further, the safety override mechanism of the vehicle judgment and execution module has two triggering conditions, one is that the cloud instruction conflicts with the vehicle-side decision instruction, such as the cloud requiring acceleration, but an obstacle appears in front; the other is that the network delay > threshold; if any one of the two conditions triggers the safety override mechanism, the vehicle-side RSS model decision based on physical rules will be adopted, and if no triggering condition is triggered, the vehicle will be controlled according to the final decision instruction of the cloud.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] The system of the present application can break through the limitations of traditional decision system sensing blind area and cloud control system scene coverage deficiency by the cooperation of the designed environment perception module, data processing module, cloud decision module, digital twin pre-play module and vehicle judgment and execution module, so as to make the digital twin technology interact with the autonomous vehicle, further improve the safety of vehicle driving according to the consistency of digital twin scene and real scene, improve the safety of autonomous vehicle driving at complex signal-free intersections, improve the real-time performance, safety and generalization ability of vehicle decision in complex intersection scenarios, reduce the probability of collision, and overall, the present application provides a comprehensive and effective solution for safety decision of autonomous driving in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 Figure 1 is a diagram of an autonomous driving complex scenario safety decision system based on digital twin.

[0037] Figure 2 Figure 4 is a flowchart of a signal-free intersection safety decision test process.

[0038] Figure 3 Figure 5 is a flowchart of a vehicle-side safety override mechanism test process. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0040] As Figure 1As shown, an automatic driving complex scene safety decision system based on digital twinning includes an environment perception module, a data processing module, a cloud decision module, a digital twinning pre-play module, and a vehicle judgment and execution module. Specifically, the environment perception module is used to perceive the road traffic environment information around the vehicle and send it to the data processing module. The environment perception module is composed of a vehicle-end perception module and a road-end perception module. The vehicle-end perception module is composed of several laser radars and cameras, and the road-end perception module is composed of laser radars, millimeter wave radars, and cameras. The raw data obtained by the environment perception module includes image data around the vehicle, point cloud image data of the environment around the vehicle, high-precision map data around the vehicle, and vehicle state data including vehicle speed, vehicle acceleration, vehicle coordinates, and vehicle heading angle.

[0041] The data processing module is used to process the raw data sent by the road-end and vehicle-end. The radar point cloud image data and visual image data are fused by the F-Point Net algorithm to generate a list of target obstacle types, length, width, height, speed, heading angle, etc., and transmitted to the cloud decision module through the ZMQ protocol via a 5G network.

[0042] The cloud decision module is used to generate candidate control instructions for the autonomous vehicle. Based on the list of target obstacle types, length, width, height, speed, heading angle, etc., transmitted by the data processing module, a deep reinforcement learning model is used to generate control instructions and send them to the digital twinning pre-play module for testing.

[0043] The digital twinning pre-play module is used to test whether the control instructions output by the cloud decision module are safe. The driving pre-play is performed on the surrounding dynamic obstacles and the vehicle itself within a time window of t seconds. If the test is qualified, the control instructions are output to the vehicle judgment and execution module. If the test is not qualified, the cloud decision module generates new control instructions.

[0044] The vehicle judgment and execution module is used to judge whether the cloud instructions are invalid and control the vehicle to execute the control instructions and give correct and timely feedback.

[0045] Example 1, as shown in Figure 2 The safety decision test for signal-free intersection passage:

[0046] Step one: After the vehicle starts driving, send "hello" to the cloud and the road to check the connection between the vehicle, the road, and the cloud. If the cloud and the road can continuously and stably reply, it is confirmed that the vehicle-road-cloud communication quality is good.

[0047] Step two: The environment perception module starts the vehicle-end and road-end sensors to cooperate in data collection.

[0048] Vehicle-end sensor: 16-line mechanical laser radar is used to collect 3D point cloud data around the vehicle, 1080P camera is used to collect image data, and the state of the vehicle (speed 20 km / h, acceleration 0.5 m / s², coordinates (X=100, Y=200), heading angle 0°) is synchronously acquired;

[0049] Road-end sensor: road-side millimeter wave radar detects the x, y direction speed of other vehicles in the intersection (such as the speed of the left-turn vehicle is 25 km / h, and the lateral speed is 5 m / s), 192-line solid-state laser radar provides high-precision obstacle three-dimensional information (such as the non-motor vehicle crossing the intersection laterally has a length, width and height of 1.8 m×0.6 m×1.2 m), and 1080P camera assists in identifying the types of traffic participants (motor vehicle, non-motor vehicle, pedestrian), and all raw data are transmitted in real time to the data processing module through the MQTT protocol;

[0050] Step three: the data processing module at the road side processes the received raw data, generates an obstacle information list (such as there is an electric bicycle with a length, width and height of 1.8 m×0.6 m×1.2 m, a speed of 15 km / h, an acceleration of 0.5 m / s², and a heading angle of 20° at the coordinates of X=300, Y=100) through a perception fusion algorithm, and the processed obstacle information list is transmitted in real time to the cloud decision module through the MQTT protocol;

[0051] Step four: the cloud decision module in the cloud inputs the obstacle information list and the state of the vehicle to construct a state space S based on a DDQN deep reinforcement learning model, and outputs a candidate control instruction “uniform straight running” (maintaining 20 km / h to pass through the intersection), and the candidate control instruction is sent to the digital twin pre-rehearsal module for testing through the MQTT protocol;

[0052] Step five: the digital twin module in the cloud constructs a digital twin scene consistent with the real intersection through the UE4 software, predicts the trajectory of the dynamic obstacle based on the LSTM algorithm, calculates a pre-rehearsal time window t=3 seconds, pre-rehearses the “uniform straight running” instruction for 3 seconds, the virtual twin vehicle runs straight at 20 km / h, and the minimum distance from the opposite electric bicycle is greater than the safety threshold (5 m), and there is no collision risk, the instruction is determined to be safe, the test is passed, and the instruction is sent to the vehicle judgment and execution module;

[0053] Step six: the vehicle judgment and execution module at the vehicle end receives the “uniform straight running” instruction from the cloud, checks the communication delay (0.3 s< threshold 1 s), and the instruction is consistent with the decision of the vehicle-end RSS model (no conflict), and the safety override mechanism is not triggered. The actuator controls the vehicle to maintain 20 km / h uniform straight running to pass through the intersection, and the real-time vehicle state is fed back to the cloud to complete the closed-loop test.

[0054] In embodiment 2,Figure 3 The cloud instruction invalidation vehicle end safety override mechanism test is shown:

[0055] Step one: the vehicle end and the road end sensor normally collect intersection data (the vehicle is ready to turn left, and there is a straight vehicle opposite), the data processing module generates an obstacle list and transmits it to the cloud;

[0056] Step two: the cloud generates a candidate instruction "turn left", but due to 5G network fluctuation, the instruction transmission delay is 1.5s (exceeding the threshold of 1s);

[0057] Step three: the vehicle judgment and execution module detects that the network delay is 1.5s> threshold, triggers the safety override mechanism, enables the vehicle end RSS model decision, and the RSS model makes a decision "wait for 3s and then turn left" based on the current environment (the distance of the opposite vehicle is 80m, and the speed is 40km / h), and controls the vehicle to pause;

[0058] Step four: when the cloud instruction delay arrives, the vehicle has executed the vehicle end decision, the system records the delay event and feeds back to the cloud, and the cloud adjusts the subsequent instruction sending strategy.

[0059] In summary, the application provides an automatic driving complex scene safety decision system based on digital twinning. The system architecture design significantly improves the decision safety and reliability of the automatic driving vehicle in complex scenes. The environment perception module can comprehensively and accurately obtain the vehicle surrounding environment information, providing rich and accurate data basis for subsequent processing. The data processing module has high real-time processing capability, ensuring that data can be timely converted into valuable decision basis. The cloud decision module generates scientific and reasonable candidate decision instructions with powerful computing resources and advanced algorithms. The digital twinning pre-play module verifies the safety of the candidate decision instruction, and excludes potential risks in advance to ensure the safety of the decision. It can realize accurate perception of complex scenes of automatic driving, efficient data processing, scientific cloud decision, advanced digital twinning pre-play and safe and reliable vehicle judgment and execution. The environment perception module can comprehensively collect surrounding environment information, providing rich and accurate data basis for subsequent processing. The data processing module can quickly filter, analyze and process massive data to extract key information. The cloud decision module makes scientific and reasonable decisions with powerful computing capability and advanced algorithms. The digital twinning pre-play module simulates various possible situations in advance to provide forward-looking reference for decision making. The safety override mechanism of the vehicle judgment and execution module ensures that the vehicle can make safe decisions based on physical rules in special situations, effectively improving the safety and reliability of automatic driving in complex scenes.

[0060] It should be understood that those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the application.

Claims

1. A safety decision-making system for complex scenarios of autonomous driving based on digital twins, characterized in that, It includes an environmental perception module, a data processing module, a cloud decision-making module, a digital twin pre-simulation module, and a vehicle judgment and execution module. The environmental perception module is used to collect environmental data and vehicle status data around the vehicle through vehicle-side sensors and road-side sensors, and send them to the data processing module via the MQTT protocol. The data processing module is used to process the raw data collected by the environmental perception module in real time, generate an information list containing the type, length, width and height, speed and heading angle of the target obstacle through a multi-sensor fusion perception algorithm, and transmit it to the cloud decision module and digital twin pre-simulation module through the MQTT protocol. The cloud-based decision-making module is used to generate candidate control commands based on the obstacle information list provided by the data processing module, which is based on the DDQN deep reinforcement learning model constrained by horizontal and vertical safety rules. The command is then sent to the digital twin pre-simulation module for testing, and unsafe commands are replanned based on the test feedback. The digital twin pre-simulation module is used to construct a digital twin autonomous driving scenario using UE4 software. It performs advanced simulation testing on the candidate control commands output by the cloud decision module in a t-second time window, feeds back high-risk commands to the cloud decision module for replanning, and sends safety commands to the vehicle judgment and execution module via the ZMQ protocol. The vehicle judgment and execution module is used to determine whether the cloud command is invalid, and execute the final cloud command or the vehicle-side RSS model decision command based on physical rules based on the security overwrite mechanism.

2. The autonomous driving safety decision-making system based on digital twins for complex scenarios as described in claim 1, characterized in that, The vehicle-side sensors of the environmental perception module include a mechanical lidar and a camera; the road-side sensors include a millimeter-wave radar, a lidar, and a camera; wherein, the millimeter-wave radar is used to detect the speed of obstacles in the x and y directions; the lidar is used to provide three-dimensional spatial information of obstacles; and the camera is used to collect obstacle type, tracking ID, and position x and y.

3. The autonomous driving safety decision-making system based on digital twins for complex scenarios according to claim 1, characterized in that, The environmental data of the environmental perception module includes image data of the vehicle's surroundings, point cloud image data, high-precision map data, and vehicle posture data.

4. The autonomous driving safety decision-making system based on digital twins for complex scenarios as described in claim 1, characterized in that, The data processing module processes data by using the visual YOLO algorithm to process image data around the vehicle in real time to generate a list of target types and locations, using the JPDA algorithm to process environmental data around the vehicle in real time to generate a list of speed targets, and using a perception fusion algorithm to combine radar point cloud image data to generate a final fused perception list, which is then transmitted to the cloud decision module via the MQTT protocol over a 5G network.

5. The autonomous driving safety decision-making system based on digital twins for complex scenarios according to claim 1, characterized in that, The deep reinforcement learning model of the cloud-based decision-making module includes a state space S and an action space A; wherein, the state space S includes the vehicle state S0. i Pedestrian Status S j The motion space A = {'accelerate', 'decelerate', 'constant speed', 'turn left', 'turn right'}.

6. The autonomous driving safety decision-making system based on digital twins for complex scenarios according to claim 1, characterized in that, The digital twin pre-simulation module uses LOD technology to perform lightweight processing of the digital twin scene, predicts the trajectory of traffic participants through the LSTM algorithm, and dynamically corrects the pre-simulated trajectory based on the actual environment feedback from the vehicle. When the deviation exceeds the threshold, it re-predicts.

7. The autonomous driving safety decision-making system based on digital twins for complex scenarios according to claim 1, characterized in that, The digital twin pre-simulation module performs advance simulations of received behavioral decision candidate instructions within a time window of t seconds. The t seconds are determined based on the actual vehicle speed, basic braking distance, minimum simulation time, and total system latency of the autonomous vehicle, as shown in the following formula: In the formula This indicates the minimum rehearsal time for advanced simulation of digital twin driving scenarios. Indicates the vehicle's actual speed. Indicates the vehicle's basic braking distance. This represents the total system latency.

8. The autonomous driving safety decision-making system based on digital twins for complex scenarios according to claim 1, characterized in that, The security overwrite mechanism of the vehicle judgment and execution module is triggered under the following conditions: conflict between cloud instructions and vehicle-side decision instructions; network latency > threshold; when any of the conditions are triggered, the RSS model decision instructions are enabled, otherwise the final cloud instructions are executed.

Images