Remote control simulation driving system based on digital physical twin

Abstract

The invention relates to the technical field of automatic driving and the technical field of digital twinning, in particular to a remote control simulation driving system based on digital and physical twinning. Comprising a sensing layer, a twinborn layer and a decision execution layer, the sensing layer comprises automatic driving vehicle sensing based on an intelligent network connection vehicle and environment sensing based on a road side unit, and is used for collecting vehicle operation data and traffic environment data and using the collected data to construct a twin layer; according to the invention, a multi-mode sensor is used for collecting data, and a digital twin environment is remotely constructed, so that a safety officer can accurately recognize the environment where a vehicle is located; the real motion state of the vehicle is restored through remote simulation of the cockpit, physical twinning of vehicle motion is achieved, and the control precision of a safety officer is improved; by utilizing a simulated driving environment, the safety and comfort of remote takeover are ensured, accurate and efficient remote operation is realized, and finally the driving takeover level is improved.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving technology and digital twin technology, and in particular to a remote-controlled simulated driving system based on digital-physical twins. Background Art

[0002] With the continuous upgrading of artificial intelligence technology and sensor technology, autonomous driving technology has developed rapidly as an effective means to improve urban traffic safety and travel efficiency. Autonomous driving technology integrates intelligent networking, artificial intelligence perception and decision-making, radar positioning and perception data fusion, high-precision map positioning, vehicle sensor monitoring and other technologies, allowing vehicles to travel without human intervention. In recent years, shared taxis providing autonomous driving services have gradually begun to operate. However, due to the large number of elements in the transportation system and the complex interaction between mixed traffic entities, the problems of limited perception range of autonomous driving of single-vehicle intelligence, insufficient collaborative decision-making ability, and poor global optimization ability have gradually become prominent, especially in failure and critical design operation domains such as severe weather, extreme working conditions, and workshop behavior conflicts. Autonomous driving vehicles may be out of control, affecting driving safety and traffic efficiency.

[0003] In this context, remote takeover of autonomous vehicles becomes a safe and efficient solution. Information about the vehicle's surroundings is collected through on-board cameras, millimeter-wave radars, lidars and other sensing devices, and transmitted to the central control platform through the mobile network. The safety officer then remotely controls the smart connected vehicle to drive it out of the dangerous area. Compared with offline manual control by the safety officer, remote takeover undoubtedly improves efficiency and versatility, making large-scale application of autonomous driving smart connected vehicles possible.

[0004] However, the current remote takeover technology still faces two defects: video, as the main means of providing information, can only provide dynamic information within the vehicle's surrounding field of view, and it is difficult to make an accurate description of the vehicle's overall environment, which hinders the safety officer's judgment of road conditions and increases the difficulty of operation; the current safety officer is set to take over in a fixed environment, lacks control over the vehicle's overall motion state, and reduces safety and comfort. Summary of the invention

[0005] In order to make up for the shortcomings of the prior art, the present invention proposes a remote-controlled simulated driving system based on digital-physical twins to solve the limitations of existing remote takeover technology in the application of autonomous driving vehicles, especially in terms of information acquisition and operation control; specifically, the purpose of the present invention is: to collect data by using multimodal sensors, remotely build a digital twin environment, and ensure that the safety officer has an accurate understanding of the vehicle's environment; through a remote simulation of the cockpit, the vehicle's real motion state is restored, the physical twin of the vehicle's motion is realized, and the safety officer's control accuracy is improved; using a simulated driving environment to ensure the safety and comfort of remote takeover, achieve accurate and efficient remote operation, and ultimately improve the level of driving takeover.

[0006] A remote-controlled simulated driving system based on digital-physical twins described in the present invention includes a perception layer, a twin layer, and a decision-making execution layer; the perception layer includes vehicle perception based on intelligent connected vehicles for autonomous driving and environmental perception based on roadside units, which is used to collect vehicle operation data and traffic environment data, and use the collected data to construct a twin layer; it contains digital twins and physical twins, and uses the multimodal data transmitted by the behavior layer to construct a super-perceptual simulated driving environment, realizing the digital twin of the vehicle's traffic environment and the physical twin dual modeling of the vehicle's motion behavior; after the decision-making execution layer determines that the vehicle is in a takeover state, it first records the control command issued by the control instruction input device, and then transmits the vehicle control command to the controlled vehicle of the behavior layer through the 5G mobile network; the controlled vehicle performs operations according to the vehicle control command, so that the simulated cockpit can achieve posture following with the controlled vehicle.

[0007] Preferably, the perception layer includes two modules: vehicle perception based on intelligent connected vehicles and environmental perception based on roadside units; the vehicle perception part captures surrounding video information through multi-view panoramic cameras around the vehicle, uses lidar to detect the surrounding three-dimensional scenes, and collects vehicle positioning coordinates, speed, acceleration and yaw angle operating data through an inertial measurement unit.

[0008] Preferably, traffic environment perception is achieved by using roadside units such as cameras and traffic sensors to obtain traffic environment data during vehicle driving, including road building information models, lane line detection, traffic light recognition, traffic sign recognition, pedestrian detection, non-motor vehicle detection, motor vehicle detection, obstacle recognition and vehicle positioning, etc., and uniformly establishes unit information with time-stamped data association and fusion in the vehicle coordinate system.

[0009] Preferably, the twin layer includes two modules: digital twin and physical twin. The digital twin module has two goals: using the historical information of the roadside unit to reproduce the meso-urban road-level driving environment, generating partial road background image models based on the historical road video data collected by each roadside unit, and extracting background pixels from the historical road image data to build an environmental base for micro-simulation; using the target vehicle laser point cloud data and 3D Gaussian fuzzy modeling technology to reconstruct the single-vehicle-level micro-road three-dimensional scene, identifying the contours and three-dimensional dimensions of the surrounding target objects through the laser point cloud, obtaining a complete three-dimensional spatial dynamic structure diagram of the complex road scene, and providing real-time road condition information centered on the target vehicle on the head-up display in front of the driver.

[0010] Preferably, the physical twin module has two goals: by collage of panoramic camera video information around the vehicle, the panoramic view centered on the target vehicle is reproduced on a circular panoramic display screen with the simulated driving console as the center, providing a full-view takeover environment for the safety officer; by reproducing the IMU sensor through the six-degree-of-freedom driving simulation driving seat to collect vehicle positioning coordinates, speed, acceleration and yaw angle operating information; through the collaborative combination of digital twin and physical twin modules, a remote driving console is jointly built to provide the safety officer with a super-perceptive fully simulated digital-physical twin driving environment.

[0011] Preferably, the decision-making execution layer includes a cloud-based decision-making layer and a roadside execution layer; after the safety officer at the cloud-based decision-making layer makes a driving judgment based on the information provided by the twin layer, the decision layer transmits the control instructions through the cabin-side controller of the simulated driving components such as the steering wheel and pedals; when it is determined that the controlled vehicle is in a takeover state, the decision layer records the vehicle control command generated by the simulated driving component, and sends the vehicle control command to the controlled vehicle through the 5G mobile network, so that the roadside execution layer redirects the controlled vehicle to drive according to the vehicle control command, so that the simulated cockpit can achieve posture following with the controlled vehicle.

[0012] The beneficial effects of the present invention are as follows:

[0013] 1. The present invention uses multimodal sensors to collect data and remotely constructs a digital twin environment to ensure that the safety officer has an accurate understanding of the vehicle's environment; through remote simulation of the cockpit, the vehicle's true motion state is restored, the physical twin of the vehicle's motion is realized, and the safety officer's control accuracy is improved; the simulated driving environment is used to ensure the safety and comfort of remote takeover, achieve accurate and efficient remote operation, and ultimately improve the level of driving takeover.

[0014] 2. By constructing a dual model of digital twins and physical twins, the present invention can provide a super-perceptive digital-physical dual twin driving environment for remote takeover safety officers, significantly improving the remote takeover capability and stability when the vehicle exceeds the designed operating domain; by real-time collection of vehicle operation data and traffic environment data, combined with digital twin and physical twin technologies, it can provide safety officers with comprehensive and accurate vehicle and environmental information, effectively ensuring the operating safety of autonomous driving vehicles in the event of a single-vehicle autonomous driving system failure or failure.

[0015] 3. The present invention utilizes 5G mobile communication technology, and the system can realize the rapid transmission of vehicle operation data and traffic environment data, as well as the real-time issuance of remote takeover control instructions, thereby improving the system response speed and information processing capabilities; through the coordinated work of the vehicle perception and execution module and the environmental perception module, the system can achieve high-precision perception of the vehicle's surrounding environment, including lane line detection, traffic light recognition, traffic sign recognition, pedestrian detection, vehicle detection, obstacle recognition and vehicle positioning, etc., to provide accurate environmental information for remote takeover. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is further described below in conjunction with the accompanying drawings and implementation modes.

[0017] Figure 1 It is a schematic diagram of a remote control simulation device of a digital physical twin of the present invention.

[0018] Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0020] like Figure 1-2 As shown, a remote-controlled simulated driving system based on digital-physical twins described in the present invention includes: a perception layer, a twin layer, and a decision-making execution layer; the perception layer includes vehicle perception based on intelligent connected vehicles for autonomous driving and environmental perception based on roadside units, which is used to collect vehicle operation data and traffic environment data, and use the collected data to construct a twin layer; the twin layer contains digital twins and physical twins, and uses the multimodal data transmitted by the behavior layer to construct a super-perceptual simulated driving environment, realizing the digital twin of the traffic environment in which the vehicle is located and the physical twin dual modeling of the vehicle's motion behavior; after the decision-making execution layer determines that the vehicle is in a takeover state, it first records the control command issued by the control instruction input device, and then transmits the vehicle control command to the controlled vehicle of the behavior layer through the 5G mobile network; the controlled vehicle performs operations according to the vehicle control command, so that the simulated cockpit can achieve posture following with the controlled vehicle.

[0021] As an embodiment of the present invention, the perception layer includes two modules: vehicle perception based on intelligent connected vehicles and environmental perception based on roadside units; the vehicle perception part captures surrounding video information through a multi-view panoramic camera around the vehicle, detects the surrounding three-dimensional scene using a lidar, and collects vehicle positioning coordinates, speed, acceleration, and yaw angle operation data through an inertial measurement unit; traffic environment perception obtains traffic environment data during vehicle driving by using roadside units such as cameras and traffic sensors, including road building information models, lane line detection, traffic light recognition, traffic sign recognition, pedestrian detection, non-motor vehicle detection, motor vehicle detection, obstacle recognition, and vehicle positioning, etc., and uniformly establishes unit information with time-stamped data association and fusion in the vehicle coordinate system.

[0022] As an implementation mode of the present invention, the twin layer includes two modules: digital twin and physical twin. The digital twin module has two objectives: using the historical information of the roadside unit to reproduce the meso-city road-level driving environment, generating a partial road background image model based on the historical road video data collected by each roadside unit, and extracting background pixels from the historical road image data to build an environmental base for micro-simulation; using the target vehicle laser point cloud data and 3D Gaussian fuzzy modeling technology to reconstruct the single-vehicle-level micro-road three-dimensional scene, identifying the contours and three-dimensional dimensions of the surrounding target objects through the laser point cloud, and obtaining a complete three-dimensional spatial dynamic structure diagram of the complex road scene. , and provide real-time road condition information centered on the target vehicle on the head-up display in front of the driver; the physical twin module has two goals: by collage of panoramic camera video information around the vehicle, the panoramic view centered on the target vehicle is reproduced on a circular panoramic display screen with the simulated driving console as the center, providing a full-view takeover environment for the safety officer; through the six-degree-of-freedom driving simulation driving seat, the IMU sensor is reproduced to collect vehicle positioning coordinates, speed, acceleration and yaw angle operation information; through the collaborative combination of digital twin and physical twin modules, a remote driving console is jointly built to provide the safety officer with a super-perceptual fully simulated digital-physical twin driving environment.

[0023] As an implementation mode of the present invention, the decision-making execution layer includes a cloud-based decision-making layer and a road-side execution layer; after the safety officer at the cloud-based decision-making layer makes a driving judgment based on the information provided by the twin layer, the decision-making layer transmits the control instructions through the cabin-end controller of the simulated driving components such as the steering wheel and pedals; when it is determined that the controlled vehicle is in a takeover state, the decision-making layer records the vehicle control command generated by the simulated driving component, and sends the vehicle control command to the controlled vehicle through the 5G mobile network, so that the road-side execution layer redirects the controlled vehicle to drive according to the vehicle control command, so that the simulated cockpit can achieve posture following with the controlled vehicle.

[0024] It can be seen from the technical solution provided by the above-mentioned invention that the present invention can provide a super-perceptive digital-physical twin driving environment for the remote takeover safety officer, and realize remote takeover control in this environment, to ensure the safe operation of the vehicle in the event of a failure or malfunction of the single-vehicle automatic driving system, and realize effective remote control of automatic driving intelligent connected vehicles.

[0025] The present invention has the following significant technical effects and advantages:

[0026] a) Enhanced remote takeover capability: By building a dual model of digital twins and physical twins, this system can provide a super-perceptual digital-physical dual-twin driving environment for remote takeover safety officers, significantly improving the remote takeover capability and stability when the vehicle exceeds the designed operating domain;

[0027] b) Improve operational safety: The system collects vehicle operation data and traffic environment data in real time, and combines digital twin and physical twin technologies to provide safety officers with comprehensive and accurate vehicle and environment information, effectively ensuring the operational safety of autonomous vehicles in the event of a single vehicle autonomous driving system failure or failure;

[0028] c) Real-time data transmission and processing: Using 5G mobile communication technology, the system can achieve rapid transmission of vehicle operation data and traffic environment data, as well as real-time issuance of remote takeover control instructions, improving system response speed and information processing capabilities;

[0029] d) High-precision environmental perception: Through the collaborative work of the vehicle perception and execution module and the environmental perception module, the system can achieve high-precision perception of the vehicle's surrounding environment, including lane line detection, traffic light recognition, traffic sign recognition, pedestrian detection, vehicle detection, obstacle recognition and vehicle positioning, etc., providing accurate environmental information for remote takeover;

[0030] e) Full-view takeover environment: The system provides the safety officer with a full-view takeover environment through video collage and laser point cloud technology, enhancing the safety officer's perception of the vehicle's traffic environment and improving the accuracy and efficiency of remote takeover;

[0031] f) Real-time driving traffic information: By using laser point cloud to identify the contours and three-dimensional dimensions of surrounding target objects, the system can provide the safety officer with real-time driving traffic information centered on the target vehicle, enhancing the safety officer's control over the vehicle's motion status;

[0032] g) Super-perceptual fully simulated driving experience: The physical twin module reproduces the vehicle positioning coordinates, speed, acceleration and yaw angle operation information collected by the IMU sensor through a six-degree-of-freedom driving simulation driving seat, and combines the digital twin super-perceptual driving environment to provide the safety officer with a super-perceptual fully simulated digital and physical twin driving environment;

[0033] h) Effective remote control: The decision-making layer can achieve effective remote control of the controlled vehicle through the control instructions issued by the cabin-side controller of the simulated driving component, so that the simulated cockpit can follow the posture of the controlled vehicle, improving the practical operability of remote takeover.

[0034] In summary, the present invention not only improves the efficiency and safety of remote takeover, but also provides a strong guarantee for the operation safety of autonomous driving vehicles, and has important practical value and broad prospects.

[0035] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A remote control simulation driving system based on digital physical twins, characterized by: It includes a perception layer, a twin layer, and a decision-making execution layer; the perception layer includes vehicle perception based on intelligent connected vehicles for autonomous driving and environmental perception based on roadside units, which are used to collect vehicle operation data and traffic environment data, and use the collected data to build a twin layer; it includes digital twins and physical twins, and uses the multimodal data transmitted by the behavior layer to build a super-perceptual simulated driving environment, realizing the digital twin of the vehicle's traffic environment and the physical twin dual modeling of the vehicle's motion behavior; after the decision-making execution layer determines that the vehicle is in a takeover state, it first records the control command issued by the control command input device, and then transmits the vehicle control command to the controlled vehicle in the behavior layer through the 5G mobile network; the controlled vehicle executes the operation according to the vehicle control command, so that the simulated cockpit can achieve posture following with the controlled vehicle.

2. A remote control simulation driving system based on digital physical twins according to claim 1, characterized in that: The perception layer includes two modules: vehicle perception based on intelligent connected vehicles and environmental perception based on roadside units. The vehicle perception part captures surrounding video information through multi-view panoramic cameras around the vehicle, uses lidar to detect the surrounding three-dimensional scenes, and collects vehicle positioning coordinates, speed, acceleration and yaw angle operation data through an inertial measurement unit.

3. A remote control simulation driving system based on digital physical twins according to claim 2, characterized in that: Traffic environment perception uses roadside units such as cameras and traffic sensors to obtain traffic environment data during vehicle driving, including road building information model, lane line detection, traffic light recognition, traffic sign recognition, pedestrian detection, non-motor vehicle detection, motor vehicle detection, obstacle recognition and vehicle positioning, etc. In the vehicle coordinate system, a unified unit information with time-stamped data association and fusion is established.

4. A remote-controlled simulated driving system based on digital-physical twins according to claim 1, characterized in that: The twin layer includes two modules: digital twin and physical twin. The digital twin module has two goals: to use the historical information of the roadside unit to reproduce the meso-urban road-level driving environment, to generate partial road background image models based on the historical road video data collected by each roadside unit, and to extract background pixels from the historical road image data to build an environmental base for micro-simulation; to use the target vehicle laser point cloud data and 3D Gaussian fuzzy modeling technology to reconstruct the single-vehicle-level micro-road three-dimensional scene, to identify the contours and three-dimensional dimensions of the surrounding target objects through the laser point cloud, to obtain a complete three-dimensional spatial dynamic structure diagram of the complex road scene, and to provide real-time road condition information centered on the target vehicle on the head-up display in front of the driver.

5. A remote control simulation driving system based on digital physical twins according to claim 4, characterized in that: The physical twin module has two goals: by collage of panoramic camera video information around the vehicle, the panoramic view centered on the target vehicle is reproduced on a circular panoramic display screen with the simulated driving console as the center, providing a full-view takeover environment for the safety officer; by reproducing the IMU sensor through the six-degree-of-freedom driving simulation driving seat to collect vehicle positioning coordinates, speed, acceleration and yaw angle operation information; through the collaborative combination of digital twin and physical twin modules, a remote driving console is jointly built to provide the safety officer with a super-perceptive fully simulated digital-physical twin driving environment.

6. A remote-controlled simulated driving system based on digital-physical twins according to claim 1, characterized in that: The decision-making execution layer includes the cloud-based decision-making layer and the road-side execution layer; after the safety officer at the cloud-based decision-making layer makes a driving judgment based on the information provided by the twin layer, the decision-making layer transmits the control instructions through the cabin-side controller of the simulated driving components such as the steering wheel and pedals; when it is determined that the controlled vehicle is in a takeover state, the decision-making layer records the vehicle control commands generated by the simulated driving components, and sends the vehicle control commands to the controlled vehicle through the 5G mobile network, so that the road-side execution layer re-enables the controlled vehicle to drive according to the vehicle control commands, so that the simulated cockpit can achieve posture following with the controlled vehicle.

Images