Real-time multi-scene automatic driving system based on AI multi-sensor fusion

Abstract

The invention provides a real-time multi-scene automatic driving system based on AI multi-sensor fusion. The system can realize multi-scene seamless high-precision real-time automatic driving of the automatic driving vehicle. Adopted key technologies comprise a multi-source heterogeneous fusion automatic driving technology based on AI assistance and an AI enabling enhanced decision execution mechanism. Aiming at the defects of the current automatic driving technology in the aspect of vehicle full situation awareness, the technology improves the multi-source fusion efficiency and accuracy through deep learning, directly maps from multi-sensor original data to a driving decision through an end-to-end learning model to improve the reaction speed, and deeply fuses multiple sensors from an information layer. Meanwhile, aiming at the contradiction between big data processing and real-time computing requirements under the current automatic driving vehicle-machine man-cloud interconnection, the edge computing technology, cloud processing and local computing are used for assistance, so that the power consumption is reduced and the real-time requirements are met. Furthermore, through access integration of an intelligent traffic system, deep combination of traffic big data and AI automatic driving is realized, and relative positioning between vehicles is realized through traffic big data interconnection characteristics. The purposes of responding to complex road conditions in real time and reducing the power consumption of a processing algorithm are achieved through information characteristics such as the vehicle machine, the man-machine and the accurate position between the vehicle machine and the specific marker. In a real-time multi-scene automatic driving system based on AI multi-sensor fusion, the generalization ability of the system is continuously enhanced, and the applicability and transportability of the system are concrete so as to adapt to wider driving scenes.

Description

Technical Field

[0001] The present invention relates to a real-time multi-scenario autonomous driving technology based on AI multi-sensor fusion, and in particular to an AI-assisted multi-source heterogeneous fusion autonomous driving technology. Background Art

[0002] The development of autonomous driving technology has gradually evolved from the initial automated assisted driving system to today's fully autonomous driving system. Early systems mainly relied on single sensor input, such as radar or camera, to detect the environment around the vehicle. As the technology developed, more types of sensors were integrated into the system, including laser radar (LiDAR), ultrasonic sensors and infrared cameras. These sensors have their own advantages and limitations. For example, cameras perform well in good lighting conditions, while radars can work stably in bad weather such as fog and rain.

[0003] Although advances in sensor technology have provided autonomous vehicles with richer environmental information, how to effectively fuse data from different sensors remains a challenge. Current multi-sensor fusion techniques mainly rely on classic data fusion frameworks, such as weighted fusion, Kalman filter, and particle filter methods. These methods perform well in dealing with scenarios with low dynamic complexity, but their efficiency and accuracy still need to be improved in highly dynamic and changeable driving environments, such as urban traffic and complex intersections. In addition, existing autonomous driving systems face challenges in real-time performance. Although fast data processing is essential for autonomous driving, in practical applications, it is often difficult to achieve ideal real-time response, especially when processing large amounts of data from multiple sensors. This is mainly due to the large amount and diversity of sensor data, as well as the limitations of computing resources.

[0004] The introduction of AI technology provides new possibilities for solving these problems. By utilizing advanced machine learning algorithms such as deep learning, a variety of sensor data can be processed and interpreted more efficiently. For example, deep learning models can learn useful feature representations from complex sensor data, improve the accuracy of data fusion and the decision-making ability of the system. However, these models usually require a lot of computing resources, and the training process is complex and time-consuming, which poses a limitation in real-time autonomous driving applications. Autonomous driving has gradually developed into an inevitable foundation for vehicle autonomous navigation, full situational awareness, and intelligent decision-making for safe driving under various sensors, computer vision, and deep learning technologies. AI-enabled autonomous driving has become a major trend.

[0005] Therefore, it is urgent to invent a new AI-based multi-sensor fusion real-time multi-scenario autonomous driving system. This system needs to be able to efficiently and accurately process and fuse multiple sensor data in a variety of environments and complex scenarios, while ensuring high real-time performance and reliability to adapt to the increasingly complex driving environment. Through technological innovation, this system will greatly improve the intelligence level and safety of autonomous driving vehicles, laying a solid foundation for the future development of autonomous driving technology. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a real-time multi-scenario autonomous driving system based on AI multi-sensor fusion, which is suitable for AI-enabled autonomous driving to achieve real-time seamless autonomous intelligent navigation for autonomous driving.

[0007] In view of the shortcomings of current autonomous driving technology in terms of vehicle full situational awareness, such as the situational awareness "gap" problem caused by the differences in spatiotemporal characteristics of multi-source heterogeneous data in improving the accuracy and reliability of environmental perception, deep learning is used to improve the efficiency and accuracy of multi-source fusion. An end-to-end learning model is used to directly map from multi-sensor raw data to driving decisions to improve the response speed, and at the same time, multi-sensors are deeply integrated from the information layer. The use of advanced fusion algorithms enabled by AI can enable autonomous driving to solve the previous performance problems in urban canyons and extremely severe weather conditions. Multi-sensors assisted by AI are more vital, have adaptive self-learning characteristics for faults and low-reliability environments, more accurately predict behavior and assess risks, and enhance the predictability and responsiveness of the entire system. The human-computer interaction system under AI technology enables natural language processing NLP and augmented reality AR technology to better provide passengers with decision-making and visual voice feedback. At the same time, in view of the contradiction between big data processing and real-time computing requirements under the current autonomous driving vehicle-machine-human-cloud interconnection, edge computing technology and cloud processing and local computing are used to assist in reducing power consumption and real-time requirements. Furthermore, through the integration of intelligent transportation systems, we can achieve a deep combination of traffic big data and AI autonomous driving, and at the same time estimate ethical and legal issues. Based on high-precision maps, we can use the interconnected characteristics of traffic big data to achieve relative positioning between vehicles, precise positions between vehicle computers, human computers, and vehicle computers and specific markers, and other information characteristics to achieve real-time response to complex road conditions and reduce the power consumption of processing algorithms. In the real-time multi-scenario autonomous driving system based on AI multi-sensor fusion, we will continuously enhance the generalization ability of the system, and make it more applicable and portable to adapt to a wider range of driving scenarios.

[0008] The technical solution of the present invention is a real-time multi-scenario autonomous driving system based on AI multi-sensor fusion, which includes the following steps:

[0009] Step S1, realizing AI-assisted multi-sensor multi-source heterogeneous space-time adaptive fusion mechanism, specifically includes the following sub-steps:

[0010] Step S11, synchronize the hardware and software timestamps of the vehicle-mounted multi-sensors GNSS (satellite navigation system), INS (inertial navigation system), WS (odometer), Image (image vision including but not limited to SLAM), and Lidar (including millimeter wave radar), calibrate and compensate for the asynchronous ones, and achieve adaptive time synchronization.

[0011] Step S12, adaptively synchronize the vehicle-mounted GNSS, INS, WS, Image, and Lidar sensors in space, establish error equations for joint calibration and motion compensation, and periodically calibrate the multiple sensors in the subsequent form of the vehicle.

[0012] Step S13, connect the 5G network, high-precision map and smart traffic related positioning and attitude information to achieve high-reliability and high-precision full situational awareness, and use AI cloud data to coordinate and assist local multi-sensor multi-source heterogeneous fusion to achieve full real-time seamless autonomous driving.

[0013] Step S14, using mainstream deep learning (including but not limited to MVO-CNN-CONVLSTM-ATTENTION) to assist GNSS / INS / WS / Image / Lidar / 5G / MM cascade Kalman adaptive filtering multi-source fusion to perceive the vehicle's high-precision position and attitude information in real time. At the same time, cluster data optimization can be achieved between autonomous driving vehicles, and information such as the relative positions between vehicles can be used to achieve autonomous driving higher than level L3.

[0014] Step S2, based on the high-precision vehicle full-situation data obtained in step 1, upload the real-time information to the AI ​​spatiotemporal big data cloud, combine cloud processing and edge computing to perform data reintegration with the AI ​​autonomous driving database AIACDS, and realize AI autonomous driving above level L3 and higher-level cloud-based smart libraries.

[0015] Step S21: After the high-precision position and posture obtained by multi-source fusion and richer multi-sensor original information are uploaded to the cloud, the multi-sensor data are re-integrated using powerful cloud computing and edge computing methods, and seamless data processing is achieved through deep learning. The more accurate position and posture results and predicted values ​​are returned to the vehicle system to iteratively replace the previous processing results, achieving long-range driving that is higher than L3 AI autonomous driving.

[0016] Step S22, establish a smart cloud database SCD, capture and track different vehicles and corresponding high-precision landmark location information through the smart cloud database, realize cloud screening and push of autonomous driving vehicles, and return auxiliary information to the vehicle system, thus forming a virtuous circle.

[0017] Step S3, implement the decision-making execution mechanism enhanced by AI empowerment.

[0018] Step S31, using the information from S1 and S2, the vehicle situation awareness is perceived by the vehicle machine AR four-dimensional real-time delivery and tracking warning, and a decision plan is provided for selection based on the evaluation criteria such as the optimal path and the minimum risk.

[0019] Step S32: After a specific plan is selected manually or the AI ​​selects the optimal plan, the execution feedback time is monitored in real time and the data is uploaded to achieve seamless AI autonomous driving higher than level L3.

[0020] Furthermore, in step S11, the adaptive time synchronization is achieved by performing hardware and software adaptive time synchronization on multiple sensors. The unified reference time of GNSS time is used to provide a time reference for each sensor, and each sensor adds timestamp information to the data collected independently according to the calibrated time, thereby achieving time synchronization.

[0021] The hardware and software adaptive time synchronization uses GNSS time as the reference time and performs data alignment for sensors with inconsistent frequency and time references through interpolation and extrapolation combined with least squares. At the same time, cloud data collection and deep learning are used to learn and predict the alignment timestamps of vehicle hardware.

[0022] In step S12, adaptive spatial synchronization is performed because the vehicle moves during spatial synchronization, so the displacement change must be compensated, and the relative displacement such as the camera must be compensated at the same time.

[0023] ΔS=Δt*v

[0024] Among them, the calculation model for speed V generally adopts a high-order polynomial to adapt to the requirements of different sensors.

[0025] In the joint calibration, the whole vehicle calibration is centered on the INS, and single sensor and inter-sensor calibration are performed. Each sensor calibration has its own error equation, and the manufacturer will also provide calibration software. The joint calibration is centered on the INS and needs to be set by yourself. The calibration formulas for each sensor and INS are as follows:

[0026]

[0027] Among them, GNSS rotates according to the coordinate axis in line with INS to compensate for the displacement, and when it is converted to the vehicle display, it continues to compensate for the displacement between the center of the vehicle body and INS.

[0028] Furthermore, in step S13, the GNSS / INS / WS / Image / Lidar / 5G / MM cascade adaptive Kalman filter fusion mechanism adopts the federal Kalman filter, and the system state equation is:

[0029]

[0030] The state variables have 15 to 21 dimensions depending on the computing power:

[0031]

[0032] In the formula, the right side from left to right is the misalignment angle, velocity error, position error, gyro bias, accelerometer bias, and gyro and accelerometer proportional factors can be further added.

[0033] In step S14, deep learning is trained, learned and predicted based on multi-sensor information. When the training accuracy and precision meet the requirements, multi-source fusion is performed and can be used as one of the error detection methods. That is, the deep learning result can be used as a sensor.

[0034] Further, in step S21, the specific operations are: 1) the multi-source heterogeneous fusion results assisted by deep learning are uploaded to the cloud as the basis of vehicle situation awareness, including the original data of each sensor; 2) the data is reprocessed and recalculated by combining the original data with the powerful cloud computing, and the sliding window accuracy is further improved based on the previous local processing results, that is, the segmented continuous iterative processing enables it to process the current moment result on the previous higher precision data set; 3) post-processing is used in combination with the real-time data source to further seamlessly process the missing or unqualified data through deep learning for unqualified data values, so as to obtain high-precision full-cycle continuously available vehicle situation awareness results and feedback to the vehicle computer, reduce the burden on the vehicle computer and provide guarantee for the vehicle computer to obtain high-precision results in real-time processing, and achieve long-term endurance maintenance of AI autonomous driving higher than L3 level.

[0035] In step S22, the specific implementation is as follows: 1) Establishing a smart cloud database SCD through the historical data of each vehicle machine and the smart transportation system; 2) Rapidly matching and searching related vehicle machines through the smart cloud database according to the position of the vehicle machine to capture and track to form a phased vehicle group and realize cloud screening and push of autonomous driving vehicles; 3) Optimizing and ensuring the relative position and absolute position of the vehicle machine according to the high-precision map, auxiliary road sign information, etc. under the relevant vehicle machines and the smart transportation system; 4) Using the currently open source deep learning large model, a friendly mechanism of continuous virtuous cycle feedback between the vehicle machine and the cloud is formed.

[0036] In step S31, the basic information of vehicle situation awareness and the intelligent cloud database are used to greatly reduce the difficulty of local processing. The cloud-based large model processing is used to perform real-time vehicle-machine AR four-dimensional delivery and tracking warning broadcasts, providing users with optional decision-making plans based on multiple conditions such as optimal path, safety, multi-modal traffic, road conditions, etc. under different weights, and showing the concentrated advantages of each plan to meet different driving purposes and emergency driving needs.

[0037] Furthermore, in step S32, 1) after manually selecting the required plan, the driving preference is recorded according to the selected plan. At the same time, the driver can also manually add or delete driving preferences to improve the fit of the plan decision, and the execution feedback time is monitored in real time and the data is uploaded; 2) in the absence of human intervention, according to the current driving purpose, the vehicle system automatically selects the optimal plan, and monitors the execution feedback time in real time and uploads the data to evaluate the performance of the whole vehicle to cope with emergency risks. In particular, in emergency risk avoidance and other situations, the AI ​​autonomous driving system will actively select and select the best risk avoidance plan based on the long-term vehicle performance evaluation, and execute the plan decision based on the cloud big data scenario for the multi-situation risk avoidance basic simulation under the current performance of the autonomous driving vehicle.

[0038] The beneficial effects of the present invention are as follows: a multi-sensor fusion real-time multi-scenario autonomous driving system based on AI in the present invention has many beneficial effects and greatly promotes the application and development of autonomous driving technology. First, the system significantly improves the accuracy and reliability of environmental perception through advanced multi-sensor fusion technology. Traditional autonomous driving systems often rely on a single type of sensor and are easily limited by the limitations of a single sensor, such as insufficient performance of cameras under low light conditions, or data interference problems of radars in complex urban environments. The present invention integrates multiple sensors such as cameras, radars, and LiDARs, and uses deep learning algorithms to efficiently fuse and process data, so that the vehicle can maintain a high degree of environmental perception under various climate and lighting conditions, thereby greatly improving driving safety. Secondly, the real-time AI driving decision engine of the present invention can make driving decisions quickly and accurately. When receiving the fused sensor data, the AI ​​engine uses advanced algorithms to immediately evaluate the road conditions and possible risks and make corresponding driving operation decisions. This high-speed data processing and decision-making capability is difficult to match with traditional autonomous driving systems. It allows autonomous driving vehicles to respond to emergencies more effectively, such as emergency avoidance of pedestrians or other vehicles, greatly enhancing driving safety. In addition, the adaptive learning and optimization mechanism of the present system enables autonomous driving vehicles to continuously improve their performance. Through real-time learning and data feedback, the AI ​​engine can adjust and optimize driving strategies to adapt to changing road and traffic conditions. This self-optimization capability means that the system becomes more accurate and efficient as it is used. Safety and redundancy design are also important features of the present invention. Through multiple backup and fault switching mechanisms, the system can maintain basic operating functions and ensure the safety of vehicles and passengers even when key components fail. This design greatly improves the reliability of the system and the trust of passengers. Finally, the present invention has a high degree of practicality and broad application prospects. The modular design of the system allows it to be easily integrated into various models and platforms. Both commercial vehicles and private cars can benefit from this advanced autonomous driving technology. This not only promotes the commercialization of autonomous driving technology, but also lays the foundation for the innovation of future transportation systems.

[0039] In summary, the present invention, through its advanced technology and design, not only improves the safety and reliability of autonomous driving, but also enhances the intelligence level of the vehicle and user experience, demonstrating the broad development potential of autonomous driving in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the adaptive time synchronization principle of an embodiment of the present invention.

[0041] Figure 2 It is a schematic diagram of the adaptive spatial synchronization principle of an embodiment of the present invention.

[0042] Figure 3 It is a flowchart of the AI ​​autonomous driving multi-source fusion algorithm according to an embodiment of the present invention.

[0043] Figure 4 Schematic diagram of deep learning according to an embodiment of the present invention.

[0044] Figure 5 It is a schematic diagram of the AI-enabled enhanced decision-making execution mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0046] The method and technology of the present invention involves installing a portable AI autonomous driving system in an autonomous driving vehicle, which includes hardware products and software algorithms, and can also only use the software algorithm part to calculate the sensor results. The system includes multiple millimeter wave radar sensors, GNSS receivers, Lidar, odometer WS and IMU and other sensor devices, which are integrated in a lightweight MEMS board. Various types of sensors are installed on the autonomous driving vehicle, including cameras, radars, LiDAR, GPS and inertial measurement units (IMU). The selection and arrangement of each sensor should be optimized to cover the entire environment around the vehicle to ensure that there are no blind spots. Advanced data fusion algorithms, adaptive cascade Kalman filters, and deep learning-based fusion technology are used to integrate and process data from different sensors in real time. These algorithms should be able to effectively handle data heterogeneity and timing differences from different sensors. For the implementation of the AI ​​driving decision engine, a decision model based on machine learning, a deep neural network and a large learning model are developed to process the fused sensor data and make driving decisions. Model training should use a large amount of driving data, including different traffic scenarios, weather conditions and road types. At the same time, the AI ​​model is optimized in real time to reduce decision delays and ensure that the vehicle can respond quickly in a complex traffic environment. There is no need to deploy dedicated hardware accelerators on the vehicle to handle AI calculations. All calculations are performed in the cloud processing center, reducing costs and power consumption. Its online learning system allows the AI ​​driving decision engine to self-adjust and optimize based on real-time driving data and user feedback. This includes dynamic adjustment of algorithms to improve the accuracy and adaptability of decisions. AI models and algorithms are regularly updated to incorporate new driving data and the latest research results to ensure that the system continues to adapt to new driving environments and regulatory changes.

[0047] At the same time, the present invention is based on the AI ​​automatic driving system, and the decision algorithm part of the core challenges and difficulties in automatic driving is systematically optimized. The decision layer is an extremely difficult part of the automatic driving system, which requires an efficient AI model and a large amount of training data. The decision layer refers to making decisions and judgments based on the perceived road information, traffic information, and vehicle information, selecting appropriate working models and formulating corresponding control strategies, and issuing instructions to the underlying control execution module. In scenarios such as lane keeping, lane departure warning, vehicle distance keeping, and obstacle warning, the decision system needs to predict the state of the vehicle and other vehicles, lanes, pedestrians, etc. encountered in the future, and different people have different driving strategies for different situations. Therefore, the optimization of the decision algorithm requires efficient artificial intelligence models and a large amount of training data to cover a variety of long-tail scenarios. The decision system of automatic driving mainly includes two parts: behavioral decision-making and path planning. The behavioral decision system models the traffic participants involved in the vehicle and predicts their behavior, and then summarizes the information from all parties and submits it to the path planning system for decision-making. The decision system can generally be divided into three layers according to the hierarchy: cognitive layer, task layer, and planning layer, which are responsible for driving scenario cognition, driving behavior decision-making, local path planning, and ultimately determine the optimal path of the vehicle form. Ultimately, the car computer and the smart cloud database complement each other and jointly promote a virtuous cycle.

[0048] The technical solution of the present invention to solve the above technical problems is as follows: A real-time multi-scenario autonomous driving system based on AI multi-sensor fusion, comprising the following steps:

[0049] Step S1, implement AI-assisted multi-sensor multi-source heterogeneous space-time adaptive fusion mechanism. First, the hardware and software of each sensor are synchronized in space-time adaptively, and then the time position is aligned through joint calibration. In the unified time coordinate reference framework, multi-source heterogeneous data fusion is performed, and deep learning is used as a "sensor" for fusion through the deep learning algorithm under AI.

[0050] Step S11, synchronize the hardware and software timestamps of the vehicle-mounted multi-sensors GNSS (satellite navigation system), INS (inertial navigation system), WS (odometer), Image (image vision including but not limited to SLAM), and Lidar (including millimeter wave radar), calibrate and compensate for the asynchronous ones, and achieve adaptive time synchronization.

[0051] Adaptive time synchronization is achieved by hardware and software adaptive time synchronization of multiple sensors. The unified reference time of GNSS time is used to provide a time reference for each sensor. Each sensor adds timestamp information to the data collected independently according to the calibrated time, thereby achieving time synchronization.

[0052] Among them, hardware and software adaptive time synchronization uses GNSS time as the reference time and performs data alignment for sensors with inconsistent frequency and time references through interpolation and extrapolation combined with least squares. At the same time, cloud data collection and deep learning are used to predict and align the timestamps of vehicle hardware learning. The adaptive time synchronization solution is as follows: Figure 1 .

[0053] Step S12, adaptively synchronize the vehicle-mounted GNSS, INS, WS, Image, and Lidar sensors in space, establish error equations for joint calibration and motion compensation, and periodically calibrate the multiple sensors in the subsequent form of the vehicle.

[0054] Adaptive Spatial Synchronization Since the vehicle moves in spatial synchronization, it is necessary to compensate for the displacement changes and at the same time compensate for the relative displacement such as the camera.

[0055] ΔS=Δt*v

[0056] Among them, the calculation model of speed V generally adopts high-order polynomials to adapt to the requirements of different sensors. Adaptive spatial synchronization schemes such as Figure 2 .

[0057] In the joint calibration, the whole vehicle calibration is centered on the INS, and single sensor and inter-sensor calibration are performed. Each sensor calibration has its own error equation, and the manufacturer will also provide calibration software. The joint calibration is centered on the INS and needs to be set by yourself. The calibration formulas for each sensor and INS are as follows:

[0058]

[0059] Among them, GNSS rotates according to the coordinate axis in line with INS to compensate for the displacement, and when it is converted to the vehicle display, it continues to compensate for the displacement between the center of the vehicle body and INS.

[0060] Furthermore, in step S13, the GNSS / INS / WS / Image / Lidar / 5G / MM cascade adaptive Kalman filter fusion mechanism adopts the federal Kalman filter, and the system state equation is:

[0061]

[0062] The state variables have 15 to 21 dimensions depending on the computing power:

[0063]

[0064] In the formula, the right side from left to right is the misalignment angle, velocity error, position error, gyro bias, accelerometer bias, and gyro and accelerometer proportional factors can be further added.

[0065] Step S13: Connect 5G network, high-precision map and smart traffic related positioning information to achieve high reliability and high precision full situational awareness, and use AI cloud data to assist local multi-sensor multi-source heterogeneous fusion to achieve full real-time seamless autonomous driving. Figure 3 shown.

[0066] Step S14, using mainstream deep learning (including but not limited to MVO-CNN-CONVLSTM-ATTENTION) to assist GNSS / INS / WS / Image / Lidar / 5G / MM cascade Kalman adaptive filtering multi-source fusion to perceive the vehicle's high-precision position and attitude information in real time. At the same time, cluster data optimization can be achieved between autonomous driving vehicles, and information such as the relative positions between vehicles can be used to achieve autonomous driving higher than level L3.

[0067] Deep learning is trained and predicted based on multi-sensor information. When the training accuracy and precision meet the requirements, multi-source fusion is performed and can be used as one of the error detection methods. In other words, the deep learning results can be used as a sensor. The corresponding deep learning algorithms are as follows: Figure 4 shown.

[0068] Step S21: After the high-precision position and posture obtained by multi-source fusion and richer multi-sensor original information are uploaded to the cloud, the multi-sensor data are re-integrated using powerful cloud computing and edge computing methods, and seamless data processing is achieved through deep learning. The more accurate position and posture results and predicted values ​​are returned to the vehicle system to iteratively replace the previous processing results, achieving long-range driving that is higher than L3 AI autonomous driving.

[0069] The specific operations are as follows: 1) The multi-source heterogeneous fusion results assisted by deep learning are uploaded to the cloud as the basis for vehicle situational awareness, including the original data of each sensor; 2) The data is reprocessed and recalculated through the original data combined with powerful cloud computing, and the sliding window accuracy is further improved based on the previous local processing results, that is, segmented continuous iterative processing enables it to process the current moment results on the previous higher-precision data set; 3) Post-processing is used in combination with real-time data sources to further seamlessly process missing or unqualified data through deep learning for unqualified data values, thereby obtaining high-precision, full-cycle continuously available vehicle situational awareness results and feeding back to the vehicle computer, reducing the burden on the vehicle computer and providing guarantees for the vehicle computer to obtain high-precision results in real-time processing, achieving long-range AI autonomous driving higher than L3 level.

[0070] Step S22, establish a smart cloud database SCD, capture and track different vehicles and corresponding high-precision landmark location information through the smart cloud database, realize cloud screening and push of autonomous driving vehicles, and return auxiliary information to the vehicle system, thus forming a virtuous circle.

[0071] The specific implementation is as follows: 1) Establish a smart cloud database SCD through the historical data of each vehicle computer and the smart transportation system; 2) According to the location of the vehicle computer, quickly match and search for related vehicle computers through the smart cloud database, capture and track to form a phased vehicle group and realize cloud screening and push of autonomous driving vehicles; 3) Optimize and ensure the relative position and absolute position of the vehicle computer based on the high-precision map, auxiliary road sign information, etc. under the relevant vehicle computers and smart transportation systems; 4) Use the currently open source deep learning large model to form a friendly mechanism for continuous virtuous cycle feedback between the vehicle computer and the cloud.

[0072] Step S3, implement the decision-making execution mechanism enhanced by AI empowerment, such as Figure 5 shown.

[0073] Step S31, using the information from S1 and S2, the vehicle situation awareness is perceived by the vehicle machine AR four-dimensional real-time delivery and tracking warning, and a decision plan is provided for selection based on the evaluation criteria such as the optimal path and the minimum risk.

[0074] By utilizing basic vehicle situation awareness information and intelligent cloud database, the difficulty of local processing is greatly reduced. The cloud-based large model processing is used to perform real-time delivery of vehicle-mounted AR four-dimensional and tracking warning broadcasts, providing users with optional decision-making plans based on multiple conditions such as optimal path, safety, multi-modal traffic, road conditions, etc. under different weights, and showing the concentrated advantages of each plan to meet different driving purposes and emergency driving needs.

[0075] Step S32: After a specific plan is selected manually or the AI ​​selects the optimal plan, the execution feedback time is monitored in real time and the data is uploaded to achieve seamless AI autonomous driving higher than level L3.

[0076] 1) After manually selecting the required plan, the driving preference is recorded according to the selected plan. The driver can also manually add or delete driving preferences to improve the fit of the plan decision, and the execution feedback time is monitored in real time and the data is uploaded; 2) In the absence of human intervention, the vehicle system automatically selects the optimal plan based on the current driving purpose, and monitors the execution feedback time in real time and uploads the data to evaluate the performance of the entire vehicle to cope with emergency risks. In particular, in emergency avoidance and other situations, the AI ​​autonomous driving system will actively select and select the best avoidance plan based on the long-term vehicle performance evaluation, and execute the plan decision based on the cloud big data scenario for the multi-situation avoidance basic simulation under the current performance of the autonomous driving vehicle.

[0077] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A real-time multi-scenario autonomous driving system based on AI multi-sensor fusion, characterized in that: The following steps are involved: Step 1, implement AI-assisted multi-sensor multi-source heterogeneous space-time adaptive fusion mechanism. Perform hardware and software timestamp synchronization on the vehicle-mounted multi-sensor GNSS (satellite navigation system), INS (inertial navigation system), WS (odometer), Image (image vision including but not limited to SLAM), Lidar (including millimeter wave radar), calibrate and compensate for the asynchronous ones, and realize adaptive time synchronization. Further adaptive spatial synchronization of the vehicle-mounted GNSS, INS, WS, Image, Lidar multi-sensors, establish error equations for joint calibration and motion compensation, and perform periodic calibration and calibration of multi-sensors in the subsequent form of the vehicle. Connect 5G network, high-precision map and smart traffic related positioning and attitude information to achieve high reliability and high precision full situational awareness, and use AI cloud data to assist local multi-sensor multi-source heterogeneous fusion to achieve full real-time seamless autonomous driving. Mainstream deep learning is used to assist GNSS / INS / WS / Image / Lidar / 5G / MM cascade Kalman adaptive filtering multi-source fusion to perceive the vehicle's high-precision position and attitude information in real time. At the same time, cluster data optimization can be achieved between autonomous driving vehicles, and information such as the relative position between vehicles can be used to achieve autonomous driving higher than L3. Step 2: Based on the high-precision vehicle full-situation data obtained in step 1, upload the real-time information to the AI ​​spatiotemporal big data cloud, and combine cloud processing and edge computing to perform data re-integration with the AI ​​autonomous driving database AIACDS to achieve AI autonomous driving above level L3 and higher-level cloud intelligence libraries. After the high-precision position and posture obtained by multi-source fusion and richer multi-sensor original information are uploaded to the cloud, the multi-sensor data is re-integrated using powerful cloud computing and edge computing methods, and seamless data processing is achieved through deep learning. The more accurate position and posture results and predicted values ​​are returned to the vehicle system to iteratively replace the previous processing results, achieving long-range AI autonomous driving higher than level L3. At the same time, a smart cloud database SCD is established, and different vehicles and corresponding high-precision landmark location information are captured and tracked through the smart cloud database, cloud screening and push of autonomous driving vehicles are achieved, and auxiliary information is returned to the vehicle system, thus forming a virtuous circle. Step 3: Implement an AI-enabled enhanced decision-making execution mechanism. Using the information from S1 and S2, the vehicle situation awareness is realized through real-time delivery of vehicle-mounted AR four-dimensional images and tracking warnings, and decision-making plans are provided for selection based on the optimal path and minimum risk criteria. The basic information of vehicle situation awareness and the intelligent cloud database greatly reduce the difficulty of local processing, and the vehicle-mounted AR four-dimensional images are delivered in real time and tracking warnings are broadcasted at the lower end of the cloud large model processing. Optional decision plans are provided for users based on multiple conditions such as optimal path, safety, multi-modal traffic, road conditions, etc. under different weights, and the concentrated advantages of each plan are displayed to meet different driving purposes and emergency driving needs. After humans select a specific plan or AI selects the optimal plan, the execution feedback time is monitored in real time and the data is uploaded to achieve seamless AI autonomous driving above level L3. 1) After manually selecting the required plan, the driving preference is recorded according to the selected plan. The driver can also manually add or delete driving preferences to improve the fit of the plan decision, and the execution feedback time is monitored in real time and the data is uploaded; 2) In the absence of human intervention, the vehicle system automatically selects the optimal plan based on the current driving purpose, and monitors the execution feedback time in real time and uploads the data to evaluate the performance of the entire vehicle to cope with emergency risks. In particular, in emergency avoidance and other situations, the AI ​​autonomous driving system will actively select and select the best avoidance plan based on the long-term vehicle performance evaluation, and execute the plan decision based on the cloud big data scenario for the multi-situation avoidance basic simulation under the current performance of the autonomous driving vehicle.

Images