End-to-end automatic driving multi-modal data fusion method and system

Abstract

The invention relates to the technical field of automatic driving, and discloses an end-to-end automatic driving multi-modal data fusion method and system, and the method comprises the steps: obtaining multiple pieces of sensor data, GPS data and vehicle body data of an automatic driving vehicle, and the multiple pieces of sensor data comprise laser radar data, camera data and millimeter wave radar data; performing pre-fusion processing on the laser radar data and the camera data to obtain a pre-fusion result; and performing post-fusion processing on the pre-fusion result, the millimeter wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result so as to realize multi-modal data fusion, and further realizing automatic driving decision through a large model technology based on the pre-fusion result and the post-fusion result. According to the invention, efficient fusion of multi-modal data in a complex driving environment is realized, the accuracy and reliability of automatic driving decision making are improved, and the stability and safety of an automatic driving system in a dynamic environment are also improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving multimodal data fusion method and system. Background Art

[0002] Existing autonomous driving technologies usually rely on staged processing, such as perception, positioning, path planning, control and other modules. These modules usually work independently, which can easily lead to information loss and delay, affecting the overall performance of the autonomous driving system. With the development of deep learning large model technology, end-to-end autonomous driving systems have begun to emerge, which can achieve more efficient autonomous driving by directly inputting raw data for driving decisions.

[0003] However, existing end-to-end autonomous driving systems usually rely on separate perception, positioning and control models. There is no overall large model to handle perception, positioning and control modules at the same time. When facing complex driving scenarios, it is easy to fail to provide sufficient environmental information, which in turn affects the accuracy and reliability of driving decisions. In addition, there is still room for improvement in the integration of processing time and space information in existing systems. Simply relying on the sensor's own timestamp to synchronize the system timestamp, such as based on the Network Time Protocol (NTP) and Simultaneous Localization and Mapping (SLAM) technologies to synchronize the system timestamp, will cause a large error when the timestamp information is transmitted from one level to another. Therefore, how to achieve efficient fusion of multimodal data in complex driving environments to further improve the accuracy and reliability of autonomous driving decisions has become an urgent problem to be solved.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide an end-to-end autonomous driving multimodal data fusion method and system, aiming to solve the technical problem that the existing technology cannot achieve efficient fusion of multimodal data in complex driving environments, so as to further improve the accuracy and reliability of autonomous driving decisions.

[0006] To achieve the above object, the present invention provides an end-to-end autonomous driving multimodal data fusion method, the method comprising the following steps:

[0007] Acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, including lidar data, camera data, and millimeter-wave radar data;

[0008] Performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result;

[0009] The front fusion result, the millimeter wave radar data, the GPS data and the vehicle body data are subjected to post-fusion processing to obtain a post-fusion result, so as to realize the fusion of multimodal data.

[0010] Optionally, after the step of acquiring a plurality of sensor data, GPS data and vehicle body data of the autonomous driving vehicle, the step further includes:

[0011] Preprocessing is performed on the plurality of sensor data to obtain the preprocessed plurality of sensor data, wherein the preprocessing includes time synchronization processing and coordinate transformation processing.

[0012] Optionally, the step of performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result includes:

[0013] Perform feature extraction on the laser radar data and the camera data respectively by using a BEVFusion algorithm to obtain a laser radar BEV feature and a camera BEV feature;

[0014] The laser radar BEV feature and the camera BEV feature are fused to obtain a front fusion result.

[0015] Optionally, the step of performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data to obtain a post-fusion result to achieve fusion of multimodal data includes:

[0016] Performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data through an extended Kalman filter algorithm to obtain a post-fusion result to achieve fusion of multimodal data;

[0017] Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is:

[0018] x k =F k *x k-1 +w k

[0019] In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes a state vector corresponding to the front fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data;

[0020] The observation equation corresponding to the extended Kalman filter algorithm is:

[0021] z k =H k *x k +v k ,

[0022] In the formula, z k is the output of the observation model, H k is the observation matrix, v k is the observation noise.

[0023] Optionally, after the step of performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multimodal data, the method further includes:

[0024] Outputting an autonomous driving decision through a trained end-to-end intelligent driving big model according to the pre-fusion result and the post-fusion result;

[0025] Among them, the trained end-to-end intelligent driving model is built based on a convolutional neural network model and a recurrent neural network model. The multiple convolutional layers of the convolutional neural network model are connected to the long short-term memory layer of the recurrent neural network model through a fusion layer, and the fusion layer is used to fuse multimodal data.

[0026] In addition, to achieve the above-mentioned purpose, the present invention also proposes an end-to-end autonomous driving multimodal data fusion system, and the end-to-end autonomous driving multimodal data fusion system includes:

[0027] A data acquisition module is used to acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, wherein the multiple sensor data include lidar data, camera data, and millimeter wave radar data;

[0028] A front fusion module, used for performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result;

[0029] The post-fusion module is used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multi-modal data.

[0030] Optionally, the data acquisition module is further used to preprocess the multiple sensor data to obtain the preprocessed multiple sensor data, and the preprocessing includes time synchronization processing and coordinate transformation processing.

[0031] Optionally, the front fusion module is further used to extract features from the laser radar data and the camera data respectively through a BEVFusion algorithm to obtain a laser radar BEV feature and a camera BEV feature;

[0032] The front fusion module is also used to fuse the laser radar BEV features and the camera BEV features to obtain a front fusion result.

[0033] Optionally, the post-fusion module is further used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data through an extended Kalman filter algorithm to obtain a post-fusion result to achieve fusion of multimodal data;

[0034] Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is:

[0035] x k =F k *x k-1 +w k

[0036] In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes a state vector corresponding to the front fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data;

[0037] The observation equation corresponding to the extended Kalman filter algorithm is:

[0038] z k =H k *x k +v k ,

[0039] In the formula, z k is the output of the observation model, H k is the observation matrix, v k is the observation noise.

[0040] Optionally, the post-fusion module is further used to output an autonomous driving decision through a trained end-to-end intelligent driving big model according to the pre-fusion result and the post-fusion result;

[0041] Among them, the trained end-to-end intelligent driving model is built based on a convolutional neural network model and a recurrent neural network model. The multiple convolutional layers of the convolutional neural network model are connected to the long short-term memory layer of the recurrent neural network model through a fusion layer, and the fusion layer is used to fuse multimodal data.

[0042] In the present invention, multiple sensor data, GPS data and vehicle body data of the autonomous driving vehicle are obtained, and the multiple sensor data include laser radar data, camera data and millimeter wave radar data; the laser radar data and the camera data are subjected to front fusion processing to obtain front fusion results; the front fusion results, the millimeter wave radar data, the GPS data and the vehicle body data are subjected to post fusion processing to obtain post fusion results, so as to realize the fusion of multimodal data. By performing front fusion processing on the laser radar data and camera data of the autonomous driving vehicle, and then performing post fusion processing on the obtained front fusion results and the millimeter wave radar data, GPS data and vehicle body data of the autonomous driving vehicle, efficient fusion of multimodal data in complex driving environments can be realized. Furthermore, based on the front fusion results and the post fusion results, the autonomous driving decision is realized through the large model technology, which improves the accuracy and reliability of the autonomous driving decision, and also improves the stability and safety of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a first embodiment of an end-to-end autonomous driving multimodal data fusion method according to the present invention;

[0044] Figure 2 It is a schematic diagram of the front fusion and the back fusion involved in the first embodiment of the end-to-end autonomous driving multimodal data fusion method of the present invention;

[0045] Figure 3 Schematic diagram of autonomous driving decision-making involved in the first embodiment of the end-to-end autonomous driving multimodal data fusion method of the present invention;

[0046] Figure 4 This is a structural block diagram of the first embodiment of the end-to-end autonomous driving multimodal data fusion system of the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0049] The embodiment of the present invention provides an end-to-end autonomous driving multimodal data fusion method, referring to Figure 1 , Figure 1This is a flow chart of the first embodiment of the end-to-end autonomous driving multimodal data fusion method of the present invention.

[0050] In this embodiment, the end-to-end autonomous driving multimodal data fusion method includes the following steps:

[0051] Step S10: Acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, where the multiple sensor data include lidar data, camera data, and millimeter wave radar data;

[0052] In this embodiment, multimodal data can be understood as a data set containing multiple different types of information, such as multiple sensor data, GPS data, and body data of an autonomous driving vehicle, wherein the multiple sensor data include Light Laser Detection and Ranging (LiDAR) data, camera data, and millimeter wave radar (Radar) data; the body data can be understood as indicator data related to the body required for autonomous driving decisions, such as steering wheel angle, braking speed, throttle speed, etc.

[0053] In a specific implementation, multiple sensor data of the autonomous driving vehicle can also be preprocessed to obtain multiple preprocessed sensor data, wherein the preprocessing includes time synchronization processing and coordinate transformation processing, so as to achieve the alignment of multimodal data in time and space. For example, the Camera data can be normalized, the LiDAR data can be processed for point cloud semantic segmentation (Semantic Segmentation) and instance segmentation (Instance Segmentation), and the Radar data can be denoised, so as to achieve the calibration and cleaning of multimodal data and ensure the temporal and spatial consistency of the data.

[0054] Step S20: performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result;

[0055] In this embodiment, the BEVFusion algorithm can be used to extract features from the LiDAR data and the camera data respectively to obtain the LiDAR BEV features and the camera BEV features, and then the LiDAR BEV features and the camera BEV features are fused to obtain the front fusion result. In a specific implementation, the BEVFusion algorithm can be used to fuse the data from the LiDAR and the Camera into a unified Bird's Eye View (BEV) layer. For example, the point cloud data provided by the LiDAR based on the surrounding environment of the vehicle is projected onto the BEV view, and then the LiDAR feature map is extracted from the BEV view; the images of vehicles, pedestrians, traffic signs, etc. captured by the Camera are mapped to the BEV view through geometric transformation, and then the Camera feature map is extracted from the BEV view. Then, the LiDAR feature map and the Camera feature map are aligned, and the two are fused through a deep learning model (such as a fusion network) to obtain the front fusion result. Furthermore, the fused feature map (i.e., the front fusion result) can be used for target detection, etc., and the detection results are fed back to the autonomous driving system for decision-making and control (for example, automatic braking, steering, etc.).

[0056] Reference Figure 2 , Figure 2 This is a schematic diagram of the front fusion and back fusion involved in the first embodiment of the end-to-end autonomous driving multimodal data fusion method of the present invention. Figure 2 In the figure, the front fusion part is represented by a solid line, and the back fusion part is represented by a dotted line. When performing front fusion, the BEVFusion algorithm can be used to fuse the data from LiDAR and Camera into a unified BEV. The Camera uses the BEV algorithm to generate BEV features, and the LIDAR uses 3D convolution (Convolution, Conv) to extract the BEV features of the LIDAR side, and then couple them with the BEV features of the Camera side. Furthermore, the channel attention mechanism can be used to use the global pooling layer and the 1*1 convolution layer as the weights of LiDAR and Camera in BEVFusion in the front fusion.

[0057] Step S30: performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multimodal data.

[0058] In this example, the result of the pre-fusion may be obtained first, and then matching association (such as ID / confidence matching) may be performed, and then the system state may be updated through an extended Kalman filter (Extended Kalman Filter, EKF) algorithm. Figure 2In the process, the EKF algorithm can be used to perform post-fusion processing on the front fusion results, millimeter-wave radar data, GPS data, and vehicle body data to obtain the post-fusion results to achieve the fusion of multi-modal data.

[0059] Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is:

[0060] x k =F k *x k-1 +w k

[0061] In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes the state vector corresponding to the previous fusion result, millimeter-wave radar data, GPS data, and vehicle body data;

[0062] The observation equation corresponding to the extended Kalman filter algorithm is:

[0063] z k =H k *x k +v k ,

[0064] In the formula, z k is the output of the observation model, H k is the observation matrix, v k is the observation noise.

[0065] Therefore, the pre-fusion results, millimeter-wave radar data, GPS data, and vehicle body data are used as observation inputs, and the state estimation is adjusted through the Kalman gain to finally obtain the fused obstacle position, size, surrounding vehicle position, speed and other information. For example, the state estimation equation and observation equation can be used to smooth and fuse obstacles detected by different modes to obtain information related to the obstacle, which is also convenient for the subsequent use of large model technology to process obstacle information and improve the perception and decision-making capabilities of the autonomous driving system in complex environments.

[0066] It is easy to understand that the output of the large model can be made more accurate after the output of the pre-fusion is processed by the post-fusion EKF algorithm. In the specific implementation, the data that needs to be post-fused can be connected in series with the pre-fusion BevFusion model through the CNN architecture to obtain the final fusion model (Fusion Model). This fusion model can effectively fuse multimodal data to achieve more accurate state estimation, and also help improve the accuracy and reliability of autonomous driving decisions in complex driving environments.

[0067] Reference Figure 3 , Figure 3 This is a schematic diagram of the autonomous driving decision involved in the first embodiment of the end-to-end autonomous driving multimodal data fusion method of the present invention. Figure 3 In the process, the trained end-to-end intelligent driving model can also output the autonomous driving decision, such as acceleration, deceleration, steering, etc., according to the pre-fusion result and the post-fusion result. Further, the decision can be converted into a specific vehicle control instruction and executed by the on-board controller to achieve more efficient autonomous driving. Among them, the trained end-to-end intelligent driving model includes a convolutional neural network model (CNN) and a recurrent neural network (RNN) model, the convolutional neural network model includes several convolutional layers and a pooling layer, the recurrent neural network model includes a long short-term memory (LSTM) layer, and a fusion layer is provided between the convolutional layer and the long short-term memory layer, and the fusion layer is used to fuse data of different modes. Among them, the model architecture and model parameter settings of the trained end-to-end intelligent driving model can refer to the end-to-end intelligent driving model of a certain car company. This embodiment focuses on improving the fusion layer used to fuse multimodal data in the large model. In addition, the specific model training method can be selected according to the actual accuracy requirements, and this embodiment does not limit this.

[0068] In the specific implementation, the acquired GPS data and sensor timestamp information can also be passed to the above-mentioned large model to synchronize the system timestamp and realize the effective integration of time and space information. The large model technology is used to integrate the time and space information in the input data, improve the adaptability and stability of the autonomous driving system in a dynamic environment, and greatly reduce the time error of obstacle transmission. In addition, the location of obstacles can be predicted based on the above-mentioned large model, and the obstacle location information can be updated in real time, so as to output the autonomous driving decision based on the obstacle location information.

[0069] In specific implementations, the system performance can also be tested in a virtual simulation environment to evaluate its performance in different driving scenarios. In addition, the stability and reliability of the system can also be tested in an actual road environment, and optimization and adjustments can be made based on the test results.

[0070] In this embodiment, by integrating multimodal data and designing an efficient end-to-end autonomous driving system, the decision-making accuracy and reliability of the autonomous driving system in complex environments are improved, and by effectively integrating time and space information, the autonomous driving system is helped to output autonomous driving decisions efficiently and in real time. In addition, the large model technology can achieve efficient data processing and driving decision-making in the end-to-end autonomous driving system, reduce computational complexity and resource consumption, and improve the stability and safety of the autonomous driving system.

[0071] In this embodiment, multiple sensor data, GPS data, and vehicle body data of the autonomous driving vehicle are obtained, and the multiple sensor data include laser radar data, camera data, and millimeter wave radar data; the laser radar data and the camera data are subjected to front fusion processing to obtain a front fusion result; the front fusion result, the millimeter wave radar data, the GPS data, and the vehicle body data are subjected to post fusion processing to obtain a post fusion result, so as to realize the fusion of multimodal data. By performing front fusion processing on the laser radar data and camera data of the autonomous driving vehicle, and then performing post fusion processing on the obtained front fusion result and the millimeter wave radar data, GPS data, and vehicle body data of the autonomous driving vehicle, efficient fusion of multimodal data in complex driving environments can be realized. Furthermore, based on the front fusion results and the post fusion results, the autonomous driving decision is realized through the large model technology, which improves the accuracy and reliability of the autonomous driving decision, and also improves the stability and safety of the autonomous driving system.

[0072] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the end-to-end autonomous driving multimodal data fusion system of the present invention.

[0073] like Figure 4 As shown, the end-to-end autonomous driving multimodal data fusion system proposed in the embodiment of the present invention includes:

[0074] A data acquisition module 10 is used to acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, wherein the multiple sensor data include laser radar data, camera data, and millimeter wave radar data;

[0075] A front fusion module 20, used for performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result;

[0076] The post-fusion module 30 is used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multi-modal data.

[0077] Optionally, the data acquisition module 10 is also used to preprocess multiple sensor data of the autonomous driving vehicle to obtain multiple preprocessed sensor data, wherein the preprocessing includes time synchronization processing and coordinate transformation processing.

[0078] Optionally, the front fusion module 20 is further used to extract features from the laser radar data and the camera data respectively through a BEVFusion algorithm to obtain a laser radar BEV feature and a camera BEV feature;

[0079] The front fusion module 20 is also used to fuse the laser radar BEV features and the camera BEV features to obtain a front fusion result.

[0080] Optionally, the post-fusion module 30 is further used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data through an extended Kalman filter algorithm to obtain a post-fusion result to achieve fusion of multi-modal data;

[0081] Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is:

[0082] x k =F k *x k-1 +w k

[0083] In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes a state vector corresponding to the front fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data;

[0084] The observation equation corresponding to the extended Kalman filter algorithm is:

[0085] z k =H k *x k +v k ,

[0086] In the formula, z k is the output of the observation model, H k is the observation matrix, v kis the observation noise.

[0087] Optionally, the post-fusion module 30 is further configured to output an autonomous driving decision through a trained end-to-end intelligent driving big model according to the pre-fusion result and the post-fusion result;

[0088] Among them, the trained end-to-end intelligent driving model is built based on a convolutional neural network model and a recurrent neural network model. The multiple convolutional layers of the convolutional neural network model are connected to the long short-term memory layer of the recurrent neural network model through a fusion layer, and the fusion layer is used to fuse multimodal data.

[0089] In this embodiment, multiple sensor data, GPS data, and vehicle body data of the autonomous driving vehicle are obtained, and the multiple sensor data include laser radar data, camera data, and millimeter wave radar data; the laser radar data and the camera data are subjected to front fusion processing to obtain a front fusion result; the front fusion result, the millimeter wave radar data, the GPS data, and the vehicle body data are subjected to post fusion processing to obtain a post fusion result, so as to realize the fusion of multimodal data. By performing front fusion processing on the laser radar data and camera data of the autonomous driving vehicle, and then performing post fusion processing on the obtained front fusion result and the millimeter wave radar data, GPS data, and vehicle body data of the autonomous driving vehicle, efficient fusion of multimodal data in complex driving environments can be realized. Furthermore, based on the front fusion results and the post fusion results, the autonomous driving decision is realized through the large model technology, which improves the accuracy and reliability of the autonomous driving decision, and also improves the stability and safety of the autonomous driving system.

[0090] Other embodiments or specific implementation methods of the end-to-end autonomous driving multimodal data fusion system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0091] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0092] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0094] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An end-to-end autonomous driving multimodal data fusion method, characterized in that: The end-to-end autonomous driving multimodal data fusion method comprises the following steps: Acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, including lidar data, camera data, and millimeter-wave radar data; Performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result; The front fusion result, the millimeter wave radar data, the GPS data and the vehicle body data are subjected to post-fusion processing to obtain a post-fusion result, so as to realize the fusion of multimodal data.

2. The end-to-end autonomous driving multimodal data fusion method according to claim 1, characterized in that: After the step of obtaining the plurality of sensor data, GPS data and vehicle body data of the autonomous driving vehicle, the method further includes: Preprocessing is performed on the plurality of sensor data to obtain the preprocessed plurality of sensor data, wherein the preprocessing includes time synchronization processing and coordinate transformation processing.

3. The end-to-end autonomous driving multimodal data fusion method according to claim 1, characterized in that: The step of performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result includes: Perform feature extraction on the laser radar data and the camera data respectively by using a BEVFusion algorithm to obtain a laser radar BEV feature and a camera BEV feature; The laser radar BEV feature and the camera BEV feature are fused to obtain a front fusion result.

4. The end-to-end autonomous driving multimodal data fusion method according to claim 1, characterized in that: The step of performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data to obtain a post-fusion result to achieve fusion of multimodal data includes: Performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data through an extended Kalman filter algorithm to obtain a post-fusion result to achieve fusion of multimodal data; Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is: x k =F k *x k-1 +w k In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes a state vector corresponding to the front fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data; The observation equation corresponding to the extended Kalman filter algorithm is: z k =H k *x k +v k , In the formula, z k is the output of the observation model, H k is the observation matrix, v k is the observation noise.

5. The end-to-end autonomous driving multimodal data fusion method according to claim 1, characterized in that: After the step of performing post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multi-modal data, the method further includes: Outputting an autonomous driving decision through a trained end-to-end intelligent driving big model according to the pre-fusion result and the post-fusion result; Among them, the trained end-to-end intelligent driving model includes a convolutional neural network model and a recurrent neural network model. The multiple convolutional layers of the convolutional neural network model are connected to the long short-term memory layer of the recurrent neural network model through a fusion layer, and the fusion layer is used to fuse multimodal data.

6. An end-to-end autonomous driving multimodal data fusion system, characterized in that: The end-to-end autonomous driving multimodal data fusion system includes: A data acquisition module is used to acquire multiple sensor data, GPS data, and body data of the autonomous driving vehicle, wherein the multiple sensor data include lidar data, camera data, and millimeter wave radar data; A front fusion module, used for performing front fusion processing on the laser radar data and the camera data to obtain a front fusion result; The post-fusion module is used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data to obtain a post-fusion result to achieve fusion of multi-modal data.

7. The end-to-end autonomous driving multimodal data fusion system according to claim 6, characterized in that: The data acquisition module is also used to preprocess multiple sensor data of the autonomous driving vehicle to obtain multiple preprocessed sensor data, wherein the preprocessing includes time synchronization processing and coordinate transformation processing.

8. The end-to-end autonomous driving multimodal data fusion system according to claim 6, characterized in that: The front fusion module is further used to extract features from the laser radar data and the camera data respectively through a BEVFusion algorithm to obtain a laser radar BEV feature and a camera BEV feature; The front fusion module is also used to fuse the laser radar BEV features and the camera BEV features to obtain a front fusion result.

9. The end-to-end autonomous driving multimodal data fusion system according to claim 6, characterized in that: The post-fusion module is further used to perform post-fusion processing on the pre-fusion result, the millimeter-wave radar data, the GPS data and the vehicle body data through an extended Kalman filter algorithm to obtain a post-fusion result to achieve fusion of multi-modal data; Among them, the state estimation equation corresponding to the extended Kalman filter algorithm is: x k =F k *x k-1 +w k In the formula, x k is the prior distribution of the system state at time k, x k-1 is the posterior distribution of the system state at time k-1, F k is the state transfer matrix, w k is Gaussian noise, and the system state includes a state vector corresponding to the front fusion result, the millimeter-wave radar data, the GPS data, and the vehicle body data; The observation equation corresponding to the extended Kalman filter algorithm is: z k =H k *x k +v k , In the formula, z k is the output of the observation model, H k is the observation matrix, v k is the observation noise.

10. The end-to-end autonomous driving multimodal data fusion system according to claim 6, characterized in that: The post-fusion module is further used to output an autonomous driving decision through a trained end-to-end intelligent driving big model according to the pre-fusion result and the post-fusion result; Among them, the trained end-to-end intelligent driving model includes a convolutional neural network model and a recurrent neural network model. The multiple convolutional layers of the convolutional neural network model are connected to the long short-term memory layer of the recurrent neural network model through a fusion layer, and the fusion layer is used to fuse multimodal data.

Images