An end-to-end autonomous driving prediction and planning method, system, device and medium

Abstract

The application discloses an end-to-end automatic driving prediction and planning method, system, device and medium, comprising the following steps: obtaining original image data based on a sensing device, performing preliminary extraction, and obtaining multi-modal features; performing multi-class perception tasks on the multi-modal features obtained through preliminary extraction based on target detection and semantic segmentation; performing multi-modal feature fusion on the multi-modal features based on different fusion strategies to obtain fused multi-modal features; performing modal perception and state planning on the fused multi-modal features to obtain preliminary prediction and planning results; and performing post-processing optimization on the obtained preliminary prediction and planning results to obtain optimal end-to-end automatic driving prediction and planning results. The application predicts and optimizes a path by collecting feature information, thereby establishing a multi-modal fusion network based on a Transformer and a prediction and planning system of auxiliary multi-modal features, ensuring the accuracy and robustness of an automatic driving perception task, and significantly reducing the collision rate.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving prediction and planning method, system, device and medium. Background Technology

[0002] Traditional autonomous driving technologies often employ a segmented architecture, placing perception, decision-making, and planning functions in separate modules. This approach can lead to information omissions or inconsistencies during data transfer between modules. In contrast, the end-to-end approach utilizes a unified deep learning model to directly map sensor data to driving actions, circumventing the limitations of modular architectures. This method effectively uncovers the correlations between multimodal sensor data and enhances perception accuracy, decision rationality, and planning scientific rigor through global optimization strategies. Furthermore, by eliminating complex inter-module interaction steps, the end-to-end model can more agilely adapt to dynamic changes in the driving environment, demonstrating significant real-time performance advantages.

[0003] Despite the immense potential of end-to-end approaches, their implementation faces numerous challenges. Multimodal data fusion is crucial for efficient perception and planning, but the heterogeneity of data from different sensors makes spatiotemporal alignment and effective interaction difficult. In complex and ever-changing urban traffic environments, the uncertainties surrounding vehicles, pedestrians, and road conditions impose stringent requirements on model robustness. Furthermore, handling long-tail scenarios in autonomous driving—ensuring model safety in rare but high-risk situations—is a critical issue that end-to-end approaches urgently need to address. Additionally, since these methods are typically based on large deep neural networks, the high-dimensional state and action spaces require significant real-time computing power, posing challenges to practical deployment.

[0004] Research on end-to-end autonomous driving technology focuses on several key methods. Convolutional Neural Networks (CNNs) are widely used to process camera image data and extract scene features and semantic information; Recurrent Neural Networks (RNNs) are used to combine historical information for temporal modeling and predict the trajectory of dynamic targets. In recent years, attention mechanisms and Transformer models have been increasingly used in multimodal data fusion, effectively capturing the correlations between different modalities. Furthermore, Reinforcement Learning (RL) continuously optimizes driving strategies through interaction with the environment, while Generative Adversarial Networks (GANs) are used for simulation training and data augmentation to improve the model's generalization ability.

[0005] While the aforementioned methods have achieved some success in perception and planning, they still have limitations in practical applications. End-to-end models require a large amount of high-quality labeled data for training, but the cost of collecting and labeling actual road data is prohibitive. Model performance may degrade significantly in scenarios outside the coverage of the training data, limiting its applicability in complex and novel environments. Furthermore, the black-box nature of end-to-end models increases the difficulty of interpreting the decision-making process, posing challenges to safety verification and regulation. Simultaneously, designing a multi-task learning framework that can simultaneously optimize perception, planning, and control tasks is also extremely challenging. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art. To achieve the above objectives, an end-to-end autonomous driving prediction and planning method, system, device and medium are adopted to solve the problems mentioned in the background art.

[0007] The first aspect of the technical solution: an end-to-end autonomous driving prediction and planning method based on multimodal fusion, comprising the following steps:

[0008] Step S1: Acquire raw image data based on sensing devices, perform preliminary extraction, and obtain multimodal features;

[0009] Step S2: Based on object detection and semantic segmentation, perform multi-class perception tasks on the initially extracted multimodal features;

[0010] Step S3: Based on different fusion strategies, perform multimodal feature fusion on the multimodal features to obtain the fused multimodal features;

[0011] Step S4: Perform modality perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results;

[0012] Step S5: Post-process and optimize the obtained preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results.

[0013] As a further aspect of the present invention, the specific steps in step S1 include:

[0014] Step S11: Obtain the raw data from the camera. Use the pre-trained ResNet-18 network as the base model. Input the raw image into ResNet-18 and transform it into a feature vector from a bird's-eye view.

[0015] Step S12: Based on step S11, a pyramid convolutional neural network is further used for processing to effectively capture features at different scales of the image and fuse them.

[0016] Step S13: Obtain the point cloud data of the LiDAR, use the SECOND algorithm to efficiently generate the point cloud network, and extract key features;

[0017] Step S14: Concatenate the camera feature vector and the lidar feature vector together to form a multimodal feature vector, which contains information from at least two sensors.

[0018] As a further aspect of the present invention, the specific steps in step S2 include:

[0019] Step S21: Based on multimodal feature vectors, perform multi-target detection tasks through deep learning networks. The multi-target detection tasks include, but are not limited to, detecting road signs, pedestrians, other vehicles, traffic participants such as traffic lights, and road surface information.

[0020] Step S22: Use multimodal feature vectors to perform semantic segmentation and identify different areas of the road and their attributes. The different areas include lanes, intersections, driving lanes, and non-motorized vehicle lanes.

[0021] As a further aspect of the present invention, the specific steps in step S3 include:

[0022] Step S31: Employ a multimodal feature fusion network based on the Transformer model. The multimodal feature fusion network includes an encoder-decoder structure, wherein the encoder is used to extract features from multiple sensors, and the decoder is used to fuse the extracted features into a unified representation.

[0023] Step S32: In the process of multimodal feature fusion, the attention mechanism is used to perform weighted fusion of features acquired by different sensors;

[0024] Step S33: Use data augmentation techniques to expand the training data, perform rotation, cropping, scaling, and flipping operations on the original image, and perform affine transformation on the lidar point cloud data;

[0025] Step S34: Input the multimodal feature fusion representation vector into the decoder. The decoder further processes the fused features through a multilayer perceptron network to obtain the final multimodal fusion representation, which contains key information from different sensors.

[0026] Step S35: Perform multi-task learning on the fused feature vectors to optimize the performance of different perception tasks.

[0027] As a further aspect of the present invention, the specific steps in step S4 include:

[0028] Step S41: During the target detection and tracking process, based on the fused feature vector, different types of targets are detected by the target detection algorithm, and target tracking is performed to provide real-time target position and motion status.

[0029] Step S42: For path planning and decision-making tasks, a deep Q-network is used to learn path planning strategies;

[0030] Step S43: Use the value function of the deep Q network to select the optimal action, and determine the action that the vehicle should perform by maximizing the Q value function;

[0031] Step S44: Based on environmental feedback, including the relative distance between the vehicle and obstacles and the status of traffic lights, update the Q-value function of the deep Q network to optimize the path planning and decision-making process;

[0032] Step S45: Test the effectiveness of modal perception and state planning in a simulation scenario, collect actual driving data to verify target detection, tracking, path planning and decision-making strategies, and continuously optimize and adjust the model based on actual feedback.

[0033] As a further aspect of the present invention, the specific steps in step S5 include:

[0034] Step S51: In the target detection task, the non-maximum suppression method is used to post-process the detection boxes;

[0035] If multiple overlapping target boxes are detected, non-maximum suppression removes redundant boxes by comparing the confidence scores of each box, and only retains the box with the highest confidence score.

[0036] Step S52: For the path planning task, a path smoothing technique is introduced to optimize the coherence of the planned path. After the path is generated, short-distance turns in the path are smoothed using interpolation methods, making the path transition more natural, reducing unnecessary sharp turns, and improving driving stability and safety.

[0037] Step S53: In the target tracking task, a target tracking technology based on Kalman filter is adopted. The Kalman filter uses information such as the target's current position, velocity and acceleration to estimate the target's trajectory in real time, overcoming misjudgments caused by changes in the target's trajectory or background interference, thereby improving the accuracy and stability of target tracking.

[0038] Step S54: Based on the post-processing optimization results, comprehensively adjust the target detection box, path planning trajectory, and target tracking trajectory to further improve the overall performance of the model. This process ensures that the outputs of target detection, path planning, and target tracking are more accurate, smooth, and stable in a multi-task environment.

[0039] The second aspect of the technical solution: A planning system comprising any one of the above-mentioned features, including:

[0040] The data acquisition module is used to acquire raw image data based on sensing devices, perform preliminary extraction, and obtain multimodal features;

[0041] The task execution module is used to perform multi-class perception tasks on the initially extracted multimodal features based on object detection and semantic segmentation.

[0042] The feature fusion module is used to fuse multimodal features based on different fusion strategies to obtain fused multimodal features;

[0043] The perception and planning module is used to perform modal perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results;

[0044] The optimization module is used to post-process and optimize the preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results.

[0045] The third aspect of the technical solution: A device, further comprising:

[0046] At least one processor;

[0047] At least one memory for storing at least one program;

[0048] When the at least one program is executed by the at least one processor, the at least one processor implements an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any of the preceding claims.

[0049] The fourth aspect of the technical solution: a storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any of the preceding claims.

[0050] Compared with the prior art, the present invention has the following technical advantages:

[0051] By employing the aforementioned technical solution, raw image data is acquired through sensing devices, and multimodal features are initially extracted. Subsequently, object detection and semantic segmentation techniques are used to perform multi-class perception tasks. Then, multimodal feature fusion is achieved based on different fusion strategies, followed by modal perception and state planning to obtain preliminary prediction and planning results. Path planning and decision-making incorporating environmental feedback ensure dynamic adjustment and safety of vehicle behavior. Post-processing optimization methods further enhance the stability and robustness of target tracking and path planning, providing strong support for the development of autonomous driving technology. Attached Figure Description

[0052] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0053] Figure 1 This is a schematic diagram illustrating the steps of the prediction and planning method according to an embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating the prediction and planning method according to an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please refer to Figure 1 and Figure 2 In this embodiment of the invention, an end-to-end autonomous driving prediction and planning method based on multimodal fusion includes the following steps:

[0057] Step S1: Acquire raw image data based on the sensing device, perform preliminary extraction to obtain multimodal features, the specific steps of which include:

[0058] Step S11: Obtain the camera's raw data I img It uses a pre-trained ResNet-18 network. As the base model, the original image is input into ResNet-18 and transformed into a feature vector F from a bird's-eye view. img The formula is:

[0059]

[0060] Step S12: Based on step S11, a pyramid convolutional neural network Conv is further used for processing, using convolutional kernels K of different sizes. i Effectively captures features F1, F2, ..., F of images at different scales. n And then they are fused together, the formula is:

[0061] F i =Conv i (I img ,K i );

[0062] Step S13: Acquire point cloud data I from the LiDAR.pc Since the irregularity and disorder of point cloud data make it difficult for neural networks to extract features, the SECOND algorithm is used to normalize the point cloud to voxel space. Next, a point cloud feature extraction network is used. Extracting key features F pc :

[0063]

[0064] Step S14: Convert the camera feature vector F img and lidar feature vector F pc The vectors are concatenated to form a multimodal feature vector F. con This vector contains information from two sensors:

[0065] F con =[F img F con ];

[0066] Step S2: Based on object detection and semantic segmentation, perform multi-class perception tasks on the initially extracted multimodal features. The specific steps include:

[0067] Step S21: Based on multimodal feature vectors, perform multi-target detection tasks through deep learning networks. The multi-target detection tasks include, but are not limited to, detecting road signs, pedestrians, other vehicles, traffic participants such as traffic lights, and road surface information.

[0068] Step S22: Use multimodal feature vectors to perform semantic segmentation and identify different areas of the road and their attributes. The different areas include lanes, intersections, driving lanes, and non-motorized vehicle lanes.

[0069] Step S3: Based on different fusion strategies, perform multimodal feature fusion to obtain fused multimodal features. The specific steps include:

[0070] Step S31: Employ a multimodal feature fusion network based on the Transformer model. This network includes an encoder-decoder structure, where the encoder extracts features from multiple sensors (camera and LiDAR), and the decoder fuses the extracted features into a unified representation. The encoder uses a self-attention mechanism to weight features from different modalities to improve the ability to capture important features.

[0071] Step S32: In the multimodal feature fusion process, an attention mechanism is used to weightedly fuse features provided by different sensors. Specifically, a self-attention module is used to correlate features from different sensors to increase attention to key regions and enhance the model's ability to fuse features from different modalities.

[0072] Q = F con W Q K = F con W K V = F con W V , F fused =AV;

[0073] Among them, W Q W K W V The learned weight matrix is ​​used to generate query, key, and value vectors, respectively, and the final multimodal feature representation F fused ;

[0074] Step S33: Use data augmentation techniques to augment the training data to improve the model's generalization ability and avoid overfitting. Specific methods include rotating, cropping, scaling, and flipping the original images, as well as performing affine transformations on the LiDAR point cloud data, to increase the diversity and representativeness of the training samples.

[0075] Step S34: The fused representation vector of the multimodal features is input into the decoder. The decoder further processes the fused features through a multilayer perceptron network to obtain the final multimodal fused representation. This representation contains key information from different sensors and provides support for subsequent perception tasks (object detection, road segmentation, object tracking).

[0076] First, the input multimodal fusion features undergo a linear transformation through the first fully connected layer:

[0077] h1=σ(F fused W1+b1);

[0078] Where W1 is the weight matrix of the first fully connected layer, with a size of d. fused ×d1, b1 are bias terms with a size of d1, and σ(·) is the activation function, which adds nonlinearity.

[0079] Next, after processing by the multilayer perceptron, the features are passed layer by layer. Assuming there are l layers, the computation process of the fully connected layer can be recursively represented as follows:

[0080] h l =σ(h) l-1 W l +b l );

[0081] After passing through a multilayer perceptron network, the final feature representation F is obtained. finalThis will provide support for subsequent perception tasks. The output of the final layer can be expressed by the following formula:

[0082] F final =h L ;

[0083] Step S35: Perform multi-task learning on the fused feature vectors to optimize the performance of different perceptual tasks. By sharing feature representations, redundancy between different tasks is reduced, improving computational efficiency and the overall performance of the model.

[0084] Step S4: Perform modality perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results. The specific steps include:

[0085] Step S41: During the target detection and tracking process, based on the fused feature vector, different types of targets (other vehicles, pedestrians, obstacles) are detected by the target detection algorithm, and target tracking is performed to provide real-time target position and motion status.

[0086] The specific steps of object detection can be represented by the following formula:

[0087]

[0088] in, These are the center coordinates of the i-th target bounding box; These are the width and height of the target; It is the class probability of the target; D(F) final This is an object detection network that outputs the location information and category of each object.

[0089] Step S42: For path planning and decision-making tasks, a Deep Q-Network (DQN) is used to learn path planning strategies. Specifically, the input state information (current vehicle position, speed, surrounding obstacles, traffic rules, etc.) is used as the current state, and the DQN network outputs the value estimate of each possible action.

[0090] Step S43: The optimal action is selected using the value function of a deep Q-network. The action (steering, acceleration, deceleration, etc.) that the vehicle should perform is determined by maximizing the Q-value function. This process also considers factors such as obstacles, traffic signs, and traffic rules in the environment to ensure that the selected action is feasible and safe in the actual environment.

[0091] Step S44: Based on environmental feedback (relative distance between vehicles and obstacles and traffic light status), update the Q-value function of the deep Q-network to optimize path planning and decision-making. An exploratory-exploitation strategy balance is employed to continuously improve decision-making and achieve long-term optimal behavior.

[0092] Step S45: Test the effectiveness of modal perception and state planning in a simulation scenario, collect actual driving data to verify target detection, tracking, path planning and decision-making strategies, and continuously optimize and adjust the model based on actual feedback.

[0093] Step S5: Post-process and optimize the obtained preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results. The specific steps include:

[0094] Step S51: In the target detection task, the non-maximum suppression method is used to post-process the detection boxes;

[0095] If multiple overlapping target boxes are detected, non-maximum suppression compares the confidence scores of each box, removes redundant boxes, and retains only the box with the highest confidence score, thereby effectively avoiding duplicate detection and improving the accuracy of target detection.

[0096] Step S52: For the path planning task, a path smoothing technique is introduced to optimize the continuity of the planned path. After the path is generated, short-distance turns in the path are smoothed using interpolation methods, making the path transition more natural, reducing unnecessary sharp turns, and improving driving stability and safety.

[0097] Step S53: In the target tracking task, a target tracking technique based on a Kalman filter is adopted. The Kalman filter uses information such as the target's current position, velocity, and acceleration to estimate the target's trajectory in real time, overcoming misjudgments caused by changes in the target's trajectory or background interference, thereby improving the accuracy and stability of target tracking.

[0098] Step S54: Based on the post-processing optimization results, comprehensively adjust the target detection box, path planning trajectory, and target tracking trajectory to further improve the overall performance of the model. This process ensures that the outputs of target detection, path planning, and target tracking are more accurate, smooth, and stable in a multi-task environment.

[0099] The second aspect of the technical solution: A planning system comprising any one of the above-mentioned features, including:

[0100] The data acquisition module is used to acquire raw image data based on sensing devices, perform preliminary extraction, and obtain multimodal features;

[0101] The task execution module is used to perform multi-class perception tasks on the initially extracted multimodal features based on object detection and semantic segmentation.

[0102] The feature fusion module is used to fuse multimodal features based on different fusion strategies to obtain fused multimodal features;

[0103] The perception and planning module is used to perform modal perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results;

[0104] The optimization module is used to post-process and optimize the preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results.

[0105] The third aspect of the technical solution: A device, further comprising:

[0106] At least one processor;

[0107] At least one memory for storing at least one program;

[0108] When the at least one program is executed by the at least one processor, the at least one processor implements an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any of the preceding claims.

[0109] The fourth aspect of the technical solution: a storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any of the preceding claims.

[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. An end-to-end autonomous driving prediction and planning method based on multimodal fusion, characterized in that, Includes the following steps: Step S1: Acquire raw image data based on the sensing device, perform preliminary extraction to obtain multimodal features, the specific steps of which include: Step S11: Obtain the raw data from the camera. Use the pre-trained ResNet-18 network as the base model. Input the raw image into ResNet-18 and transform it into a feature vector from a bird's-eye view. Step S12: Based on step S11, a pyramid convolutional neural network is further used for processing to effectively capture features at different scales of the image and fuse them. Step S13: Obtain the point cloud data of the LiDAR, use the SECOND algorithm to efficiently generate the point cloud network, and extract key features; Step S14: Concatenate the camera feature vector and the lidar feature vector together to form a multimodal feature vector, which contains information from at least two sensors; Step S2: Based on object detection and semantic segmentation, perform multi-class perception tasks on the initially extracted multimodal features; Step S3: Based on different fusion strategies, multimodal features are fused to obtain fused multimodal features. The specific steps include: Step S31: Employ a multimodal feature fusion network based on the Transformer model. The multimodal feature fusion network includes an encoder-decoder structure, wherein the encoder is used to extract features from multiple sensors, and the decoder is used to fuse the extracted features into a unified representation. Step S32: In the process of multimodal feature fusion, the attention mechanism is used to perform weighted fusion of features acquired by different sensors; Step S33: Use data augmentation techniques to expand the training data, perform rotation, cropping, scaling, and flipping operations on the original image, and perform affine transformation on the lidar point cloud data; Step S34: Input the multimodal feature fusion representation vector into the decoder. The decoder further processes the fused features through a multilayer perceptron network to obtain the final multimodal fusion representation, which contains key information from different sensors. Step S35: Perform multi-task learning on the fused feature vectors to optimize the performance of different perceptual tasks; Step S4: Perform modality perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results. The specific steps include: Step S41: During the target detection and tracking process, based on the fused feature vector, different types of targets are detected by the target detection algorithm, and target tracking is performed to provide real-time target position and motion status. Step S42: For path planning and decision-making tasks, a deep Q-network is used to learn path planning strategies; Step S43: Use the value function of the deep Q network to select the optimal action, and determine the action that the vehicle should perform by maximizing the Q value function; Step S44: Based on environmental feedback, including the relative distance between the vehicle and obstacles and the status of traffic lights, update the Q-value function of the deep Q network to optimize the path planning and decision-making process. Step S45: Test the effectiveness of modal perception and state planning in a simulation scenario, collect actual driving data to verify target detection, tracking, path planning and decision-making strategies, and continuously optimize and adjust the model based on actual feedback; Step S5: Post-process and optimize the obtained preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results. The specific steps include: Step S51: In the target detection task, the non-maximum suppression method is used to post-process the detection boxes; If multiple overlapping target boxes are detected, non-maximum suppression removes redundant boxes by comparing the confidence scores of each box, and only retains the box with the highest confidence score. Step S52: For the path planning task, a path smoothing technique is introduced to optimize the continuity of the planned path. After the path is generated, short-distance turns in the path are smoothed by interpolation to make the path transition more natural, reduce unnecessary sharp turns, and improve the stability and safety of driving. Step S53: In the target tracking task, a target tracking technology based on Kalman filter is adopted. The Kalman filter uses the target's current position, velocity and acceleration information to estimate the target's trajectory in real time, overcoming misjudgments caused by changes in the target's trajectory or background interference, thereby improving the accuracy and stability of target tracking. Step S54: Based on the post-processing optimization results, comprehensively adjust the target detection box, path planning trajectory, and target tracking trajectory to further improve the overall performance of the model.

2. The end-to-end autonomous driving prediction and planning method based on multimodal fusion according to claim 1, characterized in that, The specific steps in step S2 include: Step S21: Based on multimodal feature vectors, perform multi-target detection tasks through deep learning networks. The multi-target detection tasks include, but are not limited to, detecting road signs, pedestrians, other vehicles, traffic participants such as traffic lights, and road surface information. Step S22: Use multimodal feature vectors to perform semantic segmentation and identify different areas of the road and their attributes. The different areas include lanes, intersections, driving lanes, and non-motorized vehicle lanes.

3. A planning system comprising an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any one of claims 1 to 2, characterized in that, include: The data acquisition module is used to acquire raw image data based on sensing devices, perform preliminary extraction, and obtain multimodal features; The task execution module is used to perform multi-class perception tasks on the initially extracted multimodal features based on object detection and semantic segmentation. The feature fusion module is used to fuse multimodal features based on different fusion strategies to obtain fused multimodal features; The perception and planning module is used to perform modal perception and state planning on the fused multimodal features to obtain preliminary prediction and planning results; The optimization module is used to post-process and optimize the preliminary prediction and planning results to obtain the optimal end-to-end autonomous driving prediction and planning results.

4. An apparatus, characterized in that, Also includes: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements an end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any one of claims 1-2.

5. A storage medium storing processor-executable instructions, characterized in that: The processor-executable instructions, when executed by the processor, are used to implement the end-to-end autonomous driving prediction and planning method based on multimodal fusion as described in any one of claims 1-2.

Images