A multi-modal data processing method and device for end-to-end autonomous driving

Abstract

This application belongs to the field of autonomous driving technology, specifically relating to a multimodal data processing method and apparatus for end-to-end autonomous driving. The method includes: extracting multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of an encoder; performing a feature fusion operation on the image features and LiDAR features of each feature extraction stage after each stage; after the final stage, segmenting at least a first output stream for trajectory prediction from the split LiDAR features; performing a feature enhancement operation on the first output stream and vehicle state features to generate enhanced features; and decoding the enhanced features to generate a sequence of future trajectory points for the vehicle. This application significantly improves inference efficiency and enhances environmental robustness while maintaining perception depth.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to a multimodal data processing method and apparatus for end-to-end autonomous driving. Background Technology

[0002] In the technological development of autonomous driving systems, end-to-end deep learning architectures have become an important research direction due to their ability to integrate perception and decision-making. One of the core challenges of such systems lies in how to effectively fuse data from multimodal sensors such as cameras and LiDAR. Currently, multimodal fusion methods based on the Transformer architecture are one of the mainstream technical solutions in this field, with typical applications including the much-discussed Transfuser model.

[0003] However, this type of method reveals two obvious limitations in practical applications: Inefficient computation: Because its core self-attention mechanism has a quadratic increase in computational complexity when processing long sequence data such as high-resolution images and dense point clouds, the model's inference speed is difficult to meet the real-time requirements of autonomous driving. Insufficient environmental adaptability and robustness: It lacks an effective built-in fault tolerance mechanism for uncertainties in real-world scenarios such as sensor noise and partial occlusion, which affects the stability of the system in complex environments.

[0004] Therefore, while ensuring fusion accuracy, how to significantly improve computational efficiency and environmental adaptability has become a technical challenge that urgently needs to be overcome in this field. Summary of the Invention

[0005] This application proposes a multimodal data processing scheme for end-to-end autonomous driving to address the problems of low computational efficiency and insufficient environmental robustness in existing Transformer-based schemes.

[0006] A first aspect of this application provides a multimodal data processing method for end-to-end autonomous driving, comprising: Multi-scale features of camera images and LiDAR bird's-eye view images are extracted through multiple feature extraction stages of the encoder. After each feature extraction stage, a feature fusion operation is performed on the image features and lidar features of that stage, including: adding image modality location codes to the image features and adding lidar modality location codes to the lidar features and then concatenating them, adding a first global location code to the concatenated features, fusing them through a bidirectional Mamba fusion module, and splitting the fusion result into image features and lidar features for the next stage of processing. After the final stage, at least a first output stream for trajectory prediction is segmented from the split lidar features; Perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then concatenating them, performing differential discarding on the concatenated features, and adding a second global position code to the discarded features; The enhanced features are decoded to generate a sequence of future trajectory points for the vehicle.

[0007] In some embodiments of this application, the encoder is a ResNet34 network, which includes a Stem layer for preliminary feature extraction and four residual block stages connected in sequence: a first stage, a second stage, a third stage, and a fourth stage.

[0008] In some embodiments of this application, the bidirectional Mamba fusion module models the input feature sequence in both forward and backward directions to achieve fusion.

[0009] In some embodiments of this application, in the feature fusion operation, the image modal location encoding, the lidar modal location encoding, and the first global location encoding are all learnable parameters during model training; and / or In the feature enhancement operation, the lidar modal position encoding, the vehicle state modal position encoding, and the second global position encoding are all learnable parameters during model training.

[0010] In some embodiments of this application, the differential discarding of splicing features includes: A discard rate is set differently for feature elements corresponding to different modalities in the spliced ​​features, and a discard operation is performed on the feature elements based on the discard rate.

[0011] In some embodiments of this application, the method further includes: A second output stream for semantic segmentation prediction of the bird's-eye view is segmented from the LiDAR features obtained after splitting.

[0012] In some embodiments of this application, the second output stream is processed sequentially through a convolutional layer, a modified linear unit activation function, and another convolutional layer to output the bird's-eye view semantic segmentation prediction result.

[0013] In some embodiments of this application, decoding the enhanced features includes: The enhanced features are decoded using a Transformer decoder.

[0014] In some embodiments of this application, the method further includes: The sequence of future trajectory points of the vehicle is input into the PID controller, which generates driving commands for the lateral and longitudinal control of the vehicle.

[0015] A second aspect of this application provides a multimodal data processing apparatus for end-to-end autonomous driving, comprising: The feature extraction module is used to extract multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of the encoder; The feature fusion module is used to perform feature fusion operation on the image features and LiDAR features of each feature extraction stage after each feature extraction stage. The feature fusion module includes: adding image modality location codes to the image features and adding LiDAR modality location codes to the LiDAR features and then concatenating them; adding a first global location code to the concatenated features; fusing them through a bidirectional Mamba fusion module; and splitting the fusion result into image features and LiDAR features for processing in the next stage. The feature segmentation module is used, after the final stage, to segment at least a first output stream for trajectory prediction from the split lidar features; The feature enhancement module is used to perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then concatenating them, performing differential discarding on the concatenated features, and adding a second global position code to the discarded features; The trajectory prediction module is used to decode the enhanced features and generate a sequence of future trajectory points for the vehicle.

[0016] In summary, the multimodal data processing method and apparatus for end-to-end autonomous driving provided in the embodiments of this application firstly replace the Transformer architecture based on the self-attention mechanism with a bidirectional Mamba fusion module with linear computational complexity. This module's forward and backward bidirectional modeling mechanism fundamentally avoids the quadratic computation bottleneck and ensures sufficient capture of contextual information, thereby significantly improving inference efficiency while maintaining perception depth and successfully solving the computational efficiency problem. Secondly, by introducing a feature enhancement mechanism including modal position encoding, global position encoding, and differential discarding, accurate spatial semantic alignment of multimodal features is provided, while effectively improving the model's tolerance to sensor noise and data gaps, significantly improving environmental robustness. Furthermore, by constructing a dual-task decoding architecture of trajectory prediction and bird's-eye view semantic segmentation, the fused and enhanced multimodal features are fully utilized to output the bird's-eye view semantic segmentation results and vehicle motion trajectory point sequences in parallel, providing more comprehensive information support for system decision-making. The organic combination of these technical features lays a solid foundation for building a high-performance, highly reliable end-to-end autonomous driving system. Attached Figure Description

[0017] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a schematic diagram of the overall architecture of a modular autonomous driving system; Figure 2 This is a schematic diagram of the ALVINN model architecture; Figure 3 This is a schematic diagram of the overall architecture of the end-to-end autonomous driving system based on bidirectional Mamba proposed in this application; Figure 4 This is a schematic diagram of the internal flow of a bidirectional Mamba fusion module according to some embodiments of this application; Figure 5 This is a schematic diagram of the internal flow of the DPFSD module according to some embodiments of this application; Figure 6 This is a computer system architecture diagram applicable to the multimodal data processing method for end-to-end autonomous driving in this application; Figure 7 This is a flowchart illustrating a multimodal data processing method for end-to-end autonomous driving according to some embodiments of this application; Figure 8 This application presents an embodiment of the comparison between the BMFM model and the baseline model in terms of parameter count, inference speed, and overall performance. Figure 9This paper demonstrates the improvement of various sub-indicators of the BMFM model after gradually introducing the core modules of this application in one embodiment; Figure 10 This application illustrates the impact of different positional encodings on the performance of the BMFM model in one embodiment. Figure 11 This is a schematic diagram of a multimodal data processing apparatus for end-to-end autonomous driving, according to some embodiments of this application. Detailed Implementation

[0018] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0019] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0020] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0021] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0022] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0023] There are generally considered to be two different paths to achieving autonomous driving: modularization and end-to-end solutions. For example... Figure 1 As shown, traditional modular solutions decompose autonomous driving tasks into independent modules such as perception, planning, and control. While this makes debugging easier, it also brings problems such as the need for frequent communication and coordination between modules, complex system architecture, and difficulty in achieving optimal global performance at the system level because each module usually optimizes its local goals independently.

[0024] Therefore, research is increasingly focusing on the construction of end-to-end autonomous driving systems. The core idea of ​​an end-to-end system is to directly map sensor inputs to control commands or trajectories through a unified model. For example... Figure 2 The ALVINN model shown is an early successful implementation of this idea. By learning the mapping from various sensor data (such as image sensors, distance sensors, etc.) to vehicle control commands, it initially verified the feasibility of the end-to-end method and laid the foundation for subsequent research.

[0025] With the evolution of deep learning technology, modern end-to-end autonomous driving models need to handle more complex multimodal inputs (such as images and LiDAR) and output trajectory predictions with longer temporal ranges and higher accuracy. Against this backdrop, methods based on the Transformer architecture, with their powerful global context modeling capabilities, have become the mainstream solution for achieving multimodal feature fusion. However, its core self-attention mechanism has an inherent computational bottleneck: this mechanism needs to calculate the correlation between all pairs of elements in the input sequence, causing the computational complexity to increase quadratically with the sequence length. Faced with long sequences of data consisting of high-resolution images and dense point clouds, this bottleneck leads to significant computational inefficiency, making it difficult to meet the stringent real-time requirements of automotive platforms.

[0026] Furthermore, in complex and dynamic driving scenarios, the model also needs to effectively address practical challenges such as sensor noise and missing data, while existing methods still fall short in robust fusion of cross-modal features.

[0027] To address the aforementioned challenges, this application provides an end-to-end autonomous driving data processing scheme based on bidirectional Mamba. It aims to improve the computational efficiency and environmental adaptability of the autonomous driving system while ensuring high accuracy through efficient multi-scale bidirectional feature fusion and the introduction of a noise-robust feature enhancement mechanism.

[0028] The overall architecture of the solution is as follows: Figure 3As shown, it mainly includes an encoder, a bidirectional Mamba fusion module, a feature enhancement module (DPFSD), and a decoder.

[0029] The encoder is used to extract multi-scale features from images from the vehicle's camera and the LiDAR bird's-eye view (BEV). After each feature extraction stage of the encoder, the image features and LiDAR features are deeply fused using a bidirectional Mamba fusion module. The internal process of the bidirectional Mamba fusion module is as follows: Figure 4 As shown: First, modality-specific location encoding is added to each modality feature to preserve its spatial structure. After concatenation, global location encoding is added. Then, forward and backward sequence modeling is performed via a bidirectional Mamba module to achieve efficient long-range context awareness and cross-modal alignment. Subsequently, the fused features are split into image features and LiDAR features, and sent back to the encoder branch of the next stage for processing. This iterative process realizes multi-level and multi-scale feature interaction.

[0030] In the final stage, the fused features undergo a splitting operation: one path is used to generate a BEV feature map containing rich semantic information; the other path serves as a feature stream for trajectory prediction. The feature stream for trajectory prediction, along with the vehicle state features, is input into the Feature Enhancement Module (DPFSD), whose internal process is as follows: Figure 5 (As shown). This DPFSD module significantly improves the model's feature robustness in uncertain environments by adding specific positional encodings to features of different modalities and then concatenating them, while implementing a differentiated discarding strategy to simulate sensor noise and information loss. Finally, the enhanced features are used by the decoder (Transformer decoder) to generate a sequence of future trajectory points for the vehicle.

[0031] Figure 6 This is a computer system architecture diagram applicable to the multimodal data processing method for end-to-end autonomous driving used in this application. (See diagram for example.) Figure 6 As shown, the system mainly includes interconnected sensor units, a trajectory prediction server, and a control execution unit (autonomous vehicle). The trajectory prediction server acquires multimodal raw data of the autonomous driving environment through the sensor units, performs end-to-end processing on the multimodal raw data, and outputs prediction results including a sequence of future trajectory points for the vehicle, thereby generating control commands which are sent to the control execution unit. Wherein: The trajectory prediction server is the core computing unit deploying an end-to-end autonomous driving model based on bidirectional Mamba. During training, the model efficiently models long-range cross-modal contexts through a multi-scale bidirectional Mamba fusion module and enhances robustness to sensor noise and missing information through a differentiated dropout strategy in the DPFSD feature enhancement module, thus embedding high-precision and high-efficiency trajectory prediction capabilities into the model parameters. During runtime, the server receives multimodal data streams from sensors, performs a complete feature fusion, task segmentation, feature enhancement, and decoding process, and finally outputs trajectory point sequences and other results, which are then converted into underlying vehicle control commands.

[0032] The sensor unit is an interface unit used to collect raw data from the autonomous driving environment in real time. This unit includes at least an image sensor and a LiDAR sensor, responsible for capturing high-resolution visual images and accurate 3D point cloud data, respectively, and sending this multimodal data to a trajectory prediction server for processing.

[0033] The control execution unit (autonomous vehicle) is the physical vehicle that ultimately receives and executes driving commands. This unit receives trajectory point sequences or direct control commands (such as steering, throttle, and braking) from the trajectory prediction server through the onboard control system, and the underlying controller (such as a PID controller) converts them into specific actuator actions to achieve real-time, closed-loop control of the vehicle.

[0034] At the deployment level, the sensor units are integrated into the autonomous vehicle, and the trajectory prediction server can be physically deployed in an onboard computing platform (edge ​​computing), a roadside edge server, or a remote cloud server. Control commands can be issued via vehicle-to-cloud or vehicle-to-edge communication links, depending on network conditions and latency requirements. The entire system can serve as a complete hardware and software solution, or its core trajectory prediction capabilities can be integrated into a wider range of autonomous driving simulation systems or fleet management platforms as API services.

[0035] Figure 7 This is a flowchart illustrating a multimodal data processing method for end-to-end autonomous driving according to some embodiments of this application. In some embodiments, the multimodal data processing method for end-to-end autonomous driving may be... Figure 6 The trajectory prediction server in the system shown performs the following steps: The S710 extracts multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of the encoder.

[0036] Specifically, the input data to the encoder consists of images and LiDAR BEV (LiDAR bird's-eye view image) with a shape of [B, 1, 256, 256], where B is the batch size, representing the number of samples processed simultaneously by the forward / backward propagation model. The image is stitched together from cameras positioned at the right front, front, and left front of the vehicle, with a 3-channel shape and a channel resolution of 256*1024. The LiDAR BEV image is a single-channel image with a channel resolution of 256*256. After the image and LiDAR BEV data are input into the model, a ResNet34 model is used for feature extraction.

[0037] In some embodiments of this application, the ResNet34 model comprises a total of 34 layers, including an initial Stem layer, four sequentially connected Stage residual blocks, and a final global average pooling and fully connected layer. Wherein: Initial Stem layer: A 7×7 convolutional layer with a stride of 2 and 64 output channels is used, followed by a 3×3 max pooling layer (with a stride of 2) for preliminary feature extraction and downsampling of the input image.

[0038] The residual blocks of the four stages are connected in sequence: The first stage consists of three residual blocks, each containing two 3×3 convolutional layers with a stride of 1 and 64 output channels.

[0039] The second stage consists of four residual blocks. The first block has a step size of 2 and the number of output channels is increased to 128. The remaining blocks have a step size of 1 and the number of output channels remains at 128.

[0040] The third stage consists of six residual blocks. The first block has a step size of 2 and the number of output channels increases to 256. The remaining blocks have a step size of 1 and the number of output channels remains at 256.

[0041] Phase 4: Contains three residual blocks. The first block has a step size of 2 and the number of output channels increases to 512. The remaining blocks have a step size of 1 and the number of output channels remains at 512.

[0042] S720, after each feature extraction stage, a feature fusion operation is performed on the image features and LiDAR features of that stage, including: adding image modality location codes to the image features and adding LiDAR modality location codes to the LiDAR features and then concatenating them, adding a first global location code to the concatenated features, fusing them through a bidirectional Mamba fusion module, and splitting the fusion result into image features and LiDAR features for processing in the next stage.

[0043] After the residual block processing at each stage, the image features extracted at that stage and the LiDAR BEV features are input into the bidirectional Mamba fusion module for feature fusion. Figure 4 This is a schematic diagram of the internal processing flow of the bidirectional Mamba fusion module. For example... Figure 4 As shown: First, modality-specific location codes are added to the current image features and LiDAR feature maps to preserve spatial information between modalities and help the model distinguish the spatial locations of different modalities.

[0044] Subsequently, the feature maps of the two modalities, after adding modality-specific location encoding, were concatenated.

[0045] Next, a global token location code is added to the concatenated features to inject global spatial information into the concatenated features and enhance cross-modal spatial interaction.

[0046] The concatenated features are then input into a bidirectional Mamba module for fusion. This module models the feature sequence in both forward and backward directions, thereby capturing long-range dependencies and contextual information within the sequence.

[0047] Finally, the fused features are split into image features and LiDAR features, and fed back into the ResNet34 network for the next stage of processing.

[0048] The image modal position encoding, the lidar modal position encoding, and the global token position encoding mentioned in S720 are all learnable parameters.

[0049] S730, after the final stage, at least a first output stream for trajectory prediction is segmented from the split lidar features.

[0050] Specifically, in some embodiments of this application, corresponding LiDAR features are extracted from the fused features obtained through four different scale fusions, and at least two output streams are then segmented from the LiDAR features: a first output stream for trajectory prediction and a second output stream for bird's-eye view (BEV) semantic segmentation prediction.

[0051] The first output stream will be used together with the vehicle state features for subsequent trajectory prediction.

[0052] S740, perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then concatenating them, performing differential discarding on the concatenated features, and adding a second global position code to the discarded features.

[0053] Specifically, the first output stream and the vehicle state features are input into the dual-modal position coding feature discarding module (DPFSD module) for feature enhancement. Figure 5 This is a schematic diagram of the internal processing flow of the DPFSD module. For example... Figure 5 As shown: First, positional embedding is added for each mode. Specifically: LiDAR features are enhanced using a dedicated LiDAR position encoding, which is a learnable parameter designed to provide contextual information about spatial location. Similarly, vehicle state features are processed using vehicle state modal position encoding (also known as Ego position encoding) to reflect the sequential characteristics of state variables. The introduction of position encoding helps subsequent models capture dependencies within modes. Likewise, Ego position encoding is also a learnable parameter.

[0054] Then, a concat operation is performed on the two features after adding positional encoding.

[0055] Next, the concatenated features are selectively discarded: The differentiated discarding refers to setting different discarding rates for feature elements derived from the LiDAR bird's-eye view image and feature elements derived from the vehicle's state in the stitched features, in order to improve the model's generalization ability and adapt to the characteristics of different modalities.

[0056] The motivation for adopting a differentiated discarding strategy stems from several aspects: First, due to hardware limitations and the inherent noise characteristics of onboard sensors, the observation and perception of the surrounding environment (such as position or speed) may contain errors, failing to fully and accurately reflect the real-world scene. Second, in practical applications of navigation modules, driving commands may be lost due to communication interruptions, or human-provided commands may not be optimal in complex traffic scenarios. In such cases, the model needs to possess the ability to deeply understand and reason about the scene and surrounding dynamic agents to make reasonable decisions in the absence of explicit guidance. Furthermore, research has shown that masking some image and vehicle state features can effectively improve the overall performance of self-supervised tasks and motion planning.

[0057] Finally, after discarding the differential features, the module adds a global token position encoding to the fused features to further enhance the spatial consistency of the features. Similarly, this global token position encoding is also a learnable parameter.

[0058] S750, decode the enhanced features to generate a sequence of future trajectory points for the vehicle.

[0059] Specifically, the enhanced features are decoded using a Transformer decoder.

[0060] The Transformer decoder consists of multiple stacked decoding layers, each of which contains: Masked multi-head self-attention module: The input is the predicted sequence Q. The query Q, key K, and value V matrix are all inputs to the predicted sequence Q, aiming to model the temporal dependencies within the predicted sequence. The multi-head mechanism allows the model to model these dependencies from different perspectives. For example, one attention head focuses on the spatial distance between waypoints, while another head focuses on the directional change trends of waypoints, thereby generating trajectories that better reflect actual driving needs.

[0061] Multi-head cross-attention module: Its input consists of the processed predicted sequence Q (as the query) and the multi-scale fused LiDAR feature map (as the key K and value V). The goal is to interact the target sequence with the encoder output. The cross-attention mechanism allows the decoder to dynamically focus on the contextual information most relevant to the current waypoint prediction. Simultaneously, the multi-head mechanism enables the model to extract contextual information from different perspectives; for example, one head might focus on the spatial distribution of LiDAR features, while another head might focus on the dynamic changes in the vehicle's state features. This dynamism allows the decoder to flexibly extract information from multimodal features according to the needs of the current decoding position.

[0062] Feedforward Neural Network (FFN): It consists of two linear layers and a ReLU activation function. Its purpose is twofold: first, to enhance the non-linear expressive power of features; and second, to complement the attention mechanism: the attention mechanism captures dependencies in the sequence, while FFN focuses on feature enhancement at each position.

[0063] In some embodiments of this application, the second output stream directly passes through a convolutional layer, ReLU, and another convolutional layer to output a BEV semantic segmentation prediction.

[0064] Optionally, the method further includes: The decoded and generated waypoint trajectory of the vehicle in the future The inputs are sent to the PID controller to generate driving commands, including steering angle (Steer), throttle (Throttle), and brake (Brake). It represents the entire trajectory, that is, a set of path points. Indicates at time step A single path point. This represents the total number of path points in the trajectory.

[0065] The PID controller includes two controllers responsible for lateral control and longitudinal control, respectively. It calculates the control output based on the input error through a linear combination of proportional, integral, and derivative terms.

[0066] One embodiment of this application validates the bidirectional Mamba-based end-to-end autonomous driving model (BMFM) described in s710-s750 on the NAVSIM dataset. Specifically, this includes an evaluation of model efficiency and performance.

[0067] 1. Dataset and Data Preprocessing: The training data is the NAVSIM dataset developed based on OpenScene, with a training dataset size of 445G and a test dataset size of 217G. This dataset is a compact redistribution of the large-scale nuPlan dataset, retaining only relevant annotations and 2Hz sensor data.

[0068] The input data for BMFM consists of images from multiple viewpoints and LiDAR point clouds. The image data originates from eight 1920×1080 pixel images from multiple views, while the LiDAR data is generated through fusion of data from five sensors. The model's input architecture supports processing data sequences that include the current time frame and can optionally include frames from the previous three time steps. In this embodiment, for simplicity, only the camera image and LiDAR BEV image at the current time step are used as input, without fusing any historical frame data. The vehicle status (Ego_status) feature includes speed, acceleration, and driving commands provided by the navigation module.

[0069] The model outputs eight waypoint trajectories over the next four seconds, sampled at a frequency of 2Hz. Each waypoint is specified by two-dimensional coordinates (x, y) and a heading angle. The generated waypoint trajectories are then fed to a PID controller to generate corresponding driving commands for low-level vehicle control.

[0070] 2. Experimental setup and evaluation indicators: The experiment used an NVIDIA RTX 3090 as the computing hardware.

[0071] The evaluation indicators mainly include: Total Params: Measured in megabytes (MB). A lower value indicates a lighter model.

[0072] Training / Validation Speed: Measured in iterations per second (it / s), a higher value indicates higher computational efficiency.

[0073] Overall Performance Score (PDM Score): A comprehensive performance evaluation score; the higher the value, the better the overall performance of the model.

[0074] Detailed performance metrics include NC (Navigation Completion), DAC (Driving Advice Compliance), EP (Endpoint Error), TTC (Time To Collision), and C (Compliance), which are used to evaluate model performance from multiple dimensions.

[0075] 3. Experimental Results: To verify the effectiveness of the model and its core modules in this application, the following comparative and ablation experiments were conducted.

[0076] Efficiency and overall performance evaluation: Figure 8 The results show the comparison between the model in this application and the baseline model in terms of parameter count, inference speed and overall performance.

[0077] Module effectiveness assessment: Figure 9 This demonstrates the improvement of various sub-indicators of the model after gradually introducing each core module of this application.

[0078] Location coding strategy effectiveness evaluation: Figure 10 Ablation experiments were conducted to demonstrate the impact of different location encodings on model performance.

[0079] Figure 10 In this context, EGPE represents encoder global position coding, EMPE represents encoder modal position coding, DGPE represents decoder global position coding, and DMPE represents decoder modal position coding.

[0080] 4. Experimental Analysis: Figure 8 and Figure 9 In this comparison, the baseline model used is Transfuser, abbreviated as T. Models compared to Transfuser include: T+BiMamba: Representative model: Based on the baseline model Transfuser(T), only the bidirectional Mamba fusion module (BiMamba Fusion) proposed in this application is integrated.

[0081] Objective: To verify the individual contribution of the bidirectional Mamba fusion module to improving the efficiency and effectiveness of multimodal feature fusion. Figure 8 and Figure 9 Data shows that the model improves upon the baseline in key metrics such as PDM Score and EP (trajectory error).

[0082] T+TB: Representative model: Based on the baseline model Transfuser(T), only the prediction header of the decoding part is replaced or enhanced with a Transformer block.

[0083] Objective: To verify the effect of using a Transformer decoder (compared to simpler structures such as MLPs that may be used in the baseline) on capturing long-range dependencies and improving trajectory prediction accuracy. Figure 9 It was noted that this change significantly improved the TTC (Time to Collision) score.

[0084] T+DPFSD: Representative model: Based on the baseline model Transfuser(T), only the dual-modal location-coded feature discarding module (DPFSD) proposed in this application is integrated.

[0085] Objective: To verify the individual effect of the DPFSD module in enhancing feature robustness through a differential discarding strategy. Figure 9 It is noted that this model achieved the highest EP (trajectory error) score, demonstrating its effectiveness in mitigating poor sensor input.

[0086] BMFM: Representative model: The complete model of this application is a complete architecture that integrates the Bi-Mamba Fusion Module (BiMamba) and the Bimodal Location Coded Feature Drop Module (DPFSD) and uses the Transformer decoder.

[0087] Objective: To verify the overall performance of all core innovative modules of this application when they work together. Figure 8 and Figure 9 As shown, BMFM achieved best performance on most metrics, demonstrating the effectiveness of module integration.

[0088] Figure 8 Results analysis: The complete BMFM model achieved the highest PDM score (0.835) with a lower total parameter count (50.6 MB) than the baseline Transfuser (56.0 MB). The integration of the Bimodal Location-Encoded Feature Dropout (DPFSD) module maintained comparable training and validation speeds to the baseline. The Bi-directional Mamba (BiMamba) module achieved a balanced improvement in performance and speed.

[0089] Figure 9 Results analysis: After introducing the Transformer block (T+TB), the TTC score significantly improved from 0.917 to 0.923.

[0090] The introduction of the bidirectional Mamba module (T+BiMamba) improved the PDM score to 0.827 and the EP score to 0.780, indicating that the use of bidirectional information flow can make fuller use of multimodal data.

[0091] The introduction of the DPFSD module (T+DPFSD) achieved the highest EP score (0.783), demonstrating its effectiveness in reducing intermodal information interference and mitigating poor sensor input.

[0092] The complete BMFM model achieved the highest scores across all evaluation metrics, demonstrating the effectiveness of the ensemble approach.

[0093] Figure 10 Results analysis: Figure 10 The results show that removing any of the positional encodings (EGPE, EMPE, DGPE, DMPE) leads to a decrease in certain performance metrics of the model. This demonstrates that the strategy of combining modality-specific positional encoding and global positional encoding adopted in this application has a significant and necessary positive impact on the performance of multimodal models.

[0094] Figure 11 This is a schematic diagram of a multimodal data processing apparatus for end-to-end autonomous driving, according to some embodiments of this application. Figure 11 As shown, the multimodal data processing device 1100 for end-to-end autonomous driving includes a feature extraction module 1110, a feature fusion module 1120, a feature segmentation module 1130, a feature enhancement module 1140, and a trajectory prediction module 1150. The multimodal data processing function for end-to-end autonomous driving can be... Figure 6 The trajectory prediction server in the system shown is executed. Wherein: The feature extraction module 1110 is used to extract multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of the encoder; The feature fusion module 1120 is used to perform feature fusion operation on the image features and lidar features of each feature extraction stage after each feature extraction stage. The feature fusion operation includes: adding image modality position codes to the image features and adding lidar modality position codes to the lidar features and then concatenating them; adding a first global position code to the concatenated features; fusing them through a bidirectional Mamba fusion module; and splitting the fusion result into image features and lidar features for processing in the next stage. The feature segmentation module 1130 is used to, after the final stage, segment at least a first output stream for trajectory prediction from the split lidar features; The feature enhancement module 1140 is used to perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then splicing them together, performing differential discarding on the spliced ​​features, and adding a second global position code to the discarded features; The trajectory prediction module 1150 is used to decode the enhanced features and generate a sequence of future trajectory points for the vehicle.

[0095] In summary, the multimodal data processing method and apparatus for end-to-end autonomous driving provided in the embodiments of this application firstly replace the Transformer architecture based on the self-attention mechanism with a bidirectional Mamba fusion module with linear computational complexity. This module's forward and backward bidirectional modeling mechanism fundamentally avoids the quadratic computation bottleneck and ensures sufficient capture of contextual information, thereby significantly improving inference efficiency while maintaining perception depth and successfully solving the computational efficiency problem. Secondly, by introducing a feature enhancement mechanism including modal position encoding, global position encoding, and differential discarding, accurate spatial semantic alignment of multimodal features is provided, while effectively improving the model's tolerance to sensor noise and data gaps, significantly improving environmental robustness. Furthermore, by constructing a dual-task decoding architecture of trajectory prediction and bird's-eye view semantic segmentation, the fused and enhanced multimodal features are fully utilized to output the bird's-eye view semantic segmentation results and vehicle motion trajectory point sequences in parallel, providing more comprehensive information support for system decision-making. The organic combination of these technical features lays a solid foundation for building a high-performance, highly reliable end-to-end autonomous driving system.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0097] Although the foregoing embodiments are described within the general context of computer systems running in conjunction with operating systems and applications, those skilled in the art will recognize that other implementations can also be performed by incorporating other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0098] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A multimodal data processing method for end-to-end autonomous driving, characterized in that, include: Multi-scale features of camera images and LiDAR bird's-eye view images are extracted through multiple feature extraction stages of the encoder. After each feature extraction stage, a feature fusion operation is performed on the image features and lidar features of that stage, including: adding image modality location codes to the image features and adding lidar modality location codes to the lidar features and then concatenating them, adding a first global location code to the concatenated features, fusing them through a bidirectional Mamba fusion module, and splitting the fusion result into image features and lidar features for the next stage of processing. After the final stage, at least a first output stream for trajectory prediction is segmented from the split lidar features; Perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then concatenating them, performing differential discarding on the concatenated features, and adding a second global position code to the discarded features; The enhanced features are decoded to generate a sequence of future trajectory points for the vehicle.

2. The method according to claim 1, characterized in that: The encoder is a ResNet34 network, which includes a Stem layer for preliminary feature extraction, and four residual block stages connected in sequence: a first stage, a second stage, a third stage, and a fourth stage.

3. The method according to claim 1, characterized in that: The bidirectional Mamba fusion module models the input feature sequences in both forward and backward directions to achieve fusion.

4. The method according to claim 1, characterized in that: In the feature fusion operation, the image modal location encoding, the lidar modal location encoding, and the first global location encoding are all learnable parameters during model training; and / or In the feature enhancement operation, the lidar modal position encoding, the vehicle state modal position encoding, and the second global position encoding are all learnable parameters during model training.

5. The method according to claim 1, characterized in that, The differential discarding of spliced ​​features includes: A discard rate is set differently for feature elements corresponding to different modalities in the spliced ​​features, and a discard operation is performed on the feature elements based on the discard rate.

6. The method according to claim 1, characterized in that, The method further includes: A second output stream for semantic segmentation prediction of the bird's-eye view is segmented from the LiDAR features obtained after splitting.

7. The method according to claim 6, characterized in that, The second output stream is processed sequentially through a convolutional layer, a modified linear unit activation function, and another convolutional layer to output the semantic segmentation prediction result of the bird's-eye view.

8. The method according to claim 1, characterized in that, Decoding the enhanced features includes: The enhanced features are decoded using a Transformer decoder.

9. The method according to claim 1, characterized in that, The method further includes: The sequence of future trajectory points of the vehicle is input into the PID controller, which generates driving commands for the lateral and longitudinal control of the vehicle.

10. A multimodal data processing device for end-to-end autonomous driving, characterized in that, include: The feature extraction module is used to extract multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of the encoder; The feature fusion module is used to perform feature fusion operation on the image features and LiDAR features of each feature extraction stage after each feature extraction stage. The feature fusion module includes: adding image modality location codes to the image features and adding LiDAR modality location codes to the LiDAR features and then concatenating them; adding a first global location code to the concatenated features; fusing them through a bidirectional Mamba fusion module; and splitting the fusion result into image features and LiDAR features for processing in the next stage. The feature segmentation module is used, after the final stage, to segment at least a first output stream for trajectory prediction from the split lidar features; The feature enhancement module is used to perform feature enhancement operations on the first output stream and the vehicle state features to generate enhanced features, including: adding LiDAR modal position codes to the first output stream, adding vehicle state modal position codes to the vehicle state features and then concatenating them, performing differential discarding on the concatenated features, and adding a second global position code to the discarded features; The trajectory prediction module is used to decode the enhanced features and generate a sequence of future trajectory points for the vehicle.

Images