Transform-based multi-modal feature fusion end-to-end automatic driving method

Abstract

The invention relates to an end-to-end automatic driving method based on Transform multi-modal feature fusion, and the method comprises the steps: collecting an RGB image and an original depth image, and converting the original depth image into an HHA image; inputting the RGB image and the HHA image into a sensing module for fusion to obtain a fusion feature map, and performing global average pooling and flattening operation to obtain environment features; acquiring the real-time speed of a vehicle, an advanced navigation command and a target position, connecting in series to form measurement input, and processing based on an MLP measurement encoder to obtain measurement characteristics; adding the environment features and the measurement features element by element to obtain combined features, performing downsampling step by step, and connecting a ReLU activation function behind each layer to obtain track features; and a loss estimator is constructed, training loss is dynamically predicted in real time based on the track features, and the weight of each feature parameter is dynamically adjusted according to the loss, so that the vehicle is dynamically controlled and adjusted in real time. According to the method, the multi-modal features can be well fused, so that a more accurate end-to-end automatic driving method is realized.

Description

Technical Field

[0001] The present invention relates to the field of end-to-end autonomous driving, and in particular to an end-to-end autonomous driving method based on Transformer-based multimodal feature fusion. Background Art

[0002] The end-to-end autonomous driving approach refers to collecting data directly from the vehicle's sensors (such as camera images, radar signals, etc.), processing this data through algorithms, and ultimately outputting the vehicle's driving instructions (such as steering, acceleration, deceleration, etc.). The entire process is fully automated, does not require human intervention, and does not rely on any preset independent modules.

[0003] The core of end-to-end autonomous driving lies in integrating the three modules of perception, planning, and control through a large model, eliminating the boundaries between them and forming a whole. Specifically, its functions are as follows:

[0004] The perception end collects environmental information, such as data obtained by the vehicle's cameras, lidar and other sensors.

[0005] The control end makes decisions based on the perceived environmental information to control the vehicle's trajectory and speed.

[0006] After the planning and control end collects the perception information, the system directly outputs driving instructions through the algorithm without the need for manually defined rules.

[0007] An end-to-end autonomous driving approach offers the following advantages:

[0008] High flexibility: It does not rely on prior information such as high-precision maps, and autonomous driving can be achieved nationwide as long as there is a navigation system.

[0009] High decision-making efficiency: Through learning from large amounts of data, it is possible to find response strategies that suit the scenario more quickly.

[0010] High safety: Combined with the visual language model (VLM), it can understand complex traffic scenarios and improve the safety and reliability of the system.

[0011] The end-to-end autonomous driving approach has the following disadvantages:

[0012] High technical difficulty: A large amount of high-quality human driving data is required for training, and the training process is complex.

[0013] High cost: Requires high-performance computing resources and a large amount of training data.

[0014] Application Scenarios and Future Development Trends of End-to-End Autonomous Driving Methods: End-to-end autonomous driving methods excel in complex scenarios, particularly those involving irregular intersections and traffic lights, providing accurate navigation and driving decisions. In the future, with continued technological advancement and the accumulation of data, end-to-end autonomous driving methods will become more mature and their application will expand.

[0015] However, existing end-to-end autonomous driving methods have the following technical bottlenecks:

[0016] Insufficient multimodal data fusion: Traditional RGB and LiDAR fusion requires complex alignment, high computational cost, and local convolution operations make it difficult to capture global feature relationships.

[0017] Limited feature expression: Single-modal data (such as RGB images) lacks depth information, resulting in weak 3D scene understanding capabilities; pixel-level fusion strategies ignore cross-modal complementarity.

[0018] Rigid control strategy: Existing methods use fixed weights to fuse trajectory prediction and control signals, which cannot dynamically adapt to changes in complex scenarios, affecting safety and adaptability.

[0019] In summary, existing end-to-end autonomous driving methods still have significant deficiencies in the fusion of multimodal data, resulting in an inability to fully integrate and utilize multimodal data, making control strategies prone to deviations. Summary of the Invention

[0020] In order to solve the above technical problems, the present invention provides an end-to-end autonomous driving method based on Transformer multimodal feature fusion, which can well fuse multimodal features, thereby realizing a more accurate end-to-end autonomous driving method.

[0021] The present invention provides an end-to-end autonomous driving method based on Transformer multimodal feature fusion, which adopts the following technical solutions:

[0022] A Transformer-based end-to-end autonomous driving method for multimodal feature fusion includes the following steps:

[0023] S1. Install three cameras on the top of the vehicle for collecting RGB images. The three cameras face the front of the vehicle, 60° to the left of the front, and 60° to the right of the front.

[0024] Three depth cameras are installed on the top of the vehicle to collect raw depth images. The depth cameras are located at the same position as the camera. The three depth cameras face the front of the vehicle, 60° to the left of the front, and 60° to the right of the front. The original depth images are converted into HHA images including three dimensions: height, horizontal angle, and vertical angle.

[0025] S2: Input the RGB image and HHA image into the perception module for fusion to obtain the fusion feature map, and perform global average pooling and flattening operations on the fusion feature map to obtain the environmental feature I Feat ;

[0026] S3: Obtain the vehicle's real-time speed, high-level navigation command, and target position, input the vehicle's real-time speed, high-level navigation command, and target position into the decision module in series to form a measurement input m, and process the measurement input m based on the MLP measurement encoder to obtain the measurement feature f m ;

[0027] Environmental Characteristics I Feat and the measured characteristic f m The combined feature F is obtained by element-by-element addition, and the combined feature F is gradually downsampled. Each layer is followed by a ReLU activation function to obtain the trajectory feature f traj ;

[0028] S4. Build a loss estimator to dynamically predict the training loss in real time based on trajectory features, and dynamically adjust the weights of each feature parameter according to these losses to perform real-time dynamic control adjustments on the vehicle.

[0029] Preferably, the perception module includes a PVT backbone network and a dual attention feature fusion network;

[0030] The PVT backbone network includes OPE operation, initial normalization operation, CTB operation and terminal normalization operation;

[0031] The dual attention feature fusion network includes:

[0032] Enhance the expression of image features and depth features through self-attention feature enhancement network;

[0033] The information of image features and depth features is complemented by the cross-attention feature calibration network;

[0034] The depth-guided feature fusion network guides the depth features and fuses the image features with the depth features.

[0035] Preferably, the self-attention feature enhancement network includes:

[0036] The image features and deep features Flatten and rearrange into a one-dimensional sequence and Where L = H × W is the length of the sequence, H and W are the height and width of the feature map, and C is the number of channels of the feature;

[0037] The image sequence f rgb and the depth sequence f hha Obtain query (Q), key (K), and value (V) embeddings through linear projection;

[0038]

[0039] Among them, (Q r ,K r ,V r ) and (Q h ,K h ,V h ) are the image sequences f rgb and the depth sequence f hha The corresponding embedding, is the learned projection matrix, D represents the dimension of embedding;

[0040] The attention weights are calculated using the scaled dot product between Q and K and the weights are used to aggregate the values for each query:

[0041]

[0042] Among them, Attention represents the attention weight calculation process, softmax is the activation function, d k It is a scaling factor and is used to prevent the dot product value from being too large to affect the gradient calculation;

[0043] The self-attention feature enhancement network also includes the use of a multi-head attention mechanism to expand the traditional attention mechanism into N independent attention heads. By capturing the feature relationships at different positions through multiple independent attention heads, the expression ability of the model is enhanced. The formula is as follows:

[0044]

[0045] Among them, MultiHead is the multi-head attention calculation process, Concat represents the splicing operation, and denote the output of the i-th attention head in the image sequence and depth sequence respectively, It is the output projection matrix. The splicing operation ensures the diversity of multi-head attention features.

[0046] A dynamic position encoding generator is designed to dynamically generate spatial position information encoding for the image branch and the depth branch. The dynamic position encoding generator receives the high-resolution, low-channel feature map F output from the first stage of the PVT backbone network as input and generates a dynamic position encoding DPE through the following steps:

[0047] The input features are expanded to 4 times the original number of channels through a 1×1 convolutional layer. The expanded features are then passed through two convolutional blocks consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function for nonlinear transformation and feature enhancement to extract higher-level spatial information.

[0048] The features are downsampled to 1 / 8 of their original resolution through nearest neighbor interpolation. The flattened and rearranged features are then used to generate the final dynamic position encoding.

[0049] The generated dynamic position code is embedded into Q, K and V, and the attention weight distribution is adjusted;

[0050] Image branch position code P generated by dynamic position code generator r and the deep branch position encoding P h , respectively with (Q r ,K r ,V r ) and (Q h ,K h ,V h ) embedding is combined to calculate the attention weights of the multi-head self-attention mechanism; the attention output generated by these weights is enhanced by residual connection and nonlinear transformation to obtain enhanced image features and deep features

[0051] The enhanced features are calculated using the following formula:

[0052]

[0053] Among them, MultiHead represents the multi-head attention calculation process, which generates context-enhanced feature representation by capturing the global dependencies between different feature positions. MLP is a multi-layer perceptron, which is used to improve the expressiveness of features. It retains the original information of the input features through residual connections and incorporates global context information.

[0054] Preferably, the cross-attention feature calibration network includes:

[0055] Adopt cross-query generation method: the query Q embedding in the image branch is generated by linear projection of depth features, while the query Q embedding in the depth branch is generated by linear projection of image features;

[0056] The cross-attention feature calibration network outputs the calibrated image features and deep features The process of feature calibration through the multi-head cross attention mechanism is described by the following formula:

[0057]

[0058] Among them, MultiHead represents the multi-head attention calculation process, MLP is a multi-layer perceptron, and It is the enhanced image feature and depth feature output by the self-attention feature enhancement network. and They are and The query, key and value embeddings obtained by linear projection, is the learned projection matrix, D represents the dimension of embedding, (P r ,P h ) is the dynamic position encoding generated by the dynamic position encoding generator. The position encoding explicitly embeds the spatial position information into Q and K, enhancing the attention mechanism to model the spatial relationship of the input features.

[0059] Preferably, the deep guided feature fusion network includes:

[0060] Query Q F Embedded by calibrated deep features Generated by linear projection, and the key K F Sum V F The embedding is composed of calibrated image features After linear projection generation, the dynamic position encoding generator is in the fusion process. The deep features are explicitly guided by the image features through query embedding. The fusion feature output of the dynamic position encoding generator is Output F Representing the global information and geometric details of the scene;

[0061] The following formula is used to describe the process of feature fusion based on the multi-head cross-attention mechanism in the dynamic position encoding generator:

[0062]

[0063] in, and is the calibrated image feature and depth feature output by the cross-attention feature calibration network, (Q F ,K F ,V F )yes and The query, key and value embeddings obtained by linear projection, MultiHead represents the multi-head attention calculation process, MLP is a multi-layer perceptron, (P r ,P h ) is a dynamic position code generated by the dynamic position code generator.

[0064] Preferably, the decision module includes:

[0065] The real-time velocity v, high-level navigation command p, and target position g are concatenated to form the measurement input m, which is then processed by the MLP-based measurement encoder to generate the measurement feature f. m :

[0066]

[0067] Among them, f m The dimension is 128, representing the dynamic state and navigation information of the vehicle, Concat represents the concatenation operation, and MLP is a multi-layer perceptron;

[0068] Environmental Characteristics I Feat With the measured characteristic f m The combined feature F is obtained by element-by-element addition;

[0069] F=I Feat +f m ;

[0070] The combined feature F is shared by the trajectory prediction branch and the multi-step control prediction branch;

[0071] In the trajectory planning branch, the input combined feature F is gradually downsampled through the linear layer, and each layer is followed by a ReLU activation function to generate a compact trajectory feature f traj , then the trajectory feature f traj The input is fed into four Gated Recurrent Units (GRUs) to predict the vehicle's path points (w0, w1, w2, w3) for the next four time steps. The generated path points are used by two independent PID controllers to generate longitudinal and lateral control signals, respectively. The longitudinal controller uses the speed reference value of the path point as the target and generates the accelerator or brake signal in combination with the current vehicle speed; the lateral controller uses the direction angle of the path point as the target and generates the steering angle signal.

[0072] In the multi-step control prediction branch, the control actions for the next four time steps are predicted, and the trajectory-guided attention mechanism is used to enhance the integration of the vehicle's current state and environmental characteristics;

[0073] The control branch uses the hidden state information of GRU in the trajectory branch. The control output consists of a value head and a strategy head, and multi-step control information is generated through linear downsampling and GRU processing.

[0074] Preferably, the loss estimator comprises:

[0075] The GRU hidden features h and latent feature representation f in the control branch and trajectory branch are used to predict trajectory loss and control signal loss respectively;

[0076] The loss estimator obtains the predicted values of trajectory loss and control signal loss through the following formula:

[0077]

[0078] in, and They are loss estimators E traj and E ctrl Generates the predicted values of trajectory loss and control signal loss, the loss estimator E traj and E ctrl It consists of multiple linear layers, L align To predict the alignment loss, L align The loss estimator is optimized by the mean squared error (MSE) so that the predicted loss is close to the true loss:

[0079]

[0080] Among them, L w and L c are the true losses for trajectory prediction and control signal prediction, respectively;

[0081] The calculation of the control preference value is based on the prediction loss. The priority of each branch is adjusted by dynamically weighting the importance of the trajectory branch and the control branch. The formula is as follows:

[0082]

[0083] Among them, k c and k w Is the loss range balance coefficient, which is used to balance the difference in the numerical range of the two losses. c and k w All are set to 1.0;

[0084] By controlling the preference value Prefer, the output weights of the trajectory branch and the control branch are dynamically adjusted;

[0085] The final output of the control signal u final The control signals of the trajectory branch and the control branch are combined in a weighted manner. Assume that the output of the trajectory branch is u traj , the control branch output is u ctrl , then the final control signal u final The calculation formula is:

[0086] u final=Prefer·u ctrl +(1-Prefer)·u traj ;

[0087] When the predicted trajectory loss is large The preference value Prefer approaches 1, and the control signal generated by the control branch contributes more to the final control signal. On the contrary, when the predicted control loss is large,

[0088] The preference value Prefer approaches 0, and the control signal generated by the trajectory branch has a greater weight on the final control signal.

[0089] Preferably, the losses include:

[0090] Trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align ;

[0091] Trajectory planning loss L traj It consists of the L1 loss between the predicted waypoints and the true value waypoints and the feature loss, expressed as:

[0092]

[0093] Among them, w t and Represent the predicted waypoint and the true waypoint of the tth step, L F Represents feature loss, which is used to measure the current step feature With expert features The L2 distance between them, as an additional supervisory signal, λ F is an adjustable loss weight. For control prediction, L ctrl The loss can be expressed as:

[0094]

[0095] Loss function L ctrl The difference between the prediction and the true Beta distribution is measured by KL divergence, which is applicable to the initial moment and future time steps t, and is given by λ F Weighted feature loss improves the learning ability of the model at each time step;

[0096] RGBD-based 2D semantic segmentation is introduced as an auxiliary task, and the auxiliary loss L aux as follows:

[0097] L Aux =λ s ·L speed +λv ·L value +λ seg ·L seg ;

[0098] The above trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align According to the weight λ traj ,λ ctrl ,λ aux and λ align Calculate the weights and the total loss function is:

[0099] L=λ traj ·L traj +λ ctrl ·L ctrl +λ aux ·L aux +λ align ·L align .

[0100] In summary, the present invention has the following beneficial technical effects:

[0101] 1. This application can effectively integrate multimodal features, thereby achieving a more accurate end-to-end autonomous driving method.

[0102] 2. This application enhances robustness: the dual attention mechanism effectively integrates cross-modal global features, and the dynamic control strategy significantly improves adaptability to complex scenarios.

[0103] 3. This application adopts a lightweight PVT backbone network to make calculations more efficient, balance computing resources and real-time requirements, and is suitable for in-vehicle embedded platforms.

[0104] 4. This application provides an efficient and reliable technical solution for autonomous driving systems by building a lightweight end-to-end framework that takes into account both perception accuracy and real-time performance.

[0105] 5. This application proposes a dynamic control preference adjustment mechanism to achieve scene-adaptive trajectory and control signal fusion, breaking through the limitations of static weights.

[0106] 6. This application introduces the Transformer-based dual attention mechanism into RGB-D multimodal fusion, solving the problem of cross-modal global feature modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 It is the multimodal perception and decision-making framework architecture for end-to-end autonomous driving in the embodiment of the present invention.

[0108] Figure 2 Schematic diagram of the installation positions of the camera and the depth camera in an embodiment of the present invention.

[0109] Figure 3 This is an architectural diagram of the PVT backbone network in an embodiment of the present application.

[0110] Figure 4 This is an architectural diagram of the dual attention feature fusion network in an embodiment of the present application.

[0111] Figure 5 Schematic diagram of a dynamic position code generator in an embodiment of the present application. DETAILED DESCRIPTION

[0112] The present invention will be further described in detail below with reference to the accompanying drawings.

[0113] An embodiment of the present invention discloses an end-to-end autonomous driving method based on Transformer multimodal feature fusion.

[0114] Reference Figure 1 This Transformer-based, multimodal feature fusion, end-to-end autonomous driving approach utilizes two core modules: the perception module and the decision module. The perception module extracts environmental information features from the input sensor modalities, while the decision module generates future control signals for the vehicle based on these environmental information features. Furthermore, 2D semantic segmentation is employed as an auxiliary task, with the goal of improving the feature extraction capabilities of the perception module and the overall performance of the end-to-end autonomous driving approach through a multi-task learning strategy. The following sections detail the representation of the proposed model's input and output, and provide specific examples of the perception module, decision module, dynamic control preference adjustment mechanism, and loss function.

[0115] Specifically, the end-to-end autonomous driving method based on Transformer multimodal feature fusion includes the following steps:

[0116] S1. Install three cameras for collecting RGB images and three depth cameras for collecting original depth images on the top of the vehicle.

[0117] Specifically, refer to Figure 2 The three cameras face the front of the vehicle, 60° to the left of the front, and 60° to the right of the front respectively; H and W in the RGB image captured by the camera represent the height and width of the image respectively.

[0118] Each camera has a horizontal field of view of 120°. The overlapping fields of view of these cameras provide wide-angle coverage, ensuring that the model can fully perceive dynamic objects, static obstacles and other traffic elements in the surrounding environment, enhancing its understanding of complex driving scenarios.

[0119] The depth cameras are positioned at the same position as the front camera. They face the front, 60° to the left, and 60° to the right of the front of the vehicle, respectively. They convert the original depth image into an HHA image with three dimensions: height, horizontal angle, and vertical angle.

[0120] The original depth image is a single-channel depth map, in which the value of each pixel represents the current distance, that is, the depth value. The single-channel depth map only contains depth information and cannot express the spatial relationship in the scene structure, which limits the model's understanding of the environment. In contrast, this application uses the Depth-to-HHA method to convert the original depth image into an HHA image consisting of three dimensions: height, horizontal angle, and vertical angle. This representation enhances the visualization of depth information and can better capture the 3D spatial relationship in the environment, so that more effective features can be extracted through the deep learning model, thereby improving the accuracy of environmental perception. Finally, the converted HHA image is used as the input of the depth branch, and the other settings remain consistent with the image branch configuration.

[0121] In addition to the two modal data inputs of RGB image and HHA image, such as Figure 1 As shown in Figure 2, the system also receives three additional inputs: real-time speed, high-level navigation commands, and target position. The speed input helps the system consider the vehicle's real-time state in perception and low-level control, while the high-level navigation commands and target position are used to support path planning and high-level decision making.

[0122] In this architecture, the output of the perception module is a feature vector representing the current driving environment. Based on these features, the trajectory branch in the decision module predicts the future trajectory—the combination of waypoints the autonomous vehicle will pass through in the future, where T is a hyperparameter representing the number of waypoints to be predicted. A PID controller is then used to decode these waypoints into control information for the trajectory branch. Simultaneously, the control branch in the decision module directly predicts future vehicle control information based on these feature vectors. Finally, the proposed adaptive control fusion mechanism generates the final vehicle control commands by weightedly fusing the control information from the trajectory branch and the control branch, dynamically adjusting the weights based on real-time feedback.

[0123] S2: Input the RGB image and HHA image into the perception module for fusion to obtain the fusion feature map, and perform global average pooling and flattening operations on the fusion feature map to obtain the environmental feature I Feat ;

[0124] The perception module includes a PVT backbone network and a dual attention feature fusion network, which are optimized for feature extraction and feature fusion respectively. Figure 2As shown in the figure, the perception module extracts features from the RGB image and the HHA image respectively through the PVT backbone network, and then inputs the extracted features into the dual attention feature fusion network. The PVT backbone network enhances the expressiveness of image and depth features through a multi-head self-attention mechanism, and effectively fuses image and depth features using a cross-attention mechanism, fully exploiting the complementary information between the two modalities, further improving the expressiveness and robustness of the perception module.

[0125] Specifically, the PVT backbone network includes OPE operation, initial normalization operation, CTB operation and terminal normalization operation, please refer to Figure 3 shown.

[0126] Each stage consists of an Overlapping Block Embedding (OPE) and a Convolutional Transformer Block (CTB). OPE achieves more efficient local feature extraction by enlarging the patch window and introducing a 50% overlap between adjacent windows. This design effectively avoids the loss of spatial context that can occur with direct pooling while enhancing the correlation between features between adjacent windows, making the model more sensitive to local boundaries and details. The extracted features are then initially normalized and passed to the CTB.

[0127] Normalization operations help accelerate training and stabilize the gradient updates of the network. In CTB, the query (Q) is generated from the input features through linear projection, while the key (K) and value are obtained through convolution operations. Compared with direct linear projection, convolution can retain more local spatial information and provide stronger spatial context relevance for the subsequent attention mechanism. CTB combined with convolution can capture global features while retaining more local spatial information through convolution, thereby maintaining high sensitivity to local details and boundaries and significantly improving feature expression capabilities. After the CTB operation, the terminal normalization operation is performed and the result is output.

[0128] The image branch uses PVT-B2, pre-trained on ImageNet, and the depth branch uses the untrained PVT-B1. This differentiated design aims to fully utilize the rich contextual information of RGB images while avoiding the domain shift error introduced by pre-trained weights on the feature distribution of depth data. Due to significant differences in the distribution of depth data and RGB images, such as less texture information, directly using pre-trained weights may limit the model's ability to express depth features. Through end-to-end training, the depth branch can more naturally adapt to the distribution characteristics of depth information, complementing the image branch and improving the feature expression capabilities of the perception module.

[0129] Finally, since the feature dimension generated by the final stage of the PVT backbone network is 512, to reduce the computational complexity of subsequent modules, 3×3 convolutions are used to reduce the dimension of the features extracted by the backbone network in the image and depth branches to 256. This dimensionality reduction not only reduces computational cost but also preserves contextual information within the spatial neighborhood through the convolution operation, thus achieving a balance between performance and computational efficiency.

[0130] The PVT backbone network can efficiently extract multi-scale local and global features in image and depth branches, providing reliable input for subsequent model components.

[0131] The dual attention feature fusion network includes:

[0132] like Figure 4 As shown in the figure, a three-stage feature fusion network is designed. First, the Self-Attention Feature Enhancement Network (SA-FENet) is used to enhance the expressive power of image and depth features respectively, capturing the long-range dependencies between features. Second, the Cross-Attention Feature Calibration Network (CA-FCNet) is used to achieve information complementarity between the two feature modalities and reduce the uncertainty in a single modality. Finally, the Depth-Guided Feature Fusion Network (DG-FFNet) accurately fuses image features with depth features through the guidance of depth features. In the following paragraphs, the details of each component network will be introduced.

[0133] Self-Attention Feature Enhancement Network: Considering the advantages of the self-attention mechanism in capturing long-range dependencies and modeling global contextual information, SA-FENet leverages the self-attention mechanism in the Transformer to enhance the image and deep features extracted by the PVT backbone network. Compared with the traditional Transformer-based self-attention mechanism, SA-FENet incorporates dynamic position encoding (DPE), which not only improves the ability to capture long-range dependencies between features but also better adapts to the feature distribution of different modalities by dynamically adjusting the encoding method. This improvement addresses the problem that traditional fixed position encoding is less sensitive to changes in the input distribution.

[0134] First, image features and deep features is flattened and rearranged into a one-dimensional sequence and Where L = H × W is the length of the sequence, H and W are the height and width of the feature map, and C is the number of channels of the feature. Then, use the following formula to convert the image sequence f rgb and the depth sequence f hha The query (Q), key (K), and value (V) embeddings are obtained through linear projection.

[0135]

[0136] Among them, (Q r ,K r ,V r ) and (Q h ,K h ,V h ) are the image sequences f rgb and the depth sequence f hha The corresponding embedding, is the learned projection matrix and D represents the dimension of embedding.

[0137] Then, we use the scaled dot product between Q and K to calculate the attention weights, and use the weights to aggregate the values of each query:

[0138]

[0139] Among them, Attention represents the attention weight calculation process, and softmax is the activation function. k It is a scaling factor used to prevent the dot product value from being too large and affecting the gradient calculation.

[0140] To further improve model performance, SA-FENet adopts a multi-head attention mechanism. The multi-head attention mechanism expands the traditional attention mechanism into N independent attention heads. Through multiple independent attention heads, it captures the feature relationships at different positions and enhances the model's expressive power. The specific formula is as follows:

[0141]

[0142] Among them, MultiHead is the multi-head attention calculation process, and Concat represents the splicing operation.

[0143] and denote the output of the i-th attention head in the image sequence and depth sequence respectively, It is the output projection matrix, and the splicing operation ensures the diversity characteristics of multi-head attention.

[0144] At the same time, in order to make up for the deficiency of self-attention mechanism in modeling spatial relationships when capturing sequence information, a position encoding mechanism is introduced. By designing a dynamic position encoding generator (Dynamic Position Encoding Generator, D-PEG), it is used to dynamically generate spatial position information encoding for the image branch and the depth branch. Figure 5 As shown in the figure, D-PEG receives as input the high-resolution, low-channel feature map F output from the first stage of the PVT backbone network and generates a dynamic positional encoding (DPE) through the following steps: First, the input features are expanded to four times the original number of channels through a 1×1 convolutional layer, thereby increasing the dimensionality of the feature space and enabling the model to capture richer spatial information and diverse feature representations. Subsequently, the expanded features are passed through two convolutional blocks consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function for nonlinear transformation and feature enhancement, thereby extracting higher-level spatial information. Next, the features are downsampled to 1 / 8 the original resolution through nearest neighbor interpolation. This process effectively reduces computational complexity while preserving critical global spatial information. Finally, the flattened and reordered features are used to generate the final dynamic positional encoding.

[0145] After the generated dynamic position encoding is embedded in Q, K, and V, the distribution of attention weights is adjusted to effectively capture the dependencies of spatial information, thereby enhancing the self-attention mechanism's ability to model spatial relationships. Compared to fixed position encoding, dynamic position encoding better adapts to the distribution characteristics of input features by adaptively adjusting the position encoding. For input image features or depth features, D-PEG can dynamically adjust the position encoding based on the local and global information of the features, making it more adaptable to the needs of different input branches. This flexible adjustment method allows the position encoding to be optimized according to the specific distribution of input features.

[0146] Finally, the image branch position encoding P generated by D-PEG is used r and the deep branch position encoding P h , respectively with (Q r ,K r ,V r ) and (Q h ,K h ,V h ) embedding is combined to calculate the attention weights of the multi-head self-attention mechanism. The attention output generated by these weights is enhanced by residual connection and nonlinear transformation to obtain enhanced image features. and deep features The shapes of the enhanced image features and depth features generated are consistent with the input features, ensuring that they can be directly used for subsequent task processing. Specifically, the enhanced features can be calculated using the following formula:

[0147]

[0148] MultiHead represents the multi-head attention computation process, which generates context-enhanced feature representations by capturing the global dependencies between different feature positions. Multi-layer perceptrons (MLPs) further enhance the expressive power of features. Residual connections preserve the original information of the input features while incorporating global context.

[0149] Through this design, SA-FENet can effectively capture long-range dependencies between different feature dimensions, enhance feature representation, and improve global context perception. This enhanced feature representation provides richer and more detailed information for the subsequent Cross-Attention Feature Calibration Network (CA-FCNet), facilitating the effective interaction between image and depth features.

[0150] Cross-Attention Feature Calibration Network: The goal of cross-calibration is to reduce uncertainty in a single modality by leveraging intermodal information complementarity. In complex scenes, RGB images may not accurately capture important visual cues, such as lane markings or edge contours, while depth features may inaccurate geometric information due to sensor noise or measurement bias. By leveraging intermodal feature interaction, cross-calibration combines the strengths of both modalities, mitigates their respective limitations, enhances the robustness of feature representation, and improves adaptability to complex scenes.

[0151] CA-FCNet introduces a Transformer-based multi-head cross-attention mechanism that not only models feature dependencies within modalities but also directly models feature interactions between modalities. This mechanism calculates correlations between queries and key-value pairs across modalities, incorporating global contextual information into feature updates, significantly improving inter-modal complementarity and overall feature expressiveness.

[0152] In the implementation process, CA-FCNet adopts a cross-query generation method: the query Q embedding in the image branch is generated by the linear projection of the depth feature, while the query Q embedding in the depth branch is generated by the linear projection of the image feature. This cross-generation strategy can enhance the information complementarity between modalities during feature calibration. Finally, CA-FCNet outputs the calibrated image features. and deep features The following formula describes the process of feature calibration through the multi-head cross attention mechanism:

[0153]

[0154]

[0155] Among them, MultiHead represents the multi-head attention calculation process, and MLP is a multi-layer perceptron. and It is the enhanced image feature and depth feature output by SA-FENet. and They are and Query, key, and value embeddings obtained through linear projection.

[0156] is the learned projection matrix, and D represents the dimension of embedding. (P r ,P h ) is the dynamic position encoding generated by D-PEG. Position encoding enhances the attention mechanism’s ability to model the spatial relationship of input features by explicitly embedding spatial position information into Q and K, thereby further improving the spatial consistency of the calibration results.

[0157] Through the aforementioned multi-head cross-attention feature calibration network, CA-FCNet effectively models global contextual interactions and feature dependencies between modalities. This Transformer-based mechanism overcomes the limitations of traditional methods that are limited to modeling intra-modal features, enabling deep complementarity of information between modalities, reducing the uncertainty inherent in a single modality, and making the output features more robust and expressive.

[0158] Deep-Guided Feature Fusion Network: To effectively fuse calibrated image and depth features, DG-FFNet also introduces a Transformer-based multi-head cross-attention mechanism to dynamically model the global contextual relationships between the two modal features, thereby achieving feature optimization and fusion. Similar to CA-FCNet, DG-FFNet uses cross-modality modeling of global contextual relationships to fully leverage the guiding role of deep features on image features, significantly improving the spatial consistency and overall robustness of features during the fusion process.

[0159] In DG-FFNet, the query Q F Embedded by calibrated deep features Generated by linear projection, and the key K F Sum V F The embedding is composed of calibrated image features Generated by linear projection. At the same time, in order to improve the robustness of the fusion feature, during the fusion process of DG-FFNet, the deep features explicitly guide the image features through query embedding, so that the fused features have stronger spatial consistency and representation ability. Finally, the fusion feature output of DG-FFNet is Output FIt can more accurately represent the global information and geometric details of the scene. The following formula describes the feature fusion process based on the multi-head cross attention mechanism in DG-FFNet:

[0160]

[0161] in, and are the calibrated image features and depth features output by CA-FCNet.

[0162] (Q F ,K F ,V F )yes and The query, key, and value embeddings are obtained through linear projection. MultiHead represents the multi-head attention calculation process, and MLP is a multi-layer perceptron. (P r ,P h ) is a dynamic positional code generated by D-PEG.

[0163] Through a Transformer-based multi-head cross-attention mechanism, DG-FFNet is able to fully combine the detailed representation of image features with the spatial geometric information of deep features. Compared with traditional modal fusion methods, DG-FFNet's fused features not only have stronger expressive power and robustness, but also can better adapt to the diverse perception needs in complex scenes, providing higher-quality input for subsequent decision modules.

[0164] S3: Obtain the vehicle's real-time speed, high-level navigation command, and target position, input the vehicle's real-time speed, high-level navigation command, and target position into the decision module in series to form a measurement input m, and process the measurement input m based on the MLP measurement encoder to obtain the measurement feature f m ;

[0165] Environmental Characteristics I Feat and the measured characteristic f m The combined feature F is obtained by element-by-element addition, and the combined feature F is gradually downsampled. Each layer is followed by a ReLU activation function to obtain the trajectory feature f traj ;

[0166] Specifically, the decision module is based on the trajectory guidance control network proposed by TCP. Figure 1 As shown in Figure 1, the input RGB image and HHA image are processed by the perception module to obtain the fused feature map G. Then, the fused feature map G undergoes global average pooling (GAP) and flattening operations to obtain the environment feature I with a length of 256. Feat , used to characterize the current global environment characteristics.

[0167] At the same time, the real-time velocity v, high-level navigation command p, and target position g are concatenated to form the measurement input m, which is then processed by the MLP-based measurement encoder to generate the measurement feature f. m :

[0168]

[0169] Among them, f m The dimension is 128, representing the dynamic state and navigation information of the vehicle. Concat represents the concatenation operation, and MLP is a multi-layer perceptron.

[0170] Finally, Environmental Characteristics I Feat With the measured characteristic f m The combined feature F is obtained by element-by-element addition.

[0171] F=I Feat +f m ;

[0172] The combined feature F is shared by the trajectory prediction branch and the multi-step control prediction branch.

[0173] In the trajectory planning branch, first, the input combined feature F is gradually downsampled through a series of linear layers, each followed by a ReLU activation function, to generate a compact trajectory feature f traj . Connect the trajectory feature f traj The input is fed into four gated recurrent units (GRUs) to predict the vehicle's pathpoints (w0, w1, w2, w3) four time steps into the future. Finally, the generated pathpoints are used by two independent PID controllers to generate longitudinal and lateral control signals, respectively. The longitudinal controller uses the velocity reference value at the pathpoint as a target and combines it with the current vehicle speed to generate the accelerator or brake signal; the lateral controller uses the heading angle at the pathpoint as a target to generate the steering angle signal.

[0174] In the multi-step control prediction branch, this module directly predicts control actions four time steps into the future. It uses a trajectory-guided attention mechanism to enhance the integration of the vehicle's current state and environmental features. Specifically, the control branch leverages the hidden state information of the GRU in the trajectory branch to improve its focus on key feature areas. The final control output consists of a value head and a policy head, which are then processed by linear downsampling and the GRU to generate multi-step control information.

[0175] S4. Build a loss estimator to dynamically predict the training loss in real time based on trajectory features, and dynamically adjust the weights of each feature parameter according to these losses to perform real-time dynamic control adjustments on the vehicle.

[0176] Specifically, in autonomous driving tasks, different scenarios have significantly different requirements for trajectory prediction and control signal prediction. To enable the control strategy to dynamically adapt to these requirements and address the challenge of not being able to obtain true losses in closed-loop testing, a dynamic control preference adjustment mechanism (DCPAM) based on prediction losses is proposed. This mechanism dynamically predicts the training losses of the trajectory branch and the control branch by constructing a loss estimator, and adjusts the weights of each branch based on these losses, thereby achieving a more accurate and flexible control strategy. Compared with TCP, DCPAM can significantly improve the robustness and safety of autonomous driving systems in complex environments, effectively reducing potential driving risks.

[0177] Since the actual trajectory loss and control signal loss cannot be directly obtained in closed-loop testing, these losses are indirectly inferred through prediction losses, and the predicted losses are used to optimize the control strategy. A loss estimator module is designed, which uses the GRU hidden features h and latent feature representation f in the control branch and trajectory branch to predict trajectory loss and control signal loss, respectively. Specifically, the loss estimator obtains the predicted values of trajectory loss and control signal loss using the following formula:

[0178]

[0179] in, and They are loss estimators E traj and E ctrl Generates predictions of trajectory loss and control signal loss. Loss Estimator E traj and E ctrl It consists of multiple linear layers. At the same time, in order to ensure that the prediction loss can reflect the actual training loss as much as possible, the prediction alignment loss L is designed align , which optimizes the loss estimator by the mean squared error (MSE) so that the predicted loss is close to the true loss:

[0180]

[0181] Among them, L w and L c are the true losses for trajectory prediction and control signal prediction, respectively.

[0182] The control preference value (Prefer) is calculated based on the prediction loss, and the priority of each branch is adjusted by dynamically weighting the importance of the trajectory branch and the control branch. The specific formula is as follows:

[0183]

[0184] Among them, kc and k w Is the loss range balance coefficient, which is used to balance the difference in the value range of the two losses. c and k w All are set to 1.0. Through this formula, the preference value changes dynamically in the range of [0,1], reflecting the requirements of trajectory prediction and control signal prediction in different scenarios.

[0185] By controlling the preference value Prefer, the output weights of the trajectory branch and the control branch can be dynamically adjusted. Specifically, the final output u of the control signal final The control signals of the trajectory branch and the control branch can be combined in a weighted manner. Let the output of the trajectory branch be u traj , the control branch output is u ctrl , then the final control signal u final The calculation formula is:

[0186] u final =Prefer·u ctrl +(1-Prefer)·u traj ;

[0187] Therefore, when the predicted trajectory loss is large (i.e. ), the preference value Prefer approaches 1, and the control signal generated by the control branch contributes more to the final control signal. On the contrary, when the predicted control loss is large (i.e. ), the preference value Prefer approaches 0, and the control signal generated by the trajectory branch contributes greatly to the final control signal.

[0188] Compared to traditional static control preference assignments (such as fixed weights), this approach can dynamically adjust control strategies based on real-time scenarios. In particular, in closed-loop testing, where there is a lack of actual loss feedback, the model can adjust the control strategy based on predicted loss, improving the adaptability and robustness of the control strategy.

[0189] The loss function is specifically:

[0190] The loss includes trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align .

[0191] Trajectory planning loss L traj It consists of the L1 loss and feature loss between the predicted waypoints and the true value waypoints, which can be expressed as:

[0192]

[0193] Among them, wt and L denotes the predicted waypoint and the true waypoint of the tth step respectively. F Represents feature loss, which is used to measure the current step feature With expert features The L2 distance between λ and λ serves as an additional supervisory signal. F is an adjustable loss weight. For control prediction, L ctrl The loss can be expressed as:

[0194]

[0195] Loss function L ctrl It consists of four terms. It measures the difference between the prediction and the true Beta distribution through the Kullback-Leibler (KL) divergence, applicable to the initial moment and the future time step t. F The weighted feature loss improves the learning ability of the model at each time step.

[0196] In addition, by introducing RGBD-based 2D semantic segmentation as an auxiliary task, which has been proven to be effective in end-to-end autonomous driving. In use, the same decoder architecture as the AIM-MT baseline is used to decode the semantic segmentation results from the fusion features generated by the perception module. Specifically, seven semantic categories such as vehicles, roads, red lights, pedestrians, lane markings, sidewalks, and unmarked objects are considered. The semantic segmentation is supervised by the cross entropy loss, which further improves the environmental perception ability of the model and helps the intelligent agent better understand the surrounding environment. The final auxiliary loss L aux as follows:

[0197] L Aux =λ s ·L speed +λ v ·L value +λ seg ·L seg ;

[0198] The above trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align According to the weight λ traj ,λ ctrl ,λ aux and λ align Calculate the weights and the total loss function is:

[0199] L=λ traj ·L traj +λ ctrl ·L ctrl+λ aux ·L aux +λ align ·L align .

[0200] The training loss is dynamically predicted in real time based on trajectory features, and the weights of each feature parameter are dynamically adjusted according to these losses to make real-time dynamic control adjustments to the vehicle.

[0201] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving method based on Transformer-based multimodal feature fusion, characterized by: The following steps are involved: S1. Install three cameras on the top of the vehicle for collecting RGB images. The three cameras face the front of the vehicle, 60° to the left of the front, and 60° to the right of the front. Three depth cameras are installed on the top of the vehicle to collect raw depth images. The depth cameras are located at the same position as the camera. The three depth cameras face the front of the vehicle, 60° to the left of the front, and 60° to the right of the front. The original depth images are converted into HHA images including three dimensions: height, horizontal angle, and vertical angle. S2: Input the RGB image and HHA image into the perception module for fusion to obtain the fusion feature map, and perform global average pooling and flattening operations on the fusion feature map to obtain the environmental feature I Feat ; S3: Obtain the vehicle's real-time speed, high-level navigation command, and target position, input the vehicle's real-time speed, high-level navigation command, and target position into the decision module in series to form a measurement input m, and process the measurement input m based on the MLP measurement encoder to obtain the measurement feature f m ; Environmental Characteristics I Feat and the measured characteristic f m The combined feature F is obtained by element-by-element addition, and the combined feature F is gradually downsampled. Each layer is followed by a ReLU activation function to obtain the trajectory feature f traj ; S4. Build a loss estimator to dynamically predict the training loss in real time based on trajectory features, and dynamically adjust the weights of each feature parameter according to these losses to perform real-time dynamic control adjustments on the vehicle.

2. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 1, characterized in that: The perception module includes a PVT backbone network and a dual attention feature fusion network; The PVT backbone network includes OPE operation, initial normalization operation, CTB operation and terminal normalization operation; The dual attention feature fusion network includes: Enhance the expression of image features and depth features through self-attention feature enhancement network; The information of image features and depth features is complemented by the cross-attention feature calibration network; The depth-guided feature fusion network guides the depth features and fuses the image features with the depth features.

3. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 2, characterized in that: The self-attention feature enhancement network includes: The image features and deep features Flatten and rearrange into a one-dimensional sequence and Where L = H × W is the length of the sequence, H and W are the height and width of the feature map, and C is the number of channels of the feature; The image sequence f rgb and the depth sequence f hha Obtain query (Q), key (K), and value (V) embeddings through linear projection; Among them, (Q r ,K r ,V r ) and (Q h ,K h ,V h ) are the image sequences f rgb and the depth sequence f hha The corresponding embedding, is the learned projection matrix, D represents the dimension of embedding; The attention weights are calculated using the scaled dot product between Q and K and the weights are used to aggregate the values for each query: Among them, Attention represents the attention weight calculation process, softmax is the activation function, d k It is a scaling factor and is used to prevent the dot product value from being too large to affect the gradient calculation; The self-attention feature enhancement network also includes the use of a multi-head attention mechanism to expand the traditional attention mechanism into N independent attention heads. By capturing the feature relationships at different positions through multiple independent attention heads, the expression ability of the model is enhanced. The formula is as follows: Among them, MultiHead is the multi-head attention calculation process, Concat represents the splicing operation, and denote the output of the i-th attention head in the image sequence and depth sequence respectively, It is the output projection matrix. The splicing operation ensures the diversity characteristics of multi-head attention. A dynamic position encoding generator is designed to dynamically generate spatial position information encoding for the image branch and the depth branch. The dynamic position encoding generator receives the high-resolution, low-channel feature map F output from the first stage of the PVT backbone network as input and generates a dynamic position encoding DPE through the following steps: The input features are expanded to 4 times the original number of channels through a 1×1 convolutional layer. The expanded features are then passed through two convolutional blocks consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function for nonlinear transformation and feature enhancement to extract higher-level spatial information. The features are downsampled to 1 / 8 of their original resolution through nearest neighbor interpolation. The flattened and rearranged features are then used to generate the final dynamic position encoding. The generated dynamic position code is embedded into Q, K and V, and the attention weight distribution is adjusted; Image branch position code P generated by dynamic position code generator r and the deep branch position encoding P h , respectively with (Q r ,K r ,V r ) and (Q h ,K h ,V h ) embedding is combined to calculate the attention weights of the multi-head self-attention mechanism; the attention output generated by these weights is enhanced by residual connection and nonlinear transformation to obtain enhanced image features and deep features The enhanced features are calculated using the following formula: Among them, MultiHead represents the multi-head attention calculation process, which generates context-enhanced feature representation by capturing the global dependencies between different feature positions. MLP is a multi-layer perceptron, which is used to improve the expressiveness of features. It retains the original information of the input features through residual connections and incorporates global context information.

4. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 2, characterized in that: The cross-attention feature calibration network includes: Adopt cross-query generation method: the query Q embedding in the image branch is generated by linear projection of depth features, while the query Q embedding in the depth branch is generated by linear projection of image features; The cross-attention feature calibration network outputs the calibrated image features and deep features The process of feature calibration through the multi-head cross attention mechanism is described by the following formula: Among them, MultiHead represents the multi-head attention calculation process, MLP is a multi-layer perceptron, and It is the enhanced image feature and depth feature output by the self-attention feature enhancement network. and They are and The query, key and value embeddings obtained by linear projection, is the learned projection matrix, D represents the dimension of embedding, (P r ,P h ) is the dynamic position encoding generated by the dynamic position encoding generator. The position encoding explicitly embeds the spatial position information into Q and K, enhancing the attention mechanism to model the spatial relationship of the input features.

5. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 2, characterized in that: The depth-guided feature fusion network includes: Query Q F Embedded by calibrated deep features Generated by linear projection, and the key K F Sum V F The embedding is composed of calibrated image features After linear projection generation, the dynamic position encoding generator is in the fusion process. The deep features are explicitly guided by the image features through query embedding. The fusion feature output of the dynamic position encoding generator is Output F Representing the global information and geometric details of the scene; The following formula is used to describe the process of feature fusion based on the multi-head cross-attention mechanism in the dynamic position encoding generator: in, and is the calibrated image feature and depth feature output by the cross-attention feature calibration network, (Q F ,K F ,V F )yes and The query, key and value embeddings obtained by linear projection, MultiHead represents the multi-head attention calculation process, MLP is a multi-layer perceptron, (P r ,P h ) is a dynamic position code generated by the dynamic position code generator.

6. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 1, characterized in that: The decision-making module includes: The real-time velocity v, high-level navigation command p, and target position g are concatenated to form the measurement input m, which is then processed by the MLP-based measurement encoder to generate the measurement feature f. m : Among them, f m The dimension is 128, representing the dynamic state and navigation information of the vehicle, Concat represents the concatenation operation, and MLP is a multi-layer perceptron; Environmental Characteristics I Feat With the measured characteristic f m The combined feature F is obtained by element-by-element addition; F=I Feat +f m ; The combined feature F is shared by the trajectory prediction branch and the multi-step control prediction branch; In the trajectory planning branch, the input combined feature F is gradually downsampled through the linear layer, and each layer is followed by a ReLU activation function to generate a compact trajectory feature f traj , then the trajectory feature f traj The input is fed into four Gated Recurrent Units (GRUs) to predict the vehicle's path points (w0, w1, w2, w3) for the next four time steps. The generated path points are used by two independent PID controllers to generate longitudinal and lateral control signals, respectively. The longitudinal controller uses the speed reference value of the path point as the target and generates the accelerator or brake signal in combination with the current vehicle speed; the lateral controller uses the direction angle of the path point as the target and generates the steering angle signal. In the multi-step control prediction branch, the control actions for the next four time steps are predicted, and the trajectory-guided attention mechanism is used to enhance the integration of the vehicle's current state and environmental characteristics; The control branch uses the hidden state information of GRU in the trajectory branch. The control output consists of a value head and a strategy head, and multi-step control information is generated through linear downsampling and GRU processing.

7. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 1, characterized in that: The loss estimator includes: The GRU hidden features h and latent feature representation f in the control branch and trajectory branch are used to predict trajectory loss and control signal loss respectively; The loss estimator obtains the predicted values of trajectory loss and control signal loss through the following formula: in, and They are loss estimators E traj and E ctrl Generates the predicted values of trajectory loss and control signal loss, the loss estimator E traj and E ctrl It consists of multiple linear layers, L align To predict the alignment loss, L align The loss estimator is optimized by the mean squared error (MSE) so that the predicted loss is close to the true loss: Among them, L w and L c are the true losses for trajectory prediction and control signal prediction, respectively; The calculation of the control preference value is based on the prediction loss. The priority of each branch is adjusted by dynamically weighting the importance of the trajectory branch and the control branch. The formula is as follows: Among them, k c and k w Is the loss range balance coefficient, which is used to balance the difference in the numerical range of the two losses. c and k w All are set to 1.0; By controlling the preference value Prefer, the output weights of the trajectory branch and the control branch are dynamically adjusted; The final output of the control signal u final The control signals of the trajectory branch and the control branch are combined in a weighted manner. Assume that the output of the trajectory branch is u traj , the control branch output is u ctrl , then the final control signal u final The calculation formula is: u final =Prefer·u ctrl +(1-Prefer)·u traj ; When the predicted trajectory loss is large The preference value Prefer approaches 1, and the control signal generated by the control branch contributes more to the final control signal. On the contrary, when the predicted control loss is large, The preference value Prefer approaches 0, and the control signal generated by the trajectory branch has a greater weight on the final control signal.

8. The end-to-end autonomous driving method based on Transformer multimodal feature fusion according to claim 1, characterized in that: Losses include: Trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align ; Trajectory planning loss L traj It consists of the L1 loss between the predicted waypoints and the true value waypoints and the feature loss, expressed as: Among them, w t and Represent the predicted waypoint and the true waypoint of the tth step, L F Represents feature loss, which is used to measure the current step feature With expert features The L2 distance between them, as an additional supervisory signal, λ F is an adjustable loss weight. For control prediction, L ctrl The loss can be expressed as: Loss function L ctrl The difference between the prediction and the true Beta distribution is measured by KL divergence, which is applicable to the initial moment and future time steps t, and is given by λ F Weighted feature loss improves the learning ability of the model at each time step; RGBD-based 2D semantic segmentation is introduced as an auxiliary task, and the auxiliary loss L aux as follows: L Aux =λ s ·L speed +λ v ·L value +λ seg ·L seg ; The above trajectory planning loss L traj , control prediction loss L ctrl , auxiliary loss L aux and prediction alignment loss L align According to the weight λ traj ,λ ctrl ,λ aux and λ align Calculate the weights and the total loss function is: L=λ traj ·L traj +λ ctrl ·L ctrl +λ aux ·L aux +λ align ·L align 。

Images