High-performance and high-efficiency visual language action model for end-to-end automatic driving, automatic driving method and equipment

Abstract

The invention discloses a high-performance and high-efficiency visual language action model for end-to-end automatic driving, an automatic driving method and equipment, and the method comprises the steps: firstly, proposing a simple space-streaming time sequence Transform, carrying out the interaction of an intelligent agent query and a map query, and carrying out the interaction with a scene feature, so as to model a spatial feature, streaming query is used for spreading historical information frame by frame to efficiently model time sequence features. Secondly, providing a lightweight Transform, adopting efficient skill mixed experts in a feedforward neural network, and sparsely activating part of experts to learn professional driving skills by using a routing network; mixed sequence sparse attention is adopted in an attention layer, the most relevant token sequence is selected by using a routing network, and low important feature contribution and large resource consumption caused by structural deviation and attention averaging are eliminated; low-rank key value joint representation is adopted in a key value cache region, and a key value token is compressed and then cached to eliminate the problem that the reasoning speed is low. And thirdly, providing a diffusion planner, modeling a multi-modal trajectory through diffusion, and outputting an optimal trajectory to solve the planning uncertainty.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent automobile automatic driving, and relates to a high-performance and high-efficiency visual language action model for end-to-end automatic driving, an automatic driving method and equipment. BACKGROUND

[0002] The rapid development of automatic driving technology promotes the progress of the transportation field. The rule-driven paradigm connects the perception, prediction and planning modules in series, and has the problem of error accumulation. The data-driven end-to-end paradigm directly maps the original sensor input to the trajectory or control action, solving the problem of error accumulation. However, due to the lack of logical reasoning ability and the existence of data bias, the interpretability and robust generalization in cross-domain scenarios are poor. The above problems reduce the driving score and lead to a high collision rate, seriously affecting safety.

[0003] The visual language action model based on knowledge-driven large language models is first trained on massive data to obtain world knowledge, and then fine-tuned on specific tasks to obtain specialized models. In cross-domain tasks, it has strong robustness, generalization, transferability, and interpretability. Its structure, which consists of a visual encoder that encodes visual features, a large language model that makes decisions, and an action decoder that generates trajectories, is similar to an autonomous driving system. This provides a new approach to achieving end-to-end autonomous driving that is both good and fast. Recent research has introduced the visual language action model from the field of robotics to the field of autonomous driving. Despite these breakthroughs, applying the visual language action model to end-to-end autonomous driving is not easy, as it faces the challenge of simultaneously achieving high performance and high efficiency. This has severely hindered the development and safe iterative optimization process of large models in both academia and industry. Specifically, first, the large language model based on the Transformer architecture has a huge computational cost when processing multi-modal tokens due to multiple transformations by feedforward neural networks. At the same time, they rely on a unified feedforward neural network to handle all scenarios and driving behaviors, causing the model to favor high-frequency scenarios and making it difficult to handle rare but critical driving operations such as emergency braking or sharp turns. This lack of specialized learning paradigm outputs average driving behavior across multiple scenarios rather than the optimal strategy for the current scenario, limiting its ability to handle complex scenarios. When spatiotemporal features are applied for spatiotemporal enhancement, this error and computational cost are amplified indefinitely. DeepseekR1 improves performance while reducing resource consumption based on a hybrid expert architecture. The hybrid expert architecture has dynamic sparse activation characteristics and fine-grained task adaptability. The former reduces computational cost through sparse activation, while the latter improves specialization by adapting different experts to different tasks. The hybrid expert architecture has great potential in generating professional driving behaviors and reducing computational cost, but its application in the field of autonomous driving is almost non-existent. Second, the standard attention average requires dense computation between the query and all key-value pairs, resulting in a quadratic increase in computational cost and a long training period to focus on important features. At the same time, the average distribution of standard attention causes the model to default to equal feature contributions and assign the same attention weight, making it difficult to highlight important features and affecting decision-making. Third, before attention calculation, tokens are first converted into a large number of high-dimensional keys and values, all of which need to be stored in the key-value cache to speed up reasoning. Therefore, the key-value cache needs to cache a large number of key-value elements, and the large key-value cache presents a catastrophic growth along the time sequence direction, causing a reasoning delay bottleneck. Second, the action decoder of the visual language action model ignores the planning uncertainty caused by environmental changes and driving style differences, and defaults to a deterministic relationship between traffic participants and the environment, outputting a single trajectory, which leads to planning errors. The above problems make it difficult to achieve high performance and high efficiency simultaneously. Therefore, innovation is needed in the underlying architecture to meet both high performance and high efficiency, which is both a research hotspot and a research challenge. SUMMARY

[0004] Against this background, this invention proposes a high-performance and high-efficiency visual language action model for end-to-end autonomous driving. (See attached image) Figure 1 As shown, the overall structure of this invention mainly consists of a spatial-streaming temporal Transformer, a large language model based on a lightweight Transformer, and a diffusion planner. First, to enhance the spatiotemporal information of the visual language action model, a simplified spatial-streaming temporal Transformer is proposed in the visual part. This transforms the model by interacting with dynamic agent queries and static map queries, and by interacting with scene features to obtain spatial information. Streaming queries propagate historical information frame-by-frame, achieving efficient modeling of temporal features and avoiding the exponential increase in computational cost caused by frame-by-frame interaction with images. Second, this invention addresses the issue from three aspects: feedforward neural networks, attention mechanisms, and key-value buffers. In the language part, a lightweight Transformer is proposed as the basic framework for the large language model, as shown in the attached diagram. Figure 2 As shown, Figure 2 In this paper, (a) represents the standard Transformer, and (b) represents the lightweight Transformer of this invention. This invention proposes an efficient skill hybrid expert model by combining a parameter-efficient fine-tuning method in the feedforward neural network layer. It utilizes a routing network to dynamically and sparsely activate some experts to learn professional driving skills for the corresponding scenario while keeping the remaining experts frozen. Activated experts participate in computation with parameter updates, while frozen experts do not participate. This ensures customized driving behavior to avoid averaging driving behavior and thus enhances adaptability in multiple scenarios, while reducing computational costs. This invention combines the ideas of hybrid experts and sparse attention to propose a hybrid sequence sparse attention model. Through a routing network, the query selects the most relevant image token and increases its importance weight, reducing attention to the remaining parts. This avoids attention averaging to highlight important feature contributions and reduces the resource consumption of intensive computation. Simultaneously, it eliminates structural bias by mixing with standard attention. To improve inference speed, this invention does not directly convert tokens into a large number of high-dimensional keys and values ​​and put them into the key-value cache. Instead, it first compresses the tokens into a low-rank key-value joint representation before putting them into the key-value cache, reducing the number of key-value elements in the key-value cache and eliminating the bottleneck problem of slow inference caused by an excessively large key-value cache. Third, in the action part, this invention proposes a diffusion planner, which, based on the multimodal trajectory attributes retained by hybrid experts, models the multimodal trajectory distribution through diffusion display and outputs the optimal trajectory to fully model the planning uncertainty.

[0005] The specific steps of the high-performance and efficient visual language action modeling method proposed in this invention are as follows:

[0006] Step 1: Construct a spatial-streaming temporal Transformer. Spatial and historical information are crucial for planning; however, current visual language action models have poor spatial modeling capabilities and are limited by computational costs, failing to fully utilize temporal information. To address this issue, this invention proposes a simplified spatial-streaming temporal Transformer, the structure of which is shown in the attached figure. Figure 3 This invention combines agent querying with map querying for internal interaction and external interaction with scene features to achieve scene representation and enhance spatial capabilities. It uses streaming queries to transmit historical information frame-by-frame, achieving efficient temporal modeling. It comprises two parts: spatial modeling and temporal modeling, as detailed below:

[0007] Spatial modeling. This invention sets up a learnable initial static map query. And learnable initial dynamic agent query ,in and These represent the corresponding number of queries. This indicates the number of query channels. This invention will... and Piecing together the passages to create a scene query Based on self-attention The present invention will The interaction connects the semantic information of the agent and the map to obtain updated scene queries. Then based on cross attention ,Will With scene features Interact to obtain spatial information and obtain updated scene queries. :

[0008]

[0009]

[0010]

[0011] Timing modeling. This invention defines an end-to-end streaming memory pool. It selectively retains historical information with high confidence, among which It is the number of storage frames. This refers to the number of historical scene queries stored. To efficiently model temporal features, historical information is propagated frame-by-frame through streaming queries, thus avoiding interaction with all historical frames, reducing computational costs, and efficiently transmitting historical information. Firstly, since the target pose and coordinates dynamically change over time, this invention queries the current scene separately. and historical scene query Applying MLN transform:

[0012]

[0013] , It involves querying the transformed historical scene and the current scene. Then, and After aggregation, it can be used as a query. As keys and values, they then interact based on temporal cross-attention:

[0014]

[0015] It is temporal cross attention. This is the updated scene query. Finally, this invention utilizes an MLP (Multilayer Perceptron) to... Convert to image tokens and concatenate and align with text tokens before passing to the LLM (Large Language Model); simultaneously, use Update the streaming memory pool.

[0016] The training objective of the spatial-streaming temporal Transformer incorporates agent loss. and map loss The agent's loss consists of classification focal loss and regression L1 loss:

[0017]

[0018] The map loss consists of classification focal loss and negative log-likelihood regression loss:

[0019]

[0020] Step two: Construct a large language model based on a lightweight Transformer. The Transformer decoder in the large language model mainly consists of a feedforward neural network layer, an attention layer, and a key-value buffer. For the feedforward neural network layer, this invention proposes an efficient skill-hybrid expert architecture; for the attention layer, this invention proposes a hybrid sequence sparse attention; for the key-value buffer, this invention proposes a low-rank key-value joint representation. The forward propagation of the Transformer decoder can be simplified as follows:

[0021]

[0022]

[0023] in, It is the length of the token sequence. It is the decoder layer index. It is the first All tokens output by the layer It is the first The first layer of attention output One token, It is the first The first layer of the feedforward neural network FFN outputs the first layer of the feedforward neural network. Each token. This invention will be described in detail in the order described above.

[0024] (1) High-Efficiency Skill Hybrid Expert. Full parameter fine-tuning based on standard feedforward neural networks leads to average driving behavior and incurs huge computational costs. To simultaneously address these issues, this invention proposes a high-efficiency skill hybrid expert, as shown in the appendix. Figure 4 As shown, the efficient skill hybrid expert possesses dynamic sparse activation characteristics and fine-grained task adaptability. The former achieves the same effect as full parameter fine-tuning by sparsely activating some experts through a routing network, thus reducing computational costs. The latter decouples driving behavior patterns to achieve customized driving skills and address average driving behavior. This invention replaces the feedforward neural network layer with an efficient skill hybrid expert layer, which contains multiple experts. Each expert is a scaled-down version of the feedforward neural network, with different experts responsible for different driving skills. The efficient skill hybrid expert consists of specific skill hybrid experts, global skill hybrid experts, and a routing network.

[0025] Specific skill hybrid expert. Specific skill hybrid experts utilize sparse dynamic activation input from routing networks. Experts specializing in the relevant scenario will then learn corresponding driving skills in a customized manner. Specifically, in the Transformer's... In the layer decoder, a specific skill expert is represented as , Indicates the corresponding index. Routing network. From the routing parameter matrix and Function composition, This was acquired through training and learning. It is the dimension of the input token.

[0026] First, this invention utilizes a routing network. Calculate input Matching score with the expert for each scenario Based on this matching score, the present invention reuses... The function calculates Probability distribution of being assigned to the expert in each scenario :

[0027]

[0028] Second, the present invention is designed with The routing strategy refers to the probability distribution of matching scores from all experts. Selected The probability distribution of the expert with the highest matching score Simultaneously, this invention sets the probability distribution of the remaining expert matching scores to 0 so that they do not participate in the forward computation of the expert layer:

[0029]

[0030] in, This indicates an indicator function; it returns 1 if the condition is true, and 0 otherwise. and They are equal in numerical value.

[0031] Third, the probability distribution of the selected experts The corresponding expert's output value Perform a weighted summation to obtain the output of a hybrid expert with specific skills. :

[0032]

[0033] This means in Only among those with specific skill-mixing expertise are there One non-zero expert participated in the calculation and the remainder was set to 0, therefore It is sparse.

[0034] Global Skill Hybrid Expert. While specific skill hybrid experts perform well in scenarios with distinct features, they may exhibit planning wobbling and overfitting in scenarios with ambiguous features, which weakens the system's robustness. To mitigate this risk and ensure stable performance across diverse scenarios, this invention incorporates a global skill hybrid expert. , This represents the corresponding index. Regardless of how the routing network works, the Global Skill Hybrid Expert remains active and performs the following calculations for each input:

[0035]

[0036] The final output consists of specific skill experts, global skill experts, and residual connection values:

[0037]

[0038] The global skill hybrid expert possesses general capabilities, providing semantic background support across all scenarios and acting as a fallback for specific skill hybrid experts, ensuring that routing networks do not completely fail even when facing ambiguous scenarios. Simultaneously, it provides global regularization, making the results more robust and stable.

[0039] The expert parameter matrix is ​​inherited from the pre-trained parameter matrix of LLM. Inputting multiple frames and fine-tuning all parameters for each expert incurs significant resource consumption. To address this, this invention employs efficient low-rank decomposition of hybrid experts, freezing the expert parameter matrix and training only the low-rank parameter matrix that needs updating, thereby achieving efficient fine-tuning of some parameters. The forward propagation process of the hybrid expert layer is represented as follows:

[0040]

[0041] in, It is an expert parameter matrix inherited from LLM. This is the expert parameter matrix that needs updating. This invention addresses... Perform low-rank decomposition:

[0042]

[0043] The forward propagation process at this point is represented as:

[0044]

[0045] This invention is frozen In this case, backpropagation gradient updates only calculate the low-rank decomposition matrix, thus significantly reducing computational cost. The entire update process of the efficient skill-based hybrid expert layer can be represented as follows:

[0046]

[0047] in, It is approximately equal to the hyperparameter of a constant learning rate.

[0048] Routing network. The routing network is primarily updated via backpropagation using the loss function derived from language modeling. The loss function for language modeling is represented as the cross-entropy loss for predicting the next token:

[0049]

[0050] In the formula, These are the model parameters. Through the above method, the routing network routes the input scenario to the corresponding expert and performs dynamic sparse activation, promoting each expert's learning of specialized driving skills and significantly reducing computational costs.

[0051] The learned routing network may suffer from load imbalance, leading to two problems: 1) Route collapse: the model always selects only a few experts, preventing other experts from being fully trained. 2) Computational bottleneck: experts are distributed across multiple devices, resulting in some devices being overloaded while others are idle, leading to low resource utilization efficiency and increased latency. To address these issues, this invention incorporates a load balancing loss mechanism. It is balanced by expert loss and equipment loss composition:

[0052] 1) Ensure that every expert is fully utilized and avoid neglecting any expert:

[0053]

[0054] In the formula, For expert equilibrium factor parameters, Experts Token Selected frequency, This indicates that all tokens are associated with the expert. The average value of the probability distribution;

[0055] 2) Ensure that the computing load is evenly distributed among multiple devices to avoid any device becoming a bottleneck:

[0056]

[0057] In the formula, This represents the equipment balance factor parameter. Indicates the device index. Indicates the number of devices. Indicates equipment Average usage frequency of all experts Indicates deployment on the device A collection of experts An index representing the experts within the device. Indicates equipment The sum of the probability distributions of all experts.

[0058] (2) Sequence Hybrid Sparse Attention. The attention averaging of standard attention in large language models leads to an inability to highlight important feature contributions, high computational cost, slow training speed, and structural bias. To address these issues, this invention proposes sequence hybrid sparse attention, as shown in the appendix. Figure 5As shown, this invention first divides the entire sequence of image tokens into multiple image token subsequences, and then dynamically and sparsely selects the most relevant image token subsequences for sparse attention computation. The inspiration for sequence-mixed sparse attention comes from hybrid expert architectures and sparse attention; the former is applied to the FFN layer of a Transformer, and the latter to the attention layer of a Transformer. This invention creatively applies the idea of ​​dynamic selection of sparse activations in a hybrid expert architecture to overcome the averaging distribution of standard attention, thereby highlighting the contribution of important features and improving resource efficiency. This invention also mixes sequence-mixed sparse attention with standard attention to maintain structural consistency between pre-training and fine-tuning.

[0059] Standard attention. In standard attention, each query token pays attention to the entire sequence of image tokens, resulting in each query being related to the length... Perform intensive computations on the key token and value token:

[0060]

[0061] in, Indicates the length of the image token sequence. and These represent the key token and the value token, respectively. This indicates a query.

[0062] Sequence Hybrid Sparse Attention. Unlike standard attention, the sequence hybrid sparse attention of this invention benefits from image token sequence partitioning and routing strategies, enabling the query token to selectively focus on image token subsequences, and thus interact with the most informative key-value token subsequences. Specifically, firstly, this invention partitions the image token sequence into subsequences of length [length missing]. The image token sequences are all divided into A continuous sequence of image token subsequences, each subsequence having a length of [length missing]. , No. The value range of each image token subsequence is:

[0063]

[0064] Second, this invention performs sparse attention computation on each query and the selected key-value token subsequence:

[0065]

[0066] in, This represents the set of indices of the selected key-value token. To achieve this, based on... The core idea of ​​this invention is a routing network strategy that allows each query to focus on the most relevant subsequence of image tokens, rather than the entire image sequence. First, this invention calculates a proximity score. Used to measure queries With the Key token subsequence Correlation:

[0067]

[0068] Use query vector and the first Key token subsequence The inner product between the mean pooling operations is calculated. Then, this invention utilizes the routing network. For all intimacy scores implement Select the K sequences with the highest scores:

[0069]

[0070] In the formula, This represents an indicator function, which returns 1 if the condition is true and 0 otherwise. Finally, the set of all selected token subsequences is represented as:

[0071]

[0072] This means Of the routing values, only K non-zero subsequences are selected and participate in the calculation; the unselected subsequences are set to 0 and do not participate in the calculation. It is sparse. The above method can accurately and efficiently capture important image token sequences, highlight the contribution of important features, reduce computational costs, and improve training speed.

[0073] Sequence-mixed sparse attention and standard attention. The sequence-mixed sparse attention of this invention is designed as a replacement module for standard attention, maintaining the same number of parameters. Therefore, to minimize the performance bottleneck caused by structural bias, while maintaining compatibility with the Transformer architecture, this invention mixes sequence-mixed sparse attention with standard attention in multiple decoder layers. For a 24-layer decoder, layers 0 to 2 are set as standard attention layers to fully preserve the original information, while layers 3 to 23 are set as sequence-mixed sparse attention layers for precise sparse attention computation. This approach avoids the functional adaptability degradation caused by structural biases during pre-training and fine-tuning.

[0074] (3) Low-rank key-value joint representation. In large language models, the massive key-value cache leads to severely low inference efficiency during inference. To address this issue, this invention does not directly convert tokens into high-dimensional keys and values ​​for storage. Instead, it first compresses them into a low-rank key-value joint representation before placing them into the key-value cache. Subsequently, dimensionality restoration and attention calculation are performed simultaneously. This method eliminates the bottleneck of the massive key-value cache and improves inference speed, as shown in the appendix. Figure 6 As shown.

[0075] Traditional Transformers typically project the input token into a high-dimensional key-value pair and store the values ​​in a key-value buffer for subsequent standard attention computation. Let... It is the first The token in the th _ The input in layer attention utilizes the projection matrix Generate queries, keys, and values ​​respectively. :

[0076]

[0077]

[0078]

[0079] In the formula, It is the embedded dimension. For the number of attention heads, It is the dimension of each head. They represent the first Queries, keys, and values ​​for each token. Divided into Each attention head is used for subsequent calculations of the multi-head attention mechanism:

[0080]

[0081]

[0082]

[0083] in, They represent the first The token in the th _ Each attention head contains queries, keys, and values. The attention mechanism needs to store what... and .for Standard multi-head attention per layer There are 1 attention head, and the dimension of each attention head is 1. Therefore, each token needs to be cached. Each key and value element.

[0084] To address this, the present invention compresses the token into a low-rank key-value joint representation to store all key-value elements. Specifically, it utilizes low-rank compression techniques to project the token into a low-dimensional latent space:

[0085]

[0086] It is a low-rank key-value union representation, containing all the key and value elements that need to be stored. It is the dimension of the low-rank key-value joint representation. This is the dimension-reduced projection matrix used for compression. Subsequently, the low-rank key-value joint representation is dimension-upgraded to achieve decompression, while attention is calculated simultaneously.

[0087]

[0088]

[0089] , These represent the key values ​​after decompression. , These are the key and value up-dimensional projection matrices, respectively. This invention also performs low-rank compression and restoration on the query to reduce activation memory during training:

[0090]

[0091]

[0092] It is a compressed low-rank query. It is a low-rank query dimension. It is a dimension-reduced projection matrix. This is the query after decompression. It is an up-dimensional projection matrix. During inference, the cache only stores the low-rank key-value joint representation. Therefore, each token contains only Each key-value element. It's important to note that this invention only temporarily changes the key-value caching format; compression and decompression do not affect model performance. (Appendix) Figure 7 The text visually demonstrates compression and decompression, along with a comparison of the corresponding cache sizes. (a) indicates no compression, and (b) indicates compression.

[0093] Step 3: Constructing the diffusion planner. Current visual language action models ignore the uncertainty of planning, leading to high collision rates and low driving scores. To address this issue, this invention proposes a diffusion planner to fully model the multimodal trajectory distribution, thereby generating the trajectory with the highest confidence, as shown in the attached figure. Figure 8As shown, the diffusion planner is mainly divided into a forward diffuser and a backward diffuser. The purpose of the forward diffuser is to gradually inject noise into the original data to generate a noisy trajectory, while the backward diffuser generates a clean multimodal trajectory from the noisy trajectory. During the training phase, forward diffusion is performed first, and then the backward diffusion strategy is learned; during the inference phase, the learned backward diffusion strategy is directly used to generate multimodal trajectories.

[0094] The goal of this invention is to approximate the planned action space as a probability distribution and sample an action from this distribution at each time step to control the vehicle. The first step involves constructing a forward diffuser. The Bench2Drive dataset records various driving trajectories, which this invention uses as expert trajectories. Then, the k-means clustering method is used to group the expert trajectories into a discretized trajectory vocabulary. ,in This indicates the number of trajectory words, each trajectory word Includes a set of waypoints , Indicates the time frame of the plan. express The coordinates and heading angle of the trajectory point at any given time. This invention will... The trajectory is used as the anchor point trajectory, and in Gaussian noise is added to generate noise anchor point trajectories. Then, the noise anchor point trajectory is diffused to the set anchor Gaussian distribution. Achieving a lightweight diffusion process, This represents the index. The second step is to construct the back diffuser. During training, the back diffuser will... As input, a high-level decision-making and planning token is also utilized, which is output from a large language model. Scene characteristics Scenario query for Spatial-Streaming Temporal Transformer And historical planning query of streaming memory pools As a guiding condition for reverse diffusion Therefore, the backdiffusion process is formalized as:

[0095]

[0096] In the formula, Indicates a reverse diffuser. This represents the predicted, denoised trajectory. This represents the classification score of the predicted denoised trajectory.

[0097] In a reverse diffuser, firstly, [the following is done] As key and value, the noise anchor point trajectory Encoded as ,make and Interacting based on cross-attention to obtain advanced guidance on LLM:

[0098]

[0099] This is the updated planning query. Second, it incorporates scene features. Using these as keys and values, the planning query interacts with scene features based on variable cross-attention to obtain scene information:

[0100]

[0101] This is the updated planning query. Third, it will include scenario queries. Using these as keys and values, planning queries and scene information are combined through cross-attention interactions to obtain semantic information about the agent and the map.

[0102]

[0103] This is an updated planning query. Fourth, it includes historical planning queries. Using these as keys and values, the current planning query and historical planning queries interact based on temporal cross-attention to capture historical motion information:

[0104]

[0105] This is an updated planning query. Fifth, enhance using FFN. Trajectory feature representation capability:

[0106]

[0107] This is an updated planning query. On one hand, it is used to update the streaming memory pool; on the other hand, it is passed to... To decode the multimodal trajectory:

[0108]

[0109] Ultimately, this invention selects the trajectory with the highest confidence score. and corresponding scores This invention will use the trajectory closest to the true value. The anchor point noise trajectory is used as a positive sample. The remaining trajectories are used as negative samples. The training objective combines trajectory classification loss and trajectory regression loss:

[0110]

[0111] In the formula, Used to balance The regression loss consists of the following components. The classification loss is composed of binary cross-entropy. This invention decouples the trajectory into velocity track points and geometric track points to obtain the velocity and heading angle respectively.

[0112] Step 4, end-to-end learning.

[0113] Loss function of spatial-streaming temporal Transformer Loss by the agent and map loss composition:

[0114]

[0115] Loss function of large language models Mainly due to cross-entropy loss modeled by language Load balancing losses composition:

[0116]

[0117] The loss function of the diffusion planner is .

[0118] The total loss function of this invention is the weighted sum of all the above loss terms:

[0119]

[0120] in , , These are weighting coefficients, representing the contribution of each loss term.

[0121] Based on the above-mentioned high-performance and efficient visual language action model for constructing end-to-end autonomous driving, this invention also proposes a corresponding model, which is the model built and trained in steps one to four above.

[0122] The present invention also proposes a vehicle-mounted device in which the above-mentioned model is arranged.

[0123] The beneficial effects of this invention are:

[0124] (1) This invention proposes a high-performance and efficient visual language action model for end-to-end autonomous driving. While improving driving score, success rate, collision rate and trajectory error, it significantly reduces computing cost and improves training and inference speed, thus solving the safety problem caused by the imbalance between performance and efficiency.

[0125] (2) This invention proposes a simple spatial-streaming temporal Transformer, which enhances spatial information by enabling intelligent agents and map queries to interact with the environment. The streaming query propagates historical features frame by frame for efficient temporal modeling, solving the problems of weak spatial capabilities and low temporal efficiency.

[0126] (3) This invention proposes a lightweight Transformer large language model, which solves the problems of high average driving behavior and high computational cost by using dynamic sparse activation to efficiently learn professional driving skills through expert hybrid learning; it solves the problems of low contribution of important features, high resource consumption and structural bias caused by attention averaging by using hybrid sequence sparse attention to select the most relevant token sequence; and it solves the problem of slow inference speed by compressing key-value tokens into low-rank key-value joint representation to reduce key-value caching.

[0127] (4) This invention proposes a diffusion planner, which models multimodal trajectories based on planning uncertainty, and solves the problem of low driving scores and high collision rates caused by single trajectories under the deterministic paradigm. Attached Figure Description

[0128] Figure 1 A schematic diagram of the overall structure of a high-performance and efficient visual language action model for end-to-end autonomous driving.

[0129] Figure 2 Comparison of standard Transformer and lightweight Transformer architecture diagrams;

[0130] Figure 3 Spatial-stream temporal Transformer architecture diagram;

[0131] Figure 4 A structure diagram for highly efficient skill-based hybrid experts;

[0132] Figure 5 Sequence Mixed Sparse Attention Structure Diagram;

[0133] Figure 6 Key-value low-rank joint joint representation structure diagram;

[0134] Figure 7 A diagram comparing compression and decompression, and their corresponding cache sizes;

[0135] Figure 8 Diffusion planner structure diagram. Detailed Implementation

[0136] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0137] The specific implementation of this invention is as follows:

[0138] I. Constructing a Spatial-Streaming Temporal Transformer. Spatial and historical information are crucial for planning; however, current visual language action models have poor spatial modeling capabilities and cannot fully utilize temporal information due to computational limitations. To address this issue, this invention proposes a simplified spatial-streaming temporal Transformer, the structure of which is shown in the attached figure. Figure 3 This invention combines agent querying with map querying for internal interaction and external interaction with scene features to achieve scene representation and enhance spatial capabilities. It uses streaming querying to transmit historical information frame by frame to achieve efficient temporal modeling.

[0139] Spatial modeling. This invention sets up a learnable initial static map query. And learnable initial dynamic agent query ,in and These represent the corresponding number of queries. This indicates the number of query channels. This invention will... and Piecing together the passages to create a scene query Based on self-attention The present invention will The interaction connects the semantic information of the agent and the map to obtain updated scene queries. Then based on cross attention ,Will With scene features Interact to obtain spatial information and obtain updated scene queries. :

[0140]

[0141]

[0142]

[0143] Timing modeling. This invention defines an end-to-end streaming memory pool. It selectively retains historical information with high confidence, among which It is the number of storage frames. This refers to the number of historical scene queries stored. To efficiently model temporal features, historical information is propagated frame-by-frame through streaming queries, thus avoiding interaction with all historical frames, reducing computational costs, and efficiently transmitting historical information. Firstly, since the target pose and coordinates dynamically change over time, this invention queries the current scene separately. and historical scene query Applying MLN transform:

[0144]

[0145] , It involves querying the transformed historical scene and the current scene. Then, and After aggregation, it can be used as a query. As keys and values, they then interact based on temporal cross-attention:

[0146]

[0147] It is temporal cross attention. This is the updated scenario query. Finally, this invention utilizes MLP to... Convert to image tokens and concatenate and align with text tokens before passing to the LLM; simultaneously, use Update the streaming memory pool. The training objective of the spatial-streaming temporal Transformer incorporates agent loss. and map loss The agent loss is composed of classification focal loss ( ) and regression L1 loss ( )composition:

[0148]

[0149] Map loss is composed of classification focal loss. and negative log-likelihood regression loss composition:

[0150]

[0151] II. Constructing a Lightweight Transformer-Based Large Language Model (LLM). The Transformer decoder in the large language model mainly consists of a feedforward neural network layer, an attention layer, and a key-value buffer. For the feedforward neural network layer, this invention proposes an efficient skill-based hybrid expert architecture; for the attention layer, this invention proposes a hybrid sequence sparse attention; for the key-value buffer, this invention proposes a low-rank key-value joint representation. The forward propagation of the Transformer decoder can be simplified as follows:

[0152]

[0153]

[0154] in, It is the length of the token sequence. It is the decoder layer index. It is the first All tokens output by the layer feedforward neural network. It is the first The first layer of attention output One token, It is the first All tokens output by the layer attention layer; Attention represents attention. It is the first The first layer of the feedforward neural network FFN outputs the first layer of the feedforward neural network. Each token. This invention will be described in detail in the order described above.

[0155] (1) High-Efficiency Skill Hybrid Expert. Full parameter fine-tuning based on standard feedforward neural networks leads to average driving behavior and incurs huge computational costs. To simultaneously address these issues, this invention proposes a high-efficiency skill hybrid expert, as shown in the appendix. Figure 4 As shown, the efficient skill hybrid expert possesses dynamic sparse activation characteristics and fine-grained task adaptability. The former achieves the same effect as full parameter fine-tuning by sparsely activating some experts through a routing network, thus reducing computational costs. The latter decouples driving behavior patterns to achieve customized driving skills and address average driving behavior. This invention replaces the feedforward neural network layer with an efficient skill hybrid expert layer, which contains multiple experts. Each expert is a scaled-down version of the feedforward neural network, with different experts responsible for different driving skills. The efficient skill hybrid expert consists of specific skill hybrid experts, global skill hybrid experts, and a routing network.

[0156] Specific skill hybrid expert. Specific skill hybrid experts utilize sparse dynamic activation input from routing networks. Experts specializing in the relevant scenario will then learn corresponding driving skills in a customized manner. Specifically, in the Transformer's... In the layer decoder, a specific skill expert is represented as D represents the number of experts with a specific skill. This indicates the corresponding specific expert index. Routing network. From the routing parameter matrix

[0157] and Function composition, This was acquired through training and learning. D is the dimension of the input token, where D is the number of experts with a specific skill. yes The dimension. First, this invention utilizes the routing parameter matrix. Calculate input Matching score with the expert for each scenario Based on this matching score, the present invention reuses... The function calculates Probability distribution of being assigned to the expert in each scenario :

[0158]

[0159] Second, the present invention is designed with The routing strategy refers to selecting from the probability distribution of all expert matching scores. The probability distribution of the expert with the highest matching score Simultaneously, this invention sets the probability distribution of the remaining expert matching scores to 0 so that they do not participate in the forward computation of the expert layer:

[0160]

[0161] in, This indicates an indicator function, if the condition is... If true, the value is 1; otherwise, it is 0. and They are numerically equal. Third, the probability distribution of the selected experts... With corresponding experts Calculated output value Perform a weighted summation to obtain the output of a hybrid expert with specific skills. :

[0162]

[0163] This means in Only among those with specific skill-mixing expertise are there One non-zero expert participated in the calculation and the remainder was set to 0, therefore It is sparse.

[0164] Global Skill Hybrid Expert. While specific skill hybrid experts perform well in scenarios with distinct features, they may exhibit planning wobbling and overfitting in scenarios with ambiguous features, which weakens the system's robustness. To mitigate this risk and ensure stable performance across diverse scenarios, this invention incorporates a global skill hybrid expert. G represents the number of global experts. This represents the index of the corresponding global expert. Regardless of how the routing network works, the global skill hybrid expert remains active and applies a condition to each input. Perform calculations for the feedforward neural network:

[0165]

[0166] This indicates that the t-th token is in the t-th position. The layer's global skill expert output is mixed. The final output consists of the output of the specific skill expert, the output of the global skill expert, and the residual connection value.

[0167]

[0168] The global skill hybrid expert possesses general capabilities, providing semantic background support across all scenarios and acting as a fallback for specific skill hybrid experts, ensuring that routing networks do not completely fail even when facing ambiguous scenarios. Simultaneously, it provides global regularization, making the results more robust and stable.

[0169] The expert parameter matrix is ​​inherited from the pre-trained parameter matrix of LLM. Inputting multiple frames and fine-tuning all parameters for each expert incurs significant resource consumption. To address this, this invention employs efficient low-rank decomposition of hybrid experts, freezing the expert parameter matrix and training only the low-rank parameter matrix that needs updating, thereby achieving efficient fine-tuning of some parameters. The forward propagation process of the hybrid expert layer is represented as follows:

[0170]

[0171] in, It is an expert parameter matrix inherited from LLM. This is the expert parameter matrix that needs updating. This invention addresses... Perform low-rank decomposition:

[0172]

[0173] The forward propagation process at this point is represented as:

[0174]

[0175] This invention is frozen In this case, backpropagation gradient updates only calculate the low-rank decomposition matrix, thus significantly reducing computational cost. The entire update process of the efficient skill-based hybrid expert layer can be represented as follows:

[0176]

[0177] in, It is approximately equal to the hyperparameter of constant learning rate. It is a low-rank dimension that is set. It is the first A low-rank decomposition of a specific skill-specific hybrid expert parameter matrix. It is the first A low-rank decomposition of the global skill hybrid expert parameter matrix.

[0178] Routing networks. Routing networks are primarily updated through backpropagation of the loss function derived from language modeling. The loss function for language modeling... This is represented as the cross-entropy loss for predicting the next token:

[0179]

[0180] In the formula, Representation model (parameters are) Given real tokens Under the condition that the next token is The probability prediction is achieved through the above method. The routing network routes the input scenario to the corresponding expert and performs dynamic sparse activation, facilitating each expert's learning of specialized driving skills and significantly reducing computational costs.

[0181] The learned routing network may suffer from load imbalance, leading to two problems: 1) Route collapse: the model always selects only a few experts, preventing other experts from being fully trained. 2) Computational bottleneck: experts are distributed across multiple devices, resulting in some devices being overloaded while others are idle, leading to low resource utilization efficiency and increased latency. To address these issues, this invention incorporates a load balancing loss mechanism. It is balanced by expert loss and equipment loss composition:

[0182] 1) Ensure that every expert is fully utilized and avoid neglecting any expert:

[0183]

[0184] In the formula, For expert equilibrium factor parameters, Experts Token Selected frequency, This indicates the number of experts selected. This indicates that all tokens are associated with the expert. The average value of the probability distribution;

[0185] 2) Ensure that the computing load is evenly distributed among multiple devices to avoid any device becoming a bottleneck:

[0186]

[0187] In the formula, This represents the equipment balance factor parameter. Indicates the device index. Indicates the number of devices. Indicates equipment The average usage frequency of all experts, f y This indicates the frequency of use by experts. Indicates deployment on the device A collection of experts An index representing the experts within the device. Indicates equipment Probability distribution of all experts The sum of .

[0188] (2) Sequence Hybrid Sparse Attention. The attention averaging of standard attention in large language models leads to an inability to highlight important feature contributions, high computational cost, slow training speed, and structural bias. To address these issues, this invention proposes sequence hybrid sparse attention, as shown in the appendix. Figure 5 As shown, this invention first divides the entire sequence of image tokens into multiple image token subsequences, and then dynamically and sparsely selects the most relevant image token subsequences for sparse attention computation. The inspiration for sequence-mixed sparse attention comes from hybrid expert architectures and sparse attention; the former is applied to the FFN layer of a Transformer, and the latter to the attention layer of a Transformer. This invention creatively applies the idea of ​​dynamic selection of sparse activations in a hybrid expert architecture to overcome the averaging distribution of standard attention, thereby highlighting the contribution of important features and improving resource efficiency. This invention also mixes sequence-mixed sparse attention with standard attention to maintain structural consistency between pre-training and fine-tuning.

[0189] Standard attention. In standard attention, each query token pays attention to the entire sequence of image tokens, resulting in each query being related to the length... Perform intensive computations on the key token and value token:

[0190]

[0191] in, Indicates the length of the image token sequence. and These represent the key token and the value token, respectively. This indicates a query.

[0192] Sequence Hybrid Sparse Attention. Unlike standard attention, the sequence hybrid sparse attention of this invention benefits from image token sequence partitioning and routing strategies, enabling the query token to selectively focus on image token subsequences, and thus interact with the most informative key-value token subsequences. Specifically, firstly, this invention partitions the image token sequence into subsequences of length [length missing]. The image token sequences are all divided into A continuous sequence of image token subsequences, each subsequence having a length of [length missing]. , No. The value range of each image token subsequence is:

[0193]

[0194] Second, this invention performs sparse attention computation on each query and the selected key-value token subsequence:

[0195]

[0196] in, This represents sparse attention in mixed sequences. This represents the set of indices of the selected key-value token. To achieve this, based on... The core idea of ​​this invention is a routing network strategy that allows each query to focus on the most relevant subsequence of image tokens, rather than the entire image sequence. First, this invention calculates a proximity score. Used to measure queries With the Key token subsequence Correlation:

[0197]

[0198] Use query vector Q and the first Key token subsequence The inner product between mean pooling operations is calculated. This represents the inner product operation, and meanpool represents mean pooling. Then, this invention utilizes a routing network. For all intimacy scores implement Select the K sequences with the highest scores:

[0199]

[0200] In the formula, This represents an indicator function, which returns 1 if the condition is true and 0 otherwise. Finally, the set of all selected token subsequences is represented as:

[0201]

[0202] This means Of the routing values, only K non-zero subsequences are selected and participate in the calculation; the unselected subsequences are set to 0 and do not participate in the calculation. It is sparse. The above method can accurately and efficiently capture important image token sequences, highlight the contribution of important features, reduce computational costs, and improve training speed.

[0203] Sequence-mixed sparse attention and standard attention. The sequence-mixed sparse attention of this invention is designed as a replacement module for standard attention, maintaining the same number of parameters. Therefore, to minimize the performance bottleneck caused by structural bias, while maintaining compatibility with the Transformer architecture, this invention mixes sequence-mixed sparse attention with standard attention in multiple decoder layers. For a 24-layer decoder, layers 0 to 2 are set as standard attention layers to fully preserve the original information, while layers 3 to 23 are set as sequence-mixed sparse attention layers for precise sparse attention computation. This approach avoids the functional adaptability degradation caused by structural biases during pre-training and fine-tuning.

[0204] (3) Low-rank key-value joint representation. In large language models, the massive key-value cache leads to severely low inference efficiency during inference. To address this issue, this invention does not directly convert tokens into high-dimensional keys and values ​​for storage. Instead, it first compresses them into a low-rank key-value joint representation before placing them into the key-value cache. Subsequently, dimensionality restoration and attention calculation are performed simultaneously. This method eliminates the bottleneck of the massive key-value cache and improves inference speed, as shown in the appendix. Figure 6 As shown.

[0205] Traditional Transformers typically project the input token into a high-dimensional key-value pair and store the values ​​in a key-value buffer for subsequent standard attention computation. Let... It is the first The token in the th _ The input to the layer attention (i.e., the output of layer l-1) is used with the projection matrix. Generate queries, keys, and values ​​respectively. :

[0206]

[0207]

[0208]

[0209] In the formula, It is the embedded dimension. For the number of attention heads, It is the dimension of each head. They represent the first Queries, keys, and values ​​for each token. Divided into Each attention head is used for subsequent calculations of the multi-head attention mechanism:

[0210]

[0211]

[0212]

[0213] in, They represent the first The token in the th _ Each attention head contains queries, keys, and values. The attention mechanism needs to store what... and .for Standard multi-head attention per layer There are 1 attention head, and the dimension of each attention head is 1. Therefore, each token needs to be cached. Each key and value element.

[0214] To address this, the present invention compresses the token into a low-rank key-value union representation to store all key-value elements. Specifically, it utilizes low-rank compression technology to compress the token h... t Projecting onto a low-dimensional potential space:

[0215]

[0216] It is a low-rank key-value union representation, containing all the key and value elements that need to be stored. It is the dimension of the low-rank key-value joint representation. This is the dimension-reduced projection matrix used for compression. Subsequently, the low-rank key-value joint representation is dimension-upgraded to achieve decompression, while attention is calculated simultaneously.

[0217]

[0218]

[0219] , These represent the key values ​​after decompression. , These are the key and value up-dimensional projection matrices, respectively. This invention also performs low-rank compression and restoration on the query to reduce activation memory during training:

[0220]

[0221]

[0222] It is a compressed low-rank query. It is a low-rank query dimension. It is a dimension-reduced projection matrix. This is the query after decompression. It is an up-dimensional projection matrix. During inference, the cache only stores the low-rank key-value joint representation. Therefore, each token contains only Each key-value element. It's important to note that this invention only temporarily changes the key-value caching format; compression and decompression do not affect model performance. (Appendix) Figure 7 It visually demonstrates compression and decompression, as well as a comparison of the corresponding cache sizes.

[0223] III. Diffusion Planner. Current visual language action models ignore the uncertainty of planning, leading to high collision rates and low driving scores. To address this issue, this invention proposes a diffusion planner to fully model the multimodal trajectory distribution, thereby generating the trajectory with the highest confidence, as shown in the appendix. Figure 8 As shown, the diffusion planner is mainly divided into a forward diffuser and a backward diffuser. The purpose of the forward diffuser is to gradually inject noise into the original data to generate a noisy trajectory, while the backward diffuser generates a clean multimodal trajectory from the noisy trajectory. During the training phase, forward diffusion is performed first, and then the backward diffusion strategy is learned; during the inference phase, the learned backward diffusion strategy is directly used to generate multimodal trajectories.

[0224] The goal of this invention is to approximate the planned action space as a probability distribution and sample an action from this distribution at each time step to control the vehicle. The first step involves constructing a forward diffuser. The Bench2Drive dataset records various driving trajectories, which this invention uses as expert trajectories. Then, the k-means clustering method is used to group the expert trajectories into a discretized trajectory vocabulary. ,in This indicates the number of trajectory words, each trajectory word Includes a set of waypoints , Indicates the time frame of the plan. express The coordinates and heading angle of the trajectory point at any given time. This invention will... The trajectory is used as the anchor point trajectory, and in Gaussian noise is added to generate noise anchor point trajectories. , This indicates the number of noise trajectories. Then, the noise anchor point trajectories are diffused to a predefined anchored Gaussian distribution. Achieving a lightweight diffusion process, This represents the index. The second step is to construct the back diffuser. During training, the back diffuser will... As input, a high-level decision-making and planning token is also utilized, which is output from a large language model. Scene characteristics Scenario query for Spatial-Streaming Temporal Transformer And historical planning query of streaming memory pools As a guiding condition for reverse diffusion Therefore, the backdiffusion process is formalized as:

[0225]

[0226] In the formula, Indicates a reverse diffuser. This represents the predicted, denoised trajectory. This represents the classification score of the predicted denoised trajectory.

[0227] In a reverse diffuser, firstly, [the following is done] As a key value, the noise anchor point trajectory Encoded as ,make and Interacting based on cross-attention to obtain advanced guidance on LLM:

[0228]

[0229] This is the updated planning query for this section.

[0230] Second, scene features Using these as keys and values, the planning query interacts with scene features based on variable cross-attention to obtain scene information:

[0231]

[0232] This is the updated planning query for this section.

[0233] Third, query the scene. Using these as keys and values, planning queries and scene information are combined through cross-attention interactions to obtain semantic information about the agent and the map.

[0234]

[0235] This is the updated planning query for this section.

[0236] Fourth, historical planning queries Using these as keys and values, the current planning query and historical planning queries interact based on temporal cross-attention to capture historical motion information:

[0237]

[0238] This is the updated planning query for this section.

[0239] Fifth, enhance using feedforward neural networks (FFN). Trajectory feature representation capability:

[0240]

[0241] This is the last updated planning query. It is used to update the streaming memory pool on one hand, and to pass it to the multilayer perceptron on the other. To decode the multimodal trajectory:

[0242]

[0243] Ultimately, this invention selects the trajectory with the highest confidence score. and corresponding scores This invention will use the trajectory closest to the true value. The anchor point noise trajectory is used as a positive sample. The remaining trajectories are used as negative samples. The loss function L of the training target. DP It combines trajectory classification loss and trajectory regression loss:

[0244]

[0245] In the formula, These are weighting coefficients used to balance the weights of the components. The regression loss consists of the following components. The classification loss is composed of binary cross-entropy. This invention decouples the trajectory into velocity track points and geometric track points to obtain the velocity and heading angle respectively.

[0246] IV. End-to-end learning. Loss function of spatial-streaming temporal Transformer. Loss by the agent and map loss composition:

[0247]

[0248] Loss function of large language models Mainly due to cross-entropy loss modeled by language Load balancing losses composition:

[0249]

[0250] The loss function of the diffusion planner is The total loss function is the weighted sum of all the loss terms mentioned above:

[0251]

[0252] in , , These are weighting coefficients, representing the contribution of each loss term.

[0253] V. This invention also proposes an end-to-end autonomous driving device, which utilizes the model constructed above in conjunction with onboard cameras and other equipment to realize the high-performance and efficient visual pre-stress action model for end-to-end autonomous driving methods. The autonomous driving method includes the following processes:

[0254] First, the images captured by the vehicle-mounted camera and other visual sensors are encoded by a visual encoder and then input into the spatial-streaming temporal Transformer, which outputs an image token. At the same time, the input commands and vehicle status are encoded by a text encoder and output as text tokens. The image token and text token are input into the streaming memory pool and the content of the streaming memory pool is updated.

[0255] Next, the image token and text token processed by the streaming memory pool are concatenated and aligned and then sent to the large language model of the lightweight Transformer. After processing, the language expression result is output on the one hand, and the planning token is output on the other hand and sent to the diffusion planner. The diffusion planner outputs the optimal trajectory of the vehicle and completes the vehicle action.

[0256] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A high-performance and efficient visual language action model for end-to-end autonomous driving, characterized in that, A. Use an improved spatial-streaming temporal Transformer in the vision part: obtain spatial information by interacting with dynamic agent queries and static map queries and interacting with scene features, and use streaming queries to propagate historical information frame by frame to achieve efficient modeling of temporal features, avoiding the doubling of computational cost caused by interacting with images frame by frame; B. Use a lightweight Transformer framework for the large language model in the language part: In the feedforward neural network layer, an efficient skill hybrid expert is proposed by combining an efficient parameter fine-tuning method. The routing network is used to dynamically and sparsely activate some experts to learn professional driving skills in the corresponding scenario while keeping the remaining experts frozen. Activated experts keep their parameters updated and participate in the calculation, while frozen experts do not participate in the calculation. At the attention layer, combining the ideas of hybrid expert and sparse attention, we propose hybrid sequential sparse attention. Through the routing network, the query selects the most relevant image token and increases its importance weight, while reducing the attention of the remaining parts. At the same time, we eliminate structural bias by mixing it with standard attention. In the key-value cache, the token is first compressed into a low-rank key-value union representation and then put into the key-value cache to reduce the number of key-value elements in the key-value cache and eliminate the slow inference caused by an excessively large key-value cache. C. In the action part, a diffusion planner is used. Based on the multimodal trajectory attributes retained by the hybrid experts, the multimodal trajectory distribution is modeled by diffusion display and the optimal trajectory is output to fully model the planning uncertainty.

2. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 1, characterized in that, In Part A, the improved spatial-stream temporal Transformer includes two parts: spatial modeling and temporal modeling. Spatial Modeling: Setting up a learnable initial static map query And learnable initial dynamic agent query ,in and These represent the corresponding number of queries. Indicates the number of query channels; and Piecing together the passages to create a scene query Based on self-attention The interaction connects the semantic information of the agent and the map to obtain updated scene queries. ; Then based on cross attention With scene features Interact to obtain spatial information and get the latest current scene query. ; Timing modeling: Define an end-to-end streaming memory pool It selectively retains historical information with high confidence, among which It is the number of storage frames. It represents the number of historical scenario queries stored. To efficiently model temporal features, historical information is propagated frame by frame through streaming queries. First, queries for the current scene are performed. and historical scene query Applying MLN transform: , It includes both historical scene queries and current scene queries after the transformation; Then, and After aggregation, it can be used as a query. As keys and values, they interact based on temporal cross-attention: It is temporal cross attention. This is the updated scene query; Finally, using MLP to Convert to image tokens and concatenate and align with text tokens before passing to the LLM; simultaneously, use Update the streaming memory pool.

3. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 2, characterized in that, The training objective of the spatial-streaming temporal Transformer incorporates agent loss. and map loss Agent loss It consists of classification focal loss and regression L1 loss: ； Map loss It consists of classification focal loss and negative log-likelihood regression loss: 。 4. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 3, characterized in that, The efficient skill hybrid expert in Part B has dynamic sparse activation characteristics and fine-grained task adaptability. The former achieves the same effect as full parameter fine-tuning by sparsely activating some experts through a routing network, thereby reducing computational costs. The latter decouples driving behavior patterns to achieve customized driving skills to solve average driving behavior. The efficient skill hybrid expert includes a specific skill hybrid expert, a global skill hybrid expert, and a routing network; wherein: the specific skill hybrid expert uses the routing network to dynamically and sparsely activate the expert corresponding to the scene to which the input belongs, thereby customizing the learning of the corresponding driving skills; The global skill hybrid expert is always active and performs calculations for each input; the routing network routes the input to the corresponding expert based on the input scenario and performs dynamic sparse activation to promote each expert to learn professional driving skills, which are updated through backpropagation of language modeling loss.

5. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 4, characterized in that, The hybrid sequence sparse attention in Part B is as follows: First, the length is The image token sequences are all divided into A continuous sequence of image token subsequences, each subsequence having a length of [length missing]. , No. The value range of each image token subsequence is: Secondly, perform sparse attention computation on each query and the selected key-value token subsequence: in, A set of indices representing the selected key-value token; The selected key-value token subsequence is obtained as follows: First, calculate the intimacy score. Used to measure queries With the Key token subsequence Correlation: From the query vector and the first Key token subsequence The inner product between the mean pooling operations is calculated. Then, using the routing network For all intimacy scores implement Choose the K sequences with the highest scores.

6. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 5, characterized in that, The low-rank key-value joint representation in Part B is as follows: The token is projected into a low-dimensional latent space using low-rank compression techniques: It is a low-rank key-value union representation, containing all the key and value elements that need to be stored. It is the dimension of the low-rank key-value joint representation. It is a dimension-reduced projection matrix used for compression; Subsequently, the low-rank key-value joint representation is increased in dimensionality to achieve decompression, while attention calculation is performed simultaneously: , These represent the key values ​​after decompression. , These are the up-dimensional projection matrices of the keys and values, respectively; This also includes low-rank compression and restoration of queries to reduce activation memory during training: It is a compressed low-rank query. It is a low-rank query dimension. It is a dimension-reduced projection matrix. This is the query after decompression. It is an up-dimensional projection matrix.

7. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 6, characterized in that, The diffusion planner in section C includes a forward diffuser and a reverse diffuser; The forward diffuser uses the k-means clustering method to group expert trajectories into a discrete trajectory vocabulary. ,in This indicates the number of trajectory words, each trajectory word Includes a set of waypoints , Indicates the time frame of the plan. express The coordinates and heading angle of the trajectory point at any given time will... The trajectory is used as the anchor point trajectory and is included in the trajectory vocabulary. Gaussian noise is added to generate noise anchor point trajectories. Then, the noise anchor point trajectory is diffused to the set anchor Gaussian distribution. Achieving a lightweight diffusion process, Indicates an index; The reverse diffuser: will As input, utilize the high-level decision-making and planning token output from the large language model. Scene characteristics Scenario query for Spatial-Streaming Temporal Transformer And historical planning query of streaming memory pools As a guiding condition for reverse diffusion It also includes the following settings: First, As key and value, the noise anchor point trajectory Encoded as ,make and Interacting based on cross-attention to obtain advanced guidance on LLM: This is the updated planning query for this part; Second, scene features Using these as keys and values, the planning query interacts with scene features based on variable cross-attention to obtain scene information: This is the updated planning query for this part; Third, query the scene. Using these as keys and values, planning queries and scene information are combined through cross-attention interactions to obtain semantic information about the agent and the map. This is the updated planning query for this part; Fourth, historical planning queries Using these as keys and values, the current planning query and historical planning queries interact based on temporal cross-attention to capture historical motion information: This is the updated planning query for this part; Fifth, enhance using FFN. Trajectory feature representation capability: This part updates the planning query, which is used to update the streaming memory pool on one hand, and pass it on on the other hand. To decode the multimodal trajectory: Ultimately, the trajectory with the highest confidence score was selected. and corresponding scores .

8. The high-performance and efficient visual language action model for end-to-end autonomous driving according to claim 1, characterized in that, The loss function for end-to-end learning and training of the model is as follows: It is the loss function of the spatial-streaming temporal Transformer, which is composed of the agent's loss. and map loss composition; It is the loss function of the large language model, which consists of cross-entropy loss. and load balancing losses composition; It is the loss function of the diffusion planner; , , It is a weighting coefficient, representing the contribution weight of each loss item.

9. The autonomous driving method for end-to-end autonomous driving using a high-performance and efficient visual language action model according to claim 1, characterized in that, The process includes the following: First, the image acquired by the visual sensor is encoded by the visual encoder and then input into the spatial-streaming temporal Transformer of claim 1. After MLP processing, an image token is output. At the same time, the input command and vehicle status are encoded by the text encoder and a text token is output. The image token and the text token are transmitted to the streaming memory pool of claim 1 and the content of the streaming memory pool is updated. After processing by the streaming memory pool, the image token and text token are concatenated and aligned before being sent to the large language model of the lightweight Transformer as described in claim 1. After processing, the language expression result is output on one hand, and the planning token is output on the other hand and sent to the diffusion planner as described in claim 1. The diffusion planner outputs the optimal trajectory of the vehicle and completes the action.

10. An autonomous driving vehicle device, characterized in that, The device deploys the high-performance and efficient visual language action model for end-to-end autonomous driving as described in claim 1.

Images