Vehicle control method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a vehicle control method and device, electronic equipment, a storage medium and a program product. The control method comprises the steps that at least two current surround-view videos and a current aerial view sequence collected by a vehicle are acquired; generating a target video Token sequence and a target image Token sequence based on the at least two current look-around videos and the current aerial view sequence by using the trained visual language behavior model, and determining a target decision result based on the target video Token sequence and the target image Token sequence; wherein the length of the target video Token sequence is smaller than the length of the original video Token sequence, and the original video Token sequence is obtained by encoding at least two current look-around videos; the length of the target image Token sequence is smaller than that of the original image Token sequence, and the original image Token sequence is obtained by encoding the current aerial view sequence. According to the method, the three-dimensional space perception capability of the model during automatic driving can be enhanced, and the calculation complexity and the reasoning time delay can be reduced.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, specifically to a vehicle control method and device, electronic equipment, storage medium, and program product. Background Technology

[0002] With the development of deep learning technology, especially the advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs), artificial intelligence has made significant progress in understanding complex scenarios and executing decisions. In the field of autonomous driving, VLA (Vision-Language-Action) models are considered one of the key technologies for achieving end-to-end autonomous driving. They aim to directly output driving behavior by understanding visual environmental information and natural language commands surrounding the vehicle. However, traditional VLM models, when applied to autonomous driving, suffer from insufficient three-dimensional spatial perception and high inference latency. Summary of the Invention

[0003] One objective of this application is to provide a vehicle control method to solve the problems of insufficient three-dimensional spatial perception and high inference latency when the VLM model is applied to autonomous driving in related technologies; another objective is to provide a vehicle control device; a third objective is to provide an electronic device; a fourth objective is to provide a computer-readable storage medium; and a fifth objective is to provide a computer program product.

[0004] To achieve the above objectives, this application provides a vehicle control method, the technical solution of which is as follows: Acquire at least two current surround view video and current bird's-eye view sequences captured by the vehicle; Using a trained visual language behavior model, a target video token sequence and a target image token sequence are determined based on the at least two current surround-view videos and the current bird's-eye view sequence. A target decision result is then determined based on the target video token sequence and the target image token sequence, and the target decision result is used to control the vehicle. The length of the target video token sequence is less than the length of the original video token sequence, which is obtained by encoding the at least two current surround-view videos. Similarly, the length of the target image token sequence is less than the length of the original image token sequence, which is obtained by encoding the current bird's-eye view sequence.

[0005] Based on the aforementioned technical methods, firstly, by simultaneously introducing surround-view video and multi-frame BEV (Bird's-Eye View) features as input to the model, the model can more comprehensively and accurately perceive and understand the three-dimensional spatial layout, object relationships, and dynamic changes of the driving scene, greatly enhancing its three-dimensional spatial perception capabilities. Secondly, by efficiently compressing the surround-view video and bird's-eye view sequences through a visual language behavior model, the number of tokens can be significantly reduced, lowering the model's computational complexity and inference latency, making it more suitable for autonomous driving scenarios with extremely high real-time requirements. Simultaneously, because the visual information retains key features and significantly reduces redundant data after encoding and compression, the compressed VLA model can still achieve performance comparable to the uncompressed or lightly compressed model. Finally, by deeply fusing information from different modalities through a visual language behavior model, the rationality and efficiency of the output driving decisions are improved.

[0006] Furthermore, the visual language behavior model includes an encoding and compression module and a large language model; the step of generating a target video token sequence and a target image token sequence based on the at least two current surround-view videos and the current bird's-eye view sequence using the trained visual language behavior model, and determining the target decision result based on the target video token sequence and the target image token sequence, includes: using the encoding and compression module to encode and compress the at least two current surround-view videos and the current bird's-eye view sequence respectively to obtain the target video token sequence and the target image token sequence; and using the large language model to determine the target decision result based on the target video token sequence and the target image token sequence.

[0007] Based on the aforementioned technical means, the number of tokens input into the large language model can be significantly reduced by efficiently compressing the surround-view video and bird's-eye view sequences through the encoding and compression module. By using information from various modalities as input to the large language model, the large language model can achieve more intelligent and flexible autonomous driving behavior control based on understanding environmental information.

[0008] Furthermore, the encoding and compression module includes a first encoding and compression module and a second encoding and compression module; the step of using the encoding and compression module to encode and compress the at least two current surround-view videos and the current bird's-eye view sequence to obtain the target video token sequence and the target image token sequence includes: using the first encoding and compression module to encode and compress the at least two current surround-view videos to obtain the target video token sequence; and using the second encoding and compression module to encode and compress the current bird's-eye view sequence to obtain the target image token sequence.

[0009] Based on the aforementioned technical methods, by introducing a first encoding and compression module and a second encoding and compression module to independently and efficiently encode and compress the surround-view video and bird's-eye view sequences, not only can key information from each modality be extracted more accurately and the fusion efficiency between different modalities be improved, but the computational burden and inference latency of LLM can also be significantly reduced, thereby improving real-time performance and responsiveness. Simultaneously, this modular design facilitates subsequent optimization or replacement of certain compression strategies, thus enhancing flexibility and scalability.

[0010] Furthermore, the first encoding and compression module includes a first encoding module and a first compression module; the step of using the first encoding and compression module to encode and compress the at least two current surround-view videos to obtain the target video token sequence includes: using the first encoding module to encode the at least two current surround-view videos to obtain the original video token sequence; and using the first compression module to compress the original video token sequence to obtain the target video token sequence.

[0011] Based on the above technical means, by completing the encoding and compression operations of visual information in stages, the visual information has a good structured representation before entering the compression module. This helps the compression module to more effectively identify and retain key features, further improve compression efficiency and output quality, and provide efficient and accurate input data for subsequent LLM.

[0012] Furthermore, the first compression module includes multiple learnable video query tokens and a first selector, wherein the number of video query tokens is less than the length of the original video token sequence; the step of compressing the original video token sequence using the first compression module to obtain the target video token sequence includes: using the first selector to select the target video token sequence associated with the multiple video query tokens from the original video token sequence.

[0013] Based on the aforementioned technical means, by using learnable video query tokens and selector mechanisms, the most valuable information for downstream tasks can be extracted adaptively without relying on manual rules, making the compression process more intelligent. This maintains high information fidelity while keeping the number of tokens low, thereby reducing the computational overhead and inference latency of the model.

[0014] Furthermore, the first selector includes at least one cross-attention layer; the step of selecting the target video token sequence associated with the plurality of video query tokens from the original video token sequence using the first selector includes: using the at least one cross-attention layer to determine a first attention score between each video query token and each video token in the original video token sequence, and determining the target video token sequence based on each first attention score.

[0015] Based on the above technical means, by using at least one cross-attention layer, the context information most relevant to the query token can be selected quickly. This not only avoids redundant intra-sequence computation in self-attention, but also significantly improves the model's speed and accuracy in modeling complex relationships, thereby maintaining high information fidelity while keeping the number of tokens low.

[0016] Furthermore, the first encoding module includes a video encoder and a video projection layer; the step of using the first encoding module to encode the at least two current surround-view videos to obtain the original video token sequence includes: using the video encoder to encode the at least two surround-view videos to obtain a video token sequence; and using the video projection layer to perform feature mapping on each token in the video token sequence to obtain the original video token sequence.

[0017] Based on the above technical means, the combination of video encoder and video projection layer can not only significantly improve the coding efficiency and quality of video information, but also achieve efficient representation of video content, enhance its information expression ability before compression, lay a good foundation for subsequent compression operations, and ensure that the compressed token sequence can still retain sufficient semantic information to support high-quality decision output, thereby more effectively supporting subsequent driving behavior decisions.

[0018] Furthermore, the second encoding and compression module includes a second encoding module and a second compression module; the step of using the second encoding and compression module to encode and compress the current bird's-eye view sequence to obtain the target image token sequence includes: using the second encoding module to encode the current bird's-eye view sequence to obtain the original image token sequence; and using the second compression module to compress the original image token sequence to obtain the target image token sequence.

[0019] Based on the above-mentioned technical means, by performing structured information extraction and compression processing on the bird's-eye view sequence in stages, it is possible not only to ensure that important three-dimensional information such as spatial layout is preserved, thereby improving the model's performance in spatial perception, which is especially suitable for tasks that require judging the position and relative distance of objects, but also to reduce the consumption of computing resources and speed up the model's inference speed.

[0020] Furthermore, the second compression module includes multiple learnable image query tokens and a second selector, wherein the number of image query tokens is less than the length of the image token sequence; the step of compressing the original image token sequence using the second compression module to obtain the target image token sequence includes: using the second selector to select the target image token sequence associated with the multiple image query tokens from the original image token sequence.

[0021] Based on the above technical means, by introducing image query tokens and selector mechanisms, representative areas or features in the bird's-eye view can be extracted in a focused manner, thereby preserving the spatial information most helpful to decision-making during the compression process. This not only improves the model's perception ability and decision-making accuracy, but also significantly reduces the model's computational complexity and inference latency, making it particularly suitable for resource-constrained edge devices or application scenarios with high real-time requirements.

[0022] Furthermore, the second selector includes at least one cross-attention layer; the step of selecting the target image token sequence associated with the plurality of image query tokens from the original image token sequence using the second selector includes: using the at least one cross-attention layer to determine a second attention score between each of the image query tokens and each image token in the original image token sequence, and determining the target image token sequence based on each second attention score.

[0023] Based on the above technical means, by using at least one cross-attention layer, the context information most relevant to the query token can be selected quickly. This not only avoids redundant intra-sequence computation in self-attention, but also significantly improves the model's speed and accuracy in modeling complex relationships, thereby maintaining high information fidelity while keeping the number of tokens low.

[0024] Furthermore, the encoding and compression module further includes a language encoding module, and the control method further includes: using the language encoding module to encode the received language instructions to obtain a text token sequence; the step of using the large language model to determine the target decision result based on the target video token sequence and the target image token sequence includes: using the large language model to determine the target decision result based on the target video token sequence, the target image token sequence, and the text token sequence; wherein, the target decision result includes at least one of the following: target driving trajectory, target driving behavior.

[0025] Based on the aforementioned technical means, by introducing a language encoding module, natural language instructions are incorporated into the model's input to expand the model's input dimension. This enables the model to make decisions based on understanding environmental information and further combining user intent, thereby achieving more intelligent and flexible autonomous driving behavior control.

[0026] Furthermore, the control method further includes: determining the number of video query tokens and / or the number of image query tokens in the encoding and compression module based on the vehicle's computing resource information and the current decision task; wherein the target decision result is adapted to the current decision task.

[0027] Based on the aforementioned technical methods, by dynamically adjusting the number of video and image query tokens, not only is efficient compression of visual information achieved, but it can also flexibly adapt to different computing resource conditions and task requirements. While ensuring decision-making performance, it optimizes model operating efficiency and resource utilization, thereby enhancing model stability and controllability. Simultaneously, this dynamic adjustment method avoids unnecessary computational waste, improving overall response speed and stability, and thus better adapting to the diverse scenario requirements and real-time demands of autonomous driving environments.

[0028] Furthermore, the control method further includes: acquiring training samples; wherein the training samples include at least two surround-view videos and bird's-eye view sequences; using the visual language behavior model to be trained, generating a first video token sequence and a first image token sequence based on the at least two surround-view videos and the bird's-eye view sequences, and determining a decision result based on the first video token sequence and the first image token sequence; and updating the parameters of the visual language behavior model at least once based on the decision result to obtain the trained visual language behavior model.

[0029] Based on the aforementioned technical means, the visual language behavior model is trained through an end-to-end training method, enabling the trained visual language model to make accurate vehicle decisions.

[0030] Furthermore, the control method further includes: initializing the visual language behavior model to obtain an initialized visual language behavior model; wherein the initialized visual language behavior model includes an initialized encoding and compression module and an initialized large language model; the step of using the visual language behavior model to be trained, based on the at least two surround-view videos and the bird's-eye view sequence, to generate a first video token sequence and a first image token sequence, and based on the first video token sequence and the first image token sequence, to determine the decision result includes: using the initialized encoding and compression module to encode and compress the at least two surround-view videos and the bird's-eye view sequence respectively to obtain the first video token sequence and the first image token sequence; and using the initialized large language model to determine the decision result based on the first video token sequence and the first image token sequence.

[0031] Based on the aforementioned technical means, a reasonable initialization method can accelerate the model's convergence process and improve the model's overall performance.

[0032] Furthermore, the initialization of the visual language behavior model to obtain the initialized visual language behavior model includes at least one of the following: initializing the target query token based on prior knowledge to obtain the initialized visual language behavior model; wherein the target query token includes at least one video query token among the multiple video query tokens of the encoding and compression module, and / or at least one image query token among the multiple image query tokens of the encoding and compression module; initializing at least one image query token among the multiple image query tokens based on spatial location information to obtain the initialized visual language behavior model.

[0033] Based on the aforementioned technical methods, by introducing prior knowledge or spatial location information to initialize image query tokens and / or video query tokens, the model can be guided to learn effective feature representations more quickly, improving its generalization ability and stability, thereby exhibiting superior performance in practical applications. Simultaneously, it allows visual language behavior models to achieve better performance in fewer training epochs, thus reducing computational resource consumption and accelerating deployment.

[0034] Furthermore, the training samples also include decision labels; the step of updating the parameters of the visual language behavior model at least once based on the decision results to obtain the trained visual language behavior model includes: determining a target loss value based on the decision results and the decision labels; and updating the parameters of the visual language behavior model at least once based on the target loss value to obtain the trained visual language behavior model.

[0035] Based on the aforementioned technical means, determining the target loss value according to the decision result and decision label can improve the accuracy of the target loss value, so as to accurately determine whether the visual language behavior model has converged. The model parameters of the visual language behavior model are updated at least once according to the target loss value. Since the target loss value is determined based on the decision result, the consistency of the prediction of the same sample by the trained visual language behavior model can be improved, thereby enabling the trained visual language behavior model to make vehicle decisions more accurately.

[0036] A vehicle control device, the control device comprising: The acquisition module is used to acquire at least two current surround view videos and current bird's-eye view sequences collected by the vehicle; The processing module is used to utilize a trained visual language behavior model to determine a target video token sequence and a target image token sequence based on the at least two current surround view videos and the current bird's-eye view sequence, and to determine a target decision result based on the target video token sequence and the target image token sequence, wherein the target decision result is used to control the vehicle; The target video token sequence is shorter than the original video token sequence, which is obtained by encoding the at least two current surround view videos; the target image token sequence is shorter than the original image token sequence, which is obtained by encoding the current bird's-eye view sequence.

[0037] An electronic device includes a processor and a memory, the memory storing a computer program executable on the processor, wherein the processor, when executing the computer program, implements any of the methods described above.

[0038] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in any of the preceding claims.

[0039] A computer program product includes a computer program or instructions that, when executed by a processor, implement the method described in any of the preceding claims.

[0040] The beneficial effects of this application are: (1) By simultaneously introducing surround view video and multi-frame BEV features as input to the model, the model can perceive and understand the three-dimensional spatial layout, object relationships, dynamic changes, etc. of the driving scene more comprehensively and accurately, which greatly enhances the perception ability of three-dimensional space.

[0041] (2) By introducing a first encoding and compression module and a second encoding and compression module to independently and efficiently encode and compress the surround-view video and bird's-eye view sequences, not only can the key information of each modality be extracted more accurately and the fusion efficiency between different modalities be improved, but the number of tokens input into the LLM can also be significantly reduced, greatly reducing the computational burden and inference latency of the LLM, thereby improving real-time performance and responsiveness. At the same time, this modular design also facilitates subsequent optimization or replacement of a certain part of the compression strategy, thereby improving flexibility and scalability.

[0042] (3) By completing the encoding and compression of visual information in stages, the visual information has a good structured representation before entering the compression module, which helps the compression module to more effectively identify and retain key features, and further improve compression efficiency and output quality.

[0043] (4) By performing structured information extraction and compression processing on the bird's-eye view sequence in stages, it can not only ensure that important three-dimensional information such as spatial layout is preserved, thereby improving the model's performance in spatial perception, especially suitable for tasks that require judging the position and relative distance of objects, but also reduce the consumption of computing resources and speed up the model's reasoning speed.

[0044] (5) By using a learnable video query token and selector mechanism, the most valuable information for downstream tasks can be extracted adaptively without relying on manual rules, making the compression process more intelligent and maintaining a high information fidelity while keeping the number of tokens low.

[0045] (6) By combining the video encoder and the video projection layer, not only can the coding efficiency and quality of video information be significantly improved, but also the efficient representation of video content can be achieved, enhancing its information expression ability before compression, laying a good foundation for subsequent compression operations, and ensuring that the compressed token sequence can still retain sufficient semantic information to support high-quality decision output.

[0046] (7) By introducing image query token and selector mechanism, representative areas or features in the bird's-eye view can be extracted in a focused manner, thereby preserving the spatial information most helpful to decision-making during the compression process. This not only improves the model's perception ability and decision-making accuracy, but also significantly reduces the model's computational complexity and inference latency. It is especially suitable for resource-constrained edge devices or application scenarios with high real-time requirements.

[0047] (8) Because the key features are retained and redundant data is greatly reduced after the visual information is encoded and compressed, the compressed VLA model can still achieve the performance of uncompressed or lightly compressed models.

[0048] (9) Deeply integrate information from different modalities through a large language model to improve the rationality and efficiency of the output driving decisions.

[0049] (10) By introducing a language encoding module, natural language instructions are incorporated into the model's input to expand the model's input dimension, enabling the model to make decisions based on understanding environmental information and further combining user intent, thereby achieving more intelligent and flexible autonomous driving behavior control.

[0050] (11) By dynamically adjusting the number of video and image query tokens, not only is efficient compression of visual information achieved, but it can also flexibly adapt to different computing resource conditions and task requirements. While ensuring decision-making performance, it optimizes the model's operating efficiency and resource utilization, thereby enhancing the model's stability and controllability. At the same time, this dynamic adjustment method can also avoid unnecessary computational waste, thereby improving the overall response speed and stability, and thus better adapting to the diverse scenario requirements and real-time requirements in the autonomous driving environment.

[0051] (12) Before model training, initializing image query tokens and / or video query tokens by introducing prior knowledge or spatial location information can guide the model to learn effective feature representations more quickly, improve the model's generalization ability and stability, and thus show better performance in practical applications. At the same time, it can enable visual language behavior models to achieve better performance in fewer training rounds. Attached Figure Description

[0052] Figure 1 A schematic diagram illustrating the implementation flow of a vehicle control method provided in an embodiment of this application; Figure 2 A schematic diagram of a visual language behavior model provided in an embodiment of this application; Figure 3 A schematic diagram of a compression module provided in an embodiment of this application; Figure 4This is a schematic diagram of the composition structure of a vehicle control device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0054] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0055] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0056] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] The method provided in this application can be executed by an electronic device, which can be a laptop, tablet, desktop computer, vehicle, set-top box, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] The technical solutions in the embodiments of this application will now be clearly and completely described with reference to the accompanying drawings.

[0060] Figure 1 This is a schematic diagram illustrating the implementation flow of a vehicle control method provided in an embodiment of this application, as shown below. Figure 1 As shown, the control method includes steps S11 and S12, wherein: Step S11: Obtain at least two current surround view videos and current bird's-eye view sequences captured by the vehicle.

[0061] Here, surround-view video refers to video captured in real time by at least two cameras installed around the vehicle, covering 360 degrees of environmental information around the vehicle. For example, the vehicle may be equipped with cameras with seven perspectives: front wide-angle, front telephoto, left front side view, left rear side view, rear-facing view, right rear side view, and right front side view, to cover 360 degrees around the vehicle. In implementation, each camera can capture images at a certain frame rate (e.g., 10 FPS). In some implementations, several frames captured within a certain time period can be selected as input to form a continuous video segment (i.e., video). For example, within each time step, the video segment formed by the most recent N (e.g., 4, 6, etc.) frames is used as input for the surround-view video. It is understood that each video stream is continuous in time and space and together constitutes a comprehensive visual perception of the environment around the vehicle.

[0062] A BEV sequence includes at least two frames of BEV. BEV is a perspective that projects three-dimensional spatial information onto a two-dimensional plane, commonly used in autonomous driving systems for global perception of roads, obstacles, etc. A BEV sequence can be generated from multiple frames of BEV through LiDAR, visual sensors, or other methods. These images project the vehicle's surrounding environment from a top-down perspective, providing spatial layout information. In some implementations, BEV can be formed by integrating multi-source data (e.g., images, point clouds, radar, etc.) and using geometric transformations or neural networks to achieve perspective conversion. For example, BEV can be formed by integrating data collected by LiDAR and video frames captured by cameras. In practice, to capture dynamic information, past multi-frame BEVs can be fused as input to the VLA model to help the VLA model more accurately understand road structure, obstacle locations, etc.

[0063] In some implementations, the number of BEVs in the BEV sequence is adapted to the number of video frames in the surround view video and aligned in the time dimension. For example, if the surround view video includes 4 video frames, then the BEV sequence may also include 4 BEV frames, with each BEV frame aligned in the time dimension with one video frame.

[0064] In some implementations, the BEV sequence can be a BEV feature sequence, that is, the BEV sequence includes a sequence of at least two frames of BEV features (e.g., feature maps).

[0065] Step S12: Using the trained visual language behavior model, determine the target video token sequence and the target image token sequence based on at least two current surround view videos and the current bird's-eye view sequence. Based on the target video token sequence and the target image token sequence, determine the target decision result, which is used to control the vehicle. The length of the target video token sequence is less than the length of the original video token sequence, which is obtained by encoding at least two current surround view videos. The length of the target image token sequence is less than the length of the original image token sequence, which is obtained by encoding the current bird's-eye view sequence.

[0066] Here, the VLA model is an end-to-end multimodal artificial intelligence model that integrates visual input, natural language understanding, and action decision-making. The VLA model can process not only visual information (such as surround view video and current bird's-eye view features) but also natural speech commands (such as voice commands or text commands).

[0067] In deep learning, input data can be transformed into a series of vector representations called tokens. Each token represents a local feature or semantic fragment of the original data. For example, a sequence of visual tokens generated after encoding an image can represent the features of different regions or objects in the image.

[0068] Decision outcomes (including target decision outcomes and other decision outcomes mentioned below) may include, but are not limited to, driving trajectories and driving behaviors. Driving behaviors may include, but are not limited to, steering, acceleration, and braking. For example, in complex traffic scenarios, a reasonable driving strategy can be generated based on current road conditions, traffic rules, and user instructions. For instance, by combining pedestrian crossing areas in the surround-view video and pedestrian position information in the current bird's-eye view features, this visual language behavior model can output a deceleration decision.

[0069] In implementation, at least two current surround-view videos and the current bird's-eye view sequence can be input into the VLA to obtain the target decision result. The VLA first encodes and compresses the current bird's-eye view sequence and at least two current surround-view videos to obtain the target video token sequence and the target image token sequence; then, the target decision result is determined based on the target video token sequence and the target image token sequence.

[0070] In this embodiment, firstly, by simultaneously introducing surround-view video and multi-frame BEV features as input to the model, the model can more comprehensively and accurately perceive and understand the three-dimensional spatial layout, object relationships, and dynamic changes of the driving scene, greatly enhancing its three-dimensional spatial perception capabilities. Secondly, by efficiently compressing the surround-view video and bird's-eye view sequences through a visual language behavior model, the number of tokens can be significantly reduced, lowering the model's computational complexity and inference latency, making it more suitable for autonomous driving scenarios with extremely high real-time requirements. Simultaneously, because the visual information retains key features and significantly reduces redundant data after encoding and compression, the compressed VLA model can still achieve the performance of the uncompressed or lightly compressed model. Finally, by deeply fusing information from different modalities through a visual language behavior model, the rationality and efficiency of the output driving decisions are improved.

[0071] In some implementations, the visual language behavior model includes an encoding compression module and a large language model. Step S12 includes steps S121 and S122, wherein: Step S121: Using the encoding and compression module, encode and compress at least two current surround view video and current bird's-eye view sequences respectively to obtain target video token sequence and target image token sequence; Step S122: Using a large language model, determine the target decision result based on the target video token sequence and the target image token sequence.

[0072] Here, the core of the VLA model lies in its internal encoding and compression module and large language model.

[0073] The encoding and compression module is a core component in the VLA model used to reduce the number of visual tokens. It is a neural network structure specifically designed for efficient information extraction and compression of visual information (e.g., surround-view video and BEV sequences). The module is primarily responsible for efficiently compressing the original visual token sequences (i.e., video token sequences and image token sequences) to reduce the amount of data required for subsequent processing.

[0074] In some implementations, the encoding and compression module can encode and compress one modality of information first, and then encode and compress another modality of information. It can also encode and compress different modalities of information simultaneously. Alternatively, it can encode different modalities of information first, and then compress the encoded modalities. In practice, through compression processing, the original token sequence can be compressed into a compact token sequence with reduced length limits but retaining key information.

[0075] For example, the encoding and compression module can first encode the surround-view video to generate the original video token sequence, and then compress the original video token sequence to generate the target video token sequence; then encode the BEV sequence to generate the original image token sequence, and then compress the original image token sequence to generate the target image token sequence.

[0076] For example, the encoding and compression module can encode the surround-view video and the BEV sequence simultaneously to generate the original video token sequence and the target video token sequence; then, it can compress the original video token sequence and the target video token sequence simultaneously to generate the target video token sequence and the target image token sequence.

[0077] For example, the encoding and compression module can first encode the BEV sequence to generate the original image token sequence; then encode the surround view video to generate the original video token sequence; then compress the original image token sequence to generate the target image token sequence; and finally compress the original video token sequence to generate the target video token sequence.

[0078] In some implementations, to accelerate encoding and compression efficiency, the encoding and compression module may include a first encoding and compression module and a second encoding and compression module. The first encoding and compression module is used to process the encoding and compression of surround view video, and the second encoding and compression module is used to process the encoding and compression of BEV sequences.

[0079] In some implementations, the encoding and compression module may include, but is not limited to, an encoding module and a compression module. An encoding module is a neural network module used to convert input features into a high-dimensional feature representation. The design goal of the encoding module is to retain as much original information as possible while transforming it into a compact vector form suitable for model processing. The compression module extracts the most critical information from the original token sequence and forms a shorter target token sequence based on this information. The compression module can significantly reduce the number of tokens while maintaining important information, thereby reducing computational complexity and inference latency.

[0080] LLM (Language Processing) is a language processing system trained on large-scale data, capable of performing various natural language processing tasks, including text generation, question answering, and translation. With a large number of parameters, LLM can handle long sequences of natural language input and perform complex reasoning tasks. As the core decision-making module of the entire vehicle control method, LLM receives compact target video token sequences and compact target image token sequences from the encoding and compression module. It is responsible for deep fusion of the compressed target video token sequences and target image token sequences, ultimately outputting the decision result through deep fusion of information from different modalities.

[0081] As a core component of multimodal fusion and decision-making, the LLM receives a target video token sequence from the first encoding and compression module and a target image token sequence from the second encoding and compression module. By deeply fusing these two token sequences from different modalities, the LLM can learn the correlation between visual and spatial information, thereby generating accurate driving decisions.

[0082] LLMs can be various neural network models capable of reasoning and driving decision-making, such as GPT, DeepSeek, Qwen series, and LLaMA series. In some implementations, the LLM structure can be based on the Transformer architecture, possessing powerful semantic modeling and context-aware capabilities.

[0083] The target decision result is the driving behavior suggestion ultimately output by the LLM based on the fused information, such as the target driving trajectory and target driving behavior. In implementation, the vehicle can perform corresponding actions based on the target decision result, thereby achieving end-to-end autonomous driving control. In some implementations, this target decision result is adapted to the current decision task. The current decision task refers to the specific type of operation currently being performed by the autonomous driving system, such as trajectory prediction or driving behavior decision-making.

[0084] In this embodiment, the surround-view video and bird's-eye view sequence are efficiently compressed by the encoding compression module, which can significantly reduce the number of tokens input into the large language model. By using information from various modalities as input to the large language model, the large language model can achieve more intelligent and flexible autonomous driving behavior control based on understanding environmental information.

[0085] In some embodiments, the encoding compression module includes a first encoding compression module and a second encoding compression module; step S121 includes steps S1211 and S1212, wherein: Step S1211: Using the first encoding and compression module, encode and compress at least two current surround view videos to obtain the target video token sequence.

[0086] Here, the first encoding compression module is a neural network structure specifically designed for efficient information extraction and compression of multi-view video data (such as surround-view video). The core function of the first encoding compression module is to transform the original high-dimensional video token sequence into a compact video token sequence that is significantly shorter but retains key information. The first encoding compression module can reduce the amount of data required for subsequent processing by large language models, thereby improving inference efficiency.

[0087] In some implementations, the first encoding and compression module may include, but is not limited to, a first encoding module and a first compression module. The first encoding module is a neural network module used to convert multi-frame BEV features into high-dimensional semantic feature representations. The design purpose of the first encoding module is to retain as much original visual information as possible while transforming it into a compact vector form suitable for model processing. The first compression module extracts the most critical information from the original video token sequence and forms a shorter target video token sequence based on this information. The first compression module can significantly reduce the number of tokens while maintaining important information, thereby reducing computational complexity and inference latency. The first compression module can be any suitable module capable of implementing this function.

[0088] Encoding compression refers to filtering out the most critical information for downstream tasks from a large number of input visual tokens and representing this information as a smaller number of tokens. Encoding compression not only reduces computational complexity but also avoids redundant information interfering with decision-making.

[0089] The target video token sequence is a compact video feature sequence output after processing by the first encoding compression module. Although the target video token sequence is much shorter than the original video token sequence, it still retains key visual information about the vehicle's surrounding environment, such as road boundaries, obstacles, and pedestrians, which can effectively support subsequent behavioral decisions.

[0090] Step S1212: Using the second encoding and compression module, the current bird's-eye view sequence is encoded and compressed to obtain the target image Token sequence.

[0091] Here, the second encoding compression module is similar to the first encoding compression module. The second encoding compression module targets the BEV sequence and is used to extract the spatial information most valuable for driving behavior. Since BEV features provide a top-down view of the three-dimensional space surrounding the vehicle, the role of the second encoding compression module is to significantly reduce the dimensionality of the BEV features while retaining the aforementioned spatial layout information, so that they can be quickly input into the LLM for fusion.

[0092] In some implementations, the second encoding and compression module may include, but is not limited to, a second encoding module and a second compression module. The second encoding module is a neural network module used to convert multi-frame BEV features into high-dimensional semantic feature representations. The design objective of the second encoding module is to retain as much key semantic information as possible from the original BEV image while transforming it into a compact vector form suitable for model processing. The second compression module extracts the most representative feature information from the original image token sequence and forms a shorter target image token sequence based on this information. The second compression module can significantly reduce the number of tokens while preserving important information; the second compression module can be any suitable module capable of achieving this function.

[0093] A BEV sequence is a sequence of BEV features from multiple time frames, reflecting the top-down spatial environment changes perceived by the vehicle at different times. Information from bird's-eye view sequences is crucial for understanding dynamic traffic scenarios, identifying lane markings, and predicting the trajectories of other vehicles.

[0094] The target image token sequence is the output of the second encoding compression module. It is a compact image feature sequence with a much shorter length than the original BEV token sequence. The target image token sequence can contain key spatial information such as the vehicle's location, the positional relationships of surrounding objects, and motion trends, which helps improve the model's ability to understand the 3D environment.

[0095] In this embodiment, by introducing a first encoding compression module and a second encoding compression module to independently and efficiently encode and compress the surround-view video and bird's-eye view sequences, not only can key information from each modality be extracted more accurately and the fusion efficiency between different modalities be improved, but the computational burden and inference latency of LLM can also be significantly reduced, thereby improving real-time performance and responsiveness. Simultaneously, this modular design facilitates subsequent optimization or replacement of certain compression strategies, thus enhancing flexibility and scalability.

[0096] In some embodiments, the first encoding compression module includes a first encoding module and a first compression module; step S1211 includes steps S131 and S132, wherein: Step S131: Using the first encoding module, encode at least two current surround view videos to obtain the original video token sequence.

[0097] Here, the first encoding module is used to convert the input surround-view video data into a high-dimensional visual feature representation. The first encoding module can be, but is not limited to, based on a Vision Transformer (ViT) or other image encoders. The first encoding module can extract key visual information from each camera's viewpoint frame by frame and generate a raw video token sequence with temporal information through temporal pooling operations. In this way, the motion state of key objects such as pedestrians, traffic lights, and road signs can be captured through this video token sequence.

[0098] During implementation, the surround-view video is encoded using a first encoding module, transforming complex visual information into a structured token sequence for easier subsequent processing. This enhances the model's perception of driving scenarios, enabling more accurate identification of key objects and events in the environment, thereby improving the decision-making quality of the autonomous driving system.

[0099] Step S132: Use the first compression module to compress the original video token sequence to obtain the target video token sequence.

[0100] Here, the first compression module can be a cross-attention-like structure. It extracts the most crucial information from the original video token sequence and forms a shorter target video token sequence based on this information. This first compression module can significantly reduce the number of tokens while preserving important information, thereby reducing computational complexity and inference latency.

[0101] In some implementations, the first compression module may include, but is not limited to, multiple learnable Video Query Tokens (VDTs) and a first selector. The learnable VDTs are vector parameters automatically optimized by the VLA model during training. The VDTs are used to extract key information from the original video token sequence. The learnable VDTs do not have fixed meanings but rather form semantically specific attention focuses after training on large-scale data. The first selector's role is to select the most relevant original token as the output based on the similarity between each VDT and the original token (e.g., through an attention mechanism). In some implementations, the first selector may include at least one cross-attention layer. For example, using the original video token sequence as the key and value, and each VDT as the query, weighted aggregation is performed through at least one cross-attention layer, resulting in a target video token sequence with a length much shorter than the original sequence. For instance, assuming the original video token sequence has 700 elements, after processing by the first compression module, only 7 or fewer key tokens may remain. This not only reduces the burden of subsequent processing but also improves the model's real-time response capability, meeting the low-latency requirements of autonomous driving.

[0102] In practice, the first encoding module transforms the raw video data into a semantically meaningful high-dimensional token sequence. Then, the first compression module further extracts the most crucial information, expressing the same content in a more concise form. Therefore, the first encoding and compression modules can be executed sequentially, and their outputs must maintain consistent data format and semantic coherence to ensure the efficiency and accuracy of the entire processing flow, providing efficient and accurate input data for the subsequent large language model.

[0103] In this embodiment, by performing the encoding and compression operations on visual information in stages, the visual information already has a good structured representation before entering the compression module. This helps the compression module to more effectively identify and retain key features, further improving compression efficiency and output quality, and providing efficient and accurate input data for subsequent LLM.

[0104] In some implementations, the first encoding module includes a video encoder and a video projection layer; step S131 includes steps S1311 and S1312, wherein: Step S1311: Use a video encoder to encode at least two surround-view videos to obtain a video token sequence.

[0105] Here, the video encoder (Vision Encoder) can be a Transformer-based image encoder used to transform input video frames into high-dimensional feature representations. The working principle of the video encoder is similar to ViT: it divides each frame into multiple image blocks, linearly embeds these blocks into the model, and then extracts semantic information from the images through a multi-layer self-attention mechanism. The video encoder can not only capture the content of a single frame but also combine temporal information to generate a sequence of video tokens with temporal relationships. For example, in autonomous driving scenarios, the video stream from each camera is segmented into multiple frames. The video encoder first encodes each frame, then performs pooling operations in the temporal dimension, merging adjacent frames into a single temporal block. This reduces data redundancy and improves computational efficiency. Finally, each camera's video segment is encoded into several video tokens, and all the video tokens from all cameras are combined to generate a video token sequence.

[0106] Step S1312: Using the video projection layer, perform feature mapping on each token in the video token sequence to obtain the original video token sequence.

[0107] Here, the Vision Adapter is a trainable feature mapping layer used to map the high-dimensional visual tokens output by the video encoder to the feature space of the subsequent large language model. Since the visual model and the large language model may typically use different feature dimensions, feature alignment is necessary through the video projection layer to ensure effective fusion of information between different modalities. For example, the video encoder might output 768-dimensional visual tokens, while the large language model might expect 768-dimensional or higher tokens; the video projection layer is responsible for adjusting the dimensions to maintain consistency between the two in subsequent processing.

[0108] The video projection layer performs feature mapping on each token in the original video token sequence. This can be accomplished through simple linear transformations or by introducing more complex nonlinear transformation structures, such as multilayer perceptrons. The video projection layer not only reduces dimensionality but also enhances the consistency between video tokens and large language models, thereby making subsequent multimodal fusion more efficient.

[0109] In this embodiment, the combination of video encoder and video projection layer can not only significantly improve the coding efficiency and quality of video information, but also achieve efficient representation of video content, enhance its information expression ability before compression, lay a good foundation for subsequent compression operations, and ensure that the compressed token sequence can still retain sufficient semantic information to support high-quality decision output, thereby more effectively supporting subsequent driving behavior decisions.

[0110] In some implementations, the first compression module includes multiple learnable video query tokens and a first selector, wherein the number of video query tokens is less than the length of the video token sequence; step S132 includes step S1321, wherein: Step S1321: Using the first selector, select the target video token sequence associated with multiple video query tokens from the original video token sequence.

[0111] Here, VDTs are vector parameters automatically optimized by the model during training, used to extract key information from the original video token sequence. For example, in an autonomous driving scenario, one VDT might learn to focus on the position of vehicles ahead, while another VDT might learn to focus on the status of traffic lights. The VDTs described above are similar to the query process in information retrieval, using a small number of key questions (queries) to extract the most valuable information from a large amount of input. Because the number of VDTs is far less than the number of original tokens, this significantly reduces the amount of data required for subsequent processing, thereby reducing computational complexity and inference latency.

[0112] The first selector's role is to select the most relevant original token as the output based on the similarity between each VDT and the original token. Since the first selector selects the most relevant original token based on the similarity between each VDT and the original token, each VDT can generate a token that condenses the key visual information, and these tokens are ultimately combined to form the target video token sequence.

[0113] In implementation, the core information of the original video content can be represented with a smaller number of tokens through the operation of the first compression module. The first compression module retains the important features of the original data while avoiding interference from redundant information. The operation of the first compression module not only significantly reduces the number of tokens input to the large language model but also improves the model's processing speed and resource utilization. Furthermore, because the attention mechanism can dynamically adjust weights according to different tasks, the structure of the first compression module has good flexibility and adaptability, making it suitable for various autonomous driving scenarios.

[0114] In the embodiments of this application, by using a learnable video query token and selector mechanism, the most valuable information for downstream tasks can be extracted adaptively without relying on manual rules, making the compression process more intelligent. This maintains a high information fidelity while keeping the number of tokens low, thereby reducing the computational overhead and inference latency of the model.

[0115] In some implementations, the first selector includes at least one cross-attention layer; step S1321 includes step S13211, wherein: Step S13211: Using at least one cross-attention layer, determine the first attention score between each video query token and each video token in the original video token sequence, and determine the target video token sequence based on each first attention score.

[0116] Here, the query token is essentially a set of learnable, parameterized "information aggregators". In physical implementation, it can be a vector with dimensions consistent with the model's feature space.

[0117] The first selector uses a cross-attention-like mechanism to match each VDT with each video token in the original video token sequence. Specifically, the first selector calculates the relevance score between the VDT and each original video token (i.e., the first attention score), and then performs a weighted summation of the original video tokens based on these scores to generate a more compact but information-dense token (i.e., a video token in the target video token sequence).

[0118] In implementation, each video query token is input as a query request (Query) and the original video token sequence is input as a key-value pair into the first selector, which then outputs the target video token sequence. This target video token sequence includes the answers obtained from each query request, which is a highly condensed and compact feature representation containing information related to that query request.

[0119] In this embodiment, by using at least one cross-attention layer, the context information most relevant to the query token can be selected quickly. This not only avoids redundant intra-sequence computation in self-attention, but also significantly improves the model's speed and accuracy in modeling complex relationships, thereby maintaining high information fidelity while keeping the number of tokens low.

[0120] In some embodiments, the second encoding compression module includes a second encoding module and a second compression module; step S1212 includes steps S141 and S142, wherein: Step S141: Use the second encoding module to encode the current bird's-eye view sequence to obtain the original image token sequence.

[0121] Here, the second encoding module is a neural network module used to convert multi-frame BEVs into high-dimensional semantic feature representations. The second encoding module can be based on the Transformer architecture, convolutional neural networks, etc. By learning geometric information and semantic features in the BEV space, the second encoding module maps the input BEV sequence into a series of image tokens. Each image token can be seen as an abstract representation of a local region in the image, containing key information such as location, object category, and motion state. For example, in autonomous driving scenarios, the second encoding module might encode pedestrians, traffic lights, lane lines, and other objects around the vehicle into different tokens for subsequent modules to understand and make decisions.

[0122] The original image token sequence refers to the sequence of tokens output by the second encoding module. Due to the high resolution and long time dimension of BEV images, the original image token sequence is often very long, containing a large amount of repetitive or redundant information. This not only increases computational overhead but also affects the model's inference speed. Therefore, further compression processing of the original image token sequence is needed to improve efficiency.

[0123] Step S142: Use the second compression module to compress the original image token sequence to obtain the target image token sequence.

[0124] Here, the second compression module is similar to the first compression module, and it can also be a structure based on a cross-attention mechanism. The core idea of ​​the second compression module is to extract the most representative feature information by interacting with the original image token sequence.

[0125] In some implementations, the second compression module may include, but is not limited to, multiple learnable Image Query Tokens (IQTs) and a second selector. The learnable IQTs are vector parameters automatically optimized by the VLA model during training. The IQTs are used to extract key information from the original image token sequence. The learnable IQTs do not have fixed meanings but rather form semantically specific attention focuses after training on large-scale data. The second selector selects the most relevant original token as the output based on the similarity between each IQT and the original token (e.g., through an attention mechanism). In some implementations, the second selector may include at least one cross-attention layer. For example, the model uses the original image token sequence as the key and value, and each IQT as the query, performing weighted aggregation through at least one cross-attention layer to output a target image token sequence much shorter than the original sequence. This not only reduces the burden of subsequent data processing but also improves the model's real-time response capability, meeting the low-latency requirements of autonomous driving.

[0126] The target image token sequence is the compressed token sequence output by the second compression module. Compared to the original image token sequence, the target image token sequence has a shorter length, thus reducing the computational burden and inference latency of subsequent modules. Simultaneously, due to the attention-based compression strategy, the target image token sequence can still retain information crucial to downstream tasks (such as driving behavior prediction). For example, in autonomous driving scenarios, the target image token sequence may only contain a few tokens related to obstacles ahead or traffic lights, while ignoring irrelevant details in the background.

[0127] In implementation, firstly, the current bird's-eye view sequence is semantically encoded using a second encoding module to generate an original image token sequence containing rich information. Subsequently, the original image token sequence is compressed using an attention-driven process by a second compression module, significantly reducing the data volume while retaining key information. The combination of the second encoding module and the second compression module achieves an efficient and faithful visual representation processing workflow, meeting the real-time and computational efficiency requirements of autonomous driving systems.

[0128] In this embodiment of the application, by performing structured information extraction and compression processing on the bird's-eye view sequence in stages, not only can important three-dimensional information such as spatial layout be preserved, thereby improving the model's performance in spatial perception, which is especially suitable for tasks that require judging the position and relative distance of objects, but it can also reduce the consumption of computing resources and speed up the model's inference speed.

[0129] In some implementations, the second compression module includes multiple learnable image query tokens and a second selector, the number of image query tokens being less than the length of the image token sequence; step S142 includes step S1421, wherein: Step S1421: Using the second selector, select the target image token sequence associated with multiple image query tokens from the original image token sequence.

[0130] Here, IQTs are learnable vectors used to aggregate visual information. Each IQT has the same dimensions as the model's feature space. The parameters of the IQTs are adaptively optimized during the training process to capture key semantics or task-related patterns in the image. For example, in autonomous driving scenarios, one IQT might be specifically designed to focus on visual information about the dynamics of vehicles ahead, while another IQT might be specifically designed to capture visual information about traffic light states. Since the number of IQTs is typically much smaller than the length of the original image token sequence, significant information compression can be achieved. The design of IQTs borrows from the idea of ​​cross-attention mechanisms, enabling a small number of IQTs to efficiently extract important information from a large number of input image tokens.

[0131] The second selector can be an attention-based module responsible for selecting the most relevant sequence of target image tokens based on the similarity between the IQT and the original image tokens. Specifically, the second selector calculates the attention score between the IQT and each original image token, and selects the most representative image tokens to form the target image token sequence based on the score. This attention-based image token selection method can significantly reduce the number of image tokens without losing key information, thereby reducing the computational cost of subsequent processing.

[0132] There is a close data logic relationship between the second selector and the IQT. The IQT defines the key regions or semantic content of interest, while the second selector filters information based on the key regions or semantic content defined by the IQT. The compression module, which includes the second selector and the IQT, ensures that the compressed image token sequence is not only shorter but also retains the key visual information helpful for the decision-making task.

[0133] In this embodiment, by introducing an image query token and selector mechanism, representative areas or features in the bird's-eye view can be extracted in a focused manner, thereby preserving the spatial information most helpful for decision-making during the compression process. This not only improves the model's perception capabilities and decision-making accuracy, but also significantly reduces the model's computational complexity and inference latency, making it particularly suitable for resource-constrained edge devices or application scenarios with high real-time requirements.

[0134] In some implementations, the second selector includes at least one cross-attention layer; step S1421 includes step S14211, wherein: Step S14211: Using at least one cross-attention layer, determine the second attention score between each image query token and each image token in the original image token sequence, and determine the target image token sequence based on each second attention score.

[0135] Here, the query token is essentially a set of learnable, parameterized "information aggregators". In physical implementation, it can be a vector with dimensions consistent with the model's feature space.

[0136] The second selector uses a cross-attention mechanism to match each IQT with each image token in the original image token sequence. Specifically, the second selector calculates the relevance score between the IQT and each original image token (i.e., the second attention score), and performs a weighted summation of the original image tokens based on these scores to generate a more compact but information-dense token (i.e., an image token in the target image token sequence).

[0137] In implementation, each image query token is input as a query request, and the original image token sequence is input as a key into the second selector, which then outputs the target image token sequence. This target image token sequence includes the answers obtained from each query request, which is a compact feature representation that highly condenses the information related to the query request.

[0138] In this embodiment, by using at least one cross-attention layer, the context information most relevant to the query token can be selected quickly. This not only avoids redundant intra-sequence computation in self-attention, but also significantly improves the model's speed and accuracy in modeling complex relationships, thereby maintaining high information fidelity while keeping the number of tokens low.

[0139] In some embodiments, the encoding compression module further includes a language encoding module, and the control method further includes step S123, which includes step S1221, wherein: Step S123: Use the language encoding module to encode the received language instructions to obtain a text token sequence.

[0140] Here, the language encoding module is used to tokenize natural language instructions into text. Its role is to convert user-input natural language instructions into a sequence of text tokens that LLM can understand. The language encoding module can be implemented using an encoder based on a pre-trained language model (such as BERT or GPT), capable of extracting semantic information from the language instructions and mapping this information to a unified feature space to support fusion processing with visual tokens. Thus, by adding a language encoding module to the encoding compression module, the user-provided natural language instructions can be utilized more effectively, enhancing the understanding of the task objective and thereby improving the accuracy and contextual relevance of decisions.

[0141] Language instructions can include, but are not limited to, voice instructions and text instructions.

[0142] A text token sequence is a series of discrete vector representations generated by segmenting and embedding natural language instructions. These vectors contain the semantic information of the language instructions, and text token sequences can interact and be fused with video token sequences and BEV token sequences in the same feature space.

[0143] During implementation, upon receiving a language command, the language encoding module parses the command into a structured text token sequence and passes it to the subsequent LLM for further processing. By incorporating user intent into decision-making, it can make decisions in complex driving scenarios, thereby improving its intelligence level and user experience.

[0144] Step S1221: Using a large language model, determine the target decision result based on the target video token sequence, target image token sequence, and text token sequence; wherein, the target decision result includes at least one of the following: target driving trajectory, target driving behavior.

[0145] Here, the target decision result refers to the final control command or prediction result output by the LLM after integrating multimodal information. The target decision result may include the target driving trajectory (such as vehicle path planning in the next few seconds) and / or the target driving behavior (such as steering, acceleration, braking, etc.), and the specific target decision result depends on the task requirements.

[0146] In implementation, by introducing text token sequences, it is possible not only to make judgments based on visual information but also to combine semantic information from language commands, achieving more accurate decisions that align with user intent. For example, given a language command to avoid an obstacle ahead and pull over, a safe obstacle avoidance path (i.e., the target driving trajectory) can be calculated by combining the current surround-view video and BEV characteristics, and the corresponding driving behavior can be executed according to the avoidance path.

[0147] In this embodiment, by introducing a language encoding module, natural language instructions are incorporated into the model's input to expand the model's input dimension. This enables the model to make decisions based on understanding environmental information and further combining user intent, thereby achieving more intelligent and flexible autonomous driving behavior control.

[0148] In some embodiments, the control method further includes step S15, wherein: Step S15: Based on the vehicle's computing resource information and the current decision task, determine the number of video query tokens and / or the number of image query tokens in the encoding and compression module; wherein the target decision result is adapted to the current decision task.

[0149] Here, computing resource information can refer to the vehicle's hardware configuration, real-time load, etc. For example, the CPU model, GPU model, memory size, and current resource usage. By using this computing resource information to assess the upper limit of allocable computing power during model inference, the number of query tokens can be dynamically adjusted to avoid performance degradation or response delays due to insufficient computing resources.

[0150] The current decision-making task refers to the specific type of operation that the autonomous driving system is currently performing, such as trajectory prediction or driving behavior decision-making. Different decision-making tasks may require different numbers of query tokens, which can be flexibly adjusted to suit different tasks. For example, the number of VQTs in a trajectory prediction task is greater than the number of VQTs in a driving behavior decision-making task.

[0151] VQT is a core parameter in the video token compressor. VQTs are learnable vectors that adaptively focus on the most critical information regions in the video for the downstream decision-making task during training. For example, the first VQT might be dedicated to capturing the trajectory of vehicles ahead, while the second VQT might focus on recognizing road signs. The number of VQTs determines the length of the final compressed video token, thus affecting the model's inference speed and decision accuracy.

[0152] IQT is a core parameter in the BEV Token Compressor, responsible for extracting key spatial information from multiple BEV frames. Since BEV features have a well-defined spatial structure, IQT can be designed to incorporate prior knowledge and / or spatial location for initialization, for example, focusing on specific spatial regions or semantic categories. IQT design can improve the model's perceptual capabilities and robustness in complex 3D scenes.

[0153] In practice, there is a close relationship between computational resource information, the current decision-making task, the encoding and compression module, VQT, IQT, and the target decision result. For example, computational resource information determines the overall computational capability boundary, while the current decision-making task determines the priority of the demand for perceptual information. By adjusting the number of VQTs and IQTs, efficient compression of visual information is achieved, thereby ensuring that the target decision result can be accurately generated and adapted to the current decision-making task under limited resources.

[0154] In some implementations, a correspondence can be pre-established between various decision tasks, various computing resource information, the number of VQTs and the number of IQTs. Based on this correspondence, the number of VQTs and / or the number of IQTs that are adapted to the current decision task and computing resource information can be obtained.

[0155] In some implementations, the number of query tokens, K, is a key hyperparameter that directly determines the degree of information compression and requires a trade-off between model performance, inference speed, and computational resources. In some implementations, a performance-latency curve can be pre-established. This allows setting a minimum acceptable performance threshold and a maximum latency limit, selecting the K value that best meets the latency requirements—essentially finding the point with the highest "cost-effectiveness." During implementation, performance and latency can be determined using computational resource information and the current decision-making task.

[0156] In this embodiment, by dynamically adjusting the number of video and image query tokens, not only is efficient compression of visual information achieved, but it can also flexibly adapt to different computing resource conditions and task requirements. While ensuring decision-making performance, it optimizes model operating efficiency and resource utilization, thereby enhancing model stability and controllability. Simultaneously, this dynamic adjustment method avoids unnecessary computational waste, improving overall response speed and stability, and thus better adapting to diverse scenario requirements and real-time demands in autonomous driving environments.

[0157] In some embodiments, the control method further includes steps S101 to S103, wherein: Step S101: Obtain training samples; wherein, the training samples include at least two surround view videos and bird's-eye view sequences.

[0158] Here, training samples refer to the input data and corresponding label data used to train the VLA model. The input data in the training samples can be extracted from real-time vehicle data or generated by a simulator. Each training sample represents a perceptual input and expected output in a specific scenario.

[0159] The quality of training samples directly affects the training performance of VLA models. High-quality training samples enable VLA models to learn more accurate input-output mapping relationships, thereby improving the performance of VLA models in real-world environments. In some implementations, to ensure the diversity of training samples, different data can be collected from various traffic scenarios, weather conditions, lighting environments, etc., to create training samples.

[0160] Step S102: Using the visual language behavior model to be trained, determine the first video token sequence and the first image token sequence based on at least two surround view video and bird's-eye view sequences, and determine the decision result based on the first video token sequence and the first image token sequence.

[0161] Here, the visual language behavior model is responsible for converting the original video stream into high-dimensional video tokens and transforming BEV features into feature representations suitable for inputting a large language model, as well as compressing the high-dimensional video tokens and the BEV feature representations suitable for inputting a large language model. In implementation, the generation process of the first video token sequence and the first image token sequence can be found in the specific implementation of step S12 described above. The decision outcome may include, but is not limited to, driving trajectory and driving behavior. In implementation, the process for determining this decision outcome can be found in the specific implementation of step S12 described above.

[0162] Step S103: Based on the decision results, update the parameters of the visual language behavior model at least once to obtain the trained visual language behavior model.

[0163] Here, parameter update refers to the process of adjusting the internal parameters of the VLA model. The methods for updating model parameters can include, but are not limited to, at least one of gradient descent, momentum update, and Newton's momentum method. In implementation, those skilled in the art can determine the update method according to actual needs; this application does not impose such limitations.

[0164] In some implementations, based on preset conditions, it can be determined whether the model parameters of the visual language behavior model need to be updated. If updating the model parameters is necessary, an appropriate parameter update algorithm is used to update the model parameters. After updating the model parameters, a new decision result is determined based on the next training sample. Then, based on the new decision result, it is determined whether further updating of the model parameters is needed. If updating the model parameters is not necessary, the finally updated visual language behavior model is determined as the trained visual language behavior model. The preset conditions may include, but are not limited to, at least one of the following: number of iterations, loss value not exceeding a threshold, and loss value convergence.

[0165] In some implementations, the gradients of the parameters in each layer can be calculated using the backpropagation algorithm, and the parameters can be updated using optimizers (such as Adam, SGD, etc.). After each round of training, the VLA model becomes more adapted to the training data, thereby improving the overall performance of the VLA model.

[0166] The frequency and magnitude of parameter updates affect the convergence speed and final performance of the VLA model. In some implementations, a dynamic learning rate can be used, employing a larger learning rate in the early stages of training to quickly reduce the loss value, and then gradually decreasing the learning rate in the later stages to fine-tune the parameters of the VLA model, thereby achieving better training results.

[0167] In this embodiment of the application, the visual language behavior model is trained through an end-to-end training method, so that the trained visual language model can make accurate vehicle decisions.

[0168] In some embodiments, the control method further includes step S100, and step S102 includes steps S1021 and S1022, wherein: Step S100: Initialize the visual language behavior model to obtain the initialized visual language behavior model; wherein, the initialized visual language behavior model includes the initialized encoding and compression module and the initialized large language model.

[0169] Here, initialization refers to the process of setting the internal parameters of the VLA model to initial values. For the VLA model, initialization can include, but is not limited to, assigning values ​​to the parameters of various sub-modules such as the encoder, projection layer, compression module, and large language model. For example, a pre-trained visual language model (such as Qwen2.5-VL) can be used as the base model, and the query tokens in the encoding and compression modules can be randomly initialized on this basis. This ensures that the VLA model has some prior knowledge, thereby accelerating the convergence speed of the subsequent training process.

[0170] The initialization process is closely related to the performance of the VLA model. A reasonable initialization strategy can avoid gradient explosion or vanishing problems in the early stages of training and improve the stability of the VLA model. Furthermore, the initialization process determines the learning starting point of the VLA model at the beginning of training and affects the final generalization ability and convergence efficiency of the model determined by the initialization process.

[0171] Step S1021: Using the initialized encoding and compression module, encode and compress at least two surround view videos and the bird's-eye view sequence respectively to obtain a first video token sequence and a first image token sequence.

[0172] Here, the encoding and compression module is a core component in the VLA model used to reduce the number of visual tokens. In implementation, the process by which this encoding and compression module determines the first video token sequence and the first image token sequence can be found in the specific implementation of step S121 described above.

[0173] Step S1022: Using the initialized large language model, determine the decision result based on the first video token sequence and the first image token sequence.

[0174] Here, the LLM has a large number of parameters, enabling it to process long sequences of natural language input and perform complex reasoning tasks. The LLM is the core decision-making module of the entire vehicle control method. In implementation, the process by which the LLM determines the decision result can be found in the specific implementation of step S121 above.

[0175] In the embodiments of this application, by using a reasonable initialization method, the convergence process of the model can be accelerated and the overall performance of the model can be improved.

[0176] In some embodiments, step S100 includes step S1001 and / or step S1002, wherein: Step S1001: Based on prior knowledge, initialize the target query token to obtain the initialized visual language behavior model; wherein, the target query token includes at least one video query token among multiple video query tokens of the encoding and compression module, and / or at least one image query token among multiple image query tokens of the encoding and compression module.

[0177] Here, prior knowledge refers to feature information obtained before VLA model training through existing data, domain experience, pre-trained models, etc. For example, a pre-trained object detector can be used to extract feature vectors of specific target objects (such as pedestrians, vehicles, traffic lights, etc.), and these feature vectors can be used as initial values ​​for VQT to guide the VLA model to focus on these target objects during the learning process. As another example, feature vectors from a pre-trained segmentation model (such as segmenting lane lines or drivable areas) can be used as initial values ​​for IQT to guide the VLA model to focus on specific semantic targets during the learning process. This can accelerate the convergence speed of the VLA model and improve its recognition accuracy and robustness in autonomous driving scenarios.

[0178] Step S1002: Based on spatial location information, initialize at least one image query token among multiple image query tokens to obtain the initialized visual language behavior model.

[0179] Here, spatial location information refers to the spatial coordinates of objects or areas represented in a BEV. For example, the initial values ​​of IQT can be set to correspond to key areas such as 5-10 meters directly in front of the vehicle, the left blind spot, and the center of intersections. In this way, IQT can be endowed with a certain spatial perception ability and understanding of three-dimensional spatial layout in the early stages of training, and guided to focus more easily on areas that have an important impact on decision-making.

[0180] In implementation, prior knowledge can be used to provide task-related semantic guidance, followed by structured constraints on geometric distribution through spatial location information. Combining these two approaches enables the formation of more reasonable and physically meaningful query token representations in the early stages of VLA model training. This joint initialization strategy helps the VLA model adapt to autonomous driving scenarios more quickly, reduces invalid parameter searches, and thus achieves efficient training and high-precision inference.

[0181] It is understandable that an IQT initialized with prior knowledge is not the same as an IQT initialized with spatial location information.

[0182] In this embodiment, by introducing prior knowledge or spatial location information to initialize image query tokens and / or video query tokens, the model can be guided to learn effective feature representations more quickly, improving its generalization ability and stability, thereby exhibiting superior performance in practical applications. Simultaneously, the visual language behavior model can achieve better performance in fewer training epochs, thus reducing computational resource consumption and accelerating the deployment process.

[0183] In some implementations, the training samples also include decision labels; step S103 includes steps S1031 and S1032, wherein: Step S1031: Determine the target loss value based on the decision results and decision labels.

[0184] Here, the target loss value is a quantitative metric that measures the difference between the VLA model's predictions and the actual labels. In this application, common loss functions, such as mean squared error and cross-entropy loss, can be used to evaluate the prediction accuracy of the VLA model. By calculating the loss value, the current performance level of the VLA model can be determined, providing a basis for subsequent parameter updates.

[0185] The magnitude of the loss value reflects how well the VLA model fits the current training samples. A smaller loss value means that the VLA model's predictions differ less from the true labels, while a larger loss value indicates a greater difference between the VLA model's predictions and the true labels. Through continuous optimization, the VLA model's ability to understand input data and its decision-making accuracy can be gradually improved.

[0186] Step S1032: Based on the target loss value, update the parameters of the visual language behavior model at least once to obtain the trained visual language behavior model.

[0187] Here, the need to update the model parameters of the visual language behavior model can be determined based on the target loss value. For example, the target loss value can be compared with a threshold; if the target loss value is greater than the threshold, the model parameters of the visual language behavior model are updated; if the target loss value is not greater than the threshold, the visual language behavior model is considered the trained visual language behavior model. Alternatively, the target loss value can be compared with the previous target loss value; if the target loss value is greater than the previous target loss value, the model parameters of the visual language behavior model are updated; if the target loss value is approximately equal to the previous target loss value, the visual language behavior model is considered the trained visual language behavior model.

[0188] In this embodiment, determining the target loss value based on the decision result and decision label can improve the accuracy of the target loss value, so as to accurately determine whether the visual language behavior model has converged. The model parameters of the visual language behavior model are updated at least once based on the target loss value. Since the target loss value is determined based on the decision result, the consistency of the prediction of the trained visual language behavior model for the same sample can be improved, thereby enabling the trained visual language behavior model to make vehicle decisions more accurately.

[0189] The technical solutions provided in the embodiments of this application will be described in detail below with reference to specific application scenarios.

[0190] In autonomous driving scenarios, vehicles need to perceive and understand highly dynamic and complex 3D environments in real time. Traditional VLM models face the following challenges when applied to autonomous driving: Insufficient 3D spatial perception: Single or simply stitched video streams cannot fully express the complete 3D spatial information of a driving scene, especially the precise position, distance, and relative relationships of objects around the vehicle. Although surround-view camera systems (such as 7V systems) can provide a 360-degree view around the vehicle, how to effectively utilize this video data with spatial relationships and combine it with 3D spatial representations (such as BEV features) to enhance the model's scene understanding ability is an urgent problem to be solved.

[0191] Computational resource and latency pressures: Autonomous driving systems place extremely high demands on model inference latency. Introducing surround-view video (e.g., from seven camera views) and multi-frame BEV features generates a large number of visual tokens. For example, the video stream from each camera and multi-frame BEV features are encoded into a series of token sequences. When these token sequences are concatenated and input into the VLA model, the total token length becomes enormous. This not only significantly increases the difficulty and computational cost of model training but also makes it difficult for the model to meet the low-latency response requirements of autonomous driving scenarios.

[0192] Multimodal information fusion efficiency: How to efficiently fuse information from different sensors (such as BEV features generated by surround view cameras, radar, lidar, etc.) and different modalities (visual, language) to generate accurate and safe driving decisions is the key to the success of VLA models.

[0193] Therefore, developing a VLA model that can effectively integrate surround-view video and 3D spatial information, while solving the resulting token length explosion problem, to improve model training efficiency and reduce inference latency, is of great significance for promoting the development of autonomous driving technology.

[0194] To achieve the above objectives, this application proposes the following technical solution: A large vision-language-behavior model for autonomous driving, such as Figure 2 As shown, it includes: (1) Multimodal input module 21 Surround view video input (e.g., 7V Videos): Receives surround view video streams from multiple cameras around the vehicle. Each video stream is continuous in time and space and together constitutes a comprehensive visual perception of the vehicle's surroundings.

[0195] Multi-frame BEV Feature Input (End-to-End BEV Feature): Receives a sequence of bird's-eye view (BEV) features that fuses historical multi-frame sensor information (such as radar, lidar, or vision-generated pseudo-lidar). BEV features provide a top-down understanding of the three-dimensional spatial layout of the driving scene. The BEV feature sequence is processed concurrently with the surround-view video. Both the surround-view video and the BEV feature represent the three-dimensional spatial layout: one perpendicular to the ground, and the other parallel to the ground.

[0196] Language command input (optional, User Instruction): Receives user commands or scenario descriptions in natural language. Language commands can be preset or entered by the user; both are supported during use. This can be used in autonomous driving and in user-controlled vehicle interaction.

[0197] (2) Token encoding and compression module 22 (corresponding to the aforementioned encoding and compression module) The video token encoder and projection layer (Vision Encoder + Vision Adapter) (corresponding to the first encoding module mentioned above) encodes each lookaround video stream, converting it into a series of high-dimensional video tokens. The Vision Adapter (corresponding to the video projection layer mentioned above) maps visual features, mapping the image space to the language space through the Vision Adapter.

[0198] BEV Token Projection Layer (BEV Adapter) (corresponding to the second encoding module mentioned above): Encodes multi-frame BEV feature sequences and converts them into a series of high-dimensional BEV tokens.

[0199] Video Token Compressor (corresponding to the first compression module mentioned above): as follows Figure 3 As shown, a compression module 31 based on a cross-attention mechanism is introduced. This module uses a set of learnable query tokens 32 to interact with a large number of original video tokens output by the encoder, thereby compressing the lengthy original video token sequence into a compact video token sequence (corresponding to the aforementioned target video token sequence) that is significantly shortened (e.g., compressed by more than 100 times) but retains key information.

[0200] BEV Token Compressor (corresponding to the second compression module mentioned above): as follows Figure 3As shown, similarly, a compression module based on a cross-attention mechanism is introduced, which uses a set of learnable query tokens32 to interact with a large number of original BEV tokens output by the encoder, compressing the lengthy original BEV token sequence into a compact BEV token sequence (corresponding to the aforementioned target image token sequence) that significantly shortens the length (e.g., compresses by more than 100 times) but retains key 3D spatial information.

[0201] (3) Multimodal fusion and decision-making module 23 (Large Language Model LLM) This module receives a compact video token sequence from a video token compressor, a compact BEV token sequence from a BEV token compressor, and encoded language instruction tokens (if present). It then uses a large language model (LLM) to deeply fuse these tokens from different sources, learning the complex relationships between them.

[0202] Finally, based on the fused representation, driving behavior decisions are output, such as control signals for steering, acceleration, and braking, and a coarse trajectory prediction for the vehicle.

[0203] The following is a deductive illustration using a vehicle equipped with 7 cameras as an example.

[0204] (1) Input module: Surround View Video Input (7V Video): The vehicle is equipped with seven cameras with different viewing angles: front wide-angle, front telephoto, left front side view, left rear side view, direct rear view, right rear side view, and right front side view, covering 360 degrees around the vehicle. Each camera captures video at a certain frame rate (10 FPS). At each time step, the nearest N (N=4) frames are taken as input.

[0205] Multi-frame BEV feature input (End-to-End BEV Feature): BEV features can be extracted from the vehicle's LiDAR and surround-view 7V images through an end-to-end autonomous driving model. To capture dynamic information, a sequence of BEV feature maps from the past M (M=4) frames is fused as input.

[0206] (2) Token encoding and compression module: Video Token Encoder and Projection Layer (Vision Encoder + Vision Adapter): For each video segment from a camera, a ViT-based image encoder encodes each frame and then aggregates them into video tokens through temporal pooling (Time Patch=2). Assuming each camera's video segment generates L_vid tokens after encoding, 7 cameras will generate a total of 7*L_vid video tokens. Subsequently, the video features are passed through the Vision Projection Layer (Vision Adapter) to map the visual model's features to the feature space of the subsequent large language model.

[0207] BEV Token Projection Layer (BEV Adapter): For the BEV feature sequence extracted by the end-to-end model, point-to-point fusion is first performed in the time dimension. Assume that L_bev BEV tokens are generated after encoding multiple frames of BEV features. Then, the BEV features are mapped to the feature space of the subsequent large language model through the BEV projection layer (BEV Adapter).

[0208] The Video Token Compressor contains K_vid learnable query tokens, where K_vid is much smaller than 7*L_vid (e.g., K_vid is (7*L_vid) / 100). The original 7*L_vid video tokens serve as keys and values, and the K_vid query tokens serve as queries, interacting through one or more cross-attention layers. The output of the cross-attention layers is the compressed K_vid compact video tokens.

[0209] Details of the cross-attention mechanism: We can draw inspiration from models like Perceiver and Flamingo, aggregating large-scale input features using a small number of learnable latent variables (i.e., query tokens). Its computation can be simplified as follows: CompressedTokens=CrossAttention(Query_learnable, Key_input, Value_input).

[0210] The BEV Token Compressor, similar to the video token compressor, contains K_bev learnable query tokens, where K_bev is much smaller than L_bev (e.g., K_bev is L_bev / 100). The original L_bev BEV tokens serve as keys and values, and the K_bev query tokens serve as queries, interacting through one or more cross-attention layers. The output of the cross-attention layers is the compressed K_bev compact BEV tokens.

[0211] (3) Multimodal fusion and decision-making module (Large Language Model LLM) Token concatenation and input: The compressed K_vid compact video tokens, K_bev compact BEV tokens, and language instruction tokens (let's say L_lang) encoded by a standard text encoder (such as BERT or GPT) are concatenated to form a unified multimodal token sequence. This multimodal token sequence is then input into a large language model. The large language model learns the deep interactions and dependencies between different modal tokens through multiple layers of self-attention.

[0212] (4) Output layer: Behavioral Decision: For driving behavior prediction tasks, a standard autoregressive decoding method is used to generate a sequence of vehicle control decisions to predict the future, token by token.

[0213] Rough Trajectory: Generates a sequence of future rough trajectories token by token using standard autoregressive decoding.

[0214] This application also provides a training method for the above-mentioned VLA model, including the following steps: Data preparation: Construct a system that includes surround-view video data, corresponding multi-frame BEV feature data, and corresponding driving behavior labels (e.g., operation data of expert drivers).

[0215] Model initialization: Initialize the parameters of each component of the VLA model, including the video encoder and the multimodal fusion and decision module (LLM). The parameters of the LLM are directly initialized using the parameters of an existing large multimodal model (VLM) (such as Qwen2.5-VL). The BEV feature encoder can be initialized using the parameters of an existing end-to-end model. The token compression module is initialized randomly. This approach can accelerate convergence.

[0216] End-to-end training: Surround view video and multi-frame BEV features are input into the model and processed by the token encoding and token compression modules. The compressed tokens and language instruction tokens (if used) are then fed into the multimodal fusion and decision module. Depending on the task type (e.g., decision prediction, coarse trajectory prediction, etc.), the loss function between the model output and the ground truth label is calculated. All learnable parameters of the model, including the query tokens in the token compressor, are optimized end-to-end using the backpropagation algorithm.

[0217] Further explanation regarding Query Tokens in the Token Compressor To further clarify the core mechanism of the Token Compressor module in this application, especially regarding the nature, validity, design considerations, and quantity selection of its Query Tokens, the following detailed explanation is provided: (1) The essence and working principle of query tokens: Query tokens are essentially a set of learnable, parameterized "information aggregators". In physical implementation, it is a vector (Tensor) with the same dimension as the model feature space (if the model feature space is 4096-dimensional, then the dimension of this token is 4096-dimensional). It is not a pre-set template with specific physical meaning, but rather learns adaptively to become a "representative" of a specific semantic or task concept through large-scale data training. Its working principle can be compared to the "query" process in information retrieval: Input Tokens (Key, Value): Represents the "raw database" containing all visual information (multiple video tokens obtained after encoding each surround-view video).

[0218] Query Token: Represents a "query request" that is optimized during training to "ask" the most valuable questions for the final driving decision.

[0219] Cross-attention mechanism: This is the implementation method of the query process. The query token is compared with each input token to calculate a similarity score (attention score), and then a weighted sum is performed on all input tokens based on this score.

[0220] Output (compressed Token): This is the "answer" to the "query request," which is a highly condensed, compact feature representation that contains information related to the query topic.

[0221] For example, after training, one query token might autonomously evolve into a specialized aggregator for "vehicle dynamics ahead," while another might evolve into an aggregator for "traffic light status."

[0222] Regarding the validity and generalization capabilities of the query token, this application's mechanism guarantees its validity and generalization capabilities through the following two points: The learning focuses on "abstract patterns" rather than "concrete instances": the goal of query token learning is to recognize universal visual patterns or concepts. For example, it learns the visual features (specific outlines, movement patterns, etc.) of the abstract concept of "pedestrian," rather than memorizing the clothing of a specific pedestrian in the training set. Therefore, when encountering a pedestrian never seen before in a new scene, as long as they conform to the abstract pattern of "pedestrian," the query token can be successfully activated and its information extracted. This is the core of the generalization ability of deep learning models.

[0223] The constraint of large-scale, diverse data: Model training relies on large datasets containing various weather, lighting, road conditions, and traffic scenarios. This diversity forces query tokens to learn invariant, essential patterns within the scenarios, rather than "coincidences" specific to a particular scenario. If a query token only learns to recognize a specific scenario, it will lead to significant prediction errors (losses) in a large number of other scenarios. These "bad" parameters are corrected during backpropagation optimization.

[0224] (2) Initialization and design considerations for query tokens: To further enhance the stability and controllability of the model, this application also covers more sophisticated initialization design strategies, as some embodiments of this technical solution: (Option 1) Semantic initialization based on prior knowledge: For some query tokens, pre-trained object detectors (such as detecting vehicles, pedestrians, and traffic lights) or segmentation models (such as segmenting lane lines and drivable areas) can be used to extract feature vectors from some typical samples as their initial values. This is equivalent to "informing" these query tokens in advance to focus on specific semantic targets, which can accelerate model convergence and potentially improve performance.

[0225] (Option 2) Structured Initialization Based on Spatial Location: Especially for BEV features, which have a clear spatial structure, some query tokens can be designed to focus on specific areas within the BEV space, such as the "5-10 meter area directly in front of the vehicle," the "left blind spot," and the "center of an intersection." By assigning initial spatial encoding information of these areas to the query tokens, they are guided to learn to perceive specific spatial locations.

[0226] The introduction of these design considerations proves that the query token is not a pure "black box," but can be designed and guided according to engineering needs.

[0227] (3) Selection and Balancing of the Number of Query Tokens (K): The number of query tokens, K, is a key hyperparameter that directly determines the degree of information compression and requires a trade-off between model performance, inference speed, and computational resources. Its selection follows standard engineering practices: Relationship Analysis: 1) Small K: High compression ratio, short input sequences to LLM, fast inference speed, and low computational resource consumption. However, the disadvantage is that it may become an information bottleneck, losing too many details and causing a decrease in model performance. 2) Large K: Low compression ratio, retains more information, and the model performance may be higher. However, the disadvantage is that LLM processes long sequences, resulting in slow inference speed and high computational resource consumption.

[0228] Determination Method: The optimal value of K is not fixed but depends on the specific in-vehicle computing platform and task requirements. It is determined through multiple experiments on a validation dataset. Specifically, a minimum acceptable performance threshold and a maximum latency threshold (e.g., 100 milliseconds for autonomous driving) are set. Then, a performance-latency curve is plotted for a series of K values ​​(e.g., 32, 64, 128, 256). Finally, the K value that best meets the latency requirements is selected, i.e., the point with the highest cost-effectiveness is found.

[0229] The beneficial effects of this application are as follows: Enhanced 3D spatial understanding: By simultaneously incorporating surround-view video and multi-frame BEV features, the model can more comprehensively and accurately perceive and understand the 3D spatial layout, object relationships, and dynamic changes of the driving scene, providing a richer information foundation for subsequent decision-making.

[0230] Significantly reduced computational complexity and latency: Innovative video token compressors and BEV token compressors can drastically compress the length of raw visually relevant tokens (e.g., by more than 100 times), greatly reducing the number of tokens input to subsequent multimodal fusion modules. This not only makes the model easier to train and reduces the demand for computing resources, but also significantly reduces the model's inference latency, making it more suitable for autonomous driving scenarios with extremely high real-time requirements.

[0231] Maintaining excellent performance: Despite significant token compression, the compressor based on a cross-attention mechanism effectively preserves information crucial to downstream tasks, enabling the compressed model to achieve performance comparable to or even better than the uncompressed or lightly compressed model.

[0232] Enhancing model robustness and generalization ability: A more comprehensive understanding of the scene and a more efficient information processing mechanism help improve the model's robustness and generalization ability in various complex and unknown driving scenarios.

[0233] Based on the above embodiments, this application also provides a vehicle control device. Figure 4 This is a schematic diagram of the composition of a vehicle control device provided in an embodiment of this application, as shown below. Figure 4 As shown, the control device 40 includes an acquisition module 41 and a processing module 42, wherein: Acquisition module 41 is used to acquire at least two current surround view videos and current bird's-eye view sequences collected by the vehicle; Processing module 42 is used to utilize the trained visual language behavior model to determine a target video token sequence and a target image token sequence based on at least two current surround-view videos and a current bird's-eye view sequence, and to determine a target decision result based on the target video token sequence and the target image token sequence. The target decision result is used to control the vehicle. The length of the target video token sequence is less than the length of the original video token sequence, which is obtained by encoding at least two current surround-view videos. The length of the target image token sequence is less than the length of the original image token sequence, which is obtained by encoding the current bird's-eye view sequence.

[0234] In some implementations, the visual language behavior model includes an encoding and compression module and a large language model; the processing module 42 is further configured to use the encoding and compression module to encode and compress at least two current surround view video and current bird's-eye view sequences respectively to obtain a target video token sequence and a target image token sequence; and to use the large language model to determine the target decision result based on the target video token sequence and the target image token sequence.

[0235] In some implementations, the encoding compression module includes a first encoding compression module and a second encoding compression module; the processing module 42 is further configured to use the first encoding compression module to encode and compress at least two current surround view videos to obtain a target video token sequence; use the second encoding compression module to encode and compress the current bird's-eye view sequence to obtain a target image token sequence; and use a large language model to determine the target decision result based on the target video token sequence and the target image token sequence.

[0236] In some implementations, the first encoding and compression module includes a first encoding module and a first compression module; the processing module 42 is further configured to use the first encoding module to encode at least two current surround-view videos to obtain an original video token sequence; and use the first compression module to compress the original video token sequence to obtain a target video token sequence.

[0237] In some implementations, the first compression module includes multiple learnable video query tokens and a first selector, wherein the number of video query tokens is less than the length of the original video token sequence; the processing module 42 is further configured to use the first selector to select a target video token sequence associated with the multiple video query tokens from the original video token sequence.

[0238] In some implementations, the first selector includes at least one cross-attention layer; the processing module 42 is further configured to use the at least one cross-attention layer to determine a first attention score between each video query token and each video token in the original video token sequence, and to determine a target video token sequence based on each first attention score.

[0239] In some implementations, the first encoding module includes a video encoder and a video projection layer; the processing module 42 is further configured to use the video encoder to encode at least two surround-view videos to obtain a video token sequence; and use the video projection layer to perform feature mapping on each token in the video token sequence to obtain the original video token sequence.

[0240] In some embodiments, the second encoding and compression module includes a second encoding module and a second compression module; the processing module 42 is further configured to use the second encoding module to encode the current bird's-eye view sequence to obtain the original image token sequence; and use the second compression module to compress the original image token sequence to obtain the target image token sequence.

[0241] In some implementations, the second compression module includes a plurality of learnable image query tokens and a second selector, wherein the number of image query tokens is less than the length of the image token sequence; the processing module 42 is further configured to use the second selector to select a target image token sequence associated with the plurality of image query tokens from the original image token sequence.

[0242] In some implementations, the second selector includes at least one cross-attention layer; the processing module 42 is further configured to utilize the at least one cross-attention layer to determine a second attention score between each image query token and each image token in the original image token sequence, and to determine a target image token sequence based on each second attention score.

[0243] In some implementations, the encoding and compression module further includes a language encoding module. The processing module 42 is also used to encode the received language instructions using the language encoding module to obtain a text token sequence; and to determine the target decision result based on the target video token sequence, the target image token sequence, and the text token sequence using a large language model; wherein the target decision result includes at least one of the following: target driving trajectory, target driving behavior.

[0244] In some implementations, the processing module 42 is further configured to determine the number of video query tokens and / or the number of image query tokens in the encoding and compression module based on the vehicle's computing resource information and the current decision task; wherein the target decision result is adapted to the current decision task.

[0245] In some embodiments, the control device further includes a training module for acquiring training samples; wherein the training samples include at least two surround-view video and bird's-eye view sequences; using the visual language behavior model to be trained, a first video token sequence and a first image token sequence are generated based on the at least two surround-view video and bird's-eye view sequences, and a decision result is determined based on the first video token sequence and the first image token sequence; based on the decision result, the parameters of the visual language behavior model are updated at least once to obtain the trained visual language behavior model.

[0246] In some implementations, the training module is further used to initialize the visual language behavior model to obtain an initialized visual language behavior model; wherein, the initialized visual language behavior model includes an initialized encoding and compression module and an initialized large language model; using the initialized encoding and compression module, at least two surround-view videos and the bird's-eye view sequence are encoded and compressed respectively to obtain a first video token sequence and a first image token sequence; using the initialized large language model, a decision result is determined based on the first video token sequence and the first image token sequence.

[0247] In some implementations, the training module is further used for at least one of the following: initializing the target query token based on prior knowledge to obtain an initialized visual language behavior model; wherein the target query token includes at least one video query token among a plurality of video query tokens of the encoding and compression module, and / or at least one image query token among a plurality of image query tokens of the encoding and compression module; and initializing at least one image query token among a plurality of image query tokens based on spatial location information to obtain an initialized visual language behavior model.

[0248] In some implementations, the training samples also include decision labels; the training module is also used to determine a target loss value based on the decision results and decision labels; and to update the parameters of the visual language behavior model at least once based on the target loss value to obtain the trained visual language behavior model.

[0249] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0250] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0251] This application also provides a vehicle including a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement any of the methods described above.

[0252] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. The computer-readable storage medium can be transient or non-transient.

[0253] This application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement some or all of the steps in any of the above-described methods. The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0254] It should be noted that, Figure 5 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the hardware entity of the electronic device 500 includes: a processor 501, a communication interface 502, and a memory 503, wherein: Processor 501 typically controls the overall operation of electronic device 500.

[0255] Communication interface 502 enables electronic devices to communicate with other terminals or servers via a network.

[0256] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 501 and various modules in the electronic device 500. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 501, the communication interface 502, and the memory 503 can be performed via bus 504.

[0257] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0258] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A control method of a vehicle, characterized by, The control method comprises: acquiring at least two current surround view videos and a current bird's eye view sequence collected by the vehicle; generating a target video Token sequence and a target image Token sequence based on the at least two current surround view videos and the current bird's eye view sequence by using a trained visual language behavior model, and determining a target decision result based on the target video Token sequence and the target image Token sequence, the target decision result being used to control the vehicle. The length of the target video Token sequence is less than the length of an original video Token sequence obtained by encoding the at least two current surround view videos, and the length of the target image Token sequence is less than the length of an original image Token sequence obtained by encoding the current bird's eye view sequence.

2. The control method according to claim 1, characterized by, The visual language behavior model comprises an encoding compression module and a large language model. The generating of the target video Token sequence and the target image Token sequence based on the at least two current surround view videos and the current bird's eye view sequence by using the trained visual language behavior model, and the determining of the target decision result based on the target video Token sequence and the target image Token sequence, comprise: encoding and compressing the at least two current surround view videos and the current bird's eye view sequence respectively by using the encoding compression module to obtain the target video Token sequence and the target image Token sequence; determining the target decision result based on the target video Token sequence and the target image Token sequence by using the large language model.

3. The control method according to claim 2, characterized by, The encoding compression module comprises a first encoding compression module and a second encoding compression module. The encoding and compressing of the at least two current surround view videos and the current bird's eye view sequence respectively by using the encoding compression module to obtain the target video Token sequence and the target image Token sequence, comprise: encoding and compressing the at least two current surround view videos by using the first encoding compression module to obtain the target video Token sequence; encoding and compressing the current bird's eye view sequence by using the second encoding compression module to obtain the target image Token sequence.

4. The control method according to claim 3, characterized by The first encoding compression module comprises a first encoding module and a first compression module. The encoding and compressing of the at least two current surround view videos by using the first encoding compression module to obtain the target video Token sequence, comprise: encoding the at least two current surround view videos by using the first encoding module to obtain an original video Token sequence; compressing the original video Token sequence by using the first compression module to obtain the target video Token sequence.

5. The control method according to claim 4, characterized by The first compression module comprises a plurality of learnable video query tokens and a first selector, a number of the video query tokens being less than a length of the original video token sequence; The compression of the original video token sequence by the first compression module to obtain the target video token sequence comprises: Selecting, by the first selector, the target video token sequence associated with the plurality of video query tokens from the original video token sequence.

6. The control method according to claim 5, characterized by The first selector comprises at least one cross-attention layer; The selection of the target video token sequence associated with the plurality of video query tokens from the original video token sequence by the first selector comprises: Determining, by the at least one cross-attention layer, a first attention score between each of the video query tokens and each of the video tokens in the original video token sequence respectively, and determining the target video token sequence based on each of the first attention scores.

7. The control method according to claim 4, characterized by, The first encoding module comprises a video encoder and a video projection layer; The encoding of the at least two current surround view videos by the first encoding module to obtain the original video token sequence comprises: Encoding, by the video encoder, the at least two surround view videos to obtain a video token sequence; Mapping, by the video projection layer, each token in the video token sequence to obtain the original video token sequence.

8. The control method according to claim 3, characterized by, The second encoding compression module comprises a second encoding module and a second compression module; The encoding compression of the current bird's eye view sequence by the second encoding compression module to obtain the target image token sequence comprises: Encoding, by the second encoding module, the current bird's eye view sequence to obtain an original image token sequence; Compressing, by the second compression module, the original image token sequence to obtain the target image token sequence.

9. The control method according to claim 8, characterized by, The second compression module comprises a plurality of learnable image query tokens and a second selector, a number of the image query tokens being less than a length of the image token sequence; The compression of the original image token sequence by the second compression module to obtain the target image token sequence comprises: Selecting, by the second selector, the target image token sequence associated with the plurality of image query tokens from the original image token sequence.

10. The control method according to claim 9, characterized by, The second selector comprises at least one cross-attention layer; The selection of the target image token sequence associated with the plurality of image query tokens from the original image token sequence by the second selector comprises: The at least one cross-attention layer is used to determine a second attention score between each of the image query tokens and each of the original image tokens, and based on each of the second attention scores, the target image token sequence is determined.

11. The control method according to claim 2, characterized by, The encoding compression module further includes a language encoding module, The control method further includes encoding the received language instruction by using the language encoding module to obtain a text token sequence. The control method further includes:

12. The control method according to claim 2, characterized by The control method further includes: The control method further includes:

13. The control method according to any one of claims 1 to 12, characterized by, Based on the computing resource information of the vehicle and the current decision-making task, the number of video query tokens in the encoding compression module and / or the number of image query tokens in the encoding compression module are determined; wherein the target decision result is adapted to the current decision-making task. The control method further includes: Obtaining training samples; wherein the training samples include at least two surround view videos and a bird's eye view sequence; The visual language behavior model to be trained is used to generate a first video token sequence and a first image token sequence based on the at least two surround view videos and the bird's eye view sequence, and determine a decision result based on the first video token sequence and the first image token sequence. The parameters of the visual language behavior model are updated at least once based on the decision result to obtain the trained visual language behavior model.

14. The control method of claim 13, wherein The control method further includes initializing the visual language behavior model to obtain an initialized visual language behavior model; wherein the initialized visual language behavior model includes an initialized encoding compression module and an initialized large language model; The visual language behavior model to be trained is used to generate a first video token sequence and a first image token sequence based on the at least two surround view videos and the bird's eye view sequence, and determine a decision result based on the first video token sequence and the first image token sequence, including: using the initialized encoding compression module to perform encoding compression processing on the at least two surround view videos and the bird's eye view sequence respectively to obtain the first video token sequence and the first image token sequence; using the initialized large language model to determine the decision result based on the first video token sequence and the first image token sequence.

15. The control method according to claim 14, characterized by, The initializing the visual language behavior model to obtain an initialized visual language behavior model comprises at least one of the following: The target query Token comprises at least one of the plurality of video query Tokens of the encoding compression module and / or at least one of the plurality of image query Tokens of the encoding compression module. The at least one of the plurality of image query Tokens is initialized based on spatial position information to obtain the initialized visual language behavior model.

16. The control method according to claim 13, characterized by The training sample further comprises a decision label; The updating the parameters of the visual language behavior model based on the decision result at least once to obtain the trained visual language behavior model comprises: The target loss value is determined based on the decision result and the decision label. The parameters of the visual language behavior model are updated based on the target loss value at least once to obtain the trained visual language behavior model.

17. A control device of a vehicle characterized by comprising: The control device comprises: An acquisition module configured to acquire at least two current surround view videos and a current bird's eye view sequence collected by the vehicle; A processing module configured to generate a target video Token sequence and a target image Token sequence based on the at least two current surround view videos and the current bird's eye view sequence, and determine a target decision result based on the target video Token sequence and the target image Token sequence, the target decision result being used to control the vehicle. The length of the target video Token sequence is less than the length of an original video Token sequence, and the original video Token sequence is obtained by encoding the at least two current surround view videos; and the length of the target image Token sequence is less than the length of an original image Token sequence, and the original image Token sequence is obtained by encoding the current bird's eye view sequence.

18. An electronic device, comprising: A processor and a memory, the memory storing a computer program capable of running on the processor, and the processor implements the method in any one of claims 1 to 16 when executing the computer program.

19. A computer-readable storage medium, characterized in that, A computer program or instructions, which, when executed by a processor, implement the method in any one of claims 1 to 16.

20. A computer program product comprising computer programs or instructions, characterized in that, A computer program or instructions, which, when executed by a processor, implement the method in any one of claims 1 to 16.

Images