Automatic driving method and device based on visual language and medium

Abstract

The invention provides an automatic driving method and device based on a visual language and a medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring a navigation instruction input by a navigation map and multi-view image data acquired by a multi-view sensor; obtaining a visual language vector according to a visual language model in a preset trajectory planning model, the navigation instruction and the multi-view image data; according to an end-to-end model in the trajectory planning model, the navigation instruction, the multi-view image data and the visual language vector, obtaining a planned trajectory of the vehicle; and determining a control signal corresponding to the planned track, and performing automatic driving according to the control signal. According to the method, the processing flow of track prediction in the automatic driving process is simplified, the calculation amount consumption is reduced, the complex road condition can be processed, and the scene coverage degree of automatic driving is improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular, to an autonomous driving method, device, and medium based on vision and language. Background Art

[0002] During the autonomous driving of a vehicle, it is very important to quickly predict the vehicle's trajectory in the next few seconds according to the driving scenario.

[0003] In the related art, a trajectory prediction module and a control module are usually set in an in-vehicle computing device. The trajectory prediction module can perform trajectory prediction according to the navigation instructions of the navigation map and the image data collected by a multi-view camera to obtain the planned trajectory of the vehicle in the next few seconds. The control module can generate corresponding control signals according to the planned trajectory so that the vehicle can perform autonomous driving according to the planned trajectory and the control signals.

[0004] However, the calculation process of the above process is relatively complex, the amount of calculation consumed is large, and the model in the trajectory prediction module is relatively simple and cannot handle complex road conditions, resulting in a low scene coverage. Summary of the Invention

[0005] The embodiments of this application provide an autonomous driving method, device, and medium based on vision and language, which can simplify the processing flow of trajectory prediction, reduce the amount of calculation consumed, and can handle complex road conditions, improving the scene coverage of autonomous driving.

[0006] In a first aspect, the embodiments of this application provide an autonomous driving method based on vision and language, including:

[0007] Obtain the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensor;

[0008] According to the vision and language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data, obtain a vision and language vector;

[0009] According to the end-to-end model in the trajectory planning model, the navigation instructions, the multi-view image data, and the vision and language vector, obtain the planned trajectory of the vehicle;

[0010] Determine the control signal corresponding to the planned trajectory and perform autonomous driving according to the control signal.

[0011] In a possible implementation manner, the vision and language model is a large vision and language model LVLM. The obtaining a vision and language vector according to the vision and language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data includes:

[0012] Input the navigation instruction and the multi-view image data into the vision-language sub-model VLM of the LVLM model, and encode to obtain corresponding image tokens and text tokens;

[0013] Input the image tokens and text tokens into the large language sub-model LLM of the LVLM model to obtain a meta-action feature vector based on vision-language.

[0014] In a possible implementation manner, the inputting the navigation instruction and the multi-view image data into the vision-language sub-model VLM of the LVLM model, and encoding to obtain corresponding image tokens and text tokens includes:

[0015] Input the multi-view image data into the vision encoder of the VLM model for encoding and compression to obtain scene tokens and traffic participant tokens;

[0016] Input the scene tokens and traffic participant tokens into the deformable attention network of the VLM model for decoding and splicing to obtain a full-scene representation vector, and determine corresponding image tokens according to the full-scene representation vector;

[0017] Input the navigation instruction into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

[0018] In a possible implementation manner, it further includes:

[0019] Obtain a user instruction input by the user;

[0020] Correspondingly, input the navigation instruction and the user instruction into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

[0021] In a possible implementation manner, the vision-language model is the BEVFormer model, and the obtaining a vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instruction, and the multi-view image data includes:

[0022] Input the multi-view image data into the BEVFormer model to extract corresponding BEV feature vectors;

[0023] Input the navigation instruction into the preset text model to extract corresponding text feature vectors;

[0024] Perform vector fusion on the BEV feature vectors and the text feature vectors to obtain a vision-language vector.

[0025] In a possible implementation, obtaining the planned trajectory of the vehicle according to the end-to-end model in the trajectory planning model, the navigation instruction, the multi-view image data, and the vision-language vector includes:

[0026] Input the navigation instruction and the multi-view image data into the end-to-end model, so that the end-to-end model uses the navigation instruction, the multi-view image data, and the historical trajectory of the vehicle to perform scene processing to obtain the current scene vector;

[0027] Input the vision-language vector into the end-to-end model, so that the end-to-end model uses the current scene vector and the vision-language vector to perform trajectory prediction to obtain the planned trajectory of the vehicle within a preset duration later.

[0028] In a possible implementation, it further includes:

[0029] Obtain multi-view image sample data, and parse the multi-view image sample data to obtain sample segment data;

[0030] Obtain the sample label corresponding to each sample segment data, the sample label includes a navigation instruction label, a meta-action label, and a scene label, the navigation instruction label includes one or more navigation instructions, and different variants of the same navigation instruction, the meta-action label includes a lateral meta-action and / or a longitudinal meta-action, and the scene label includes a driving scene description;

[0031] Train the initial neural network model according to the sample segment data and the corresponding sample label to obtain the preset trajectory planning model, the initial neural network model includes an initial vision-language model and an initial end-to-end model.

[0032] In a second aspect, an embodiment of the present application provides an in-vehicle computing device, including:

[0033] An acquisition module, configured to acquire a navigation instruction input by a navigation map and multi-view image data collected by a multi-view sensor;

[0034] A processing module, configured to obtain a vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instruction, and the multi-view image data; obtain the planned trajectory of the vehicle according to the end-to-end model in the trajectory planning model, the navigation instruction, the multi-view image data, and the vision-language vector; determine the control signal corresponding to the planned trajectory, and perform autonomous driving according to the control signal.

[0035] In a third aspect, an embodiment of the present application provides another in-vehicle computing device, including:

[0036] A processor, and a memory communicatively connected to the processor;

[0037] The memory is used to store computer-executable instructions;

[0038] The processor is used to execute the computer-executable instructions stored in the memory, such that the processor performs the operations as described in the first aspect and / or various possible implementation manners of the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored therein, and when the computer-executable instructions are executed by a processor, they are used to implement the operations as described in the first aspect and / or various possible implementation manners of the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it is used to implement the operations as described in the first aspect and / or various possible implementation manners of the first aspect.

[0041] An embodiment of the present application provides a vision-language-based autonomous driving method, device, and medium. The trajectory planning model may include two parts: a vision-language model and an end-to-end model. By changing the architecture of the trajectory planning model and adjusting the model parameter quantity, the end-to-end model can directly output control signals corresponding to the planned trajectory, simplifying the processing flow of trajectory prediction in the autonomous driving process, reducing the computational consumption, and enabling more efficient trajectory prediction. Further, the vision-language model can perform semantic understanding and spatial understanding based on navigation instructions and multi-view image data to obtain a vision-language vector that can fully represent the current scene information. The end-to-end model can perform further scene understanding based on navigation instructions, multi-view image data, and in combination with the vision-language vector to obtain more accurate scene information. The combination of the low-frequency vision-language model and the high-frequency end-to-end model can output a more accurate planned trajectory and corresponding control signals. Through such an arrangement, the trajectory planning model can have the ability to perform semantic understanding and spatial understanding of the driving scene, can fully understand the driving scene information to handle complex road conditions, and improves the scene coverage of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0043] Figure 1 It is a system architecture diagram of an embodiment of the present application;

[0044] Figure 2 It is a flowchart of a vision-language-based autonomous driving method according to an embodiment of the present application;

[0045] Figure 3 The structural schematic diagram of an in-vehicle computing device according to an embodiment of the present application;

[0046] Figure 4 The structural schematic diagram of an in-vehicle computing device according to another embodiment of the present application.

[0047] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and more detailed descriptions will be provided hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of Specific Embodiments

[0048] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0049] In the technical solution of the present application, the processing of collection, storage, use, processing, transmission, provision, and disclosure of information such as financial data or user data complies with the provisions of relevant laws and regulations and does not violate public order and good customs.

[0050]

[0051] It should be noted that in the embodiments of the present application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned, and they should be considered exemplary. Their purpose is only to illustrate the feasibility in the implementation of the technical solution of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0052] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0053] The vision-language based autonomous driving method, device, and medium of this application can be used in the field of autonomous driving technology, and can also be used in any field other than the autonomous driving technology field, such as the vehicle control technology field, etc. The application field of the vision-language based autonomous driving method, device, and medium of this application is not limited.

[0054] The vision-language based autonomous driving method, device, and medium of this application can be applied to the scenario of a vehicle performing autonomous driving. As long as a vehicle is equipped with a navigation map and multi-view cameras, the vision-language based autonomous driving method, device, and medium of this application can be applied.

[0055] First, the nouns involved in this application are explained:

[0056] VLM (Vision-Language Model) is a multi-modal model that combines visual and language information. Such models aim to understand and generate content related to vision and language. VLM models are typically used to handle tasks involving images and text, such as image description generation, visual question answering (VQA), image-text matching, etc. The core idea of the VLM model is to enhance the model's performance in multi-modal tasks by jointly learning visual and language features.

[0057] LLM (Large Language Model) refers to those large-scale deep learning models that are trained to process and generate natural language text. These models are typically based on the Transformer architecture and, through training on large-scale text data, can perform various natural language processing (NLP) tasks, such as text generation, translation, question answering, summarization, etc.

[0058] LVLM (Large Vision-Language Model) is a multi-modal model that combines a large language model (LLM) and a vision model. Such models aim to simultaneously process visual and language information to perform complex tasks involving images and text. The LVLM model can perform well in multi-modal tasks through large-scale training data and advanced deep learning architectures.

[0059] "End-to-End" is a term widely used in the fields of computer science and engineering, especially in machine learning and deep learning. It refers to a system design approach where a model or system processes directly from input to output without intermediate manual intervention or explicit feature engineering. The goal of this approach is to automate the entire processing flow through a unified model architecture.

[0060] BEVFormer is a model for autonomous driving and computer vision tasks, especially in the field of Bird's Eye View (BEV) perception. BEV perception is a key task in autonomous driving systems as it provides a top-down view to better understand the environment around the vehicle. In autonomous driving, BEV perception is the basis for path planning, obstacle detection, and navigation. BEVFormer can help autonomous driving systems better understand complex traffic environments.

[0061] A token, in the context of natural language processing, refers to a symbol used to represent a word or phrase. A token can be a single character or a sequence of multiple characters.

[0062] Image tokens and text tokens are the basic units for processing and analyzing image and text data, commonly used in machine learning and natural language processing (NLP). In computer vision and deep learning, image tokens are typically used to divide image data into smaller, manageable parts for processing and analysis. In natural language processing (NLP), text tokenization is the process of breaking text into smaller units (tokens), which are the basis for a model to understand and process text.

[0063] During the process of vehicle autonomous driving, it is very important to quickly predict the vehicle's trajectory in the next few seconds according to the driving scenario. In related technologies, a trajectory prediction module and a control module are usually set in the in-vehicle computing device. The trajectory prediction module can perform trajectory prediction based on the navigation instructions of the navigation map and the image data collected by the multi-view camera to obtain the planned trajectory of the vehicle in the next few seconds. The control module can generate corresponding control signals according to the planned trajectory so that the vehicle can perform autonomous driving according to the planned trajectory and control signals.

[0064] However, the computational process of the above process is relatively complex, consuming a large amount of computing power, and the model in the trajectory prediction module is relatively simple and cannot handle complex road conditions, resulting in a low scene coverage.

[0065] Based on the above technical problems, the inventive concept of this application lies in: how to provide an autonomous driving solution that can simplify the processing flow of trajectory prediction, reduce the consumption of computational resources, handle complex road conditions, and improve the scene coverage of autonomous driving.

[0066] An embodiment of this application provides an autonomous driving method, device, and medium based on vision-language. The trajectory planning model can be split into two parts: a vision-language model and an end-to-end model. By changing the model architecture and adjusting the model parameter quantity, the end-to-end model can directly output the control signal corresponding to the planned trajectory, simplifying the processing flow of trajectory prediction during autonomous driving, reducing the consumption of computational resources, and enabling more efficient trajectory prediction. In addition, the vision-language model and the end-to-end model also endow the trajectory planning model with the ability to semantically and spatially understand the driving scene, enabling it to fully understand driving scene information to handle complex road conditions and improving the scene coverage of autonomous driving.

[0067] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0068] Figure 1 As shown in FIG. 1, which is the system architecture diagram of an embodiment of this application, the in-vehicle computing device can obtain the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensors; according to the vision-language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data, obtain the vision-language vector; according to the end-to-end model in the trajectory planning model, the navigation instructions, the multi-view image data, and the vision-language vector, obtain the planned trajectory of the vehicle; determine the control signal corresponding to the planned trajectory, and perform autonomous driving according to the control signal.

[0069] Figure 2 As shown in the flowchart of the autonomous driving method based on vision-language in an embodiment of this application, this embodiment describes the autonomous driving method based on vision-language with the execution subject being the in-vehicle computing device. As Figure 2 shown, the autonomous driving method based on vision-language can include the following steps:

[0070] S201: Obtain the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensors.

[0071] In this embodiment, the in-vehicle computing device can be communicatively connected to the navigation map and the multi-view sensors respectively to obtain the navigation instructions and the multi-view image data respectively. The in-vehicle computing device can be a device with computing capabilities on the vehicle side, and no restrictions are imposed here.

[0072] In this embodiment, the navigation instruction can be an instruction such as lane change or turning issued when the navigation map performs navigation.

[0073] In this embodiment, the multi-view sensor can be a multi-view camera set at different positions of the vehicle, which can collect image data from different perspectives during the driving process of the vehicle.

[0074] S202: Obtain a visual language vector according to the visual language model in the preset trajectory planning model, the navigation instruction, and the multi-view image data.

[0075] In this embodiment, the visual language model can be a model for understanding and analyzing visual image data and text language data. The visual language model can be obtained by performing data migration and training on existing visual language models, such as the CLIP model, the BEVFormer model, etc.

[0076] In this embodiment, the visual language model can perform scene understanding on the navigation instruction and the multi-view image data to obtain a visual language vector that can fully represent the current scene information.

[0077] S203: Obtain the planned trajectory of the vehicle according to the end-to-end model in the trajectory planning model, the navigation instruction, the multi-view image data, and the visual language vector.

[0078] In this embodiment, the end-to-end model can be obtained by performing data migration and training on an existing end-to-end model (E2E), and can perform trajectory prediction according to the input navigation instruction, multi-view image data, and visual language vector.

[0079] In this embodiment, the running frequency of the visual language model is relatively low. It uses the navigation instructions and multi-view image data for a period of time to obtain the visual language vector for that period. The running frequency of the end-to-end model is relatively high, and it can predict a more accurate planned trajectory at the current moment according to the current navigation instruction, multi-view image data, and in combination with the visual language vector.

[0080] S204: Determine the control signal corresponding to the planned trajectory and perform autonomous driving according to the control signal.

[0081] In this embodiment, the in-vehicle computing device can directly determine the corresponding control signal according to the planned trajectory and send the control signal to the corresponding components to perform autonomous driving according to the planned trajectory.

[0082] In this embodiment, the trajectory planning model may include two parts: a vision-language model and an end-to-end model. By changing the architecture of the trajectory planning model and adjusting the model parameter quantity, the end-to-end model can directly output the control signal corresponding to the planned trajectory, simplifying the processing flow of trajectory prediction in the autonomous driving process, reducing the computational consumption, and enabling more efficient trajectory prediction. Further, the vision-language model can perform semantic understanding and spatial understanding based on the navigation instruction and multi-view image data to obtain a vision-language vector that can fully represent the current scene information. The end-to-end model can perform further scene understanding based on the navigation instruction, multi-view image data, and in combination with the vision-language vector to obtain more accurate scene information. The combination of the low-frequency vision-language model and the high-frequency end-to-end model can output a more accurate planned trajectory and the corresponding control signal. Through such a setting, the trajectory planning model can have the ability to perform semantic understanding and spatial understanding of the driving scene, can fully understand the driving scene information to handle complex road conditions, and improves the scene coverage of autonomous driving.

[0083] In a possible implementation manner, the vision-language model may be a large vision-language model LVLM. Then, step S202 of obtaining the vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instruction, and the multi-view image data may include:

[0084] S11: Input the navigation instruction and the multi-view image data into the vision-language sub-model VLM of the LVLM model, and encode to obtain the corresponding image tokens and text tokens.

[0085] S12: Input the image tokens and the text tokens into the large language sub-model LLM of the LVLM model to obtain the meta-action feature vector based on vision-language.

[0086] Exemplarily, CLIP (ViT-B / 32) may be used as the backbone of the above VLM model, and ViT-L / 14 in CLIP may be used as the vision encoder to encode the multi-view image data into image tokens, and the text encoder may be used to encode the navigation instruction into text tokens.

[0087] Exemplarily, QWEN-7b may be used as the backbone of the above LLM model, and then the hidden states vector of the last layer of the model is output, passed through two layers of MLP (multi-layer perceptron), and the final meta-action feature vector is output to the E2E module.

[0088] In this embodiment, when the vision-language model is the LVLM model, the LVLM model may include a VLM model and an LLM model. The VLM model can perform semantic understanding and spatial understanding based on the input navigation instructions and multi-view image data to facilitate scene understanding and generate image tokens and text tokens. The LLM model can make high-level decisions based on the input image tokens and text tokens and predict the meta-action feature vector corresponding to the scene.

[0089] In a possible embodiment, the above step S11 of inputting the navigation instructions and multi-view image data into the vision-language sub-model VLM of the LVLM model and encoding to obtain corresponding image tokens and text tokens may include:

[0090] S21: Input the multi-view image data into the vision encoder of the VLM model for encoding and compression to obtain scene tokens and traffic participant tokens.

[0091] S22: Input the scene tokens and traffic participant tokens into the deformable attention network of the VLM model for decoding and splicing to obtain a full-scene representation vector, and determine the corresponding image tokens according to the full-scene representation vector.

[0092] S23: Input the navigation instructions into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

[0093] In this embodiment, the multi-view image data in the above step S21 may be multi-view image data acquired by the in-vehicle computing device within a period of time. The in-vehicle computing device can process the multi-view image data of a period of time into a set of multi-view image sequences I ∈ R^(Nimg×H×W×3), and input the navigation instructions and the multi-view image sequences into the VLM model. Nimg, H, and W respectively represent the number, height, and width of the images.

[0094] In this embodiment, after receiving the multi-view image sequence I, the vision encoder can extract scene features based on the transformer structure, and these features are further encoded and compressed by the driving vision adapter to generate corresponding scene tokens (scene tokens) Eimg∈R^(Nimg×Mimg×C) and traffic participant tokens (traffic participant tokens) Jimg∈R^(Nimg×Himg×C). Mimg, Himg, and C respectively represent the number of scene tokens per image, the number of traffic participant tokens per image, and the feature dimension of the LLM model.

[0095] In this embodiment, in order to enable the VLM model to distinguish the image features of different views and build spatial understanding, a simple but effective surround view prompt can be designed in advance for the driving scenario. Taking the front view as an example, the corresponding prompt is: <FRONT VIEW>:\n <imgs>\n, where <imgs>It is a special token of QWEN-LLM and will be replaced by image tokens during the generation process.

[0096] Exemplarily, the scenario token / traffic participant token can be obtained in the following way: First, perform video encoding (Video Encoder) to extract frames from video clips, and then encode them through a visual encoder to obtain a series of features. Specifically:

[0097] (1) Use the pre-trained CLIP (ViT-B / 32) as the backbone network to transfer the image representation to the video representation.

[0098] (2) CLIP first extracts non-overlapping image patches and then performs a linear projection to map them to one-dimensional tokens.

[0099] (3) Use the Transformer architecture to simulate the interaction between each patch in the input image to obtain the final representation.

[0100] (4) In the manner of ViT and CLIP, use the output marked with [class] as the image representation.

[0101] In this embodiment, the attributes of traffic participants (position, class score, direction, etc.) can be decoded from traffic participant queries through an MLP-based decoder head. To enrich traffic participant features for motion prediction, a deformable attention network introducing an attention mechanism can be used for traffic participant-traffic participant and traffic participant-scene interactions. The deformable attention network can learn traffic participant-level features and scene-level features from the shared image feature map. Then, the traffic participant vector and the scene vector are concatenated and passed through a 2-layer MLP to finally output the full-scene representation vector.

[0102] Exemplarily, ViT-L / 14 in CLIP can be used as the visual encoder, and each image is resized to a size of H = W = 224, generating 576 image tokens. A randomly initialized self-query is used to learn implicit scene features valuable for planning. The traffic participant query first interacts with the traffic participant query through the Transformer decoder. During this process, the self-query serves as the query in the attention mechanism, while the traffic participant query serves as the key and value. The self-position and the traffic participant position are encoded through a single-layer MLP and used as the query position embedding and the key position embedding. The position embedding provides information about the relative position relationship between the traffic participant and the ego vehicle.

[0103] In this embodiment, the text encoder can encode the navigation instruction into text tokens Etxt ∈ R^(Mtxt×C), where Mtxt is the number of text tokens.

[0104] In this embodiment, the VLM model may include a visual encoder, a deformable attention network, and a text encoder. Through the text encoder, the navigation instruction can be encoded and feature-extracted to obtain text tokens that can represent semantic information. Through the visual encoder, the scene tokens and traffic participant tokens can be decoded and compressed. While extracting the scene features and traffic participant features, the computational consumption is reduced and the computational efficiency is improved. The deformable attention network can learn the correlation between the scene and the traffic participants, and splice the traffic participant vector and the scene vector to obtain a more comprehensive and accurate full-scene representation vector, improving the accuracy of subsequent decision-making based on the full-scene representation vector and the text tokens.

[0105] In a possible implementation, the method may further include:

[0106] Obtain a user instruction input by the user.

[0107] Correspondingly, the above step S23 may include: inputting the navigation instruction and the user instruction into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

[0108] In this embodiment, the user instruction may be voice information, text information, or other control signals. When the user instruction is an instruction other than text information, the user instruction can be first processed into text information, and then the corresponding text information is input into the text encoder for encoding.

[0109] In this embodiment, in some emergency situations, the user can also input relevant user instructions. At this time, the text encoder can encode the navigation instruction and the user instruction to obtain text tokens that can represent semantic information.

[0110] In a possible implementation, the vision-language model may also be the BEVFormer model. Then, the above step S202 of obtaining the vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instruction, and the multi-view image data may include:

[0111] S31: Input the multi-view image data into the BEVFormer model to extract the corresponding BEV feature vector.

[0112] S32: Input the navigation instruction into the preset text model to extract the corresponding text feature vector.

[0113] S33: Perform vector fusion on the BEV feature vector and the text feature vector to obtain the vision-language vector.

[0114] In this embodiment, the text model can be any model capable of encoding and feature extraction for navigation instructions, without any limitation here.

[0115] Exemplarily, the BEVFormer model can be first used to obtain the global BEV feature (Global BEV Feature) corresponding to the multi-view image data. In the retrieval task, all parameters of the BEVFormer model are frozen, and the generated features are directly used for downstream post-processing and retrieval.

[0116] Secondly, semantic representation fusing is performed. To achieve zero-shot retrieval, comprehensive semantic information needs to be extracted from the text input. The semantic understanding ability of pre-trained models in the general context can be utilized to fine-tune these models to adapt to the field of autonomous driving, obtaining a text model capable of encoding and feature extraction for navigation instructions. Through sentence embedding, the graph embedding is fused with the navigation text embedding sequence to form a synthetic representation for the supervision of retrieval tasks and caption generation.

[0117] Finally, shared cross-modal prompting is performed, which includes cross-modal interaction. Cross-modal interaction is an independent structure that bridges the BEV and text branches through the cross-attention method. It consists of a set of learnable long sequence tokens, aiming to map the BEV features and text features to the same manifold space, promoting the alignment of different modal information in the two branches. Through interaction, the BEV features and text features are fused to ensure that the resulting features come from the same embedding space and maintain the same shape of the feature map.

[0118] In this embodiment, the BEVFormer model is a dedicated camera-based BEV perception model, including two key modules: spatial attention and temporal attention. These modules enable the model to aggregate spatial and temporal information, thereby better representing movable obstacles. The BEVFormer model can also be used to perform temporal and spatial understanding of multi-view image data, obtaining a BEV feature vector that can fully represent the current scene information. Then, using any text model, the text feature vector corresponding to the navigation instruction can be extracted, and by fusing the BEV feature vector and the text feature vector, a vision-language vector that can fully represent the current scene information can be obtained.

[0119] In a possible embodiment, step S203 of obtaining the planned trajectory of the vehicle according to the end-to-end model, navigation instruction, multi-view image data, and vision-language vector in the trajectory planning model may include:

[0120] S41: Input the navigation instruction and multi-view image data into the end-to-end model, so that the end-to-end model processes the scene using the navigation instruction, multi-view image data, and the historical trajectory of the vehicle to obtain the current scene vector.

[0121] S42: Input the vision-language vector into the end-to-end model, so that the end-to-end model uses the current scene vector and the vision-language vector for trajectory prediction to obtain the planned trajectory of the vehicle within a preset time duration in the future.

[0122] In this embodiment, the navigation instruction and multi-view image data in the above step S41 can be the navigation instruction and multi-view image data obtained by the in-vehicle computing device at the current moment.

[0123] In this embodiment, the historical trajectory of the vehicle can be the driving trajectory before the current moment of the vehicle.

[0124] In this embodiment, before inputting the vision-language vector into the end-to-end model, the vision-language vector also needs to be processed into a format that the end-to-end model can apply.

[0125] In this embodiment, the preset time duration can be flexibly set by those skilled in the art according to the actual situation. For example, the preset time duration can be 1s or 2s, and there is no limitation here.

[0126] In this embodiment, the vision-language model can perform semantic understanding and spatial understanding based on the navigation instruction and multi-view image data for a period of time to obtain the vision-language vector. The calculation frequency is relatively low and cannot meet the real-time requirements of autonomous driving, while the end-to-end model is a real-time model with a relatively high calculation frequency. Therefore, the end-to-end model also needs to perform scene understanding again based on the current navigation instruction and multi-view image data to obtain the current scene vector that can represent the driving scene at the current moment, and perform trajectory prediction based on the current scene vector and the vision-language vector to obtain a more accurate planned trajectory.

[0127] In a possible embodiment, the method may further include:

[0128] S51: Obtain multi-view image sample data, and parse the multi-view image sample data to obtain sample segment data.

[0129] S52: Obtain the sample label corresponding to each sample segment data. The sample label includes a navigation instruction label, a meta-action label, and a scene label. The navigation instruction label includes one or more navigation instructions and different variants of the same navigation instruction. The meta-action label includes a lateral meta-action and / or a longitudinal meta-action. The scene label includes a driving scene description.

[0130] S53: Train the initial neural network model based on the sample segment data and the corresponding sample labels to obtain a preset trajectory planning model. The initial neural network model includes an initial vision-language model and an initial end-to-end model.

[0131] In this embodiment, the collected sample data can be parsed into segments, and appropriate navigation instructions and optional attention instructions can be labeled for each segment. The parsing process takes a series of frames as input and splits these frames into segments, with each segment corresponding to a navigation instruction. For example, if the agent starts turning left at frame T0 and ends at frame Tn, then (T0, Tn) is marked as a new segment and attached with the navigation instruction "Turn left at the next intersection".

[0132] In this embodiment, three types of navigation instructions (follow, turn, and others) are mainly considered, with a total of 56 different instructions. Considering the inherent richness of natural language, for each type of instruction, N different variants can be generated using the QWEN-MAX API. Each variant has the same semantic meaning but differs in wording. This makes language interpretation more comprehensive and flexible, capable of adapting to different expressions of the same instruction. At the same time, considering that in many cases, an instruction may consist of two to three consecutive instructions, such as "Turn right at this intersection, then go straight to the next intersection and turn right", navigation instruction labels including multiple navigation instructions can also be constructed to simulate real driving scenarios based on navigation.

[0133] In this embodiment, the meta-actions include lateral decision-making and longitudinal decision-making. Lateral meta-actions include turning left, going straight, and turning right, while longitudinal meta-actions cover accelerating, maintaining, decelerating, and stopping. Lateral meta-actions are determined based on the lateral displacement during the predicted future time step T, and longitudinal meta-actions are determined by the change in speed during the prediction period. The final meta-action is composed of lateral and longitudinal meta-actions together.

[0134] In this embodiment, the driving scene description can include information related to the driving scene such as traffic conditions, environment (e.g., city, countryside, etc.), road type (e.g., paved road, highway), weather conditions, time of day, and road conditions (e.g., whether the road is flat or if there are any obstacles).

[0135] In this embodiment, the multi-view image sample data can first be parsed into sample segment data, and corresponding navigation instruction labels, meta-action labels, and scene labels can be labeled for the sample segment data respectively. Through pre-training and fine-tuning on different data sets, the generalization ability of the model across scenarios and the ability of transfer learning are improved, enabling the trajectory planning model to accurately make meta-action decisions and trajectory predictions based on the input navigation instructions and multi-view image data.

[0136] The following uses a specific embodiment to elaborate on the vision-language-based autonomous driving method of the present application.

[0137] In a specific embodiment, a vehicle is driving autonomously on the road. A navigation map and multi-view cameras are installed on the vehicle. The specific autonomous driving process is as follows:

[0138] In the first step, the in-vehicle computing device obtains the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensors.

[0139] In the second step, the in-vehicle computing device inputs the multi-view image data into the visual encoder of the VLM model for encoding and compression to obtain scene tokens and traffic participant tokens; inputs the scene tokens and traffic participant tokens into the deformable attention network of the VLM model for decoding and splicing to obtain a full-scene representation vector; inputs the navigation instructions into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

[0140] In the third step, the in-vehicle computing device inputs the full-scene representation vector and the text tokens into the LLM model to obtain a vision-language-based meta-action feature vector.

[0141] In the fourth step, the in-vehicle computing device inputs the navigation instructions and the multi-view image data into the end-to-end model, so that the end-to-end model uses the navigation instructions, the multi-view image data, and the historical trajectory of the vehicle to perform scene processing to obtain the current scene vector.

[0142] In the fifth step, the in-vehicle computing device inputs the vision-language vector into the end-to-end model, so that the end-to-end model uses the current scene vector and the vision-language vector to perform trajectory prediction to obtain the planned trajectory of the vehicle within a preset time period.

[0143] In the sixth step, the in-vehicle computing device determines the control signal corresponding to the planned trajectory and sends the control signal to the corresponding components, so that the vehicle drives autonomously according to the planned trajectory within the preset time period.

[0144] Figure 3 It is a schematic structural diagram of an in-vehicle computing device according to an embodiment of the present application. As Figure 3 shown, the in-vehicle computing device includes: an acquisition module 31, configured to obtain the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensors; a processing module 32, configured to obtain a vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data; obtain the planned trajectory of the vehicle according to the end-to-end model, the navigation instructions, the multi-view image data, and the vision-language vector in the trajectory planning model; determine the control signal corresponding to the planned trajectory, and perform autonomous driving according to the control signal.

[0145] The in-vehicle computing device provided by the embodiment of the present application can execute the technical solutions shown in the above method embodiments. The implementation principles and beneficial effects are similar, and will not be elaborated here.

[0146] Figure 4 It is a schematic structural diagram of an in-vehicle computing device according to another embodiment of the present application. As Figure 4 shown, the in-vehicle computing device includes: a processor 401 and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer-executable instructions; the processor 401 executes the computer-executable instructions stored in the memory 402 to implement the steps of the vision-language-based autonomous driving method in the above various method embodiments.

[0147] In the above in-vehicle computing device, the memory 402 and the processor 401 are directly or indirectly electrically connected to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines, such as being connected through a bus. The memory 402 stores computer-executable instructions for implementing a data access control method, including at least one software function module that can be stored in the memory 402 in the form of software or firmware. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402.

[0148] The memory 402 can be, but is not limited to, a random access memory (Random Access Memory, abbreviated as RAM), a read-only memory (Read Only Memory, abbreviated as ROM), a programmable read-only memory (Programmable Read-Only Memory, abbreviated as PROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), an electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. Among them, the memory 402 is used to store programs, and the processor 401 executes the programs after receiving the execution instructions. Further, the software programs and modules in the memory 402 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide a running environment for other software components.

[0149] The processor 401 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor 401 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of the method embodiments of the present application.

[0151] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the method embodiments of the present application.

[0152] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0153] Furthermore, it should be noted that although the steps in the flowchart are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0154] It should be understood that the above device embodiments are illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0155] In addition, unless otherwise specified, in each embodiment of the present application, each functional unit / module can be integrated into one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0156] In the above embodiments, the descriptions of the various embodiments each have their own emphasis. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered that the scope described in this specification.

[0157] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the appended claims.

[0158] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.< / imgs> < / imgs>

Claims

1. A vision-language-based autonomous driving method, characterized in that, Including: Obtain the navigation instructions input by the navigation map and the multi-view image data collected by the multi-view sensors; According to the vision-language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data, obtain a vision-language vector; According to the end-to-end model in the trajectory planning model, the navigation instructions, the multi-view image data, and the vision-language vector, obtain the planned trajectory of the vehicle; Determine the control signal corresponding to the planned trajectory and perform autonomous driving according to the control signal.

2. The vision-language-based autonomous driving method according to claim 1, wherein The vision-language model is the large vision-language model LVLM. The step of obtaining a vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data includes: Input the navigation instructions and the multi-view image data into the vision-language sub-model VLM of the LVLM model to encode and obtain corresponding image tokens and text tokens; Input the image tokens and text tokens into the large language sub-model LLM of the LVLM model to obtain a meta-action feature vector based on vision and language.

3. The vision-language-based autonomous driving method according to claim 2, wherein The step of inputting the navigation instructions and the multi-view image data into the vision-language sub-model VLM of the LVLM model to encode and obtain corresponding image tokens and text tokens includes: Input the multi-view image data into the vision encoder of the VLM model for encoding and compression to obtain scene tokens and traffic participant tokens; Input the scene tokens and traffic participant tokens into the deformable attention network of the VLM model for decoding and splicing to obtain a full-scene representation vector, and determine corresponding image tokens according to the full-scene representation vector; Input the navigation instructions into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

4. The vision-language-based autonomous driving method according to claim 3, wherein It also includes: Obtain the user instructions input by the user; Correspondingly, input the navigation instructions and the user instructions into the text encoder of the VLM model for encoding to obtain corresponding text tokens.

5. The vision-language based autonomous driving method according to claim 1, characterized in that The vision-language model is the BEVFormer model. The step of obtaining a vision-language vector according to the vision-language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data includes: Input the multi-view image data into the BEVFormer model to extract the corresponding BEV feature vector; Input the navigation instructions into the preset text model to extract the corresponding text feature vector; Perform vector fusion on the BEV feature vector and the text feature vector to obtain a vision-language vector.

6. The vision-language based autonomous driving method according to any one of claims 1-5, characterized in that, The step of obtaining the planned trajectory of the vehicle according to the end-to-end model in the trajectory planning model, the navigation instructions, the multi-view image data, and the vision-language vector includes: Input the navigation instructions and the multi-view image data into the end-to-end model, so that the end-to-end model uses the navigation instructions, the multi-view image data, and the historical trajectory of the vehicle to perform scene processing to obtain the current scene vector; Input the visual language vector into the end-to-end model, so that the end-to-end model uses the current scene vector and the visual language vector for trajectory prediction to obtain the planned trajectory of the vehicle within a preset duration in the future.

7. The visual language-based autonomous driving method according to any one of claims 1-5, characterized in that Further comprising: Obtain multi-view image sample data and parse the multi-view image sample data to obtain sample segment data; Obtain the sample label corresponding to each sample segment data, the sample label includes a navigation instruction label, a meta-action label, and a scene label, the navigation instruction label includes one or more navigation instructions and different variants of the same navigation instruction, the meta-action label includes a lateral meta-action and / or a longitudinal meta-action, and the scene label includes a driving scene description; Train an initial neural network model according to the sample segment data and the corresponding sample label to obtain the preset trajectory planning model, and the initial neural network model includes an initial visual language model and an initial end-to-end model.

8. An in-vehicle computing device, characterized in that, Comprising: An acquisition module for acquiring navigation instructions input by a navigation map and multi-view image data collected by a multi-view sensor; A processing module for obtaining a visual language vector according to the visual language model in the preset trajectory planning model, the navigation instructions, and the multi-view image data; obtaining the planned trajectory of the vehicle according to the end-to-end model, the navigation instructions, the multi-view image data, and the visual language vector in the trajectory planning model; determining a control signal corresponding to the planned trajectory, and performing autonomous driving according to the control signal.

9. A vehicle-mounted computing device, characterized in that, Comprising: A processor and a memory communicatively connected to the processor; The memory is used for storing computer execution instructions; The processor is used for executing the computer execution instructions stored in the memory, so that the processor executes the visual language-based autonomous driving method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by a processor, they are used to implement the visual language-based autonomous driving method according to any one of claims 1-7.

Images