Automatic driving track generation method and device, storage medium and electronic device

Abstract

The invention provides an automatic driving track generation method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining multi-view image data collected by a camera of a vehicle, and obtaining the attribute parameters of the camera; the multi-view image data is adopted to generate two-dimensional image features, and the two-dimensional image features comprise local visual features of multiple views; the two-dimensional image features and the attribute parameters are adopted to generate bird's-eye view BEV features, the two-dimensional image features are adopted to generate high-level semantic information, the high-level semantic information comprises meta-actions and enhanced semantics, the meta-actions are used for describing rough actions of path planning, and the enhanced semantics are used for conducting semantic interpretation on the meta-actions; and generating an automatic driving path of the vehicle by adopting the BEV features and the high-level semantic information. According to the embodiment of the invention, the technical problem of low reliability of an automatic driving path generated by an end-to-end driving model in related technologies is solved, and the safety and robustness of an automatic driving system are improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more specifically, to a method, apparatus, storage medium, and electronic device for generating autonomous driving trajectories. Background Technology

[0002] Among related technologies, autonomous driving technology has developed rapidly in recent years. Its core tasks include environmental perception, path planning, and vehicle control. Among these, path planning is the final link connecting perception and control, and it directly determines the vehicle's driving behavior and safety.

[0003] Traditional path planning methods are typically based on a hierarchical architecture. First, a perception module extracts environmental features, and then a path is generated using manually designed rules or traditional optimization algorithms. However, this approach suffers from several drawbacks. Firstly, feature extraction is limited by the perception module, which cannot fully extract the high-level semantic information implicit in the scene (such as potential traffic intentions or complex scene relationships). Secondly, rule design struggles to adapt to diverse scenarios; manually customized rules and optimization methods often fail when faced with complex and varied driving environments, exhibiting poor generalization ability.

[0004] To address these issues, end-to-end deep learning methods have emerged in recent years. These methods rely on algorithms to learn features and patterns from data, reducing human intervention and generating driving paths directly from sensor data. However, a significant drawback of these methods is that the models are essentially black boxes, lacking interpretability and making it difficult to understand and determine the underlying logic of path generation. For example, the underlying logic for a vehicle passing through an intersection may not be based on traffic rules, but rather on judgments made from the actions of the vehicle in front and those around it, leading to poor reliability in path generation.

[0005] No efficient and accurate solution has yet been found to address the aforementioned issues in the relevant technologies. Summary of the Invention

[0006] This invention provides a method, apparatus, storage medium, and electronic device for generating autonomous driving trajectories to solve technical problems in related technologies.

[0007] According to an embodiment of the present invention, a method for generating an autonomous driving trajectory is provided, comprising: acquiring multi-view image data collected by a vehicle's camera and acquiring attribute parameters of the camera; generating two-dimensional image features using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple perspectives; generating bird's-eye view BEV features using the two-dimensional image features and the attribute parameters; generating high-level semantic information using the two-dimensional image features, wherein the high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe coarse actions in path planning, and the enhanced semantics are used to semantically interpret the meta-actions; and generating an autonomous driving path for the vehicle using the BEV features and the high-level semantic information.

[0008] Optionally, generating BEV features using the two-dimensional image features and the attribute parameters includes: reading the intrinsic parameters and extrinsic parameters relative to the vehicle for each camera from the attribute parameters; and projecting the two-dimensional image features onto the vehicle's body coordinate system using the intrinsic and extrinsic parameters to obtain the BEV features.

[0009] Optionally, generating high-level semantic information using the two-dimensional image features includes: acquiring input commands from the vehicle and reading preset surround-view prompts, wherein the surround-view prompts include prompts for views from multiple perspectives, and the input commands include user commands and navigation commands; inputting the two-dimensional image features, the input commands, and the surround-view prompts into a multimodal large model to generate corresponding global visual semantic features and text features, wherein the multimodal large model includes a visual encoder, a language encoder, a text encoder, and a cross-attention network, the cross-attention network being used to align the features of the two-dimensional image features and the surround-view prompts; and generating high-level semantic information using the global visual semantic features, the text features, and the surround-view prompts.

[0010] Optionally, generating high-level semantic information using the global visual semantic features, the text features, and the lookaround cue includes: generating a text token using the text features, generating an image tag using the global visual semantic features; inputting the text token, the image tag, and the lookaround cue into a large language model, and outputting high-level semantic information.

[0011] Optionally, before inputting the two-dimensional image features and the look-around cue into the multimodal large model, the method further includes: configuring an adaptation layer between the visual encoder and the language encoder of the multimodal large model, wherein the adaptation layer includes a visual adaptation module and a language adaptation module, the visual adaptation module is used to adjust the visual features output by the visual encoder through additional convolutional layers and fully connected layers so that the adjusted visual features are adapted to different visual tasks in the autonomous driving environment, and the language adaptation module is used to adjust the input format of the language encoder using prefix cues corresponding to the visual task so that the adjusted input format is related to natural language instructions in the autonomous driving scenario.

[0012] Optionally, generating the autonomous driving path of the vehicle using the BEV features and the high-level semantic information includes: detecting dynamic objects based on the BEV features and generating a local map; extracting meta-action features from the high-level semantic information using a meta-action encoder and converting the meta-action features into embedding vectors; predicting the future trajectory of the dynamic objects in the local map; inputting the future trajectory and the embedding vectors into a path planning model, and outputting the autonomous driving path of the vehicle using the embedding vectors as prior information.

[0013] Optionally, before inputting the future trajectory and the meta-action features into the path planning model, the method further includes: acquiring sample data, wherein the sample data includes: sample multi-view image data, sample vehicle driving data, sample high-precision map data, and sample meta-actions related to the path; configuring a multi-task loss function, wherein the multi-task loss function includes: a path generation error loss function, a meta-action consistency loss function, and a logical consistency loss function, wherein the path generation error loss function is used to optimize the difference between the model-predicted path and the real trajectory using mean squared error, the meta-action consistency loss function is used to optimize the selection of meta-actions using a classification loss function to ensure that the path planning decision is consistent with the generated meta-actions, and the logical consistency loss function is used to constrain the consistency between the explanatory information in path planning and the generated semantic information; and jointly training an initial model using the sample data and the multi-task loss function to obtain the path planning model.

[0014] According to another embodiment of the present invention, an apparatus for generating an autonomous driving trajectory is provided, comprising: an acquisition module for acquiring multi-view image data collected by a camera of a vehicle and acquiring attribute parameters of the camera; a first generation module for generating two-dimensional image features using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple perspectives; a second generation module for generating bird's-eye view BEV features using the two-dimensional image features and the attribute parameters, and generating high-level semantic information using the two-dimensional image features, wherein the high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe coarse actions of path planning, and the enhanced semantics are used to semantically interpret the meta-actions; and a third generation module for generating an autonomous driving path for the vehicle using the BEV features and the high-level semantic information.

[0015] Optionally, the second generation module includes: a reading unit, used to read the intrinsic parameters and extrinsic parameters relative to the vehicle from the attribute parameters; and a projection unit, used to project the two-dimensional image features onto the vehicle's body coordinate system using the intrinsic and extrinsic parameters to obtain BEV features.

[0016] Optionally, the second generation module includes: a processing unit, configured to acquire input commands from the vehicle and read preset surround-view prompts, wherein the surround-view prompts include prompts for multiple viewpoints, and the input commands include user commands and navigation commands; a first generation unit, configured to input the two-dimensional image features, the input commands, and the surround-view prompts into a multimodal large model to generate corresponding global visual semantic features and text features, wherein the multimodal large model includes a visual encoder, a language encoder, a text encoder, and a cross-attention network, the cross-attention network being used to align the features of the two-dimensional image features and the surround-view prompts; and a second generation unit, configured to generate high-level semantic information using the global visual semantic features, the text features, and the surround-view prompts.

[0017] Optionally, the second generation unit includes: a generation subunit, used to generate a text token using the text features and generate an image tag using the global visual semantic features; and an output subunit, used to input the text token, the image tag, and the lookaround cue into a large language model and output high-level semantic information.

[0018] Optionally, the second generation module further includes: a configuration unit, configured to configure an adaptation layer between the visual encoder and the language encoder of the multimodal large model before the first generation unit inputs the two-dimensional image features and the look-around cue into the multimodal large model, wherein the adaptation layer includes a visual adaptation module and a language adaptation module, the visual adaptation module is configured to adjust the visual features output by the visual encoder through additional convolutional layers and fully connected layers so that the adjusted visual features are adapted to different visual tasks in the autonomous driving environment, and the language adaptation module is configured to adjust the input format of the language encoder using prefix cues corresponding to the visual task so that the adjusted input format is related to natural language instructions in the autonomous driving scenario.

[0019] Optionally, the third generation module includes: a processing subunit, configured to detect dynamic objects based on the BEV features and generate a local map, extract meta-action features of the high-level semantic information using a meta-action encoder, and convert the meta-action features into embedding vectors; a prediction subunit, configured to predict the future trajectory of the dynamic object in the local map; and an output subunit, configured to input the future trajectory and the embedding vector into a path planning model, and output the autonomous driving path of the vehicle using the embedding vector as prior information.

[0020] Optionally, the third generation module further includes: an acquisition subunit, configured to acquire sample data before the output subunit inputs the future trajectory and the meta-action features into the path planning model, wherein the sample data includes: sample multi-view image data, sample vehicle driving data, sample high-precision map data, and sample meta-actions related to the path; a configuration subunit, configured to configure a multi-task loss function, wherein the multi-task loss function includes: a path generation error loss function, a meta-action consistency loss function, and a logical consistency loss function, wherein the path generation error loss function is used to optimize the difference between the model's predicted path and the real trajectory using mean squared error, the meta-action consistency loss function is used to optimize the selection of meta-actions using a classification loss function to ensure that the path planning decision is consistent with the generated meta-actions, and the logical consistency loss function is used to constrain the consistency between the explanatory information in path planning and the generated semantic information; and a training subunit, configured to jointly train an initial model using the sample data and the multi-task loss function to obtain the path planning model.

[0021] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0022] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0023] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0024] The beneficial effects of this invention are: 1. By utilizing image data collected by multi-view surround-view cameras of the vehicle, and mapping the visual input of multiple cameras into semantic features, combined with vehicle state and temporal features, and combining multimodal fusion technology with a deep learning model with logical reasoning ability, not only is the efficient generation of autonomous vehicle driving paths achieved, but also more reasonable and interpretable planning decisions are generated, thereby improving the safety and robustness of the autonomous driving system. 2. By generating logical and interpretable driving paths through high-level semantic information, the reliability of the autonomous driving paths generated by the autonomous driving module is guaranteed. This solves the technical problem of low reliability of autonomous driving paths generated by end-to-end driving models in related technologies, and improves the safety and robustness of the autonomous driving system. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a hardware structure block diagram of a car according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for generating an autonomous driving trajectory according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the functional module structure involved in the embodiments of the present invention; Figure 4 This is a schematic diagram of the processing flow of an embodiment of the present invention; Figure 5 This is a structural block diagram of an autonomous driving trajectory generation device according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1 The method embodiment provided in Embodiment 1 of this application can be executed in an automobile, server, processor, security controller, autonomous driving / assisted driving / intelligent driving controller, or similar processing device. Taking its operation in an automobile as an example, Figure 1 This is a hardware structure block diagram of a car according to an embodiment of the present invention. For example... Figure 1 As shown, a car may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the vehicle may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned automobile. For example, the automobile may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store vehicle programs, such as application software programs and modules, like the vehicle program corresponding to a method for generating an autonomous driving trajectory for a vehicle in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the vehicle program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a vehicle's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0031] This embodiment provides a method for generating autonomous driving trajectories. Figure 2 This is a flowchart of a method for generating an autonomous driving trajectory according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain multi-view image data captured by the vehicle's camera and obtain the attribute parameters of the camera; In this embodiment, multi-view image data can be acquired using the vehicle's vision cameras. Six onboard surround-view cameras capture image data from different perspectives, including front, rear, front left, rear left, front right, and rear right. The image input from each perspective is processed by a 2D convolutional neural network (CNN) to extract local visual features from each image. During this process, a 2D convolutional backbone network is utilized. 2D visual images acquired by cameras from each frame and viewpoint Perform image feature encoding to obtain the coded features of each frame and each viewpoint of the camera image. .

[0032] Step S202: Generate two-dimensional image features using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple viewpoints; The two-dimensional image features in this embodiment include local visual features from six perspectives: front, back, left front, left back, right front, and right back.

[0033] Step S203: Generate bird's-eye view BEV features using the two-dimensional image features and the attribute parameters, and generate high-level semantic information using the two-dimensional image features. The high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to semantically interpret the meta-actions. Optionally, meta-actions include lateral meta-actions and longitudinal meta-actions. The action set of longitudinal meta-actions includes: acceleration, maintaining, and deceleration. The action set of lateral meta-actions includes: right turn, going straight, and left turn. Enhanced semantics then interprets the corresponding meta-actions. For example, "The vehicle decelerates while going straight (meta-action) because it is approaching the intersection and the light is red (interpretation)." Enhanced semantics is used to explain the logical basis of the meta-action.

[0034] Step S204: Generate the autonomous driving path of the vehicle using the BEV features and the high-level semantic information; The autonomous driving path in this embodiment is a finely planned path that the vehicle can understand and directly execute.

[0035] Through the above steps, multi-view image data collected by the vehicle's camera and attribute parameters of the camera are obtained; two-dimensional image features are generated using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple perspectives; bird's-eye view BEV features are generated using the two-dimensional image features and the attribute parameters, and high-level semantic information is generated using the two-dimensional image features, wherein the high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe coarse actions in path planning, and the enhanced semantics are used to semantically interpret the meta-actions; the autonomous driving path of the vehicle is generated using the BEV features and the high-level semantic information. By generating a logical and interpretable driving path through the high-level semantic information, the reliability of the autonomous driving path generated by the autonomous driving module is ensured, solving the technical problem of low reliability of autonomous driving paths generated by end-to-end driving models in related technologies, and improving the safety and robustness of the autonomous driving system.

[0036] In one embodiment of this example, generating BEV features using the two-dimensional image features and the attribute parameters includes: reading the intrinsic parameters of each camera and the extrinsic parameters relative to the vehicle from the attribute parameters; and projecting the two-dimensional image features onto the vehicle's body coordinate system using the intrinsic and extrinsic parameters to obtain the BEV features.

[0037] In this embodiment, a 2D convolutional neural network backbone structure is used to encode the multi-view image data collected by the six-camera surround view camera, and extract the local visual features of each view. Then, by combining the intrinsic parameters of each camera and the extrinsic parameters relative to the vehicle body, the image features of each view are converted into a unified surround view BEV (Bird's Eye View) view feature representation in the vehicle coordinate system through a geometric projection method, so as to provide standardized input features for downstream path planning.

[0038] Optional, 2D convolutional backbone network Input 2D visual images from cameras at various viewpoints Shared weights can be used, but are not limited to, commonly used 2D convolutional network structures with a certain depth, such as ResNet, EfficientNet, SwinTransformer, VoVNetV2, ConvNeXt, and their CBNet combined backbone network architecture.

[0039] In this embodiment, generating high-level semantic information using the two-dimensional image features includes: acquiring the vehicle's input commands and reading preset surround-view prompts, wherein the surround-view prompts include prompts for multiple viewpoints, and the input commands include user commands and navigation commands; inputting the two-dimensional image features, the input commands, and the surround-view prompts into a multimodal large model to generate corresponding global visual semantic features and text features, wherein the multimodal large model includes a visual encoder, a language encoder, a text encoder, and a cross-attention network, the cross-attention network being used to align the features of the two-dimensional image features and the surround-view prompts; and generating high-level semantic information using the global visual semantic features, the text features, and the surround-view prompts.

[0040] This embodiment introduces LLaVa (Large Language and Vision Assistant) as a multimodal large model to enhance semantic understanding of scenes by fusing visual and linguistic features. Image data from different perspectives collected by a six-camera surround-view camera in the vehicle is used as the visual input to the LLaVa model. LLaVa processes these images using a CLIP (Contrastive Language-Image Pretraining)-based visual encoder to generate global visual semantic features. The multimodal large model introduced in this embodiment combines the advantages of computer vision and natural language processing, enabling it to extract high-level semantic information from visual input and utilize its vision-language multimodal alignment capability to achieve scene understanding. By pre-training on massive image-text pairs, the multimodal large model learns rich semantic knowledge, allowing it to accurately understand visual information and generate reasonable semantic descriptions even in scenarios with few samples or even no labeled data. This generalization ability is very suitable for the varied and complex scenarios in autonomous driving. Multimodal large models can efficiently align visual and linguistic features to form a unified multimodal representation, effectively integrating semantic information from surround-view camera input and driving scenarios (such as traffic rules and road signs), providing a more comprehensive environmental understanding for path planning. Unlike traditional perception models, multimodal large models can generate action descriptions and parsing (such as "pedestrians are crossing the road," "construction area ahead," "accelerate to turn right"). This information not only provides additional guidance for the planning model but also significantly improves the interpretability of the planning results.

[0041] In one example, generating high-level semantic information using the global visual semantic features, the text features, and the lookaround cue includes: generating a text token using the text features, generating an image tag using the global visual semantic features; inputting the text token, the image tag, and the lookaround cue into a large language model, and outputting high-level semantic information.

[0042] To handle the relative positions of multi-view images in three-dimensional space, it is necessary to combine the intrinsic parameters of each camera from different viewpoints. and external parameters relative to the vehicle body coordinate system The visual encoder uses a multi-view image sequence As input, image features are extracted, encoded and compressed by the driving vision adapter, and image labels are obtained. . , and These are the number of images, the number of image tags per image, and the feature dimensions of LVLM. and These are the image's height and width, respectively. The text includes a lookaround indicator.<front><FRONT RIGHT><FRONT LEFT><BACK RIGHT><BACK LEFT> <back>The system inputs images and text, along with basic user instruction prompts such as "Look around and plan a meta-action for safe driving." Both images and text are fed into Large Vision-Language Models (LVLMs) to predict high-level decisions. Finally, the meta-action encoder encodes the meta-actions and scene descriptions in the decision-making process and outputs meta-action features. ,in It is a feature dimension of the end-to-end planning model.

[0043] In one example, the size of each image can be adjusted to... This results in 576 image tags. For multi-image inputs, this leads to an excessive number of image tags, which not only slows down VLM training and inference but also causes model crashes and decoding failures. Therefore, this example performs additional encoding and compression of the image features to reduce the number of image tags. A set of image queries is used. To encode image features and output image labels, MHSA refers to bullish self-attention. It is a visual encoder. It is a multilayer perceptron composed of linear layers and GELU activation layers, used to project image features. and They represent the first One image Image tagging and image querying.

[0044] Optionally, before inputting the two-dimensional image features and the look-around cue into the multimodal large model, the method further includes: configuring an adaptation layer between the visual encoder and the language encoder of the multimodal large model, wherein the adaptation layer includes a visual adaptation module and a language adaptation module, the visual adaptation module is used to adjust the visual features output by the visual encoder through additional convolutional layers and fully connected layers so that the adjusted visual features are adapted to different visual tasks in the autonomous driving environment, and the language adaptation module is used to adjust the input format of the language encoder using prefix cues corresponding to the visual task so that the adjusted input format is related to the natural language instructions in the autonomous driving scenario.

[0045] To achieve lightweight fine-tuning, an adapter layer can be introduced between the vision and language encoders of LLaVa (a multimodal large model). The adapter layer is primarily responsible for adapting the pre-trained LLaVa model to meet the specific needs of autonomous driving tasks. The specific design includes: a vision adaptation module: adjusting visual features through additional convolutional and fully connected layers to adapt them to different vision tasks in the autonomous driving environment (such as obstacle detection and lane recognition); and a language adaptation module: using task-specific prompts to adjust the input format of the language model, making it more relevant to natural language instructions in autonomous driving scenarios, such as "The vehicle is currently slowing down because the red light is on." To improve the performance of the LLaVa model in autonomous driving tasks, this embodiment designs a multi-task loss function, including the following parts: Description generation loss: calculating the difference between the generated scene description and the labeled descriptions in the Deep Drive-X Dataset (BDD-X), using cross-entropy loss to optimize the description generation quality; and interpretation generation loss: optimizing the semantic consistency between the generated scene interpretation and the real labeled interpretation, using evaluation metrics such as BLEU or ROUGE for training. Meta-action consistency loss: Ensure that the generated meta-actions are logically consistent with the support of visual and linguistic input, and avoid decisions that are inconsistent with the scene description.

[0046] In the multimodal large model, its language encoder adopts the LLaMA (Large Language Model) architecture, which can process natural language instructions, such as: "You are..."<FRONT RIGHT> Questions like "What does the camera see?" or "What is the reason for the current path selection?" help the model understand the context and provide more information during path generation. To generate stable results, a fixed and effective question-and-answer format is designed. A simple and effective surround-view cue is designed for driving scenarios. Taking the front view as an example, the corresponding cue is:<FRONT VIEW> :\n \n, where These are special markers in LLM that are replaced by image markers during generation. The prompts include: scene description, traffic signal verification, vulnerable road user identification, motion intent prediction, meta-action planning, and planning parsing. This embodiment aligns visual features (2D image features) and linguistic instructions (look-around cues) using a cross-attention mechanism. The LLaVa model deeply fuses visual and linguistic features through this mechanism, enhancing its scene understanding capabilities and generating action descriptions and parsing based on linguistic input. For example, "The vehicle slows down while going straight (meta-action) because it is approaching an intersection and the light is red (explanation)." These action descriptions and parsing provide high-level semantic guidance for path planning decisions, further improving the accuracy and safety of path planning.

[0047] In one embodiment of this example, generating an autonomous driving path for the vehicle using the BEV features and the high-level semantic information includes: detecting dynamic objects based on the BEV features and generating a local map; extracting meta-action features from the high-level semantic information using a meta-action encoder and converting the meta-action features into embedding vectors; predicting the future trajectory of the dynamic object in the local map; inputting the future trajectory and the embedding vector into a path planning model, and outputting the autonomous driving path for the vehicle using the embedding vector as prior information.

[0048] Meta-motion encoder Using a set of learnable embeddings Achieve a one-to-one mapping from meta-actions to meta-action features, where This refers to the number of meta-actions. The process of generating meta-action features is as follows: ,in For the first The image corresponds to the viewfinder; among which ,for The text token obtained after text encoder analysis of the sequence of consecutive meta-actions and the overall scene analysis of the entire sequence. This represents the number of text tokens. Subsequently, the meta-action features are input into the path planning module and combined with the action prediction results to plan the trajectory.

[0049] The path planning model directly predicts the vehicle's future autonomous driving path based on the input multimodal BEV features and the generated meta-action features. It consists of three modules: an end-to-end perception module that detects dynamic objects and generates local maps; an end-to-end motion prediction module that predicts the future trajectories of dynamic objects; and an end-to-end planning module (path planning model), which uses a set of planning labels to interact with scene features through an attention mechanism to predict the autonomous driving trajectory. Figure 3 This is a schematic diagram of the functional module structure involved in the embodiment of the present invention. It also illustrates a multimodal large model, which, combined with its Adapter layer, generates high-level semantic information (meta-actions and their semantic interpretations) with semantic enhancement. The meta-action encoder obtains meta-action features, which are then input to the end-to-end planning module. Since the meta-action features are in the form of embedded vectors, fine-tuning LLaVa can be easily combined with other end-to-end models.

[0050] In one example, before inputting the future trajectory and the meta-action features into the path planning model, the method further includes: acquiring sample data, wherein the sample data includes: sample multi-view image data, sample vehicle driving data, sample high-precision map data, and sample meta-actions related to the path; configuring a multi-task loss function, wherein the multi-task loss function includes: a path generation error loss function, a meta-action consistency loss function, and a logical consistency loss function. The path generation error loss function is used to optimize the difference between the model's predicted path and the actual trajectory using mean squared error. The meta-action consistency loss function is used to optimize the selection of meta-actions using a classification loss function to ensure that the path planning decision is consistent with the generated meta-actions. The logical consistency loss function is used to constrain the consistency between the explanatory information in path planning and the generated semantic information. The initial model is jointly trained using the sample data and the multi-task loss function to obtain the path planning model.

[0051] The constructed path planning model needs to be pre-trained to optimize its path generation and logical reasoning capabilities. The specific steps are as follows: Data preparation: Data from six onboard cameras and vehicle driving data are used, combined with high-precision map annotations of key elements in the driving scene (such as lane lines, traffic lights, etc.). The training data also includes path-related action descriptions and analyses (such as reasons for vehicle deceleration, obstacle avoidance decisions, etc.). During training, a multi-task loss function is used for joint optimization. The loss functions include: Path generation error loss: Optimizing the difference between the model's predicted path and the actual trajectory using mean squared error (MSE). Meta-action consistency loss: Ensuring consistency between path planning decisions and generated meta-actions, optimizing the selection of meta-actions using a classification loss function. Logical consistency loss: Constraining the consistency between the interpretation in path planning and the generated semantic information, optimizing the model's interpretability. By jointly training the model using multi-task loss functions, path generation, logical reasoning, and interpretability are simultaneously optimized. Finally, the model's accuracy and interpretability are evaluated using validation and test sets, ensuring its stable operation in complex autonomous driving environments.

[0052] This embodiment proposes an autonomous driving path planning method based on a multimodal large model and an end-to-end deep learning framework. This method utilizes panoramic perception information captured by a six-camera surround-view camera, combined with multimodal fusion technology and a deep learning model with logical reasoning capabilities. On the basis of achieving efficient generation of autonomous vehicle driving paths, it enhances the interpretability of the path generation process through logical reasoning, providing a safer and more reliable technical solution for autonomous driving in training processes and complex driving scenarios. Figure 4 This is a schematic diagram of the processing flow of an embodiment of the present invention, including the following steps: Step 1: Using a 2D convolutional neural network backbone structure, feature encoding is performed on the multi-view image data collected by the six-camera surround view camera to extract the local visual features of each view. Then, combining the intrinsic parameters of each camera and the extrinsic parameters relative to the vehicle body, the image features of each view are converted into a unified surround view BEV view feature representation in the vehicle coordinate system through geometric projection, providing standardized input features for downstream path planning.

[0053] Step 2: Introduce a multimodal large model (e.g., a visual language model) and design an Adapter layer for lightweight fine-tuning and training, tailored to the needs of autonomous driving scenarios. The Adapter layer adapts a small amount of labeled data to combine the generalization ability of the multimodal large model with the requirements of actual driving scenarios, improving the model's performance on specific tasks while retaining its general reasoning capabilities.

[0054] Step 3: Generate high-level semantic action descriptions and parsings by analyzing multi-view image data. The multimodal large model improves its understanding of complex driving environments by integrating image features with linguistic knowledge (such as traffic rules and dynamic scene descriptions). The high-level semantic information generated by the model mainly consists of two parts: one is the "meta-actions" used to guide path planning, divided into lateral and longitudinal parts. The longitudinal part includes "acceleration," "maintaining," and "deceleration," while the lateral part includes "right turn," "straight ahead," and "left turn." The other part is path interpretation, which includes scene descriptions and high-level decision-making information after logical reasoning (such as target lane selection and obstacle avoidance strategies). Additionally, scene description information is used to enhance the scene semantics of the BEV features generated in Step S1.

[0055] Step 4: Construct an end-to-end planning model (path planning model). Using the generated "meta-actions" as prior information, and combining multimodal features from the BEV perspective, directly output the vehicle's target driving path and current actions within several future time steps. Through model structure design, ensure the accuracy and consistency of path generation, and retain interpretable intermediate results during path generation, such as obstacle avoidance criteria or lane selection logic. The path generation model is pre-trained. Multi-view image data from a six-camera surround-view system and the vehicle's actual driving trajectory are collected, along with key semantic information from the driving scene annotated using high-precision maps. Through multi-task loss function design, the path generation error and meta-action execution consistency loss are jointly optimized to improve the safety and robustness of the path planning model.

[0056] This embodiment utilizes panoramic perception information captured by a six-camera surround-view system, combined with multimodal fusion technology and a deep learning model with logical reasoning capabilities. This not only enables efficient generation of the vehicle's driving path but also enhances the interpretability of the path generation process through logical reasoning, providing a safer and more reliable technical solution for autonomous driving in complex driving scenarios.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0058] Example 2 This embodiment also provides an apparatus for generating autonomous driving trajectories, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0059] Figure 5 This is a structural block diagram of an autonomous driving trajectory generation device according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: The acquisition module 51 is used to acquire multi-view image data captured by the vehicle's camera and to acquire the attribute parameters of the camera; The first generation module 52 is used to generate two-dimensional image features using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple viewpoints. The second generation module 53 is used to generate bird's-eye view BEV features using the two-dimensional image features and the attribute parameters, and to generate high-level semantic information using the two-dimensional image features. The high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to provide semantic interpretation of the meta-actions. The third generation module 54 is used to generate the autonomous driving path of the vehicle using the BEV features and the high-level semantic information.

[0060] Optionally, the second generation module includes: a reading unit, used to read the intrinsic parameters and extrinsic parameters relative to the vehicle from the attribute parameters; and a projection unit, used to project the two-dimensional image features onto the vehicle's body coordinate system using the intrinsic and extrinsic parameters to obtain BEV features.

[0061] Optionally, the second generation module includes: a processing unit, configured to acquire input commands from the vehicle and read preset surround-view prompts, wherein the surround-view prompts include prompts for multiple viewpoints, and the input commands include user commands and navigation commands; a first generation unit, configured to input the two-dimensional image features, the input commands, and the surround-view prompts into a multimodal large model to generate corresponding global visual semantic features and text features, wherein the multimodal large model includes a visual encoder, a language encoder, a text encoder, and a cross-attention network, the cross-attention network being used to align the features of the two-dimensional image features and the surround-view prompts; and a second generation unit, configured to generate high-level semantic information using the global visual semantic features, the text features, and the surround-view prompts.

[0062] Optionally, the second generation unit includes: a generation subunit, used to generate a text token using the text features and generate an image tag using the global visual semantic features; and an output subunit, used to input the text token, the image tag, and the lookaround cue into a large language model and output high-level semantic information.

[0063] Optionally, the second generation module further includes: a configuration unit, configured to configure an adaptation layer between the visual encoder and the language encoder of the multimodal large model before the first generation unit inputs the two-dimensional image features and the look-around cue into the multimodal large model, wherein the adaptation layer includes a visual adaptation module and a language adaptation module, the visual adaptation module is configured to adjust the visual features output by the visual encoder through additional convolutional layers and fully connected layers so that the adjusted visual features are adapted to different visual tasks in the autonomous driving environment, and the language adaptation module is configured to adjust the input format of the language encoder using prefix cues corresponding to the visual task so that the adjusted input format is related to natural language instructions in the autonomous driving scenario.

[0064] Optionally, the third generation module includes: a processing subunit, configured to detect dynamic objects based on the BEV features and generate a local map, extract meta-action features of the high-level semantic information using a meta-action encoder, and convert the meta-action features into embedding vectors; a prediction subunit, configured to predict the future trajectory of the dynamic object in the local map; and an output subunit, configured to input the future trajectory and the embedding vector into a path planning model, and output the autonomous driving path of the vehicle using the embedding vector as prior information.

[0065] Optionally, the third generation module further includes: an acquisition subunit, configured to acquire sample data before the output subunit inputs the future trajectory and the meta-action features into the path planning model, wherein the sample data includes: sample multi-view image data, sample vehicle driving data, sample high-precision map data, and sample meta-actions related to the path; a configuration subunit, configured to configure a multi-task loss function, wherein the multi-task loss function includes: a path generation error loss function, a meta-action consistency loss function, and a logical consistency loss function, wherein the path generation error loss function is used to optimize the difference between the model's predicted path and the real trajectory using mean squared error, the meta-action consistency loss function is used to optimize the selection of meta-actions using a classification loss function to ensure that the path planning decision is consistent with the generated meta-actions, and the logical consistency loss function is used to constrain the consistency between the explanatory information in path planning and the generated semantic information; and a training subunit, configured to jointly train an initial model using the sample data and the multi-task loss function to obtain the path planning model.

[0066] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0067] Example 3 Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0068] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, acquire multi-view image data captured by the vehicle's camera, and acquire the attribute parameters of the camera; S2, using the multi-view image data to generate two-dimensional image features, wherein the two-dimensional image features include local visual features from multiple viewpoints; S3, using the two-dimensional image features and the attribute parameters to generate bird's-eye view BEV features, and using the two-dimensional image features to generate high-level semantic information, the high-level semantic information including meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to semantically interpret the meta-actions; S4, using the BEV features and the high-level semantic information to generate the autonomous driving path for the vehicle.

[0069] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0070] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0071] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0072] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, acquire multi-view image data captured by the vehicle's camera, and acquire the attribute parameters of the camera; S2, using the multi-view image data to generate two-dimensional image features, wherein the two-dimensional image features include local visual features from multiple viewpoints; S3, using the two-dimensional image features and the attribute parameters to generate bird's-eye view BEV features, and using the two-dimensional image features to generate high-level semantic information, the high-level semantic information including meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to semantically interpret the meta-actions; S4, using the BEV features and the high-level semantic information to generate the autonomous driving path for the vehicle.

[0073] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0074] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0076] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a," "an," and "described" as used herein may also mean including the plural forms. The terms "comprising," "including," "containing," and "having" are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a specific order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0077] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / back> < / front>

Claims

1. A method for generating an autonomous driving trajectory, characterized in that, include: Acquire multi-view image data captured by the vehicle's camera, and acquire the attribute parameters of the camera; Two-dimensional image features are generated using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple viewpoints; The two-dimensional image features and the attribute parameters are used to generate bird's-eye view BEV features, and the two-dimensional image features are used to generate high-level semantic information. The high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to semantically interpret the meta-actions. The autonomous driving path of the vehicle is generated using the BEV features and the high-level semantic information.

2. The method according to claim 1, characterized in that, Generating BEV features using the two-dimensional image features and the attribute parameters includes: Read the intrinsic parameters and extrinsic parameters relative to the vehicle for each camera from the attribute parameters; The two-dimensional image features are projected onto the vehicle's body coordinate system using the intrinsic and extrinsic parameters to obtain BEV features.

3. The method according to claim 1, characterized in that, Generating high-level semantic information using the two-dimensional image features includes: The system acquires input commands from the vehicle and reads preset surround view prompts, wherein the surround view prompts include prompts for multiple viewpoints, and the input commands include user commands and navigation commands. The two-dimensional image features, the input command, and the lookaround cue are input into a multimodal large model to generate global visual semantic features and text features. The multimodal large model includes a visual encoder, a language encoder, a text encoder, and a cross-attention network. The cross-attention network is used to align the features of the two-dimensional image features and the lookaround cue. High-level semantic information is generated using the global visual semantic features, the text features, and the lookaround cue.

4. The method according to claim 3, characterized in that, Generating high-level semantic information using the global visual semantic features, the text features, and the lookaround cue includes: Text tokens are generated using the aforementioned text features, and image tags are generated using the aforementioned global visual semantic features; The text token, the image tag, and the lookaround cue are input into the large language model, and high-level semantic information is output.

5. The method according to claim 3, characterized in that, Before inputting the two-dimensional image features and the look-around cue into the multimodal large model, the method further includes: An adaptation layer is configured between the visual encoder and the language encoder of the multimodal large model. The adaptation layer includes a visual adaptation module and a language adaptation module. The visual adaptation module is used to adjust the visual features output by the visual encoder through additional convolutional layers and fully connected layers so that the adjusted visual features are adapted to different visual tasks in the autonomous driving environment. The language adaptation module is used to adjust the input format of the language encoder using prefix cues corresponding to the visual tasks so that the adjusted input format is related to the natural language instructions in the autonomous driving scenario.

6. The method according to claim 1, characterized in that, Generating the autonomous driving path for the vehicle using the BEV features and the high-level semantic information includes: Dynamic objects are detected based on the BEV features and a local map is generated. Meta-action encoders are used to extract meta-action features of the high-level semantic information and convert the meta-action features into embedding vectors. Predict the future trajectory of the dynamic object in the local map; The future trajectory and the embedded vector are input into the path planning model, and the autonomous driving path of the vehicle is output using the embedded vector as prior information.

7. The method according to claim 6, characterized in that, Before inputting the future trajectory and the meta-action features into the path planning model, the method further includes: Acquire sample data, wherein the sample data includes: multi-view image data, vehicle driving data, high-precision map data, and path-related meta-actions. Configure a multi-task loss function, which includes: a path generation error loss function, a meta-action consistency loss function, and a logical consistency loss function. The path generation error loss function is used to optimize the difference between the model's predicted path and the actual trajectory using mean squared error. The meta-action consistency loss function is used to optimize the selection of meta-actions using a classification loss function to ensure that the path planning decision is consistent with the generated meta-actions. The logical consistency loss function is used to constrain the consistency between the explanatory information in path planning and the generated semantic information. The path planning model is obtained by jointly training the initial model using the sample data and the multi-task loss function.

8. An apparatus for generating autonomous driving trajectories, characterized in that, include: The acquisition module is used to acquire multi-view image data captured by the vehicle's camera and to acquire the attribute parameters of the camera; The first generation module is used to generate two-dimensional image features using the multi-view image data, wherein the two-dimensional image features include local visual features from multiple viewpoints; The second generation module is used to generate bird's-eye view BEV features using the two-dimensional image features and the attribute parameters, and to generate high-level semantic information using the two-dimensional image features. The high-level semantic information includes meta-actions and enhanced semantics, wherein the meta-actions are used to describe the coarse actions of path planning, and the enhanced semantics are used to provide semantic interpretation of the meta-actions. The third generation module is used to generate the autonomous driving path of the vehicle using the BEV features and the high-level semantic information.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 7 when it is run.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 7.

Images