Method and device for generating automatic driving trajectory

Abstract

The embodiments of this specification provide a method and apparatus for generating an autonomous driving trajectory. The method includes: obtaining multimodal data collected by vehicle-mounted sensors; the multimodal data includes data of multiple different data types; inputting the data of the multiple different data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of multiple different data types; fusing the features of the multiple different data types to obtain a high-dimensional feature space vector; inputting the feature space vector into a large multimodal model for inference, and using a trajectory decoder to convert the inference results of the large multimodal model into an autonomous driving trajectory.

Description

Technical Field

[0001] This specification relates to the field of autonomous driving technology, and in particular to a method and device for generating an autonomous driving trajectory. Background Art

[0002] With the continuous development of autonomous driving technology, how to realize autonomous driving planning has become a hot topic in current research.

[0003] In recent years, the application of large language models (LLMs) has continued to expand, with precedents showing their successful application in the field of autonomous driving. Leveraging their advantages in generalized reasoning, logical deduction, and contextual modeling, their application to autonomous driving has become a growing trend.

[0004] To address the problem of generating autonomous driving trajectories in the field of autonomous driving, related technologies transform the trajectory planning task into a language modeling problem, thereby utilizing large language models to generate autonomous driving trajectories based on geographic coordinate descriptions.

[0005] However, the final output of existing large language models is in text form. Such results can only serve as a formal prompt and have little impact on actual autonomous driving. Summary of the Invention

[0006] In view of this, one or more embodiments of this specification provide a method and device for generating an autonomous driving trajectory, an electronic device, and a storage medium to solve the problems existing in the related art.

[0007] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0008] According to a first aspect of an embodiment of this specification, a method for generating an autonomous driving trajectory is provided, the method comprising:

[0009] Acquire multimodal data collected by vehicle-mounted sensors; the multimodal data includes data of multiple different data types;

[0010] Inputting the data of the multiple different data types into encoders corresponding to the respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of the multiple different data types;

[0011] Fusing the features of the multiple different data types to obtain a high-dimensional feature space vector;

[0012] The feature space vector is input into a multimodal large model for inference, and a trajectory decoder is used to convert the inference result of the multimodal large model into an autonomous driving trajectory.

[0013] Optionally, the acquiring of multimodal data collected by the vehicle-mounted sensor includes at least one of the following:

[0014] Acquire a video image of the vehicle's surroundings captured by an on-board image acquisition device;

[0015] Obtain vehicle motion data collected by on-board sensors;

[0016] Obtain vehicle navigation data received by the vehicle's navigation device.

[0017] Optionally, the step of inputting the data of the multiple different data types into encoders corresponding to the respective data types for encoding to obtain multimodal features with unified feature representation includes at least one of the following:

[0018] Input the video image into the visual transformer for encoding processing to obtain image features;

[0019] The vehicle motion data is input into the trajectory autoencoder for encoding processing to obtain motion features;

[0020] Inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features;

[0021] The image features, motion features, and navigation features have a unified feature representation.

[0022] Optionally, the step of fusing the features of the multiple different data types to obtain a high-dimensional feature space vector includes:

[0023] Obtaining basic prompt words for guiding the multimodal large model to reason in the expected direction;

[0024] The basic prompt word is fused with the features of the multiple different data types to obtain a high-dimensional feature space vector.

[0025] Optionally, the multimodal large model also outputs text information related to the current driving situation, so that the autonomous driving or the driver can make driving decisions based on the text information.

[0026] Optionally, the text information includes at least one of a description of the vehicle's surroundings, a description of the vehicle's motion status, and a description of navigation instructions.

[0027] Optionally, the method further includes:

[0028] Acquire training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include expected autonomous driving trajectories based on the temporal behavior data and environmental perception data;

[0029] A multimodal large model is trained based on the training samples; wherein, a thinking chain is introduced in the process of training the multimodal large model, and a plurality of reasoning tasks are constructed in the thinking chain to help the multimodal large model learn the mapping relationship between the input samples and the output labels.

[0030] Optionally, the multiple reasoning tasks include:

[0031] Inferring the intentional command task of expected driving behavior under input samples;

[0032] Reasoning about scene description tasks that describe the environment perception data in the input sample;

[0033] Inference is the object labeling task of labeling the objects of interest present in the context perception data in the input sample.

[0034] Optionally, a visual expert model is also introduced in the process of training the multimodal large model. The visual expert model is used to identify the objects of interest in the environmental perception data, and input the recognition results into the multimodal large model to assist the multimodal large model in learning multiple reasoning tasks of the mapping relationship between the input samples and the output labels.

[0035] Optionally, the inference result of the multimodal large model includes a plurality of groups of longitudinal dynamic control variables and lateral motion planning parameters that are continuous in time sequence;

[0036] Wherein, the longitudinal dynamics control variable includes the intensity of the quantified throttle or brake operation;

[0037] The lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0038] Optionally, the multimodal large model includes an end-to-end multimodal large model.

[0039] According to a second aspect of an embodiment of this specification, a device for generating an autonomous driving trajectory is provided, the device comprising:

[0040] An acquisition unit, which acquires multimodal data collected by the vehicle-mounted sensor; the multimodal data includes data of multiple different data types;

[0041] an encoding unit, inputting the data of the plurality of different data types into encoders corresponding to the respective data types for encoding processing, thereby obtaining multimodal features with unified feature representation; the multimodal features include features of the plurality of different data types;

[0042] A fusion unit, fusing the features of the multiple different data types to obtain a high-dimensional feature space vector;

[0043] A generation unit inputs the feature space vector into a multimodal large model for inference, and uses a trajectory decoder to convert the inference result of the multimodal large model into an autonomous driving trajectory.

[0044] According to a third aspect of the embodiments of this specification, there is provided an electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0045] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0046] According to a fourth aspect of the embodiments of this specification, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-readable instructions, and the machine-readable instructions implement the above method when called and executed by a processor.

[0047] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0048] A trajectory decoder is added on the basis of the multimodal large model, and the trajectory decoder is used to convert the inference results of the multimodal large model into an autonomous driving trajectory to achieve precise vehicle control. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method for generating an autonomous driving trajectory provided by an exemplary embodiment of this specification;

[0050] Figure 2 A schematic diagram of the architecture of a multimodal large model provided as an exemplary embodiment of this specification;

[0051] Figure 3 A schematic diagram of training a large multimodal model provided as an exemplary embodiment of this specification;

[0052] Figure 4 This is a schematic diagram of the structure of an electronic device in which an autonomous driving trajectory generation device is located, provided in an exemplary embodiment of this specification;

[0053] Figure 5A block diagram of an apparatus for generating an autonomous driving trajectory according to an exemplary embodiment of this specification is provided. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0055] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0056] As mentioned previously, in the field of autonomous driving, the problem of generating autonomous driving trajectories is addressed by transforming the trajectory planning task into a language modeling problem, thereby utilizing large language models to generate autonomous driving trajectories based on geographic coordinate descriptions. However, existing large language models ultimately output text, which serves only as a formal prompt and has little impact on actual autonomous driving.

[0057] In view of this, this specification provides a solution for generating an autonomous driving trajectory, which is described in detail below with reference to the accompanying drawings.

[0058] See Figure 1 , Figure 1 This is a flow chart of a method for generating an automatic driving trajectory provided by an exemplary embodiment of this specification. Figure 1 As shown, the method may include the following steps:

[0059] Step 110: Acquire multimodal data collected by the vehicle-mounted sensor; the multimodal data includes data of multiple different data types.

[0060] This manual uses Multimodal Large Language Models (MLLMs).

[0061] Multimodal large models extend the semantic understanding capabilities of traditional large language models (LLMs) to the field of visual cognition. LLMs leverage their strengths in generalized reasoning, logical deduction, and contextual modeling to build a foundational framework for cross-modal cognition, enabling visual-linguistic cross-modal alignment.

[0062] In this embodiment, it is first necessary to obtain data of multiple different data types collected by several different vehicle-mounted sensors. These data are generally called multimodal data.

[0063] In an exemplary embodiment, acquiring multimodal data collected by the vehicle-mounted sensor may include at least one of the following:

[0064] Acquire a video image of the vehicle's surroundings captured by an on-board image acquisition device;

[0065] Obtain vehicle motion data collected by on-board sensors;

[0066] Obtain vehicle navigation data received by the vehicle's navigation device.

[0067] In this embodiment, the image acquisition device may include, but is not limited to, cameras mounted on the vehicle. Multiple cameras may be deployed at different locations on the vehicle to capture video images from different angles around the vehicle, such as a front camera for capturing images of the vehicle's front environment, a rear camera for capturing images of the vehicle's rear environment, and side cameras for capturing images of the vehicle's sides. Multiple cameras can capture a 360° surround view, providing a richer and more accurate environmental reference for generating autonomous driving trajectories.

[0068] The vehicle motion data includes, for example, but is not limited to, the vehicle's speed, acceleration, heading angle (used to characterize the direction of the vehicle's front), body posture, etc. The vehicle motion data can be used to describe the vehicle's motion state and environmental changes.

[0069] The vehicle navigation data may include map data provided by navigation satellites, vehicle positioning data, etc. A more accurate autonomous driving trajectory can be planned using the vehicle navigation data.

[0070] Step 120: Input the data of the multiple different data types into encoders corresponding to the respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of the multiple different data types.

[0071] In this embodiment, since the multimodal data are collected by different vehicle-mounted sensors, there are large differences in data standards such as scale and format, making them difficult to use directly. For this reason, encoding processing can be performed through their respective corresponding encoders to unify the feature representation between the multimodal data.

[0072] In an exemplary embodiment, inputting the data of the multiple different data types into encoders corresponding to the respective data types for encoding to obtain multimodal features with unified feature representation may include at least one of the following:

[0073] Input the video image into the visual transformer for encoding processing to obtain image features;

[0074] The vehicle motion data is input into the trajectory autoencoder for encoding processing to obtain motion features;

[0075] Inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features;

[0076] The image features, motion features, and navigation features have a unified feature representation.

[0077] like Figure 2 As shown in Figure 1, video images can be input into the Vision Transformer (ViT) for encoding. The Vision Transformer is a model based on the self-attention mechanism that can effectively capture complex features and relationships in video images.

[0078] Vehicle motion data can be input into a trajectory autoencoder (TAE) for encoding. The trajectory autoencoder specifically consists of a trajectory autoencoder (TAE) and an ego encoder (Ego Encoder).

[0079] The trajectory encoder is a deep learning model used to encode and decode time series data and extract high-level features from it. The above-mentioned vehicle motion data is a type of time series data, which includes vehicle motion data collected by on-board sensors over a continuous period of time (such as 5 seconds).

[0080] The goal of the self-encoder is to encode the ego-state of the vehicle motion data (such as position, velocity, pose, etc.) and map it to a latent space through which the ego-state can be modeled and thus reacted in the decision-making process.

[0081] In this embodiment, the trajectory encoder and the autoencoder are used in conjunction. The trajectory autoencoder focuses more on how to extract high-level features from time series data, while the autoencoder focuses on modeling the vehicle's self-state. The combination of the two helps improve the perception, prediction, and decision-making capabilities of autonomous driving.

[0082] Vehicle navigation data can be input into a map encoder (such as SD Map Encoder) for encoding. The map encoder is responsible for parsing and understanding the map data and providing navigation support for the vehicle.

[0083] After being processed by their respective encoders, the above multimodal data can be converted into multimodal features with unified feature representation.

[0084] Step 130 : Fusing the features of the multiple different data types to obtain a high-dimensional feature space vector.

[0085] In this embodiment, multimodal data feature fusion can be performed using a multilayer perceptron (MLP). An MLP can be composed of multiple layers (such as an input layer, a hidden layer, and an output layer). Each layer can contain a number of neurons connected by weights. Since there are no loops between different layers, it is a feedforward neural network.

[0086] The input layer is used to receive multimodal features, and each data type of feature in the multimodal features can correspond to a neuron.

[0087] Hidden layers are located between the input and output layers. The neurons in the hidden layers can perform nonlinear changes to the output values of the neurons in the input layer to extract the relationships between multimodal features. This nonlinear change can be achieved through activation functions.

[0088] The output layer is used to integrate the processed multimodal features into a unified, high-dimensional feature space vector to better capture the complex relationship between multimodal data.

[0089] In an exemplary embodiment, fusing the features of the multiple different data types to obtain a high-dimensional feature space vector may further include:

[0090] Obtaining basic prompt words for guiding the multimodal large model to reason in the expected direction;

[0091] The basic prompt word is fused with the features of the multiple different data types to obtain a high-dimensional feature space vector.

[0092] like Figure 2 As shown, the multi-layer perceptron also includes basic prompts when performing feature fusion. The basic prompts can be preset instructions or questions used to guide the output of the multimodal large model. By integrating the basic prompts into the feature space vector, the multimodal large model can understand the reasoning task requirements, thereby better generating an expected autonomous driving trajectory.

[0093] Step 140: Input the feature space vector into a multimodal large model for inference, and use a trajectory decoder to convert the inference result of the multimodal large model into an autonomous driving trajectory.

[0094] like Figure 2 As shown in Figure 3, the fused feature space vector can be input into a large multimodal model for inference.

[0095] By integrating vision-related image data, motion-related vehicle motion data, and vehicle navigation data into the multimodal large model, the multimodal large model can more comprehensively understand the current driving situation and output inference results that are consistent with the current driving situation.

[0096] On the one hand, the integration of multimodal data can not only improve the accuracy and reliability of the autonomous driving system, but also enhance its adaptability to complex environments.

[0097] On the other hand, in response to the inference results output by the multimodal large model, this embodiment adds a trajectory decoder based on the multimodal large model. The trajectory decoder uses the inference results of the multimodal large model to generate a visualized autonomous driving trajectory.

[0098] In an exemplary embodiment, Figure 2 As shown, in addition to outputting inference results for the trajectory decoder to generate an autonomous driving trajectory, the multimodal large model can also output text information related to the current driving situation, so that the autonomous driving or the driver can make driving decisions based on the text information.

[0099] Exemplarily, the text information includes at least one of a description of the vehicle's surroundings, a description of the vehicle's motion state, and a description of navigation instructions.

[0100] Textual reasoning utilizes the powerful language generation capabilities of large multimodal models to convert complex driving information into easy-to-understand text form, improving the interpretability of the generated autonomous driving trajectory and thus providing decision support for autonomous driving or drivers.

[0101] In an exemplary embodiment, the multimodal large model may include an end-to-end multimodal large model.

[0102] The end-to-end multimodal big model is also called the multimodal big model with full-process integrated architecture. Compared with the multimodal big model with traditional layered architecture, the end-to-end multimodal big model improves the overall system efficiency.

[0103] In the traditional layered architecture, each functional module is optimized independently, and features need to be transferred between different functional modules during the model inference process. The end-to-end architecture achieves global optimization by building a shared feature extraction backbone network and eliminates redundant links in feature transfer between functional modules, thereby improving computing resource utilization. In this way, compared with the traditional layered architecture, the overall system performance is improved.

[0104] In practical applications, large multimodal models mainly rely on the capabilities of the base model itself and lack knowledge of autonomous driving. Therefore, the reasoning results given may not conform to driving logic and have poor interpretability.

[0105] For this purpose, this manual Figure 1 Based on the embodiment shown, the training process of the multimodal large model is improved, and a thought chain is introduced during the training process to guide the fine-tuning of the multimodal large model in the field of autonomous driving. The specific embodiments are as follows:

[0106] In an exemplary embodiment, the method further includes:

[0107] Acquire training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include expected autonomous driving trajectories based on the temporal behavior data and environmental perception data;

[0108] A multimodal large model is trained based on the training samples; wherein, a thinking chain is introduced in the process of training the multimodal large model, and a plurality of reasoning tasks are constructed in the thinking chain to help the multimodal large model learn the mapping relationship between the input samples and the output labels.

[0109] exist Figure 3 In the diagram of training a large multimodal model, by jointly modeling temporal behavior data and environmental perception data, a probability distribution prediction of the expected future autonomous driving trajectory is achieved to form a complete perception-decision closed loop.

[0110] The time series behavior data may include vehicle motion data within a continuous historical period (such as the driving speed, acceleration, heading angle, vehicle body posture, etc. shown above). The environmental perception data may include video images and spatiotemporal context information.

[0111] During the training process, the input samples in the training samples are used as model input and input into the multimodal large model that introduces the thinking chain. The multiple reasoning tasks constructed by the thinking chain are used to guide the multimodal large model to gradually learn the mapping relationship between the input samples and the corresponding output labels.

[0112] In an exemplary embodiment, the plurality of reasoning tasks include:

[0113] Intent Command task, Scene Description task, and Major Objects task.

[0114] Among them, the intention command task is used to infer the expected driving behavior under the input samples.

[0115] The expected driving behavior may refer to a clear statement of the vehicle's expected direction of movement in the current scenario. For example:

[0116] Continue driving in the current lane, or make a left turn, right turn, or go straight.

[0117] Determine whether to maintain the current speed, or perform deceleration, acceleration, etc.

[0118] These intention instructions provide clear guidance for subsequent autonomous driving trajectory planning, ensuring the consistency and safety of driving decisions.

[0119] Among them, the scene description task is used to infer the scene description of the environmental perception data in the input sample.

[0120] The scene description may be a concise and clear description of the current driving scene, taking into account factors such as the status of traffic lights, the dynamics of surrounding vehicles or pedestrians, and lane markings. For example:

[0121] "The traffic light ahead is red. There are no other vehicles on the left or right lanes, but there are pedestrians crossing the right crosswalk."

[0122] "The traffic light ahead is green, and there is a car in the left lane preparing to merge into this lane."

[0123] These scenario descriptions help to fully understand the current driving environment and provide strong support for making accurate driving decisions.

[0124] Among them, the object labeling task is used to infer the object labeling task of labeling the objects of interest present in the environmental perception data in the input sample.

[0125] Object annotation can refer to identifying and marking road users, i.e., objects that require special attention during autonomous driving, and clearly identifying the specific locations of these objects in the video image of the driving scene. A brief description of each object is provided, explaining its current behavior and its importance to driving decisions. For example:

[0126] Pedestrian: "In the right crosswalk, crossing the street to the left. Importance: Slow down to ensure safe passage."

[0127] Vehicle: "A car on the left is accelerating and preparing to merge into this lane. Importance: Avoid it. You may need to adjust your speed or lane."

[0128] Through the above-mentioned reasoning tasks, not only can detailed driving intentions and scene descriptions be generated, but traffic participants and their behaviors can also be accurately identified, providing more accurate and safe decision support for the autonomous driving system.

[0129] The above reasoning process simulates the thinking mode of human drivers, improves the transparency and explainability of autonomous driving, and enhances the ability to cope with complex driving situations.

[0130] In an exemplary embodiment, a visual expert model is also introduced into the process of training the multimodal large model;

[0131] like Figure 3 As shown, the visual expert model is used to identify the objects of interest in the environmental perception data and input the recognition results into the multimodal large model to assist the multimodal large model in learning multiple reasoning tasks of the mapping relationship between the input samples and the output labels.

[0132] This embodiment can accurately and efficiently identify objects (traffic participants such as pedestrians, other vehicles, traffic lights, lane signs, etc.) present in environmental perception data (such as video images) by adding a visual expert model, helping the multimodal large model to perform the above-mentioned reasoning tasks.

[0133] For example, for an intention-command task, if the visual expert model recognizes that the current vehicle lane is a through lane, the current traffic light is red for straight ahead, and a pedestrian is crossing the crosswalk, it can infer that the expected driving behavior is to brake and stop at the stop line in the through lane and wait for the green light to pass.

[0134] For the scene description task, based on the recognition results of the above-mentioned visual expert model, it can be generated that the current straight traffic light is red and there are pedestrians crossing the crosswalk.

[0135] For the description task, based on the recognition results of the above-mentioned visual expert model, the location of traffic lights, pedestrians, other vehicles, etc. can be marked on the video image.

[0136] In an exemplary embodiment, the inference results output by the multimodal large model may include a plurality of temporally continuous groups of longitudinal dynamic control variables and lateral motion planning parameters;

[0137] Wherein, the longitudinal dynamics control variable includes the intensity of the quantified throttle or brake operation;

[0138] The lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0139] Different from the traditional direct trajectory regression method in the local coordinate system, this embodiment designs a dual-channel intermediate representation including longitudinal dynamic control quantity and lateral motion planning parameters. The intensity of the throttle or braking operation quantified by the dynamic control quantity is used to map the speed change requirement, and the steering mechanism angle described by the lateral motion planning parameters reflects the curvature adjustment amplitude of the autonomous driving trajectory.

[0140] The design of this dual-channel intermediate representation is inspired by modeling human driving behavior: longitudinal control determines velocity, while lateral parameters constrain direction. This dual-channel intermediate representation not only improves the interpretability of the decision-making process but also mitigates the curse of dimensionality caused by high-dimensional feature inputs during autonomous driving trajectory prediction through parameter decoupling. (Increasing the dimensionality of input features exacerbates data sparsity, computational complexity, and overfitting.)

[0141] Returning to step 140, the multimodal large model can generate several groups of longitudinal dynamic control quantities S and lateral motion planning parameters C within the next T seconds based on the input feature space vector and the mapping relationship learned under the guidance of the aforementioned thinking chain. Each group of longitudinal dynamic control quantities and lateral motion planning parameters can be regarded as the control variables of a trajectory point; then the trajectory decoder converts these several groups of longitudinal dynamic control quantities S and lateral motion planning parameters C within the next T seconds into autonomous driving trajectories, so that the autonomous driving system can control the vehicle motion based on the autonomous driving trajectory.

[0142] In an exemplary embodiment of this specification, an electronic device capable of implementing the above method is also provided.

[0143] Figure 4 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 4At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor reading the corresponding computer program from the non-volatile memory into the memory and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0144] Please refer to Figure 5 In one software embodiment, a device for generating an autonomous driving trajectory is provided, the device comprising:

[0145] An acquisition unit 510 acquires multimodal data collected by an onboard sensor; the multimodal data includes data of multiple different data types;

[0146] The encoding unit 520 inputs the data of the multiple different data types into encoders corresponding to the respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of the multiple different data types;

[0147] A fusion unit 530 performs feature fusion on the features of the multiple different data types to obtain a high-dimensional feature space vector;

[0148] The generating unit 540 inputs the feature space vector into a multimodal large model for inference, and uses a trajectory decoder to convert the inference result of the multimodal large model into an autonomous driving trajectory.

[0149] Optionally, the acquiring unit 510 includes at least one of the following:

[0150] Acquire a video image of the vehicle's surroundings captured by an on-board image acquisition device;

[0151] Obtain vehicle motion data collected by on-board sensors;

[0152] Obtain vehicle navigation data received by the vehicle's navigation device.

[0153] Optionally, the encoding unit 520 includes at least one of the following:

[0154] Input the video image into the visual transformer for encoding processing to obtain image features;

[0155] The vehicle motion data is input into the trajectory autoencoder for encoding processing to obtain motion features;

[0156] Inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features;

[0157] The image features, motion features, and navigation features have a unified feature representation.

[0158] Optionally, the fusion unit 540 includes:

[0159] Obtain basic prompt words for guiding the multimodal large model to reason in the expected reasoning direction; perform feature fusion on the basic prompt words and the features of the multiple different data types to obtain a high-dimensional feature space vector.

[0160] Optionally, the multimodal large model also outputs text information related to the current driving situation, so that the autonomous driving or the driver can make driving decisions based on the text information.

[0161] Optionally, the text information includes at least one of a description of the vehicle's surroundings, a description of the vehicle's motion status, and a description of navigation instructions.

[0162] Optionally, the device further includes:

[0163] a sample acquisition unit for acquiring training samples, wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include expected autonomous driving trajectories based on the temporal behavior data and environmental perception data;

[0164] A model training unit trains a multimodal large model based on the training samples; wherein, a thought chain is introduced in the process of training the multimodal large model, and a plurality of reasoning tasks are constructed in the thought chain to help the multimodal large model learn the mapping relationship between the input samples and the output labels.

[0165] Optionally, the multiple reasoning tasks include:

[0166] Inferring the intentional command task of expected driving behavior under input samples;

[0167] Reasoning about scene description tasks that describe the environment perception data in the input sample;

[0168] Inference is the object labeling task of labeling the objects of interest present in the context perception data in the input sample.

[0169] Optionally, a visual expert model is also introduced in the process of training the multimodal large model. The visual expert model is used to identify the objects of interest in the environmental perception data, and input the recognition results into the multimodal large model to assist the multimodal large model in learning multiple reasoning tasks of the mapping relationship between the input samples and the output labels.

[0170] Optionally, the inference result of the multimodal large model includes a plurality of groups of longitudinal dynamic control variables and lateral motion planning parameters that are continuous in time sequence;

[0171] Wherein, the longitudinal dynamics control variable includes the intensity of the quantified throttle or brake operation;

[0172] The lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0173] Optionally, the multimodal large model includes an end-to-end multimodal large model.

[0174] The implementation process of the functions and effects of each module in the above-mentioned device is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method for generating the autonomous driving trajectory. For relevant matters, please refer to the partial description of the method implementation method, which will not be repeated here.

[0175] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0176] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0177] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0178] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0179] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0181] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0183] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0184] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for generating an autonomous driving trajectory, characterized in that: The method comprises: Acquire multimodal data collected by vehicle-mounted sensors; the multimodal data includes data of multiple different data types; Inputting the data of the multiple different data types into encoders corresponding to the respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of the multiple different data types; Fusing the features of the multiple different data types to obtain a high-dimensional feature space vector; Inputting the feature space vector into a large multimodal model for inference, and converting the inference results of the large multimodal model into an autonomous driving trajectory using a trajectory decoder; wherein the large multimodal model extends the semantic understanding capabilities of traditional large-scale language models to the field of visual cognition; The multimodal large model is trained in the following way: Acquire training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include expected autonomous driving trajectories based on the temporal behavior data and environmental perception data; Training a large multimodal model based on the training samples; wherein a thought chain is introduced in the process of training the large multimodal model, and a plurality of reasoning tasks are constructed in the thought chain to help the large multimodal model learn the mapping relationship between the input samples and the output labels; The multiple reasoning tasks include: Inferring the intentional command task of expected driving behavior under input samples; Reasoning about scene description tasks that describe the environment perception data in the input sample; Inference is the object labeling task of labeling the objects of interest present in the context perception data in the input sample.

2. The method according to claim 1, characterized in that The acquiring of multimodal data collected by the vehicle-mounted sensor includes at least one of the following: Acquire a video image of the vehicle's surroundings captured by an on-board image acquisition device; Obtain vehicle motion data collected by on-board sensors; Obtain vehicle navigation data received by the vehicle's navigation device.

3. The method according to claim 2, characterized in that The step of inputting the data of the plurality of different data types into encoders corresponding to the respective data types for encoding to obtain multimodal features with unified feature representation includes at least one of the following: Input the video image into the visual transformer for encoding processing to obtain image features; The vehicle motion data is input into the trajectory autoencoder for encoding processing to obtain motion features; Inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features; The image features, motion features, and navigation features have a unified feature representation.

4. The method according to claim 1, wherein The feature fusion of the multiple different data types to obtain a high-dimensional feature space vector includes: Obtaining basic prompt words for guiding the multimodal large model to reason in the expected direction; The basic prompt word is fused with the features of the multiple different data types to obtain a high-dimensional feature space vector.

5. The method according to claim 1, wherein The multimodal large model also outputs text information related to the current driving situation, so that the automatic driving or the driver can make driving decisions based on the text information.

6. The method according to claim 5, characterized in that The text information includes at least one of a description of the vehicle's surroundings, a description of the vehicle's motion state, and a description of navigation instructions.

7. The method according to claim 1, characterized in that A visual expert model is also introduced in the process of training the multimodal large model. The visual expert model is used to identify the objects of interest in the environmental perception data and input the recognition results into the multimodal large model to assist the multimodal large model in learning multiple reasoning tasks of the mapping relationship between the input samples and the output labels.

8. The method according to claim 1, characterized in that The inference results of the multimodal large model include a plurality of groups of longitudinal dynamic control variables and lateral motion planning parameters that are continuous in time sequence; Wherein, the longitudinal dynamics control variable includes the intensity of the quantified throttle or brake operation; The lateral motion planning parameters include the angle of the vehicle's steering mechanism.

9. The method according to claim 1, characterized in that The multimodal large model includes an end-to-end multimodal large model.

10. A device for generating an automatic driving trajectory, characterized in that: The device comprises: An acquisition unit, which acquires multimodal data collected by the vehicle-mounted sensor; the multimodal data includes data of multiple different data types; an encoding unit, inputting the data of the plurality of different data types into encoders corresponding to the respective data types for encoding processing, thereby obtaining multimodal features with unified feature representation; the multimodal features include features of the plurality of different data types; A fusion unit, fusing the features of the multiple different data types to obtain a high-dimensional feature space vector; a generation unit that inputs the feature space vector into a large multimodal model for inference, and converts the inference results of the large multimodal model into an autonomous driving trajectory using a trajectory decoder; wherein the large multimodal model extends the semantic understanding capabilities of traditional large-scale language models to the field of visual cognition; The multimodal large model is obtained by training the following units: a sample acquisition unit for acquiring training samples, wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include expected autonomous driving trajectories based on the temporal behavior data and environmental perception data; A model training unit trains a multimodal large model based on the training samples; wherein, a thought chain is introduced in the process of training the multimodal large model, and a plurality of reasoning tasks are constructed in the thought chain to help the multimodal large model learn the mapping relationship between the input samples and the output labels; the plurality of reasoning tasks include: an intention instruction task of reasoning about the expected driving behavior under the input samples; a scene description task of reasoning about the scene description of the environmental perception data in the input samples; and an object labeling task of reasoning about the objects of interest existing in the environmental perception data in the input samples.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

12. A machine-readable storage medium, characterized in that Machine-readable instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images