Automatic driving vehicle trajectory planning method

Abstract

The invention relates to an automatic driving vehicle trajectory planning method, which comprises the following steps of: extracting comprehensive spatio-temporal information representation of a driving scene from automatic driving multi-modal data by adopting an end-to-end branch, and predicting and generating a trajectory sequence predicted by a vehicle in a future set time period; a visual language model branch is adopted to embed visual features related to the text instruction into a large language model, and a plurality of different track sequences in a future set time period are obtained through prediction; and fusing the trajectory sequences output by the two branches according to the loss value and the danger situation of the current driving scene to obtain a final planning trajectory. Compared with the prior art, the method has the advantages of high safety and reliability.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for trajectory planning of an autonomous driving vehicle. Background Art

[0002] In the process of promoting the construction of the autonomous driving industry and its related supporting facilities, foundation intelligence will play a vital role. Among them, the foundation model, as a large-scale machine learning model, is pre-trained with large-scale data and can adapt to a wide range of downstream tasks by fine-tuning for specific tasks or specific fields. It is an important part of basic intelligence. In the field of natural language processing (NLP), large foundation models mainly include large language models (LLM) such as BERT and GPT series. Their application in autonomous driving is an inevitable development trend of artificial intelligence technology and has become a golden track for major manufacturers and teams to continue to compete.

[0003] In terms of interpretability, the academic and industrial communities are currently exploring the research of embedding language knowledge into decision-making in the field of autonomous driving: through pre-training with rich text instruction data, large language models can extract and interpret various elements in open driving scenarios, realize open scene understanding and reasoning in autonomous driving, and deduce natural language instructions for driving decision-making and planning. Regarding this topic, some studies have built autonomous driving models based on a GPT-like approach to directly output planning trajectories or control instructions, while other studies have tried to use Transformer to build end-to-end autonomous driving systems with good results. While completing the driving control task, natural language prompts are added to the decision-making process of the autonomous driving control module, which improves the interpretability of the results. In addition, autonomous driving can also apply real physical world models to learn background knowledge about how the world works by observing a small number of events that are related or unrelated to the task, grasp the internal laws and predict the evolution process. Using a deep generative model as a world model, it is possible to learn large-scale image data, fully unleash the model's reasoning ability, and build a general basic model for autonomous driving vision.

[0004] In terms of social interaction, it is extremely challenging for autonomous vehicles to predict the intentions of surrounding traffic participants and achieve socially compatible game interactions and behavior planning. Current autonomous vehicles are indeed lacking in these aspects. Sometimes they are too conservative and wait for a long time for safety goals, and sometimes they are too aggressive and compete with other traffic participants. The survey report shows that the public has a strong sense of distrust in the behavior of autonomous vehicles. 60% of the respondents do not believe in the safety and reliability of autonomous vehicles, especially in scenes where pedestrians are crossing the road. They are more worried that autonomous vehicles will collide with pedestrians and cause safety accidents. Summary of the invention

[0005] The purpose of the present invention is to provide a trajectory planning method for an autonomous driving vehicle in order to overcome the defects of the above-mentioned prior art, thereby overcoming the potential safety hazard problem caused by the long-tail distribution of edge scenes faced by autonomous driving vehicles.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for trajectory planning of an autonomous driving vehicle, the method comprising:

[0008] End-to-end branch: Extract features from multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene; Perform multi-scale feature learning on the comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic characteristics of the road, and extract the interactive characteristics of the traffic participants in the driving scene through the interactive network; Combine the historical dynamic information of the vehicle and the interactive characteristics of the traffic participants in the driving scene, and use the time series prediction module to predict the probability distribution of the vehicle's driving trajectory; Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, generate the trajectory sequence predicted by the vehicle within the set time period in the future;

[0009] Visual language model branch: extracts visual features related to text instructions from autonomous driving image data, embeds the visual features into a large language model through an adaptation network, combines them with coded prompts, outputs the environmental understanding of the driving scene in the form of structured text, and predicts several different trajectory sequences within a set period of time in the future;

[0010] Fusion of the end-to-end branch and the visual language model branch: Calculate the loss value of the trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. According to the size of the loss value and the dangerous situation of the current driving scene, the trajectory sequences output by the two branches are fused to obtain the final planning trajectory.

[0011] Preferably, the feature extraction of the autonomous driving multimodal data to obtain a comprehensive spatiotemporal information representation of the driving scene specifically includes: encoding the input continuous multi-frame point cloud data and image data respectively, concatenating the encoded point cloud feature L and image feature I into a sequence and a multimodal feature F of the image driving scene, inputting the multimodal feature F into a spatiotemporal Transformer network, and obtaining a bird's-eye view feature B of the driving scene through multimodal fusion, which is a comprehensive spatiotemporal information representation of the driving scene.

[0012] Preferably, the comprehensive spatiotemporal information representation of the driving scene is represented by multi-scale feature learning to capture the characteristics of each traffic participant and the basic characteristics of the road, and the interactive characteristics of the traffic participants in the driving scene are extracted through the interactive network, specifically including:

[0013] The query of each agent in the driving scene and the bird's-eye view feature B after spatiotemporal network encoding are input into the Transformer-based variable attention network to obtain the feature A of each agent in the driving scene;

[0014] Query the road facilities in the traffic scene M query The bird’s-eye view feature B after spatiotemporal network encoding is input into the Transformer-based variable attention network to obtain the road facility feature M in the traffic scene;

[0015] Query Ego query The characteristics A of each intelligent agent in the driving scenario and the characteristics M of the road facilities in the traffic scenario are input into the interactive network based on the graph neural network, and cross-attention calculation is performed to capture the feature interaction R of the multi-order neighborhood centered on the vehicle.

[0016] Preferably, the time series prediction module is a time series prediction module based on a U-Net network.

[0017] Preferably, the generating of the trajectory sequence predicted by the vehicle within a future set period of time based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction result specifically includes:

[0018] The comprehensive spatiotemporal information representation of the driving scene and the preliminary probability distribution prediction results are combined through the occupancy grid network to generate road scene constraints;

[0019] According to the road scene constraints and the preliminary probability distribution prediction results, the short-term trajectory prediction results are obtained through the trajectory planning network;

[0020] A Transformer-based regression trajectory generation network is used to convert the short-term trajectory prediction results into long-term trajectory planning results, which is the trajectory sequence predicted by the vehicle within a set period in the future.

[0021] Preferably, in the visual language model branch, the training process of the large language model includes:

[0022] Expanding datasets to train large language models using counterfactual-based visual instruction generation strategies; and / or

[0023] Using a multi-task encoder integrated with multiple visual encoders to extract latent visual information and using a knowledge-enhanced network to extract prior knowledge from visual features; and / or

[0024] Auxiliary losses are used to supervise the training process, and a reinforcement learning strategy with human feedback is combined to update the pre-trained large language model.

[0025] Preferably, the trajectory sequences output by the two branches are fused according to the loss value and the dangerous situation of the current driving scene, and the fusion process includes the following situations:

[0026] 1) When the visual language model branch does not generate an alarm and the loss value is less than the set threshold, only short-term collision detection is performed, and correction and optimization are performed based on the planned trajectory to obtain the final trajectory prediction result;

[0027] 2) When the visual language model branch does not generate an alarm warning but the loss value exceeds the set threshold, a relatively conservative driving strategy is selected between the two branches;

[0028] 3) When the visual language model branch generates an alarm but the loss value is less than the set threshold, trajectory verification is performed and an alternative safe trajectory is planned;

[0029] 4) When the visual language model branch generates an alarm but the loss value exceeds the set threshold, a new safest trajectory is re-judged or planned.

[0030] Preferably, the dangerous situation of the driving scene is judged according to a visual language model, 0 indicates that there is no danger in the current scene, and 1 indicates that the current scene is a dangerous scene.

[0031] Preferably, after the end-to-end branch is merged with the visual language model branch to obtain the final planning trajectory, the final planning trajectory is further subjected to temporal logic security verification, specifically:

[0032] The signal timing logic STL is used to characterize traffic rules and safety criteria. The situation is judged by combining scene extraction with traffic rules to generate the first STL specification φ1 that characterizes traffic rules. The safety criteria and driving scene information are integrated into the second STL specification φ2. The safety quantification values ​​of the first and second specifications φ1 and φ2 are obtained using the STL robust function.

[0033] If the safety quantization values ​​corresponding to the current planned trajectory meet the requirements, the vehicle controller will output the reference trajectory after safety verification, otherwise the backup trajectory planning algorithm will be used to re-plan the trajectory.

[0034] Preferably, before the end-to-end branch is merged with the visual language model branch, the method further includes: converting the trajectory sequence output by the visual language model branch from a two-dimensional point coordinate system to a three-dimensional world coordinate system that is the same as the end-to-end branch output through a depth estimation algorithm to achieve coordinate alignment.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) The present invention fuses the trajectory sequence output by the end-to-end branch and the trajectory sequence output by the visual language model branch. The fusion process takes into account the trajectory error value, collision risk and dangerous situation of the driving scene. When the end-to-end branch cannot accurately perceive the surrounding strange objects in extreme, escape or dangerous scenes, a drivable path can still be planned based on the visual language model branch, which effectively solves the safety hazard problem caused by the long-tail distribution of edge scenes faced by autonomous driving vehicles, and has higher safety and reliability.

[0037] 2) The present invention integrates the visual language model branch driven by the large language model with the end-to-end branch, realizing the interactive decision-making participation of intelligent driving AI, which can effectively reduce the probability of manual takeover of the vehicle and improve the platform operation efficiency.

[0038] 3) The present invention adopts a counterfactual-based visual instruction generation strategy to expand the data set to train a large language model, adopts a multi-task encoder integrated with multiple visual encoders to extract potential visual information, and uses a knowledge enhancement network to extract prior knowledge from visual features to enhance the visual perception ability of the large language model, or adopts an auxiliary loss to supervise the training process, and combines a reinforcement learning strategy with human feedback to update the pre-trained large language model, which solves the problem that the generated text content of the large language model is often inconsistent with the corresponding visual content.

[0039] 4) Through the safety verification of the temporal logic planning trajectory, the risk of traffic accidents is effectively reduced, the passenger travel experience is improved, and it serves as a good demonstration for the operation of intelligent connected vehicles across the country. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the method architecture of the present invention;

[0041] Figure 2 It is an end-to-end branch architecture;

[0042] Figure 3 Branching architecture for visual language models. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] Example

[0045] This embodiment integrates the end-to-end branch and the visual language model branch to build a safe and explainable end-to-end AI driving model, and constructs an autonomous driving vehicle trajectory planning method, which includes:

[0046] End-to-end branch: Extract features from multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene; Perform multi-scale feature learning on the comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic characteristics of the road, and extract the interactive characteristics of the traffic participants in the driving scene through the interactive network; Combine the historical dynamic information of the vehicle and the interactive characteristics of the traffic participants in the driving scene, and use the time series prediction module to predict the probability distribution of the vehicle's driving trajectory; Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, generate the trajectory sequence predicted by the vehicle within the set time period in the future;

[0047] Visual language model branch: extracts visual features related to text instructions from autonomous driving image data, embeds the visual features into a large language model through an adaptation network, combines them with coded prompts, outputs the environmental understanding of the driving scene in the form of structured text, and predicts several different trajectory sequences within a set period of time in the future;

[0048] Fusion of the end-to-end branch and the visual language model branch: Calculate the loss value of the trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. According to the size of the loss value and the dangerous situation of the current driving scene, the trajectory sequences output by the two branches are fused to obtain the final planning trajectory.

[0049] Next, combined with the model structure (such as Figure 1 The method of this embodiment is further described in detail.

[0050] Step 1: Prepare the autonomous driving dataset and build the automatic annotation tool.

[0051] In order to build a scene understanding and planning dataset for traffic scenarios, data collection and integration are carried out from two aspects. On the one hand, multiple open source large-scale surround view datasets from home and abroad are collected, covering urban road scenes and various extreme scenes, including: NuScenes (Boston, Singapore) and Waymo dataset (USA), etc. On the other hand, image and video data taken over by intelligent driving vehicles are collected to build a large surround view sequence dataset, and data annotation is performed based on this dataset. First, the categories of long-tail objects and dangerous scenes are pre-defined, such as: traffic lights with strange shapes, animals crossing the road, children and the elderly on the road, etc., and image-based and text-based search methods are used to mine the above scenes in the dataset to form a new dangerous scene dataset for model training. For each dangerous scene, a large language model is used to give a detailed scene description as well as the category, state and impact of each key target object on the vehicle. The constructed dataset can be used for subsequent end-to-end large model training and performance verification.

[0052] Afterwards, the method of mind chain guided training is adopted, and the visual language model is used to give a detailed text description of the driving scene, including weather conditions, road types, lane passability and key target coordinates, etc. At the same time, based on the analysis of the impact of key targets on the driving of the vehicle, the future driving suggestions and strategies for the vehicle are given. By analyzing a video sequence, the complete driving action sequence of the vehicle is obtained, such as: acceleration, deceleration, slow straight driving, changing lanes to the left, changing lanes to the right, etc., as well as the target area that the vehicle is expected to reach (for example, 1 to 3 seconds later) in the 2D image coordinate system.

[0053] Step 2: Overall end-to-end model architecture design and training, such as Figure 2 As shown. In an autonomous driving system, in order to obtain a comprehensive spatiotemporal information representation of the driving scene, it is necessary to fuse multimodal data from multiple sensors. Specifically, multiple frames of continuous point cloud data and image data can be used as input. First, the point cloud sequence and image sequence are encoded separately to extract their feature representations. Then, the features of these two modalities are fused, and the fused features can contain rich information from both the point cloud and the image, and fully describe the three-dimensional spatial structure and visual details of the driving scene.

[0054] In the planning of autonomous driving systems, it is crucial to consider the interactive relationship between various traffic elements. The system needs to analyze the dynamics of participants such as vehicles, pedestrians, and bicycles on the road in real time, while also considering the influence of environmental factors such as traffic signals and road signs. Only by fully understanding and simulating these complex interactive relationships can the autonomous driving system make safe and efficient decisions. To this end, we need to design an interactive module. Multi-scale learning is performed from multimodal driving scene information to capture the characteristics of each traffic participant and the basic characteristics of the road, and then based on this, the interactive network analysis scene is constructed to extract the interactive features of the traffic participants. This comprehensive interactive analysis helps the system better adapt to the complex and changing traffic environment and improve driving safety and comfort.

[0055] ① Representation of spatiotemporal information of multimodal driving scenarios:

[0056] In an autonomous driving system, in order to obtain a comprehensive spatiotemporal information representation of the driving scene, it is necessary to fuse multimodal data from multiple sensors. Specifically, multiple frames of continuous point cloud data and image data can be used as input. First, the point cloud sequence and image sequence are encoded separately to extract their feature representations. Then, the features of these two modalities are fused. The fused features can contain rich information from both the point cloud and the image, and fully describe the three-dimensional spatial structure and visual details of the driving scene.

[0057] F=C(I,L) (1)

[0058] B = TSTransformerr(B query , F, F) (2)

[0059] Among them, I represents the features encoded by the image, L represents the features encoded by the point cloud, C represents the feature splicing operation, F represents the output features of the splicing, and B query I represents the initialization query, TSTransformer represents the spatiotemporal Transformer network, and B represents the bird’s-eye view feature after spatiotemporal network encoding. First, I and L are concatenated and fused to obtain the multimodal feature F of the driving scene, and then TSTransformer is used to generate the multimodal feature F of the driving scene. r The network transforms the multimodal fusion feature F into a bird's-eye view feature B of the driving scene.

[0060] ② Dynamic interactive learning of multiple traffic elements

[0061] In the planning of autonomous driving systems, it is crucial to consider the interactions between various traffic elements. The system needs to analyze the dynamics of participants such as vehicles, pedestrians, and bicycles on the road in real time, while also considering the impact of environmental factors such as traffic signals and road signs. Only by fully understanding and simulating these complex interactions can the autonomous driving system make safe and efficient decisions. First, we need to perform multi-scale learning from multi-modal driving scene information to capture the characteristics of each traffic participant and the basic characteristics of the road.

[0062] A=DATransformer(A query , B, B) (3)

[0063] M=DATransformer(M query , B, B) (4)

[0064] Among them, A query represents the query of each agent in the driving scene, B represents the bird's-eye view feature after spatiotemporal network coding, M query represents the query of road facilities in traffic scenarios, A represents the characteristics of each agent in the driving scenario, M represents the characteristics of road facilities in the traffic scenario, and DATrasformer represents the variable attention network based on Transformer.

[0065] Under the condition of obtaining the characteristics of each important traffic element, we designed an interaction module. Here, the vehicle is used as the example center, and the interaction characteristics of traffic participants extracted from the interaction network analysis scene are used. This comprehensive interaction analysis helps the system better adapt to the complex and changing traffic environment and improve driving safety and comfort. The interactive generation method here enables the model to capture the subtle dependencies between trajectories and scenes. The generated trajectories can meet multiple constraints such as road structure, avoid obstacles, and follow traffic rules, thereby enhancing the social interaction attributes of the autonomous driving system.

[0066] R=Interaction(Ego query , A, M) (5)

[0067] Among them, Ego query Represents the query of the ego-car, Interaction represents the Transformer-based interaction module, which performs cross-attention calculation to capture the feature interaction R of the multi-order neighborhood centered on the ego-car instance.

[0068] ③Multi-agent trajectory prediction with integrated planning

[0069] In the autonomous driving system, it is necessary to directly predict the future movement trajectory of each traffic participant to provide input for the final motion planning and control. The interactive features R obtained above are input into the prediction module to predict the future movement of each traffic participant. In particular, the historical dynamic information Ego of the vehicle is incorporated here. state Therefore, the planning of the ego vehicle is integrated into the prediction module TUNet, and a preliminary probability distribution prediction P of the future driving trajectory of the ego vehicle is made.

[0070] P=TUNet(MLP(Egos tate ), R) (6)

[0071] Among them, TUNet represents the time series prediction module based on the U-Net network, which integrates the bird's-eye view feature B of the driving scene with the prediction result P through the occupancy grid network to generate the road scene constraint O, and then safely and effectively generates the future driving trajectory u of the vehicle:

[0072] O=OccHead(P,B,B) (7)

[0073] u=PlanHead(P,O) (8)

[0074] Among them, OccHead represents the occupancy grid network, and PlanHead is the short-term trajectory planning network based on Transformer.

[0075] Specifically, according to the collision constraints of the road scene and the motion prediction results obtained by the preliminary probability distribution, a short-term trajectory sequence is generated through the Transformer-based short-term trajectory planning network planning: the probability field is used to continuously map from the action space to the probability distribution, and each trajectory action is encoded into a high-dimensional input, embedded in the Transformer decoder network, and combined with the current driving scene information to output the planning results.

[0076] In particular, in order to achieve long-term planning, a regressive future driving trajectory planning is designed. After obtaining the trajectory planning results in the short-term domain, the results are encoded into an input sequence, and the Transformer network based on the attention mechanism is used to obtain the trajectory points for a period of time in the future through decoding prediction. The long-term planning result u * The expression is:

[0077] u * =AgGen(u t ,u t+1 ,…,u t+T ) (9)

[0078] Among them, AgGen represents the regression trajectory generation network based on Transformer, ut ,u t+1 , ..., u t+T Represents the short-term results planned by the trajectory planning network PlanHead.

[0079] Step 3: Visual language model architecture design and training, such as Figure 3 shown.

[0080] The Transformer-based query network is used to extract the visual input most relevant to the text instructions, and then the visual features are embedded into the large language model through the adaptation network. Combined with the encoded prompts, the model can output the environmental understanding of the traffic scene in the form of structured text, and provide reasonable driving semantic instructions for planning tasks, improving the generalization ability to cope with unknown scenes. The visual language models used are QwenVL, InternVL, mPLUG-Owl, etc. During the network training process, the pre-trained model parameter weights are frozen. By adding an adaptation network to be trained in each layer of the Transformer architecture, the ability to understand the scene is improved without increasing the training cost, greatly reducing the number of trainable parameters for downstream tasks.

[0081] In addition to driving instructions in semantic form, the visual language model can also infer the target area that the vehicle is expected to reach (for example, 1 to 3 seconds later) in the 2D image coordinate system. The future trajectory of the vehicle obtained by the end-to-end branch is in the world coordinate system. In order to align the outputs of the two branches, the depth estimation method is used to project the 2D point coordinates to obtain the coordinate values ​​in the 3D world coordinate system. The advantage of this architecture is that in extreme, escape or dangerous scenarios, when the end-to-end branch cannot accurately perceive the surrounding strange objects (such as puddles, ice and snow, etc.), it can still plan a drivable path based on the output of the visual language model. The specific projection transformation method from the image coordinate system to the world coordinate system is as follows: First, the image coordinates are back-projected to the camera coordinate system. The depth information needs to be estimated. An algorithm based on a convolutional neural network can be used to obtain the distance from the pixel to the camera.

[0082]

[0083] Where (x, y) represents the coordinates of the trajectory point in the image coordinate system; f represents the focal length of the camera, Represents the perspective projection matrix, with a dimension of 3×4; Z c Indicates the depth value that needs to be estimated; (X c , Y c , Z c ) represents the coordinates of the trajectory point in the mapped camera coordinate system.

[0084] After that, transform the camera coordinates to world coordinates:

[0085]

[0086] in, represents the camera external parameter matrix, R is the rotation matrix with a dimension of 3×3, and T is the translation matrix with a dimension of 3×1; (X w , Y w , Z w ) represents the coordinates of the trajectory point in the mapped world coordinate system.

[0087] In addition, due to the insufficient amount of training data, noise in the training data, and lack of diversity, the multimodal large language model often generates text content that is inconsistent with the corresponding visual content, which is called the hallucination problem of the large language model. To solve the hallucination problem of the model, improvements can be made in three aspects: data, model, and training process. First, the data set is expanded by using a counterfactual-based visual instruction generation strategy, and the multimodal large language model trained on the calibration data set is proven to be less prone to hallucinations. Secondly, in terms of the model, a multi-task encoder can be used to integrate the potential visual information extracted by multiple visual encoders, and a knowledge enhancement module can be used to extract prior knowledge from the visual input, thereby enhancing the visual perception ability of the large language model. Finally, during the model training process, an auxiliary loss can be used to supervise the training process, and a reinforcement learning strategy with human feedback can be combined to update the pre-trained large language model to improve visual features and reduce the occurrence of hallucinations.

[0088] Step 4: Fusion of visual language model and end-to-end model.

[0089] The VLM branch prediction results in several different trajectory sequences within the next 3 seconds. n represents the number of predicted trajectory sequences, corresponding to aggressive and conservative driving styles on different lanes. Specifically, different driving styles can be judged by the velocity change rate, acceleration change rate, and whether sudden acceleration and deceleration occur. The end-to-end branch outputs a trajectory sequence with the highest confidence in the next 3 seconds [(x1, y1), (x2, y2), (x3, y3)]. The average L2 displacement loss is calculated between the trajectory sequence output from the end-to-end and each trajectory output by the VLM, and the trajectories output by the two branches are fused according to the size of the loss value and the dangerous situation of the current scene. Among them, the dangerous situation of the scene is judged by the VLM, 0 means that there is no danger in the current scene, and 1 means that the current scene is a dangerous scene. The specific fusion process includes the following 4 cases:

[0090] 1) When the VLM branch does not generate an alarm and the calculated L2 loss value is less than the set threshold ∈, it means that the trajectories obtained by the VLM and the end-to-end branches are basically the same, and the driving scene at this time is relatively safe. Therefore, when it is handed over to the subsequent trajectory safety verification module, only short-term collision detection is performed to avoid lowering the performance of the AI ​​system. Correction and optimization can be performed on the basis of the existing planned trajectory to obtain the final trajectory prediction result.

[0091] 2) When the VLM branch does not generate an alarm but the calculated L2 loss value exceeds the set threshold ∈, it means that the trajectories obtained by the VLM and the end-to-end branch are inconsistent. Although the VLM did not alarm, it may be a missed alarm. In order to ensure driving safety and minimize the possibility of false negative samples (FN), the subsequent safety verification module can select a relatively conservative driving strategy in the two branches.

[0092] 3) When the VLM branch generates an alarm but the calculated L2 loss value is less than the set threshold ∈, it indicates that there may be driving risks in the current scenario. The trajectories planned by the VLM and the end-to-end branches are consistent. It is possible that both models are planned correctly, the VLM model has a false alarm, or both branches are planned incorrectly. Therefore, at this time, the safety verification module needs to perform strict trajectory verification and plan an alternative safety trajectory.

[0093] 4) When the VLM branch generates an alarm but the calculated L2 loss value exceeds the set threshold ∈, it means that the vehicle is in a dangerous driving scenario and the trajectories planned by the two branches are inconsistent. This is the most serious situation and requires the back-end safety verification module to re-judge or plan the safest trajectory.

[0094] Step 5: Safety verification of the planned trajectory based on the temporal logic method.

[0095] Develop safety quantification and verification technology, that is, describe safety requirements through high-level specifications of the target system, generate corresponding safety quantifiers, and analyze and verify whether the running trajectory planned by the AI ​​model meets safety requirements. Safety quantification technology first seeks a suitable formal logic language to accurately describe the safety criteria of the target system.

[0096] In order to solve the formal safety verification problem of autonomous driving, we firstly propose to use signal temporal logic (STL) to characterize traffic rules and safety criteria. By extracting scenarios and combining them with traffic rules to make situation judgment, we can generate STL specification φ1 that represents traffic rules. Similarly, we integrate safety criteria with driving scenario information and summarize them into STL specification φ2. Then, we use STL robust semantics to construct a safety quantification framework of legitimacy and rationality for specifications φ1 and φ2 respectively. By observing factors such as road models, traffic signal models and traffic participants in real time, we use data-driven AI models to plan trajectories, input the reference trajectory into the legitimacy and rationality safety quantifier, and use STL robust functions to obtain the safety quantification values ​​of STL specifications φ1 and φ2, so as to quantify the risk safety of the scenario. When STL specifications φ1 and φ2 meet the requirements, we can design a correct controller to track the reference trajectory; when the robustness of STL specifications φ1 or φ2 is negative, we use the backup trajectory planning algorithm to re-plan the trajectory.

[0097] Step 6: Model deployment and real vehicle testing.

[0098] Efficient deployment of end-to-end driving models requires comprehensive consideration of latency and waiting time, existing and memory usage, scalability and high throughput, hardware resource utilization, and model size and performance evaluation to optimize and solve. Through the optimization combination method, the model compression and pruning scheme is explored, and a new pruning metric is introduced to focus on identifying the interdependent structure within the model. Based on the importance evaluation results, the granularity is further refined, and the second-order importance of a single parameter and the group importance of the coupling structure are added to comprehensively consider, while maximizing the model compression efficiency and maintaining the performance of the model. At the same time, a low-precision inference algorithm is used to design the control model parameters and memory usage, speculative sampling is performed on the model, and low-bit quantization is performed on the activation value outliers. FP8 is tried for model training, which greatly improves the inference speed while ensuring controllable accuracy loss. Relying on the computing unit configuration of the smart car, and ensuring that the computing platform is integrated with other electronic control units (ECUs) and sensors of the vehicle, ensure that data can be transmitted and processed in real time. At the same time, interfaces with sensors and vehicle control systems are developed to ensure that sensor data can be transmitted to the model in real time, and model output is transmitted to the vehicle control system. Standardized communication protocols are used to ensure stable communication between components. Specifically, the ROS method is used to parse the large model algorithm into CAN signals, and the CAN communication protocol is used between the industrial computer and the controller.

[0099] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for trajectory planning of an autonomous driving vehicle, characterized in that: The method includes: End-to-end branch: Extract features from multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene; Perform multi-scale feature learning on the comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic characteristics of the road, and extract the interactive characteristics of the traffic participants in the driving scene through the interactive network; Combine the historical dynamic information of the vehicle and the interactive characteristics of the traffic participants in the driving scene, and use the time series prediction module to predict the probability distribution of the vehicle's driving trajectory; Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, generate the trajectory sequence predicted by the vehicle within the set time period in the future; Visual language model branch: extracts visual features related to text instructions from autonomous driving image data, embeds the visual features into a large language model through an adaptation network, combines them with coded prompts, outputs the environmental understanding of the driving scene in the form of structured text, and predicts several different trajectory sequences within a set period of time in the future; Fusion of the end-to-end branch and the visual language model branch: Calculate the loss value of the trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. According to the size of the loss value and the dangerous situation of the current driving scene, the trajectory sequences output by the two branches are fused to obtain the final planning trajectory.

2. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: The feature extraction of autonomous driving multimodal data to obtain a comprehensive spatiotemporal information representation of the driving scene specifically includes: encoding the input continuous multi-frame point cloud data and image data respectively, concatenating the encoded point cloud feature L and image feature I into a sequence and a multimodal feature F of the image driving scene, inputting the multimodal feature F into a spatiotemporal Transformer network, and obtaining a bird's-eye view feature B of the driving scene through multimodal fusion, which is a comprehensive spatiotemporal information representation of the driving scene.

3. The method for trajectory planning of an autonomous driving vehicle according to claim 2, characterized in that: The multi-scale feature learning of the comprehensive spatiotemporal information representation of the driving scene captures the characteristics of each traffic participant and the basic characteristics of the road, and extracts the interactive characteristics of the traffic participants in the driving scene through the interactive network, specifically including: The query of each agent in the driving scene and the bird's-eye view feature B after spatiotemporal network encoding are input into the Transformer-based variable attention network to obtain the feature A of each agent in the driving scene; Query the road facilities in the traffic scene M query The bird’s-eye view feature B after spatiotemporal network encoding is input into the Transformer-based variable attention network to obtain the road facility feature M in the traffic scene; Query Ego query The characteristics A of each intelligent agent in the driving scenario and the characteristics M of the road facilities in the traffic scenario are input into the interactive network based on the graph neural network, and cross-attention calculation is performed to capture the feature interaction R of the multi-order neighborhood centered on the vehicle.

4. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: The time series prediction module is a time series prediction module based on a U-Net network.

5. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: The method of generating a trajectory sequence predicted by the vehicle within a set period of time in the future based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results specifically includes: The comprehensive spatiotemporal information representation of the driving scene and the preliminary probability distribution prediction results are combined through the occupancy grid network to generate road scene constraints; According to the road scene constraints and the preliminary probability distribution prediction results, the short-term trajectory prediction results are obtained through the trajectory planning network; A Transformer-based regression trajectory generation network is used to convert the short-term trajectory prediction results into long-term trajectory planning results, which is the trajectory sequence predicted by the vehicle within a set period in the future.

6. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: In the visual language model branch, the training process of the large language model includes: Expanding datasets to train large language models using counterfactual-based visual instruction generation strategies; and / or Using a multi-task encoder integrated with multiple visual encoders to extract latent visual information and using a knowledge-enhanced network to extract prior knowledge from visual features; and / or Auxiliary losses are used to supervise the training process, and a reinforcement learning strategy with human feedback is combined to update the pre-trained large language model.

7. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: The trajectory sequences output by the two branches are fused according to the loss value and the dangerous situation of the current driving scene. The fusion process includes the following situations: 1) When the visual language model branch does not generate an alarm and the loss value is less than the set threshold, only short-term collision detection is performed, and correction and optimization are performed based on the planned trajectory to obtain the final trajectory prediction result; 2) When the visual language model branch does not generate an alarm warning but the loss value exceeds the set threshold, a relatively conservative driving strategy is selected between the two branches; 3) When the visual language model branch generates an alarm but the loss value is less than the set threshold, trajectory verification is performed and an alternative safe trajectory is planned; 4) When the visual language model branch generates an alarm but the loss value exceeds the set threshold, a new safest trajectory is re-judged or planned.

8. The method for trajectory planning of an autonomous driving vehicle according to claim 7, characterized in that: The dangerous situation of the driving scene is judged according to the visual language model, 0 indicates that there is no danger in the current scene, and 1 indicates that the current scene is a dangerous scene.

9. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: After the end-to-end branch is integrated with the visual language model branch to obtain the final planning trajectory, the final planning trajectory is also subject to temporal logic security verification, specifically: The signal timing logic STL is used to characterize traffic rules and safety criteria. The situation is judged by combining scene extraction with traffic rules to generate the first STL specification φ1 that characterizes traffic rules. The safety criteria and driving scene information are integrated into the second STL specification φ2. The safety quantification values ​​of the first and second specifications φ1 and φ2 are obtained using the STL robust function. If the safety quantization values ​​corresponding to the current planned trajectory meet the requirements, the vehicle controller will output the reference trajectory after safety verification, otherwise the backup trajectory planning algorithm will be used to re-plan the trajectory.

10. The method for trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: Before the end-to-end branch is merged with the visual language model branch, the method further includes: converting the trajectory sequence output by the visual language model branch from a two-dimensional point coordinate system to a three-dimensional world coordinate system that is the same as the end-to-end branch output through a depth estimation algorithm to achieve coordinate alignment.

Images