An automatic driving vehicle trajectory planning method

Abstract

The application relates to an automatic driving vehicle trajectory planning method, which comprises the following steps: adopting an end-to-end branch to extract comprehensive space-time information representation of a driving scene from automatic driving multi-modal data, and predicting a predicted trajectory sequence of a self vehicle in a future setting period; adopting a visual language model branch to embed visual features related to a text instruction into a large language model, and predicting a plurality of different trajectory sequences in the future setting period; and fusing the trajectory sequences output by the two branches according to the loss value and the dangerous situation of the current driving scene, so as to obtain a final planning trajectory. Compared with the prior art, the application has the advantages of high safety and reliability.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for trajectory planning of an autonomous driving vehicle. Background Art

[0002] Foundation Intelligence will play a crucial role in promoting the autonomous driving industry and the development of related supporting infrastructure. Foundation models, as large-scale machine learning models, are pre-trained on large amounts of data and fine-tuned for specific tasks or domains, adapting to a wide range of downstream tasks. These models are a crucial component of foundation intelligence. In the field of natural language processing (NLP), large foundation models primarily include large language models (LLMs) such as BERT and the GPT series. Their application in autonomous driving is an inevitable development trend in artificial intelligence technology and has become a golden opportunity for major manufacturers and teams to continuously compete.

[0003] In terms of explainability, academic and industrial research in the autonomous driving field is currently exploring how to embed language knowledge into decision-making. Pre-trained with rich textual instruction data, large language models can extract and interpret various elements in open driving scenarios, enabling open-scene understanding and reasoning in autonomous driving, and inferring natural language instructions for driving decision-making and planning. Some research is building autonomous driving models based on approaches similar to GPT, directly outputting planned trajectories or control commands. Others have demonstrated promising results by using Transformers to build end-to-end autonomous driving systems. While completing driving control tasks, these systems also incorporate natural language prompts into the decision-making process of autonomous driving control modules, improving the explainability of the results. Furthermore, autonomous driving can also utilize models of the real physical world. By observing a small number of events, both relevant and irrelevant to the task, it can learn background knowledge about how the world works, capture underlying laws, and predict evolutionary processes. Using deep generative models as world models allows for learning from large-scale image data, fully unleashing the model's reasoning capabilities and building a universal foundational model for autonomous driving vision.

[0004] In terms of social interaction, autonomous vehicles face significant challenges in predicting the intentions of surrounding traffic participants and implementing socially compatible game interactions and behavioral planning. Current autonomous vehicles are deficient in these areas, sometimes acting overly conservatively and waiting for extended periods to maintain safety, and sometimes acting too aggressively and cutting in front of other traffic participants. A survey report indicates a strong public distrust of the behavior of autonomous vehicles, with 60% of respondents expressing distrust of their safety and reliability, particularly in pedestrian crossing scenarios. They are particularly concerned that autonomous vehicles will collide with pedestrians and cause accidents. Summary of the Invention

[0005] The purpose of the present invention is to provide a trajectory planning method for an autonomous driving vehicle in order to overcome the defects of the above-mentioned prior art and to overcome the safety hazard problem caused by the long-tail distribution of edge scenes faced by autonomous driving vehicles.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for trajectory planning of an autonomous driving vehicle, the method comprising:

[0008] End-to-end branch: Feature extraction is performed on multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene. Multi-scale feature learning is performed on this comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic road characteristics. The interactive characteristics of traffic participants in the driving scene are extracted through an interactive network. The time series prediction module is used to perform probability distribution prediction of the ego vehicle's driving trajectory, combining the historical dynamic information of the ego vehicle and the interactive characteristics of traffic participants in the driving scene. Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, a trajectory sequence predicted for the ego vehicle within a set future time period is generated.

[0009] The visual language model branch extracts visual features related to text instructions from autonomous driving image data, embeds these visual features into a large language model through an adaptation network, and combines them with coded prompts to output an understanding of the driving scene in the form of structured text, predicting several different trajectory sequences within a set time period in the future.

[0010] Fusion of the end-to-end branch and the visual language model branch: The loss value is calculated for each trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. The trajectory sequences output by the two branches are fused based on the size of the loss value and the dangerous situation of the current driving scene to obtain the final planned trajectory.

[0011] Preferably, the feature extraction of the autonomous driving multimodal data to obtain a comprehensive spatiotemporal information representation of the driving scene specifically includes: encoding the input continuous multi-frame point cloud data and image data respectively, concatenating the encoded point cloud features L and image features I into a sequence and a multimodal feature F of the image driving scene, inputting the multimodal feature F into a spatiotemporal Transformer network, and obtaining a bird's-eye view feature B of the driving scene through multimodal fusion, which is a comprehensive spatiotemporal information representation of the driving scene.

[0012] Preferably, the multi-scale feature learning of the comprehensive spatiotemporal information representation of the driving scene is performed to capture the characteristics of each traffic participant and the basic road characteristics, and the interactive characteristics of the traffic participants in the driving scene are extracted through the interactive network, specifically including:

[0013] The query of each agent in the driving scene and the bird's-eye view feature B after spatiotemporal network encoding are input into the Transformer-based variable attention network to obtain the feature A of each agent in the driving scene;

[0014] Query the road facilities in the traffic scene M query The bird's-eye view feature B after spatiotemporal network encoding is input into the Transformer-based variable attention network to obtain the road facility feature M in the traffic scene;

[0015] Query Ego by car query The features A of each intelligent agent in the driving scenario and the features M of the road facilities in the traffic scenario are input into the interactive network based on the graph neural network, and cross-attention calculation is performed to capture the feature interaction R of the multi-order neighborhood centered on the vehicle.

[0016] Preferably, the time series prediction module is a time series prediction module based on a U-Net network.

[0017] Preferably, generating a trajectory sequence predicted for the vehicle within a set future time period based on the comprehensive spatiotemporal information representation of the driving scenario and the probability distribution prediction results specifically includes:

[0018] The comprehensive spatiotemporal information representation of the driving scene and the preliminary probability distribution prediction results are combined through the occupancy grid network to generate road scene constraints;

[0019] According to the road scene constraints and the preliminary probability distribution prediction results, the short-term trajectory prediction results are obtained through the trajectory planning network;

[0020] A Transformer-based regression trajectory generation network is used to convert the short-term trajectory prediction results into long-term trajectory planning results, which is the trajectory sequence predicted for the vehicle within a set period in the future.

[0021] Preferably, in the visual language model branch, the training process of the large language model includes:

[0022] Expanding datasets to train large language models using counterfactual-based visual instruction generation strategies; and / or

[0023] Adopting a multi-task encoder integrated with multiple visual encoders to extract latent visual information and utilizing a knowledge-enhanced network to extract prior knowledge from visual features; and / or

[0024] Auxiliary losses are used to supervise the training process, and a reinforcement learning strategy with human feedback is combined to update the pre-trained large language model.

[0025] Preferably, the trajectory sequences output by the two branches are fused according to the loss value and the dangerous situation of the current driving scene. The fusion process includes the following:

[0026] 1) When the visual language model branch does not generate an alarm and the loss value is less than the set threshold, only short-term collision detection is performed, and correction and optimization are performed based on the planned trajectory to obtain the final trajectory prediction result;

[0027] 2) When the visual language model branch does not generate an alarm warning but the loss value exceeds the set threshold, the more conservative driving strategy of the two branches is selected;

[0028] 3) When the visual language model branch generates an alarm warning but the loss value is less than the set threshold, trajectory verification is performed and an alternative safe trajectory is planned;

[0029] 4) When the visual language model branch generates an alarm warning but the loss value exceeds the set threshold, a new safest trajectory is re-judged or planned.

[0030] Preferably, the dangerous situation of the driving scene is judged based on a visual language model, where 0 indicates that the current scene does not have any danger, and 1 indicates that the current scene is a dangerous scene.

[0031] Preferably, after the end-to-end branch is integrated with the visual language model branch to obtain the final planning trajectory, the final planning trajectory is further subjected to temporal logic security verification, specifically:

[0032] Traffic rules and safety criteria are represented using signal temporal logic (STL). Situational judgment is performed by combining scene extraction with traffic rules to generate the first STL specification φ1 that represents traffic rules. Safety criteria are then integrated with driving scenario information to form the second STL specification φ2. The safety quantification values ​​of the first and second specifications φ1 and φ2 are obtained using STL robust functions.

[0033] If the safety quantization values ​​corresponding to the current planned trajectory meet the requirements, the vehicle controller will output the safety-verified reference trajectory. Otherwise, the backup trajectory planning algorithm will be used to re-plan the trajectory.

[0034] Preferably, before the end-to-end branch is fused with the visual language model branch, the process further includes: converting the trajectory sequence output by the visual language model branch from a two-dimensional point coordinate system to the same three-dimensional world coordinate system as the end-to-end branch output through a depth estimation algorithm to achieve coordinate alignment.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) The present invention fuses the trajectory sequence output by the end-to-end branch with the trajectory sequence output by the visual language model branch. The fusion process takes into account the trajectory error value, collision risk, and dangerous situations in the driving scene. When the end-to-end branch cannot accurately perceive surrounding strange objects in extreme, escape, or dangerous scenarios, a drivable path can still be planned based on the visual language model branch. This effectively solves the safety hazard problem caused by the long-tail distribution of edge scenarios faced by autonomous vehicles, and has higher safety and reliability.

[0037] 2) The present invention integrates the visual language model branch driven by the large language model with the end-to-end branch, realizing interactive decision-making participation of intelligent driving AI, which can effectively reduce the probability of manual takeover of the vehicle and improve the platform operation efficiency.

[0038] 3) The present invention adopts a counterfactual-based visual instruction generation strategy to expand the data set to train a large language model, adopts a multi-task encoder integrated with multiple visual encoders to extract potential visual information, and uses a knowledge enhancement network to extract prior knowledge from visual features to enhance the visual perception ability of the large language model, or adopts an auxiliary loss to supervise the training process, and combines a reinforcement learning strategy with human feedback to update the pre-trained large language model, which solves the problem that the generated text content of the large language model is often inconsistent with the corresponding visual content.

[0039] 4) Through the safety verification of temporal logic planning trajectories, the risk of traffic accidents is effectively reduced, the passenger travel experience is improved, and it serves as a good demonstration for the operation of intelligent connected vehicles across the country. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the method architecture of the present invention;

[0041] Figure 2 It is an end-to-end branch architecture;

[0042] Figure 3 Branching architecture for visual language models. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0044] Example

[0045] This embodiment integrates a secure and explainable end-to-end AI driving model with an end-to-end branch and a visual language model branch to construct a trajectory planning method for an autonomous driving vehicle. The method includes:

[0046] End-to-end branch: Feature extraction is performed on multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene. Multi-scale feature learning is performed on this comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic road characteristics. The interactive characteristics of traffic participants in the driving scene are extracted through an interactive network. The time series prediction module is used to perform probability distribution prediction of the ego vehicle's driving trajectory, combining the historical dynamic information of the ego vehicle and the interactive characteristics of traffic participants in the driving scene. Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, a trajectory sequence predicted for the ego vehicle within a set future time period is generated.

[0047] The visual language model branch extracts visual features related to text instructions from autonomous driving image data, embeds these visual features into a large language model through an adaptation network, and combines them with coded prompts to output an understanding of the driving scene in the form of structured text, predicting several different trajectory sequences within a set time period in the future.

[0048] Fusion of the end-to-end branch and the visual language model branch: The loss value is calculated for each trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. The trajectory sequences output by the two branches are fused based on the size of the loss value and the dangerous situation of the current driving scene to obtain the final planned trajectory.

[0049] Next, combined with the model structure (such as Figure 1 The method of this embodiment is further described in detail.

[0050] Step 1: Prepare the autonomous driving dataset and build the automatic annotation tool.

[0051] To build a scene understanding and planning dataset for a traffic scene, data collection and integration are carried out from two aspects. On the one hand, a large number of open-source large-scale surround view datasets from home and abroad are collected, covering urban road scenes and various extreme scenes, including NuScenes (Boston, Singapore) and Waymo dataset (USA) and the like. On the other hand, image and video data of a vehicle taken over by an intelligent driver are collected to build a large surround view sequence dataset, and data labeling is carried out on the basis of the dataset. First, the long-tail objects and dangerous scene categories are pre-defined, such as strange traffic lights, animals crossing the road, children and old people on the road and the like, and an image and text-based search method is used to mine the above-mentioned scenes in the dataset to form a new dangerous scene dataset for model training. For each dangerous scene, a large language model is used to give a detailed scene description and the category, state and influence on the vehicle of each key target object. The dataset constructed can be used for subsequent training and performance verification of an end-to-end large model.

[0052] Then, a thought chain guided training method is used, and a visual language model is used to give a detailed text description of the driving scene, including weather conditions, road types, lane passable conditions and key target object coordinates and the like. At the same time, on the basis of analyzing the influence of the key target object on the vehicle driving, driving suggestions and strategies for the vehicle in the future are given, and through the analysis of a video sequence, a complete driving action sequence of the vehicle is obtained, such as acceleration, deceleration, slow straight driving, left lane change, right lane change and the like, as well as the target area to be reached by the vehicle in the 2D image coordinate system (for example, after 1 to 3 seconds).

[0053] Step 2: overall end-to-end model architecture design and training, as shown in Figure 2 In an autonomous driving system, in order to obtain a comprehensive spatio-temporal information representation of a driving scene, multi-modal data from multiple sensors need to be fused. Specifically, continuous multiple frames of point cloud data and image data can be used as input. First, the point cloud sequence and the image sequence are encoded respectively to extract their feature representations. Then, the features of the two modalities are fused, and the fused features can contain rich information from the point cloud and the image, comprehensively describing the three-dimensional spatial structure and visual details of the driving scene.

[0054] In the planning of autonomous driving systems, it is crucial to consider the interaction between various traffic elements. The system needs to analyze the dynamics of vehicles, pedestrians, bicycles and other participants on the road in real time, while also considering the influence of environmental factors such as traffic signals and road signs. Only by fully understanding and simulating these complex interactions can the autonomous driving system make safe and efficient decisions. To this end, we need to design an interaction module. From the multi-modal driving scene information, we learn the features of each traffic participant and the road base features at multiple scales, and then based on this, we construct an interaction network to analyze the interaction features of the extracted traffic participants in the scene. This comprehensive interaction analysis helps the system better adapt to complex and variable traffic environments, improving driving safety and comfort.

[0055] ① Spatio-temporal information representation of multi-modal driving scene:

[0056] In autonomous driving systems, in order to obtain a comprehensive spatio-temporal information representation of the driving scene, it is necessary to fuse multi-modal data from multiple sensors. Specifically, continuous multi-frame point cloud data and image data can be used as input. First, the point cloud sequence and image sequence are encoded respectively to extract their feature representations. Then, the features of the two modalities are fused, and the fused features can contain rich information from both point clouds and images, comprehensively describing the three-dimensional spatial structure and visual details of the driving scene.

[0057] F = C(I, L) (1)

[0058] B = TS Transformer(B query , F, F) (2)

[0059] Where I represents the feature encoded from images, L represents the feature encoded from point clouds, C represents the feature concatenation operation, F represents the output of concatenation, B query represents the initial query, TS Transformer represents the spatio-temporal Transformer network, and B represents the bird's eye view feature encoded by the spatio-temporal network. First, I and L are concatenated C to fuse the multi-modal features F of the driving scene, and then the TSTransformer network is used to transform the multi-modal fusion features F into the bird's eye view feature B of the driving scene. r

[0060] ② Multi-traffic element dynamic interaction learning

[0061] ​When designing autonomous driving systems, it's crucial to consider the interactions between various traffic elements. The system must analyze the dynamics of vehicles, pedestrians, cyclists, and other road users in real time, while also factoring in environmental factors like traffic signals and road signs. Only by fully understanding and modeling these complex interactions can autonomous driving systems make safe and efficient decisions. First, we need to capture the characteristics of each traffic participant and the underlying road characteristics through multi-scale learning from multimodal driving scene information.

[0062] A=DATransformer(A query , B, B) (3)

[0063] M=DATransformer(M query , B, B) (4)

[0064] Among them, A query represents the query of each agent in the driving scene, B represents the bird's-eye view feature after spatiotemporal network coding, M query represents the query of road facilities in the traffic scene, A represents the characteristics of each agent in the driving scene, M represents the characteristics of road facilities in the traffic scene, and DATrasformer represents the variable attention network based on Transformer.

[0065] Based on the characteristics of key traffic elements, we designed an interaction module. Using the ego vehicle as the example, we leveraged an interaction network to analyze the interaction characteristics of traffic participants extracted from the scene. This comprehensive interaction analysis helps the system better adapt to complex and changing traffic environments, improving driving safety and comfort. This interactive generation approach enables the model to capture subtle dependencies between trajectories and the scene, ensuring that the generated trajectories adhere to various constraints, such as road structure, obstacle avoidance, and compliance with traffic regulations, thereby enhancing the social interaction properties of the autonomous driving system.

[0066] R=Interaction(Ego query , A, M) (5)

[0067] Among them, Ego query Represents the query of the ego vehicle, and Interaction represents the Transformer-based interaction module, which performs cross-attention calculation to capture the feature interaction R of the multi-order neighborhood centered on the ego vehicle instance.

[0068] ③Multi-agent trajectory prediction based on integrated planning

[0069] In an autonomous driving system, it is necessary to directly predict the future motion trajectory of each traffic participant to provide input for the final motion planning and control. The interactive features R obtained above are input into the prediction module to predict the future motion of each traffic participant. In particular, the historical dynamic information Ego of the ego vehicle is incorporated here. state Thus, the ego vehicle planning is integrated into the prediction module TUNet, and a preliminary probability distribution prediction P of the ego vehicle's future driving trajectory is made.

[0070] P=TUNet(MLP(Egos tate ), R) (6)

[0071] TUNet represents a time series prediction module based on the U-Net network. It combines the bird's-eye view features B of the driving scene with the prediction results P through an occupancy grid network to generate the road scene constraints O, and then safely and effectively generates the future driving trajectory u of the vehicle:

[0072] O=OccHead(P,B,B) (7)

[0073] u=PlanHead(P,O) (8)

[0074] Among them, OccHead represents the occupancy grid network, and PlanHead is the short-term trajectory planning network based on Transformer.

[0075] Specifically, based on the collision constraints of the road scene and the motion prediction results obtained from the preliminary probability distribution, a short-term trajectory sequence is generated through the Transformer-based short-term trajectory planning network: a probability field is used to continuously map from the action space to the probability distribution, and each trajectory action is encoded into a high-dimensional input, embedded in the Transformer decoder network, and combined with the current driving scene information to output the planning result.

[0076] In particular, in order to achieve long-term planning, a regressive future driving trajectory planning is designed. After obtaining the trajectory planning results in the short-term domain, the results are encoded into an input sequence. The Transformer network based on the attention mechanism is used to obtain the trajectory points for a period of time in the future through decoding prediction. The long-term planning result u * The expression is:

[0077] u * =AgGen(u t ,u t+1 ,…,u t+T ) (9)

[0078] Among them, AgGen represents the regression trajectory generation network based on Transformer, ut , u t+1 ,..., u t+T represents the short-term result planned by the trajectory planning network PlanHead.

[0079] Step 3: Visual language model architecture design and training, as shown in Figure 3 .

[0080] The Transformer-based query network is used to extract the most relevant visual input to the text instruction, and then the visual features are embedded into the large language model through the adaptation network. Combined with the encoded prompt, the model can output the environmental understanding of the traffic scene in a structured text form, and provide reasonable driving semantic instructions for the planning task, improving the generalization ability to unknown scenes. The visual language models used include QwenVL, InternVL, mPLUG-Owl, etc. During network training, the pre-trained model parameters are frozen, and a trainable adaptation network is added to each layer in the Transformer architecture, which improves the understanding of the scene without increasing the training cost, and greatly reduces the number of trainable parameters in the downstream task.

[0081] In addition to semantic driving instructions, the visual language model can also infer the target area that the vehicle is expected to reach (such as 1-3 seconds later) in the 2D image coordinate system. The vehicle future trajectory obtained by the end-to-end branch is in the world coordinate system. In order to align the outputs of the two branches, a depth estimation method is used to project the 2D point coordinates to obtain the coordinate values in the 3D world coordinate system. The advantage of this architecture is that in extreme, escape or dangerous scenarios, when the end-to-end branch cannot accurately perceive the surrounding singular objects (such as potholes, ice and snow, etc.), it can still plan a drivable path based on the output of the visual language model. The specific projection conversion method from image coordinate system to world coordinate system is as follows: First, the image coordinates are back-projected to the camera coordinate system, which requires depth information estimation. Convolutional neural network-based algorithms can be used to obtain the distance from the pixel to the camera.

[0082]

[0083] where (x, y) represents the trajectory point coordinates in the image coordinate system; f represents the camera focal length value, represents the perspective projection matrix, with a dimension of 3x4; Z c represents the depth value to be estimated; (X c , Y c , Z c ) represents the mapped trajectory point coordinates in the camera coordinate system.

[0084] Then, the camera coordinates are converted to world coordinates:

[0085]

[0086] in, Represents the camera external parameter matrix, R is the rotation matrix with a dimension of 3×3, and T is the translation matrix with a dimension of 3×1; (X w , Y w , Z w ) represents the coordinates of the trajectory point in the mapped world coordinate system.

[0087] Furthermore, due to insufficient training data, noise in the training data, and a lack of diversity, large multimodal language models often generate text that is inconsistent with the corresponding visual content, a phenomenon known as the hallucination problem of large language models. To address this hallucination problem, improvements can be made in three areas: data, model, and training. First, by expanding the dataset using a counterfactual-based visual instruction generation strategy, large multimodal language models trained on a calibration dataset have been shown to be less susceptible to hallucinations. Second, in terms of the model, a multi-task encoder can be used to integrate the latent visual information extracted by multiple visual encoders, and a knowledge augmentation module can be used to extract prior knowledge from the visual input, thereby enhancing the visual perception capabilities of the large language model. Finally, during model training, an auxiliary loss can be used to supervise the training process, and a reinforcement learning strategy with human feedback can be combined to update the pre-trained large language model, improving visual features and reducing the occurrence of hallucinations.

[0088] Step 4: Fusion of visual language model and end-to-end model.

[0089] The VLM branch prediction results in several different trajectory sequences within the next 3 seconds. n represents the number of predicted trajectory sequences, corresponding to aggressive and conservative driving styles in different lanes. Specifically, different driving styles can be judged by the velocity change rate, acceleration change rate, and whether sudden acceleration and deceleration occur. The end-to-end branch outputs a trajectory sequence with the highest confidence in the next 3 seconds [(x1, y1), (x2, y2), (x3, y3)]. The average L2 displacement loss is calculated between the trajectory sequence output from the end-to-end and each trajectory output by the VLM, and the trajectories output by the two branches are fused according to the size of the loss value and the dangerous situation of the current scene. Among them, the dangerous situation of the scene is judged by the VLM, 0 means that there is no danger in the current scene, and 1 means that the current scene is a dangerous scene. The specific fusion process includes the following 4 cases:

[0090] 1) When the VLM branch does not generate an alarm and the calculated L2 loss value is less than the set threshold ∈, it indicates that the trajectories obtained by the VLM and the end-to-end branch are basically consistent, and the driving scene at this time is relatively safe. Therefore, when the trajectory safety verification module is handed over to the subsequent trajectory safety verification module, only short-term collision detection is performed to avoid degrading the performance of the AI ​​system. Correction and optimization can be performed based on the existing planned trajectory to obtain the final trajectory prediction result.

[0091] 2) If the VLM branch does not generate an alarm but the calculated L2 loss exceeds the set threshold ∈, this indicates that the trajectories obtained by the VLM and the end-to-end branch are inconsistent. Although the VLM did not generate an alarm, it may be a missed alarm. To ensure driving safety and minimize the possibility of false negatives (FN), the subsequent safety verification module can select the more conservative driving strategy of the two branches.

[0092] 3) If the VLM branch generates an alarm but the calculated L2 loss is less than the set threshold ∈, this indicates a potential driving risk in the current scenario. The VLM and end-to-end branch plan the same trajectory. This could be because both models plan correctly, but the VLM model generates a false alarm, or because both branches plan incorrectly. Therefore, the safety verification module must perform rigorous trajectory verification and plan a safe backup trajectory.

[0093] 4) When the VLM branch generates an alarm but the calculated L2 loss value exceeds the set threshold ∈, it indicates that the vehicle is in a dangerous driving scenario and the trajectories planned by the two branches are inconsistent. This is the most serious situation and requires the back-end safety verification module to re-evaluate or plan the safest trajectory.

[0094] Step 5: Safety verification of the planned trajectory based on temporal logic method.

[0095] Developing safety quantification and verification technology involves describing security requirements through high-level specifications of the target system, generating corresponding safety quantifiers, and analyzing and verifying whether the planned operational trajectory of the AI ​​model meets safety requirements. Safety quantification technology primarily requires finding an appropriate formal logic language to accurately describe the safety criteria of the target system.

[0096] To address the formal safety verification problem of autonomous driving, we first propose to use Signal Temporal Logic (STL) to represent traffic rules and safety criteria. By extracting scenarios and combining them with traffic rules for situational assessment, we generate an STL specification φ1 representing the traffic rules. Similarly, we fuse safety criteria with driving scenario information to form an STL specification φ2. We then utilize STL robust semantics to construct a safety quantification framework for the legality and rationality of specifications φ1 and φ2, respectively. Trajectory planning is performed using a data-driven AI model, leveraging real-time observations of factors such as road models, traffic signal models, and traffic participants. The reference trajectory is input into a legality and rationality safety quantifier, and the STL robust function is used to obtain the safety quantification values ​​of the STL specifications φ1 and φ2, thereby quantifying the scenario risk safety. When the STL specifications φ1 and φ2 meet the requirements, a correct controller can be designed to track the reference trajectory. When the robustness of the STL specifications φ1 or φ2 is negative, an alternative trajectory planning algorithm is employed to re-plan the trajectory.

[0097] Step 6: Model deployment and real vehicle testing.

[0098] Efficient deployment of end-to-end driving models requires comprehensive optimization and resolution, taking into account latency and wait time, existing and memory usage, scalability and high throughput, hardware resource utilization, and model size and performance assessment. We explore model compression and pruning solutions through an optimized combination approach. We introduce a new pruning metric to focus on identifying interdependent structures within the model. Based on importance assessment results, we refine the granularity by incorporating the second-order importance of individual parameters and the group importance of coupled structures. This approach maximizes model compression efficiency while maintaining performance. Furthermore, we employ a low-precision inference algorithm to control the number of model parameters and memory usage. We perform speculative sampling and low-bit quantization for activation outliers, and explore the use of FP8 for model training. This significantly improves inference speed while maintaining manageable accuracy loss. Leveraging the intelligent vehicle's computing unit configuration, we ensure that the computing platform is integrated with the vehicle's other electronic control units (ECUs) and sensors to ensure real-time data transmission and processing. Furthermore, we develop interfaces with sensors and the vehicle control system to ensure real-time transmission of sensor data to the model and transmission of model outputs to the vehicle control system. Standardized communication protocols are used to ensure stable communication between components. Specifically, the ROS method is used to parse large model algorithms into CAN signals, and the CAN communication protocol is used between the industrial computer and the controller.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for trajectory planning of an autonomous driving vehicle, characterized in that: The method includes: End-to-end branch: Feature extraction is performed on multimodal data of autonomous driving to obtain a comprehensive spatiotemporal information representation of the driving scene. Multi-scale feature learning is performed on this comprehensive spatiotemporal information representation of the driving scene to capture the characteristics of each traffic participant and the basic road characteristics. The interactive characteristics of traffic participants in the driving scene are extracted through an interactive network. The time series prediction module is used to perform probability distribution prediction of the ego vehicle's driving trajectory, combining the historical dynamic information of the ego vehicle and the interactive characteristics of traffic participants in the driving scene. Based on the comprehensive spatiotemporal information representation of the driving scene and the probability distribution prediction results, a trajectory sequence predicted for the ego vehicle within a set future time period is generated. The visual language model branch extracts visual features related to text instructions from autonomous driving image data, embeds these visual features into a large language model through an adaptation network, and combines them with coded prompts to output an understanding of the driving scene in the form of structured text, predicting several different trajectory sequences within a set time period in the future. Fusion of the end-to-end branch and the visual language model branch: The loss value is calculated for each trajectory sequence output by the end-to-end branch and each trajectory sequence output by the visual language model branch. The trajectory sequences output by the two branches are fused based on the size of the loss value and the dangerous situation of the current driving scene to obtain the final planned trajectory.

2. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: The feature extraction of the autonomous driving multimodal data to obtain a comprehensive spatiotemporal information representation of the driving scene specifically includes: encoding the input continuous multi-frame point cloud data and image data respectively, and encoding the encoded point cloud features. and image features Splicing and fusion to obtain multimodal features of driving scenes , the multimodal features Input into the spatiotemporal Transformer network, and obtain the bird's-eye view features of the driving scene through multimodal fusion , which is a comprehensive spatiotemporal information representation of the driving scene.

3. The method for trajectory planning of an autonomous driving vehicle according to claim 2, wherein: The multi-scale feature learning of the comprehensive spatiotemporal information representation of the driving scene captures the characteristics of each traffic participant and the basic road characteristics, and extracts the interactive features of the traffic participants in the driving scene through the interactive network, specifically including: Query each agent in the driving scene And the bird's-eye view features after spatiotemporal network coding Input into the Transformer-based variable attention network to obtain the characteristics of each agent in the driving scene ; Query road facilities in traffic scenarios And the bird's-eye view features after spatiotemporal network coding Input into the Transformer-based variable attention network to obtain road facility features in traffic scenes ; Query your car , Characteristics of each intelligent agent in the driving scenario and road facility characteristics in traffic scenarios Input into the interactive network based on the graph neural network, and perform cross-attention calculation to capture the feature interactions of the multi-order neighborhood centered on the vehicle .

4. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: The time series prediction module is a time series prediction module based on the U-Net network.

5. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: The method generates a trajectory sequence predicted for the vehicle within a set future time period based on the comprehensive spatiotemporal information representation of the driving scenario and the probability distribution prediction results, specifically including: The comprehensive spatiotemporal information representation of the driving scene and the preliminary probability distribution prediction results are fused through the occupancy grid network to generate road scene constraints; According to the road scene constraints and the preliminary probability distribution prediction results, the short-term trajectory prediction results are obtained through the trajectory planning network; A Transformer-based regression trajectory generation network is used to convert the short-term trajectory prediction results into long-term trajectory planning results, which is the trajectory sequence predicted for the vehicle within a set period in the future.

6. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: In the visual language model branch, the training process of the large language model includes: Expanding datasets to train large language models using counterfactual-based visual instruction generation strategies; and / or Adopting a multi-task encoder integrated with multiple visual encoders to extract latent visual information and utilizing a knowledge-enhanced network to extract prior knowledge from visual features; and / or Auxiliary losses are used to supervise the training process, and a reinforcement learning strategy with human feedback is combined to update the pre-trained large language model.

7. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: The trajectory sequences output by the two branches are fused according to the loss value and the dangerous situation of the current driving scene. The fusion process includes the following situations: 1) When the visual language model branch does not generate an alarm and the loss value is less than the set threshold, only short-term collision detection is performed, and correction and optimization are performed based on the planned trajectory to obtain the final trajectory prediction result; 2) When the visual language model branch does not generate an alarm but the loss value exceeds the set threshold, the more conservative driving strategy of the two branches is selected; 3) When the visual language model branch generates an alarm but the loss value is less than the set threshold, trajectory verification is performed and an alternative safe trajectory is planned; 4) When the visual language model branch generates an alarm warning but the loss value exceeds the set threshold, the safest trajectory is re-judged or planned.

8. The method for trajectory planning of an autonomous driving vehicle according to claim 7, wherein: The dangerous situation of the driving scene is judged based on a visual language model, where 0 indicates that the current scene is not dangerous and 1 indicates that the current scene is a dangerous scene.

9. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: After the end-to-end branch is integrated with the visual language model branch to obtain the final planning trajectory, the final planning trajectory is also subject to temporal logic security verification, specifically: Traffic rules and safety guidelines are characterized by using signal timing logic (STL). Situation judgment is performed by combining scene extraction with traffic rules to generate the first STL protocol to characterize traffic rules. , integrating safety criteria with driving scenario information into the second STL specification , using STL robust functions to obtain the first STL specification and the second STL specification Safety quantitative value; If the safety quantization values ​​corresponding to the current planned trajectory meet the requirements, the vehicle controller will output the safety-verified reference trajectory. Otherwise, the backup trajectory planning algorithm will be used to re-plan the trajectory.

10. The method for trajectory planning of an autonomous driving vehicle according to claim 1, wherein: Before fusing the end-to-end branch with the visual language model branch, the process also includes: using a depth estimation algorithm to convert the trajectory sequence output by the visual language model branch from a two-dimensional point coordinate system to the same three-dimensional world coordinate system as the end-to-end branch output to achieve coordinate alignment.

Images