Double-agent driving decision planning method and system based on multi-modal large model

Abstract

The invention relates to a dual-agent driving decision planning method and system based on a multi-modal large model, and the method comprises the steps: Q1, scene construction: converting the information of a map, a vehicle and other traffic participants into a Frenet coordinate system in a unified manner, integrating the information into a multi-layer comprehensive aerial view, and generating corresponding text prompt information; q2, a trajectory generation intelligent agent: performing comprehensive processing on the input corresponding text prompt information based on a multi-modal large language model, identifying a key obstacle of a vehicle through scene analysis, performing behavior decision, and generating a driving trajectory; and Q3, judging the generated driving track by using a track judgment agent, analyzing problems existing in the track by using an anti-thinking chain, optimizing the generated driving track, and updating the track. According to the method, a closed-loop mechanism of generation-judgment-optimization is formed, uncertainty of single agent decision is effectively avoided, and it is guaranteed that the finally output path is safe and reliable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a multi-modal large model-based dual-agent driving decision planning method and system. BACKGROUND

[0002] In the field of autonomous driving, path planning is one of the core tasks for autonomous decision-making and action of vehicles. The goal is to design a safe, comfortable and optimal path from the current state to the target state based on surrounding environment information, road conditions and driving tasks. With the development of autonomous driving technology, path planning methods have evolved, but there are still many challenges.

[0003] Traditional rule-based path planning methods rely on pre-defined rule sets and logical conditions to ensure controllability and consistency of system behavior through explicit specifications. However, the application effect of this method is restricted by the completeness of the rule set and the complexity of the environment, and the manually formulated rules are often difficult to cover all possible traffic scenarios. Learning-based path planning methods can achieve more flexible decision-making and planning in complex traffic environments through analysis and learning of large amounts of traffic data. However, this type of method is often considered a "black box" problem, and the decision-making process lacks transparency and interpretability, and performs poorly when faced with rare or unseen scenarios. In recent years, the emergence of large models has brought profound changes to multiple industries. With its development, large models have gradually acquired the ability to process multi-modal inputs such as text, images and videos, and their generalization has been fully verified. The application of multi-modal large models in autonomous driving decision planning has attracted attention.

[0004] In the prior art, a Chinese patent (application number: 202411912670.0, publication number: CN 119756400A) discloses an automatic driving vehicle path planning method, device, equipment and medium based on a large language model. The method comprises: acquiring traffic scene RGB images, traffic scene depth images, laser radar point cloud data and inertial measurement unit data of an automatic driving vehicle; performing feature extraction on the traffic scene RGB images, traffic scene depth images, laser radar point cloud data and inertial measurement unit data respectively to construct multi-modal traffic scene information; calling a preset large language model to determine the motion state of the automatic driving vehicle and the natural language description of the traffic target around the automatic driving vehicle according to the multi-modal traffic scene information; using the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the automatic driving vehicle and the target position. This scheme fails to effectively learn the relationship between the ego vehicle and the road topology, especially in complex scenes such as intersections or curves, the generated planning trajectory is often easy to rush out of the road boundary, bringing significant safety hazards. The planning module of the automatic driving system has multiple and complex input information, including map details, ego vehicle state, and various attributes of other traffic participants. However, many important scene information cannot be fully described and effectively input into the large language model, resulting in low information expression efficiency, which further affects the accuracy and rationality of the planning result. The path planning result generated directly by the large language model has uncertainty, and the trajectory may not conform to human driving habits or may have collision risk with other traffic participants, thereby failing to ensure the rationality and safety of the automatic driving behavior.

[0005] In the prior art, a Chinese patent (application number: 202510710028.2, publication number: CN 120396986A) discloses an automatic driving vehicle decision planning method based on deep reinforcement learning. The method comprises the steps of: during the driving of the ego vehicle on a structured road, collecting the position information of the environment vehicle, introducing the environment vehicle historical position data into the long short-term memory network to obtain the surrounding vehicle driving intention hidden state, and generating a planning trajectory based on a polynomial curve through the SAC algorithm according to the obtained hidden state and the ego vehicle motion information fusion. In this scheme, the whole model is still a "black box" model, and the whole decision planning process lacks transparency and explainability, it is difficult to locate the problem, and it is also not conducive to gaining the trust of automatic driving users. In addition, this method performs objectively on the trained data, but in rare or even unseen scenarios, its generalization ability is insufficient, and in some complex practical application scenarios, due to the influence of environmental uncertainty, noise and other factors, its performance may decline, and additional technical means are needed to improve and optimize. SUMMARY

[0006] In view of the shortcomings of the prior art, the present invention provides a multimodal large-scale dual-agent driving decision planning method and system, which not only forms a closed-loop mechanism of "generation-evaluation-optimization" to effectively avoid the uncertainty of single agent decision-making, but also ensures that the final output path is safe and reliable.

[0007] To achieve the above and other related objectives, the present invention provides the following technical solution: A driving decision-making and planning method based on a multimodal large model dual-agent system, the method comprising: Q1. Scene Construction: Convert the information of the map, vehicles and other traffic participants to the Frenet coordinate system, then integrate them into a multi-layer comprehensive bird's-eye view, and generate corresponding text prompts; Q2. Trajectory Generation Agent: Based on a multimodal large language model, the agent comprehensively processes the corresponding text prompts input, identifies key obstacles of the vehicle through scene analysis, makes behavioral decisions, and generates a driving trajectory. Q3. Trajectory Evaluation Agent: Evaluates the generated driving trajectory, uses a reflective thinking chain to analyze problems in the trajectory, optimizes the generated driving trajectory, and updates the trajectory.

[0008] Furthermore, in step Q1, the process of uniformly converting the information from the map, vehicles, and other traffic participants to the Frenet coordinate system, then integrating it into a multi-layered comprehensive bird's-eye view, and generating corresponding text prompts includes: Q11. Select a guide line based on navigation routing information and decompose the vehicle's motion in Cartesian space into two independent one-dimensional motions: longitudinal motion along the guide line and lateral motion perpendicular to the guide line. Q12. Map elements mapped to the Frenet coordinate system, including lane lines, road boundaries, and pedestrian crossings, as well as obstacles such as vehicles, motor vehicles, pedestrians, and cones, along with vehicle and historical trajectory information, are drawn in a BEV image using different colors and shapes. Q13. Based on BEV images, describe the pixel coordinates and velocity information of each object in the image and generate corresponding text prompts.

[0009] Furthermore, in step Q11, the coordinates of any point P in the Cartesian coordinate system are known to be (x, y), and the guide line is represented as L[x(s), y(s), θ(s)], where x(s), y(s), and θ(s) are the x and y coordinates of the point corresponding to the arc length s on the guide line and the tangent direction, respectively. The nearest matching point P on the guide line L is Pn. r The state is (x(s) r ),y(s r ),θ(sr )), s r For P r The arc length on the guide line can be converted to Frenet coordinates using the following formula: , Where s and l are the vertical and horizontal positions in Frenet coordinates, respectively.

[0010] Furthermore, in step Q13, describing the pixel coordinates and velocity information of each object in the image includes: Q131. Describe the pixel coordinates of dynamic obstacles in the image, their motion state and speed information, the position, speed and historical trajectory points of the vehicle, and the information of traffic lights; Q132. Identify targets that have a key impact on the vehicle's driving from the perception results, infer the vehicle's behavior strategy, and generate corresponding trajectory point data.

[0011] Furthermore, in step Q2, the process of making behavioral decisions and generating a driving trajectory includes: Q21. Using the CLIP-ViT-L-336px model as a visual encoder, perform understanding and feature extraction on BEV images. bev =CLIP(X bev ), where X bev For the input image information, F bev For extracted visual features; Q22. Text prompt information is encoded using a Tonkenizer to obtain text features: Token text =Tonkenizer(prompts); Q23. The output features of the visual encoder are processed through a projection layer to align the dimensions of image and text features, ultimately yielding multimodal fused features: Token mllm =F bev +Token text ; Q24. Analyze the current driving scenario, taking into account road conditions and interactions with other traffic participants, to identify key obstacles; Q25. Based on the understanding of the scene and the prediction of key obstacles, combined with the real-time information of the vehicle, make high-level behavioral decisions; Q26. Based on behavioral decisions, generate a time-sampled trajectory point T in the Frenet coordinate system: {(s1,l1),(s2,l2),...,(s... t ,l tAnd these trajectory points are projected back to Cartesian space in the following way, and executed in a real environment: (x i ,y i )=(x(s i )-l i *sin(θ(s i )),y(s i )+l i *cos(θ(s i ))), Where i = 1, 2, ..., t, the leader line is represented as L[x(s), y(s), θ(s)].

[0012] Furthermore, in step Q25, the high-level behavioral decision is decomposed into lateral decisions and vertical decisions. The lateral decisions include, but are not limited to, KEEP THE LANE, CHANGE TO LEFT LANE, and CHANGE TO RIGHTLANE. The vertical decisions include, but are not limited to, STOP, CONSTANT SPEED, ACCELERATION, DECELERATION, and DECELERATION TO ZERO.

[0013] Furthermore, in step Q3, optimizing the generated driving trajectory and updating the trajectory includes: Q31. The trajectory output by the trajectory generation agent is given to the trajectory evaluation agent for anomaly analysis; Q32. Collision Risk Analysis: Calculate the direction of the vehicle's trajectory points and those of other obstacles using α = arctan(Δy / Δx), and then construct the vehicle's bounding box Polgon at each moment based on their respective lengths and widths. i and obstacle enclosure Polgon i j Calculate Polgon i and Polgon i j Are there any overlaps between them? Q33. Abnormal Parking Analysis: Calculate the average speed of the vehicle's planned trajectory points based on the vehicle's coordinates. , Among them, v avg <v min v min A low speed threshold indicates that the planned trajectory expects the vehicle to remain stationary or limp at a very low speed, and at this time there are no traffic lights or the traffic lights are in a green state, and there are no other obstacles affecting the vehicle, which is an abnormal parking state. Q34. Comfort Anomaly Analysis: The generated trajectory is fitted using a cubic polynomial. , Where t is time, and a0, a1, a2, a3 and b0, b1, b2, b3 are constant parameters, the acceleration a and curvature k at any point on the trajectory can be calculated. , If a>acc max or a<-dec max , among which, acc max and -dec max If k represents the maximum acceleration and maximum deceleration respectively, it indicates that the planned trajectory has an abnormal situation of rapid acceleration or deceleration. <k max ,k max The threshold representing the maximum curvature indicates an anomaly where the planned trajectory experiences sharp turns.

[0014] Furthermore, optimizing and updating the generated driving trajectory also includes: Q35. For the different types of abnormal situations analyzed above, the following optimization strategies are given: 1) Collision risk: When a potential collision risk is detected, the longitudinal speed of the vehicle is automatically judged and adjusted according to the current scene environment, or the vehicle is avoided by lateral offset; 2) Abnormal parking: If the system detects a situation where the vehicle should not stop, the acceleration strategy is automatically triggered to restore the vehicle to normal driving speed; 3) Abnormal comfort: When the trajectory shows a sudden acceleration, sudden deceleration or sharp turn that does not meet the comfort requirements, the longitudinal or lateral speed is adjusted to ensure driving comfort and safety.

[0015] Furthermore, optimizing and updating the generated driving trajectory also includes: Q36. Based on the optimization strategy provided by anomaly analysis, adjust the original trajectory to generate a new trajectory and avoid the above-mentioned anomaly problem.

[0016] To achieve the above and other related objectives, the present invention also provides a multimodal large model dual-agent driving decision-making and planning system, including a computer device programmed or configured to execute the steps of any of the multimodal large model dual-agent driving decision-making and planning methods described above.

[0017] The present invention has the following positive effects: 1. This invention proposes a dual-agent framework for autonomous driving path planning based on a multimodal large language model. This framework, through the collaborative work of two decision-making agents, is responsible for reasoning about traffic scenarios to generate planned trajectories, and for analyzing and optimizing the generated trajectories, forming a closed-loop mechanism of "generation-evaluation-optimization." This effectively avoids the uncertainty of single-agent decision-making and ensures the safety and reliability of the final output path.

[0018] 2. This invention transforms the input of large models from traditional Cartesian space to Frenet space, effectively utilizing the structural features of roads and eliminating the interference of road shape on model learning. Simultaneously, it converts the position and speed information of the vehicle and other traffic participants into mileage along the road and lateral offset from the road, thereby more effectively representing the relative relationships between the vehicle and other traffic participants, as well as the spatial layout between different traffic participants and lanes.

[0019] 3. This invention enriches the model with scene information by combining multi-layered, comprehensive bird's-eye view images with textual descriptions of key information, achieving a refined expression of driving scenarios. It not only intuitively presents various elements and complex information within the scene but also enhances detail, making scene visualization clearer. In this way, we can effectively display rich and diverse input information while avoiding the problem of exceeding the maximum input limit due to excessively long text tokens.

[0020] 4. This invention designs a decision-making and planning chain based on human driving habits, including scenario analysis, behavioral decision-making, and trajectory generation, as well as a reflective reasoning chain for anomaly analysis, strategy optimization, and trajectory updating. This collaboratively ensures the safety of autonomous driving decision-making and planning, and avoids the illusion problem of large models. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the process for generating corresponding text prompt information according to the present invention; Figure 3 This is a framework diagram of the dual-agent autonomous driving decision-making and planning of the present invention; Figure 4 A data diagram illustrating the scenario constructed for this invention; Figure 5 This is a diagram of the dual-agent large-scale model architecture of the present invention. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] Example 1: As Figure 1 As shown, a driving decision-making and planning method based on a multimodal large model dual-agent system is presented, the method comprising: Q1. Scene Construction: Convert the information of the map, vehicles and other traffic participants to the Frenet coordinate system, then integrate them into a multi-layer comprehensive bird's-eye view, and generate corresponding text prompts; Q2. Trajectory Generation Agent: Based on a multimodal large language model, the agent comprehensively processes the corresponding text prompts input, identifies key obstacles of the vehicle through scene analysis, makes behavioral decisions, and generates a driving trajectory. Q3. Trajectory Evaluation Agent: Evaluates the generated driving trajectory, uses a reflective thinking chain to analyze problems in the trajectory, optimizes the generated driving trajectory, and updates the trajectory.

[0024] In this embodiment, as Figure 2 As shown, in step Q1, the process of uniformly converting the information of the map, vehicles, and other traffic participants to the Frenet coordinate system, then integrating it into a multi-layered comprehensive bird's-eye view, and generating corresponding text prompts includes: Q11. Select a guide line based on navigation routing information and decompose the vehicle's motion in Cartesian space into two independent one-dimensional motions: longitudinal motion along the guide line and lateral motion perpendicular to the guide line. Q12. Map elements mapped to the Frenet coordinate system, including lane lines, road boundaries, and pedestrian crossings, as well as obstacles such as vehicles, motor vehicles, pedestrians, and cones, along with vehicle and historical trajectory information, are drawn in a BEV image using different colors and shapes. Q13. Based on BEV images, describe the pixel coordinates and velocity information of each object in the image and generate corresponding text prompts.

[0025] In this embodiment, in step Q11, the coordinates of any point P in the Cartesian coordinate system are known to be (x, y), and the guide line is represented as L[x(s), y(s), θ(s)], where x(s), y(s), and θ(s) are the x and y coordinates of the point corresponding to the arc length s on the guide line and the tangent direction, respectively. The nearest matching point P on the guide line L is Pn. r The state is (x(s)r ),y(s r ),θ(s r )), s r For P r The arc length on the guide line can be converted to Frenet coordinates using the following formula: , Where s and l are the vertical and horizontal positions in Frenet coordinates, respectively.

[0026] In this embodiment, in step Q13, describing the pixel coordinates and velocity information of each object in the image includes: Q131. Describe the pixel coordinates of dynamic obstacles in the image, their motion state and speed information, the position, speed and historical trajectory points of the vehicle, and the information of traffic lights; Q132. Identify targets that have a key impact on the vehicle's driving from the perception results, infer the vehicle's behavior strategy, and generate corresponding trajectory point data.

[0027] In this embodiment, step Q2, which involves making behavioral decisions and generating a driving trajectory, includes: Q21. Using the CLIP-ViT-L-336px model as a visual encoder, perform understanding and feature extraction on BEV images. bev =CLIP(X bev ), where X bev For the input image information, F bev For extracted visual features; Q22. Text prompt information is encoded using a Tonkenizer to obtain text features: Token text =Tonkenizer(prompts); Q23. The output features of the visual encoder are processed through a projection layer to align the dimensions of image and text features, ultimately yielding multimodal fused features: Token mllm =F bev +Token text ; Q24. Analyze the current driving scenario, taking into account road conditions and interactions with other traffic participants, to identify key obstacles; Q25. Based on the understanding of the scene and the prediction of key obstacles, combined with the real-time information of the vehicle, make high-level behavioral decisions; Q26. Based on behavioral decisions, generate a time-sampled trajectory point T in the Frenet coordinate system: {(s1,l1),(s2,l2),...,(s... t ,l t And these trajectory points are projected back to Cartesian space in the following way, and executed in a real environment: (x i ,y i )=(x(s i )-l i *sin(θ(s i )),y(s i )+l i *cos(θ(s i ))), Where i = 1, 2, ..., t, the leader line is represented as L[x(s), y(s), θ(s)].

[0028] In this embodiment, in step Q25, the high-level behavioral decision is decomposed into lateral decision and vertical decision. The lateral decision includes, but is not limited to, KEEP THE LANE, CHANGE TO LEFT LANE, and CHANGE TO RIGHTLANE. The vertical decision includes, but is not limited to, STOP, CONSTANT SPEED, ACCELERATION, DECELERATION, and DECELERATION TO ZERO.

[0029] In this embodiment, in step Q3, optimizing the generated driving trajectory and updating the trajectory includes: Q31. The trajectory output by the trajectory generation agent is given to the trajectory evaluation agent for anomaly analysis; Q32. Collision Risk Analysis: Calculate the direction of the vehicle's trajectory points and those of other obstacles using α = arctan(Δy / Δx), and then construct the vehicle's bounding box Polgon at each moment based on their respective lengths and widths. i and obstacle enclosure Polgon i j Calculate Polgon i and Polgon i j Are there any overlaps between them? Q33. Abnormal Parking Analysis: Calculate the average speed of the vehicle's planned trajectory points based on the vehicle's coordinates. , Among them, v avg <v min v minA low speed threshold indicates that the planned trajectory expects the vehicle to remain stationary or limp at a very low speed, and at this time there are no traffic lights or the traffic lights are in a green state, and there are no other obstacles affecting the vehicle, which is an abnormal parking state. Q34. Comfort Anomaly Analysis: The generated trajectory is fitted using a cubic polynomial. , Where t is time, and a0, a1, a2, a3 and b0, b1, b2, b3 are constant parameters, the acceleration a and curvature k at any point on the trajectory can be calculated. , If a>acc max or a<-dec max , among which, acc max and -dec max If k represents the maximum acceleration and maximum deceleration respectively, it indicates that the planned trajectory has an abnormal situation of rapid acceleration or deceleration. <k max ,k max The threshold representing the maximum curvature indicates an anomaly where the planned trajectory experiences sharp turns.

[0030] In this embodiment, optimizing and updating the generated driving trajectory further includes: Q35. For the different types of abnormal situations analyzed above, the following optimization strategies are given: 1) Collision risk: When a potential collision risk is detected, the longitudinal speed of the vehicle is automatically judged and adjusted according to the current scene environment, or the vehicle is avoided by lateral offset; 2) Abnormal parking: If the system detects a situation where the vehicle should not stop, the acceleration strategy is automatically triggered to restore the vehicle to normal driving speed; 3) Abnormal comfort: When the trajectory shows a sudden acceleration, sudden deceleration or sharp turn that does not meet the comfort requirements, the longitudinal or lateral speed is adjusted to ensure driving comfort and safety.

[0031] In this embodiment, optimizing and updating the generated driving trajectory further includes: Q36. Based on the optimization strategy provided by anomaly analysis, adjust the original trajectory to generate a new trajectory and avoid the above-mentioned anomaly problem.

[0032] In this embodiment, the present invention provides a multimodal large model dual-agent driving decision planning system, including a computer device that is programmed or configured to execute the steps of any of the multimodal large model dual-agent driving decision planning methods described above.

[0033] Example 2: Based on the multimodal large model dual-agent driving decision planning method in Example 1, the present invention will be further explained and described below.

[0034] like Figure 1 As shown, the present invention provides the following technical solution: a dual-agent autonomous driving decision-making and planning method and system based on a multimodal large model, comprising the following steps: Step 1: Construct a scene based on the map, obstacles, and vehicle information required for autonomous driving decision-making and planning, and convert it into the input required for a multimodal large model; Step 11: Select a guide line based on the navigation routing information. Decompose the vehicle's motion in Cartesian space into two independent one-dimensional motions: longitudinal motion along the guide line and lateral motion perpendicular to the guide line. The advantage of performing decision-making and planning tasks in the Frenet coordinate system is that it can effectively utilize the structural features of the road, thereby achieving a clearer scene understanding. Given that the coordinates of any point P in the Cartesian coordinate system are (x, y), and the guide line is represented as L[x(s), y(s), θ(s)], where x(s), y(s), and θ(s) are the x and y coordinates of the point corresponding to the arc length s on the guide line and the tangent direction, respectively, P is the nearest matching point P on the guide line L. r The state is (x(s) r ),y(s r ),θ(s r )), s r For P r The arc length on the guide line can be converted to Frenet coordinates using the following formula: , Here, s and l represent the longitudinal and lateral positions in Frenet coordinates, respectively. By reducing the dimensionality of the planning task, the complexity of the problem is greatly simplified. Simultaneously, it transforms the position and speed information of the vehicle and other traffic participants into mileage along the road and lateral offset from the road, thus more effectively representing the relative relationships between the vehicle and other traffic participants, as well as the spatial layout between different traffic participants and lanes.

[0035] Step 12: Map elements mapped to the Frenet coordinate system, such as lane lines, road boundaries, and pedestrian crossings, as well as obstacles like vehicles, motor vehicles, pedestrians, and traffic cones, and information about the vehicle and its historical trajectory, are drawn onto a BEV image using different colors and shapes. For example... Figure 3As shown, the white boxes represent the vehicle's position, direction, and shape; blue, magenta, red, and yellow boxes represent information about vehicles, motor vehicles, pedestrians, and other obstacles, respectively. Simultaneously, the corresponding colored dashed lines represent the historical trajectory points of these objects. Orange lines represent lane dividers, green lines represent road boundaries, and cyan lines represent pedestrian crossings. This method not only intuitively presents various elements and complex information in the scene but also enhances the expression of details, making the scene visualization clearer. Through this method, we can effectively display rich and diverse input information while avoiding the problem of exceeding the maximum input limit due to excessively long text tokens.

[0036] Step 13: Thanks to the BEV images, which clearly describe the rich and detailed input information in image form, the text prompts only need to briefly describe each object in the image. The text prompts primarily emphasize the pixel coordinates of each object in the image to aid in localization; simultaneously, they quantitatively describe the speed information of each object and provide the status of relevant traffic facilities such as traffic lights. Specifically, the text data generated for each BEV image includes: 1) Captions: describing the pixel coordinates and motion status and speed information of dynamic obstacles (such as vehicles, pedestrians, etc.) in the image, the vehicle's position, speed, and historical trajectory points, and traffic light information; 2) Reasoning: identifying targets that have a key impact on the vehicle's driving from the perception results, inferring the behavioral strategies the vehicle needs to take, and generating corresponding trajectory point data. This text prompt method not only effectively simplifies the input but also ensures accurate information delivery, further improving the accuracy and robustness of path planning.

[0037] Step 2: Based on a multimodal large language model (MLLM), the input data is comprehensively processed. Then, according to human driving thought processes and the functional decomposition of autonomous driving planning methods, we designed a thought process chain for autonomous driving planning: through scene analysis, behavioral decision-making, and trajectory planning, a driving trajectory is gradually generated. This process considers various dynamic factors to ensure the safety and rationality of the generated trajectory. Step 21: Using the CLIP-ViT-L-336px model as the visual encoder, the BEV image is understood and its features are extracted. By extracting its intermediate layer features, the local and global information of the image can be captured more effectively: F bev =CLIP(X bev ), where X bev For the input image information, F bev This encoding process, which extracts visual features, allows for a full expression of the details and overall structure of the image, enhancing a comprehensive understanding of the image content.

[0038] Step 22: The text prompt information is encoded using a Tonkenizer to obtain the text feature: Token. mllm =F bev +Token text ; Step 23: The output features of the visual encoder are then processed through a projection layer to align the dimensions of the image and text features, ultimately yielding the multimodal fused features: Token. mllm =F bev +Token text This alignment process ensures that image and text features can be effectively fused in the same representation space, thereby providing a unified input for subsequent large-scale language models, enabling image and text information to work together and improve the overall performance of the model.

[0039] Step 24: Analyze the current driving scenario, taking into account road conditions and interactions with other traffic participants, to identify key obstacles that will significantly impact the vehicle's future driving strategy.

[0040] Step 25: Based on the understanding of the scene and the prediction of key obstacles, combined with the vehicle's real-time information, make high-level behavioral decisions. This decision is further broken down into lateral and vertical decisions. Lateral decisions include, but are not limited to, KEEPTHE LANE, CHANGE TO LEFT LANE, and CHANGE TO RIGHT LANE; vertical decisions include, but are not limited to, STOP, CONSTANT SPEED, ACCELERATION, DECELERATION, and DECELERATION TO ZERO. By combining lateral and vertical decisions, flexible and diverse behavioral strategies can be generated.

[0041] Step 26: Based on the behavioral decision, generate a time-sampled trajectory point T in the Frenet coordinate system: {(s1,l1),(s2,l2),...,(s... t ,l t And these trajectory points are projected back to Cartesian space in the following way, and executed in a real environment: (x i ,y i )=(x(s i )-l i *sin(θ(s i )),y(s i )+l i *cos(θ(s i ))), Where i = 1, 2, ..., t, the leader line is represented as L[x(s), y(s), θ(s)].

[0042] Step 3: Further evaluate the generated trajectory, using a reflective reasoning chain to analyze potential problems and provide targeted optimization suggestions. Through this series of steps, a safe and reasonable driving trajectory is ultimately derived.

[0043] Step 31: The trajectory generated by the trajectory generation agent is given to the trajectory evaluation agent for anomaly analysis. The trajectory of the vehicle generated by the trajectory generation agent is represented as T: {(x1,y1),(x2,y2),...(x t ,y t The future trajectories of other obstacles are represented as T. j :{(x1 j ,y1 j ),(x2 j ,y2 j ),...(x t j ,y t j )}, where (x i j ,y i j ) represents the j-th obstacle at time t. i The module provides the location coordinates. In this module, we comprehensively analyze the generated trajectory information, key obstacle data, vehicle status information, and traffic light information to determine if there are any potential anomalies and classify these anomalies.

[0044] Step 32: Collision Risk Analysis: Calculate the direction of the trajectory points of the vehicle and other obstacles using α = arctan(Δy / Δx), and then construct the vehicle's bounding box Polgon at time t'. i and obstacle enclosure Polgon i j Calculate Polgon i and Polgon i j Whether there is overlap between them; if the polygon of the vehicle overlaps with the polygon of any obstacle at any time, it means that the generated trajectory has a collision risk.

[0045] Step 33: Abnormal Parking Analysis: Calculate the average speed of the vehicle's planned trajectory points based on the vehicle's coordinates. , Among them, v avg <v minv min A low-speed threshold indicates that the planned trajectory expects the vehicle to remain stationary or limp at an extremely low speed, and at this time there are no traffic lights or the traffic lights are in a green state, and there are no other obstacles affecting the vehicle, which is an abnormal parking state.

[0046] Step 34: Comfort Anomaly Analysis: The generated trajectory is fitted using a cubic polynomial. , Where t is time, and a0, a1, a2, a3 and b0, b1, b2, b3 are constant parameters, the acceleration a and curvature k at any point on the trajectory can be calculated. , If a>acc max or a<-dec max , among which, acc max and -dec max If k represents the maximum acceleration and maximum deceleration respectively, it indicates that the planned trajectory has an abnormal situation of rapid acceleration or deceleration. <k max ,k max The threshold representing the maximum curvature indicates an anomaly where the planned trajectory experiences sharp turns.

[0047] Step 35: For the different types of anomalies analyzed above, such as... Figure 4 or Figure 5 As shown, the corresponding optimization strategies are as follows: 1) Collision risk: When a potential collision risk is detected, the longitudinal speed of the vehicle is automatically judged and adjusted according to the current scene environment, or the vehicle is avoided by lateral offset; 2) Abnormal parking: If the system detects a situation where the vehicle should not stop, the acceleration strategy is automatically triggered to restore the vehicle to the normal driving speed; 3) Abnormal comfort: When the trajectory shows a sudden acceleration, sudden deceleration or sharp turn that does not meet the comfort requirements, the longitudinal or lateral speed is adjusted to ensure driving comfort and safety.

[0048] Step 36: Based on the optimization strategy provided by the anomaly analysis, adjust the original trajectory to generate a new, safer, and more reasonable trajectory, avoiding the aforementioned anomalies. This feedback mechanism effectively improves the stability of the autonomous driving system and the driving experience in complex scenarios.

[0049] In summary, this invention not only forms a closed-loop mechanism of "generation-evaluation-optimization" to effectively avoid the uncertainty of decision-making by a single intelligent agent, but also ensures that the final output path is safe and reliable.

[0050] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A driving decision-making and planning method based on a multimodal large model dual-agent system, characterized in that, The method includes: Q1. Scene Construction: Convert the information of the map, vehicles and other traffic participants to the Frenet coordinate system, then integrate them into a multi-layer comprehensive bird's-eye view, and generate corresponding text prompts; Q2. Trajectory Generation Agent: Based on a multimodal large language model, the agent comprehensively processes the corresponding text prompts input, identifies key obstacles of the vehicle through scene analysis, makes behavioral decisions, and generates a driving trajectory. Q3. Trajectory Evaluation Agent: Evaluates the generated driving trajectory, uses a reflective thinking chain to analyze problems in the trajectory, optimizes the generated driving trajectory, and updates the trajectory.

2. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 1, characterized in that, In step Q1, the process of uniformly converting information from maps, vehicles, and other traffic participants to the Frenet coordinate system, integrating it into a multi-layered comprehensive bird's-eye view, and generating corresponding text prompts includes: Q11. Select a guide line based on navigation routing information and decompose the vehicle's motion in Cartesian space into two independent one-dimensional motions: longitudinal motion along the guide line and lateral motion perpendicular to the guide line. Q12. Map elements mapped to the Frenet coordinate system, including lane lines, road boundaries, and pedestrian crossings, as well as obstacles such as vehicles, motor vehicles, pedestrians, and cones, along with vehicle and historical trajectory information, are drawn in a BEV image using different colors and shapes. Q13. Based on BEV images, describe the pixel coordinates and velocity information of each object in the image and generate corresponding text prompts.

3. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 2, characterized in that, In step Q11, the coordinates of any point P in the Cartesian coordinate system are known to be (x, y), and the guide line is represented as L[x(s), y(s), θ(s)], where x(s), y(s), and θ(s) are the x and y coordinates of the point corresponding to the arc length s on the guide line and the tangent direction, respectively. The nearest matching point P on the guide line L is Pn. r The state is (x(s) r ),y(s r ),θ(s r )), s r For P r The arc length on the guide line can be converted to Frenet coordinates using the following formula: ， Where s and l are the vertical and horizontal positions in Frenet coordinates, respectively.

4. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 2, characterized in that, In step Q13, describing the pixel coordinates and velocity information of each object in the image includes: Q131. Describe the pixel coordinates of dynamic obstacles in the image, their motion state and speed information, the position, speed and historical trajectory points of the vehicle, and the information of traffic lights; Q132. Identify targets that have a key impact on the vehicle's driving from the perception results, infer the vehicle's behavior strategy, and generate corresponding trajectory point data.

5. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 1, characterized in that, In step Q2, the process of making behavioral decisions and generating a driving trajectory includes: Q21. Using the CLIP-ViT-L-336px model as a visual encoder, perform understanding and feature extraction on BEV images. bev =CLIP(X bev ), where X bev For the input image information, F bev For the extracted visual features; Q22. Text prompt information is encoded using a Tonkenizer to obtain text features: Token text =Tonkenizer(prompts); Q23. The output features of the visual encoder are processed through a projection layer to align the dimensions of image and text features, ultimately yielding multimodal fused features: Token mllm =F bev +Token text ; Q24. Analyze the current driving scenario, taking into account road conditions and interactions with other traffic participants, to identify key obstacles; Q25. Based on the understanding of the scene and the prediction of key obstacles, combined with the real-time information of the vehicle, make high-level behavioral decisions; Q26. Based on behavioral decisions, generate a time-sampled trajectory point T in the Frenet coordinate system: {(s1,l1),(s2,l2),...,(s... t ,l t And these trajectory points are projected back to Cartesian space in the following way, and executed in a real environment: (x i ,y i )=(x(s i )-l i *sin(θ(s i )),y(s i )+l i *cos(θ(s i ))), Where i = 1, 2, ..., t, the leader line is represented as L[x(s), y(s), θ(s)].

6. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 5, characterized in that, In step Q25, the high-level behavioral decision is decomposed into lateral decisions and vertical decisions. The lateral decisions include, but are not limited to, KEEP THE LANE, CHANGE TO LEFT LANE, and CHANGE TO RIGHT LANE. The vertical decisions include, but are not limited to, STOP, CONSTANT SPEED, ACCELERATION, DECELERATION, and DECELERATION TO ZERO.

7. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 1, characterized in that, In step Q3, optimizing the generated driving trajectory and updating the trajectory includes: Q31. The trajectory output by the trajectory generation agent is given to the trajectory evaluation agent for anomaly analysis; Q32. Collision Risk Analysis: Calculate the direction of the vehicle's trajectory points and those of other obstacles using α = arctan(Δy / Δx), and then construct the vehicle's bounding box Polgon at each moment based on their respective lengths and widths. i and obstacle enclosure Polgon i j Calculate Polgon i and Polgon i j Are there any overlaps between them? Q33. Abnormal Parking Analysis: Calculate the average speed of the vehicle's planned trajectory points based on the vehicle's coordinates. ， Among them, v avg <v min v min A low speed threshold indicates that the planned trajectory expects the vehicle to remain stationary or limp at a very low speed, and at this time there are no traffic lights or the traffic lights are in a green state, and there are no other obstacles affecting the vehicle, which is an abnormal parking state. Q34. Comfort Anomaly Analysis: The generated trajectory is fitted using a cubic polynomial. ， Where t is time, and a0, a1, a2, a3 and b0, b1, b2, b3 are constant parameters, the acceleration a and curvature k at any point on the trajectory can be calculated. ， If a>acc max or a<-dec max , among which, acc max and -dec max If k represents the maximum acceleration and maximum deceleration respectively, it indicates that the planned trajectory has an abnormal situation of rapid acceleration or deceleration. <k max ,k max The threshold representing the maximum curvature indicates an anomaly where the planned trajectory experiences sharp turns.

8. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 7, characterized in that, The optimization and updating of the generated driving trajectory also includes: Q35. For the different types of abnormal situations analyzed above, the following optimization strategies are given: 1) Collision risk: When a potential collision risk is detected, the longitudinal speed of the vehicle is automatically judged and adjusted according to the current scene environment, or the vehicle is avoided by lateral offset; 2) Abnormal parking: If the system detects a situation where the vehicle should not stop, the acceleration strategy is automatically triggered to restore the vehicle to normal driving speed; 3) Abnormal comfort: When the trajectory shows a sudden acceleration, sudden deceleration or sharp turn that does not meet the comfort requirements, the longitudinal or lateral speed is adjusted to ensure driving comfort and safety.

9. The driving decision-making and planning method based on a multimodal large model dual-agent system according to claim 8, characterized in that, The optimization and updating of the generated driving trajectory also includes: Q36. Based on the optimization strategy provided by anomaly analysis, adjust the original trajectory to generate a new trajectory and avoid the above-mentioned anomaly problem.

10. A multimodal large-model dual-agent driving decision-making and planning system, comprising computer equipment, characterized in that, The computer device is programmed or configured to perform the steps of the multimodal large model dual-agent driving decision-making and planning method as described in any one of claims 1 to 9.

Images