A method and system for autonomous driving decision planning based on a visual large language model

Abstract

The present invention provides an autonomous driving decision-making and planning method based on a visual large language model, the method comprising: step 1, obtaining video data and perception data output by a sensor, and uploading the data to a cloud platform; step 2, converting obstacle information, road information, and vehicle data into vector data, performing vector encoding, converting vehicle status information into text data, performing text encoding, performing visual encoding on the video data, and extracting environmental features; step 3, generating the coordinates of a trajectory point to be traveled through a visual large language model thinking chain based on the vector data, text data, and environmental features; step 4, performing trajectory fitting and smoothing based on the behavior decision and the trajectory point coordinates, then detecting rationality and safety, and splicing the detected trajectory with the previous frame trajectory for use in vehicle driving.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving computing, and in particular to an autonomous driving decision-making planning method and system based on a large visual language model. Background Art

[0002] With the rapid development of autonomous driving technology today, achieving safe and efficient path planning in extremely complex and ever-changing environments has become a key research topic. Traditional path planning methods typically rely primarily on high-precision maps and sensor data. High-precision maps provide detailed road information, while sensor data enables real-time perception of the surrounding environment. However, these methods inevitably have limitations when dealing with dynamic environments and making real-time decisions. For example, high-precision maps may not reflect temporary changes in the road in a timely manner, and sensor data may be subject to interference in complex environments, resulting in inaccurate perception results.

[0003] Existing technical solutions directly use result-level data, losing the rich linguistic information in the driving environment and relying heavily on the accuracy of real-time data, including information about the vehicle's own state and road conditions. Sensor errors or data delays may affect planning accuracy. Furthermore, methods that determine target vehicle driving planning information through function mapping relationships based on current driving state information and a pre-built vehicle kinematic model struggle to handle complex and changing traffic environments and unstructured roads, and their generalization capabilities need improvement. Other technical solutions use rasterized bird's-eye views to represent dynamic and static objects in the scene. However, due to the resolution limitations of bird's-eye views, the accuracy of environmental information is lost. Rasterized representations also suffer from high data redundancy, low computational efficiency, and poor representation of road structure and topological relationships. Furthermore, end-to-end models lack transparency and interpretability, and rely solely on model outputs without rule-based safety assessments and post-processing, potentially introducing uncertainty risks. Summary of the Invention

[0004] In view of this, the present invention provides an autonomous driving decision-making planning method and system based on a visual large language model to address technical issues such as the lack of transparency and explainability of existing technologies, the need to improve generalization capabilities in uncommon or unseen new environments, the reliance of decision-making planning on the output results of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information of the original data in the perception results, which is not conducive to the decision-making planning module's understanding and cognition of the scene.

[0005] The present invention provides an autonomous driving decision-making and planning method based on a visual large language model, the method comprising: step 1, obtaining video data and perception data output by a sensor and uploading them to a cloud platform, wherein the perception data includes obstacle information, road information, vehicle data, and vehicle status information; step 2, converting the obstacle information, road information, and vehicle data into vector data and performing vector encoding, converting the vehicle status information into text data and performing text encoding, performing visual encoding on the video data, and extracting environmental features; step 3, generating the coordinates of a trajectory point to be driven through a visual large language model thinking chain based on the vector data, text data, and environmental features; step 4, performing trajectory fitting and smoothing based on the behavior decision and the trajectory point coordinates using the following function: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0006] Furthermore, the step 1 includes: step 11, uploading multiple image data to the cloud platform in real time; step 12, identifying obstacle information and road information, and uploading the identification results to the cloud platform; step 13, uploading the vehicle's own data to the cloud platform.

[0007] Furthermore, the step 2 includes: step 21, performing vector conversion on the obstacle information and the vehicle data, which is expressed as: A=[A0, A1, ..., A n ], where A0 represents the information of the vehicle, A1,...,A n Indicates the information of other obstacles, n is the total number of obstacles; Step 22, convert the moving target information of multiple historical frames to the reference coordinate system of the current frame; Step 23, perform vector conversion on the target information, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed; Step 24, use a recurrent neural network to encode the target information: F A =LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t represents the number of historical frames considered; Step 25, vectorize the map element information obtained by the sensor, expressed as: M i =[c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c represents the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc., [x1,y1,x2,y2...,x l ,y l ] are all the points on the map element, l is the total number of points; Step 26, the map element is encoded through the fully connected layer to obtain the map feature: F M =FC(M); Step 27, splicing the images acquired by the visual sensor in the spatial dimension, and using the pre-trained visual encoder to extract image features F I Step 28: convert the navigation information into text data, and use the byte-level encoding method to segment and encode the text data to obtain F T ; Step 29, use the linear network to transform the target feature F A 、Map feature F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

[0008] Furthermore, step 3 includes: step 31, the visual large language model analyzes the driving scene based on the input vector data, text data and environmental characteristics, and screens out key obstacles that have an impact on the vehicle's driving strategy based on the current road conditions and the interaction between other traffic participants around it; step 32, based on the scene understanding and the prediction of key obstacles, combined with the vehicle's own data, behavioral decision-making is made; step 33, the coordinates of the trajectory points to be driven are generated based on the behavioral decision.

[0009] Furthermore, the step 31 includes: step 311, the visual large language model identifies key obstacles that affect driving behavior from traffic participants based on input vector data, text data and environmental features; step 312, sorting them according to the degree of influence on driving behavior; step 313, predicting the impact results that the key obstacles will produce according to the sorting.

[0010] Furthermore, step 32 includes: step 321, making behavioral decisions based on scene understanding and prediction of key obstacles, combined with the vehicle's own data; step 322, decomposing the behavioral decision results into horizontal decisions and vertical decisions; step 323, forming a final behavioral decision through the combination of horizontal decisions and vertical decisions for vehicle use.

[0011] Furthermore, the lateral decision-making includes maintaining the lane, changing lanes left or right, and avoiding the lane, and the longitudinal decision-making includes stopping, driving at a constant speed, accelerating, and decelerating.

[0012] Furthermore, the step 33 is specifically as follows: according to the behavior decision, generate N trajectory point coordinates in the future T time period according to the time interval T0: T ego =[(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )].

[0013] Furthermore, the step 4 includes: step 41, the visual large language model fits and smoothes the generated trajectory coordinates to form a trajectory to be driven; step 42, sampling the trajectory, using the coordinates and directions of the sampling points, combined with the vehicle's own data to construct the vehicle's bounding box at each future moment, and expand the safety distance, and judge whether the vehicle's bounding box at the same moment overlaps with other obstacle bounding boxes. If the vehicle and all obstacles do not overlap at all times, then there is no collision risk for the trajectory, and calculate whether the vehicle's bounding box at each future moment intersects with the lane boundary line. If not, then there is no risk of the trajectory exceeding the road boundary; step 43, calculate whether the trajectory has an acceleration acc exceeding the vehicle limit max and deceleration dec max , and less than the vehicle turning radius R min point, if not, the trajectory is executable; step 44, when the trajectory does not have a collision risk and is executable, the trajectory is spliced ​​with the previous frame trajectory, and the spliced ​​trajectory is used to control the vehicle to travel.

[0014] The present invention provides an autonomous driving decision-making and planning system based on a visual large language model. The system includes: a vehicle side, including a perception module and a planning module. The perception module is used to obtain video data and perception data output by the sensor and upload them to the cloud platform. The planning module is used to perform trajectory fitting and smoothing based on the behavior decision and trajectory point coordinates through the following function: x = f(t) = a0 + a1t + a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving; the cloud platform is connected to the vehicle side through the network, and is used to convert obstacle information, road information, and vehicle data into vector data, perform vector encoding, convert vehicle status information into text data, perform text encoding, perform visual encoding on video data, extract environmental features, and generate the coordinates of the trajectory points to be driven through the visual large language model thinking chain based on the vector data, text data and environmental features, and send them to the vehicle side.

[0015] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of an autonomous driving decision-making and planning method based on a visual large language model provided by the present invention;

[0017] Figure 2 This is a schematic diagram of an autonomous driving decision-making and planning system based on a visual large language model provided by the present invention;

[0018] Figure 3 This is a flow chart of a method for processing data uploaded by a vehicle end by a cloud platform provided by the present invention;

[0019] Figure 4 It is a flow chart of a method for generating a vehicle's to-be-traveled trajectory provided by the present invention;

[0020] Figure 5 This is a schematic diagram of the visual language model architecture provided by the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Example 1

[0023] The present invention provides an autonomous driving decision-making planning method and system based on a visual large language model, such as Figure 2 As shown, the system includes a vehicle side and a cloud platform, wherein the vehicle side includes a perception module and a planning module, the cloud platform exchanges information with the vehicle side through the network, and a visual language model is set on the cloud platform, such as Figure 1 As shown, the method includes the following steps.

[0024] Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0025] The vehicle side includes a perception module for acquiring video data and perception data output by sensors and uploading them to the cloud platform.

[0026] Step 2: Obstacle information, road information, and vehicle data are converted into vector data and vector-encoded. Vehicle status information and navigation information are converted into text data and text-encoded. Video data is visually encoded to extract environmental features.

[0027] Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain;

[0028] The cloud platform is connected to the vehicle via a network and uses a visual large language model thinking chain to generate trajectory point coordinates.

[0029] Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function:

[0030] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0031] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0032] T=argmin(w0cost″+w1cost″′),

[0033]

[0034] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0035] The vehicle side also includes a planning module, which is used to perform trajectory fitting and smoothing through a function based on behavioral decisions and trajectory point coordinates, then detect rationality and safety, and splice the detected trajectory with the previous frame trajectory for vehicle driving use.

[0036] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene.

[0037] Example 2

[0038] The present invention provides an autonomous driving decision-making planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0039] Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0040] The vehicle captures video data using a multi-camera system (also known as a visual sensor) and uploads it to the cloud platform. Simultaneously, the vehicle's autonomous driving system uploads the perception module's output, including obstacle information, lane information, and the vehicle's status, including real-time location coordinates, speed, and direction, to the cloud. Typically, these multimodal features are aligned before being input into the cloud platform's large visual language model for processing. The specific steps for this step are as follows.

[0041] Step 11, uploading multiple channels of image data to the cloud platform in real time;

[0042] The vehicle-side surround view camera and visual sensor send multiple images to the cloud platform in real time.

[0043] Step 12: Identify obstacle information and road information, and upload the identification results to the cloud platform.

[0044] At the same time, the vehicle side will also process the data collected by the camera, identify the information of obstacles through the 3D target detection module, including the category, 3D coordinates, size, direction, speed and other information of the obstacles, and identify lane lines, crosswalk lines, road edges, merging points, diverging points and other road information through the vehicle line detection module, and upload these perception result data to the cloud platform.

[0045] Step 13: Upload the vehicle's own data to the cloud platform.

[0046] In addition, the vehicle's own data, including vehicle coordinates, direction, speed, gear, throttle, brake, steering wheel angle and other information are also uploaded to the cloud simultaneously.

[0047] Step 2: Obstacle information, road information, and vehicle data are converted into vector data and vector-encoded. Vehicle status information and navigation information are converted into text data and text-encoded. Video data is visually encoded to extract environmental features.

[0048] Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain;

[0049] Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function:

[0050] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0051] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0052] T=argmin(w0cost″+w1cost″′),

[0053]

[0054] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0055] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene.

[0056] Example 3

[0057] The present invention provides an autonomous driving decision-making planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0058] Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0059] Step 2: Obstacle information, road information, and vehicle data are converted into vector data and vector-encoded. Vehicle status information and navigation information are converted into text data and text-encoded. Video data is visually encoded to extract environmental features.

[0060] like Figure 3 As shown in FIG, the specific steps for the cloud platform to process the data uploaded by the vehicle are as follows.

[0061] Step 21, the obstacle information and the vehicle data are vectorized and expressed as: A=

[0062] [A0,A1,...,A n ], where A0 represents the information of the vehicle, A1,...,A n Represents the information of other obstacles, where n is the total number of obstacles;

[0063] Step 22: convert the moving target information of multiple historical frames into the reference coordinate system of the current frame;

[0064] Step 23, convert the target information into a vector, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i ,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed;

[0065] Step 24: Use a recurrent neural network to encode the target information: F A =

[0066] LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t is the number of historical frames considered;

[0067] Step 25: vectorize the map element information obtained by the sensor, expressed as: M i =

[0068] [c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c represents the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc.

[0069] [x1,y1,x2,y2...,x l ,y l] are all the points on the map element, and l is the total number of points;

[0070] Step 26: Map elements are encoded through the fully connected layer to obtain map features: F M =FC(M);

[0071] Step 27: splice the images acquired by the visual sensor in the spatial dimension and use the pre-trained visual encoder to extract the image features F I ;

[0072] Step 28: Convert the navigation information into text data, and use byte-level encoding method to segment and encode the text data to obtain the text encoding feature F T ;

[0073] Step 29: Use a linear network to transform the target feature F A 、Map feature F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

[0074] Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain;

[0075] Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function:

[0076] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0077] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0078] T=argmin(w0cost″+w1cost″′),

[0079]

[0080] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0081] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene.

[0082] Example 4

[0083] The present invention provides an autonomous driving decision-making planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0084] Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0085] Step 2: Obstacle information, road information, and vehicle data are converted into vector data and vector-encoded. Vehicle status information and navigation information are converted into text data and text-encoded. Video data is visually encoded to extract environmental features.

[0086] Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain;

[0087] like Figure 5 As shown, the system designs a visual language model thinking chain for autonomous driving planning based on human driving thinking habits and functional decomposition of autonomous driving planning methods, gradually gives the final planning results, and generates trajectory point coordinates. The transparent thinking process greatly increases the interpretability of the system. The specific process is as follows.

[0088] Step 31: The visual language model analyzes the driving scene based on the input vector data, text data, and environmental features. It then identifies key obstacles that may affect the vehicle's driving strategy based on the current road conditions and the interactions with other traffic participants.

[0089] In this step, through multimodal input, the visual language model can fully understand the surrounding environmental information, identify key targets that have a real-value impact on our driving behavior from among many traffic participants, and arrange them in order of their impact on us. At the same time, it predicts the impact of these key targets on us. The specific steps are as follows.

[0090] Step 311 , the visual language model identifies key obstacles that affect driving behavior from traffic participants based on the input vector data, text data, and environmental features;

[0091] Step 312: sorting by the degree of influence on driving behavior;

[0092] Step 313: predict the impact of the key obstacles according to the ranking.

[0093] For example, suppose a pedestrian suddenly crosses the vehicle in front of the right side. If the vehicle continues to drive in its current state, there is a 90% chance that it will collide with the pedestrian in 1.5 seconds.

[0094] Step 32: Make behavioral decisions based on scene understanding and prediction of key obstacles, combined with the vehicle's own data;

[0095] This step makes a high-level behavioral decision based on the previous understanding of the scene and the prediction of key obstacles, combined with the vehicle information. The specific steps are as follows.

[0096] Step 321: Make behavioral decisions based on scene understanding and prediction of key obstacles, combined with the vehicle's own data;

[0097] Step 322, decomposing the behavioral decision result into horizontal decision and vertical decision;

[0098] In step 323 , a final behavior decision is formed by combining the lateral decision and the longitudinal decision for use by the vehicle.

[0099] Lateral decisions include lane keeping, lane changing, and lane avoidance. Longitudinal decisions include stopping, driving at a constant speed, accelerating, and decelerating. These decisions can be increased or decreased based on actual conditions. By combining lateral and longitudinal decisions, a flexible and diverse final behavior decision can be achieved.

[0100] Step 33: Generate the coordinates of the trajectory points to be driven according to the behavior decision.

[0101] The specific process of this step is: according to the behavioral decision, generate the coordinates of N trajectory points in the future T time period according to the time interval T0: T ego =

[0102] [(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )]. Where T0=0.5, T=3, N=6, that is, the coordinates of 6 trajectory points in the next 3 seconds are generated at time intervals of 0.5 seconds. Of course, these values ​​can be adjusted according to actual conditions.

[0103] Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function:

[0104] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0105] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0106] T=argmin(w0cost″+w1cost″′),

[0107]

[0108] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0109] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene.

[0110] Example 5

[0111] The present invention provides an autonomous driving decision-making planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0112] Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0113] Step 2: Obstacle information, road information, and vehicle data are converted into vector data and vector-encoded. Vehicle status information and navigation information are converted into text data and text-encoded. Video data is visually encoded to extract environmental features.

[0114] Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain;

[0115] Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function:

[0116] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0117] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0118] T=argmin(w0cost″+w1cost″′),

[0119]

[0120] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0121] like Figure 4 As shown, the specific process of this step is as follows.

[0122] Step 41: The visual language model is used to fit and smooth the generated trajectory coordinates to form a trajectory to be driven;

[0123] Generally, the trajectory output by the visual language model on the cloud platform is the coordinates of points at 0.5-second intervals. The trajectory points are relatively sparse and lack information such as curvature. The vehicle side will fit and interpolate the trajectory with a fifth-order polynomial and calculate the direction and curvature of each point. Specifically, with the time t of the trajectory as the independent variable, the fifth-order polynomial equations f(t) and g(t) for the x-coordinate and y-coordinate are constructed respectively: x = f(t) = a0 + a1t + a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +

[0124] b4t 4 +b5t 5 , where a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively. In order to ensure the smoothness of the fitted trajectory, we set the target equation as:

[0125] argmin(w0cost″+w1cost″′)

[0126]

[0127] Where w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′, respectively. The objective equation is solved using the quadratic programming optimization method to obtain the fitted trajectory equation. By interpolating the time t at 0.1 second intervals, the dense trajectory point coordinates are obtained, and the curvature at each point can be calculated: Where x′, y′, x″, and y″ are the first-order derivative and second-order derivative of x and y with respect to mileage s, respectively.

[0128] Step 42: Sample the trajectory. Using the coordinates and directions of the sampling points, combined with the vehicle's own data, construct the bounding box of the ego vehicle at each future moment. This box is then expanded to a safe distance. A check is performed to determine whether the ego vehicle's bounding box overlaps with the bounding boxes of other obstacles at the same moment. If the ego vehicle and all obstacles do not overlap at all times, then the trajectory has no collision risk. The ego vehicle's bounding box at each future moment is then calculated to determine whether it intersects the lane boundary. If not, then the trajectory has no risk of exceeding the road boundary.

[0129] There are two main aspects of trajectory safety detection: whether the future trajectory has a risk of collision with other traffic participants and whether there is a risk of exceeding the road boundary.

[0130] Step 43: Calculate whether the trajectory has an acceleration acc exceeding the vehicle limit. max and deceleration dec max , and less than the vehicle turning radius R min If there is no point, the trajectory is executable;

[0131] The executability mentioned here refers to the rationality of the detection trajectory, that is, the trajectory is executable and the trajectory is reasonable.

[0132] Step 44 : When the trajectory does not have the risk of collision and the risk of exceeding the road boundary and is executable, the trajectory is spliced ​​with the trajectory of the previous frame, and the vehicle is controlled to travel using the spliced ​​trajectory.

[0133] However, if there is a problem with the detection, it will be fed back to the visual language model for tuning, while continuing to use the planning trajectory generated by the vehicle-side autonomous driving system.

[0134] The present invention provides a method and system for autonomous driving decision-making and planning based on a visual large language model. This technical solution uses the visual large language model system on the vehicle and a cloud platform to exchange information. The vehicle acquires sensor data and uploads it to the cloud platform. The visual large language model on the cloud platform then generates a trajectory, which is uploaded to the vehicle. Finally, the vehicle generates the final planned trajectory for use by the vehicle. The present invention addresses technical issues such as the lack of transparency and interpretability of existing technologies, the need to improve generalization capabilities in unusual or unseen new environments, the reliance of decision-making and planning on the outputs of upstream perception and prediction modules, which leads to error accumulation and time delays, and the loss of rich semantic information in the original data in the perception results, which is not conducive to the decision-making and planning module's understanding and cognition of the scene.

[0135] In summary, embodiments of the present invention provide a method and system for autonomous driving decision-making and planning based on a large visual language model. This technical solution adopts a vehicle-cloud-integrated autonomous driving decision-making and planning framework. The vehicle uploads rich driving data to the cloud, uses the cloud-based large visual language model to gain an in-depth understanding of the driving scene, and sends behavioral decisions and planned trajectories to the vehicle. The vehicle detects and smoothes the cloud-based trajectory through the on-board autonomous driving system and post-processing module to ensure the safety and rationality of the final trajectory. This technical solution also comprehensively processes multimodal data for complex traffic scenarios, allowing the decision-making and planning system to fully and deeply understand and recognize the driving environment from different perspectives. Result-level traffic participant information and map element information are vectorized and encoded, effectively filtering environmental noise and better describing the topological structure of the environment. At the same time, the original image features of the multi-camera data are extracted through CLIP, preserving rich semantic information and avoiding information loss. Combined with the text encoding of navigation and other information, the data is input into the large model through feature alignment, improving decision-making and planning performance in complex environments. In addition, this technical solution solves the problem of the lack of explainability of the autonomous driving decision-making and planning process. It uses the reasoning ability of large models and combines the logic of human driving to design a decision-making and planning thinking chain. Through scene analysis, it identifies traffic participants who have a greater impact on the vehicle, and makes reasonable behavioral decisions based on their possible impact on the vehicle in the future, and finally generates specific trajectory point coordinates. The transparent reasoning process solves the explainability problem of autonomous driving and helps enhance users' trust in the autonomous driving system, which is crucial for the widespread application and popularization of autonomous driving technology.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A decision-making and planning method for autonomous driving based on a large visual language model, characterized in that: The method comprises: Step 1: Obtain video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information. Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding. Convert vehicle status information into text data and perform text encoding. Perform visual encoding on video data and extract environmental features. Step 3: Generate the coordinates of the trajectory points to be driven based on the vector data, text data, and environmental features through the visual language model thinking chain; Step 4: Based on the behavior decision and trajectory point coordinates, perform trajectory fitting and smoothing using the following function: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

2. The autonomous driving decision-making and planning method based on a visual large language model according to claim 1, characterized in that: The step 1 comprises: Step 11, uploading multiple channels of image data to the cloud platform in real time; Step 12: Identify obstacle information and road information, and upload the identification results to the cloud platform; Step 13: Upload the vehicle's own data to the cloud platform.

3. The autonomous driving decision-making and planning method based on a visual large language model according to claim 1, characterized in that: The step 2 includes: Step 21, the obstacle information and the vehicle data are vectorized and expressed as: A=[A0,A1,...,A n ], where A0 represents the information of the vehicle, A1,...,A n Represents the information of other obstacles, where n is the total number of obstacles; Step 22: convert the moving target information of multiple historical frames into the reference coordinate system of the current frame; Step 23, convert the target information into a vector, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i ,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed; Step 24: Use a recurrent neural network to encode the target information: F A =LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t is the number of historical frames considered; Step 25: vectorize the map element information obtained by the sensor, expressed as: M i =[c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c represents the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc., [x1,y1,x2,y2...,x l ,y l ] are all the points on the map element, and l is the total number of points; Step 26: Map elements are encoded through the fully connected layer to obtain map features: F M =FC(M); Step 27: splice the images acquired by the visual sensor in the spatial dimension and use the pre-trained visual encoder to extract the image features F I ; Step 28: Convert the navigation information into text data, and use a byte-level encoding method to segment and encode the text data to obtain F T ; Step 29: Use a linear network to transform the target feature F A 、Map feature F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

4. The autonomous driving decision-making and planning method based on a visual large language model according to claim 1, characterized in that: The step 3 comprises: Step 31: The visual language model analyzes the driving scene based on the input vector data, text data, and environmental features. It then identifies key obstacles that may affect the vehicle's driving strategy based on the current road conditions and the interactions with other traffic participants. Step 32: Make behavioral decisions based on scene understanding and prediction of key obstacles, combined with the vehicle's own data; Step 33: Generate the coordinates of the trajectory points to be driven according to the behavior decision.

5. The autonomous driving decision-making and planning method based on a visual large language model according to claim 4, characterized in that: The step 31 includes: Step 311 , the visual language model identifies key obstacles that affect driving behavior from traffic participants based on the input vector data, text data, and environmental features; Step 312: sorting by the degree of influence on driving behavior; Step 313: predict the impact of the key obstacles according to the ranking.

6. The autonomous driving decision-making and planning method based on a visual large language model according to claim 4, characterized in that: The step 32 includes: Step 321: Make behavioral decisions based on scene understanding and prediction of key obstacles, combined with the vehicle's own data; Step 322, decomposing the behavioral decision result into horizontal decision and vertical decision; In step 323 , a final behavior decision is formed by combining the lateral decision and the longitudinal decision for use by the vehicle.

7. The autonomous driving decision-making and planning method based on a visual large language model according to claim 6, characterized in that: The lateral decisions include lane keeping, lane changing, and lane avoidance, and the longitudinal decisions include parking, constant speed driving, accelerating, and decelerating.

8. The autonomous driving decision-making and planning method based on a visual large language model according to claim 4, characterized in that: The step 33 is specifically as follows: according to the behavior decision, generate N trajectory point coordinates in the future T time period according to the time interval T0: ego =[(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )].

9. The autonomous driving decision-making and planning method based on a visual large language model according to claim 1, characterized in that: The step 4 comprises: Step 41: The visual language model is used to fit and smooth the generated trajectory coordinates to form a trajectory to be driven; Step 42: Sample the trajectory. Using the coordinates and directions of the sampling points, combined with the vehicle's own data, construct the bounding box of the ego vehicle at each future moment. This box is then expanded to a safe distance. A check is performed to determine whether the ego vehicle's bounding box overlaps with the bounding boxes of other obstacles at the same moment. If the ego vehicle and all obstacles do not overlap at all times, then the trajectory has no collision risk. The ego vehicle's bounding box at each future moment is then calculated to determine whether it intersects the lane boundary. If not, then the trajectory has no risk of exceeding the road boundary. Step 43: Calculate whether the trajectory has an acceleration acc exceeding the vehicle limit. max and deceleration dec max , and less than the vehicle turning radius R min If there is no point, the trajectory is executable; Step 44 : When the trajectory does not have a collision risk and is executable, the trajectory is spliced ​​with the trajectory of the previous frame, and the vehicle is controlled to travel using the spliced ​​trajectory.

10. A system for implementing the autonomous driving decision-making and planning method based on a visual large language model according to claims 1-9, characterized in that: The system comprises: The vehicle side includes a perception module and a planning module. The perception module is used to obtain the video data and perception data output by the sensor and upload them to the cloud platform. The planning module is used to perform trajectory fitting and smoothing based on the behavior decisions and trajectory point coordinates using the following function: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then, after checking the rationality and safety, the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving; The cloud platform is connected to the vehicle through the network. It is used to convert obstacle information, road information, and vehicle data into vector data and perform vector encoding. It also converts vehicle status information into text data and performs text encoding. It also performs visual encoding on video data and extracts environmental features. Based on the vector data, text data, and environmental features, it generates the coordinates of the trajectory points to be traveled through the visual large language model thinking chain and sends them to the vehicle.

Images