Automatic driving decision planning method and system based on visual big language model

Abstract

The invention provides an automatic driving decision planning method based on a visual big language model, and the method comprises the steps: 1, obtaining video data and perception data outputted by a sensor, and uploading the video data and perception data to a cloud platform; step 2, converting obstacle information, road information and vehicle data into vector data, performing vector coding, converting vehicle state information into text data, performing text coding, performing visual coding on video data, and extracting environment features; 3, according to the vector data, the text data and the environment characteristics, generating coordinates of track points to be driven through a visual large language model thinking chain; and step 4, according to the behavior decision and the track point coordinates, track fitting smoothing is carried out, then rationality and safety are detected, and the track passing the detection is spliced with the track of the previous frame for vehicle driving.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving computing, and in particular to an autonomous driving decision planning method and system based on a visual large language model. Background Art

[0002] With the rapid development of autonomous driving technology in today's era, how to effectively achieve safe and efficient path planning in extremely complex and changeable environments has become a key issue in related research fields. Traditional path planning methods usually rely mainly on high-precision maps and sensor data. High-precision maps can provide detailed road information, and sensor data can perceive the surrounding environment in real time. However, these methods inevitably have certain limitations when dealing with dynamic environments and making real-time decisions. For example, high-precision maps may not be able to reflect temporary changes in the road in a timely manner, and sensor data may be interfered with in complex environments, resulting in inaccurate perception results.

[0003] Existing technical solutions directly use result-level data, which loses the rich language information in the driving environment and relies heavily on the accuracy of real-time data, including the vehicle's own status and road status information. Sensor errors or data delays may affect the accuracy of planning. In addition, the method of determining the target vehicle's driving planning information through function mapping relationships based on the current driving status information and the pre-built vehicle kinematic model is difficult to handle complex and changeable traffic environments and unstructured roads, and its generalization ability needs to be improved. Other technical solutions use rasterized bird's-eye views to represent dynamic and static targets in the scene. Due to the limitation of the resolution of the bird's-eye view, the accuracy of environmental information is lost. At the same time, the rasterized expression method has large data redundancy, low computational efficiency, and is not conducive to the representation of road structure and topological relationships. In addition, the end-to-end model lacks transparency and interpretability, and is closely based on the output of the model without rule-based safety judgment and post-processing, which may pose a risk of uncertainty. Summary of the invention

[0004] In view of this, the present invention provides an autonomous driving decision planning method and system based on a visual large language model to solve the technical problems such as the lack of transparency and explainability of the prior art, the need to improve the generalization ability in uncommon or unseen new environments, the reliance of decision planning on the results output by upstream perception and prediction modules, which leads to error accumulation and time delay, and the loss of rich semantic information of the original data in the perception results, which is not conducive to the decision planning module's understanding and cognition of the scene.

[0005] The present invention provides an automatic driving decision-making planning method based on a visual large language model, the method comprising: step 1, obtaining video data and perception data output by a sensor, and uploading them to a cloud platform, wherein the perception data comprises obstacle information, road information, vehicle data, and vehicle status information; step 2, converting the obstacle information, road information, and vehicle data into vector data, performing vector encoding, converting the vehicle status information into text data, performing text encoding, performing visual encoding on the video data, and extracting environmental features; step 3, generating the coordinates of a trajectory point to be driven through a visual large language model thinking chain according to the vector data, text data, and environmental features; step 4, performing trajectory fitting smoothing through the following function according to the behavior decision and the trajectory point coordinates: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y = g(t) = b0 + b1t + b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0006] Furthermore, the step 1 includes: step 11, uploading multiple image data to the cloud platform in real time; step 12, identifying obstacle information and road information, and uploading the identification results to the cloud platform; step 13, uploading the vehicle's own data to the cloud platform.

[0007] Furthermore, the step 2 includes: step 21, performing vector conversion on the obstacle information and the vehicle data, which is expressed as: A = [A0, A1, ..., A n ], where A0 represents the information of the vehicle, A1,...,A n represents the information of other obstacles, and n is the total number of obstacles; step 22, converting the moving target information of multiple historical frames to the reference coordinate system of the current frame; step 23, performing vector transformation on the target information, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center point, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed; Step 24, use a recurrent neural network to encode the target information: F A =LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t represents the number of historical frames considered; Step 25, vectorizing the map element information obtained by the sensor, expressed as: M i =[c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c is the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc., [x1,y1,x2,y2...,x l ,y l ] are all the points on the map element, l is the total number of points; Step 26, the map element is encoded through the fully connected layer to obtain the map feature: F M =FC(M); Step 27, splicing the images acquired by the visual sensor in the spatial dimension, and using the pre-trained visual encoder to extract image features F I Step 28, converting the navigation information into text data, using a byte-level encoding method to segment and encode the text data to obtain F T ; Step 29, use the linear network to transform the target feature F A 、Map Features F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

[0008] Furthermore, the step 3 includes: step 31, the visual large language model analyzes the driving scene according to the input vector data, text data and environmental characteristics, and screens out key obstacles that have an impact on the driving strategy of the vehicle according to the current road conditions and the interaction relationship with other surrounding traffic participants; step 32, based on the scene understanding and the prediction of key obstacles, combined with the vehicle's own data, behavioral decisions are made; step 33, the coordinates of the trajectory points to be driven are generated according to the behavioral decisions.

[0009] Furthermore, the step 31 includes: step 311, the visual large language model identifies key obstacles that have an impact on driving behavior from traffic participants based on input vector data, text data and environmental features; step 312, sorting them according to the degree of impact on driving behavior; step 313, predicting the impact results that the key obstacles will have according to the sorting.

[0010] Furthermore, the step 32 includes: step 321, making a behavioral decision based on scene understanding and prediction of key obstacles combined with the vehicle's own data; step 322, decomposing the behavioral decision result into lateral decisions and longitudinal decisions; step 323, forming a final behavioral decision through a combination of lateral decisions and longitudinal decisions for use by the vehicle.

[0011] Furthermore, the lateral decisions include lane keeping, lane changing, and lane avoidance; and the longitudinal decisions include parking, driving at a constant speed, accelerating, and decelerating.

[0012] Furthermore, the step 33 is specifically as follows: according to the behavior decision, the coordinates of N trajectory points in the future T time period are generated according to the time interval T0: T ego =[(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )].

[0013] Furthermore, the step 4 includes: step 41, the visual large language model fits and smoothes the generated trajectory coordinates to form a to-be-traveled trajectory; step 42, sampling the trajectory, using the coordinates and directions of the sampling points, combined with the vehicle's own data to construct the vehicle's bounding box at each future moment, and expand the safety distance, and determine whether the vehicle's bounding box overlaps with other obstacle bounding boxes at the same moment. If the vehicle and all obstacles do not overlap at all moments, then there is no collision risk for the trajectory, and calculate whether the vehicle's bounding box at each future moment intersects with the lane boundary line. If not, then there is no risk of the trajectory exceeding the road boundary; step 43, calculate whether the trajectory has an acceleration acc exceeding the vehicle limit max and deceleration dec max , and is smaller than the vehicle turning radius R min point, if not, the trajectory is executable; step 44, when the trajectory does not have a collision risk and is executable, the trajectory is spliced ​​with the previous frame trajectory, and the spliced ​​trajectory is used to control the vehicle to travel.

[0014] The present invention provides an automatic driving decision-making and planning system based on a visual large language model, the system comprising: a vehicle side, including a perception module and a planning module, the perception module is used to obtain video data and perception data output by a sensor, and upload them to a cloud platform, and the planning module is used to perform trajectory fitting and smoothing according to the behavior decision and trajectory point coordinates through the following function: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y = g(t) = b0 + b1t + b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively, and then detect the rationality and safety, and splice the detected trajectory with the previous frame trajectory for vehicle driving; the cloud platform is connected to the vehicle side through the network, and is used to convert obstacle information, road information, and vehicle data into vector data, perform vector encoding, convert vehicle status information into text data, perform text encoding, perform visual encoding on video data, extract environmental features, and generate the coordinates of the trajectory points to be driven through the visual large language model thinking chain according to the vector data, text data and environmental features, and send them to the vehicle side.

[0015] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of an autonomous driving decision-making planning method based on a visual large language model provided by the present invention;

[0017] Figure 2 It is a schematic diagram of an autonomous driving decision-making planning system based on a visual large language model provided by the present invention;

[0018] Figure 3 It is a flow chart of a method for the cloud platform provided by the present invention to process data uploaded by the vehicle end;

[0019] Figure 4 It is a schematic flow chart of a method for generating a trajectory of a vehicle to be driven provided by the present invention;

[0020] Figure 5 It is a schematic diagram of the visual large language model architecture provided by the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Embodiment 1

[0023] The present invention provides an automatic driving decision planning method and system based on a visual large language model, such as Figure 2 As shown, the system includes a vehicle side and a cloud platform, wherein the vehicle side includes a perception module and a planning module, the cloud platform exchanges information with the vehicle side through the network, and a visual large language model is set on the cloud platform, such as Figure 1 As shown, the method includes the following steps.

[0024] Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0025] The vehicle end includes a perception module, which is used to obtain video data and perception data output by the sensor and upload them to the cloud platform.

[0026] Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information and navigation information into text data and perform text encoding; perform visual encoding on video data and extract environmental features;

[0027] Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features;

[0028] The cloud platform is connected to the vehicle via a network, and uses the visual large language model thinking chain to generate trajectory point coordinates.

[0029] Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function:

[0030] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0031] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0032] T=argmin(w0cost″+w1cost″′),

[0033]

[0034] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0035] The vehicle side also includes a planning module, which is used to perform trajectory fitting and smoothing through a function according to the behavior decision and the coordinates of the trajectory points, and then detect rationality and safety, and splice the detected trajectory with the previous frame trajectory for vehicle driving.

[0036] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module.

[0037] Embodiment 2

[0038] The present invention provides an automatic driving decision planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0039] Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0040] The vehicle side acquires video data through a multi-eye camera, that is, a visual sensor, and uploads it to the cloud platform. At the same time, the vehicle-side automatic driving system will synchronously upload the perception results output by the perception module, including obstacle information, vehicle line information, and the vehicle status, including real-time position coordinates, speed, direction, etc. to the cloud. Usually, these multimodal features need to be aligned before being input into the cloud platform's visual language model for processing. The specific steps of this step are as follows.

[0041] Step 11, uploading multiple channels of image data to the cloud platform in real time;

[0042] The vehicle-side panoramic camera and visual sensor send multiple images to the cloud platform in real time.

[0043] Step 12: Identify obstacle information and road information, and upload the identification results to the cloud platform.

[0044] At the same time, the vehicle will also process the data collected by the camera, identify the information of obstacles through the 3D target detection module, including the category, 3D coordinates, size, direction, speed and other information of the obstacles, and identify lane lines, pedestrian crossing lines, road edges, confluence points, divergence points and other road information through the vehicle line detection module, and upload these perception result data to the cloud platform.

[0045] Step 13: Upload the vehicle's own data to the cloud platform.

[0046] In addition, the vehicle's own data, including vehicle coordinates, direction, speed, gear, throttle, brake, steering wheel angle and other information are also uploaded to the cloud simultaneously.

[0047] Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information and navigation information into text data and perform text encoding; perform visual encoding on video data and extract environmental features;

[0048] Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features;

[0049] Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function:

[0050] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0051] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0052] T=argmin(w0cost″+w1cost″′),

[0053]

[0054] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0055] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module.

[0056] Embodiment 3

[0057] The present invention provides an automatic driving decision planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0058] Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0059] Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information and navigation information into text data and perform text encoding; perform visual encoding on video data and extract environmental features;

[0060] like Figure 3 As shown, the specific steps for the cloud platform to process the data uploaded by the vehicle are as follows.

[0061] Step 21, the obstacle information and the vehicle data are vectorized and expressed as: A =

[0062] [A0,A1,...,A n ], where A0 represents the information of the vehicle, A1,...,A n Represents the information of other obstacles, where n is the total number of obstacles;

[0063] Step 22, converting the moving target information of the historical multiple frames into the reference coordinate system of the current frame;

[0064] Step 23, convert the target information into a vector, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i ,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center point, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed;

[0065] Step 24: Use a recurrent neural network to encode the target information: F A =

[0066] LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t is the number of historical frames considered;

[0067] Step 25, vectorize the map element information obtained by the sensor, expressed as: M i =

[0068] [c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c represents the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc.

[0069] [x1,y1,x2,y2...,x l ,y l] are all the points on the map element, l is the total number of points;

[0070] Step 26: The map elements are encoded through the fully connected layer to obtain map features: F M =FC(M);

[0071] Step 27, splicing the images acquired by the visual sensor in the spatial dimension, and extracting the image features F using the pre-trained visual encoder I ;

[0072] Step 28: convert the navigation information into text data, and use a byte-level encoding method to segment and encode the text data to obtain a text encoding feature F T ;

[0073] Step 29: Use a linear network to transform the target feature F A 、Map Features F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

[0074] Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features;

[0075] Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function:

[0076] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0077] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0078] T=argmin(w0cost″+w1cost″′),

[0079]

[0080] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0081] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module.

[0082] Embodiment 4

[0083] The present invention provides an automatic driving decision planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0084] Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0085] Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information and navigation information into text data and perform text encoding; perform visual encoding on video data and extract environmental features;

[0086] Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features;

[0087] like Figure 5 As shown, the system designs a visual large language model thinking chain for autonomous driving planning based on the thinking habits of human drivers and the functional decomposition of the autonomous driving planning method, gradually gives the final planning results, and generates trajectory point coordinates. The transparent thinking process greatly increases the interpretability of the system. The specific process is as follows.

[0088] Step 31, the visual language model analyzes the driving scene based on the input vector data, text data and environmental features, and selects key obstacles that affect the driving strategy of the vehicle based on the current road conditions and the interaction relationship with other traffic participants around.

[0089] In this step, through multimodal input, the visual large language model can fully understand the surrounding environmental information, identify the key targets that have a real impact on our driving behavior from many traffic participants, and arrange them in order according to the degree of impact on us. At the same time, it predicts what kind of impact these key targets will have on us. The specific steps are as follows.

[0090] Step 311, the visual large language model identifies key obstacles that have an impact on driving behavior from traffic participants based on the input vector data, text data and environmental features;

[0091] Step 312, sorting according to the degree of influence on driving behavior;

[0092] Step 313: predict the impact results that the key obstacles will produce according to the ranking.

[0093] For example, suppose a pedestrian suddenly crosses the vehicle in front of you on the right. If your vehicle continues to drive in its current state, there is a 90% chance that it will collide with the pedestrian in 1.5 seconds.

[0094] Step 32, based on scene understanding and prediction of key obstacles, combined with the vehicle's own data, behavior decisions are made;

[0095] This step makes a high-level behavioral decision based on the previous understanding of the scene and the prediction of key obstacles, combined with the vehicle information. The specific steps are as follows.

[0096] Step 321, making a behavior decision based on scene understanding and prediction of key obstacles combined with the vehicle's own data;

[0097] Step 322, decomposing the behavior decision result into horizontal decision and vertical decision;

[0098] Step 323, through the combination of lateral decision and longitudinal decision, a final behavior decision is formed for the vehicle to use.

[0099] The lateral decisions include lane keeping, lane change, lane avoidance, and longitudinal decisions include parking, constant speed driving, acceleration, and deceleration. The above decisions can also be increased or decreased according to actual conditions. Through the combination of lateral and longitudinal decisions, flexible and diverse final behavior decisions can be obtained.

[0100] Step 33, generating the coordinates of the trajectory points to be driven according to the behavior decision.

[0101] The specific process of this step is: according to the behavioral decision, generate the coordinates of N trajectory points in the future T time period according to the time interval of T0: T ego =

[0102] [(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )]. Among them, T0=0.5, T=3, N=6, that is, the coordinates of 6 trajectory points in the next 3 seconds are generated at a time interval of 0.5 seconds. Of course, these values ​​can be adjusted according to actual conditions.

[0103] Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function:

[0104] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0105] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0106] T=argmin(w0cost″+w1cost″′),

[0107]

[0108] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0109] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module.

[0110] Embodiment 5

[0111] The present invention provides an automatic driving decision planning method based on a visual large language model, such as Figure 1 As shown, the method includes:

[0112] Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information.

[0113] Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information and navigation information into text data and perform text encoding; perform visual encoding on video data and extract environmental features;

[0114] Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features;

[0115] Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function:

[0116] x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 ,

[0117] y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 ,

[0118] T=argmin(w0cost″+w1cost″′),

[0119]

[0120] Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

[0121] like Figure 4 As shown, the specific process of this step is as follows.

[0122] Step 41, the visual large language model performs fitting and smoothing on the generated trajectory coordinates to form a to-be-traveled trajectory;

[0123] Generally, the trajectory output by the visual language model on the cloud platform is the coordinates of points at intervals of 0.5 seconds. The trajectory points are relatively sparse and lack information such as curvature. The vehicle side will fit and interpolate the trajectory with a fifth-order polynomial and calculate the direction and curvature of each point. Specifically, taking the time t of the trajectory as the independent variable, the fifth-order polynomial equations f(t) and g(t) of the x-coordinate and y-coordinate are constructed respectively: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y = g(t) = b0 + b1t + b2t 2 +b3s 3 +

[0124] b4t 4 +b5t 5 , where a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively. In order to ensure the smoothness of the trajectory after fitting, we set the target equation as:

[0125] argmin(w0cost″+w1cost″′)

[0126]

[0127] Where w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′, respectively. The objective equation is solved using the quadratic programming optimization method to obtain the fitted trajectory equation. The dense trajectory point coordinates are obtained by interpolating the time t at a time interval of 0.1 seconds, and the curvature at each point can be calculated: Among them, x′, y′, x″, y″ are the first-order derivative and second-order derivative of x and y with respect to mileage s, respectively.

[0128] Step 42, sampling the trajectory, using the coordinates and directions of the sampling points and the vehicle's own data to construct the vehicle's bounding box at each future moment, and expanding the safe distance, to determine whether the vehicle's bounding box overlaps with other obstacle bounding boxes at the same moment. If the vehicle and all obstacles do not overlap at all moments, then there is no collision risk for the trajectory. Calculate whether the vehicle's bounding box at each future moment intersects with the lane boundary line. If not, then there is no risk of the trajectory exceeding the road boundary.

[0129] There are two main aspects of trajectory safety detection: whether the future trajectory has a risk of collision with other traffic participants and whether there is a risk of exceeding the road boundary.

[0130] Step 43, calculate whether the trajectory has an acceleration acc that exceeds the vehicle limit max and deceleration dec max , and is smaller than the vehicle turning radius R min If there is no point, the trajectory is executable;

[0131] The executability mentioned here refers to the rationality of the detection trajectory, that is, the trajectory is executable and the trajectory is reasonable.

[0132] Step 44: when the trajectory does not have the risk of collision and the risk of exceeding the road boundary and is executable, the trajectory is spliced ​​with the trajectory of the previous frame, and the vehicle is controlled to travel using the spliced ​​trajectory.

[0133] However, if there is a problem with the detection, it will be fed back to the visual language model for tuning, while continuing to use the planning trajectory generated by the vehicle-side autonomous driving system.

[0134] The present invention provides an automatic driving decision-making planning method and system based on a visual big language model. The technical solution exchanges information through the visual big language model system of the vehicle side and the cloud platform. The vehicle side obtains sensor data and uploads it to the cloud platform. The visual big language model on the cloud platform generates a trajectory and uploads it to the vehicle side. Finally, the vehicle side generates the final planning trajectory for vehicle driving. The present invention solves the technical problems that the prior art lacks transparency and interpretability, the generalization ability in uncommon or unseen new environments needs to be improved, the decision-making planning relies on the output results of the upstream perception and prediction modules, resulting in error accumulation and time delay, and the perception results lose the rich semantic information of the original data, which is not conducive to the understanding and cognition of the scene by the decision-making planning module.

[0135] In summary, the embodiment of the present invention provides an automatic driving decision planning method and system based on a visual large language model. The technical solution adopts a vehicle-cloud combined automatic driving decision planning framework. The vehicle uploads rich driving data to the cloud, deeply understands the driving scene through the cloud visual language large model, and sends the behavior decision and planning trajectory to the vehicle. The vehicle detects and smoothes the cloud trajectory through the on-board automatic driving system and post-processing module to ensure the safety and rationality of the final trajectory. The technical solution also comprehensively processes multi-modal data for complex traffic scenes, allowing the decision planning system to fully and deeply understand and recognize the driving environment from different angles, and vectorizes and encodes the result-level traffic participant information and map element information, effectively filters environmental noise, and better describes the topological structure information of the environment. At the same time, the original image features of the multi-camera data are extracted through CLIP, which retains rich semantic information and avoids information loss. Combined with the text encoding of navigation and other information, it is input into the large model through feature alignment to improve the decision planning performance in complex environments. In addition, this technical solution lacks explainability in the autonomous driving decision-making and planning process. By utilizing the reasoning ability of large models and combining it with the logic of human driving, the decision-making and planning thinking chain is designed. Through scene analysis, traffic participants who have a greater impact on the vehicle are identified, and reasonable behavioral decisions are made based on their possible impact on the vehicle in the future. Ultimately, specific trajectory point coordinates are generated. The transparent reasoning process solves the explainability problem of autonomous driving and helps enhance users' trust in the autonomous driving system, which is crucial for the widespread application and popularization of autonomous driving technology.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A decision-making and planning method for autonomous driving based on a visual large language model, characterized in that: The method comprises: Step 1: Obtain the video data and perception data output by the sensor and upload them to the cloud platform. The perception data includes obstacle information, road information, vehicle data, and vehicle status information. Step 2: Convert obstacle information, road information, and vehicle data into vector data and perform vector encoding; convert vehicle status information into text data and perform text encoding; perform visual encoding on video data and extract environmental features; Step 3: Generate the coordinates of the trajectory points to be driven through the visual language model thinking chain based on the vector data, text data and environmental features; Step 4: Based on the behavior decision and trajectory point coordinates, the trajectory fitting and smoothing is performed using the following function: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then the rationality and safety are detected, and the detected trajectory is spliced ​​with the previous frame trajectory for vehicle driving.

2. The autonomous driving decision planning method based on a visual large language model according to claim 1, characterized in that: The step 1 comprises: Step 11, uploading multiple channels of image data to the cloud platform in real time; Step 12: Identify obstacle information and road information, and upload the identification results to the cloud platform; Step 13: Upload the vehicle's own data to the cloud platform.

3. The autonomous driving decision planning method based on a visual large language model according to claim 1, characterized in that: The step 2 comprises: Step 21, the obstacle information and the vehicle data are vectorized and expressed as: A = [A0, A1, ..., A n ], where A0 represents the information of the vehicle, A1,...,A n Represents the information of other obstacles, where n is the total number of obstacles; Step 22, converting the moving target information of the historical multiple frames into the reference coordinate system of the current frame; Step 23, convert the target information into a vector, expressed as: A i =[c,x,y,z,θ,l,w,h,v] i ,i=0,1,...,n, where c represents the type of target, x, y, z are the position coordinates of the target center point, θ is the direction angle, l, w, h are the length, width and height respectively, and v is the speed; Step 24: Use a recurrent neural network to encode the target information: F A =LSTM(A f ,A f-1 ,A f-2 ,...,A f-t ), where f represents the current frame and t is the number of historical frames considered; Step 25, vectorize the map element information obtained by the sensor, expressed as: M i =[c,x1,y1,x2,y2,...,x l ,y l ] i ,i=1,2,...,m, where m is the total number of map elements and c is the type of map element, such as lane line, road edge line, stop line, crosswalk line, etc., [x1,y1,x2,y2...,x l ,y l ] are all the points on the map element, l is the total number of points; Step 26: The map elements are encoded through the fully connected layer to obtain map features: F M =FC(M); Step 27, splicing the images acquired by the visual sensor in the spatial dimension, and extracting the image features F using the pre-trained visual encoder I ; Step 28, converting the navigation information into text data, segmenting and encoding the text data using a byte-level encoding method, and obtaining F T ; Step 29: Use a linear network to transform the target feature F A 、Map Features F M and image features F I Convert to text encoding feature F T In the same spatial dimension, multi-modal feature alignment is achieved: F = Linear (F A )+Linear(F M )+Linear(F I )+F T , all features are fused and processed in a unified representation space.

4. The autonomous driving decision planning method based on a visual large language model according to claim 1, characterized in that: The step 3 comprises: Step 31, the visual language model analyzes the driving scene based on the input vector data, text data and environmental features, and selects key obstacles that affect the driving strategy of the vehicle based on the current road conditions and the interaction relationship with other traffic participants around. Step 32, based on scene understanding and prediction of key obstacles, combined with the vehicle's own data, behavior decisions are made; Step 33, generating the coordinates of the trajectory points to be driven according to the behavior decision.

5. The autonomous driving decision planning method based on a visual large language model according to claim 4, characterized in that: The step 31 comprises: Step 311, the visual large language model identifies key obstacles that have an impact on driving behavior from traffic participants based on the input vector data, text data and environmental features; Step 312, sorting according to the degree of influence on driving behavior; Step 313: predict the impact results that the key obstacles will produce according to the ranking.

6. The autonomous driving decision planning method based on a visual large language model according to claim 4, characterized in that: The step 32 comprises: Step 321, making a behavior decision based on scene understanding and prediction of key obstacles combined with the vehicle's own data; Step 322, decomposing the behavior decision result into horizontal decision and vertical decision; Step 323, through the combination of lateral decision and longitudinal decision, a final behavior decision is formed for the vehicle to use.

7. The autonomous driving decision planning method based on a visual large language model according to claim 6, characterized in that: The lateral decisions include lane keeping, lane changing, and lane avoidance; the longitudinal decisions include parking, constant speed driving, accelerating, and decelerating.

8. The autonomous driving decision planning method based on a visual large language model according to claim 4, characterized in that: The step 33 is specifically as follows: according to the behavior decision, generate N trajectory point coordinates in the future T time period according to the time interval T0: T ego =[(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x5,y5),……(x N ,y N )].

9. The autonomous driving decision planning method based on a visual large language model according to claim 1, characterized in that: The step 4 comprises: Step 41, the visual large language model performs fitting and smoothing on the generated trajectory coordinates to form a to-be-traveled trajectory; Step 42, sampling the trajectory, using the coordinates and directions of the sampling points and the vehicle's own data to construct the vehicle's bounding box at each future moment, and expanding the safe distance, to determine whether the vehicle's bounding box overlaps with other obstacle bounding boxes at the same moment. If the vehicle and all obstacles do not overlap at all moments, then there is no collision risk for the trajectory. Calculate whether the vehicle's bounding box at each future moment intersects with the lane boundary line. If not, then there is no risk of the trajectory exceeding the road boundary. Step 43, calculate whether the trajectory has an acceleration acc that exceeds the vehicle limit max and deceleration dec max , and is smaller than the vehicle turning radius R min If there is no point, the trajectory is executable; Step 44: when the trajectory does not have a collision risk and is executable, the trajectory is spliced ​​with the trajectory of the previous frame, and the vehicle is controlled to travel based on the spliced ​​trajectory.

10. A system for implementing the autonomous driving decision planning method based on a visual large language model according to claims 1-9, characterized in that: The system comprises: The vehicle side includes a perception module and a planning module. The perception module is used to obtain the video data and perception data output by the sensor and upload them to the cloud platform. The planning module is used to perform trajectory fitting and smoothing based on the behavior decisions and trajectory point coordinates through the following functions: x=f(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 , y=g(t)=b0+b1t+b2t 2 +b3s 3 +b4t 4 +b5t 5 , T=argmin(w0cost″+w1cost″′), Among them, a0, a1, a2, a3, a4, a5 and b0, b1, b, b3, b4, b5 are the coefficients of f(t) and g(t) respectively, w0 and w1 are the weight coefficients of the second-order derivative cost function cost″ and the third-order derivative cost function cost″′ respectively. Then check the rationality and safety, and splice the detected trajectory with the previous frame trajectory for vehicle driving; The cloud platform is connected to the vehicle through the network. It is used to convert obstacle information, road information, and vehicle data into vector data and perform vector encoding. It converts vehicle status information into text data and performs text encoding. It performs visual encoding on video data and extracts environmental features. Based on the vector data, text data, and environmental features, it generates the coordinates of the trajectory points to be driven through the visual large language model thinking chain and sends them to the vehicle.

Images