A method, system and storage medium for predicting risk situation of autonomous driving vehicles based on multimodal information fusion and large model deduction

Abstract

The present invention discloses a risk situation prediction method, system and storage medium for autonomous driving vehicles based on multimodal information fusion and large model deduction. The scheme integrates multimodal information from cameras, lidars and multiple sensors, and transmits and aggregates visible light images and infrared images. The data from multiple sensors are fused to form dynamic spatiotemporal information of surround view. The dynamic and static features of 2D-3D surround view images are fused using a neighbor cross attention mechanism; a prediction model is established using a large model, and the spatiotemporal risk information and text prompt information are analyzed to generate short-term and long-term prediction results. By evaluating the cost of each short-term plan and the corresponding long-term response plan, the one with the smallest total cost is selected as the optimal strategy to guide the safe driving of intelligent vehicles.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle technology, and in particular to a method, system and storage medium for deducing the long-term and short-term driving risk situation of intelligent vehicles based on a large model. The method utilizes multi-modal sensor information, combined with the long-term and short-term risk prediction results generated by the large model, and uses an optimization method to perform safe driving planning for intelligent vehicles. Background Art

[0002] With the development of autonomous driving technology, the safety and reliability of intelligent vehicles have received increasing attention. Traditional autonomous driving systems often make decisions based on fixed rules or simple models, and cannot fully consider complex traffic environments and obstacle perception information, so there are certain limitations. To address this problem, the present invention proposes a method for deducing autonomous driving risk situations based on a multimodal information fusion large model, which can more accurately predict the risk situation of the vehicle, thereby achieving safer driving planning. Summary of the invention

[0003] The present invention provides a method, system and storage medium for predicting the risk situation of autonomous driving based on multimodal information fusion and large model deduction. By combining multimodal information and large model deduction, the prediction and response of autonomous driving risks are realized. Based on the integration of multimodal information from multiple sensors, the method uses a large model to establish a prediction model and generate short-term and long-term prediction results. By evaluating the cost of each short-term plan and the corresponding long-term response plan, the one with the smallest total cost is selected as the optimal strategy to guide the safe driving of intelligent vehicles. The technical solution mainly includes three parts: (1) Multimodal information fusion: Integrate multimodal information from cameras, laser radars, and sensors to improve the accuracy and completeness of data. (2) Large model deduction: Use a large model to analyze spatiotemporal risk information and text prompt information to generate short-term and long-term prediction results. (3) Optimal strategy selection: By evaluating the cost of each short-term plan and the corresponding long-term response plan, the one with the smallest total cost is selected as the optimal strategy to guide the safe driving of intelligent vehicles.

[0004] The present invention proposes a method for predicting the risk situation of autonomous driving based on multimodal information fusion and large model deduction. The main steps of the method include:

[0005] Step 1: Obtain multimodal information: Obtain traffic environment perception and obstacle perception information through sensors such as cameras and lidar. Use RFN-Nest to transmit and aggregate visible light images and infrared images. Fuse data from multiple sensors to form dynamic spatiotemporal information of surround view. Use the neighbor cross attention mechanism to fuse the dynamic and static features of 2D-3D surround view images.

[0006] Step 2: Big model deduction: Understand and predict the action description, action proof and control signal of traffic objects through the big model. Generate intelligent text prompts, including situation understanding, reasoning, self-agent intention, prediction and scene type. Predict the semantic behavior of the self-vehicle and surrounding vehicles through the big model, and output the autonomous driving motion plan.

[0007] Step 3: Long- and short-term risk prediction and optimal strategy selection: Construct the input parameters of the large behavior prediction model, which include time horizon, bifurcation time, risk tolerance level, etc., as well as the model itself used to make predictions. Make short- and long-term intention predictions, predict the behavior of obstacles and overall behavior trends. Assess the probability and severity of risks, and calculate trajectory costs and belief updates. Select the final plan by evaluating trajectory costs, belief updates, and the optimal contingency plan.

[0008] Furthermore, the specific steps of establishing multimodal input of information in step 1 are:

[0009] S1.1 uses RFN-Nest to transfer and aggregate information between visible light images and infrared images.

[0010] S1.2 uses the neighbor cross attention mechanism to fuse the dynamic and static features of 2D-3D surround view images.

[0011] Furthermore, step S1.1 uses RFN-Nest to transfer and aggregate information between visible light images and infrared images, extracts multi-scale deep features from source images using convolution operations and maximum pooling operations, and outputs a fused multi-scale feature image through Transformer. Specifically, the steps are as follows:

[0012] In the first step, the feature extractor is responsible for extracting multi-level features from the input image. Multi-scale deep features are extracted from the source image using convolution operations and maximum pooling operations.

[0013] The input source image is I ir and I vi (I ir Indicates infrared image, I vi Represents a visible light image). By utilizing the maximum pooling operation in the encoder network, multi-scale deep features can be extracted from the source image. Compared with shallow features that contain more detailed information, deep features convey richer semantic information.

[0014] In the second step, the features from the visible light and infrared band images are fused into a single multi-scale feature map. The convolutional neural network (CNN) and the Transformer model are used to transfer and aggregate information between the visible light image and the infrared image. Among them, the CNN branch is specifically used to extract local feature representations, and the Transformer branch uses the Transformer block based on axial attention to capture the global context embedded in the input data.

[0015] At the same time, in order to better measure the quality of feature fusion, the loss function is redefined in this process, and the fusion loss function D fuse , including: pixel-level loss D pixel and structural similarity loss D ssim The specific expression is:

[0016] D fuse =l×D pixel +m×D ssim

[0017]

[0018] F=Decoder(FusionBlock(MultiScaleFeatures))

[0019] D ssim =1-SSIM(F,I)

[0020] Where λ and μ are weight coefficients, F is the reconstructed image after multi-scale feature fusion, I is the original image, is the Frobenius norm. D pixel Ensuring that the reconstructed image closely resembles the original input image at the individual pixel level imposes a constraint on the fidelity of pixel information during the reconstruction process. This constraint helps preserve fine-grained details and accuracy in the reconstructed image, ensuring that it retains the essential features of the input image at the granular level.

[0021] After fixing the encoder and decoder, RFN training is performed. The present invention proposes a new loss function D FRN ;

[0022] D RFN =D feature +dD detail

[0023] Among them, D detail and D feature They represent the background detail preservation loss function and the target feature enhancement loss function respectively, and the parameter δ represents the control D detail and D featureand handles orders of magnitude differences in the overall loss function.

[0024] D detail =[1-SSIM(I f ,I vi )] 2 +[1-SSIM(I f ,I ir )] 2

[0025]

[0026] Among them, I f Represents the fused image, which is visible light I vi and infrared input image I ir The fused output image, SSIM structural similarity index is an image quality evaluation index used to compare two images. M refers to the number of scales used for deep feature extraction, and and Respectively represent the fused image, the input visual band image and the input infrared band image. ξ m , vi and ir Represents the trade-off parameter for coordinating the size of the loss.

[0027] In the third step, the fused image is reconstructed through a decoder based on nested connections. The fused multi-scale deep features are fed into the decoder network for reconstructing the input image. The use of short cross-layer connections ensures the comprehensive utilization of multi-scale deep features in the image reconstruction process.

[0028] Further, step S1.2: adopting the neighbor cross attention mechanism to fuse the dynamic and static features of the 2D-3D surround view image, the specific implementation includes: aggregating the original point cloud in the 3D lidar point cloud sequence, and using the (relative) timestamp as an additional feature dimension to improve the detection performance. This setting compensates for the sparsity of the point cloud when using a single frame as the input for 3D object detection. The cross attention is effectively deployed to fuse the spatiotemporal semantic features from the input sequence with the precise positioning information from the current frame.

[0029] The dual-path architecture is designed to process the current point cloud and the aggregated point cloud separately, where the extracted features are gradually fused. The multi-frame branch is called the "dynamic branch" and the single-frame branch is called the "static branch" to highlight the rich motion information and accurate location information in each branch. The specific steps are as follows:

[0030] The first step is to initialize the features

[0031] The input dynamic branch BEV feature map is represented as Φ d, the static branch BEV feature map is represented by Φ S C, W, and H indicate the number of channels, width, and height of the dynamic branch BEV feature map, respectively. Aggregate neighbor features Before performing feature fusion, check whether the dimensions of the feature maps of the two branches are the same. If their dimensions are different, the dynamic branch feature map Φ d Perform additional convolutional layers to ensure that it reaches the same level as the static branch feature map Φ S Same dimensions.

[0032] The second step is feature fusion

[0033] Feature fusion is performed step by step through the Neighbor Cross Attention (NCA) module to obtain richer and more accurate environmental information. The specific expression is:

[0034] In the lth fusion module (l∈{1,2,…,N},N∈R), the input dynamic branch features and static branch features Use the neighbor cross attention mechanism module NCA respectively l and CNN blocks of CNN l After convolution processing, the output can be expressed as:

[0035]

[0036] in, The features of Specifically, we first use the convolutional layer to process the static feature map to obtain a sequence of m-dimensional feature vectors Φ s ∈R n×m , n represents the length of the feature sequence, m represents the dimension of each feature vector, and then U is obtained by linear projection s ∈R n×q ; The dynamic feature map is processed by the same method to obtain key J d ∈R n×q With value W d ∈R n×v , n represents the length of the dynamic feature sequence, q represents the dimension of the dynamic feature map after linear projection, v represents the dimension of the value after linear projection, and the cross attention of query i is E c i , the specific query mechanism is:

[0037]

[0038] in, To query the neighborhood of i of size k, To offset the position, residual connections and additional linear layers are used to process the output; at the same time, to enhance the static features, Φ is processed in the form of self-attention.s (queries, keys and values ​​are all generated by it), the output is concatenated with the output of the cross attention and input into the convolutional layer.

[0039] Step 3: Dynamic-static interaction

[0040] After convolution processing, the feature map becomes dense and rich in information. In order to fully integrate the features, an additional interaction module is introduced. Given a static feature map and a dynamic feature map Φ s ,Φ d ∈Φ C×W×H First, we concatenate along the channel dimension to obtain the concatenated feature F, and then use convolution to process Φ separately. s ,Φ d Get the static feature map Φ' s and dynamic feature map Φ' d , Φ' s , Φ' d The feature map Φ' after concatenation of F along the channel dimension c , then for Φ' s and Φ' d Through the convolution operation, two feature maps are generated respectively, and then respectively compared with Φ' c Concatenate and convolve to get the final output three-dimensional feature map (Where C is the number of channels, W is the width of the feature map, H is the height of the feature map, and the value at each position is a real number R.) The algorithm flow of this step ensures the effective fusion of dynamic and static features, providing more reliable input data for subsequent large model learning, thereby providing richer and more accurate support for the behavioral decision-making and planning of intelligent systems.

[0041] Step 4: Output updated features

[0042] The output feature map has a smaller size and is denser than the initial BEV feature. Through feature fusion and dynamic-static interaction, dynamic and static information can be preserved so that the intelligent system can better perceive and understand the environment and identify and respond to potential risks and dangerous situations in a timely manner.

[0043] Furthermore, in the process of building the prediction model in step 2, the large model is used to combine image information with language information, and assist in time series analysis to understand and predict the semantic behavior and scene behavior around the vehicle. The specific steps include:

[0044] S2.1 Use large models to understand the state of the traffic environment;

[0045] S2.2 Generate intelligent text prompts using large models;

[0046] S2.3 outputs the autonomous driving motion plan;

[0047] S2.4 Use the large model to predict the semantic behavior and risk situation of the vehicle and surrounding vehicles.

[0048] Furthermore, step S2.1 uses the large model to understand the traffic environment state and obtain the characteristics of traffic objects, as follows:

[0049] First, multiple frames of fused images and collected traffic information are input into the large model, and the images are converted into text information through language prompts, including action descriptions, action proofs and control signals of traffic objects.

[0050] After prompting, the large model observes the agent's self-state and outputs images as motion poses and simply encodes the natural language into a hotspot vector, which is specifically expressed as follows:

[0051] O-OneHotEncode(ObservationList)

[0052] Among them, the self-state O includes the historical trajectory of the vehicle and its current state, such as position, steering, speed and acceleration; OneHotEncode represents One-Hot encoding of the observation list, and the ObservationList observation list includes various sensor data of the vehicle, such as the position and speed of adjacent vehicles.

[0053] S2.2 Get text prompts, the specific implementation is as follows:

[0054] The prompt output provides a further understanding of traffic objects, such as situational understanding, reasoning, self-subject intention, trajectory prediction, and scene type, which can be specifically expressed as follows:

[0055] S-OneHotEncode(ScenarioList),

[0056] I'-OneHotEncode(IntentionList)

[0057] Among them, S represents environmental observation, IntentionList represents intention list;

[0058] Because the vehicle's intention is not unique, different weights will be assigned to the intention through the understanding of the big model, which can be expressed as follows:

[0059] weight i -length(IntentionInfoList)-i

[0060] Among them, weight iRepresents the weight assigned to the length of the intent list IntentionList. In the case of multiple intents, different intentions may be assigned different weights to reflect their importance in driving decisions.

[0061] S2.3, the specific implementation of outputting the autonomous driving motion plan is as follows:

[0062] The goal of autonomous driving motion planning based on a large model is to plan a feasible future driving trajectory T using the self-state O and the environment observation S as input. The process of motion planning using a large model F LLM It can be expressed as:

[0063] {T,R}=F LLM (K(O,S))

[0064] Specifically, the large model F LLM Output the feasible future driving trajectory T and the rules and constraints R that need to be followed. The planned trajectory T is represented as a set of waypoints with t time steps:

[0065] T={(x1,y1),...,(x t ,y t )}, T∈R k×2

[0066] Among them, (x i ,y i ) are 2D waypoint coordinates representing the expected position of the vehicle at time step i. The ego state O typically includes the historical trajectory of the vehicle and its current state, such as velocity and acceleration. The environment observation S contains the output of the perception and prediction system, for example, the detected object bounding boxes and their future motion.

[0067] Furthermore, step 3 performs long-term and short-term risk prediction and optimal strategy selection based on optimization methods, which is specifically implemented as follows:

[0068] The first step is to build the input of the behavior prediction model:

[0069] The specific content of the input includes: the status information M of each obstacle a,q,i (·), time range L, fork time k b , risk tolerance level ζ, reference path Z, prior belief κ for each intention, initial belief

[0070] Step 2: Long- and short-term risk measurement

[0071] risk It is defined as the combination of the probability of a collision and the severity of the collision, that is, the product of the two quantities, specifically expressed as:

[0072]

[0073] Among them, t represents the prediction stage, a represents obstacles, and q represents different directions. represents the probability of the planned trajectory colliding with any surrounding obstacles, The severity associated with potential collisions along the planning horizon at each planning time step t. The collision probability is calculated by estimating the overlap of the predicted pattern with the planned trajectory η of the ego vehicle in time and space, specifically expressed as:

[0074]

[0075] Among them, this is is a mixed Gaussian probability density function, which represents the collision probability density of the collision object a at the position (x, y) of time step t. The integration domain D is defined as a circle in two-dimensional space, whose center is located at the predicted vehicle posture At stage t within the prediction range, the radius r is the sum of the vehicle and obstacle radii, is in region D, The double integral of is used to calculate the collision probability.

[0076] The severity of a crash is defined as:

[0077]

[0078] Among them, mass v and mass a represent the masses of the ego vehicle and the obstacle vehicle, respectively, and and are their corresponding velocities, and θ is the collision angle. By properly adjusting and normalizing the parameters, the collision severity is scaled in the range of 0 to 1. A severity value of 0 corresponds to a collision that causes no damage, while a severity value of 1 indicates the highest level of damage.

[0079] In addition, the discounted opportunity constraint formula for obstacle a defines the risk, and the discounted risk at each stage t is specifically expressed as:

[0080]

[0081] Here, α is the discount factor.

[0082] After calculating the risk of each obstacle on the planned trajectory of the ego-vehicle, the predicted risk Λ at the current time step is defined as maximizing the risk of all obstacles a at all planning horizon stages,

[0083]

[0084] Among them, z0 represents the predicted trajectory, a represents the obstacle, and q represents different directions.

[0085] The maximum operator max ensures that the worst-case risk on the planned trajectory is below a threshold when When , we get the pruned path C(η,Z) is a trajectory cost function that evaluates the various costs of a trajectory η relative to the reference path Z. These costs include trajectory selection bias cost, speed cost bias, progress cost, and comfort cost.

[0086] Step 3: Calculate trajectory cost

[0087]

[0088] C=w1×deviation(f,Z)+w2×speed_deviation+w3×progress(f)+w4×comfort

[0089] in, is the trajectory cost, ComputeCost is a function of the computational cost, w and w i is the weight vector, is the trajectory point, and Z is the reference path. These costs include: The trajectory deviation cost deviation(f,Z) between the end point of the planned trajectory and the reference path: defined as c d =|dd ref |,c d is the trajectory deviation cost, d is the actual distance, and d ref is the reference distance; speed cost deviation speed_deviation: defined as c s is the speed cost deviation, is the actual speed, is the reference speed; progress cost progress(f): defined as represents the total travel distance along the reference path Γ, is the progress position, is the actual position; and the comfort cost comfort: is defined as c j is the comfort cost, is the mean square sum of the penalty longitudinal jolt, is the mean square sum of lateral turbulence, N is the total number of points, and is used as a comfort index. Through the above steps, we can finally find the emergency plan η that meets the conditions from the priority queue γ tb:T ,The priority queue is sorted by the cost of the trajectories from low to high ,so the optimal contingency plan will be at the beginning of the queue.

[0090] Step 4: Update your beliefs

[0091] Specifically, the Bayesian update rule is used to update the belief in the intent pattern λ according to the new observation and prediction model, which is specifically expressed as:

[0092]

[0093] Where κ is a possible intention, I(κ) is the belief held on the set κ of possible intentions, Represents the prior belief of the self-carrier on the intention input at time k. represents the updated posterior belief. represents the probability density function of the Gaussian mixture model, with mean μ k and covariance∑ k , and the state under observation

[0094] Among them, the target intention estimation function of the mixed Gaussian model is specifically expressed as:

[0095]

[0096] m a,t,i (x,y)=Ν(μ i (k),∑ i (k))

[0097] in, represents the target intention function in the Gaussian mixture model, φ i represents the weight of the i-th mixture component, m a,t,i (x, y) represents the probability density function of the i-th component, Ν(μ i (k),∑ i (k)) represents a normal distribution with mean μ i (k), the covariance is ∑ i (k).

[0098] This updating process takes into account the observation information at the current moment and the probability estimates of the prediction model to adjust the belief about the intention pattern.

[0099] The fifth step is to combine the final plan, update the probability distribution of the permutations, and evaluate the total cost of the entire plan. First, the combination will share the plan and the optimal emergency plan under different intention modes {η kb:T (κ1) best,..., η kb:T (κ |Z| ) best,...,} into a complete plan η:

[0100]

[0101] Then, the updated belief is used to adjust the probability distribution σ(γ j ) +

[0102]

[0103] This step ensures that the selection of different intention patterns in the final plan reflects the current belief distribution.

[0104] Then, the total cost of the entire plan is evaluated. This total cost usually takes into account the cost of shared plans and contingency plans, as well as other possible factors such as security and smoothness. It can be expressed as:

[0105]

[0106] stg j (η)≤I j ,j=1,...,n,

[0107] Finally, plan η is added to the final priority queue and sorted from lowest to highest total cost.

[0108]

[0109] Specifically, η * The optimal plan finally selected, represents the initial trajectory point in the optimal path, k b represents the b-th trajectory point, Indicates that from the kth b The trajectory from the trajectory point to the time T, where κ n Represents a sub-trajectory point.

[0110] Through these steps, the algorithm can generate a complete final plan under different intention modes, and evaluate the total cost of the plan based on the updated beliefs and probability distribution of the permutations and combinations, so as to select the optimal plan to add to the final priority queue.

[0111] The present invention also proposes a risk situation prediction system for autonomous driving vehicles based on multimodal information fusion and large model deduction, including a multimodal information acquisition module, a large model deduction module, and a long-term and short-term risk prediction module;

[0112] The multimodal information acquisition module implements the content of step 1;

[0113] The large model deduction module implements the content of step 2;

[0114] The long-term and short-term risk prediction module implements the content of step 3.

[0115] The present invention also includes a storage medium, which contains the program code of the above prediction method.

[0116] Beneficial effects of the present invention:

[0117] 1. The multimodal information fusion module in the present invention uses multimodal information from cameras, lidars, sensors, etc. to fully perceive vehicles, pedestrians, obstacles, etc. in the traffic environment. In this process, RFN-Nest technology transmits and aggregates visible light images and infrared images to form dynamic spatiotemporal information of surround view. Through the neighbor cross-attention mechanism, the dynamic and static features of 2D-3D surround view images are fused to ensure accurate perception of the surrounding environment. By integrating multi-source data, the present invention reduces the perceptual errors that may be caused by single modal information, improves the accuracy and completeness of information, and provides rich and comprehensive input data for subsequent large model deductions. The accuracy and completeness of the perceived data provide a more reliable basis for the prediction model, thereby improving the accuracy and stability of the prediction results. This helps to ensure that the autonomous driving system has a comprehensive understanding of the complex traffic environment and improves the system's adaptability in a dynamic environment.

[0118] 2. The large model deduction module in the present invention understands the state of the traffic environment by analyzing the actions, control signals and environmental information of traffic objects, and generates intelligent text prompts. By describing, proving and predicting the actions of traffic objects, the large model can generate short-term and long-term prediction results, covering the semantic behavior and risk situation of the vehicle and surrounding vehicles. Large model deduction plays a key role in the prediction model. Through multi-frame image and traffic information input, the large model can convert pictures into text information and reason and predict the behavior of traffic objects. This includes the understanding of the intentions and scene types of the self-subject and motion planning based on thinking chain reasoning and decision-making processes. Through large model deduction, the behavior of the vehicle and surrounding vehicles can be predicted in the short and long term. This prediction is not only based on historical data, but also combined with traffic environment and vehicle intentions. The intelligent reasoning ability and adaptability of the large model improve the prediction accuracy and provide reliable guidance for the safe driving of vehicles.

[0119] 3. The long-term and short-term risk prediction and optimal strategy selection module in the present invention performs short-term and long-term intention prediction by constructing the input of the behavior prediction model, including time range, bifurcation time, risk tolerance level, etc. The behavior of obstacles and overall behavior trends are predicted to provide a planning basis for the autonomous driving system. The system evaluates the probability and severity of collisions, and selects the optimal emergency plan by calculating trajectory costs and belief updates. This process ensures the safety of vehicle driving and reduces driving risks. Through the Bayesian update rule, the belief in the intention pattern is updated according to the new observation and prediction model to ensure more accurate predictions. Combined with the updated belief distribution and cost evaluation results, the optimal strategy is selected to ensure the safe driving of the vehicle in a complex traffic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 This is a structural diagram of the method for deducing the risk situation of autonomous driving based on multimodal information by using the fusion large model described in the present invention.

[0121] Figure 2 This is a schematic diagram of using RFN-Nest to perform information transmission and aggregation on visible light images and infrared images according to the present invention.

[0122] in:

[0123] (a) is a flowchart showing the use of RFN-Nest to transfer and aggregate information between visible light images and infrared images according to the present invention.

[0124] (b) is a diagram showing the network structure of the RFN-Nest encoder and decoder used in the present invention.

[0125] (c) is a diagram showing the network structure of the Transformer used in the present invention.

[0126] Figure 3 This is a flowchart of the present invention for fusing the dynamic and static features of 2D-3D surround images using the neighbor cross attention mechanism.

[0127] in:

[0128] (a) represents the overall architecture of fusing the dynamic and static features of 2D-3D surround view images using the neighbor cross attention mechanism as described in the present invention.

[0129] (b) represents the network structure using the neighbor cross attention mechanism described in the present invention.

[0130] (c) represents the network structure of dynamic and static interactions described in the present invention.

[0131] Figure 4 A flowchart of using a large model to predict long-term and short-term risks according to the present invention;

[0132] Figure 5 This is a scene example diagram of the method for deducing the risk situation of autonomous driving based on multimodal information by using the fusion big model of the present invention;

[0133] Figure 6 This is a diagram showing the results of the fusion model of the present invention inferring the risk situation of autonomous driving based on multimodal information.

[0134] (a) represents the real-time dynamic interface of the autonomous driving risk situation system deduced by the fusion big model described in the present invention based on multimodal information.

[0135] (b) represents the language prompt interface of the autonomous driving risk situation system deduced by the fusion large model described in the present invention based on multimodal information.

[0136] (c) represents the risk perception interface of the autonomous driving risk situation system deduced by the fusion big model described in the present invention based on multimodal information.

[0137] (d) represents the optimal trajectory prompt interface of the autonomous driving risk situation system deduced by the fusion large model described in the present invention based on multimodal information. DETAILED DESCRIPTION

[0138] The method for predicting risk situations of autonomous driving based on multimodal information fusion and large model deduction proposed in the present invention first collects multimodal traffic environment perception and obstacle perception information from cameras, lidars and sensors to construct the original input data of the model; secondly, through RFN-Nest and neighbor cross-attention mechanism, the visible light image and infrared image are fused into dynamic spatiotemporal information of surround view to obtain the optimal data fusion results under different conditions; then, the large model is used to analyze the actions, situational understanding and intentions of traffic objects, a behavior prediction model is constructed, and the semantic behavior and risk situation of the vehicle and surrounding vehicles are predicted; then, through the long-term and short-term risk prediction and optimal strategy selection model, the short-term and long-term risk situations are predicted, and the optimal response plan is selected; finally, the Bayesian update rule is used to update the belief of the intention pattern, adjust the prediction results and response strategies, and finally realize the safe driving of intelligent vehicles.

[0139] The present invention will be further described below in conjunction with the accompanying drawings.

[0140] like Figure 1 As shown, the autonomous driving risk situation prediction system based on multimodal information fusion large model deduction described in the present invention includes a data acquisition device, a data preprocessing module, a prediction model and a visualization module.

[0141] The data acquisition device includes sensors and detection equipment for collecting road and environmental information around the vehicle and the vehicle's motion data. Specifically, the collected vehicle motion data includes vehicle position and speed, acceleration and angular velocity, path and trajectory, direction and heading angle, timestamp, motion state, etc.; the original road and environmental data includes geographic data and road network data. Geographic data describes the data of geographical features on the earth's surface, including longitude, latitude, etc., and road network data includes the topological structure and attribute information of the road.

[0142] The main function of the visualization module is to present the prediction results and analysis data in an intuitive and easy-to-understand way. Through graphical interfaces, charts, etc., it displays the risk sources and potential dangers around the vehicle in real time, and displays the risk prediction results, including the probability, severity and impact range of the risk. It also displays the predicted trajectory of the vehicle and the selected optimal driving path, thereby helping users understand the decision-making process of the autonomous driving system.

[0143] The data preprocessing module is used to realize multi-scale image fusion and feature extraction, unified representation of dynamic-static information, spatiotemporal correlation data encoding and fusion, etc. Through the processing and fusion of different sensor data, the integration and unified representation of multi-modal information are realized, which provides basic data and feature representation for the subsequent intelligent vehicle driving risk situation deduction. The following is the specific content of data preprocessing:

[0144] Step 1) Construct the multimodal information input of the model: First, the visible light image and infrared image are fused to improve the accuracy and robustness of target detection and recognition, such as Figure 2 (a); Secondly, the fused 2D image is used to form a static surround image, which is then fused with the 3D radar image to form all-round, time-space related environmental perception information, enriching the system's dynamic perception of the environment, thereby providing richer and more accurate input data for large model learning. The specific process is as follows:

[0145] In the first step, the feature extractor is responsible for extracting multi-level features from the input image. Multi-scale deep features are extracted from the source image using convolution operations and maximum pooling operations.

[0146] like Figure 2 (b), specifically, the visible light image and the infrared image are fused, and feature extraction is performed using an encoder network, where “k×k, (in, out)” means that the convolution kernel size is k×k, the input channel is in, and the output channel is out. ir and I vi They represent the infrared image and visible light image in the source image respectively, and O represents the output of RFN-Nest, that is, the fused image.

[0147] In the second step, the features from the visible light and infrared band images are fused into a single multi-scale feature map. Figure 2 (c) Specifically, the fuser uses a convolutional neural network (CNN) and a Transformer network to transfer and aggregate information between visible light images and infrared images. The CNN branch is specifically used to extract local feature representations, and the Transformer branch uses an axial attention-based Transformer block to capture the global context embedded in the input data. In this process, the fusion loss function D fuse It is expressed as:

[0148] D fuse =l×D pixel +m×D ssim

[0149]

[0150] D ssim =1-SSIM(F,I)

[0151] F=Decoder(FusionBlock(MultiScaleFeatures))

[0152] Where D pixel and D ssim represents pixel loss and structural similarity loss, F is the reconstructed image after multi-scale feature fusion, I is the original image, λ and μ are weight coefficients, is the Frobenius norm.

[0153] D pixel Ensuring that the reconstructed image closely resembles the original input image at the individual pixel level imposes a constraint on the fidelity of pixel information during the reconstruction process. This constraint helps preserve fine-grained details and accuracy in the reconstructed image, ensuring that it retains the essential features of the input image at the granular level.

[0154] After fixing the encoder and decoder, RFN training is performed. The present invention proposes a new loss function D FRN

[0155] D FRN =D feature +dD detail

[0156] Among them, D detail and D feature They represent the background detail preservation loss function and the target feature enhancement loss function respectively. The parameter δ represents the control D detail and D featureand handles orders of magnitude differences in the overall loss function.

[0157] D detail =[1-SSIM(I f ,I vi )] 2 +[1-SSIM(I f ,I ir )] 2

[0158]

[0159] Among them, I f Represents the fused image, which is visible light I vi and infrared input image I ir The fused output image, SSIM structural similarity index is an image quality evaluation index used to compare two images. M refers to the number of scales used for deep feature extraction, and and Respectively represent the fused image, the input visual band image and the input infrared band image. ξ m , vi and ir Represents the trade-off parameter for coordinating the size of the loss.

[0160] In the third step, the fused image is reconstructed through a decoder based on nested connections. The fused multi-scale deep features are fed into the decoder network for reconstructing the input image. The use of short cross-layer connections ensures the comprehensive utilization of multi-scale deep features in the image reconstruction process. Specifically, the fusion network and the decoding process are as follows: Figure 2 (b)

[0161]

[0162] The input is the multi-scale extracted features, the subscript of I: ir is the feature of the infrared image, and vi is the feature of the visible light image. The two features are used as the input of convolution layer 1 and convolution layer 2 respectively, and also as the input of convolution layer 6. The output of convolution layer 1 and convolution layer 2 is connected and used as the input of convolution layer 3. In the decoder, the fused multi-scale deep features are fed into the decoder network The image is reconstructed step by step through two convolutional layers. In each row, these blocks are connected by short connections, similar to the dense block architecture. This short connection method allows the features to be effectively transferred and comprehensively utilized between different levels, enhancing the quality of the reconstructed image. The cross-layer links in the decoder connect the multi-scale deep features, ensuring that the features of all scales can be fully utilized during the decoding process. The output of the decoder network is a fused image, which is finally generated after the multi-scale feature reconstruction process.

[0163] Step 2): Use the neighbor cross attention mechanism to fuse the dynamic and static features of the 2D-3D surround image, such as Figure 3 (a)(b)(c) is shown in the figure. The specific steps are as follows:

[0164] The first step is to initialize the features

[0165] The input dynamic branch BEV feature map is represented as Φ d , the static branch BEV feature map is represented by Φ S C, W, and H indicate the number of channels, width, and height of the dynamic branch BEV feature map, respectively. Aggregate neighbor features Before performing feature fusion, check whether the dimensions of the feature maps of the two branches are the same. If their dimensions are different, the dynamic branch feature map Φ d Perform additional convolutional layers to ensure that it reaches the same level as the static branch feature map Φ S Same dimensions.

[0166] The second step is feature fusion

[0167] like Figure 3 (b), feature fusion is performed step by step through the neighbor cross attention mechanism module, which is specifically expressed as:

[0168] In the lth fusion module (l∈{1,2,…,N},N∈R), the input dynamic branch features and static branch features Use the neighbor cross attention mechanism module NCA respectively l and CNN block CNN l After processing, the output can be expressed as:

[0169] Φ s l+1 =CNN l (NCA l (Φ d l ,Φ s l ))

[0170] Specifically, The features of are used as queries, according to The feature generation keys and values ​​of Φ s l+1 is the output feature of the static branch in the l-th layer fusion module. Since the position of the same object in the two feature maps should be close, it is more important to use local information for cross attention, which can also reduce the calculation. Specifically, the convolutional layer is first used to split the static feature map into tokens to obtain a sequence of feature vectors Φ s ∈R n×m, n represents the length of the feature sequence, and m represents the dimension of each feature vector. Then, U is obtained by linear projection. s ∈R n×q , n represents the length of the feature sequence, q represents the dimension of the feature vector after linear projection; similarly, the key J is obtained by splitting into tokens and linearly projecting the dynamic feature map d ∈R n×q With value W d ∈R n×v , n represents the length of the dynamic feature sequence, q represents the dimension of the dynamic feature map after linear projection, and v represents the dimension of the value after linear projection. c i , its specific query mechanism is:

[0171]

[0172] in, To query the neighborhood of i of size k, is the position offset, The output of the linear layer is further processed using residual connections and additional linear layers. At the same time, to enhance static features, a similar method is used to process Φ in the form of self-attention. s (queries, keys and values ​​are all generated by it), the output is concatenated with the output of the cross attention and input into the convolutional layer.

[0173] Step 3: Dynamic-static interaction

[0174] After the concatenated feature map is processed by convolution and neighbor cross attention mechanism, the feature map becomes dense and rich in information. Figure 3 As shown in (c), in order to fully integrate the features, an additional interaction module is introduced. Given a static feature map and a dynamic feature map Φ s ,Φ d ∈Φ C×W×H First, concatenate along the channel dimension to get F, and use convolution to process them separately to get the static feature map Φ' s , dynamic feature map Φ' d , the feature map Φ' after splicing along the channel dimension c , and then generate two feature maps through convolution operation, which are related to Φ' c Concatenate and convolve to get the final output three-dimensional feature map (where C is the number of channels, W is the width of the feature map, H is the height of the feature map, and the value at each position is a real number R) and is input into the detection head.

[0175] Furthermore, the prediction model: in the process of predicting the model, the large model is used to combine the image information with the language information, and assist in time series analysis, to understand and predict the semantic behavior and scene behavior around the vehicle, such as Figure 4 , Figure 5 As shown, the specific steps include:

[0176] S2.1 Use large models to understand the state of the traffic environment;

[0177] S2.2 Generate intelligent text prompts using large models;

[0178] S2.3 outputs the autonomous driving motion plan;

[0179] S2.4 uses a large model to predict the semantic behavior and long-term and short-term risks of the vehicle and surrounding vehicles, and selects the optimal strategy.

[0180] Furthermore, step S2.1 uses the large model to understand the traffic environment state and obtain the characteristics of traffic objects. The specific implementation includes:

[0181] First, the multi-frame fusion images and the collected traffic information are input into the large model, and the images are converted into text information through language prompts, including the action description, action proof and control signal of the traffic object. The example is described as follows:

[0182] Question 1: The current speed of the vehicle is 2.15 m / s. What is the current movement of the vehicle?

[0183] Hint 1: The car is about to turn right.

[0184] Question 2: Why does the vehicle behave this way?

[0185] Tip 2: Because the right turn indicator ahead is green.

[0186] Question 3: What is the turning angle of the vehicle?

[0187] Tip 3: 0.35.

[0188] Question 4: Where is the current location of the vehicle?

[0189] Hint 3: The vehicle is located at (12.05,6.12).

[0190] After prompting, the large model observes the agent's self-state and outputs images as motion poses and simply encodes the natural language into a hotspot vector, which is specifically expressed as follows:

[0191] O-OneHotEncode(ObservationList)

[0192] Among them, the self-state O includes the historical trajectory of the vehicle and its current state, such as position, steering, speed and acceleration; OneHotEncode represents One-Hot encoding of the observation list, and the ObservationList observation list includes various sensor data of the vehicle, such as the position and speed of adjacent vehicles.

[0193] The specific implementation of generating text prompts using the large model in S2.2 is as follows:

[0194] The prompt output provides a further understanding of the traffic object, such as situational understanding, reasoning, the self-agent's intention, trajectory prediction results, and scene type. The example is described as follows:

[0195] Question 1: What are the positions and velocities of nearby vehicles?

[0196] Hint 1: Neighboring vehicle 1 (10.05, 6.12), speed 1.98 m / s: Neighboring vehicle 2 (9.05, 5.00), speed 2.40 m / s, pedestrian (None, None), speed None.

[0197] Question 2: Which lane is the vehicle in?

[0198] Tip 2: The vehicle is in the through / right turn lane.

[0199] Output scene information and intentions through prompts, specifically expressed as:

[0200] S-OneHotEncode(ScenarioList),

[0201] I'-OneHotEncode(IntentionList)

[0202] Among them, S represents the description of the traffic scene, OneHotEncode represents the One-Hot encoding of the observation scenario list, ScenarioList is the observation scenario list, including various sensor data of the vehicle, such as the position and speed of adjacent vehicles; I' represents the description of the vehicle intention, OneHotEncode represents the One-Hot encoding of the vehicle intention list, and IntentionList includes steering, acceleration, etc.

[0203] Because the vehicle's intention is not unique, different weights will be assigned to the intention through the understanding of the big model, which can be expressed as follows:

[0204] weight i -length(IntentionInfoList)-i

[0205] Among them, weight i Represents the weight assigned to the length of the IntentionList. In the case of multiple intentions, different intentions may be assigned different weights to reflect their importance in driving decisions.

[0206] In S2.3, the specific implementation of outputting the autonomous driving motion plan is as follows:

[0207] The goal of autonomous driving motion planning based on a large model is to use the self-state O and the environmental observation S as input, and plan a feasible future driving trajectory T through the processing function K. The motion planning process can be expressed as:

[0208] {T,R}=F LLM (K(O,S))

[0209] Specifically, the large model F LLM Output the feasible future driving trajectory T and the rules and constraints R that need to be followed. The planned trajectory T is represented as a set of waypoints with t time steps:

[0210] T={(x1,y1),...,(x t ,y t )}, T∈R t×2

[0211] Where (x i ,y i ) are 2D waypoint coordinates representing the expected position of the vehicle at time step i. The ego state O typically includes the historical trajectory of the vehicle and its current state, such as velocity and acceleration. The environment observation S contains the output of the perception and prediction system, for example, the detected object bounding boxes and their future motion.

[0212] S2.4, as Figure 4 As shown in the figure, the specific implementation of long-term and short-term risk prediction and optimal strategy selection based on the optimization method is as follows:

[0213] The first step is to build the input of the large model:

[0214] The specific content of the input includes: the status information M of each obstacle a,q,t (·), time range L, fork time k b , risk tolerance level ζ, reference path Z, prior belief κ for each intention, initial belief

[0215] Step 2: Long- and short-term risk measurement

[0216] Risk is defined as the combination of the probability of a collision and the severity of the collision, that is, the product of the two quantities, specifically expressed as:

[0217]

[0218] Among them, t represents the prediction stage, a represents obstacles, and q represents different directions. represents the probability of the planned trajectory colliding with any surrounding obstacles, The severity associated with potential collisions along the planning horizon at each planning time step t. The collision probability is calculated by estimating the overlap of the predicted pattern with the planned trajectory η of the ego vehicle in time and space, specifically expressed as:

[0219]

[0220] Among them, this is is a mixed Gaussian probability density function, which represents the collision probability density of the collision object a at the position (x, y) of time step t. The integration domain D is defined as a circle in two-dimensional space, whose center is located at the predicted vehicle posture At stage t within the prediction range, the radius r is the sum of the vehicle and obstacle radii, is in region D, The double integral of is used to calculate the collision probability.

[0221] The crash severity levels are defined as:

[0222]

[0223] Among them, mass v and mass a represent the masses of the ego vehicle and the obstacle vehicle, respectively, and and are the corresponding velocities of the ego vehicle and the obstacle vehicle, and θ is the collision angle. By properly adjusting and normalizing the parameters, the collision severity is scaled in the range of 0 to 1. A severity value of 0 corresponds to a collision that causes no damage, while a severity value of 1 indicates the highest level of damage.

[0224] In addition, the discounted opportunity constraint formula for obstacle a defines the risk, and the discounted risk at each stage t is specifically expressed as:

[0225]

[0226] Here, α is the discount factor.

[0227] After calculating the risk of each obstacle on the planned trajectory of the ego-vehicle, the predicted risk Λ at the current time step is defined as maximizing the risk of all obstacles a at all planning horizon stages,

[0228]

[0229] Among them, z0 represents the predicted trajectory.

[0230] The maximum operator max ensures that the worst-case risk on the planned trajectory is below a threshold when Get the trimmed trajectory path C(η,Z) is a trajectory cost function that evaluates the various costs of a trajectory η relative to the reference path Z. These costs include trajectory selection bias cost, speed cost bias, progress cost, and comfort cost.

[0231] Step 3: Calculate trajectory cost

[0232]

[0233] in, is the trajectory cost, ComputeCost is the function of the computational cost, w is the weight vector, is the trajectory point and Z is the reference path.

[0234] Specifically, the cost function contains sub-costs focusing on different aspects of program performance, such as safety, passenger comfort, progress, and tracking, and is expressed as:

[0235] C=w1×deviation(f,Z)+w2×speed_deviation+w3×progress(f)+w4×comfort

[0236] Among them, C represents the cost function, which is the abbreviation of the cost function ComputeCost, and the weight vector w i ∈R + ,i=1,2,3,4,representing the weight associated with each cost item. These costs include,the trajectory deviation cost deviation(f,Z) between the end point of the planned trajectory and the reference path: c d =|dd ref |,c d is the trajectory deviation and cost, d is the actual distance; speed cost deviation c s is the speed cost deviation, is the actual speed, is the reference speed; the progress cost represents the total travel distance along the reference path Γ, is the progress position, is the actual location; and the comfort cost Penalized longitudinal jolt mean square sum and the mean square sum of lateral jolt N is the total number of points, which is used as a comfort index. Through the above steps, we can finally find the emergency plan η that meets the conditions from the priority queue γ. tb :T, the priority queue is sorted by the cost of the trajectories from low to high, so the optimal contingency plan will be at the beginning of the queue.

[0237] Step 4: Update your beliefs

[0238] Specifically, the Bayesian update rule is used to update the belief in the intent pattern λ according to the new observation and prediction model, which is specifically expressed as:

[0239]

[0240] Where κ is a possible intention, I(κ) is the belief held on the set κ of possible intentions, Represents the prior belief of the self-carrier on the intention input at time k. represents the posterior probability. represents the probability density function of the Gaussian mixture model, with mean μ k and covariance∑ k , and the state under observation

[0241] Among them, the target intention estimation function of the mixed Gaussian model is specifically expressed as:

[0242]

[0243] m a,t,i (x,y)=Ν(μ i (k),∑ i (k))

[0244] in, represents the target intention function in the Gaussian mixture model, φ i represents the weight of the i-th mixture component, m a,t,i (x, y) represents the probability density function of the i-th component, Ν(μ i (k),∑ i (k)) represents a normal distribution with mean μ i (k), the covariance is ∑ i (k).

[0245] This updating process takes into account the observation information at the current moment and the probability estimates of the prediction model to adjust the belief about the intention pattern.

[0246] The fourth step is to combine the final plan, update the probability distribution of the permutations, and evaluate the total cost of the entire plan. and the optimal contingency plan {η under different intention modes λ kb:T(κ1) best,..., η kb:T (κ |Z| ) best,...,} into a complete plan η:

[0247]

[0248] Among them, η represents the final combination plan, Indicates the optimal trajectory points under different intention modes.

[0249] Then, using the updated belief To adjust the probability distribution of the combination of different intention patterns σ(γ j ) +

[0250]

[0251] in, Indicates updating the permutations and combinations according to the belief distribution at the current time.

[0252] This step ensures that the selection of different intention patterns in the final plan reflects the current belief distribution.

[0253] Then, the total cost of the entire plan is evaluated. This total cost usually takes into account the cost of shared plans and contingency plans, as well as other possible factors such as security and smoothness. It can be expressed as:

[0254]

[0255] stg j (η)≤I j ,j=1,...,n,

[0256] in, is the initial trajectory point The cost function, σ(γ j ) + is the probability distribution of the permutations and combinations of intention patterns, is the cost function of the intent pattern, g j (η) is the state constraint function, I j is the threshold of the constraint.

[0257] Finally, the combined plan η is added to the final priority queue γ final and sort them from low to high in terms of total cost.

[0258]

[0259] Specifically, η * The optimal plan finally selected, represents the initial trajectory point in the optimal path, k b represents the b-th trajectory point, Indicates that from the kth b The trajectory between trajectory points to T time, where κ n Represents a sub-trajectory point.

[0260] Through these steps, the algorithm can generate a complete final plan under different intention modes, and evaluate the total cost of the plan based on the updated beliefs and probability distribution of the permutations and combinations, so as to select the optimal plan to add to the final priority queue.

[0261] Figure 6 (a), (b), (c), and (d) are the display diagrams of the long-term and short-term driving risk situation deduction results of the large-scale model of the intelligent vehicle described in this application. It should be noted that this diagram is only an example of an interface diagram for the long-term and short-term driving risk situation deduction of the intelligent vehicle of the large-scale model, and only contains necessary functions. It can be improved according to specific needs in the future, and it is not a constraint condition of this application.

[0262] Reference Figure 6 (a), the interface consists of four parts. Frame ① shows the function interface of long-term and short-term risk prediction interface. Other function interfaces are not explained here and can be further improved in the future; Frame ② shows the current signal strength and time; Frame ③ shows the four display interface buttons of the long-term and short-term risk prediction system: "Real-time dynamics", "Language prompts", "Risk perception", and "Optimal trajectory". By selecting different function buttons, you can view specific display information; Frame ④ displays the specific information of the function button selected in Frame ③.

[0263] In detail, the "Real-time Dynamics" interface (refer to Figure 6 (a) It can fully display the current road environment of the autonomous driving vehicle and the specific location of the surrounding moving objects, so that the driver can perceive and understand the surrounding environment information in a specific and comprehensive manner; the "language prompt" interface (refer to Figure 6 (b) shows the big model’s understanding of the vehicle and its surroundings, which can guide the driver’s behavior and is crucial for risk perception; the “risk perception” interface (refer to Figure 6 (c)) shows the trajectories of surrounding high-collision risk objects and a large number of short-term risks for the vehicle; the “optimal trajectory” interface (refer to Figure 6 (d)) provides the driver with safe driving solutions in different scenarios by analyzing the different intentions of the predicted surrounding vehicles.

[0264] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A risk situation prediction method for autonomous driving vehicles based on multimodal information fusion and large model deduction, characterized in that: The steps include: Step S1: obtaining multimodal information: obtaining traffic environment perception and obstacle perception information; Step S2: large model deduction: using the large model to understand the acquired multimodal information, generating intelligent text prompts, and outputting motion planning trajectories through the large model; The specific implementation of S2 includes the following: S2.1, obtain traffic object characteristics: First, multiple frames of dynamic and static fusion images and collected traffic information are input into the large model, and the images are converted into text information through language prompts, including the action description, action proof and control signal of the traffic object; After prompting, the large model observes the agent's self-state and outputs the image as a motion posture and encodes the natural language into a hotspot vector, which is specifically expressed as follows: O-OneHotEncode(ObservationList) Among them, the self-state O includes the historical trajectory of the vehicle and its current state; OneHotEncode represents the One-Hot encoding of the observation list, and the ObservationList observation list includes various sensor data of the vehicle; S2.2, get text prompt: The further understanding of traffic objects through prompt output is specifically expressed as follows: S-OneHotEncode(ScenarioList), I-OneHotEncode(IntentionList) Among them, S represents the description of the traffic scene, OneHotEncode represents the One-Hot encoding of the observation scene list, and ScenarioList is the observation scene list, including various sensor data of the vehicle; I represents the description of the vehicle intention, OneHotEncode represents the One-Hot encoding of the vehicle intention list, and the IntentionList intention list includes steering, acceleration, etc.; As vehicle intentions are not unique, different weights are assigned to intentions through the understanding of the big model, which is specifically expressed as follows: weight i -length(IntentionInfoList)-i Among them, weight i Represents the weight assigned to the length of the intention list IntentionList. In the case of multiple intentions, different intentions are assigned different weights to reflect their importance in driving decisions; S2.3, autonomous driving motion planning output: The goal of autonomous driving motion planning based on a large model is to use the self-state O and the environmental observation S as input, and plan a feasible future driving trajectory T through the processing function K. The motion planning process is expressed as: {T,R}=F LLM (K(O,S)) F LLM is a large model, T is the output of the feasible driving trajectory in the future, and R is the rules and constraints that need to be followed; Represent the driving trajectory T as a set of waypoints of t time steps: T={(x 1, y1),...,(x t, y t )},T∈R t×2 Where (x i, y i ) are the 2D waypoint coordinates representing the expected position of the vehicle at time step i; Step S3: Long-term and short-term risk prediction: construct the input of the behavior prediction model, perform short-term and long-term intention prediction, and predict the behavior of obstacles and the overall behavior trend.

2. The method for predicting risk situation of an autonomous driving vehicle according to claim 1, characterized in that: The implementation of step S1 includes the following: S1.1 uses RFN-Nest to transfer and aggregate the multimodal information obtained by cameras, lidars, sensors, etc. to visible light images and infrared images, extracts multi-scale deep features from the source images using convolution operations and maximum pooling operations, and outputs the fused multi-scale feature images through Transformer; S1.2 fuses data from multiple sensors to form dynamic spatiotemporal information of surround view, and uses the neighbor cross attention mechanism to fuse the dynamic and static features of 2D-3D surround view images.

3. The method for predicting risk situation of an autonomous driving vehicle according to claim 2, characterized in that: The S1.2 specifically includes the following steps: S1.2.1, Initialization Features The input dynamic branch BEV feature map is represented as Φ d , the static branch BEV feature map is represented by Φ S , C, W and H represent the number of channels, width and height of the dynamic branch BEV feature map respectively. Before performing feature fusion, the aggregated neighbor features check whether the dimensions of the feature maps of the two branches are the same. If their dimensions are different, the dynamic branch feature map Φ d Perform additional convolutional layers to ensure that it reaches the same level as the static branch feature map v S Same dimensions; S1.2.2, Feature Fusion Feature fusion is performed step by step through the neighbor cross attention mechanism (NCA) to obtain rich and accurate environment information, which is specifically stated as follows: In the lth fusion module, l∈{1,2,…,N}, the input dynamic branch features and static branch features Use the neighbor cross attention mechanism module NCA respectively l and CNN block CNN l After processing, the output is expressed as: F s l+1 =CNN l (NCA l (F d l ,F s l )) Specifically, As the query, we use The features of U are used to generate keys and values. First, the static feature map is tokenized using a convolutional layer to obtain a sequence of m-dimensional feature vectors, and then U is obtained through linear projection. s ∈R n×q ; Similarly, the key J is obtained by tokenizing and linearly projecting the dynamic feature map d ∈R n×q With value W d ∈R n×v , the cross attention E of query i c i , the specific query mechanism is: in, To query the neighborhood of i of size k, is the position offset, To enhance static features, Φ is processed in the form of self-attention s , queries, keys and values ​​are generated by it, and the output is concatenated with the output of the cross attention and input into the convolutional layer; S1.2.3, Dynamic-static interaction In order to fully integrate the features, an additional interaction module is introduced; given the static feature map and the dynamic feature map Φ s ,Φ d ∈Φ C×W×H First, we concatenate along the channel dimension to obtain the concatenated feature F, and then use convolution to process Φ separately. s ,Φ d Get the static feature map Φ' s and dynamic feature map Φ' d , Φ' s , Φ' d The feature map Φ' after concatenation of F along the channel dimension c , then for Φ' s and Φ' d Through the convolution operation, two feature maps are generated respectively, and then combined with Φ' c The final output three-dimensional feature map is obtained by splicing and convolution. Among them, C is the number of channels, W is the width of the feature map, H is the height of the feature map, and the value at each position is a real number R.

4. The method for predicting risk situation of an autonomous driving vehicle according to claim 1, characterized in that: The specific implementation of S3 includes the following: S3.1, construct the input of the large model: The specific input content includes: the prediction model M of each obstacle a,q,i (·), time range L, fork time k b , risk tolerance level ζ, reference path Z, prior belief κ for each intention, initial belief S3.2, Long- and Short-term Risk Measurement Risk is defined as the combination of the probability of a collision and the severity of the collision, that is, the product of the two quantities, specifically expressed as: Among them, t represents the prediction stage, a represents obstacles, and q represents different directions. represents the probability of the planned trajectory colliding with any surrounding obstacles, Severity levels associated with potential collisions along the planning horizon at each planning time step t; The probability of a collision is calculated by estimating the overlap of the predicted pattern and the ego vehicle’s planned trajectory η in time and space, which can be expressed as: Among them, this is is a mixed Gaussian probability density function, which represents the collision probability density of the collision object a at the position (x, y) of the time step t. The integration domain D is defined as a circle, whose center is located at the predicted vehicle posture in two-dimensional space. At stage t within the prediction range, the radius r is the sum of the vehicle and obstacle radii, is in region D The double integral of is used to calculate the collision probability; The severity of a crash is defined as: Among them, mass v and mass a Represent the mass of the self-vehicle and the obstacle vehicle, respectively, and speed v and speed a are their corresponding velocities, and θ is the collision angle; by properly adjusting and normalizing the parameters, the collision severity is scaled in the range of 0 to 1, where a severity value of 0 corresponds to a collision that causes no damage, and a severity value of 1 indicates the highest level of damage; In addition, the discounted opportunity constraint formula for obstacle a defines the risk, and the discounted risk at each stage t is specifically expressed as: Where α is the discount factor; After calculating the risk of each obstacle on the ego vehicle’s planned trajectory, the predicted risk Λ at the current time step is defined as maximizing the risk of all obstacles a at all planning horizon stages: The maximum operator max ensures that the worst-case risk on the planned trajectory is below a threshold Finally, the trimmed trajectory path is obtained C(η,Z) is a trajectory cost function that is used to evaluate the various costs of a trajectory η relative to a reference path Z.

5. The method for predicting risk situation of an autonomous driving vehicle according to claim 1, characterized in that: The specific implementation of S3 also includes: S3.3, Calculate trajectory cost in, is the trajectory cost, ComputeCost is the function of the computational cost, w is the weight vector, is the trajectory point, Z is the reference path; The cost function contains sub-costs focusing on different aspects of program performance, including safety, passenger comfort, progress, and tracking, and is expressed as: C=w1·deviation(η,Z)+w2·speed_deviation+w3·progress(η)+w4·comfort Among them, C represents the cost function, which is the abbreviation of the cost function ComputeCost, and the weight vector w i ∈R + ,i=1,2,3,4, represents the weight associated with each cost item, these costs include, the trajectory deviation cost deviation(η,Z) between the end point of the planned trajectory and the reference path: c d =|dd ref |,c d is the trajectory deviation and cost, d is the actual distance; speed cost deviation speed_deviation: c s is the speed cost deviation, is the actual speed, is the reference speed; the progress cost represents the total travel distance along the reference path Γ, is the progress position, is the actual location; and the comfort cost comfort: Penalized longitudinal jolt mean square sum and the mean square of lateral jitter d t ”' 2 , N is the total number of points, used as a comfort index; S3.4, Belief Update Use the Bayesian update rule to update the belief about the intent pattern λ based on the new observations and prediction model, which is specifically stated as: where κ is the possible intention, represents the prior belief of the self-carrier about the intention input at time k, represents the posterior belief, represents the probability density function of the Gaussian mixture model, with mean μ k , covariance∑ k And the state under observation Among them, the Gaussian mixture model performs target intention estimation function as follows: m a,t,i (x,y)=N(μ i (k),∑ i (k)) in, represents the target intention function in the Gaussian mixture model, φ i represents the weight of the i-th mixture component, m a,t,i (x, y) represents the probability density function of the i-th component, N(μ i (k),∑ i (k)) represents a normal distribution with mean μ i (k), the covariance is ∑ i (k); S3.5, assemble the final plan, update the probability distribution of the permutations and combinations, evaluate the total cost of the entire plan, and select the optimal plan.

6. The method for predicting risk situation of an autonomous driving vehicle according to claim 5, characterized in that: The S3.5 is specifically implemented as follows: S3.5.1, Sharing Plan and the optimal emergency plan under different intention modes {η kb:T (κ1) best,..., η kb:T (κ |Z| ) best,..., } into a complete plan η: S3.5.2, Using Updated Beliefs To adjust the probability distribution of the combination of different intention patterns σ(γ j ) + , ensuring that the selection of different intention patterns in the final plan reflects the current belief distribution; S3.5.3, Evaluate the total cost of the entire program; S3.5.4, sort from low to high according to the total cost to obtain the final priority queue.

7. An autonomous driving vehicle risk situation prediction system based on multimodal information fusion and large model deduction, characterized in that: It includes multimodal information acquisition module, large model deduction module and long-term and short-term risk prediction module; The multimodal information acquisition module implements the content of step S1 described in claim 1; The large model deduction module implements the content of step S2 described in claim 1; The long-term and short-term risk prediction module implements the content of step S3 described in claim 1.

8. A storage medium, characterized in that: The program code of the prediction method according to claim 1 is stored therein.

Images