Driving situation prediction and adaptive strategy generation system based on cloud multi-mode large model

Abstract

The invention discloses a driving situation prediction and adaptive strategy generation system based on a cloud multi-mode large model. Comprising a multi-modal data acquisition and situation prediction module, a self-adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud driving intelligent monitoring and feedback module and a dynamic risk assessment and path optimization module. Various input data from the vehicle and the environment are processed through the cloud, risk factors such as sudden obstacles, weather deterioration and complex traffic conditions in the automatic driving situation are predicted in advance, cloud computing provides powerful data processing capacity and storage resources, a large model deployed at the cloud can train data collected by the vehicle, and the risk factors can be predicted in advance. Therefore, the perception and prediction capabilities of the automatic driving system are improved, various coping strategies aiming at the risks are generated, and the vehicle is helped to make a safer and more intelligent driving decision in a complex scene through dynamic generation and remote optimization of an adaptive strategy.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model. Background Art

[0002] Current multimodal large language models (MLLMs) show great potential in improving the interpretability of autonomous driving systems. These models are able to generate control predictions and natural language explanations, but they also face challenges such as high data annotation costs and large domain differences.

[0003] For example, the Chinese patent with the publication number CN116394979A discloses an automatic driving decision control method based on roadside fusion perception. The invention relates to an automatic driving decision control method based on roadside fusion perception, including: step S1: obtaining the road condition information and environmental information around the vehicle and the vehicle's own state information; step S2: determining the vehicle's driving behavior according to the road condition information and environmental information around the vehicle and the vehicle's own state information; step S3: planning the vehicle's optimal driving path according to the determined driving behavior; step S4: controlling the longitudinal trajectory of the vehicle's movement and the lateral trajectory of the vehicle's movement, so that the vehicle travels according to the optimal driving path. The present invention can control the vehicle's driving behavior based on the current scene, plan the vehicle's optimal path based on the driving behavior, and control the vehicle to travel according to the optimal path.

[0004] The autonomous driving system used in this patent faces problems such as data scarcity, significant domain differences, high training costs, and catastrophic forgetting when dealing with complex and changing traffic environments. These problems limit the system's generalization ability and real-time performance, especially in driving scenarios that require rapid response.

[0005] Existing autonomous driving technology already has high processing capabilities under normal conditions, but in dynamic and complex driving scenarios (such as sudden obstacles, extreme weather, traffic jams, accidents, etc.), vehicles may still face problems of improper handling. Existing systems often rely on local sensors and algorithms to make decisions, and lack the ability to perceive traffic, weather, and environmental changes in a wider range in advance. Summary of the invention

[0006] The purpose of the present invention is to provide a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model to solve the above-mentioned problems existing in the prior art.

[0007] The specific application is as follows:

[0008] A driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model, including a multimodal data acquisition and scenario prediction module, an adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud-based driving intelligent monitoring and feedback module, and a dynamic risk assessment and path optimization module;

[0009] The multimodal data collection and situation prediction module collects vehicle driving data, weather data and traffic data through cameras, lidars, sensors and GPS deployed on the vehicle, and uses multimodal large model technology to perform situation analysis and prediction on the collected vehicle driving data, weather data and traffic data to generate multimodal large model analysis results;

[0010] The adaptive strategy generation and remote optimization module generates multiple strategies for dealing with complex situations through the analysis results of the multimodal large model, automatically selects and optimizes multiple strategies for the multiple strategies, and generates the best strategy;

[0011] The collaborative driving and multi-vehicle strategy coordination module is used to coordinate strategies for coping with complex situations among multiple autonomous driving vehicles, ensuring that multiple vehicles can work together in the same complex situation, avoiding conflicts between multiple vehicles and optimizing overall safety;

[0012] The cloud-based driving intelligent monitoring and feedback module intelligently monitors the vehicle's driving status, environmental changes and response strategies through the cloud, and provides real-time feedback;

[0013] The dynamic risk assessment and path optimization module, through the prediction and analysis of the multi-modal large model, conducts real-time assessment of the dynamic risks in the vehicle's driving path and generates an optimal path optimization solution;

[0014] The multimodal data collection and situation prediction module, the adaptive strategy generation and remote optimization module, the collaborative driving and multi-vehicle strategy coordination module, the cloud-based driving intelligent monitoring and feedback module and the dynamic risk assessment and path optimization module are interconnected.

[0015] Further, the multimodal data acquisition and situation prediction module includes an image and sensor data fusion submodule, a situation prediction submodule and a danger warning submodule;

[0016] The image and sensor data fusion submodule generates a three-dimensional environment perception result by fusing and analyzing the vehicle driving data, weather data and traffic data through a multi-modal large model;

[0017] The situation prediction submodule predicts possible risk situations in combination with the three-dimensional environment perception results;

[0018] The danger warning submodule, when the situation prediction submodule predicts an impending dangerous situation, issues a warning signal in advance to notify the vehicle and passengers to take preventive measures;

[0019] The image and sensor data fusion submodule includes an image data processing unit, a lidar data processing unit and a data fusion unit;

[0020] The image data processing unit uses a convolutional neural network to perform image features of different levels on the two-dimensional image data captured by the camera, and the convolutional neural network includes ResNet-50 and EfficientNet. The ResNet-50 and the EfficientNet are connected in series, and the output of the ResNet-50 is used as the input of the EfficientNet;

[0021] The laser radar data processing unit is used to extract features from the three-dimensional point cloud data captured by the laser radar using the PointNet++ network to obtain spatial information features and geometric features of each point to form three-dimensional point cloud features;

[0022] The data fusion unit is used to fuse the image features and the three-dimensional point cloud features to form multimodal features. The fusion adopts the EarlyFusion strategy plus the self-attention mechanism, and the formula is:

[0023] F fusion =SelfAttention(F img , F lidar , F 其他传感器 ),

[0024] Among them, F fusion is the fused multimodal feature, SelfAttention is the self-attention function, F fusion is the image feature, F lidar is the 3D point cloud feature, F 其他传感器 are other sensor features;

[0025] The specific process implemented by the situation prediction submodule is as follows:

[0026] The space around the vehicle is modeled and predicted using the BEVTransformer model, which is able to handle long-range dependencies and effectively capture global information in complex scenes:

[0027] H = Transformer (F fusion ),

[0028] Where H is the spatial representation around the vehicle, and the predictions generated based on this representation will be used for subsequent driving decisions;

[0029] The probability distribution of potential risk scenarios is evaluated by using the Bayesian neural network (BNN) model:

[0030] P(y|H)=∫P(y|z)P(z|H)dz, where z is a latent variable used to describe potential risk scenarios.

[0031] Further, the adaptive strategy generation and remote optimization module includes a multi-strategy generation submodule, a real-time remote optimization submodule and an adaptive driving submodule;

[0032] The multi-strategy generation submodule is used to detect that the vehicle encounters different situations and generate multiple response strategies, wherein the multiple response strategies include path replanning, speed adjustment and emergency braking, and the different situations include complex situations or potential dangers encountered by the vehicle;

[0033] The real-time remote optimization submodule optimizes the multiple response strategies through the cloud, dynamically adjusts the vehicle's driving plan, and sends control instructions corresponding to the optimized strategies to the vehicle in real time.

[0034] Furthermore, the adaptive driving submodule generates adaptive driving strategies of different levels based on different scenarios;

[0035] The specific implementation process of the multi-strategy generation submodule is as follows:

[0036] Use the deep Q network in reinforcement learning to generate different driving strategies. The driving strategy generation process will consider different driving scenarios and select the optimal strategy based on the current environment state:

[0037] Q(s,a)=E[r+βmax' a Q(s',a')],

[0038] Among them, s represents the current state of the vehicle, including speed, position, and sensor data, a represents the set of actions that can be taken, including braking, steering, acceleration, and deceleration, β is the discount factor, r is the immediate reward, and max' a Q(s', a') represents the maximum Q value of all possible actions in the next state s', E represents the learning rate, and the driving scenarios include emergency braking, obstacle avoidance and deceleration;

[0039] The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule is as follows:

[0040] The generated multiple response strategy sets are optimized through genetic algorithm GA, and multiple objective functions are used for multi-objective optimization:

[0041] Min{f1(a),f2(a),...,f n (a)},

[0042] where f n (a) is the loss function for a certain goal, Min represents the final selection of the optimal strategy;

[0043] The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule and dynamically adjusting the vehicle's driving plan is as follows:

[0044] Real-time remote strategy optimization is achieved through differential evolution algorithm and distributed deep learning model, and each vehicle is dynamically adjusted according to the optimization strategy provided by the cloud.

[0045] Furthermore, the collaborative driving and multi-vehicle strategy coordination module includes a collaborative situation prediction submodule, a strategy coordination submodule and a collective path planning submodule;

[0046] The collaborative situation prediction submodule is used to analyze the multimodal data of all vehicles through the cloud when multiple vehicles are driving in the same area, predict the global driving situation, and generate a multi-vehicle collaborative strategy;

[0047] The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle according to the status and driving mission of each vehicle to ensure the mutual coordination among multiple vehicles;

[0048] The collective path planning submodule is used to generate a collective path planning when multiple vehicles need to pass through different situational sections together.

[0049] Furthermore, the specific implementation process of the collaborative situation prediction submodule is as follows:

[0050] Using the global situation prediction algorithm based on the graph neural network GNN, the multi-modal data of multiple vehicles is globally modeled. The status of each vehicle is used as a node, which is updated through the data of neighboring nodes, and finally a global collaborative strategy and global graph structure are formed:

[0051]

[0052] in, is the feature representation of node v at layer k, N(v) is the set of neighbor nodes, W k is the network weight, c vu is the normalization coefficient;

[0053] Based on the global graph structure, a distributed optimization Lagrange multiplier algorithm is used to generate multi-vehicle collaborative path planning;

[0054] The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle to ensure the mutual coordination between multiple vehicles. The specific implementation process is as follows:

[0055] Through real-time data feedback, multi-vehicle strategies are continuously adjusted, and a model based on game theory is used to coordinate multi-vehicle strategies.

[0056] Furthermore, the cloud-based driving intelligent monitoring and feedback module includes a global monitoring and scheduling submodule, a feedback optimization submodule, and an emergency remote takeover submodule;

[0057] The global monitoring and scheduling submodule continuously monitors the operation of vehicles through traffic data and weather data, and automatically triggers strategies for adjustment if dangerous situations or abnormal conditions are detected;

[0058] The feedback optimization submodule is used to feed back the execution results and data of the global monitoring and scheduling submodule to the cloud. The execution results and data are used for future scenario prediction and strategy optimization generation, forming a self-learning closed loop;

[0059] The emergency remote takeover submodule is used to remotely take over vehicle control via the cloud and implement emergency operations in extreme situations, including vehicle collisions or other serious dangers.

[0060] Furthermore, the specific implementation process of continuously monitoring the operation of vehicles through traffic data and weather data in the global monitoring and scheduling submodule is as follows:

[0061] The vehicle's position, speed, and acceleration are monitored in real time through the Bayesian online learning algorithm and Kalman filter, and abnormal conditions are detected based on the dynamic Bayesian network DBN. The Kalman filter update formula is:

[0062] x t+1 =Ax t +Bu t +w t ,

[0063] P t+1 =AP t A T +Q,

[0064] Among them, x t is the system status, u t is the predicted state estimate, P t is the covariance matrix, A is the state transfer matrix, B is the control input matrix, w t is the process noise;

[0065] The specific implementation process of the feedback optimization submodule is as follows:

[0066] Regression analysis is used to analyze the results and data preprocessing of the global monitoring and scheduling submodule in real time, and an autoregressive sliding average model is used to perform time series analysis on the vehicle's historical dangerous situations or abnormal state data to generate feature data for future situation prediction and strategy optimization of the vehicle;

[0067] Based on the feature data of the vehicle's future scenario prediction and strategy optimization, combined with external environment data, random forest regression is used to analyze road conditions and weather information to form a self-learning closed loop;

[0068] The specific implementation process of the emergency remote takeover submodule is as follows:

[0069] When a potential extreme situation is detected, remote control is performed through a rule-based PID controller. The PID control formula is:

[0070]

[0071] Where, e(t) is the error signal, K p , K i , K d They are the proportional coefficient, integral coefficient and differential coefficient respectively.

[0072] Further, the dynamic risk assessment and path optimization module includes a dynamic risk assessment submodule, a path planning and optimization submodule, and an emergency route generation submodule;

[0073] The dynamic risk assessment submodule is used to analyze potential risks on the road in real time and optimize the vehicle's path according to the severity of the risk;

[0074] The path planning and optimization submodule generates an optimal driving path for the vehicle based on the evaluation results of the dynamic risk assessment submodule and ensures that the path planning is dynamically updated according to the actual situation during the vehicle driving process;

[0075] The emergency route generation submodule is used to generate an emergency path planning route when the vehicle encounters a high-risk situation, and optimize the path planning route in real time.

[0076] Furthermore, the specific implementation process of the dynamic risk assessment submodule is as follows:

[0077] Adopt factor analysis method to pre-process data in real time, extract the characteristics of vehicle position, speed and acceleration, and use autoregressive sliding average model to conduct time series analysis on historical vehicle driving data to generate real-time vehicle characteristic data;

[0078] Based on the real-time characteristic data of the vehicle and in combination with external environmental data, Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and generate risk assessment results;

[0079] Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and predict potential dangerous areas on the road through confidence intervals:

[0080] f(x) = GPR(m(x), k(x, x')), where m(x) is the mean function and k(x, x') is the kernel function used to measure the similarity between road points;

[0081] The specific implementation process of the path planning and optimization submodule is as follows:

[0082] The A* algorithm is combined with dynamic programming (DP) to generate the optimal driving route. Through multi-dimensional analysis of the current traffic conditions and road network, alternative routes to bypass potential risk areas are calculated, and the traffic flow and estimated driving time of each route are predicted and evaluated to generate a set of optimal detour routes.

[0083] The goal of dynamic programming (DP) is to find the optimal route for the preferred detour path set while ensuring safety through the shortest path algorithm:

[0084] J * (s) = min u {g(s,u)+J * (f(s,u))},

[0085] Among them, J * (s) is the optimal cost of state s, and g(s,u) is the cost function for taking action u.

[0086] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0087] The embodiment of the present invention provides a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model. The multimodal data collection and scenario prediction module collects vehicle driving data, weather data and traffic data through cameras, lidars, sensors and GPS deployed on the vehicle, and uses multimodal large model technology to perform scenario analysis and prediction on the collected vehicle driving data, weather data and traffic data to generate a multimodal large model analysis result; the adaptive strategy generation and remote optimization module generates a variety of strategies for dealing with complex situations through the multimodal large model analysis results, automatically selects and optimizes a variety of strategies for the various strategies, and generates the best strategy; the collaborative driving and multi-vehicle strategy coordination module is used to coordinate strategies for dealing with complex situations among multiple autonomous driving vehicles, ensuring that multiple vehicles can work together in the same complex situation, avoiding conflicts between multiple vehicles and optimizing overall safety. ; The cloud-based driving intelligent monitoring and feedback module intelligently monitors the vehicle's driving status, environmental changes and response strategies through the cloud, and provides real-time feedback; the dynamic risk assessment and path optimization module conducts real-time assessment of the dynamic risks in the vehicle's driving path through the prediction and analysis of the multimodal large model, and generates the optimal path optimization solution; the present invention processes a variety of input data from the vehicle and the environment through the cloud, and predicts risk factors in autonomous driving scenarios in advance, such as sudden obstacles, weather deterioration, complex traffic conditions, etc. Cloud computing provides powerful data processing capabilities and storage resources. The large model deployed in the cloud can train the data collected by the vehicle, thereby improving the perception and prediction capabilities of the autonomous driving system, and generate a variety of response strategies for these risks. It also dynamically generates and remotely optimizes through adaptive strategies to help vehicles make safer and smarter driving decisions in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a system architecture diagram of a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided by an embodiment of the present invention;

[0089] Figure 2 It is a multimodal data collection and situation prediction module diagram of a driving situation prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided by an embodiment of the present invention;

[0090] Figure 3 It is an adaptive strategy generation and remote optimization module diagram of a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided by an embodiment of the present invention;

[0091] Figure 4 It is a diagram of a collaborative driving and multi-vehicle strategy coordination module of a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided by an embodiment of the present invention;

[0092] Figure 5 It is a cloud-based driving intelligent monitoring and feedback module diagram of a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided by an embodiment of the present invention;

[0093] Figure 6 It is a dynamic risk assessment and path optimization module diagram of a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0094] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0095] Example 1

[0096] The embodiment of the present invention provides a driving situation prediction and adaptive strategy generation system based on a cloud-based multimodal large model, such as Figure 1 As shown, it includes multimodal data collection and situation prediction module, adaptive strategy generation and remote optimization module, collaborative driving and multi-vehicle strategy coordination module, cloud-based driving intelligent monitoring and feedback module and dynamic risk assessment and path optimization module;

[0097] The multimodal data collection and situation prediction module collects vehicle driving data, weather data and traffic data through cameras, lidars, sensors and GPS deployed on the vehicle, and uses multimodal large model technology to perform situation analysis and prediction on the collected vehicle driving data, weather data and traffic data to generate multimodal large model analysis results;

[0098] The adaptive strategy generation and remote optimization module generates multiple strategies for dealing with complex situations through the analysis results of the multimodal large model, automatically selects and optimizes multiple strategies for the multiple strategies, and generates the best strategy;

[0099] The collaborative driving and multi-vehicle strategy coordination module is used to coordinate strategies for coping with complex situations among multiple autonomous driving vehicles, ensuring that multiple vehicles can work together in the same complex situation, avoiding conflicts between multiple vehicles and optimizing overall safety;

[0100] The cloud-based driving intelligent monitoring and feedback module intelligently monitors the vehicle's driving status, environmental changes and response strategies through the cloud, and provides real-time feedback;

[0101] The dynamic risk assessment and path optimization module, through the prediction and analysis of the multi-modal large model, conducts real-time assessment of the dynamic risks in the vehicle's driving path and generates an optimal path optimization solution;

[0102] The multimodal data collection and situation prediction module, the adaptive strategy generation and remote optimization module, the collaborative driving and multi-vehicle strategy coordination module, the cloud-based driving intelligent monitoring and feedback module and the dynamic risk assessment and path optimization module are interconnected.

[0103] Furthermore, the multimodal data acquisition and situation prediction module includes an image and sensor data fusion submodule, a situation prediction submodule and a danger warning submodule, such as Figure 2 As shown;

[0104] The image and sensor data fusion submodule generates a three-dimensional environment perception result by fusing and analyzing the vehicle driving data, weather data and traffic data through a multi-modal large model;

[0105] The situation prediction submodule predicts possible risk situations in combination with the three-dimensional environment perception results;

[0106] The danger warning submodule, when the situation prediction submodule predicts an impending dangerous situation, issues a warning signal in advance to notify the vehicle and passengers to take preventive measures;

[0107] The image and sensor data fusion submodule includes an image data processing unit, a lidar data processing unit and a data fusion unit;

[0108] The image data processing unit uses a convolutional neural network to perform image features of different levels on the two-dimensional image data captured by the camera, and the convolutional neural network includes ResNet-50 and EfficientNet. The ResNet-50 and the EfficientNet are connected in series, and the output of the ResNet-50 is used as the input of the EfficientNet;

[0109] The laser radar data processing unit is used to extract features from the three-dimensional point cloud data captured by the laser radar using the PointNet++ network to obtain spatial information features and geometric features of each point to form three-dimensional point cloud features;

[0110] The data fusion unit is used to fuse the image features and the three-dimensional point cloud features to form multimodal features. The fusion adopts the EarlyFusion strategy plus the self-attention mechanism, and the formula is:

[0111] F fusion =SelfAttention(F img , F lidar , F 其他传感器 ),

[0112] Among them, F fusionis the fused multimodal feature, SelfAttention is the self-attention function, F fusion is the image feature, F lidar is the 3D point cloud feature, F 其他传感器 are other sensor features;

[0113] The specific process implemented by the situation prediction submodule is as follows:

[0114] The space around the vehicle is modeled and predicted using the BEVTransformer model, which is able to handle long-range dependencies and effectively capture global information in complex scenes:

[0115] H = Transformer (F fusion ),

[0116] Where H is the spatial representation around the vehicle, and the predictions generated based on this representation will be used for subsequent driving decisions;

[0117] The probability distribution of potential risk scenarios is evaluated by using the Bayesian neural network (BNN) model:

[0118] P(y|H)=∫P(y|z)P(z|H)dz, where z is a latent variable used to describe potential risk scenarios.

[0119] Furthermore, the adaptive strategy generation and remote optimization module includes a multi-strategy generation submodule, a real-time remote optimization submodule and an adaptive driving submodule. Figure 3 As shown;

[0120] The multi-strategy generation submodule is used to detect that the vehicle encounters different situations and generate multiple response strategies, wherein the multiple response strategies include path replanning, speed adjustment and emergency braking, and the different situations include complex situations or potential dangers encountered by the vehicle;

[0121] The real-time remote optimization submodule optimizes the multiple response strategies through the cloud, dynamically adjusts the vehicle's driving plan, and sends control instructions corresponding to the optimized strategies to the vehicle in real time.

[0122] Specifically, the multimodal data collection and situation prediction module interacts with the adaptive strategy generation and remote optimization module in the following process:

[0123] Interaction description: The multimodal data acquisition and situation prediction module is responsible for acquiring and processing the sensor data of the vehicle and the environment, generating real-time perception results of the environment and situation prediction information. This information is directly input into the adaptive strategy generation module to generate a strategy that adapts to the current situation.

[0124] Interaction algorithms and models:

[0125] Data sharing and transmission mechanism:

[0126] Feature F after multimodal data fusion fusion The scenario prediction result H will be passed as input to the adaptive strategy generation module:

[0127] InputtoModule:{F fusion , H},

[0128] Among them, F fusion represents the current vehicle's environmental perception information, and H represents the situation prediction results in the short term in the future, including sudden obstacles and bad weather;

[0129] Risk assessment transfer based on Bayesian network:

[0130] The scenario prediction results generated by the multimodal data acquisition module can be used to model potential risks based on the Bayesian network and pass the risk estimate to the strategy generation module:

[0131]

[0132] The strategy generation module generates a variety of response strategies based on risk probability;

[0133] The state input that strategy generation depends on:

[0134] The state space S includes the prediction information from the situation prediction module, such as the risk level on the path, the complexity of the environment, etc. Based on these states, the policy generation module uses deep Q learning or policy gradient algorithm to generate corresponding response actions.

[0135] Furthermore, the adaptive driving submodule generates adaptive driving strategies of different levels based on different scenarios;

[0136] The specific implementation process of the multi-strategy generation submodule is as follows:

[0137] Use the deep Q network in reinforcement learning to generate different driving strategies. The driving strategy generation process will consider different driving scenarios and select the optimal strategy based on the current environment state:

[0138] Q(s,a)=E[r+βmax' a Q(s', a')],

[0139] Among them, s represents the current state of the vehicle, including speed, position, and sensor data, a represents the set of actions that can be taken, including braking, steering, acceleration, and deceleration, β is the discount factor, r is the immediate reward, and max' aQ(s', a') represents the maximum Q value of all possible actions in the next state s', E represents the learning rate, and the driving scenarios include emergency braking, obstacle avoidance and deceleration;

[0140] The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule is as follows:

[0141] The generated multiple response strategy sets are optimized through genetic algorithm GA, and multiple objective functions are used for multi-objective optimization:

[0142] Min{f1(a),f2(a),...,f n (a)},

[0143] where f n (a) is the loss function for a certain goal, Min represents the final selection of the optimal strategy;

[0144] The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule and dynamically adjusting the vehicle's driving plan is as follows:

[0145] Real-time remote strategy optimization is achieved through differential evolution algorithm and distributed deep learning model, and each vehicle is dynamically adjusted according to the optimization strategy provided by the cloud.

[0146] Furthermore, the cooperative driving and multi-vehicle strategy coordination module includes a cooperative situation prediction submodule, a strategy coordination submodule and a collective path planning submodule, such as Figure 4 As shown;

[0147] The collaborative situation prediction submodule is used to analyze the multimodal data of all vehicles through the cloud when multiple vehicles are driving in the same area, predict the global driving situation, and generate a multi-vehicle collaborative strategy;

[0148] The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle according to the status and driving mission of each vehicle to ensure the mutual coordination among multiple vehicles;

[0149] The collective path planning submodule is used to generate a collective path planning when multiple vehicles need to pass through different situational sections together.

[0150] Specifically, the process of interaction between the adaptive strategy generation and remote optimization module and the cooperative driving and multi-vehicle strategy coordination module is as follows:

[0151] Interaction Description:

[0152] In the multi-vehicle collaborative scenario, the adaptive strategy generation module will interact the generated single-vehicle strategy with the collaborative driving module to ensure the compatibility of strategies among multiple vehicles and avoid conflicts.

[0153] Interaction algorithms and models:

[0154] Strategy coordination based on game theory:

[0155] Multi-vehicle collaboration is optimized through a model based on game theory. The strategy generation of each vehicle needs to consider the behavior of other vehicles. Using the Nash equilibrium model, the system can find the optimal strategy combination of multiple vehicles.

[0156] Strategy adjustment and information exchange among multiple workshops:

[0157] The collaborative driving module uses a distributed optimization algorithm to adjust the strategy of each vehicle to ensure global optimality. For example, a distributed consensus algorithm based on Lagrange multipliers is used for optimization. The system coordinates the strategies of all vehicles through distributed optimization.

[0158] Global situation prediction and sharing of collaborative driving:

[0159] The situation prediction information H of each vehicle i It will be integrated through the cloud to generate global situational prediction information H global Adjust the collective paths and behaviors of multiple vehicles:

[0160] H global =Fusion(H1, H2, ..., H n ),

[0161] Through the embedded node aggregation mechanism of the GNN graph neural network, the system can capture the interactive relationship between vehicles and perform global optimization.

[0162] Furthermore, the specific implementation process of the collaborative situation prediction submodule is as follows:

[0163] Using the global situation prediction algorithm based on the graph neural network GNN, the multi-modal data of multiple vehicles is globally modeled. The status of each vehicle is used as a node, which is updated through the data of neighboring nodes, and finally a global collaborative strategy and global graph structure are formed:

[0164]

[0165] in, is the feature representation of node v at layer k, N(v) is the set of neighbor nodes, W k is the network weight, c vu is the normalization coefficient;

[0166] Based on the global graph structure, a distributed optimization Lagrange multiplier algorithm is used to generate multi-vehicle collaborative path planning;

[0167] The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle to ensure the mutual coordination between multiple vehicles. The specific implementation process is as follows:

[0168] Through real-time data feedback, multi-vehicle strategies are continuously adjusted, and a model based on game theory is used to coordinate multi-vehicle strategies.

[0169] Furthermore, the cloud-based driving intelligent monitoring and feedback module includes a global monitoring and scheduling submodule, a feedback optimization submodule, and an emergency remote takeover submodule. Figure 5 As shown;

[0170] The global monitoring and scheduling submodule continuously monitors the operation of vehicles through traffic data and weather data, and automatically triggers strategies for adjustment if dangerous situations or abnormal conditions are detected;

[0171] The feedback optimization submodule is used to feed back the execution results and data of the global monitoring and scheduling submodule to the cloud. The execution results and data are used for future scenario prediction and strategy optimization generation, forming a self-learning closed loop;

[0172] The emergency remote takeover submodule is used to remotely take over vehicle control via the cloud and implement emergency operations in extreme situations, including vehicle collisions or other serious dangers.

[0173] Specifically, the interaction process between the collaborative driving and multi-vehicle strategy coordination module and the cloud driving intelligent monitoring and feedback module is as follows:

[0174] Interaction Description:

[0175] The interaction between the collaborative driving module and the cloud-based intelligent monitoring module ensures the timeliness of global monitoring and feedback. The cloud-based system can perform remote optimization and emergency adjustments based on the multi-vehicle status information provided by the collaborative driving module.

[0176] Interaction algorithms and models:

[0177] Feedback optimization based on global monitoring:

[0178] The cloud monitoring module obtains the status and environmental information of each vehicle in real time. i ,When danger or abnormality is detected, the cooperative strategy of multiple vehicles is automatically adjusted: the cooperative adjustment based on feedback can be achieved through model predictive control;

[0179] Global risk assessment and remote takeover in the cloud:

[0180] When the cloud monitoring system detects an extremely dangerous situation, such as an impending collision or a complex situation that cannot be handled, the cloud system can trigger the remote takeover function. Using the anomaly detection mechanism based on the Kalman filter, the cloud system can quickly assess the risk and issue remote control instructions. Once an anomaly is detected, the cloud monitoring will calculate new control inputs based on the vehicle status and remotely take over the vehicle.

[0181] Furthermore, the specific implementation process of continuously monitoring the operation of vehicles through traffic data and weather data in the global monitoring and scheduling submodule is as follows:

[0182] The vehicle's position, speed, and acceleration are monitored in real time through the Bayesian online learning algorithm and Kalman filter, and abnormal conditions are detected based on the dynamic Bayesian network DBN. The Kalman filter update formula is:

[0183] x t+1 =Ax t +Bu t +w t ,

[0184] P t+1 =AP t A T +Q,

[0185] Among them, x t is the system status, u t is the predicted state estimate, P t is the covariance matrix, A is the state transfer matrix, B is the control input matrix, w t is the process noise;

[0186] The specific implementation process of the feedback optimization submodule is as follows:

[0187] Regression analysis is used to analyze the results and data preprocessing of the global monitoring and scheduling submodule in real time, and an autoregressive sliding average model is used to perform time series analysis on the vehicle's historical dangerous situations or abnormal state data to generate feature data for future situation prediction and strategy optimization of the vehicle;

[0188] Based on the feature data of the vehicle's future scenario prediction and strategy optimization, combined with external environment data, random forest regression is used to analyze road conditions and weather information to form a self-learning closed loop;

[0189] The specific implementation process of the emergency remote takeover submodule is as follows:

[0190] When a potential extreme situation is detected, remote control is performed through a rule-based PID controller. The PID control formula is:

[0191]

[0192] Where, e(t) is the error signal, K p , K i , K d They are the proportional coefficient, integral coefficient and differential coefficient respectively.

[0193] Furthermore, the dynamic risk assessment and path optimization module includes a dynamic risk assessment submodule, a path planning and optimization submodule, and an emergency route generation submodule. Figure 6 As shown;

[0194] The dynamic risk assessment submodule is used to analyze potential risks on the road in real time and optimize the vehicle's path according to the severity of the risk;

[0195] The path planning and optimization submodule generates an optimal driving path for the vehicle based on the evaluation results of the dynamic risk assessment submodule, while ensuring that the path planning is dynamically updated according to the actual situation during the vehicle driving process;

[0196] The emergency route generation submodule is used to generate an emergency path planning route when the vehicle encounters a high-risk situation, and optimize the path planning route in real time.

[0197] Specifically, the interaction process between the cloud-based driving intelligent monitoring and the feedback module and the dynamic risk assessment and path optimization module is as follows:

[0198] Interaction Description:

[0199] The dynamic risk assessment and path optimization module is responsible for dynamically adjusting the vehicle's driving path based on the global information provided by the cloud monitoring module. This interaction ensures the real-time performance of the system in risk assessment and path selection.

[0200] Interaction algorithms and models:

[0201] Path optimization based on risk assessment:

[0202] The cloud monitoring module provides global risk assessment results global The dynamic risk assessment module adjusts the vehicle path in real time based on this information. Path optimization is performed using the A* algorithm or dynamic programming (DP) based on the Bellman equation;

[0203] Real-time generation of emergency paths:

[0204] When the dynamic risk assessment module finds that the vehicle path has a high risk, such as an obstacle or an accident ahead, the system will generate an emergency path and notify the vehicle. The path planning algorithm combines risk assessment and generates the optimal emergency path through a fast search algorithm such as RRT* or PRM. The algorithm ensures that the vehicle avoids high-risk areas through iterative search.

[0205] Furthermore, the specific implementation process of the dynamic risk assessment submodule is as follows:

[0206] Adopt factor analysis method to pre-process data in real time, extract the characteristics of vehicle position, speed and acceleration, and use autoregressive sliding average model to conduct time series analysis on historical vehicle driving data to generate real-time vehicle characteristic data;

[0207] Based on the real-time characteristic data of the vehicle and in combination with external environmental data, Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and generate risk assessment results;

[0208] Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and predict potential dangerous areas on the road through confidence intervals:

[0209] f(x) = GPR(m(x), k(x, x')), where m(x) is the mean function and k(x, x') is the kernel function used to measure the similarity between road points;

[0210] The specific implementation process of the path planning and optimization submodule is as follows:

[0211] The A* algorithm is combined with dynamic programming (DP) to generate the optimal driving route. Through multi-dimensional analysis of the current traffic conditions and road network, alternative routes to bypass potential risk areas are calculated, and the traffic flow and estimated driving time of each route are predicted and evaluated to generate a set of optimal detour routes.

[0212] The goal of dynamic programming (DP) is to find the optimal route for the preferred detour path set while ensuring safety through the shortest path algorithm:

[0213] J * (s) = min u {g(s,u)+J * (f(s,u))},

[0214] Among them, J * (s) is the optimal cost of state s, and g(s,u) is the cost function for taking action u.

[0215] Example 2: Scenario prediction and adaptive strategy generation for sudden road obstacles

[0216] 1. Background description: An autonomous vehicle is driving on a highway and an obstacle suddenly appears in front of it, such as an accident or a temporary roadblock.

[0217] The system needs to predict dangerous situations in advance and automatically generate response strategies.

[0218] 2. Situation prediction and warning: The system detects obstacles ahead through multimodal sensors and analyzes them through a large multimodal model in the cloud, predicting that the obstacle will affect the vehicle's driving path.

[0219] The system issues danger warnings in advance and is ready to generate multiple response strategies.

[0220] 3. Adaptive strategy generation: The system generates three strategies based on road conditions, vehicle speed, etc.: emergency braking, lane change avoidance, and speed reduction and waiting for obstacle removal.

[0221] The cloud system selects the best strategy - lane change avoidance, and sends instructions to the vehicle to adjust the vehicle's path.

[0222] 4. Feedback and optimization: The system records the execution results and feeds them back to the cloud to optimize future scenario prediction models.

[0223] Example 3: Multiple vehicles working together to respond to traffic emergencies

[0224] 1. Background description: When multiple autonomous driving vehicles are driving in the city, a sudden traffic accident occurs. The system needs to coordinate the driving strategies of multiple vehicles to avoid secondary collisions and traffic jams.

[0225] 2. Collaborative scenario prediction: The cloud system analyzes the multimodal data of all relevant vehicles, predicts the impact range of the traffic accident ahead, and evaluates the impact of the accident on the driving path of each vehicle.

[0226] 3. Multi-vehicle strategy coordination: The system generates different response strategies for each vehicle: some vehicles take emergency braking to avoid, and some vehicles take detours in advance.

[0227] The system coordinates the speed and driving routes of all vehicles to ensure smooth traffic at the accident scene.

[0228] 4. Dynamic adjustment and feedback: The system dynamically adjusts the driving strategy of each vehicle according to real-time traffic conditions to ensure safe passage, and feeds back the results for future collaborative optimization.

[0229] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0230] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0231] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0232] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0233] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0234] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

Claims

1. A driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model, characterized by: It includes multimodal data collection and situation prediction module, adaptive strategy generation and remote optimization module, collaborative driving and multi-vehicle strategy coordination module, cloud-based driving intelligent monitoring and feedback module, and dynamic risk assessment and path optimization module; The multimodal data collection and situation prediction module collects vehicle driving data, weather data and traffic data through cameras, lidars, sensors and GPS deployed on the vehicle, and uses multimodal large model technology to perform situation analysis and prediction on the collected vehicle driving data, weather data and traffic data to generate multimodal large model analysis results; The adaptive strategy generation and remote optimization module generates multiple strategies for dealing with complex situations through the analysis results of the multimodal large model, automatically selects and optimizes multiple strategies for the multiple strategies, and generates the best strategy; The collaborative driving and multi-vehicle strategy coordination module is used to coordinate strategies for coping with complex situations among multiple autonomous driving vehicles, ensuring that multiple vehicles can work together in the same complex situation, avoiding conflicts between multiple vehicles and optimizing overall safety; The cloud-based driving intelligent monitoring and feedback module intelligently monitors the vehicle's driving status, environmental changes and response strategies through the cloud, and provides real-time feedback; The dynamic risk assessment and path optimization module, through the prediction and analysis of the multi-modal large model, conducts real-time assessment of the dynamic risks in the vehicle's driving path and generates an optimal path optimization solution; The multimodal data collection and situation prediction module, the adaptive strategy generation and remote optimization module, the collaborative driving and multi-vehicle strategy coordination module, the cloud-based driving intelligent monitoring and feedback module and the dynamic risk assessment and path optimization module are interconnected.

2. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 1 is characterized in that: The multimodal data acquisition and situation prediction module includes an image and sensor data fusion submodule, a situation prediction submodule and a danger warning submodule; The image and sensor data fusion submodule generates a three-dimensional environment perception result by fusing and analyzing the vehicle driving data, weather data and traffic data through a multi-modal large model; The situation prediction submodule predicts possible risk situations in combination with the three-dimensional environment perception results; The danger warning submodule, when the situation prediction submodule predicts an impending dangerous situation, issues a warning signal in advance to notify the vehicle and passengers to take preventive measures; The image and sensor data fusion submodule includes an image data processing unit, a lidar data processing unit and a data fusion unit; The image data processing unit uses a convolutional neural network to perform image features of different levels on the two-dimensional image data captured by the camera, and the convolutional neural network includes ResNet-50 and EfficientNet. The ResNet-50 and the EfficientNet are connected in series, and the output of the ResNet-50 is used as the input of the EfficientNet; The laser radar data processing unit is used to extract features from the three-dimensional point cloud data captured by the laser radar using the PointNet++ network to obtain spatial information features and geometric features of each point to form three-dimensional point cloud features; The data fusion unit is used to fuse the image features and the three-dimensional point cloud features to form multimodal features. The fusion adopts the EarlyFusion strategy plus the self-attention mechanism. The formula is: F fusion =SelfAttention(F img ,F lidar ,F 其他传感器 ), Among them, F fusion is the fused multimodal feature, SelfAttention is the self-attention function, F fusion is the image feature, F lidar is the 3D point cloud feature, F 其他传感器 are other sensor features; The specific process implemented by the situation prediction submodule is as follows: The space around the vehicle is modeled and predicted using the BEVTransformer model, which is able to handle long-range dependencies and effectively capture global information in complex scenes: H=Transformer(F fusion ), Where H is the spatial representation around the vehicle, and the predictions generated based on this representation will be used for subsequent driving decisions; The probability distribution of potential risk scenarios is evaluated by using the Bayesian neural network (BNN) model: P(y|H)=∫P(y|z)P(z|H)dz, Among them, z is a latent variable used to describe potential risk scenarios.

3. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 2 is characterized in that: The adaptive strategy generation and remote optimization module includes a multi-strategy generation submodule, a real-time remote optimization submodule and an adaptive driving submodule; The multi-strategy generation submodule is used to detect that the vehicle encounters different situations and generate multiple response strategies, wherein the multiple response strategies include path replanning, speed adjustment and emergency braking, and the different situations include complex situations or potential dangers encountered by the vehicle; The real-time remote optimization submodule optimizes the multiple response strategies through the cloud, dynamically adjusts the vehicle's driving plan, and sends control instructions corresponding to the optimized strategies to the vehicle in real time.

4. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 3 is characterized in that: The adaptive driving submodule generates adaptive driving strategies of different levels based on different situations; The specific implementation process of the multi-strategy generation submodule is as follows: Use the deep Q network in reinforcement learning to generate different driving strategies. The driving strategy generation process will consider different driving scenarios and select the optimal strategy based on the current environment state: Q(s,a)=E[r+βmax' a Q(s',a')], Among them, s represents the current state of the vehicle, including speed, position, and sensor data, a represents the set of actions that can be taken, including braking, steering, acceleration, and deceleration, β is the discount factor, r is the immediate reward, and max' a Q(s', a') represents the maximum Q value of all possible actions in the next state s', E represents the learning rate, and the driving scenarios include emergency braking, obstacle avoidance and deceleration; The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule is as follows: The generated multiple response strategy sets are optimized through genetic algorithm GA, and multiple objective functions are used for multi-objective optimization: Min{f1(a),f2(a),...,f n (a)}, where f n (a) is the loss function for a certain goal, Min represents the final selection of the optimal strategy; The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule and dynamically adjusting the vehicle's driving plan is as follows: Real-time remote strategy optimization is achieved through differential evolution algorithm and distributed deep learning model, and each vehicle is dynamically adjusted according to the optimization strategy provided by the cloud.

5. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 4 is characterized in that: The collaborative driving and multi-vehicle strategy coordination module includes a collaborative situation prediction submodule, a strategy coordination submodule and a collective path planning submodule; The collaborative situation prediction submodule is used to analyze the multimodal data of all vehicles through the cloud when multiple vehicles are driving in the same area, predict the global driving situation, and generate a multi-vehicle collaborative strategy; The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle according to the status and driving mission of each vehicle to ensure the mutual coordination among multiple vehicles; The collective path planning submodule is used to generate a collective path planning when multiple vehicles need to pass through different situational sections together.

6. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 5 is characterized in that: The specific implementation process of the collaborative situation prediction submodule is as follows: Using the global situation prediction algorithm based on the graph neural network GNN, the multi-modal data of multiple vehicles is globally modeled. The status of each vehicle is used as a node, which is updated through the data of neighboring nodes, and finally a global collaborative strategy and global graph structure are formed: in, is the feature representation of node v at layer k, N(v) is the set of neighbor nodes, W k is the network weight, c vu is the normalization coefficient; Based on the global graph structure, a distributed optimization Lagrange multiplier algorithm is used to generate multi-vehicle collaborative path planning; The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle to ensure the mutual coordination between multiple vehicles. The specific implementation process is as follows: Through real-time data feedback, multi-vehicle strategies are continuously adjusted, and a model based on game theory is used to coordinate multi-vehicle strategies.

7. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 6 is characterized in that: The cloud-based driving intelligent monitoring and feedback module includes a global monitoring and scheduling submodule, a feedback optimization submodule, and an emergency remote takeover submodule; The global monitoring and scheduling submodule continuously monitors the operation of vehicles through traffic data and weather data, and automatically triggers strategies for adjustment if dangerous situations or abnormal conditions are detected; The feedback optimization submodule is used to feed back the execution results and data of the global monitoring and scheduling submodule to the cloud. The execution results and data are used for future scenario prediction and strategy optimization generation, forming a self-learning closed loop; The emergency remote takeover submodule is used to remotely take over vehicle control via the cloud and implement emergency operations in extreme situations, including vehicle collisions or other serious dangers.

8. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 7 is characterized in that: The specific implementation process of continuously monitoring the operation of vehicles through traffic data and weather data in the global monitoring and scheduling submodule is as follows: The vehicle's position, speed, and acceleration are monitored in real time through the Bayesian online learning algorithm and Kalman filter, and abnormal conditions are detected based on the dynamic Bayesian network DBN. The Kalman filter update formula is: x t+1 =Ax t +Bu t +w t , P t+1 =AP t From T +Q, Among them, x t is the updated state estimate, u t is the predicted state estimate, P t is the covariance matrix, A is the state transfer matrix, B is the control input matrix, w t is the process noise; The specific implementation process of the feedback optimization submodule is as follows: Regression analysis is used to analyze the results and data preprocessing of the global monitoring and scheduling submodule in real time, and an autoregressive sliding average model is used to perform time series analysis on the vehicle's historical dangerous situations or abnormal state data to generate feature data for vehicle future situation prediction and strategy optimization; Based on the feature data of the vehicle's future scenario prediction and strategy optimization, combined with external environment data, random forest regression is used to analyze road conditions and weather information to form a self-learning closed loop; The specific implementation process of the emergency remote takeover submodule is as follows: When a potential extreme situation is detected, remote control is performed through a rule-based PID controller. The PID control formula is: Where, e(t) is the error signal, K p , K i , K d They are the proportional coefficient, integral coefficient and differential coefficient respectively.

9. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 8 is characterized in that: The dynamic risk assessment and path optimization module includes a dynamic risk assessment submodule, a path planning and optimization submodule, and an emergency route generation submodule; The dynamic risk assessment submodule is used to analyze potential risks on the road in real time and optimize the vehicle's path according to the severity of the risk; The path planning and optimization submodule generates an optimal driving path for the vehicle based on the evaluation results of the dynamic risk assessment submodule, while ensuring that the path planning is dynamically updated according to the actual situation during the vehicle driving process; The emergency route generation submodule is used to generate an emergency path planning route when the vehicle encounters a high-risk situation, and optimize the path planning route in real time.

10. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 9, characterized in that: The specific implementation process of the dynamic risk assessment submodule is as follows: Adopt factor analysis method to pre-process data in real time, extract the characteristics of vehicle position, speed and acceleration, and use autoregressive sliding average model to conduct time series analysis on historical vehicle driving data to generate real-time vehicle characteristic data; Based on the real-time characteristic data of the vehicle and in combination with external environmental data, Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and generate risk assessment results; Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and predict potential dangerous areas on the road through confidence intervals: f(x)=GPR(m(x),k(x,x')), Among them, m(x) is the mean function, k(x,x') is the kernel function, which is used to measure the similarity between road points; The specific implementation process of the path planning and optimization submodule is as follows: The A* algorithm is combined with dynamic programming (DP) to generate the optimal driving route. Through multi-dimensional analysis of the current traffic conditions and road network, alternative routes to bypass potential risk areas are calculated, and the traffic flow and estimated driving time of each route are predicted and evaluated to generate a set of optimal detour routes. The goal of dynamic programming (DP) is to find the optimal route for the preferred detour path set while ensuring safety through the shortest path algorithm: J * (s)=min u {g(s,u)+J * (f(s,u))}, Among them, J * (s) is the optimal cost of state s, and g(s,u) is the cost function for taking action u.

Images