Cloud intelligent driving control system based on multi-modal large model

Abstract

The invention provides a cloud intelligent driving control system based on a multi-modal large model, and belongs to the technical field of automatic driving. The system comprises a cloud multi-modal large model analysis and control module, a real-time remote driving takeover module, an advanced prediction and risk avoidance module, a cloud intelligent driving collaboration and optimization module, a multi-modal fault diagnosis and remote repair module and a real-time communication and interaction enhancement module. The system realizes cloud remote monitoring and control of the autonomous vehicle through interaction and cooperation among different modules, and assists the autonomous vehicle through remote control, so that the autonomous vehicle runs more safely; the early prediction and risk avoidance module of the system sets the prediction of the sight line of the vehicle and the prediction of the sight line of the vehicle, and sets corresponding prediction methods for the vehicles in two different environments for prediction in a mode of extracting different matrix samples, so that the prediction accuracy of the cloud system and the safety of the automatic driving vehicle are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a cloud-based intelligent driving control system based on a multimodal large model. Background Art

[0002] With the rapid development of autonomous driving technology, operational problems of autonomous vehicles in complex environments are gradually emerging; for example, when a vehicle is operating in a dynamic and complex urban environment, in severe weather or in traffic emergencies, the system may face problems such as perception failure, difficult path planning or complex decision-making; in this case, the local driving system may not be able to handle all situations, and it will need to rely on remote manual driving intervention or intelligent cloud systems to assist in decision-making or take over the vehicle, which will be the key to solving such problems.

[0003] Traditional cloud-based driving control systems usually rely on a single data source or slower data transmission, and it is difficult to achieve real-time and efficient remote takeover and collaboration in a dynamic environment. Even if efficient network transmission functions are used, there is still a certain delay in the transmission from the cloud-based intelligent driving control system to the autonomous driving vehicle. In order to solve the above problems, the present invention combines multimodal large model technology to process multiple inputs such as images, text, voice, and sensor data to achieve more intelligent and comprehensive cloud-based driving control and support. The advance prediction function is enhanced in the multimodal large model system to further ensure driving safety. Summary of the invention

[0004] The purpose of the present invention is to overcome the above problems existing in the prior art and to greatly improve the technical effect thereof on the basis of the original technology; for this purpose, the present invention provides a cloud-based intelligent driving control system based on a multimodal large model, the system comprising:

[0005] Cloud-based multimodal large model analysis and control module; real-time remote driving takeover module; early prediction and risk avoidance module; cloud-based intelligent driving coordination and optimization module; multimodal fault diagnosis and remote repair module; real-time communication and enhanced interaction module.

[0006] The cloud-based multimodal large model analysis and control module is used to process multimodal data uploaded by the autonomous driving vehicle, analyze the vehicle's current environment, generate remote driving control strategies, and send control instructions to the vehicle through low-latency communication; the cloud-based multimodal large model analysis and control module includes: image and video processing unit, sensor data fusion unit and natural language processing NLP unit; the image and video processing unit is used to process real-time video streams and static image data uploaded by the vehicle, and identify obstacles, road signs and pedestrian elements around the vehicle through computer vision technology; the image and video processing unit includes target recognition and three-dimensional images and videos; target recognition of image and video information is achieved through YOLOv5, which is a deep learning model for object detection. It performs image processing through convolutional neural network CNN and can detect and identify targets in real time; image and video information is detected through the YOLOv5 target detection model The steps of target recognition are: extracting features from the input image through multiple convolutional layers; using the feature map output by the convolutional layer to generate candidate regions, which is also called an anchor box; classifying the targets in the anchor box through a classifier and adjusting the size of the anchor box; using non-maximum suppression (NMS) to eliminate duplicate boxes; the non-maximum suppression (NMS) is a commonly used post-processing technique for eliminating duplicate bounding boxes in target detection. The steps of eliminating duplicate bounding boxes by non-maximum suppression (NMS) are: 11) first, sorting all detected boxes according to target confidence; 12) selecting the box with the highest confidence as the reserved box; 13) using IoU to calculate the overlap between the current reserved box and other boxes, the IoU refers to the bounding box error calculated by loss, and the other boxes are boxes outside the reserved box; 14) if the IoU of other boxes exceeds a preset threshold, remove them from the candidate boxes; 15) repeating steps 12) to 14) until all boxes are processed.

[0007] The BEV Transformer model is used to realize three-dimensional image and video information; the BEV Transformer model is used to convert the image from the two-dimensional camera perspective into a three-dimensional bird's-eye view to complete the three-dimensionalization of image and video information; the BEV Transformer model uses the Transformer architecture to learn the spatial relationship of the image through the self-attention mechanism and generate a three-dimensional view of the vehicle's surrounding environment; the steps for the BEV Transformer model to realize three-dimensional image and video information are: 21) input images from different perspectives; 22) map the features of the two-dimensional image to the three-dimensional space through position encoding; 23) use the self-attention mechanism to calculate the correlation between different perspectives; 24) output the three-dimensional environment representation around the vehicle; the formula for learning spatial relationships through the self-attention mechanism is:

[0008]

[0009] Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix.

[0010] The sensor data fusion unit is used to integrate multiple sensor data from the vehicle, including radar, lidar, GPS, ultrasound and accelerometer, etc.; by integrating multiple sensor data, a more accurate three-dimensional environmental model is generated to help the system make more accurate remote driving strategies in complex environments; the extended Kalman filter EKF and point cloud fusion algorithm ICP are used to achieve the fusion of different sensor data; the fusion steps are: 31) preliminary filtering of the lidar point cloud data and GPS data to remove noise; 32) EKF is used to estimate the vehicle's position and speed; 33) the ICP algorithm is used to align the point cloud data at different time points to obtain more accurate environmental perception.

[0011] The natural language processing (NLP) unit is used to parse the voice input and text information of passengers and vehicles, and generate corresponding driving instructions according to driving needs; the natural language processing (NLP) unit is also used to process the alarm information issued by the vehicle and automatically trigger the remote driving takeover procedure; the BERT model is used to process natural language. The BERT model uses the Transformer architecture for language understanding. It can process the voice instructions of passengers and the alarm text information of the vehicle and generate corresponding instruction feedback; the steps for processing natural language through the BERT model are as follows: 41) The passenger and vehicle voice data are preprocessed, including: feature extraction and conversion of speech recognition into text, etc.; 42) The text is input into the BERT model and encoded through a multi-layer Transformer to capture the contextual semantics; 43) The BERT model outputs the parsed driving needs and triggers the corresponding driving instructions.

[0012] The real-time remote driving takeover module is used to take over the control of the vehicle in complex and emergency situations; the real-time remote driving takeover module can handle scenarios that the vehicle's automatic driving cannot cope with, including extreme weather, complex traffic conditions and emergencies. The cloud-based intelligent driving control system analyzes multimodal data to generate a takeover strategy and adjusts the vehicle's parameters including speed, direction and acceleration to ensure safe driving of the vehicle; the real-time remote driving takeover module includes: an emergency takeover trigger unit, an intelligent takeover strategy generation unit and a feedback and adjustment unit; wherein the emergency takeover trigger unit is used to automatically send a takeover request to the cloud when the vehicle detects unpredictable environmental conditions, including: such as sensor failure, system failure and extreme weather; the emergency takeover trigger unit is provided with a takeover trigger mechanism, and the takeover trigger mechanism is based on the takeover strategy of the DQN reinforcement learning algorithm; the DQN is an algorithm that combines deep learning and reinforcement learning, which aims to solve complex decision-making problems, and is particularly suitable for decision-making problems in high-dimensional state space. The steps of activating the takeover trigger mechanism through DQN are: 51) defining the state s of the vehicle in different environments t , including vehicle speed, acceleration, and sensor readings; 52) In each state, the vehicle selects action a t , including requesting to take over and continuing autonomous driving; 53) According to the execution of the action, the vehicle receives a reward of r t , including avoiding accidents and improving driving efficiency; 54) Use Q-Learning to update the vehicle's decision network. The Q-Learning is the basis of DQN. Q-Learning is a model-free reinforcement learning algorithm that helps the vehicle choose the optimal action in a specific state by learning an action value function Q value. The formula of the action value function Q value is:

[0013] Q(s t , a t )=Q(s t , a t )+α[r t +γmax at+1 Q(s t+1 , a t+1 )-Q(s t , a t )]

[0014] Among them, α is the learning rate, r t To perform action a t The immediate reward obtained after, γ is the discount factor, the Q(s t , a t ) is the updated Q value; Q(s t , a t ) is in s t In the state, select action a t The old Q value of st+1 Indicates the new state reached after executing the action; max at+1 Q(s t+1 , a t+1 ) indicates that in the new state s t+1 Next, select the action that maximizes the Q value among all actions.

[0015] The intelligent takeover strategy generation unit generates the optimal vehicle control strategy based on cloud multimodal data analysis, and sends it to the vehicle through a low-latency communication link, taking over the control functions of vehicle acceleration, braking and steering; the intelligent takeover strategy generation unit is mainly used to generate remote takeover strategies, and generates remote takeover control strategies through the MPC model; the MPC is an optimization algorithm for dynamic system control, suitable for multi-step prediction and real-time optimization; the MPC steps are: 61) Establish a vehicle dynamics model and input the vehicle state, including: vehicle speed, position and steering angle; 62) Within a certain prediction time window, predict the future state based on the current state, and optimize the control variables, including: acceleration, braking and steering; 63) Send the optimized control strategy to the vehicle; the MPC optimization formula is:

[0016]

[0017] Among them, x(k+1) is the predicted vehicle state, x ref is the target state, u k is the control input, Q and R are weight matrices, ||·|| represents the norm, By adjusting the control input u k To find the control strategy that minimizes the objective cost function.

[0018] The feedback and adjustment unit is used to receive real-time feedback on the vehicle's status and continuously adjust the takeover strategy according to environmental changes to ensure that the vehicle can safely respond to emergencies.

[0019] The advance prediction and risk avoidance module is used to predict in advance the risks that the vehicle will face in the future, and based on the predicted risks, when it is judged that the predicted risks are greater than the risk threshold, generate a plan to avoid the risks; the risk avoidance plan includes parking, deceleration, lane change and re-planning the route; the future period refers to a time period that the cloud automatically adjusts according to the vehicle speed; the advance prediction and risk avoidance module is a supplement to the real-time remote driving takeover module to predict the vehicle state; the advance prediction and risk avoidance module includes a prediction unit for the line of sight of the overtaking vehicle and a prediction unit for the line of sight of the vehicle; the prediction of the line of sight of the overtaking vehicle refers to the prediction basis including information other than the images and videos collected by the vehicle, and the information other than the images and videos collected by the vehicle is obtained through the cloud system, including other vehicles and pedestrians that block the target vehicle from obtaining information due to obstacles; the prediction of the line of sight of the overtaking vehicle is predicted using the long short-term memory neural network LSTM, and the prediction steps are: 71) First, extract the historical overtaking vehicles in the cloud system The target vehicle information and other target information of the sight line of the vehicle, the other targets include vehicles and pedestrians; extract the speed magnitude, speed direction, acceleration magnitude and acceleration direction information of the target vehicle, extract the speed magnitude, speed direction, acceleration magnitude and acceleration direction information of other targets, extract the distance information of the target vehicle and other targets, and extract the intersection point information where the target vehicle and other targets meet; the intersection point information includes the distance between the intersection point and the target vehicle and the position information relative to the target vehicle; the direction is to establish a rectangular coordinate system with the target vehicle as the origin, and then determine the speed direction, acceleration direction of different targets, and the position information of the intersection point relative to the target vehicle; 72) the target vehicle information and intersection point information extracted in 71) are used as the first row of the matrix, the extracted other target information and intersection point information are used as the second row of the matrix, and the distance information of the target vehicle and other targets is used as the third row of the matrix, that is, a matrix with three rows is formed, and the matrix is ​​set to X i , i refers to the extracted i-th group of target vehicle data information; 73) The distance when the target vehicle meets other targets is set to Y i , i represents the i-th group of target vehicles and other targets, and X i and Y i The combination of the predictions of the overtaking vehicle's sight line {X i , Y i}; 74) {X i , Y i} Input long short-term memory neural network LSTM, X i As the input of the LSTM neural network, Y iThe long short-term memory neural network LSTM is trained as the output of the LSTM neural network, and the steps of LSTM data preprocessing, defining the model architecture, training the model, adjusting the parameters and evaluating the model are performed. After the training is completed, the LSTM neural network model of the overtaking vehicle line of sight is obtained; 75) extracting the real-time target vehicle information and other target information, and organizing the target vehicle information and other target information into a matrix X according to steps 71) and 72). i data format, and input the sorted information into the LSTM neural network model of the overtaking vehicle's line of sight after preprocessing to obtain the output result, and determine whether to start the risk avoidance module according to whether the output result is within the threshold range; the threshold range means that the predicted shortest distance when the target vehicle and other targets meet is greater than the dangerous distance, and the dangerous distance is set to 3 meters according to traffic safety regulations; 76) if the output result is no longer within the threshold range, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit of the real-time remote driving takeover module, and the optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit, and the risk avoidance module issues a risk avoidance instruction to the vehicle, and the vehicle executes operations including: deceleration, lane change and detour.

[0020] In addition, the prediction of vehicle sight lines is also carried out using the long short-term memory neural network LSTM model. The difference is that in constructing the combination {X i , Y i}, X i A matrix consisting of a target vehicle and multiple other targets is constructed, that is, the first row of the matrix is ​​the attribute of the target vehicle, and the following rows are the attributes corresponding to other targets, and the attributes of the distance to the target vehicle are added after the attributes of other targets, respectively. The intersection information of the target vehicle is rewritten as the average value of the intersection information with other targets, and the distance information of the target vehicle is added as the average value of the distances with other targets; and Y i Constructed into classified data, the vehicle is in the matrix X i The accident is classified as 1, and the vehicle is in the matrix X i If no accident occurs, it is classified as 0; the neural network LSTM model is trained through the set combination, and the trained LSTM model is used to determine in real time whether the target vehicle has risks. If there are risks, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit of the real-time remote driving takeover module. The optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit, and the risk avoidance instruction is issued to the vehicle through the risk avoidance module, and the vehicle executes operations including deceleration, lane changing and detour.

[0021] The cloud-based intelligent driving coordination and optimization module is used to coordinate the cluster operation, path optimization and coordinated driving of multiple autonomous driving vehicles; the cloud-based intelligent driving coordination and optimization module can dynamically assign tasks to multiple vehicles to ensure that each vehicle effectively utilizes road resources when performing pick-up and drop-off tasks, reduce congestion and optimize overall travel efficiency; the cloud-based intelligent driving coordination and optimization module includes a task allocation unit and a coordinated driving unit. The task allocation unit is used to dynamically allocate pick-up and drop-off tasks for multiple autonomous driving vehicles based on vehicle status, real-time traffic information and passenger needs; the cloud-based system performs scheduling based on environmental changes and task priorities to ensure the shortest path, minimum energy consumption and maximum vehicle utilization; using A ＊ The algorithm combines deep reinforcement learning to optimize the path; through A ＊ The algorithm guides the search process to find the optimal path, and deep reinforcement learning is used in A ＊ The optimal path found by the algorithm is dynamically adjusted according to the dynamically changing environment; the cooperative driving unit is used to coordinate the cooperative driving of multiple vehicles on highways and urban roads, ensure the safe distance, speed matching and path optimization between the fleets, reduce mutual interference between vehicles, and improve overall efficiency; the cooperative algorithm based on V2X communication and Byzantine consensus protocol realizes multi-vehicle cooperative driving, the steps are: 81) vehicles share data including their location information and speed through V2X communication; 82) the cloud calculates the optimal fleet driving path and safe distance through the Byzantine consensus algorithm; 83) vehicles perform cooperative driving according to cloud instructions; the Byzantine consensus protocol formula is:

[0022] x i (t+1)=∑ j∈N(i) a ij x j (t)

[0023] Among them, x j (t) represents the state of node j at time t, x i (t+1) represents the state of node i at time t+1 obtained by summation operation; N(i) represents the set of neighboring vehicles of i, a ij represents the weight coefficient, and ∑ represents the summation operation.

[0024] The multimodal fault diagnosis and remote repair module is used to monitor the fault data uploaded by the vehicle in real time, including sensor failure and system anomalies, and diagnose the fault through the cloud multimodal large model; the multimodal fault diagnosis and remote repair module supports remote repair and system restart, and generates alternative driving solutions to help the vehicle complete the task, ensuring that the vehicle can continue to operate safely under fault conditions; the multimodal fault diagnosis and remote repair module includes: a fault prediction unit and a remote repair and alternative solution generation unit; the fault prediction unit is used to predict possible system faults or sensor failures based on historical vehicle operation data and sensor data, and give early warnings; a fault detection algorithm based on a CNN-LSTM network is used The method is used to predict faults. The prediction steps are: 91) Use CNN to extract multimodal data, including spatial features of images and sensor data; 92) Use LSTM to process time series data and detect anomalies in sensor data; 93) Output the fault detection results and trigger the repair process; The remote repair and alternative solution generation unit is based on the strategy testing and alternative solutions of the autonomous driving simulation platform; First, simulate vehicle failure conditions in the cloud, and test different repair strategies through simulation, including restarting the system and adjusting sensor configuration; Second, if the repair cannot be completed quickly, generate the used driving plan, such as reducing the functional complexity of the driving system by disabling certain sensors to ensure that the basic safe driving function of the vehicle is still effective.

[0025] The real-time communication and enhanced interaction module is used to establish a high-speed and low-latency communication link between the cloud and the vehicle to ensure the real-time transmission and execution of remote driving instructions; the real-time communication and enhanced interaction module also supports interaction with passengers in the vehicle, providing real-time feedback on vehicle status, fault repair progress and driving control; the real-time communication and enhanced interaction module includes: a 5G satellite communication unit and a passenger interaction and feedback unit; the 5G satellite communication unit is used to ensure that the information transmission between the vehicle and the cloud has low latency and high bandwidth characteristics when remote driving takes over and cloud-assisted control is performed; the passenger interaction and feedback unit provides passengers with real-time driving status, system failures and estimated arrival time information through smartphone applications and in-vehicle interfaces, and allows passengers to make requests or adjust the itinerary in an emergency; the 5G satellite communication unit and the passenger interaction and feedback unit are based on adaptive TCP communication control, and the low-latency communication architecture is based on the adaptive TCP congestion control protocol, which controls the efficiency of data packet transmission by adjusting the congestion window size.

[0026] Furthermore, the cloud-based multimodal large model analysis and control module, real-time remote driving takeover module, early prediction and risk avoidance module, cloud-based intelligent driving coordination and optimization module, multimodal fault diagnosis and remote repair module and real-time communication and enhanced interaction module of the present invention interact and collaborate with each other, and through the interaction and collaboration of different modules, the cloud-based multimodal large model system monitors and controls the remote autonomous driving vehicle.

[0027] The beneficial effects of the present invention are:

[0028] The present invention provides a cloud-based intelligent driving control system based on a multimodal large model, which includes a cloud-based multimodal large model analysis and control module; a real-time remote driving takeover module; an advance prediction and risk avoidance module; a cloud-based intelligent driving coordination and optimization module; a multimodal fault diagnosis and remote repair module; a real-time communication and enhanced interaction module; the system realizes remote cloud-based monitoring and control of autonomous driving vehicles through interaction and coordination between different modules; the advance prediction and risk avoidance module of the system sets the prediction of the line of sight of an overtaking vehicle and the prediction of the line of sight of the vehicle, and respectively sets corresponding prediction methods for vehicles in two different environments by extracting different matrix samples, thereby increasing the accuracy of cloud-based system predictions and the safety of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the cloud-based intelligent driving control system based on a multimodal large model of the present invention.

[0030] Figure 2 It is a schematic diagram of the cloud-based intelligent driving control process based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention, and cannot be used to limit the present invention.

[0032] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0033] like Figure 1, which is a schematic diagram of a cloud-based intelligent driving control system based on a multimodal large model according to an embodiment of the present invention, the schematic diagram includes: a cloud-based multimodal large model analysis and control module S100; a real-time remote driving takeover module S200; an advance prediction and risk avoidance module S300; a cloud-based intelligent driving coordination and optimization module S400; a multimodal fault diagnosis and remote repair module S500; a real-time communication and enhanced interaction module S600; wherein the cloud-based multimodal large model analysis and control module S100 includes: an image and video processing unit S101, a sensor data fusion unit S102 and a natural language processing NLP unit S103; a real-time remote driving takeover module S200; a Block S200 includes: an emergency takeover trigger unit S201, an intelligent takeover strategy generation unit S202 and a feedback and adjustment unit; the advance prediction and risk avoidance module S300 includes: a prediction unit S301 for the overtaking vehicle's line of sight and a prediction unit S302 for the vehicle's line of sight; the cloud-based intelligent driving coordination and optimization module S400 includes: a task allocation unit S401 and a coordinated driving unit S402; the multimodal fault diagnosis and remote repair module S500 includes a fault prediction unit S501 and a remote repair and alternative solution generation unit S502; the real-time communication and enhanced interaction module S600 includes: a 5G satellite communication unit S601 and a passenger interaction and feedback unit S602.

[0034] Among them, the cloud-based multimodal large model analysis and control module S100 is used to process the multimodal data uploaded by the autonomous driving vehicle, analyze the current environment of the vehicle, generate a remote driving control strategy, and send control instructions to the vehicle through low-latency communication; the cloud-based multimodal large model analysis and control module S100 includes: an image and video processing unit S101, a sensor data fusion unit S102, and a natural language processing NLP unit S103; the image and video processing unit S101 is used to process the real-time video stream and static image data uploaded by the vehicle, and identify obstacles, road signs and pedestrian elements around the vehicle through computer vision technology; the image and video processing unit S101 includes target recognition and three-dimensional images and videos; target recognition of image and video information is achieved through YOLOv5, which is a deep learning model for object detection. It performs image processing through convolutional neural network CNN and can detect and identify targets in real time; through YOLOv5 The steps of the target detection model for target recognition of image and video information are: extracting features from the input image through multiple convolutional layers; using the feature map output by the convolutional layer to generate candidate regions, which is also called anchor boxes; classifying the targets in the anchor boxes through a classifier and adjusting the size of the anchor boxes; using non-maximum suppression (NMS) to eliminate duplicate boxes; the non-maximum suppression (NMS) is a commonly used post-processing technique for eliminating duplicate bounding boxes in target detection. The steps of non-maximum suppression (NMS) to eliminate duplicate bounding boxes are: 11) first, sorting all detected boxes according to target confidence; 12) selecting the box with the highest confidence as the reserved box; 13) using IoU to calculate the overlap between the current reserved box and other boxes, the IoU refers to the bounding box error calculated by loss, and the other boxes are boxes outside the reserved box; 14) if the IoU of other boxes exceeds a preset threshold, remove them from the candidate boxes; 15) repeating steps 12) to 14) until all boxes are processed.

[0035] The BEV Transformer model is used to realize three-dimensional image and video information; the BEV Transformer model is used to convert the image from the two-dimensional camera perspective into a three-dimensional bird's-eye view to complete the three-dimensionalization of the image and video information; the BEV Transformer model uses the Transformer architecture to learn the spatial relationship of the image through the self-attention mechanism and generate a three-dimensional view of the vehicle's surrounding environment; the steps of the BEV Transformer model to realize three-dimensional image and video information are: 21) input images from different perspectives; 22) map the features of the two-dimensional image to the three-dimensional space through position encoding; 23) use the self-attention mechanism to calculate the correlation between different perspectives; 24) output the three-dimensional environment representation around the vehicle; the formula for learning the spatial relationship through the self-attention mechanism is:

[0036]

[0037] Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix.

[0038] The sensor data fusion unit S102 is used to integrate various sensor data from the vehicle, including radar, lidar, GPS, ultrasound and accelerometer; by integrating various sensor data, a more accurate three-dimensional environmental model is generated to help the system make more accurate remote driving strategies in complex environments; the extended Kalman filter EKF and point cloud fusion algorithm ICP are used to achieve the fusion of different sensor data; the fusion steps are: 31) preliminary filtering of the lidar point cloud data and GPS data to remove noise; 32) estimation of the vehicle's position and speed through EKF; 33) use the ICP algorithm to align the point cloud data at different time points to obtain more accurate environmental perception.

[0039] The natural language processing NLP unit S103 is used to parse the voice input and text information of passengers and vehicles, and generate corresponding driving instructions according to driving needs; the natural language processing NLP unit S103 is also used to process the alarm information issued by the vehicle and automatically trigger the remote driving takeover procedure; the BERT model is used to process natural language. The BERT model performs language understanding through the Transformer architecture. It can process the voice instructions of passengers and the alarm text information of the vehicle and generate corresponding instruction feedback; the steps for processing natural language through the BERT model are as follows: 41) The passenger and vehicle voice data are preprocessed, including: feature extraction and speech recognition conversion into text; 42) The text is input into the BERT model and encoded through a multi-layer Transformer to capture contextual semantics; 43) The BERT model outputs the parsed driving needs and triggers corresponding driving instructions.

[0040] The real-time remote driving takeover module S200 is used to take over the control of the vehicle in complex and emergency situations; the real-time remote driving takeover module S200 can handle scenarios that the vehicle's automatic driving cannot cope with, including extreme weather, complex traffic conditions and emergencies. The cloud-based intelligent driving control system generates a takeover strategy after analyzing multimodal data, and adjusts the vehicle's parameters including speed, direction and acceleration to ensure safe driving of the vehicle; the real-time remote driving takeover module S200 includes: an emergency takeover trigger unit S201, an intelligent takeover strategy generation unit S202 and a feedback and adjustment unit S203; wherein the emergency takeover trigger unit S201 is used to automatically send a takeover request to the cloud when the vehicle detects unpredictable environmental conditions, including: such as sensor failure, system failure and extreme weather; the emergency takeover trigger unit S201 is provided with a takeover trigger mechanism, and the takeover trigger mechanism is based on the takeover strategy of the DQN reinforcement learning algorithm; the DQN is an algorithm that combines deep learning and reinforcement learning, which aims to solve complex decision-making problems, and is particularly suitable for decision-making problems in high-dimensional state space. The steps of activating the takeover trigger mechanism through DQN are: 51) defining the state s of the vehicle in different environments t , including vehicle speed, acceleration, and sensor readings; 52) In each state, the vehicle selects action a t , including requesting to take over and continuing autonomous driving; 53) According to the execution of the action, the vehicle receives a reward of r t , including avoiding accidents and improving driving efficiency; 54) Use Q-Learning to update the vehicle's decision network. The Q-Learning is the basis of DQN. Q-Learning is a model-free reinforcement learning algorithm that helps the vehicle choose the optimal action in a specific state by learning an action value function Q value. The formula of the action value function Q value is:

[0041] Q(s t , a t )=Q(s t , a t )+α[r t +γmax at+1 Q(s t+1 , a t+1 )-Q(s t , a t )]

[0042] Among them, α is the learning rate, r t To perform action a t The immediate reward obtained after, γ is the discount factor, the Q(s t , a t ) is the updated Q value; Q(s t , a t ) is in st In the state, select action a t The old Q value of s t+1 Indicates the new state reached after executing the action; max at+1 Q(s t+1 , a t+1 ) indicates that in the new state s t+1 Next, select the action that maximizes the Q value among all actions.

[0043] The intelligent takeover strategy generation unit S202 generates the optimal vehicle control strategy based on cloud multimodal data analysis, and sends it to the vehicle through a low-latency communication link to take over the control functions including vehicle acceleration, braking and steering; the intelligent takeover strategy generation unit S202 is mainly used to generate remote takeover strategies, and generates remote takeover control strategies through the MPC model; the MPC is an optimization algorithm for dynamic system control, suitable for multi-step prediction and real-time optimization; the MPC steps are:

[0044] 62) Establish a dynamic model of the vehicle and input the vehicle state, including: vehicle speed, position and steering angle; 62) Predict the future state based on the current state within a certain prediction time window, and optimize the control variables, including: acceleration, braking and steering; 63) Send the optimized control strategy to the vehicle.

[0045] The MPC optimization formula is:

[0046]

[0047] Among them, x(k+1) is the predicted vehicle state, x ref is the target state, u k is the control input, Q and R are weight matrices, ||·|| represents the norm, By adjusting the control input u k To find the control strategy that minimizes the objective cost function.

[0048] The feedback and adjustment unit S203 is used to receive real-time feedback on the vehicle's status and continuously adjust the takeover strategy according to environmental changes to ensure that the vehicle can safely respond to emergencies.

[0049] Among them, the advance prediction and risk avoidance module S300 is used to predict in advance the risks that the vehicle will face in a period of time in the future, and based on the predicted risks, when it is judged that the predicted risks are greater than the risk threshold, a plan to avoid the risks is generated; the risk avoidance plan includes parking, deceleration, changing lanes and re-planning the route; the future period of time refers to a time period that the cloud automatically adjusts according to the vehicle speed; the advance prediction and risk avoidance module S300 is a supplement to the real-time remote driving takeover module S200 to predict the vehicle state; the advance prediction and risk avoidance module S300 includes a prediction unit S301 for the line of sight of the overtaking vehicle and a prediction unit S302 for the line of sight of the vehicle; the prediction of the line of sight of the overtaking vehicle refers to the prediction basis including information other than the images and videos collected by the vehicle, and the information other than the images and videos collected by the vehicle is obtained through the cloud system, including other vehicles and pedestrians that block the target vehicle from obtaining information due to obstacles; the prediction of the line of sight of the overtaking vehicle is predicted using a long short-term memory neural network LSTM, and the prediction steps are: 71) first First, extract the target vehicle information and other target information of the overtaking vehicle line of sight in the cloud system in history, the other targets include vehicles and pedestrians; extract the speed, speed direction, acceleration and acceleration direction information of the target vehicle, extract the speed, speed direction, acceleration and acceleration direction information of other targets, extract the distance information of the target vehicle and other targets, and extract the intersection point information where the target vehicle and other targets meet; the intersection point information includes the distance between the intersection point and the target vehicle and the position information relative to the target vehicle; the direction is to establish a rectangular coordinate system with the target vehicle as the origin, and then determine the speed direction, acceleration direction and position information of the intersection point relative to the target vehicle of different targets; 72) The target vehicle information and intersection point information extracted in 71) are used as the first row of the matrix, the extracted other target information and intersection point information are used as the second row of the matrix, and the distance information of the target vehicle and other targets is used as the third row of the matrix, that is, a matrix with three rows is formed, and the matrix is ​​set to X i , i refers to the extracted i-th group of target vehicle data information; 73) The distance when the target vehicle meets other targets is set to Y i , i represents the i-th group of target vehicles and other targets, and X i and Y i The combination of the predictions of the overtaking vehicle's sight line {X i , Y i}; 74) {X i , Y i} Input long short-term memory neural network LSTM, X i As the input of the LSTM neural network, Y iThe long short-term memory neural network LSTM is trained as the output of the LSTM neural network, and the steps of LSTM data preprocessing, defining the model architecture, training the model, adjusting the parameters and evaluating the model are performed. After the training is completed, the LSTM neural network model of the overtaking vehicle line of sight is obtained; 75) extracting the real-time target vehicle information and other target information, and organizing the target vehicle information and other target information into a matrix X according to steps 71) and 72). i data format, and input the sorted information into the LSTM neural network model of the overtaking vehicle's line of sight after preprocessing to obtain the output result, and determine whether to start the risk avoidance module according to whether the output result is within the threshold range; the threshold range means that the predicted shortest distance when the target vehicle and other targets meet is greater than the dangerous distance, and the dangerous distance is set to 3 meters according to traffic safety regulations; 76) If the output result is no longer within the threshold range, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit S202 of the real-time remote driving takeover module S200, and the optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit S202, and the risk avoidance module issues a risk avoidance instruction to the vehicle, and the vehicle executes operations including: deceleration, lane change and detour.

[0050] The prediction of vehicle sight lines is also carried out using the long short-term memory neural network LSTM model. The difference is that in constructing the combination {X i , Y i}, X i A matrix consisting of a target vehicle and multiple other targets is constructed, that is, the first row of the matrix is ​​the attribute of the target vehicle, and the following rows are the attributes corresponding to other targets, and the attributes of the distance to the target vehicle are added after the attributes of other targets, respectively. The intersection information of the target vehicle is rewritten as the average value of the intersection information with other targets, and the distance information of the target vehicle is added as the average value of the distances with other targets; and Y i Constructed into classified data, the vehicle is in the matrix X i The accident is classified as 1, and the vehicle is in the matrix X i If no accident occurs, it is classified as 0; the neural network LSTM model is trained through the set combination, and the trained LSTM model is used to determine in real time whether the target vehicle is at risk. If there is a risk, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit S202 of the real-time remote driving takeover module S200. The optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit S202, and the risk avoidance instruction is issued to the vehicle through the risk avoidance module, and the vehicle executes operations including deceleration, lane changing and detour.

[0051] It should be pointed out that the reason for dividing the prediction module into the prediction unit S301 for the overtaking vehicle's line of sight and the prediction unit S302 for the vehicle's line of sight is that during the driving process of the vehicle, the probability of the risk of other targets that the vehicle cannot detect and other targets that the vehicle detects is very different; for other targets that the vehicle can detect, the autonomous driving vehicle can make advance predictions based on its own intelligent driving algorithm and make self-adjustments to reduce the probability of risk; but for other targets that the vehicle cannot detect, since obstacles block the vehicle's line of sight, the vehicle cannot make predictions in advance, thereby reducing the risk; therefore, it is necessary to divide the prediction of vehicle driving risks into two parts: the prediction of the overtaking vehicle's line of sight and the prediction of the vehicle's line of sight; the present invention also distinguishes between the prediction of the overtaking vehicle's line of sight and the prediction of the vehicle's line of sight when extracting samples, so that the prediction results will be more accurate, which is convenient for the cloud to process the prediction results and generate corresponding driving strategies.

[0052] Among them, the cloud-based intelligent driving coordination and optimization module S400 is used to coordinate the cluster operation, path optimization and coordinated driving of multiple autonomous driving vehicles; the cloud-based intelligent driving coordination and optimization module S400 can dynamically assign tasks to multiple vehicles to ensure that each vehicle effectively utilizes road resources when performing pick-up and drop-off tasks, reduces congestion and optimizes overall travel efficiency; the cloud-based intelligent driving coordination and optimization module S400 includes a task allocation unit S401 and a coordinated driving unit S402, and the task allocation unit S401 is used to dynamically assign pick-up and drop-off tasks to multiple autonomous driving vehicles according to vehicle status, real-time traffic information and passenger needs; the cloud-based system is scheduled according to environmental changes and task priorities to ensure the shortest path, minimum energy consumption and maximum vehicle utilization; A ＊ The algorithm combines deep reinforcement learning to optimize the path; through A ＊ The algorithm guides the search process to find the optimal path, and deep reinforcement learning is used in A ＊ The optimal path found by the algorithm is dynamically adjusted according to the dynamically changing environment; the cooperative driving unit S402 is used to coordinate the cooperative driving of multiple vehicles on highways and urban roads, ensure the safe distance, speed matching and path optimization between the fleets, reduce mutual interference between vehicles, and improve overall efficiency; the cooperative algorithm based on V2X communication and Byzantine consensus protocol realizes multi-vehicle cooperative driving, the steps are: 81) vehicles share data including their location information and speed through V2X communication; 82) the cloud calculates the optimal fleet driving path and safe distance through the Byzantine consensus algorithm; 83) vehicles perform cooperative driving according to cloud instructions; the Byzantine consensus protocol formula is:

[0053] x i (t+1)=∑ j∈N(i) a ij xj (t)

[0054] Among them, x j (t) represents the state of node j at time t, x i (t+1) represents the state of node i at time t+1 obtained by summation operation; N(i) represents the set of neighboring vehicles of i, a ij represents the weight coefficient, and ∑ represents the summation operation.

[0055] Among them, the multimodal fault diagnosis and remote repair module is used to monitor the fault data uploaded by the vehicle in real time, including sensor failure and system anomalies, and diagnose the fault through the cloud multimodal large model; the multimodal fault diagnosis and remote repair module S500 supports remote repair and system restart, and generates alternative driving solutions to help the vehicle complete the task, ensuring that the vehicle can continue to operate safely under fault conditions; the multimodal fault diagnosis and remote repair module S500 includes: a fault prediction unit S501 and a remote repair and alternative solution generation unit S502; the fault prediction unit S501 is used to predict possible system faults or sensor failures based on historical vehicle operation data and sensor data, and give early warnings; it adopts a CNN-LST-based remote repair and alternative solution generation unit S502; the multimodal fault diagnosis and remote repair module S500 can be used to predict possible system faults or sensor failures based on historical vehicle operation data and sensor data, and give early warnings ... The fault detection algorithm of the M network is used to predict faults, and the prediction steps are: 91) Use CNN to extract multimodal data, including spatial features of images and sensor data; 92) Use LSTM to process time series data and detect anomalies in sensor data; 93) Output the fault detection results and trigger the repair process; The remote repair and alternative solution generation unit S502 is based on the strategy testing and alternative solutions of the autonomous driving simulation platform; First, simulate the vehicle failure situation in the cloud, and test different repair strategies through simulation, including restarting the system and adjusting the sensor configuration; Second, if the repair cannot be completed quickly, generate the used driving plan, such as reducing the functional complexity of the driving system by disabling certain sensors to ensure that the basic safe driving function of the vehicle is still effective.

[0056] The real-time communication and enhanced interaction module S600 is used to establish a high-speed and low-latency communication link between the cloud and the vehicle to ensure the real-time transmission and execution of remote driving instructions; the real-time communication and enhanced interaction module S600 also supports interaction with passengers in the vehicle to provide real-time feedback on vehicle status, fault repair progress and driving control; the real-time communication and enhanced interaction module S600 includes: a 5G satellite communication unit S601 and a passenger interaction and feedback unit S602; the 5G satellite communication unit S601 is used to ensure that the information transmission between the vehicle and the cloud has low latency and high bandwidth characteristics when remote driving takes over and cloud-assisted control; the passenger interaction and feedback unit S602 provides passengers with real-time driving status, system failures and estimated arrival time information through smartphone applications and vehicle-mounted interfaces, and allows passengers to make requests or adjust the itinerary in an emergency; the 5G satellite communication unit S601 and the passenger interaction and feedback unit S602 are based on adaptive TCP communication control, and the low-latency communication architecture is based on the adaptive TCP congestion control protocol, and the transmission efficiency of data packets is controlled by adjusting the congestion window size.

[0057] The cloud-based multimodal large model analysis and control module S100, the real-time remote driving takeover module S200, the advance prediction and risk avoidance module S300, the cloud-based intelligent driving coordination and optimization module S400, the multimodal fault diagnosis and remote repair module S500 and the real-time communication and enhanced interaction module S600 interact and collaborate with each other, for example: the collection of multimodal data of automatic vehicles and the fusion of the cloud-based system algorithm; in the system, the vehicle's cameras, lidar, GPS and sensors will continuously collect multimodal data, and the collected data will first be pre-processed and fused through the cloud-based multimodal large model analysis and control module, and then transmitted to other modules, such as the real-time remote driving takeover module and the cloud-based intelligent driving coordination and optimization module; for example, the cloud-based multimodal large model The cloud system collaborates with the remote driving takeover module. When the vehicle encounters complex or extreme situations, the cloud system will generate a remote takeover strategy through multimodal data analysis and the MPC model. This strategy is sent to the vehicle through the real-time communication module and executed by the real-time remote driving takeover module. Finally, the multimodal fault diagnosis and remote repair module and the remote driving takeover module are linked. When the vehicle has a sensor failure or system abnormality, the cloud will perform fault diagnosis through the multimodal fault diagnosis and remote repair module and link it with the remote driving takeover module. After the fault diagnosis, the system will generate an alternative driving strategy, remotely take over control of the vehicle, and use fewer functions to ensure vehicle safety. Through the interaction and collaboration of different modules, the cloud-based multimodal large model system can jointly complete the monitoring and control of remote autonomous driving vehicles.

[0058] like Figure 2The schematic diagram of the cloud-based intelligent driving control process based on the multimodal large model of the present invention is shown in the flowchart. In the flowchart, the vehicle first collects vehicle multimodal data, and sends the collected multimodal data including image information, video information, voice information and text information to the cloud-based multimodal large model analysis and control module S100. The cloud-based multimodal large model analysis and control module S100 analyzes the multimodal data, and respectively detects the road environment, pedestrians and obstacles through YOLOv5 image and video analysis; Transformer perceives the environment around the image, converts the two-dimensional image into three-dimensional, and generates a three-dimensional bird's-eye view; through the fusion of multimodal data, the data transmitted by different devices are integrated, including the fusion of lidar, GPS and sensor data, etc., to achieve the relationship combination between different data; the voice command or alarm text is parsed through NLP voice and text processing; by training the LSTM neural network model and predicting through the LSTM neural network, an LSTM neural network model for the line of sight of the overtaking vehicle and an LSTM neural network model for the line of sight of the vehicle are established respectively; then, the cloud-based multimodal large model analysis and control module S100 generates control strategies on the cloud through different methods of analyzing the corresponding data; the generated cloud-based control strategies include the control strategies of the real-time remote driving takeover module S200, the advance prediction and risk avoidance module S300, the cloud-based intelligent driving coordination and optimization module S400 and the multimodal fault diagnosis and remote repair module S500.

[0059] Among them, the control strategy of the real-time remote driving takeover module S200 generated in the cloud is: generate a strategy for taking over vehicle control through the MPC prediction system, and send the control strategy to the autonomous driving vehicle; the control strategy of the advance prediction and risk avoidance module S300 is to predict vehicle risks in advance through the LSTM model, including the vehicle risks of overtaking vehicles and the vehicle risks of vehicles in the vehicle's line of sight, and then issue risk avoidance instructions to the vehicle through the risk avoidance module; the control strategy of the cloud-based intelligent driving coordination and optimization module S400 is to achieve multi-vehicle coordinated adjustment through multi-vehicle collaborative driving and path planning to ensure the safety of multi-vehicle driving; the control strategy of the multi-modal fault diagnosis and remote repair module S500 is fault diagnosis CNN-LSTM and remote repair, which performs remote fault repair on the autonomous driving vehicle or generates alternative driving plans.

[0060] It should be pointed out that from S100 to S500, they are all closely connected with the real-time communication and enhanced interaction module. The cloud system sends data and control instructions to the autonomous driving vehicle through 5G satellite communication; the autonomous driving vehicle feeds back driving data to the cloud in real time; at the same time, the autonomous driving vehicle transmits the passengers' voice and text information to the cloud, which is analyzed and fed back by the cloud, enhancing the interaction with the passengers and increasing the user experience on the basis of driving safety.

Claims

1. A cloud-based intelligent driving control system based on a multimodal large model, characterized in that: The system comprises: Cloud-based multimodal large model analysis and control module; Real-time remote driving takeover module; Module for predicting and avoiding risks in advance; Cloud-based intelligent driving coordination and optimization module; Multimodal fault diagnosis and remote repair module; Real-time communication and enhanced interaction module; The cloud-based multimodal large model analysis and control module is used to process the multimodal data uploaded by the autonomous driving vehicle, analyze the vehicle's current environment, generate a remote driving control strategy, and send control instructions to the vehicle through low-latency communication; The real-time remote driving takeover module is used to take over the control of the vehicle in complex and emergency situations; the real-time remote driving takeover module can handle scenarios that the vehicle's automatic driving cannot cope with, including extreme weather, complex traffic conditions and emergencies. The cloud-based intelligent driving control system generates a takeover strategy after analyzing multimodal data, and adjusts the vehicle's parameters including speed, direction and acceleration to ensure safe driving of the vehicle; The advance prediction and risk avoidance module is used to predict in advance the risks that the vehicle will face in the future, and generate a risk avoidance plan when the predicted risk is greater than the risk threshold according to the predicted risk; the risk avoidance plan includes parking, deceleration, lane change and re-planning the route; the future period refers to a time period that is automatically adjusted by the cloud according to the vehicle speed; The cloud-based intelligent driving coordination and optimization module is used to coordinate the cluster operation, path optimization and coordinated driving of multiple autonomous driving vehicles; the cloud-based intelligent driving coordination and optimization module can dynamically allocate tasks for multiple vehicles to ensure that each vehicle effectively utilizes road resources when performing pick-up and drop-off tasks, reduce congestion and optimize overall travel efficiency; The multimodal fault diagnosis and remote repair module is used to monitor the fault data uploaded by the vehicle in real time, including sensor failure and system anomalies, and diagnose the fault through a large multimodal model in the cloud; the multimodal fault diagnosis and remote repair module supports remote repair and system restart, as well as generates alternative driving solutions to help the vehicle complete the task, ensuring that the vehicle can continue to operate safely under fault conditions; The real-time communication and enhanced interaction module is used to establish a high-speed and low-latency communication link between the cloud and the vehicle to ensure the real-time transmission and execution of remote driving instructions; the real-time communication and enhanced interaction module also supports interaction with passengers in the vehicle, providing real-time feedback on vehicle status, fault repair progress and driving control.

2. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The cloud-based multimodal large model analysis and control module includes: an image and video processing unit, a sensor data fusion unit and a natural language processing (NLP) unit; the image and video processing unit is used to process the real-time video stream and static image data uploaded by the vehicle, and identify obstacles, road signs and pedestrian elements around the vehicle through computer vision technology; the image and video processing unit includes target recognition and three-dimensional images and videos; target recognition of image and video information is achieved through YOLOv5, which is a deep learning model for object detection. It performs image processing through convolutional neural network (CNN) and can detect and identify targets in real time; the steps of performing target recognition on image and video information through the YOLOv5 target detection model are: extracting features from the input image through multiple convolutional layers; using the features output by the convolutional layer The image generates a candidate region, which is also called an anchor box; the target in the anchor box is classified by a classifier, and the size of the anchor box is adjusted; non-maximum suppression (NMS) is used to eliminate duplicate boxes; the non-maximum suppression (NMS) is a commonly used post-processing technique for eliminating duplicate bounding boxes in target detection. The steps of eliminating duplicate bounding boxes by non-maximum suppression (NMS) are as follows: 11) First, all detected boxes are sorted according to the target confidence; 12) The box with the highest confidence is selected as the reserved box; 13) The overlap between the current reserved box and other boxes is calculated using IoU, where IoU refers to the bounding box error calculated by loss, and the other boxes are boxes outside the reserved box; 14) If the IoU of other boxes exceeds a preset threshold, they are removed from the candidate boxes; 15) Steps 12) to 14) are repeated until all boxes are processed; The BEV Transformer model is used to realize three-dimensional image and video information; the BEV Transformer model is used to convert the image from the two-dimensional camera perspective into a three-dimensional bird's-eye view to complete the three-dimensionalization of the image and video information; the BEV Transformer model uses the Transformer architecture to learn the spatial relationship of the image through the self-attention mechanism and generate a three-dimensional view of the vehicle's surrounding environment; the steps of the BEV Transformer model to realize three-dimensional image and video information are: 21) input images from different perspectives; 22) map the features of the two-dimensional image to the three-dimensional space through position encoding; 23) use the self-attention mechanism to calculate the correlation between different perspectives; 24) output the three-dimensional environment representation around the vehicle; the formula for learning the spatial relationship through the self-attention mechanism is: Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix; The sensor data fusion unit is used to integrate multiple sensor data from the vehicle, including radar, lidar, GPS, ultrasound and accelerometer; by integrating multiple sensor data, a more accurate three-dimensional environmental model is generated to help the system make more accurate remote driving strategies in complex environments; the extended Kalman filter EKF and the point cloud fusion algorithm ICP are used to achieve the fusion of different sensor data; the fusion steps are: 31) preliminary filtering of the lidar point cloud data and GPS data to remove noise; 32) the position and speed of the vehicle are estimated by EKF; 33) the point cloud data at different time points are aligned using the ICP algorithm to obtain more accurate environmental perception; The natural language processing (NLP) unit is used to parse the voice input and text information of passengers and vehicles, and generate corresponding driving instructions according to driving needs; the natural language processing (NLP) unit is also used to process the alarm information issued by the vehicle and automatically trigger the remote driving takeover procedure; the BERT model is used to process natural language. The BERT model performs language understanding through the Transformer architecture. It can process the voice instructions of passengers and the alarm text information of the vehicle and generate corresponding instruction feedback; the steps for processing natural language through the BERT model are as follows: 41) the passenger and vehicle voice data are preprocessed, including: feature extraction and speech recognition conversion into text; 42) the text is input into the BERT model, encoded through a multi-layer Transformer to capture contextual semantics; 43) the BERT model outputs the parsed driving needs and triggers corresponding driving instructions.

3. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The real-time remote driving takeover module includes: an emergency takeover trigger unit, an intelligent takeover strategy generation unit and a feedback and adjustment unit; wherein, the emergency takeover trigger unit is used to automatically send a takeover request to the cloud when the vehicle detects unpredictable environmental conditions, including: sensor failure, system failure and extreme weather; the emergency takeover trigger unit is provided with a takeover trigger mechanism, and the takeover trigger mechanism is based on the takeover strategy of the DQN reinforcement learning algorithm; the DQN is an algorithm that combines deep learning and reinforcement learning, and is designed to solve complex decision-making problems, and is particularly suitable for decision-making problems in high-dimensional state space. The steps of activating the takeover trigger mechanism through DQN are: 51) defining the state s of the vehicle in different environments t , including vehicle speed, acceleration, and sensor readings; 52) In each state, the vehicle selects action a t , including requesting to take over and continuing autonomous driving; 53) According to the execution of the action, the vehicle receives a reward of r t , including avoiding accidents and improving driving efficiency; 54) Use Q-Learning to update the vehicle's decision network. The Q-Learning is the basis of DQN. Q-Learning is a model-free reinforcement learning algorithm that helps the vehicle choose the optimal action in a specific state by learning an action value function Q value. The formula of the action value function Q value is: Q(s t ,a t )=Q(s t ,a t )+α[r t +γmax at+1 Q(s t+1 ,a t+1 )-Q(s t ,a t )] Among them, α is the learning rate, r t To perform action a t The immediate reward obtained after, γ is the discount factor, the Q(s t , a t ) is the updated Q value; Q(s t , a t ) is in s t In the state, select action a t The old Q value of s t+1 Indicates the new state reached after executing the action; max at+1 Q(s t+1 , a t+1 ) indicates that in the new state s t+1 Next, select the action that maximizes the Q value among all actions; The intelligent takeover strategy generation unit generates the optimal vehicle control strategy based on cloud multimodal data analysis, and sends it to the vehicle through a low-latency communication link to take over the control functions of vehicle acceleration, braking and steering; the intelligent takeover strategy generation unit is mainly used to generate remote takeover strategies, and generates remote takeover control strategies through the MPC model; the MPC is an optimization algorithm for dynamic system control, suitable for multi-step prediction and real-time optimization; the MPC steps are: 61) Establish a dynamic model of the vehicle and input the vehicle state, including vehicle speed, position and steering angle; 62) Predict the future state based on the current state within a certain prediction time window and optimize the control variables, including acceleration, braking and steering; 63) Send the optimized control strategy to the vehicle; The MPC optimization formula is: Among them, x(k+1) is the predicted vehicle state, x ref is the target state, u k is the control input, Q and R are weight matrices, ||·|| represents the norm, By adjusting the control input u k To find the control strategy that minimizes the objective cost function; The feedback and adjustment unit is used to receive real-time status feedback of the vehicle and continuously adjust the takeover strategy according to environmental changes to ensure that the vehicle can safely respond to emergencies.

4. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The advance prediction and risk avoidance module includes: the advance prediction and risk avoidance module is a supplement to the real-time remote driving takeover module to predict the vehicle state; the advance prediction and risk avoidance module includes a prediction unit for the overtaking vehicle line of sight and a prediction unit for the vehicle line of sight; the prediction of the overtaking vehicle line of sight refers to the prediction basis including information other than the image and video collected by the vehicle, and the information other than the image and video collected by the vehicle is obtained through the cloud system, including other vehicles and pedestrians that block the target vehicle from obtaining information due to obstacles; the prediction of the overtaking vehicle line of sight is predicted using a long short-term memory neural network LSTM, and the prediction steps are: 71) First, extract the target vehicle information and other target information of the overtaking vehicle line of sight in the history of the cloud system, and the other targets include vehicles and pedestrians; extract the speed, speed direction, acceleration size and The acceleration direction information is used to extract the speed magnitude, speed direction, acceleration magnitude and acceleration direction information of other targets, extract the distance information between the target vehicle and other targets, and extract the intersection point information where the target vehicle and other targets meet; the intersection point information includes the distance between the intersection point and the target vehicle and the position information relative to the target vehicle; the direction is to establish a rectangular coordinate system with the target vehicle as the origin, and then determine the speed direction, acceleration direction of different targets, and the position information of the intersection point relative to the target vehicle; 72) The target vehicle information and intersection point information extracted in 71) are used as the first row of the matrix, the other target information and intersection point information extracted are used as the second row of the matrix, and the distance information of the target vehicle and other targets is used as the third row of the matrix, that is, a matrix with three rows is formed, and the matrix is ​​set to X i , i refers to the extracted i-th group of target vehicle data information; 73) The distance when the target vehicle meets other targets is set to Y i , i represents the i-th group of target vehicles and other targets, and X i and Y i The combination of the predictions of the overtaking vehicle's sight line {X i , Y i }; 74) {X i , Y i } Input long short-term memory neural network LSTM, X i As the input of the LSTM neural network, Y i The long short-term memory neural network LSTM is trained as the output of the LSTM neural network, and the steps of LSTM data preprocessing, defining the model architecture, training the model, adjusting the parameters and evaluating the model are performed. After the training is completed, the LSTM neural network model of the overtaking vehicle line of sight is obtained; 75) extracting the real-time target vehicle information and other target information, and organizing the target vehicle information and other target information into a matrix X according to steps 71) and 72). i data format, and the sorted information is pre-processed and input into the LSTM neural network model of the overtaking vehicle line of sight to obtain the output result, and whether to start the risk avoidance module is determined according to whether the output result is within the threshold range; the threshold range means that the predicted shortest distance when the target vehicle and other targets meet is greater than the dangerous distance, and the dangerous distance is set to 3 meters according to the traffic safety regulations; 76) If the output result is no longer within the threshold range, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit of the real-time remote driving takeover module, and the optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit, and the risk avoidance instruction is issued to the vehicle through the risk avoidance module, and the vehicle executes operations including: deceleration, lane change and detour; The prediction of the vehicle sight line is also performed using the long short-term memory neural network LSTM model. The difference is that in constructing the combination {X i , Y i }, X i A matrix consisting of a target vehicle and multiple other targets is constructed, that is, the first row of the matrix is ​​the attribute of the target vehicle, and the following rows are the attributes corresponding to other targets, and the attributes of the distance to the target vehicle are added after the attributes of other targets, respectively. The intersection information of the target vehicle is rewritten as the average value of the intersection information with other targets, and the distance information of the target vehicle is added as the average value of the distances with other targets; and Y i Constructed into classified data, the vehicle is in the matrix X i The accident is classified as 1, and the vehicle is in the matrix X i If no accident occurs, it is classified as 0; the neural network LSTM model is trained through the set combination, and the trained LSTM model is used to determine in real time whether the target vehicle has risks. If there are risks, the risk avoidance module is started, and the risk avoidance module is connected to the intelligent takeover strategy generation unit of the real-time remote driving takeover module. The optimal vehicle control strategy is generated according to the analysis result of the intelligent takeover strategy generation unit, and the risk avoidance instruction is issued to the vehicle through the risk avoidance module, and the vehicle executes operations including deceleration, lane changing and detour.

5. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The cloud-based intelligent driving coordination and optimization module includes a task allocation unit and a coordinated driving unit. The task allocation unit is used to dynamically allocate the pick-up and drop-off tasks of multiple autonomous driving vehicles according to vehicle status, real-time traffic information and passenger needs; the cloud-based system performs scheduling according to environmental changes and task priorities to ensure the shortest path, minimum energy consumption and maximum vehicle utilization; A ＊ The algorithm combines deep reinforcement learning to optimize the path; through A ＊ The algorithm guides the search process to find the optimal path, and deep reinforcement learning is used in A ＊ The optimal path found by the algorithm is dynamically adjusted according to the dynamically changing environment; the cooperative driving unit is used to coordinate the cooperative driving of multiple vehicles on highways and urban roads, ensure the safe distance, speed matching and path optimization between the fleets, reduce mutual interference between vehicles, and improve overall efficiency; the cooperative algorithm based on V2X communication and Byzantine consensus protocol realizes multi-vehicle cooperative driving, the steps are: 81) vehicles share data including their location information and speed through V2X communication; 82) the cloud calculates the optimal fleet driving path and safe distance through the Byzantine consensus algorithm; 83) vehicles perform cooperative driving according to cloud instructions; the Byzantine consensus protocol formula is: x i (t+1)=∑ j∈N(i) a ij x j (t) Among them, x j (t) represents the state of node j at time t, x i (t+1) represents the state of node i at time t+1 obtained by summation operation; N(i) represents the set of neighboring vehicles of i, a ij represents the weight coefficient, and ∑ represents the summation operation.

6. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The multimodal fault diagnosis and remote repair module includes: a fault prediction unit and a remote repair and alternative solution generation unit; the fault prediction unit is used to predict possible system faults or sensor failures based on historical vehicle operation data and sensor data, and to issue early warnings; a fault detection algorithm based on a CNN-LSTM network is used to predict faults, and the prediction steps are: 91) using CNN to extract multimodal data, including spatial features of image and sensor data; 92) using LSTM to process time series data and detect anomalies in sensor data; 93) outputting fault detection results and triggering a repair process; the remote repair and alternative solution generation unit is based on strategy testing and alternative solutions of an autonomous driving simulation platform; first, simulating vehicle fault conditions in the cloud, and testing different repair strategies through simulation, including restarting the system and adjusting sensor configuration; second, if the repair cannot be completed quickly, a used driving solution is generated, such as reducing the functional complexity of the driving system by disabling certain sensors to ensure that the basic safe driving function of the vehicle is still valid.

7. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The real-time communication and enhanced interaction module includes: a 5G satellite communication unit and a passenger interaction and feedback unit; the 5G satellite communication unit is used to ensure that the information transmission between the vehicle and the cloud has low latency and high bandwidth characteristics when remote driving takes over and cloud-assisted control is used; the passenger interaction and feedback unit provides passengers with real-time driving status, system failures and estimated arrival time information through smartphone applications and in-vehicle interfaces, and allows passengers to make requests or adjust itineraries in emergency situations; the 5G satellite communication unit and the passenger interaction and feedback unit are based on adaptive TCP communication control, and the low-latency communication architecture is based on the adaptive TCP congestion control protocol, which controls the efficiency of data packet transmission by adjusting the congestion window size.

8. The cloud-based intelligent driving control system based on a multimodal large model according to claim 1 is characterized in that: The cloud-based multimodal large model analysis and control module, the real-time remote driving takeover module, the early prediction and risk avoidance module, the cloud-based intelligent driving coordination and optimization module, the multimodal fault diagnosis and remote repair module, and the real-time communication and enhanced interaction module interact and collaborate with each other, and through the interaction and collaboration of different modules, the cloud-based multimodal large model system monitors and controls the remote autonomous driving vehicle.

Images