Adaptive driving behavior evaluation and training system based on multi-modal large model

Abstract

The invention discloses an adaptive driving behavior evaluation and training system based on a multi-modal large model. Comprising a multi-mode driving data acquisition and situation awareness module, a data storage module, a driving behavior real-time evaluation and feedback module, an adaptive training strategy generation module, a cloud large model driven self-learning module and a driver psychological state monitoring and adjusting module. The driving behavior real-time evaluation and feedback module is used for carrying out real-time evaluation on the driving behavior of the driver through a machine learning evaluation large model and providing instant feedback; according to the invention, a multi-modal large model technology is combined, multi-modal data of a driving vehicle and a driver are collected in real time, personalized training tasks and learning strategies are generated according to changes of driving situations and real-time evaluation of driver performance, the training tasks are dynamically adjusted in different complex situations, and the training efficiency is improved. Training tasks are adjusted by analyzing the psychological state of the driver, and the system can provide detailed feedback for behaviors of the driver.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence driving training, and specifically refers to an adaptive driving behavior evaluation and training system based on a multimodal large model. Background Art

[0002] In recent years, significant progress has been made in the technology of multimodal large models. These models are able to process and understand data from different modalities, such as images, videos, audio, and text. Multimodal alignment enables them to perform various tasks more efficiently, including image classification, aligning text with the corresponding video, and speech detection. The development of these technologies has provided new perspectives and tools for the field of autonomous driving.

[0003] Traditional driving training and assessment systems usually rely only on human coaches and scheduled driving courses, and are unable to dynamically adjust in real time based on changes in the driving environment and driver behavior.

[0004] The development of multimodal large model technology has brought new opportunities and challenges to the field of driving training, especially in the field of adaptive driving behavior assessment and training. These technological advances have made it possible to build a platform that can understand and predict driving behavior, thereby improving the safety and efficiency of driving systems.

[0005] Existing driving assessment systems have difficulty providing real-time feedback, and cannot quickly and accurately generate personalized training tasks based on complex driving scenarios. They are unable to combine cloud platform technology to optimize the multimodal large models that driving training relies on, and lack the ability to adjust training tasks by analyzing the driver's psychological state. Summary of the invention

[0006] In order to solve the problems in the above-mentioned prior art that the driving assessment system is difficult to provide real-time feedback, cannot quickly and accurately generate personalized training tasks according to complex driving situations, and cannot combine cloud platform technology to optimize the multimodal large model on which driving training relies, the present invention proposes an adaptive driving behavior assessment and training system based on a multimodal large model to improve the above-mentioned problems.

[0007] The specific application is as follows:

[0008] An adaptive driving behavior assessment and training system based on a multimodal large model, including a multimodal driving data collection and situational awareness module, a data storage module, a driving behavior real-time assessment and feedback module, an adaptive training strategy generation module, a cloud-based large model-driven self-learning module, and a driver psychological state monitoring and adjustment module;

[0009] The multimodal driving data collection and situational awareness module is used to collect multimodal data of the vehicle and the driver and perceive the driving situation;

[0010] The data storage module is used to store the data collected by the multimodal driving data collection and situational awareness module;

[0011] The driving behavior real-time evaluation and feedback module is used to evaluate the driver's driving behavior in real time and provide immediate feedback through a machine learning evaluation model;

[0012] The adaptive training strategy generation module is used to generate personalized training tasks and learning strategies through deep learning training large models;

[0013] The cloud-based big model drives the self-learning module, which performs long-term tracking and self-learning on the machine learning evaluation big model and the deep learning training big model based on multimodal data through the cloud platform;

[0014] The driver psychological state monitoring and adjustment module is used to monitor the driver's physiological and psychological state and adjust the driving task and course content when necessary.

[0015] Further, the multimodal driving data acquisition and situational awareness module includes a vehicle-mounted multimodal data acquisition submodule and a driving situational awareness submodule;

[0016] The vehicle-mounted multimodal data acquisition submodule collects the driver's behavior, the driver's physiological data, the vehicle environment and the vehicle status in real time through the camera, LiDAR radar, GPS, seat sensor and eye tracker sensor;

[0017] The driving situation perception submodule perceives the current driving situation based on the data of the vehicle-mounted multimodal data acquisition submodule, and analyzes the driving difficulty in combination with real-time traffic conditions, weather data and external information, wherein the driving situation at least includes driving on urban roads, rural roads, rainy days and nighttime;

[0018] The data storage module includes a distributed storage unit, a data backup unit and a data recovery unit;

[0019] The distributed storage unit is used to perform distributed storage of the database on multiple servers to ensure data security and reliability;

[0020] The data backup unit is used to regularly back up data to prevent data loss;

[0021] The data recovery unit is used to quickly recover data when the data is lost or damaged.

[0022] Furthermore, the driving behavior real-time evaluation and feedback module includes a driving behavior evaluation submodule, a situational adaptability evaluation submodule, and a real-time feedback and suggestion submodule;

[0023] The driving behavior evaluation submodule evaluates the driver's performance in real time and identifies potential errors based on the driver's operating habits through a machine learning evaluation model, wherein the operating habits at least include steering, shifting, accelerating, and braking;

[0024] The situational adaptability evaluation submodule evaluates the driver's performance in different driving situations through the machine learning evaluation model to form an evaluation result, wherein the performance in different driving situations at least includes reaction speed and operation accuracy in bad weather, night driving and congested traffic;

[0025] The real-time feedback and suggestion submodule provides real-time feedback and suggestions during driving according to the evaluation results of the situational adaptability evaluation submodule. The real-time feedback and suggestions provided by the system during driving at least include correction and guidance through voice or visual prompts when the driver exceeds the speed limit, makes a wrong operation or reacts slowly.

[0026] Furthermore, the specific implementation process of the machine learning evaluation large model is as follows:

[0027] L1: Obtain the current camera image, LiDAR laser radar data, map image and voice data collected by the multimodal driving data collection and situational awareness module;

[0028] L2: Process the LiDAR data to obtain a two-dimensional LiDAR image, process the map image to obtain a two-dimensional grayscale image, fuse the image features of the camera image, the features of the two-dimensional LiDAR image, and the map features of the two-dimensional grayscale image to obtain the fusion features at the current moment;

[0029] L3: Calculate the Euclidean similarity between the fusion feature at the current moment and the fusion feature of the previous frame of interest, and determine whether the fusion feature at the current moment is the frame of interest based on the Euclidean similarity;

[0030] L4: If the fused feature at the current moment is a frame of interest, the pre-trained main model is used to process the fused feature at the current moment to obtain the fine-grained feature at the current moment, and then multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frame of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The main model uses a ResNet network;

[0031] L5: If the fused features at the current moment are not frames of interest, the pre-trained auxiliary model is used to process the fused features at the current moment to obtain the coarse-grained features at the current moment, and the coarse-grained features are transformed to obtain fine-grained features. Then, multiple two-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The auxiliary model uses the MobileNetV2 network.

[0032] L6: Use natural language technology to extract the speech features of the current speech data, and use the evaluation network model to fuse the evaluation features and speech features at the current moment to obtain the target evaluation result at the current moment.

[0033] Furthermore, the specific implementation of L2 includes:

[0034] Using the conversion relationship between the LiDAR radar coordinate system and the camera imaging coordinate system, the LiDAR laser radar data is projected onto the pixel two-dimensional plane to obtain a two-dimensional laser radar pixel. The characteristics of the two-dimensional laser radar pixel include: x, y, z, r, g and a; x, y, z are the three-dimensional coordinates of the pixel center point, r is the reflection coefficient of the LiDAR laser radar, g is the point frequency of the LiDAR laser radar, and a is the angular resolution of the LiDAR laser radar;

[0035] Extract image features of the camera image, the image features include resolution HW, frame rate F, red component R, green component G and blue component B;

[0036] Extracting image features of the map image, the map features including the vehicle longitude m and the vehicle latitude n;

[0037] The fusion features at the current moment include: resolution HW, frame rate F, red component R, green component G, blue component B, x, y, z, LiDAR radar reflection coefficient r, LiDAR radar point frequency g, LiDAR radar angular resolution a, vehicle longitude m and vehicle latitude n;

[0038] In the L3, calculating the Euclidean similarity between the fused feature at the current moment and the fused feature of the previous frame of interest, and judging whether the fused feature at the current moment is the frame of interest according to the Euclidean similarity includes:

[0039] Calculate the Euclidean similarity P between the fusion feature at the current moment and the fusion feature of the previous frame of interest t :

[0040]

[0041] Among them, H t H is the three-dimensional vector after caching the fusion features at the current moment.last_i is the three-dimensional vector after caching the fusion features of the previous frame of interest, d(H t ,H last_i ) are two vectors H t and H last_i The Euclidean distance between

[0042] Determine the Euclidean similarity P t Is it greater than a preset threshold? If so, the fused feature at the current moment is a non-interesting frame. If not, the fused feature at the current moment is an interesting frame.

[0043] In the L4, multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest with the fine-grained features of the current moment to obtain the evaluation features of the current moment, including:

[0044] L41: Get the fine-grained features F at the current moment output by the main model t ;

[0045] L42: Calculate the current time The fine-grained feature map at position p2 and the fine-grained feature map M of the cached frame of interest at position p2 p2 The association matrix K(d1,d2) is:

[0046]

[0047] in, The fine-grained feature map Transformed into a three-dimensional matrix, β2(M d2 ) The fine-grained feature map M d2 Transformed into a three-dimensional matrix, is a three-dimensional matrix, and the fine-grained features of the cached frame of interest are the fine-grained features of the previous frame of interest;

[0048] L43: Calculate the cached feature map aligned to the d1 position

[0049]

[0050] Cache feature map With fine-grained feature maps Fusion is performed to obtain the evaluation features of the d1 position

[0051]

[0052] Among them, β v(.) is a 3*3 convolution operation, concat(.) represents the connection operation in the channel dimension, Q(.) represents the convolution operation of three consecutive three-dimensional convolution kernels, and the size of the three consecutive three-dimensional convolution kernels is 3×3;

[0053] L44: Evaluate features of all d1 positions Evaluative features that make up the present moment

[0054] The calculation method for obtaining the evaluation features at the current moment in L5 is the same as the calculation method for obtaining the evaluation features at the current moment in L4.

[0055] Further, the adaptive training strategy generation module includes a personalized training task push submodule and a dynamic training adjustment submodule;

[0056] The personalized training task push submodule evaluates the driver's profile through deep learning and big data mining technology, and builds a deep learning training model;

[0057] The dynamic training adjustment submodule dynamically adjusts the difficulty and content of the training tasks according to the real-time performance of the driver, optimizes the deep learning training model, and adjusts the training content of the driver at different learning stages through the output of the deep learning training model.

[0058] Furthermore, the deep learning training large model implementation process is:

[0059] S1: Acquire historical behavior data of the driver, and obtain a target training model according to the historical behavior data;

[0060] S2: collecting target operation data, target physiological data, target vehicle status data and target environment data of the driver in real time through the multimodal driving data collection and situational awareness module;

[0061] S3: Processing the target operation data, the target physiological data, the target vehicle status data and the target environment data to obtain target integrated data;

[0062] S4: according to the target integrated data, multimodal feature fusion and deep learning methods are used to judge the intention, cognitive state and visual attention of the driver to obtain a target judgment result;

[0063] S5: issuing a target voice reminder to the driver and displaying target reminder information on a vehicle-mounted display according to the target judgment result and the target training model;

[0064] S6: receiving target feedback data of the driver on the target voice reminder and the target reminder information to evaluate the training effect and obtain target training evaluation data;

[0065] S7: adjusting the target training model according to the target training evaluation data, wherein the output of the target training model is used to dynamically adjust the difficulty and content of the driver training task.

[0066] Furthermore, the specific implementation process of the target training model in S1 is as follows:

[0067] S11: Build a driver profile based on the driver’s historical behavior data and basic information;

[0068] S12: Based on the driver profile, a target training model is constructed through big data mining technology and deep learning technology;

[0069] S13: The output of the target training model is real-time training driving evaluation, personalized course recommendation and dynamic adjustment of the difficulty and content of driver training tasks. The input of the target training model is the driver's historical behavior data and real-time target integration data. The real-time training driving evaluation includes acceleration driving, deceleration driving, stable driving and lane change driving.

[0070] Furthermore, the cloud-based large model-driven self-learning module includes a learning progress tracking and evaluation submodule and a cloud-based self-learning model optimization submodule;

[0071] The learning progress tracking and evaluation submodule continuously tracks the driver's learning progress, evaluates his performance, analyzes his progress or bottlenecks, and provides periodic summaries and feedback;

[0072] The cloud-based self-learning model optimization submodule continuously learns a large amount of driving data through the cloud-based multimodal large model to optimize the machine learning evaluation large model and the deep learning training large model;

[0073] The cloud-based multimodal large model uses a variational autoencoder (VAE) to continuously learn and optimize the driver behavior model. VAE achieves reconstruction and generation of driver behavior by maximizing likelihood estimation.

[0074] The cloud-based multimodal large model utilizes a federated learning architecture to enhance the self-learning capability of the cloud platform. Local models of different vehicles are shared to the cloud platform through gradient updates without transmitting the original data.

[0075] Furthermore, the driver's psychological state monitoring and adjustment module includes a physiological data collection submodule, an emotion analysis and state adjustment submodule, and a fatigue driving warning submodule;

[0076] The physiological data collection submodule collects the driver's physiological state data in real time through seat sensors, heart rate monitoring sensors, and eye tracking sensors, and uses the physiological state data to assess the driver's psychological state;

[0077] The emotion analysis and state adjustment submodule is used to dynamically adjust the difficulty and rhythm of the driving training task through the deep emotion analysis model, and adaptively adjust the driver's training task through the reinforcement learning algorithm to ensure that the task difficulty matches the driver's emotional state;

[0078] The dynamic adjustment of the difficulty and rhythm of the driving training task at least includes that when the driver is in a state of tension or fatigue, the system will reduce high-intensity training tasks and provide a more relaxed driving course. The deep sentiment analysis model adopts a BERT model. The BERT model analyzes the driver's psychological state by extracting physiological data features. The input data of the BERT model is the driver's voice, heart rate and skin electrical response. The multimodal analysis formula of the BERT model is:

[0079] Y = softmax(W t .BERT(x t )+W s .s t +b),

[0080] Among them, x t The text data of speech conversion, s t is the physiological data corresponding to heart rate and skin electrical response, W t and W s are the weights corresponding to text data and physiological data, b is the correction coefficient, and softmax is the activation function;

[0081] The formula for dynamically adjusting the difficulty of the driving training task is:

[0082] D ajusted =f(D current , S fatigue ),

[0083] Among them, D ajusted is the adjusted driving task, S fatigue is the driver’s fatigue level;

[0084] The fatigue driving warning submodule is used to detect driver fatigue and attention. If the fatigue and attention exceed the preset threshold, the system will ensure driving safety through voice prompts and active intervention, and the active intervention includes automatic driving takeover.

[0085] The beneficial effects of the adaptive driving behavior evaluation and training system based on a multimodal large model of the present invention are as follows:

[0086] The present invention collects multimodal data of vehicles and drivers through a multimodal driving data collection and situational awareness module, and perceives driving situations; a data storage module stores the data collected by the multimodal driving data collection and situational awareness module; a driving behavior real-time evaluation and feedback module evaluates the driver's driving behavior in real time and provides immediate feedback through a machine learning evaluation big model; an adaptive training strategy generation module generates personalized training tasks and learning strategies through a deep learning training big model; a cloud big model driven self-learning module performs long-term tracking and self-learning on the machine learning evaluation big model and the deep learning training big model based on multimodal data through a cloud platform; a driver psychological state monitoring and adjustment module monitors the driver's physiological and psychological state, and adjusts driving tasks and course content when necessary; the present invention combines multimodal big model technology, collects multimodal data of driving vehicles and drivers in real time, and generates personalized training tasks and learning strategies according to changes in driving situations and real-time evaluation of driver performance, and adjusts training tasks by analyzing the driver's psychological state. The system can provide detailed feedback on the driver's behavior and optimize the driving training process through a cloud platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a system diagram of the adaptive driving behavior evaluation and training system based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0088] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] Example 1

[0090] Figure 1 This is a system diagram of the adaptive driving behavior evaluation and training system based on a multimodal large model of the present invention.

[0091] The system includes a multimodal driving data collection and situational awareness module, a data storage module, a driving behavior real-time evaluation and feedback module, an adaptive training strategy generation module, a cloud-based large model-driven self-learning module, and a driver psychological state monitoring and adjustment module;

[0092] The multimodal driving data collection and situational awareness module is used to collect multimodal data of the vehicle and the driver and perceive the driving situation;

[0093] The data storage module is used to store the data collected by the multimodal driving data collection and situational awareness module;

[0094] The driving behavior real-time evaluation and feedback module is used to evaluate the driver's driving behavior in real time and provide immediate feedback through a machine learning evaluation model;

[0095] The adaptive training strategy generation module is used to generate personalized training tasks and learning strategies through deep learning training large models;

[0096] The cloud-based big model drives the self-learning module, which performs long-term tracking and self-learning on the machine learning evaluation big model and the deep learning training big model based on multimodal data through the cloud platform;

[0097] The driver psychological state monitoring and adjustment module is used to monitor the driver's physiological and psychological state and adjust the driving task and course content when necessary.

[0098] Further, the multimodal driving data acquisition and situational awareness module includes a vehicle-mounted multimodal data acquisition submodule and a driving situational awareness submodule;

[0099] The vehicle-mounted multimodal data acquisition submodule collects the driver's behavior, the driver's physiological data, the vehicle environment and the vehicle status in real time through the camera, LiDAR radar, GPS, seat sensor and eye tracker sensor;

[0100] The driving situation perception submodule perceives the current driving situation based on the data of the vehicle-mounted multimodal data acquisition submodule, and analyzes the driving difficulty in combination with real-time traffic conditions, weather data and external information, wherein the driving situation at least includes driving on urban roads, rural roads, rainy days and nighttime;

[0101] Specifically, multimodal sensor fusion is a key technology that involves integrating data collected from multiple sensors, including cameras, LiDAR, radar, GPS, and seat sensors, to gain a global understanding of the driving situation. The fusion methods used include Kalman filtering and Bayesian fusion algorithms;

[0102] The Kalman filter is used to fuse dynamic time series data, such as GPS and inertial sensor data, to accurately estimate the trajectory of the vehicle. The formula is:

[0103] x t =x t-1 +K t (z t -Hx t-1 ),

[0104] Among them, x t is the estimate of the vehicle state at the current moment, K t is the Kalman gain, zt is the sensor observation value, and H is the observation matrix.

[0105] The Bayesian fusion algorithm achieves the fusion of multimodal data by estimating the prior and posterior probabilities of different sensor data. The fusion formula is:

[0106]

[0107] Among them, x is the fused driving situation state, Z is the data from each sensor, and P(x|Z) is the posterior probability.

[0108] Situational awareness is achieved through deep learning models, specifically based on convolutional neural networks (CNNs) and long short-term memory (LSTMs) for processing time series data:

[0109] Convolutional Neural Network (CNN): used to process visual data, including cameras and LiDAR, and automatically extract important features in driving situations, including pedestrians, obstacles, and lane markings;

[0110] Long Short-Term Memory Network (LSTM): used to process time series data and combine the current driving status with past driving information to judge driving behavior in complex scenarios.

[0111] Context-aware external data processing:

[0112] Weather and traffic data integration: The system obtains real-time weather data, including temperature, humidity, rainfall, and traffic data, such as congestion and traffic light status, through the API interface. These data are integrated into the situational analysis model through the decision tree algorithm to optimize the understanding of driving situations.

[0113] The data storage module includes a distributed storage unit, a data backup unit and a data recovery unit;

[0114] The distributed storage unit is used to perform distributed storage of the database on multiple servers to ensure data security and reliability;

[0115] The data backup unit is used to regularly back up data to prevent data loss;

[0116] The data recovery unit is used to quickly recover data when the data is lost or damaged.

[0117] Furthermore, the driving behavior real-time evaluation and feedback module includes a driving behavior evaluation submodule, a situational adaptability evaluation submodule, and a real-time feedback and suggestion submodule;

[0118] The driving behavior evaluation submodule evaluates the driver's performance in real time and identifies potential errors based on the driver's operating habits through a machine learning evaluation model, wherein the operating habits at least include steering, shifting, accelerating, and braking;

[0119] The situational adaptability evaluation submodule evaluates the driver's performance in different driving situations through the machine learning evaluation model to form an evaluation result, wherein the performance in different driving situations at least includes reaction speed and operation accuracy in bad weather, night driving and congested traffic;

[0120] The real-time feedback and suggestion submodule provides real-time feedback and suggestions during driving according to the evaluation results of the situational adaptability evaluation submodule. The real-time feedback and suggestions provided by the system during driving at least include correction and guidance through voice or visual prompts when the driver exceeds the speed limit, makes a wrong operation or reacts slowly.

[0121] Specifically, the system gives real-time driving advice through voice assistants or visual prompts on the dashboard, and the feedback is in the form of a score curve based on driving behavior. The model can determine the direction of driver improvement through a linear regression formula.

[0122] Furthermore, the specific implementation process of the machine learning evaluation large model is as follows:

[0123] L1: Obtain the current camera image, LiDAR laser radar data, map image and voice data collected by the multimodal driving data collection and situational awareness module;

[0124] L2: Process the LiDAR data to obtain a two-dimensional LiDAR image, process the map image to obtain a two-dimensional grayscale image, fuse the image features of the camera image, the features of the two-dimensional LiDAR image, and the map features of the two-dimensional grayscale image to obtain the fusion features at the current moment;

[0125] L3: Calculate the Euclidean similarity between the fusion feature at the current moment and the fusion feature of the previous frame of interest, and determine whether the fusion feature at the current moment is the frame of interest based on the Euclidean similarity;

[0126] L4: If the fused feature at the current moment is a frame of interest, the pre-trained main model is used to process the fused feature at the current moment to obtain the fine-grained feature at the current moment, and then multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frame of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The main model uses a ResNet network;

[0127] L5: If the fused features at the current moment are not frames of interest, the pre-trained auxiliary model is used to process the fused features at the current moment to obtain the coarse-grained features at the current moment, and the coarse-grained features are transformed to obtain fine-grained features. Then, multiple two-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The auxiliary model uses the MobileNetV2 network.

[0128] L6: Use natural language technology to extract the speech features of the current speech data, and use the evaluation network model to fuse the evaluation features and speech features at the current moment to obtain the target evaluation result at the current moment.

[0129] Furthermore, the specific implementation of L2 includes:

[0130] Using the conversion relationship between the LiDAR radar coordinate system and the camera imaging coordinate system, the LiDAR laser radar data is projected onto the pixel two-dimensional plane to obtain a two-dimensional laser radar pixel. The characteristics of the two-dimensional laser radar pixel include: x, y, z, r, g and a; x, y, z are the three-dimensional coordinates of the pixel center point, r is the reflection coefficient of the LiDAR laser radar, g is the point frequency of the LiDAR laser radar, and a is the angular resolution of the LiDAR laser radar;

[0131] Extract image features of the camera image, the image features include resolution HW, frame rate F, red component R, green component G and blue component B;

[0132] Extracting image features of the map image, the map features including the vehicle longitude m and the vehicle latitude n;

[0133] The fusion features at the current moment include: resolution HW, frame rate F, red component R, green component G, blue component B, x, y, z, LiDAR radar reflection coefficient r, LiDAR radar point frequency g, LiDAR radar angular resolution a, vehicle longitude m and vehicle latitude n;

[0134] In the L3, calculating the Euclidean similarity between the fused feature at the current moment and the fused feature of the previous frame of interest, and judging whether the fused feature at the current moment is the frame of interest according to the Euclidean similarity includes:

[0135] Calculate the Euclidean similarity P between the fusion feature at the current moment and the fusion feature of the previous frame of interest t :

[0136]

[0137] Among them, H t H is the three-dimensional vector after caching the fusion features at the current moment.last_i is the three-dimensional vector after caching the fusion features of the previous frame of interest, d(H t ,H last_i ) are two vectors H t and H last_i The Euclidean distance between

[0138] Determine the Euclidean similarity P t Is it greater than a preset threshold? If so, the fused feature at the current moment is a non-interesting frame. If not, the fused feature at the current moment is an interesting frame.

[0139] In the L4, multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest with the fine-grained features of the current moment to obtain the evaluation features of the current moment, including:

[0140] L41: Get the fine-grained features F at the current moment output by the main model t ;

[0141] L42: Calculate the current time The fine-grained feature map at position p2 and the fine-grained feature map M of the cached frame of interest at position p2 p2 The association matrix K(d1,d2) is:

[0142]

[0143] in, The fine-grained feature map Transformed into a three-dimensional matrix, β2(M d2 ) The fine-grained feature map M d2 Transformed into a three-dimensional matrix, is a three-dimensional matrix, and the fine-grained features of the cached frame of interest are the fine-grained features of the previous frame of interest;

[0144] L43: Calculate the cached feature map aligned to the d1 position

[0145]

[0146] Cache feature map With fine-grained feature maps Fusion is performed to obtain the evaluation features of the d1 position

[0147]

[0148] Among them, β v(.) is a 3*3 convolution operation, concat(.) represents the connection operation in the channel dimension, Q(.) represents the convolution operation of three consecutive three-dimensional convolution kernels, and the size of the three consecutive three-dimensional convolution kernels is 3×3;

[0149] L44: Evaluate features of all d1 positions Evaluative features that make up the present moment

[0150] The calculation method for obtaining the evaluation features at the current moment in L5 is the same as the calculation method for obtaining the evaluation features at the current moment in L4.

[0151] Specifically, a unified description of the evaluation characteristics of the driver's adaptability in different situations is given, using the fuzzy logic control method. The fuzzy rule base is defined as follows:

[0152] If “Road congestion” is “severe” and “reaction time” is “slow”, then “driving risk” is “high”;

[0153] If Weather Conditions is Bad and Operational Accuracy is Low, then Training Needs is High;

[0154] The fuzzy reasoning process fuzzifies, infers and defuzzifies the input variables and finally outputs the driver adaptability assessment score.

[0155] Further, the adaptive training strategy generation module includes a personalized training task push submodule and a dynamic training adjustment submodule;

[0156] The personalized training task push submodule evaluates the driver's profile through deep learning and big data mining technology, and builds a deep learning training model;

[0157] The dynamic training adjustment submodule dynamically adjusts the difficulty and content of the training tasks according to the real-time performance of the driver, optimizes the deep learning training model, and adjusts the training content of the driver at different learning stages through the output of the deep learning training model.

[0158] Furthermore, the deep learning training large model implementation process is:

[0159] S1: Acquire historical behavior data of the driver, and obtain a target training model according to the historical behavior data;

[0160] S2: collecting target operation data, target physiological data, target vehicle status data and target environment data of the driver in real time through the multimodal driving data collection and situational awareness module;

[0161] S3: Processing the target operation data, the target physiological data, the target vehicle status data and the target environment data to obtain target integrated data;

[0162] S4: according to the target integrated data, multimodal feature fusion and deep learning methods are used to judge the intention, cognitive state and visual attention of the driver to obtain a target judgment result;

[0163] S5: issuing a target voice reminder to the driver and displaying target reminder information on a vehicle-mounted display according to the target judgment result and the target training model;

[0164] S6: receiving target feedback data of the driver on the target voice reminder and the target reminder information to evaluate the training effect and obtain target training evaluation data;

[0165] S7: adjusting the target training model according to the target training evaluation data, wherein the output of the target training model is used to dynamically adjust the difficulty and content of the driver training task.

[0166] Furthermore, the specific implementation process of the target training model in S1 is as follows:

[0167] S11: Build a driver profile based on the driver’s historical behavior data and basic information;

[0168] S12: Based on the driver profile, a target training model is constructed through big data mining technology and deep learning technology;

[0169] S13: The output of the target training model is real-time training driving evaluation, personalized course recommendation and dynamic adjustment of the difficulty and content of driver training tasks. The input of the target training model is the driver's historical behavior data and real-time target integration data. The real-time training driving evaluation includes acceleration driving, deceleration driving, stable driving and lane change driving.

[0170] Furthermore, the cloud-based large model-driven self-learning module includes a learning progress tracking and evaluation submodule and a cloud-based self-learning model optimization submodule;

[0171] The learning progress tracking and evaluation submodule continuously tracks the driver's learning progress, evaluates his performance, analyzes his progress or bottlenecks, and provides periodic summaries and feedback;

[0172] The cloud-based self-learning model optimization submodule continuously learns a large amount of driving data through the cloud-based multimodal large model to optimize the machine learning evaluation large model and the deep learning training large model;

[0173] The cloud-based multimodal large model uses a variational autoencoder (VAE) to continuously learn and optimize the driver behavior model. VAE achieves reconstruction and generation of driver behavior by maximizing likelihood estimation.

[0174] The cloud-based multimodal large model utilizes a federated learning architecture to enhance the self-learning capability of the cloud platform. Local models of different vehicles are shared to the cloud platform through gradient updates without transmitting the original data.

[0175] Specifically, the gradient update process is as follows:

[0176]

[0177] Among them, θ is the global model parameter, g i is the gradient of the ith device.

[0178] Furthermore, the driver's psychological state monitoring and adjustment module includes a physiological data collection submodule, an emotion analysis and state adjustment submodule, and a fatigue driving warning submodule;

[0179] The physiological data collection submodule collects the driver's physiological state data in real time through seat sensors, heart rate monitoring sensors, and eye tracking sensors, and uses the physiological state data to assess the driver's psychological state;

[0180] The emotion analysis and state adjustment submodule is used to dynamically adjust the difficulty and rhythm of the driving training task through the deep emotion analysis model, and adaptively adjust the driver's training task through the reinforcement learning algorithm to ensure that the task difficulty matches the driver's emotional state;

[0181] The dynamic adjustment of the difficulty and rhythm of the driving training task at least includes that when the driver is in a state of tension or fatigue, the system will reduce high-intensity training tasks and provide a more relaxed driving course. The deep sentiment analysis model adopts a BERT model. The BERT model analyzes the driver's psychological state by extracting physiological data features. The input data of the BERT model is the driver's voice, heart rate and skin electrical response. The multimodal analysis formula of the BERT model is:

[0182] Y = softmax(W t .BERT(x t )+W s .s t +b),

[0183] Among them, x t The text data of speech conversion, s t is the physiological data corresponding to heart rate and skin electrical response, W t and W sare the weights corresponding to text data and physiological data, b is the correction coefficient, and softmax is the activation function;

[0184] The formula for dynamically adjusting the difficulty of the driving training task is:

[0185] D ajusted =f(D current , S fatigue ),

[0186] Among them, D ajusted is the adjusted driving task, S fatigue is the driver’s fatigue level;

[0187] The fatigue driving warning submodule is used to detect driver fatigue and attention. If the fatigue and attention exceed the preset threshold, the system will ensure driving safety through voice prompts and active intervention, and the active intervention includes automatic driving takeover.

[0188] Embodiment 2, driver's psychological state monitoring and dynamic adjustment:

[0189] 1. Background description: A student is taking advanced driving training. The system needs to generate personalized training tasks based on the student's driving performance and dynamically adjust the course content;

[0190] 2. Operation process:

[0191] The system collects students' driving behaviors (such as steering, braking, acceleration, etc.) through multimodal sensors, and evaluates their performance in real time based on environmental data (such as road conditions and weather).

[0192] The system identifies that the trainee has deficiencies in lane-merging operations, so it pushes a series of lane-merging practice tasks to help the trainee improve his or her lane-merging skills. After the trainee completes the training, the system generates personalized feedback and adjusts the difficulty of subsequent training tasks based on the trainee's progress.

[0193] Example 3, Adaptive Course Generation in Advanced Driving Training:

[0194] 1. Background description: The trainee feels nervous during driving due to bad weather. The system needs to monitor the trainee's psychological state and adjust the training content;

[0195] 2. Operation process:

[0196] The system detected the student’s elevated heart rate through physiological sensors and combined it with driving behavior analysis to determine that the student was in a state of tension;

[0197] The system automatically reduces the intensity of training tasks, provides simpler driving scenarios, and encourages students to relax through voice prompts. After the students' condition returns to normal, the system gradually increases the difficulty of training tasks.

[0198] The present invention collects multimodal data of vehicles and drivers through a multimodal driving data collection and situational awareness module, and perceives driving situations; a data storage module stores the data collected by the multimodal driving data collection and situational awareness module; a driving behavior real-time evaluation and feedback module evaluates the driver's driving behavior in real time and provides immediate feedback through a machine learning evaluation big model; an adaptive training strategy generation module generates personalized training tasks and learning strategies through a deep learning training big model; a cloud big model driven self-learning module performs long-term tracking and self-learning on the machine learning evaluation big model and the deep learning training big model based on multimodal data through a cloud platform; a driver psychological state monitoring and adjustment module monitors the driver's physiological and psychological state, and adjusts driving tasks and course content when necessary; the present invention combines multimodal big model technology, collects multimodal data of driving vehicles and drivers in real time, and generates personalized training tasks and learning strategies according to changes in driving situations and real-time evaluation of driver performance, and adjusts training tasks by analyzing the driver's psychological state. The system can provide detailed feedback on the driver's behavior and optimize the driving training process through a cloud platform.

[0199] The present invention and its embodiments are described above, which is not restrictive. What is shown in the accompanying drawings is only one of the embodiments of the present invention, and the actual content is not limited to this. In short, if ordinary technicians in this field are inspired by it and do not deviate from the purpose of the invention, they can creatively design structural methods and embodiments similar to the technical scheme, which should all fall within the scope of protection of the present invention.

Claims

1. An adaptive driving behavior evaluation and training system based on a multimodal large model, characterized by: It includes multimodal driving data collection and situational awareness module, data storage module, driving behavior real-time evaluation and feedback module, adaptive training strategy generation module, cloud-based large model driven self-learning module and driver psychological state monitoring and adjustment module; The multimodal driving data collection and situational awareness module is used to collect multimodal data of the vehicle and the driver and perceive the driving situation; The data storage module is used to store the data collected by the multimodal driving data collection and situational awareness module; The driving behavior real-time evaluation and feedback module is used to evaluate the driver's driving behavior in real time and provide immediate feedback through a machine learning evaluation model; The adaptive training strategy generation module is used to generate personalized training tasks and learning strategies through deep learning training large models; The cloud-based big model drives the self-learning module, which performs long-term tracking and self-learning on the machine learning evaluation big model and the deep learning training big model based on multimodal data through the cloud platform; The driver psychological state monitoring and adjustment module is used to monitor the driver's physiological and psychological state and adjust the driving task and course content when necessary.

2. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 1 is characterized in that: The multimodal driving data collection and situational awareness module includes a vehicle-mounted multimodal data collection submodule and a driving situational awareness submodule; The vehicle-mounted multimodal data acquisition submodule collects the driver's behavior, the driver's physiological data, the vehicle environment and the vehicle status in real time through the camera, LiDAR radar, GPS, seat sensor and eye tracker sensor; The driving situation perception submodule perceives the current driving situation based on the data of the vehicle-mounted multimodal data acquisition submodule, and analyzes the driving difficulty in combination with real-time traffic conditions, weather data and external information, wherein the driving situation at least includes driving on urban roads, rural roads, rainy days and nighttime; The data storage module includes a distributed storage unit, a data backup unit and a data recovery unit; The distributed storage unit is used to perform distributed storage of the database on multiple servers to ensure data security and reliability; The data backup unit is used to regularly back up data to prevent data loss; The data recovery unit is used to quickly recover data when the data is lost or damaged.

3. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 2 is characterized in that: The driving behavior real-time evaluation and feedback module includes a driving behavior evaluation submodule, a situational adaptability evaluation submodule and a real-time feedback and suggestion submodule; The driving behavior evaluation submodule evaluates the driver's performance in real time and identifies potential errors based on the driver's operating habits through a machine learning evaluation model, wherein the operating habits at least include steering, shifting, accelerating and braking; The situational adaptability evaluation submodule evaluates the driver's performance in different driving situations through the machine learning evaluation model to form an evaluation result, wherein the performance in different driving situations at least includes reaction speed and operation accuracy in bad weather, night driving and congested traffic; The real-time feedback and suggestion submodule provides real-time feedback and suggestions during driving according to the evaluation results of the situational adaptability evaluation submodule. The real-time feedback and suggestions provided by the system during driving at least include correction and guidance through voice or visual prompts when the driver exceeds the speed limit, makes a wrong operation or reacts slowly.

4. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 3 is characterized in that: The specific implementation process of the machine learning evaluation large model is as follows: L1: Acquire the current camera image, LiDAR laser radar data, map image and voice data collected by the multimodal driving data collection and situational awareness module; L2: Process the LiDAR data to obtain a two-dimensional LiDAR image, process the map image to obtain a two-dimensional grayscale image, fuse the image features of the camera image, the features of the two-dimensional LiDAR image, and the map features of the two-dimensional grayscale image to obtain the fusion features at the current moment; L3: Calculate the Euclidean similarity between the fusion feature at the current moment and the fusion feature of the previous frame of interest, and determine whether the fusion feature at the current moment is the frame of interest based on the Euclidean similarity; L4: If the fused feature at the current moment is a frame of interest, the pre-trained main model is used to process the fused feature at the current moment to obtain the fine-grained feature at the current moment, and then multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frame of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The main model uses a ResNet network; L5: If the fused features at the current moment are not frames of interest, the pre-trained auxiliary model is used to process the fused features at the current moment to obtain the coarse-grained features at the current moment, and the coarse-grained features are transformed to obtain fine-grained features. Then, multiple two-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest and the fine-grained features at the current moment to obtain the evaluation features at the current moment. The auxiliary model uses the MobileNetV2 network. L6: Use natural language technology to extract the speech features of the current speech data, and use the evaluation network model to fuse the evaluation features and speech features at the current moment to obtain the target evaluation result at the current moment.

5. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 4 is characterized in that: The specific implementation of L2 includes: Using the conversion relationship between the LiDAR radar coordinate system and the camera imaging coordinate system, the LiDAR laser radar data is projected onto the pixel two-dimensional plane to obtain a two-dimensional laser radar pixel. The characteristics of the two-dimensional laser radar pixel include: x, y, z, r, g and a; x, y, z are the three-dimensional coordinates of the pixel center point, r is the reflection coefficient of the LiDAR laser radar, g is the point frequency of the LiDAR laser radar, and a is the angular resolution of the LiDAR laser radar; Extract image features of the camera image, the image features include resolution HW, frame rate F, red component R, green component G and blue component B; Extracting image features of the map image, the map features including the vehicle longitude m and the vehicle latitude n; The fusion features at the current moment include: resolution HW, frame rate F, red component R, green component G, blue component B, x, y, z, LiDAR radar reflection coefficient r, LiDAR radar point frequency g, LiDAR radar angular resolution a, vehicle longitude m and vehicle latitude n; In the L3, calculating the Euclidean similarity between the fused feature at the current moment and the fused feature of the previous frame of interest, and judging whether the fused feature at the current moment is the frame of interest according to the Euclidean similarity includes: Calculate the Euclidean similarity P between the fusion feature at the current moment and the fusion feature of the previous frame of interest t : Among them, H t H is the three-dimensional vector after caching the fusion features at the current moment. last_i is the three-dimensional vector after caching the fusion features of the previous frame of interest, d(H t ,H last_i ) are two vectors H t and H last_i The Euclidean distance between Determine the Euclidean similarity P t Is it greater than a preset threshold? If so, the fused feature at the current moment is a non-interesting frame. If not, the fused feature at the current moment is an interesting frame. In the L4, multiple three-dimensional convolution kernels are used to fuse the fine-grained features of the time-aligned cached frames of interest with the fine-grained features of the current moment to obtain the evaluation features of the current moment, including: L41: Get the fine-grained features F at the current moment output by the main model t ; L42: Calculate the current time The fine-grained feature map at position p2 and the fine-grained feature map M of the cached frame of interest at position p2 p2 The association matrix K(d1,d2) is: in, The fine-grained feature map Transformed into a three-dimensional matrix, β2(M d2 ) The fine-grained feature map M d2 Transformed into a three-dimensional matrix, is a three-dimensional matrix, and the fine-grained features of the cached frame of interest are the fine-grained features of the previous frame of interest; L43: Calculate the cached feature map aligned to the d1 position Cache feature map With fine-grained feature maps Fusion is performed to obtain the evaluation features of the d1 position Among them, β v (.) is a 3*3 convolution operation, concat(.) represents the connection operation in the channel dimension, Q(.) represents the convolution operation of three consecutive three-dimensional convolution kernels, and the size of the three consecutive three-dimensional convolution kernels is 3×3; L44: Evaluate features of all d1 positions Evaluative features that make up the present moment The calculation method for obtaining the evaluation features at the current moment in L5 is the same as the calculation method for obtaining the evaluation features at the current moment in L4.

6. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 5 is characterized in that: The adaptive training strategy generation module includes a personalized training task push submodule and a dynamic training adjustment submodule; The personalized training task push submodule evaluates the driver's profile through deep learning and big data mining technology, and builds a deep learning training model; The dynamic training adjustment submodule dynamically adjusts the difficulty and content of the training tasks according to the real-time performance of the driver, optimizes the deep learning training model, and adjusts the training content of the driver at different learning stages through the output of the deep learning training model.

7. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 6 is characterized in that: The implementation process of the deep learning training large model is as follows: S1: Acquire historical behavior data of the driver, and obtain a target training model according to the historical behavior data; S2: collecting target operation data, target physiological data, target vehicle status data and target environment data of the driver in real time through the multimodal driving data collection and situational awareness module; S3: Processing the target operation data, the target physiological data, the target vehicle status data and the target environment data to obtain target integrated data; S4: according to the target integrated data, multimodal feature fusion and deep learning methods are used to judge the intention, cognitive state and visual attention of the driver to obtain a target judgment result; S5: issuing a target voice reminder to the driver and displaying target reminder information on a vehicle-mounted display according to the target judgment result and the target training model; S6: receiving target feedback data of the driver on the target voice reminder and the target reminder information to evaluate the training effect and obtain target training evaluation data; S7: adjusting the target training model according to the target training evaluation data, wherein the output of the target training model is used to dynamically adjust the difficulty and content of the driver training task.

8. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 7 is characterized in that: The specific implementation process of the target training model in S1 is: S11: Build a driver profile based on the driver’s historical behavior data and basic information; S12: Based on the driver profile, a target training model is constructed through big data mining technology and deep learning technology; S13: The output of the target training model is real-time training driving evaluation, personalized course recommendation and dynamic adjustment of the difficulty and content of driver training tasks. The input of the target training model is the driver's historical behavior data and real-time target integration data. The real-time training driving evaluation includes acceleration driving, deceleration driving, stable driving and lane change driving.

9. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 8 is characterized in that: The cloud-based large model driven self-learning module includes a learning progress tracking and evaluation submodule and a cloud-based self-learning model optimization submodule; The learning progress tracking and evaluation submodule continuously tracks the driver's learning progress, evaluates his performance, analyzes his progress or bottlenecks, and provides periodic summaries and feedback; The cloud-based self-learning model optimization submodule continuously learns a large amount of driving data through the cloud-based multimodal large model to optimize the machine learning evaluation large model and the deep learning training large model; The cloud-based multimodal large model uses a variational autoencoder (VAE) to continuously learn and optimize the driver behavior model. VAE achieves reconstruction and generation of driver behavior by maximizing likelihood estimation. The cloud-based multimodal large model utilizes a federated learning architecture to enhance the self-learning capability of the cloud platform. Local models of different vehicles are shared to the cloud platform through gradient updates without transmitting the original data.

10. The adaptive driving behavior evaluation and training system based on multimodal large model according to claim 9, characterized in that: The driver's psychological state monitoring and adjustment module includes a physiological data acquisition submodule, an emotion analysis and state adjustment submodule, and a fatigue driving warning submodule; The physiological data collection submodule collects the driver's physiological state data in real time through seat sensors, heart rate monitoring sensors, and eye tracking sensors, and uses the physiological state data to assess the driver's psychological state; The emotion analysis and state adjustment submodule is used to dynamically adjust the difficulty and rhythm of the driving training task through the deep emotion analysis model, and adaptively adjust the driver's training task through the reinforcement learning algorithm to ensure that the task difficulty matches the driver's emotional state; The dynamic adjustment of the difficulty and rhythm of the driving training task at least includes that when the driver is in a state of tension or fatigue, the system will reduce high-intensity training tasks and provide a more relaxed driving course. The deep sentiment analysis model adopts a BERT model. The BERT model analyzes the driver's psychological state by extracting physiological data features. The input data of the BERT model is the driver's voice, heart rate and skin electrical response. The multimodal analysis formula of the BERT model is: Y=softmax(W t .BERT(x t )+W s .s t +b), Among them, x t The text data of speech conversion, s t is the physiological data corresponding to heart rate and skin electrical response, W t and W s are the weights corresponding to text data and physiological data, b is the correction coefficient, and softmax is the activation function; The formula for dynamically adjusting the difficulty of the driving training task is: D ajusted =f(D current ,S fatigue ), Among them, D ajusted is the adjusted driving task, S fatigue is the driver’s fatigue level; The fatigue driving warning submodule is used to detect driver fatigue and attention. If the fatigue and attention exceed the preset threshold, the system will ensure driving safety through voice prompts and active intervention, and the active intervention includes automatic driving takeover.

Images