Automatic driving trust state early warning method and system based on multi-modal features, terminal and storage medium

Abstract

The invention relates to the technical field of automatic driving, and discloses an automatic driving trust state early warning method and system based on multi-modal features, a terminal and a storage medium, and the method comprises the steps: obtaining an eye movement data signal, driving behavior data and individual feature information of a driver; performing feature extraction on the eye movement data signals, the driving behavior data and the individual feature information to obtain eye movement features, driving features and individual features; performing feature fusion on the eye movement features, the driving features and the individual features to obtain to-be-detected multi-modal features, and inputting the to-be-detected multi-modal features into a trust prediction model for prediction to obtain a trust state prediction result; and performing dynamic feedback on the driver according to the trust level of the trust state prediction result and the individual features to obtain a dynamic feedback reminding result. According to the method, the trust state is accurately predicted through the trust prediction model, dynamic evaluation of the trust state of the driver is achieved, and feedback is conducted on the driver according to the trust state.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multimodal feature-based autonomous driving trust status warning method, system, terminal, and computer-readable storage medium. Background Art

[0002] Intelligent driving technology enables vehicles to autonomously perform core driving operations such as acceleration, deceleration, steering, and path planning in a variety of road scenarios. Despite rapid development in recent years, fully autonomous driving under all operating conditions remains a long way off, limited by the uncertainty of complex traffic environments and the dynamic nature of diverse driving tasks. Therefore, for the foreseeable future, intelligent driving systems will remain in the human-machine collaborative control phase. In this phase, the driver and the intelligent system must jointly participate in vehicle control, sharing control to achieve optimal configuration of information processing, environmental perception, and decision-making processes, thereby ensuring the safety of driving tasks. Within this collaborative framework, the driver's level of trust in the autonomous driving system becomes a key factor affecting system safety and interaction efficiency, directly impacting the driver's behavioral intervention, risk decision-making, and emergency response capabilities. Existing research shows that any deviation in the driver's trust in the system may lead to potential safety hazards. Specifically, excessive trust can easily cause the driver to lose perception of the surrounding environment and reduce his or her ability to respond to emergencies, while insufficient trust may lead to excessive intervention in the system, continuous monitoring and increased cognitive load, resulting in accumulated fatigue and even traffic accidents due to human operational errors. Currently, trust assessment tools mainly rely on traditional subjective scale tools. Although subjective scale tools are widely used in psychological research, they have problems such as poor real-time performance, strong subjectivity and limited application scenarios, making them difficult to adapt to the rapidly changing driving environment.

[0003] Currently, trust assessment tools mainly rely on traditional subjective scale tools, but they have poor real-time performance and the trust assessment results obtained are difficult to adapt to the rapidly changing driving environment. As a result, subjective scale tools cannot accurately predict the user's trust status and cannot issue warnings for the driver's operating behavior based on the trust status, which has become a problem that needs to be urgently solved.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for warning the trust status of autonomous driving based on multimodal features, aiming to solve the problem in the existing technology that subjective scale tools cannot accurately predict the user's trust status and warn the driver's operating behavior based on the trust status.

[0006] To achieve the above objectives, the present invention provides an autonomous driving trust status early warning method based on multimodal features, the autonomous driving trust status early warning method based on multimodal features comprising the following steps: Acquiring an eye movement data signal, driving behavior data, and individual characteristic information of the driver, and performing feature extraction on the eye movement data signal, the driving behavior data, and the individual characteristic information to obtain eye movement features, driving features, and individual features; Performing feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, inputting the multimodal feature to be measured into a trust prediction model for prediction, and obtaining a trust state prediction result; Dynamic feedback is provided to the driver according to the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result.

[0007] Optionally, the multimodal feature-based autonomous driving trust status warning method, wherein the step of obtaining the driver's eye movement data signal, driving behavior data, and individual feature information, and performing feature extraction on the eye movement data signal, the driving behavior data, and the individual feature information to obtain eye movement features, driving features, and individual features, specifically includes: Obtaining the driver's eye movement data signals collected by the eye tracking device, the driver's driving behavior data collected by the on-board controller, and the driver's individual characteristic information collected in advance; performing eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing on the eye movement data signal in sequence to obtain eye movement features; performing interpolation filling and outlier removal on the driving behavior data to obtain target driving behavior data, and performing time-series alignment between the eye movement data signal and the target driving behavior data to obtain driving characteristics; Normalizing and one-hot encoding the individual feature information to obtain individual features; The individual characteristic information includes gender information, age information, driving experience information, emotional state information and individual trait information.

[0008] Optionally, in the multimodal feature-based autonomous driving trust status warning method, the eye movement point processing includes interpolation processing and smoothing processing; The eye movement characteristics include fixation time, total fixation times, average pupil diameter, and saccade times; The eye movement data signal is sequentially subjected to eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing to obtain eye movement features, specifically including: Performing interpolation and smoothing processing on the eye movement data signal according to a linear interpolation formula and a preset interval length to obtain a fixation time; Extracting gaze points from the eye movement data signal according to a preset angular velocity threshold to obtain a total number of gazes; performing pupil processing on the eye movement data signal according to a preset pupil diameter to obtain an average pupil diameter; Blink recognition processing is performed on the eye movement data signal according to a preset time threshold to obtain the number of eye saccades.

[0009] Optionally, the multimodal feature-based autonomous driving trust status warning method, wherein the eye movement feature, the driving feature, and the individual feature are fused to obtain a multimodal feature to be measured, and the multimodal feature to be measured is input into a trust prediction model for prediction to obtain a trust status prediction result, specifically comprising: Fusing the eye movement feature, the driving feature, and the individual feature to obtain a multimodal feature and a multimodal feature to be measured; Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set, and training a machine learning model based on the training set to obtain a trust prediction model; Inputting the multimodal features to be measured into the trust prediction model for prediction to obtain a trust status prediction result; The test set is used to evaluate the performance of the trust prediction model.

[0010] Optionally, in the multimodal feature-based autonomous driving trust status warning method, the machine learning model includes any one of a random forest model, an extreme random tree model, an extreme gradient boosting model, and a lightweight gradient boosting machine model; The trust prediction model includes a first trust prediction model, a second trust prediction model, a third trust prediction model and a fourth trust prediction model; The multimodal features are divided according to a preset division ratio to obtain a training set and a test set, and a machine learning model is trained according to the training set to obtain a trust prediction model, specifically including: Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set; performing multiple extractions on the training set using a self-service resampling method to obtain multiple target training sets, determining multiple decision trees for the multiple target training sets using a CART algorithm, generating a random forest model based on the multiple decision trees, and training the random forest model based on the multiple target training sets to obtain a first trust prediction model; determining a plurality of feature sets of the training set according to a preset double randomness mechanism, generating an extreme random tree model according to the plurality of feature sets, and training the extreme random tree model according to the training set to obtain a second trust prediction model; Training the extreme gradient boosting model according to the training set and the preset node partitioning strategy to obtain a third trust prediction model; Training the lightweight gradient boosting machine model according to preset model parameters and the training set to obtain a fourth trust prediction model; Performance tests are performed on the first trust prediction model, the second trust prediction model, the third trust prediction model, and the fourth trust prediction model to obtain trust prediction models.

[0011] Optionally, in the multimodal feature-based autonomous driving trust status warning method, the dynamic feedback reminder result includes voice and graphical interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results; The trust level includes any one of first trust, second trust and third trust; The performing of dynamic feedback on the driver based on the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result specifically includes: Dividing the trust status prediction result according to a preset threshold strategy to obtain any one of the first trust, the second trust, and the third trust; Dynamic feedback is provided to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results.

[0012] Optionally, the multimodal feature-based autonomous driving trust status warning method, wherein the dynamic feedback is provided to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphical interface feedback results, ambient light feedback results, odor cue feedback results, and tactile feedback results, further comprising: Obtaining a driver's confidence level, and if the confidence level is at a first confidence level, performing feedback adjustment on the confidence level based on the voice and graphical interface feedback results and the odor cue feedback results to obtain a target confidence level; If the trust level is at the third trust level, the driver is reminded according to the tactile feedback result and the ambient light feedback result so that the driver can drive safely.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides an autonomous driving trust status warning system based on multimodal features, wherein the autonomous driving trust status warning system based on multimodal features: a feature extraction module, configured to obtain the driver's eye movement data signal, driving behavior data, and individual feature information, and perform feature extraction on the eye movement data signal, the driving behavior data, and the individual feature information to obtain eye movement features, driving features, and individual features; a trust state prediction module, configured to perform feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and input the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result; The trust status feedback module is used to provide dynamic feedback to the driver based on the trust level of the trust status prediction result and the individual characteristics to obtain a dynamic feedback reminder result.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an autonomous driving trust status warning program based on multimodal features, and when the autonomous driving trust status warning program based on multimodal features is executed by a processor, the steps of the autonomous driving trust status warning method based on multimodal features as described above are implemented.

[0015] In the present invention, a driver's eye movement data signals, driving behavior data, and individual characteristic information are obtained, and feature extraction is performed on the eye movement data signals, driving behavior data, and individual characteristic information to obtain eye movement features, driving features, and individual characteristics. The eye movement features, driving features, and individual characteristics are then fused to obtain a multimodal feature to be measured. The multimodal feature to be measured is input into a trust prediction model for prediction to obtain a trust state prediction result. Dynamic feedback is then provided to the driver based on the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result. The present invention accurately predicts the trust state through the trust prediction model, achieving dynamic assessment of the driver's trust state and providing feedback to the driver based on the trust state. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a preferred embodiment of the autonomous driving trust status warning method based on multimodal features of the present invention; Figure 2 This is a structural diagram of a preferred embodiment of the autonomous driving trust status warning system based on multimodal features of the present invention; Figure 3 It is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Currently, trust assessment tools mainly rely on traditional subjective scale tools, but they have poor real-time performance and the trust assessment results obtained are difficult to adapt to the rapidly changing driving environment. As a result, it is impossible to accurately predict the user's trust status through subjective scale tools and to provide early warnings to the driver's operating behavior based on the trust status. Therefore, a method for early warning of autonomous driving trust status based on multimodal features is needed. The trust prediction model can accurately predict the trust status, realize dynamic evaluation of the driver's trust status, and provide feedback to the driver based on the trust status.

[0019] The autonomous driving trust status warning method based on multimodal features described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the autonomous driving trust status warning method based on multimodal features includes the following steps: Step S10: Acquire the driver's eye movement data signal, driving behavior data, and individual characteristic information, perform feature extraction on the eye movement data signal, the driving behavior data, and the individual characteristic information, and obtain eye movement features, driving features, and individual features.

[0020] The step S10 includes: Step S11: acquiring the driver's eye movement data signal collected by the eye tracking device, the driver's driving behavior data collected by the vehicle controller, and the driver's individual characteristic information collected in advance; Step S12: performing eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing on the eye movement data signal in sequence to obtain eye movement features; Step S13: performing interpolation filling and outlier removal on the driving behavior data to obtain target driving behavior data, and performing time series alignment between the eye movement data signal and the target driving behavior data to obtain driving characteristics; Step S14: normalize and one-hot encode the individual feature information to obtain individual features.

[0021] Specifically, the driver's eye movement data signals collected by an eye tracking device (using a head-mounted eye tracker or an infrared-based depth camera, which can be obtained through a camera device integrated in the vehicle in practical applications, for example: obtaining the driver's gaze behavior on the road environment, the human-vehicle interaction interface and other driving-related information during driving, and transmitting the signal to the data collection device in real time through Bluetooth or an on-board sensor network), the driver's driving behavior data collected by the on-board controller (including but not limited to steering wheel angle, acceleration and deceleration behavior, braking operation, lane keeping and other control information) and the driver's individual characteristic information collected in advance (for example: 1. Gender information (male and female); 2. Age information, which refers to the user's natural age (unit: years); 3. Driving experience information, including years of driving, driving frequency and common driving scene types (such as urban roads, highways or rural roads, etc.); 4. Emotional state information, obtained through a self-assessment scale or a contactless facial expression recognition module, which is divided into positive emotions, neutral emotions and negative emotions). 5. Individual trait information, which is divided into cautious and adventurous types, representing the user's basic tendency towards risk avoidance or risk taking in driving behavior, respectively, and can be obtained by measuring a scale for measuring individual personality traits). The eye movement data signal is sequentially processed with eye movement points, gaze points, pupils, and blink recognition to obtain eye movement features. The driving behavior data is interpolated and filled and outliers are removed to obtain target driving behavior data. The eye movement data signal is time-aligned with the target driving behavior data to obtain driving characteristics (driving behavior data is synchronously collected from the vehicle operating system, including steering wheel angle, throttle and brake operation, vehicle speed change, lane deviation, and takeover times. After the original data is interpolated, filled, outliers are removed, and time-aligned, driving characteristics with strong stability are extracted). The individual feature information is normalized and one-hot encoded to obtain individual characteristics, wherein the individual feature information includes gender information, age information, driving experience information, emotional state information, and individual trait information.

[0022] In this embodiment, before users participate in the intelligent driving task, individual characteristic information is centrally processed to ensure that such static variables can efficiently and stably participate in the trust state modeling process in the subsequent model. Specifically, gender is converted into a binary vector representation through one-hot encoding as a binary variable; age is normalized as a continuous variable using min-max normalization to reduce the impact of its numerical scale on model weight learning. Driving experience is composed of years of driving, driving frequency, and typical driving scenarios. The years of driving information is normalized, and the frequency and scenario types are categorized and vectorized through one-hot encoding. Emotional states are acquired through self-assessment questionnaires or facial expression recognition modules and are divided into three categories: positive, neutral, and negative. They are also processed using one-hot encoding to avoid errors in the sorting of category features. Individual trait information is assessed through composite psychological scales (such as the widely used Impulsive Behavior Scale and Sensation Seeking Scale). The results are divided into cautious and adventurous types and represented in the form of binary one-hot encoding. Through the above encoding and normalization operations, the system uniformly converts multi-source heterogeneous individual feature information into numerical vector form, which not only improves the model's ability to model individual differences, but also enhances the prediction performance and generalization stability.

[0023] The step S12 includes: Step S121: performing interpolation and smoothing processing on the eye movement data signal according to a linear interpolation formula and a preset interval length to obtain a fixation time; Step S122: extracting gaze points from the eye movement data signal according to a preset angular velocity threshold to obtain a total number of gazes; Step S123: performing pupil processing on the eye movement data signal according to a preset pupil diameter to obtain an average pupil diameter; Step S124: performing blink recognition processing on the eye movement data signal according to a preset time threshold to obtain the number of eye saccades.

[0024] Specifically, the eye movement data signal is interpolated and smoothed (noise reduction) according to a linear interpolation formula and a preset interval length to obtain the gaze time, the gaze point of the eye movement data signal is extracted according to a preset angular velocity threshold to obtain the total number of gazes, the eye movement data signal is pupil processed according to a preset pupil diameter (linear interpolation is performed after outliers are removed, and then signal noise reduction is completed using a sliding mean filtering method) to obtain the average pupil diameter, the eye movement data signal is blink recognized according to a preset time threshold (the preset time threshold includes a maximum time threshold and a minimum time threshold) to obtain the number of saccades, the eye movement point processing includes interpolation processing and smoothing processing, and the eye movement features include gaze time, total number of gazes, average pupil diameter and number of saccades.

[0025] In this embodiment, the maximum interval length is set to 75ms. If the signal is missing when the interval length is less than the maximum interval length, linear interpolation is used to compensate the data, and sliding mean filtering is used to reduce the data noise. The linear interpolation formula is as follows: ; in, The horizontal coordinate of the first endpoint of the interpolation interval. The ordinate of the first endpoint of the interpolation interval. The horizontal coordinates of the second endpoints of the interpolation interval are represented by The ordinate of the second endpoint of the interpolation interval, The horizontal coordinate of any interpolation point in the interpolation interval, The vertical coordinate of any interpolation point in the interpolation interval is used to approximate the point to be interpolated by using the straight line formed by the two end points of the interpolation interval. The sliding mean filter is used to smooth the input discrete time signal to reduce noise interference and signal fluctuations. The implementation steps of the sliding mean filter are: Assume that the number of input sampling points is The output is ,use The sliding filter first calculates the previous The mean of the sample points is used as the initial filter output: ; in, Indicates the The output value at the sampling point is represents the first sampling point, represents the second sampling point, Indicates that the index is The sampling points, Indicates the size of the sliding window, and then the filter is updated (smoothed) to obtain the fixation time. For each new input signal point , use the recursive formula to update the filter output: ; in, Indicates that the filter The output value at the sampling point is express The output value at the sampling point is Indicates that the input signal is The value of the sampling point, Indicates that the input signal is The value of the sampling point.

[0026] For example, for gaze point extraction, the angular velocity calculation window length is set to 20ms, and the angular velocity threshold is set to 30° / s. An angular velocity greater than 30° / s is classified as a saccade, and an angular velocity less than or equal to 30° / s is classified as a fixation. The maximum inter-fixation time threshold is set to 75ms, and the maximum inter-fixation angular velocity threshold is set to 0.5° / s. When the interval between two adjacent fixations is less than 75ms and the angle between them is less than 0.5° / s, the two adjacent fixations are merged into one. The minimum fixation time threshold (preset pupil diameter) is set to 60ms, and fixations with a duration below the minimum fixation time threshold are removed to obtain the total number of fixations. Pupil processing includes data interpolation and signal noise reduction. The minimum pupil diameter is set to 2mm, and data smaller than the minimum pupil diameter is defined as an outlier and removed. After outliers are removed, linear interpolation is performed, and signal noise reduction is performed using a sliding mean filter to obtain the average pupil diameter. Blink processing: Set the maximum time threshold to 350ms and the minimum time threshold to 75ms. When the left and right eyes are closed at the same time and the closure time is between 75ms and 350ms (preset time threshold), it is classified as a blink. After the feature extraction module completes the preprocessing of the eye movement data signal, it extracts the eye movement features and uses the correlation coefficient method to obtain eye movement features that are highly correlated with mental workload. Finally, the eye movement features include the gaze time, the total number of gazes, the average pupil diameter, and the number of eye saccades.

[0027] Step S20: Fusing the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and inputting the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result.

[0028] The step S20 includes: Step S21: performing feature fusion on the eye movement feature, the driving feature, and the individual feature to obtain a multimodal feature and a multimodal feature to be measured; Step S22: Divide the multimodal features according to a preset division ratio to obtain a training set and a test set, and train a machine learning model based on the training set to obtain a trust prediction model; Step S23: input the multimodal features to be tested into the trust prediction model for prediction to obtain a trust state prediction result; the test set is used to perform performance evaluation on the trust prediction model.

[0029] Specifically, the eye movement features, the driving features, and the individual features are fused to obtain multimodal features and multimodal features to be tested (multimodal features (sample data) are constructed using variables such as eye movement features, driving behavior features, and individual features screened out from the feature processing module for model training and testing). The multimodal features are divided according to a preset division ratio (the sample data are divided into a training set and a test set in a ratio of 7:3, wherein a 5-fold cross-validation strategy is adopted to validate the training set) to obtain a training set and a test set. A machine learning model (random forest model; extreme random tree model; extreme gradient boosting model (XGBoost, eXtreme Gradient Boosting); lightweight gradient boosting machine (lightgbm, Light Gradient Boosting Machine)) is trained based on the training set to obtain a trust prediction model. The multimodal features to be tested are input into the trust prediction model for prediction to obtain a trust state prediction result. The test set is used to evaluate the performance of the trust prediction model.

[0030] In this embodiment, in machine learning tasks, indicators such as accuracy, recall, and precision can be used to measure the performance of the trust prediction model in classification tasks. The confusion matrix describes the comparison between the prediction results of the trust prediction model and the true labels, including the true positives ( )、True counterexample( ), false positives ( ) and false counterexamples ( ) four parts, Accuracy: The overall classification accuracy of the model is measured by calculating the proportion of samples correctly classified by the trust prediction model to the total number of samples. The higher the accuracy, the more consistent the classification results of the model are with the true labels. Accuracy ( ) is calculated as follows: ; Recall rate: It is calculated by calculating the ratio of the number of samples correctly predicted as positive by the model to the number of true positive samples. It is used to measure the model's ability to identify positive samples. The value range is 0 to 1. The higher the value, the more comprehensive and accurate the trust prediction model can find all true positive samples and reduce missed judgments. Recall rate ( ) is calculated as follows: ; Precision: refers to the proportion of positive samples among the results predicted by the model, reflecting the accuracy of the model in predicting positive samples. The value is between 0 and 1. The closer to 1, the lower the misjudgment rate of the model in predicting positive samples, and the more accurate the recognition of positive samples. Precision ( ) is calculated as follows: ; The step S22 includes: Step S221: Divide the multimodal features according to a preset division ratio to obtain a training set and a test set; Step S222: performing multiple extractions on the training set using a self-service resampling method to obtain multiple target training sets, determining multiple decision trees for the multiple target training sets using a CART algorithm, generating a random forest model based on the multiple decision trees, and training the random forest model based on the multiple target training sets to obtain a first trust prediction model; Step S223: determining multiple feature sets of the training set according to a preset double randomness mechanism, generating an extreme random tree model according to the multiple feature sets, and training the extreme random tree model according to the training set to obtain a second trust prediction model; Step S224: training the extreme gradient boosting model according to the training set and the preset node partitioning strategy to obtain a third trust prediction model; Step S225: training the lightweight gradient boosting machine model according to the preset model parameters and the training set to obtain a fourth trust prediction model; Step S226: Perform performance testing on the first trust prediction model, the second trust prediction model, the third trust prediction model, and the fourth trust prediction model to obtain a trust prediction model.

[0031] Specifically, the multimodal features are divided according to a preset division ratio to obtain a training set and a test set, the training set is subjected to multiple extractions by a self-service resampling method to obtain multiple target training sets, and multiple decision trees of the multiple target training sets are determined according to the CART (Classification and Regression Trees) algorithm (Gini Impurity, Gini impurity) index, split the node according to the classification logic with the smallest Gini index value until the threshold is reached or the sample size is insufficient to support the resampling technology), generate a random forest model based on multiple decision trees (each target training set can generate a complete decision tree, and multiple decision trees constitute a random forest), train the random forest model based on multiple target training sets to obtain a first trust prediction model, determine multiple feature sets of the training set according to a preset double randomness mechanism (at each splitting node, randomly select a subset from the training set to form multiple feature sets), generate an extreme random tree model based on multiple feature sets, train the extreme random tree model based on the training set to obtain a second trust prediction model, train the extreme gradient boosting model based on the training set and the preset node partitioning strategy to obtain a third trust prediction model. Model, train the lightweight gradient boosting machine model according to the preset model parameters and the training set (using a small batch training method to divide the training set into multiple small batches) to obtain a fourth trust prediction model, and perform performance tests on the first trust prediction model, the second trust prediction model, the third trust prediction model and the fourth trust prediction model to obtain a trust prediction model (a systematic evaluation of the performance of four mainstream machine learning models, namely the random forest model, the extreme random tree model, the extreme gradient boosting model and the lightweight gradient boosting machine model, in the driver trust state prediction task is performed to obtain a trust prediction model), the machine learning model includes any one of the random forest model, the extreme random tree model, the extreme gradient boosting model and the lightweight gradient boosting machine model, and the trust prediction model includes the first trust prediction model, the second trust prediction model, the third trust prediction model and the fourth trust prediction model.

[0032] In this embodiment, the training process of the random forest model is: through the self-service resampling technology, for example: from the number of training sets ( ) were randomly selected with replacement Samples generate target training sets, and each target training set corresponds to constructing a decision tree. In general, the number of randomly selected target training sets ( ) is much smaller than , to ensure that the error can be calculated for data that has not participated in the training. Afterwards, when each node in the decision tree is split, m attributes are randomly selected from the attribute set A with medium probability to form an attribute subset. The optimal attribute in each subset is used to split the node. When constructing each decision tree, the Gini index based on the (CART, Classification and Regression Trees) algorithm is introduced. The node is split according to the classification logic with the smallest Gini index value until the threshold is reached or the sample size is insufficient to support the resampling technology. The first trust prediction model is obtained. The calculation formula of Gini is as follows: ; in, represents the Gini coefficient, represents the total number of categories, Indicates the The number of samples of the class.

[0033] As an example, the training process of the extreme random tree model: in the process of splitting each tree node, the extreme random tree model introduces a double randomness mechanism, feature random selection: at each splitting node, a subset is randomly selected from all features to form multiple feature sets, and the partition threshold is randomly generated: for each of the above candidate features, instead of the optimal partition point based on the distribution statistics of the training set, several candidate partition thresholds are randomly sampled within the value range of the feature. Subsequently, the classification effect is calculated for each candidate feature and its corresponding random threshold combination. Gini impurity or information gain is usually used as the evaluation index, and the optimal partitioning method is selected to complete the splitting of the current node. An extreme random tree model is generated based on multiple feature sets, and the extreme random tree model is trained based on the training set to obtain a second trust prediction model. The training process of the extreme random tree model further enhances the diversity and robustness of the model and effectively alleviates the risk of overfitting. The growth process of each tree in the extreme random tree model is controlled by the following hyperparameters, such as the maximum depth of the decision tree (max_depth), which is used to prevent the tree structure from over-expanding and causing the model complexity to expand. It is usually set to 30 or an appropriate value. The minimum number of samples an internal node contains (min_samples_split). If the number of samples is less than this value, no further splitting is performed. The minimum number of samples required on a leaf node (min_samples_leaf) prevents the tree from generating too many isolated nodes and improves model stability. The number of trees that make up the model (n_estimators) is that the more the number, the more stable the model. The value is usually in the range of 100-500. These parameters jointly determine the structural complexity and modeling ability of each tree. In the specific implementation, they can be automatically tuned through cross-validation to achieve a balance between prediction performance and computational efficiency.

[0034] Furthermore, the training process of the extreme gradient boosting model uses additive modeling to construct the prediction function. The model continuously fits the residual of the previous round of model through iterations to form a prediction function ( ), the calculation formula of the prediction function is as follows: ; in, Indicates the A regression tree obtained by round learning, Represents the function space of CART (Classification and Regression Trees), Indicates the total number of categories; The extreme gradient boosting model introduces a second-order differentiable objective function design. On the basis of traditional loss functions (such as square loss and logarithmic loss), it introduces a model complexity control term. The regular term is used to suppress the excessive complexity of the tree structure and effectively alleviate the overfitting problem. To improve the optimization efficiency, the extreme gradient boosting model uses a second-order Taylor expansion to approximate the loss function, introduces the first-order gradient (Gradient) and the second-order derivative (Hessian) for optimization and update, so that the algorithm achieves a good balance between accuracy and efficiency. The extreme gradient boosting model introduces a node partitioning strategy based on information gain (Gain). After evaluating the feature partitioning, The contribution to the decrease of the loss function determines the position of the splitting point. The specific process is as follows: calculate the gradient gain of all possible splitting points for each candidate feature of the training set; select the feature and splitting point with the largest gain as the optimal splitting of the current node; repeat the above process until the termination conditions (such as maximum depth, minimum number of samples, etc.) are met to generate a complete tree structure. The node splitting strategy is combined with engineering optimization methods (such as block structure caching, feature parallelism, candidate splitting point sorting, etc.) to significantly improve the splitting efficiency and model performance. The extreme gradient boosting model is trained according to the candidate features of the training set and the preset node splitting strategy to obtain the third trust prediction model.

[0035] After training is completed, the extreme gradient boosting model combines the regression trees constructed in each round into the optimal prediction extreme gradient boosting model. For a given input training set, the model will output the predicted value through each subtree in turn, and accumulate and sum them to output the final trust status prediction result.

[0036] In this embodiment, the lightweight gradient boosting model training process is as follows: a model is constructed by iteratively training multiple decision trees. In terms of model parameter (preset model parameter) settings, the preset model parameters include: setting the number of trees (num_boost_round) to 200 to balance model complexity and training efficiency; limiting the maximum depth of the tree (max_depth) to 6 to avoid overfitting the model to the training data while ensuring that the model can effectively capture the data features, using the histogram algorithm and one-sided gradient sampling (GOSS, Gradient-based One-Side Sampling) technology reduces the amount of computation and memory consumption and improves the generalization ability of the model. At the same time, a smaller learning rate (learning_rate=0.05) is set to make the lightweight gradient boosting machine model converge more robustly during training, and the validation set indicators are monitored by early stopping method (early_stopping_rounds=20). If the evaluation indicators on the validation set no longer improve after 20 consecutive rounds, training is stopped to prevent overfitting. The small batch training method is adopted to divide the training data into multiple small batches and input them into the model for training in sequence. In each round of training, the lightweight gradient boosting machine model calculates the gradient according to the difference between the current model prediction result and the true label, and then constructs a new decision tree to fit the gradient, continuously updates the model parameters, and gradually improves the model performance to obtain the fourth trust prediction model. During the training process, multi-classification evaluation indicators (such as multi-classification accuracy and macro-average score) are used to monitor the model performance. Among them, the loss function is used to measure the difference between the multi-classification prediction probability and the true label.

[0037] Step S30 : providing dynamic feedback to the driver based on the trust level of the trust status prediction result and the individual characteristics to obtain a dynamic feedback reminder result.

[0038] The step S30 includes: Step S31: Divide the trust status prediction result according to a preset threshold strategy to obtain any one of the first trust, the second trust, and the third trust; Step S32: providing dynamic feedback to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphical interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results.

[0039] Specifically, the trust state prediction results are divided according to a preset threshold strategy to obtain any one of the first trust (low trust), the second trust (moderate trust) and the third trust (excessive trust). Dynamic feedback is provided to the driver based on any one of the first trust, the second trust and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information and the individual trait information, to obtain voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results and tactile feedback results (for example, in high-risk situations such as tunnel lane changes and congestion obstacle avoidance, the system can increase the "third trust" judgment sensitivity to enhance risk warning capabilities; in low-risk situations such as routine cruising, the tolerance can be relaxed to avoid redundant intervention). The dynamic feedback reminder results include voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results and tactile feedback results, and the trust level includes any one of the first trust, the second trust and the third trust.

[0040] In this embodiment, the system can use the in-vehicle voice system to announce the prompt content, and the language style can be flexibly adjusted according to the user's mood and cognitive load. For example, an encouraging tone can be used to guide low-trust (first-trust) drivers to build confidence, or a serious tone can be used to warn over-reliant drivers. At the same time, risk information, system status or intervention suggestions can be synchronously displayed in combination with graphical interfaces such as the central control screen and head-up display (HUD). Ambient light feedback results: Contextual visual guidance is achieved through the in-vehicle ambient light. For example, when the third trust state is detected, the light can be adjusted to a cool color (such as blue light) and flash at a low frequency to alert the driver. If a low-trust state is detected, warm-toned stable lighting (such as orange light) can be enabled to relieve the driver's tension and improve their acceptance of the system. Smell prompt feedback results: The system can be integrated with a micro-smell release device to release aromatic odors with mood-regulating effects at key state nodes. For example, when the driver is highly anxious or the system trust value is too low, a soothing lavender scent can be released; when the driver is tired, a soothing lavender scent can be released. When the driver is in a tired state or has a low level of trust but needs to respond quickly, refreshing scents such as mint and citrus can be released to improve attention. Tactile feedback results: Steering wheel vibration and seat vibration are low-interference, highly perceptible prompts. They can be activated when the system recognizes that the driver's attention is distracted or the trust is shifted, quickly establishing a feedback loop to guide the driver to pay attention to the interactive task. For example, when the trust is too high and the hands are not touching the steering wheel, the steering wheel vibration reminds the driver to stay alert and be ready to take over. For example: If the system detects that the driver has not paid attention to the road for a long time (such as sight deviation or not holding the steering wheel), it will trigger a seat vibration warning. If the driver still does not respond, the vibration amplitude and frequency will increase, accompanied by visual and sound prompts.

[0041] Furthermore, step S30 further includes obtaining the driver's trust level. If the trust level is at the first level, feedback adjustment is performed on the trust level based on the voice and graphical interface feedback results and the odor prompt feedback results to obtain a target trust level (for example, when the driver is highly anxious or the system trust value is too low, a soothing lavender scent can be released; when the driver is in a fatigued state or the trust level is low but a quick response is required, refreshing scents such as mint and citrus can be released to improve attention). If the trust level is at the third level, the driver is reminded based on the tactile feedback results and the ambient light feedback results so that the driver can drive safely (for example, when the trust is too high and the hands are not touching the steering wheel, the steering wheel vibrates to remind the driver to stay alert and be ready to take over) so that the driver can drive safely.

[0042] Further, if Figure 2 As shown, based on the above-mentioned autonomous driving trust state warning method based on multimodal features, the present invention also provides an autonomous driving trust state warning system based on multimodal features, wherein the autonomous driving trust state warning system based on multimodal features includes: a feature extraction module 51 for acquiring the driver's eye movement data signal, driving behavior data, and individual feature information, and performing feature extraction on the eye movement data signal, the driving behavior data, and the individual feature information to obtain eye movement features, driving features, and individual features; A trust state prediction module 52 is configured to perform feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and input the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result; The trust status feedback module 53 is configured to provide dynamic feedback to the driver based on the trust level of the trust status prediction result and the individual characteristics, thereby obtaining a dynamic feedback reminder result.

[0043] Further, if Figure 3 As shown, based on the above-mentioned multimodal feature-based autonomous driving trust status warning method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 3 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0044] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard drive or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped with the terminal. Furthermore, the memory 20 may include both the internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software installed in the terminal and various types of data, such as program code of the terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, the memory 20 stores an autonomous driving trust status warning program 40 based on multimodal features. The autonomous driving trust status warning program 40 based on multimodal features can be executed by the processor 10, thereby implementing the autonomous driving trust status warning method based on multimodal features described in this application.

[0045] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the multimodal feature-based autonomous driving trust status warning method.

[0046] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0047] In one embodiment, when the processor 10 executes the multimodal feature-based autonomous driving trust state warning program 40 in the memory 20, the following steps are implemented: Acquiring an eye movement data signal, driving behavior data, and individual characteristic information of the driver, and performing feature extraction on the eye movement data signal, the driving behavior data, and the individual characteristic information to obtain eye movement features, driving features, and individual features; Performing feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, inputting the multimodal feature to be measured into a trust prediction model for prediction, and obtaining a trust state prediction result; providing dynamic feedback to the driver based on the trust level of the trust state prediction result and the individual characteristics, to obtain a dynamic feedback reminder result; The step of obtaining the driver's eye movement data signal, driving behavior data, and individual characteristic information, and extracting features from the eye movement data signal, the driving behavior data, and the individual characteristic information to obtain eye movement features, driving features, and individual features specifically includes: Obtaining the driver's eye movement data signals collected by the eye tracking device, the driver's driving behavior data collected by the on-board controller, and the driver's individual characteristic information collected in advance; performing eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing on the eye movement data signal in sequence to obtain eye movement features; performing interpolation filling and outlier removal on the driving behavior data to obtain target driving behavior data, and performing time-series alignment between the eye movement data signal and the target driving behavior data to obtain driving characteristics; Normalizing and one-hot encoding the individual feature information to obtain individual features; The individual characteristic information includes gender information, age information, driving experience information, emotional state information and individual trait information.

[0048] Wherein, the eye movement point processing includes interpolation processing and smoothing processing; The eye movement characteristics include fixation time, total fixation times, average pupil diameter, and saccade times; The eye movement data signal is sequentially subjected to eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing to obtain eye movement features, specifically including: Performing interpolation and smoothing processing on the eye movement data signal according to a linear interpolation formula and a preset interval length to obtain a fixation time; Extracting gaze points from the eye movement data signal according to a preset angular velocity threshold to obtain a total number of gazes; performing pupil processing on the eye movement data signal according to a preset pupil diameter to obtain an average pupil diameter; Blink recognition processing is performed on the eye movement data signal according to a preset time threshold to obtain the number of eye saccades.

[0049] The step of fusing the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and inputting the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result specifically includes: Fusing the eye movement feature, the driving feature, and the individual feature to obtain a multimodal feature and a multimodal feature to be measured; Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set, and training a machine learning model based on the training set to obtain a trust prediction model; The multimodal features to be tested are input into the trust prediction model for prediction to obtain a trust state prediction result; the test set is used to perform performance evaluation on the trust prediction model.

[0050] The machine learning model includes any one of a random forest model, an extreme random tree model, an extreme gradient boosting model, and a lightweight gradient boosting machine model; The trust prediction model includes a first trust prediction model, a second trust prediction model, a third trust prediction model and a fourth trust prediction model; The multimodal features are divided according to a preset division ratio to obtain a training set and a test set, and a machine learning model is trained according to the training set to obtain a trust prediction model, specifically including: Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set; performing multiple extractions on the training set using a self-service resampling method to obtain multiple target training sets, determining multiple decision trees for the multiple target training sets using a CART algorithm, generating a random forest model based on the multiple decision trees, and training the random forest model based on the multiple target training sets to obtain a first trust prediction model; or determining multiple feature sets of the training set according to a preset double randomness mechanism, generating an extreme random tree model according to the multiple feature sets, and training the extreme random tree model according to the training set to obtain a second trust prediction model; or training the extreme gradient boosting model according to the training set and a preset node partitioning strategy to obtain a third trust prediction model; or training the lightweight gradient boosting machine model according to preset model parameters and the training set to obtain a fourth trust prediction model; Performance tests are performed on the first trust prediction model, the second trust prediction model, the third trust prediction model, and the fourth trust prediction model to obtain trust prediction models.

[0051] The dynamic feedback reminder results include voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results and tactile feedback results; The trust level includes any one of first trust, second trust and third trust; The performing of dynamic feedback on the driver based on the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result specifically includes: Dividing the trust status prediction result according to a preset threshold strategy to obtain any one of the first trust, the second trust, and the third trust; Dynamic feedback is provided to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results.

[0052] The method further includes providing dynamic feedback to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphical interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results. Obtaining a driver's confidence level, and if the confidence level is at a first confidence level, performing feedback adjustment on the confidence level based on the voice and graphical interface feedback results and the odor cue feedback results to obtain a target confidence level; If the trust level is at the third trust level, the driver is reminded according to the tactile feedback result and the ambient light feedback result so that the driver can drive safely.

[0053] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an autonomous driving trust status warning program based on multimodal features, and when the autonomous driving trust status warning program based on multimodal features is executed by a processor, the steps of the autonomous driving trust status warning method based on multimodal features as described above are implemented.

[0054] In summary, the present invention provides a method, system, terminal, and storage medium for warning the trust status of autonomous driving based on multimodal features. The method comprises: obtaining a driver's eye movement data signal, driving behavior data, and individual feature information; performing feature extraction on the eye movement data signal, driving behavior data, and individual feature information to obtain eye movement features, driving features, and individual features; performing feature fusion on the eye movement features, driving features, and individual features to obtain a multimodal feature to be measured; inputting the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust status prediction result; and providing dynamic feedback to the driver based on the trust level of the trust status prediction result and the individual features to obtain a dynamic feedback reminder result. The present invention accurately predicts the trust status through a trust prediction model, realizes a dynamic assessment of the driver's trust status, and provides feedback to the driver based on the trust status.

[0055] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal system comprising the element.

[0056] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0057] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for early warning of autonomous driving trust status based on multimodal features, characterized in that: The autonomous driving trust status warning method based on multimodal features includes: Acquiring an eye movement data signal, driving behavior data, and individual characteristic information of the driver, and performing feature extraction on the eye movement data signal, the driving behavior data, and the individual characteristic information to obtain eye movement features, driving features, and individual features; Performing feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, inputting the multimodal feature to be measured into a trust prediction model for prediction, and obtaining a trust state prediction result; Dynamic feedback is provided to the driver according to the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result.

2. The autonomous driving trust status warning method based on multimodal features according to claim 1 is characterized in that: The step of obtaining the driver's eye movement data signal, driving behavior data, and individual characteristic information, and extracting features from the eye movement data signal, the driving behavior data, and the individual characteristic information to obtain eye movement features, driving features, and individual features specifically includes: Obtaining the driver's eye movement data signals collected by the eye tracking device, the driver's driving behavior data collected by the on-board controller, and the driver's individual characteristic information collected in advance; performing eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing on the eye movement data signal in sequence to obtain eye movement features; performing interpolation filling and outlier removal on the driving behavior data to obtain target driving behavior data, and performing time-series alignment between the eye movement data signal and the target driving behavior data to obtain driving characteristics; Normalizing and one-hot encoding the individual feature information to obtain individual features; The individual characteristic information includes gender information, age information, driving experience information, emotional state information and individual trait information.

3. The autonomous driving trust status warning method based on multimodal features according to claim 2 is characterized in that: The eye movement point processing includes interpolation processing and smoothing processing; The eye movement characteristics include fixation time, total fixation times, average pupil diameter, and saccade times; The eye movement data signal is sequentially subjected to eye movement point processing, gaze point extraction, pupil processing, and blink recognition processing to obtain eye movement features, specifically including: Performing interpolation and smoothing processing on the eye movement data signal according to a linear interpolation formula and a preset interval length to obtain a fixation time; Extracting gaze points from the eye movement data signal according to a preset angular velocity threshold to obtain a total number of gazes; performing pupil processing on the eye movement data signal according to a preset pupil diameter to obtain an average pupil diameter; Blink recognition processing is performed on the eye movement data signal according to a preset time threshold to obtain the number of eye saccades.

4. The autonomous driving trust status warning method based on multimodal features according to claim 1, characterized in that: The step of fusing the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and inputting the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result specifically includes: Fusing the eye movement feature, the driving feature, and the individual feature to obtain a multimodal feature and a multimodal feature to be measured; Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set, and training a machine learning model based on the training set to obtain a trust prediction model; Inputting the multimodal features to be measured into the trust prediction model for prediction to obtain a trust status prediction result; The test set is used to perform performance evaluation on the trust prediction model.

5. The autonomous driving trust status warning method based on multimodal features according to claim 4 is characterized in that: The machine learning model includes any one of a random forest model, an extreme random tree model, an extreme gradient boosting model, and a lightweight gradient boosting machine model; The trust prediction model includes a first trust prediction model, a second trust prediction model, a third trust prediction model and a fourth trust prediction model; The multimodal features are divided according to a preset division ratio to obtain a training set and a test set, and a machine learning model is trained according to the training set to obtain a trust prediction model, specifically including: Dividing the multimodal features according to a preset division ratio to obtain a training set and a test set; performing multiple extractions on the training set using a self-service resampling method to obtain multiple target training sets, determining multiple decision trees for the multiple target training sets using a CART algorithm, generating a random forest model based on the multiple decision trees, and training the random forest model based on the multiple target training sets to obtain a first trust prediction model; determining a plurality of feature sets of the training set according to a preset double randomness mechanism, generating an extreme random tree model according to the plurality of feature sets, and training the extreme random tree model according to the training set to obtain a second trust prediction model; Training the extreme gradient boosting model according to the training set and the preset node partitioning strategy to obtain a third trust prediction model; Training the lightweight gradient boosting machine model according to preset model parameters and the training set to obtain a fourth trust prediction model; Performance tests are performed on the first trust prediction model, the second trust prediction model, the third trust prediction model, and the fourth trust prediction model to obtain trust prediction models.

6. The autonomous driving trust status warning method based on multimodal features according to claim 2, characterized in that: The dynamic feedback reminder results include voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results and tactile feedback results; The trust level includes any one of first trust, second trust and third trust; The performing of dynamic feedback on the driver based on the trust level of the trust state prediction result and the individual characteristics to obtain a dynamic feedback reminder result specifically includes: Dividing the trust status prediction result according to a preset threshold strategy to obtain any one of the first trust, the second trust, and the third trust; Dynamic feedback is provided to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphic interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results.

7. The autonomous driving trust status warning method based on multimodal features according to claim 6 is characterized in that: The step of providing dynamic feedback to the driver based on any one of the first trust, the second trust, and the third trust, as well as the gender information, the age information, the driving experience information, the emotional state information, and the individual trait information, to obtain voice and graphical interface feedback results, ambient light feedback results, odor prompt feedback results, and tactile feedback results, further comprising: obtaining a driver's confidence level, and if the confidence level is at a first confidence level, performing feedback adjustment on the confidence level based on the voice and graphical interface feedback results and the odor cue feedback results to obtain a target confidence level; If the trust level is at the third trust level, the driver is reminded according to the tactile feedback result and the ambient light feedback result so that the driver can drive safely.

8. An autonomous driving trust status warning system based on multimodal features, characterized by: The autonomous driving trust status warning system based on multimodal features includes: a feature extraction module, configured to obtain the driver's eye movement data signal, driving behavior data, and individual feature information, and perform feature extraction on the eye movement data signal, the driving behavior data, and the individual feature information to obtain eye movement features, driving features, and individual features; a trust state prediction module, configured to perform feature fusion on the eye movement features, the driving features, and the individual features to obtain a multimodal feature to be measured, and input the multimodal feature to be measured into a trust prediction model for prediction to obtain a trust state prediction result; The trust status feedback module is used to provide dynamic feedback to the driver based on the trust level of the trust status prediction result and the individual characteristics to obtain a dynamic feedback reminder result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an autonomous driving trust status warning program based on multimodal features stored in the memory and runnable on the processor. When the autonomous driving trust status warning program based on multimodal features is executed by the processor, the steps of the autonomous driving trust status warning method based on multimodal features as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an autonomous driving trust status warning program based on multimodal features. When the autonomous driving trust status warning program based on multimodal features is executed by a processor, the steps of the autonomous driving trust status warning method based on multimodal features as described in any one of claims 1 to 7 are implemented.

Images