Driver state monitoring incremental learning test method and system

Abstract

The invention relates to the field of intelligent driving and auxiliary driving systems, in particular to a driver state monitoring incremental learning test method and system. According to the method, multi-modal physiological signals are collected through the eye tracker, the pulse sensor and the electrocardio sensor, features are extracted through the convolutional neural network and the long-short-term memory network, information is fused through an attention mechanism, a model is trained under a federated learning framework, incremental learning updating is achieved, and the adaptability of the model is improved. The robustness of the model is tested by generating an adversarial sample and extreme scene data, and high accuracy is ensured. The system makes significant breakthrough in the aspects of privacy protection, real-time performance and the like, and a comprehensive evaluation report shows excellent performance. The method not only solves the problems in the prior art, but also opens up a new research direction for the field of intelligent driving safety, and has wide application prospects.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving and assisted driving systems, in particular to a method and system for incremental learning testing of driver state monitoring. Background Art

[0002] With the rapid development of the automotive industry and the continuous progress of intelligent driving technology, the driver state monitoring system has become one of the key technologies to ensure driving safety. Traditional driver state monitoring methods mainly rely on single-modal physiological signals, such as eye movement data or heart rate data, to judge the fatigue or distraction state of the driver. Although these methods can achieve certain effects under specific conditions, they face many challenges such as insufficient accuracy, poor generalization ability, and insufficient privacy protection.

[0003] In recent years, with the development of deep learning technology, driver state monitoring methods based on multi-modal physiological signal fusion have gradually attracted attention. Such methods can capture the state characteristics of drivers more comprehensively by fusing multiple physiological signals such as electrooculogram, heart rate, and electroencephalogram, improving the accuracy of monitoring. However, there are still some significant technical problems in existing multi-modal fusion methods. First of all, most methods adopt a centralized learning framework, which requires uploading the physiological data of all drivers to a central server for processing, inevitably leading to serious privacy and security issues. Secondly, existing methods generally lack the ability of incremental learning and cannot effectively adapt to the data characteristics of newly added drivers or newly emerging driving states, resulting in a gradual decline in the performance of the model during long-term use. In addition, existing methods often show poor robustness in the face of complex and changing actual driving environments, especially when there are interference factors such as light changes and sensor noise, the monitoring accuracy will decrease significantly.

[0004] On the other hand, with the development of vehicle networking technology, the driver state monitoring system is facing increasingly severe cyber security threats. Malicious attackers may deceive the monitoring system by constructing adversarial samples, leading to incorrect judgments by the system and threatening driving safety. However, existing driver state monitoring methods still lack effective defense mechanisms in dealing with such security threats.

[0005] In view of the above technical problems, there is an urgent need for a driver state monitoring method that can comprehensively solve various challenges such as accuracy, privacy protection, adaptability, and robustness. The present invention precisely addresses this need and proposes an innovative method and system for incremental learning testing of driver state monitoring. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to construct a driver state monitoring system with high accuracy, strong adaptability, and high robustness while protecting the privacy of drivers, and can effectively cope with potential adversarial attack threats.

[0007] The present invention proposes a driver state monitoring incremental learning test method, including:

[0008] Data acquisition step:

[0009] Collect electrooculogram, heart rate and electrocardiogram signals of the driver through an eye tracker, a pulse sensor and an electrocardiogram sensor;

[0010] Obtain the time series data of the electrooculogram, heart rate and electrocardiogram signals;

[0011] Data processing step:

[0012] Based on the time series data, use a convolutional neural network and a long short-term memory network to extract multi-modal physiological signal features;

[0013] According to the multi-modal physiological signal features, fuse the feature information of different modalities through an attention mechanism;

[0014] Based on the fused feature information, use a federated learning framework to train a driver state monitoring model;

[0015] According to the newly collected driver data, perform incremental learning to update the driver state monitoring model;

[0016] Testing step:

[0017] Generate adversarial samples and extreme scenario data for testing the robustness of the driver state monitoring model;

[0018] Based on the adversarial samples and extreme scenario data, evaluate the performance of the driver state monitoring model;

[0019] Output step:

[0020] Generate a comprehensive evaluation report including model accuracy, adaptability and robustness.

[0021] Preferably, in the data processing step, using a convolutional neural network and a long short-term memory network to extract multi-modal physiological signal features specifically includes:

[0022] Use a convolutional neural network to extract the spatial features of the electrooculogram signal;

[0023] Use a long short-term memory network to extract the temporal features of the heart rate and electrocardiogram signals;

[0024] Concatenate the spatial features and the temporal features to obtain multi-modal physiological signal features.

[0025] Preferably, in the data processing step, fusing the feature information of different modalities through an attention mechanism specifically includes:

[0026] Calculate the attention weights for each modal feature;

[0027] Based on the attention weights, perform weighted summation on each modal feature;

[0028] Obtain the fused feature representation.

[0029] Preferably, in the data processing step, a federated learning framework is used to train the driver state monitoring model, specifically including:

[0030] Locally train sub-models on multiple edge devices;

[0031] Upload the parameters of the sub-models to the central server;

[0032] Aggregate the sub-model parameters on the central server;

[0033] Distribute the aggregated model parameters to each edge device.

[0034] Preferably, in the data processing step, incremental learning is performed to update the driver state monitoring model, specifically including:

[0035] Obtain newly collected driver data;

[0036] Based on the new data, fine-tune the existing model parameters;

[0037] Evaluate the performance of the model on the new and old data;

[0038] According to the evaluation results, dynamically adjust the learning rate.

[0039] Preferably, in the testing step, generating adversarial samples specifically includes:

[0040] Construct a surrogate model to approximate the original driver state monitoring model;

[0041] Based on the surrogate model, use the fast gradient sign method to generate adversarial samples;

[0042] Iteratively optimize the adversarial samples until the original model is successfully deceived.

[0043] Preferably, in the testing step, generating extreme scenario data specifically includes:

[0044] Simulate image noise interference;

[0045] Simulate extreme brightness conditions;

[0046] Simulate extreme resolution situations;

[0047] Simulate extreme contrast conditions;

[0048] Simulate extreme color interference.

[0049] Preferably, evaluating the performance of the driver state monitoring model in the testing step specifically includes:

[0050] Calculating the accuracy of the model on adversarial samples;

[0051] Calculating the accuracy of the model on extreme scenario data;

[0052] Evaluating the adaptability of the model to new category data;

[0053] Evaluating the degree of catastrophic forgetting of the model.

[0054] Preferably, it further includes a model compression step, including:

[0055] Training a teacher model;

[0056] Using the teacher model to guide the training of a student model;

[0057] Transferring the knowledge of the teacher model to the student model through knowledge distillation technology;

[0058] Obtaining a lightweight compressed model.

[0059] The present invention also provides a driver state monitoring incremental learning test system for implementing the above method, including:

[0060] A data acquisition module, for:

[0061] Collecting electrooculogram, heart rate and electrocardiogram signals of the driver through an eye tracker, a pulse sensor and an electrocardiogram sensor;

[0062] Obtaining time series data of the electrooculogram, heart rate and electrocardiogram signals;

[0063] A feature extraction module, for:

[0064] Based on the time series data, using a convolutional neural network and a long short-term memory network to extract multi-modal physiological signal features;

[0065] A feature fusion module, for:

[0066] According to the multi-modal physiological signal features, fusing feature information of different modalities through an attention mechanism;

[0067] A model training module, for:

[0068] Based on the fused feature information, training a driver state monitoring model using a federated learning framework;

[0069] Performing incremental learning to update the driver state monitoring model according to newly collected driver data;

[0070] A test data generation module, configured to:

[0071] Generate adversarial samples and extreme scenario data for testing the robustness of the driver state monitoring model;

[0072] A model evaluation module, configured to:

[0073] Evaluate the performance of the driver state monitoring model based on the adversarial samples and extreme scenario data;

[0074] A result output module, configured to:

[0075] Generate a comprehensive evaluation report including model accuracy, adaptability, and robustness.

[0076] Through the organic combination of innovative technologies such as multi-modal physiological signal fusion, federated learning, incremental learning, and adversarial sample testing, the present invention achieves the following remarkable technical effects:

[0077] First, the present invention adopts multi-modal physiological signal fusion and deep learning models, significantly improving the accuracy of driver state monitoring. By simultaneously analyzing electrooculogram, heart rate, and electrocardiogram signals, the system can more comprehensively capture the physiological and psychological state changes of the driver, effectively identifying dangerous driving behaviors such as fatigue and distraction. The test results show that the method of the present invention has an accuracy improvement of nearly 10 percentage points compared with traditional single-modal methods, reaching a high accuracy of 95.8%.

[0078] Second, the present invention introduces a federated learning framework, innovatively solving the problem of privacy protection for driver physiological data. This framework allows the model to be trained without directly accessing the original data, effectively reducing the risk of data leakage. Through the application of differential privacy technology, the present invention controls the degree of privacy protection (measured by the ε value) at 0.1, far superior to traditional centralized learning methods, providing strong protection for driver data security.

[0079] Third, the incremental learning mechanism designed by the present invention significantly improves the adaptability and scalability of the system. By continuously learning the data characteristics of new drivers, the system can quickly adapt to individual differences and newly emerging driving states. In the case of only a small number of new samples, the method of the present invention can still maintain a high accuracy of 89.5%, an increase of nearly 20 percentage points compared with traditional methods. This feature enables the system to maintain or even improve its performance during long-term use, greatly extending the practical life of the model.

[0080] Fourth, through an innovative adversarial sample generation and testing mechanism, the present invention greatly enhances the robustness of the system. When facing possible malicious attacks, the accuracy of this system only drops by 4.6 percentage points, while the traditional method may drop by more than 20 percentage points. This advantage ensures that the system can still maintain stable and reliable performance in a complex and ever-changing actual driving environment, providing a more reliable guarantee for driving safety.

[0081] Finally, through model compression technology, the present invention not only provides a comprehensive performance improvement but also maintains good real-time performance. The average inference time of 15 milliseconds fully meets the requirements of real-time monitoring, enabling the system to operate efficiently in a vehicle-mounted environment with limited computing resources.

[0082] In summary, the driver state monitoring incremental learning test method and system proposed by the present invention have achieved significant technological breakthroughs in multiple aspects such as accuracy, privacy protection, adaptability, robustness, and real-time performance. These innovations not only solve many challenges faced by the existing technology but also open up new research directions for the field of intelligent driving safety. The wide application of the present invention will help significantly reduce traffic accidents caused by driver state problems and has important practical significance for improving road traffic safety. Brief Description of the Drawings

[0083] Figure 1 is the overall flowchart of the driver state monitoring system of the present invention;

[0084] Figure 2 is the structural diagram of the data acquisition module of the present invention;

[0085] Figure 3 is the structural diagram of the feature extraction module of the present invention;

[0086] Figure 4 is the structural diagram of the model training module of the present invention;

[0087] Figure 5 is the structural diagram of the test data generation module of the present invention;

[0088] Figure 6 is the structural diagram of the model evaluation module of the present invention; Detailed Embodiment

[0089] Please refer to the attached Figure 1-6 , the present invention provides a driver state monitoring incremental learning test method and system. The method aims to improve the accuracy, real-time performance, and robustness of driver state monitoring, thereby enhancing driving safety.

[0090] Specifically, the method of the present invention includes the following steps:

[0091] S1: Data acquisition

[0092] In the data acquisition step, electrooculogram, heart rate and electrocardiogram signals of the driver are collected through an eye tracker, a pulse sensor and an electrocardiogram sensor, and time series data of these physiological signals are obtained.

[0093] Preferably, the sampling frequency of the eye tracker is 60Hz, the sampling frequency of the pulse sensor is 100Hz, and the sampling frequency of the electrocardiogram sensor is 250Hz. These sampling frequencies can ensure the time resolution of the signals without causing data redundancy.

[0094] S2: Data processing

[0095] In the data processing step, first, based on the obtained time series data, a convolutional neural network (CNN) and a long short-term memory network (LSTM) are used to extract multi-modal physiological signal features. In one embodiment of the present invention, CNN is used to extract the spatial features of electrooculogram signals, while LSTM is used to extract the temporal features of heart rate and electrocardiogram signals. The structure of CNN can include 3 convolutional layers and 2 pooling layers, and LSTM can adopt a bidirectional structure with 128 neurons in the hidden layer. These two network structure designs can effectively capture the spatio-temporal features of physiological signals.

[0096] Then, according to the extracted multi-modal physiological signal features, the feature information of different modalities is fused through an attention mechanism. The mathematical expression of the attention mechanism is as follows:

[0097]

[0098] where α i is the attention weight of the i-th modality, e i is the feature vector of the i-th modality, and n is the number of modalities. In this way, the model can adaptively adjust the importance of different physiological signals and improve the effect of feature fusion.

[0099] Based on the attention weights, weighted summation is performed on the features of each modality to obtain the fused feature information.

[0100] After that, based on the fused feature information, a driver state monitoring model (the driver state monitoring model can be any model suitable for classification tasks) is trained using a federated learning framework. The federated learning framework allows model training using data distributed on different devices while protecting the privacy of the driver.

[0101] The present invention uses the FedAvg algorithm for parameter aggregation, and its update formula is:

[0102]

[0103] where w t+1 is the global model parameter at time t + 1, is the k-th local model parameter, n k is the data volume of the k-th device, n is the total data volume, and K is the number of devices participating in the training.

[0104] Finally, according to the newly collected driver data, incremental learning is performed to update the driver state monitoring model. The incremental learning adopts the Elastic Weight Consolidation (EWC) algorithm, and its loss function is:

[0105]

[0106] where L B (θ) is the loss function of the new task, F i is the Fisher information matrix (the Fisher information matrix is a known matrix obtained by calculation in the implementation, used to measure the importance of parameters) is the optimal parameter of the old task, θ i is the i-th model parameter, and λ is the regularization coefficient. In the present invention, the value of λ is 0.4, which is an empirical value obtained through a large number of experiments and can achieve a good balance between retaining old knowledge and learning new knowledge.

[0107] S3: Testing

[0108] In the testing step, the present invention generates adversarial samples and extreme scenario data for testing the robustness of the driver state monitoring model. The generation of adversarial samples adopts the Fast Gradient Sign Method (FGSM), and its formula is:

[0109]

[0110] where x adv is the adversarial sample, x is the original sample, ∈ is the perturbation size, J is the loss function, θ is the model parameter, and y is the true label. In an embodiment of the present invention, the value of ∈ is 0.1, which can effectively deceive the original model without significantly changing the appearance of the sample.

[0111] The relationship between the loss function J and L(θ): J and L(θ) are both loss functions, but they have different uses. L(θ) is a specific loss function used in the Elastic Weight Consolidation (EWC) algorithm in incremental learning, while J is a general loss function used in the generation of adversarial samples. Conceptually, they are both functions that measure the deviation between the model prediction and the true label, but the application scenarios are different.

[0112] The corresponding relationship between the parameter θ and θ i : θ in the FGSM formula and θ i in the EWC algorithm have the same concept and both represent the model parameter. However, there is an important difference: in the EWC loss function, θ iLet \(x_i\) represent the \(i\)-th specific parameter, while \(\theta\) in the FGSM formula represents the set of parameters of the entire model.

[0113] The FGSM formula actually generates adversarial samples by calculating the gradient of the loss function with respect to the input \(x\), rather than the gradient with respect to the parameter \(\theta\). This is a different direction from the model training process (taking the gradient with respect to the parameters). So, they are two different loss functions.

[0114] The generation of extreme scenario data includes simulating situations such as image noise, extreme brightness, extreme resolution, extreme contrast, and extreme colors. For example, for extreme brightness, the image pixel values can be multiplied by a factor \(\alpha\) (\(0.2\leq\alpha\leq5\)) to simulate different lighting conditions.

[0115] Based on the generated adversarial samples and extreme scenario data, the present invention evaluates the performance of the driver state monitoring model. The evaluation metrics include accuracy, robustness, adaptability, and the degree of catastrophic forgetting, etc. Among them, the robustness can be measured by the degree of reduction in the accuracy of the model on the adversarial samples, the adaptability can be evaluated by the performance of the model on new category data, and the degree of catastrophic forgetting can be quantified by the degree of performance degradation of the model on old tasks.

[0116] S4: Output

[0117] Finally, in the output step, the present invention generates a comprehensive evaluation report including the accuracy, adaptability, and robustness of the model. This report not only includes the numerical values of various metrics but also includes visual performance curves and comparison charts to intuitively display the various aspects of the model's performance.

[0118] The method of the present invention effectively improves the accuracy, privacy protection, adaptability, and robustness of the driver state monitoring model through innovative technologies such as the fusion of multi-modal physiological signals, federated learning, incremental learning, and adversarial sample testing. This is of great significance for enhancing driving safety and reducing traffic accidents.

[0119] In a preferred embodiment of the present invention, the specific implementation process of the federated learning framework is as follows:

[0120] First, local sub-models are trained on multiple edge devices. These edge devices can be in-vehicle computing units installed on different vehicles. Each device uses locally collected driver physiological data for model training, avoiding the direct sharing of sensitive data and effectively protecting the privacy of drivers.

[0121] Next, upload the parameters of the sub-model to the central server. Preferably, the upload process uses Secure Multi-party Computation (SMC) technology to further enhance the security of data transmission. In one embodiment of the present invention, SMC uses a homomorphic encryption algorithm, and its encryption function can be expressed as:

[0122] E(x) = (g x ·r n ) mod n 2 ,

[0123] where E(x) is the encryption function, x is the plaintext data to be encrypted, g and r are randomly selected integers, and n is the product of two large prime numbers, which is part of the public key (the two large prime numbers here refer to the two large prime numbers p and q used in RSA encryption, and their product n = p×q is used in the encryption process). This encryption method allows calculations to be performed in the encrypted domain, and the correct aggregation result can be obtained without decryption.

[0124] Subsequently, aggregate the sub-model parameters on the central server. The aggregation process uses a weighted average method, and the weights are proportional to the data volume of each device. Specifically, the aggregated model parameter w can be expressed as:

[0125]

[0126] where N is the number of devices participating in the aggregation, n i is the data volume of the i-th device, and w i is the model parameter of the i-th device.

[0127] Finally, distribute the aggregated model parameters to each edge device. The distribution process also uses a secure transmission protocol to ensure that the parameters are not tampered with or stolen during transmission.

[0128] Through this federated learning framework, the method of the present invention not only improves the model performance but also ensures data privacy and security. This is particularly important for applications such as driver state monitoring that involve personal sensitive information.

[0129] In the process of incrementally learning and updating the driver state monitoring model, the present invention adopts an innovative method.

[0130] First, obtain the newly collected driver data, which may come from new drivers or data of existing drivers in new driving environments.

[0131] Based on new data, this method updates the model by fine-tuning the parameters of the existing model. The fine-tuning process uses the Elastic Weight Consolidation (EWC) algorithm, which prevents catastrophic forgetting by adding a regularization term to the loss function. The loss function of EWC can be expressed as:

[0132] As mentioned before.

[0133] Next, this method evaluates the performance of the model on new and old data. The evaluation metrics include accuracy, F1-score, confusion matrix, etc. Preferably, for the driver state classification task, the weighted F1-score is used as the main evaluation metric to balance the problem of uneven sample numbers in different categories.

[0134] Finally, according to the evaluation results, this method dynamically adjusts the learning rate. The learning rate adjustment strategy uses the Cosine Annealing method, and its mathematical expression is:

[0135]

[0136] where η t is the learning rate at the t-th iteration, η min and η max are the minimum and maximum learning rates respectively, T is the total number of iterations, t is the current iteration number, and the whole represents the angular parameter of the cosine annealing formula. This learning rate adjustment strategy can converge quickly in the initial stage of training and make fine adjustments in the later stage, effectively improving the model performance.

[0137] In the test step of the present invention, the generation process of adversarial samples is of great significance. First, a surrogate model is constructed to approximate the original driver state monitoring model. The structure of the surrogate model is similar to that of the original model, but its parameters are randomly initialized. By repeatedly querying the output of the original model, the surrogate model is trained to make its behavior approximate that of the original model.

[0138] Based on the surrogate model, this method uses the Fast Gradient Sign Method (FGSM) to generate adversarial samples. The core idea of FGSM is to add a small perturbation in the direction of the input sample, and the perturbation direction is consistent with the gradient of the loss function. Its mathematical expression is:

[0139] As mentioned before.

[0140] Subsequently, this method iteratively optimizes the adversarial samples until the original model is successfully deceived. The optimization process uses the Projected Gradient Descent (PGD) algorithm, and its iterative formula is:

[0141]

[0142] Π x+S Indicates projecting the result into the ∈ neighborhood S of the original sample x, x t is the original sample at the moment, and α is the step size. In the present invention, the number of iterations is set to 10, and the step size α is 0.01. These parameter values ​​are obtained through a large number of experimental optimizations, which can control the computational overhead while ensuring the quality of the adversarial sample.

[0143] The adversarial samples generated by this method can effectively test the robustness of the driver state monitoring model, discover potential vulnerabilities of the model, and provide a basis for further optimization of the model.

[0144] The method of the present invention adopts a comprehensive and innovative approach to evaluate the performance of the driver state monitoring model. First, the method calculates the accuracy of the model on adversarial samples. This indicator reflects the sensitivity of the model to small perturbations and is an important criterion for measuring the robustness of the model. Furthermore, the method uses Average Confidence Reduction (ACR) as a supplementary indicator, and its calculation formula is:

[0145]

[0146] Where N is the number of samples, C(x i )and They are the prediction confidence of the original sample and the adversarial sample, respectively. The larger the ACR value, the higher the sensitivity of the model to adversarial samples and the worse the robustness.

[0147] Next, this method calculates the accuracy of the model on extreme scene data. Extreme scenes include special driving environments such as low light, high reflection, rain and snow. By testing the model performance in these scenes, the adaptability and generalization ability of the model can be comprehensively evaluated. In one embodiment of the present invention, the weighted average accuracy (WAA) is used as a comprehensive indicator:

[0148]

[0149] Among them, M is the number of extreme scene categories, w i is the weight of each scenario, Acc i is the accuracy of the model in the i-th scenario, and the weights can be set according to the actual occurrence frequency of various extreme scenarios.

[0150] In addition, this method also evaluates the adaptability of the model to new category data. This is of great significance for detecting emerging driver states (such as distractions caused by using new types of mobile devices). The evaluation uses the Few-shot Learning method to calculate the classification accuracy of the model on new categories with only a small number of samples. Preferably, a 5-shot 5-way classification task is used, that is, only 5 samples are provided for each new category, and there are a total of 5 new categories.

[0151] Finally, this method evaluates the degree of catastrophic forgetting of the model. This reflects the ability of the model to retain old knowledge when learning new knowledge. The evaluation uses the Average Accuracy Decrease (AAD) metric:

[0152]

[0153] where K is the number of old tasks, and are the accuracies of the model on the k-th old task before and after learning the new task, respectively. The smaller the AAD value, the lower the degree of catastrophic forgetting of the model.

[0154] Through these comprehensive evaluation metrics, the method of the present invention can deeply analyze various aspects of the performance of the driver state monitoring model, providing strong support for model optimization.

[0155] Furthermore, the present invention also sets a model compression step as an additional step after the data processing step (S2). Model compression, as an additional step after data processing, is used to optimize the model size and inference speed. The present invention adopts the knowledge distillation technology to obtain a lightweight but high-performance compressed model.

[0156] First, a teacher model is trained. The teacher model adopts a complex network structure, such as ResNet-101 or DenseNet-201, to fully extract features related to driver states.

[0157] Next, the teacher model is used to guide the training of the student model. The student model adopts a more lightweight network structure, such as MobileNetV2 or ShuffleNetV2. In a preferred embodiment of the present invention, the loss function of knowledge distillation is designed as follows:

[0158] L = αL CE (y, σ(z s )) + βL KL (σ(z s / T), σ(z t / T)),

[0159] where L CEis the cross-entropy loss, L KL is the KL divergence loss, y is the true label, z s and z t are the output logits of the student model and the teacher model respectively, which are used in the knowledge distillation process. σ is the softmax function, T is the temperature parameter, and α and β are coefficients. Preferably, α is set to 0.5, β is set to 0.5, and T is set to 3. These parameter values are optimized through a large number of experiments and can achieve a good balance between model performance and size.

[0160] Through the knowledge distillation technique, this method transfers the knowledge of the teacher model to the student model. This includes not only the hard labels (i.e., the final classification results) but also the soft labels (i.e., the class probability distributions). The soft labels contain richer information and help the student model learn more detailed feature representations.

[0161] Finally, this method obtains a lightweight compressed model. Compared with the original model, the compressed model can reduce the model size by more than 80% while maintaining similar performance, and the inference speed can be increased by 3 - 5 times. This is of great significance for deploying the driver state monitoring system on in-vehicle devices with limited computing resources.

[0162] The present invention also provides a driver state monitoring incremental learning test system corresponding to the above method. The system includes multiple functional modules, and each module is responsible for a specific task.

[0163] Among them, the data acquisition module 1 is used to collect the electrooculogram, heart rate, and electrocardiogram signals of the driver through an eye tracker, a pulse sensor, and an electrocardiogram sensor, and obtain the time series data of these physiological signals. This module can be integrated into the in-vehicle device to collect the physiological data of the driver in real time.

[0164] The feature extraction module 2 extracts multi-modal physiological signal features based on the time series data using a convolutional neural network and a long short-term memory network. This module is responsible for preprocessing the original physiological signals and extracting deep features.

[0165] The feature fusion module 3 fuses the feature information of different modalities through an attention mechanism. This module calculates the importance weights of each modality feature and performs weighted fusion on the features according to the weights to generate a comprehensive feature representation.

[0166] The model training module 4 trains the driver state monitoring model using the federated learning framework and performs incremental learning to update the model according to the newly collected driver data. This module is responsible for coordinating multiple devices for distributed training and realizing the continuous update of the model.

[0167] The test data generation module 5 is used to generate adversarial samples and extreme scenario data to test the robustness of the model. This module creates various challenging test scenarios for evaluating the performance of the model in complex environments.

[0168] The model evaluation module 6 evaluates the performance of the driver state monitoring model based on adversarial samples and extreme scenario data. This module comprehensively evaluates key metrics such as the accuracy, robustness, and adaptability of the model.

[0169] The result output module 7 generates a comprehensive evaluation report containing the accuracy, adaptability, and robustness of the model. This module visualizes the evaluation results and generates a detailed performance analysis report.

[0170] Through the collaborative work of these modules, this system can comprehensively evaluate and optimize the driver state monitoring model, improving its performance and reliability in practical applications. Each module is designed for specific requirements of driver state monitoring, ensuring the efficiency and accuracy of the system. This modular design makes the system have good scalability and maintainability, and can adapt to the development of future technologies and the emergence of new requirements.

[0171] To verify the effectiveness and superiority of the driver state monitoring incremental learning test method and its system proposed by the present invention, a series of simulation tests were conducted. The tests were based on real driving scenario data, including various states such as normal driving, fatigued driving, and distracted driving. The test results of the examples and comparative examples will be introduced in detail below.

[0172] Example 1 adopted the method proposed by the present invention, including core technologies such as multi-modal physiological signal fusion, federated learning, incremental learning, and adversarial sample testing. Comparative example 1 adopted a traditional single-modal (only eye movement data) driver state monitoring method, without incremental learning and robustness testing. Comparative example 2 adopted a multi-modal fusion method, but without federated learning and incremental learning.

[0173] The test metrics include: model accuracy, real-time performance, privacy protection level, adaptability, and robustness.

[0174] The detection criteria and methods for these metrics are as follows:

[0175] 1. Model accuracy: Using the 10-fold cross-validation method, calculate the average accuracy of the model on the test set.

[0176] 2. Real-time performance: Measure the average inference time of the model under standard hardware configurations.

[0177] 3. Privacy protection level: Evaluate the risk of data leakage during model training, measured by the differential privacy ε value.

[0178] 4. Adaptability: Test the performance of the model on newly added driver data and calculate the accuracy after 5-shot learning.

[0179] 5. Robustness: Calculate the degree of reduction in the accuracy of the model on adversarial samples.

[0180] The test results are shown in the following table:

[0181] Index Example 1 Comparative Example 1 Comparative Example 2 Model accuracy 95.8 87.3 92.1 Real-time performance (ms) 15 8 22 Privacy protection (ε) 0.1 N / A 1.5 Adaptability 89.5 72.1 80.3 Robustness 91.2 63.5 78.9

[0182] It can be seen from the test results that the method (Example 1) proposed in the present invention performs optimally in most metrics. The specific analysis is as follows:

[0183] In terms of the model accuracy, Example 1 reached 95.8%, significantly higher than Comparative Example 1 and Comparative Example 2. This proves the effectiveness of multi-modal physiological signal fusion and incremental learning, which can capture driver state characteristics more comprehensively and improve the recognition accuracy.

[0184] In terms of real-time performance, although Example 1 is slightly inferior to Comparative Example 1, considering that it processes more modal data, the inference time of 15 ms still meets the requirements of real-time monitoring. Compared with Comparative Example 2, Example 1 significantly improves the inference speed through model compression technology.

[0185] Privacy protection is a major highlight of the present invention. By adopting the federated learning framework, Example 1 controls the differential privacy ε value at 0.1, far superior to the centralized learning method of Comparative Example 2. Since Comparative Example 1 does not involve data sharing, this item was not evaluated.

[0186] In the adaptability test, Example 1 performed excellently and still achieved an accuracy of 89.5% with only a small number of new samples. This benefits from the incremental learning strategy adopted in the present invention, enabling the model to quickly adapt to new driver data.

[0187] The robustness test results show that when facing adversarial samples, the accuracy of Example 1 only decreased by 4.6 percentage points, while Comparative Example 1 and Comparative Example 2 decreased by 23.8 and 13.2 percentage points respectively. This proves that the adversarial sample training method proposed in the present invention can effectively improve the anti-interference ability of the model.

[0188] In summary, the incremental learning test method and system for driver state monitoring proposed in the present invention show obvious advantages in terms of accuracy, privacy protection, adaptability, and robustness. This comprehensive performance improvement is of great significance for improving driving safety. Especially the outstanding performance in privacy protection and model adaptability makes the system more suitable for large-scale deployment and long-term use in actual driving environments.

[0189] It should be noted that although the method of the present invention is slightly higher in computational complexity than the single-modal method, through the optimized model compression technology, good real-time performance is still ensured. This balance between performance and efficiency makes the method of the present invention have higher practical value in practical applications.

[0190] Future research directions can focus on further improving the real-time performance of the model and exploring more diverse physiological signal fusion methods to cope with more complex driving scenarios and driver states.

[0191] It should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A driver status monitoring incremental learning test method, characterized in that , including the following steps: S1: Data Acquisition Including: collecting the driver's eye electrogram, heart rate and electrocardiogram signals through an eye tracker, a pulse sensor and an electrocardiogram sensor; Acquire time series data of the electrooculogram, heart rate and electrocardiogram signals; S2: Data Processing The method comprises: extracting multimodal physiological signal features based on the time series data using a convolutional neural network and a long short-term memory network; According to the multimodal physiological signal characteristics, the feature information of different modalities is fused through an attention mechanism; Based on the fused feature information, the driver status monitoring model is trained using the federated learning framework; Performing incremental learning to update the driver status monitoring model according to newly collected driver data; S3: Testing Including: generating adversarial samples and extreme scenario data to test the robustness of the driver state monitoring model; Based on the adversarial samples and extreme scenario data, evaluating the performance of the driver state monitoring model; S4: Output Includes: Generates a comprehensive evaluation report that includes model accuracy, adaptability, and robustness.

2. The method according to claim 1, characterized in that In the data processing step, using convolutional neural network and long short-term memory network to extract multimodal physiological signal features specifically includes: Using a convolutional neural network to extract the spatial features of the electrooculogram signal; Extracting the temporal features of the heart rate and electrocardiogram signals using a long short-term memory network; The spatial features and temporal features are spliced ​​together to obtain multimodal physiological signal features.

3. The method according to claim 1, characterized in that In the data processing step, the feature information of different modalities is fused through the attention mechanism, which specifically includes: Calculate the attention weight of each modality feature; Based on the attention weights, the features of each modality are weighted and summed to obtain fused feature information.

4. The method according to claim 1, characterized in that In the data processing step, using the federated learning framework to train the driver status monitoring model specifically includes: Train sub-models locally on multiple edge devices; Uploading the sub-model parameters to a central server; aggregating the sub-model parameters on the central server; Distribute the aggregated model parameters to each edge device.

5. The method according to claim 1, characterized in that In the data processing step, performing incremental learning to update the driver state monitoring model specifically includes: Obtain newly collected driver data; Based on the new data, fine-tune existing model parameters; Evaluate the performance of the model on new and old data; According to the evaluation results, the learning rate is adjusted dynamically.

6. The method according to claim 1, characterized in that In the test step, generating adversarial samples specifically includes: Construct a substitute model to approximate the original driver state monitoring model; Based on the substitution model, generating adversarial samples using the fast gradient symbol method; The adversarial sample is iteratively optimized until the original model is successfully deceived.

7. The method according to claim 1, characterized in that ,In the test steps, generating extreme scenario data specifically includes: Simulate image noise interference; Simulate extreme brightness conditions; Simulate extreme resolution situations; Simulate extreme contrast conditions; Simulates extreme color interference.

8. The method according to claim 1, characterized in that In the test step, evaluating the performance of the driver status monitoring model specifically includes: Calculate the accuracy of the model on adversarial examples; Calculate the accuracy of the model on extreme scenario data; Evaluate the model's ability to adapt to new categories of data; Evaluate the degree of catastrophic forgetting of the model.

9. The method according to claim 1, characterized in that , further comprising a model compression step, which is an additional step after the data processing step, for optimizing the model size and inference speed, and comprises: Train the teacher model; Using the teacher model to guide the training of the student model; By using knowledge distillation technology, the knowledge of the teacher model is transferred to the student model; A lightweight compressed model is obtained.

10. A driver status monitoring incremental learning test system for executing the method according to any one of claims 1 to 9, characterized in that ,include: Data acquisition module (1), used for: Collect the driver's eye electrogram, heart rate and electrocardiogram signals through an eye tracker, pulse sensor and electrocardiogram sensor; and Acquire time series data of the electrooculogram, heart rate and electrocardiogram signals; Feature extraction module (2), used to: Based on the time series data, a convolutional neural network and a long short-term memory network are used to extract multimodal physiological signal features; Feature fusion module (3), used to: According to the multimodal physiological signal characteristics, the feature information of different modalities is fused through an attention mechanism; Model training module (4), used to: Based on the fused feature information, the driver status monitoring model is trained using the federated learning framework; Performing incremental learning to update the driver status monitoring model according to newly collected driver data; The test data generation module (5) is used to: Generate adversarial samples and extreme scenario data to test the robustness of the driver state monitoring model; Model evaluation module (6), used to: Based on the adversarial samples and extreme scenario data, evaluating the performance of the driver state monitoring model; The result output module (7) is used to: Generate a comprehensive evaluation report that includes model accuracy, adaptability, and robustness.

Images