Selective deep transfer learning method based on driving pattern difference

Abstract

The invention discloses a driving pattern difference-based selective deep transfer learning method, which considers the damage of individual difference of drivers to the universality of a fatigue driving identification model, and aims to solve the problem of low accuracy of identifying a fatigue driving state of an unknown driver in the prior art. Through the method, the system can extract driving pattern feature knowledge from a known driver model and migrate the driving pattern feature knowledge to a new driver to form a personalized fatigue driving identification model.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning, transfer learning and deep learning, and in particular to a fatigue driving identification model based on driving pattern (driving behavior pattern) data, specifically a selective deep transfer learning method through driving pattern differences to improve the identification accuracy of the fatigue driving state of an unknown driver. Background Art

[0002] Existing fatigue driving detection technologies usually rely on the driver's physiological data or analyze facial expressions through on-board cameras to identify fatigue states. Although such methods can achieve certain fatigue state recognition effects, they have problems such as unstable accuracy, reliance on previously acquired fatigue calibration data, and high requirements for the driving environment. In particular, the individual differences between drivers are not taken into consideration enough. The existing model has poor generalization ability for new drivers and is difficult to effectively adapt to the fatigue state of each driver. At the same time, with the differences between individual drivers and changes in the driving environment, the existing model has poor generalization ability for new drivers and is difficult to effectively adapt to the fatigue state of each driver.

[0003] In recent years, fatigue driving identification methods based on driving patterns have gradually emerged. Driving patterns refer to personalized driving behavior characteristics extracted by analyzing the driver's operating data. However, due to the large differences in driving patterns between different drivers, the existing model cannot accurately identify the fatigue state of new drivers. Therefore, there is an urgent need for a method that can improve the generalization ability of fatigue driving identification models in the absence of fatigue data of unknown drivers. Summary of the invention

[0004] The present invention provides a selective deep transfer learning method based on driving pattern differences, which aims to solve the problem of low accuracy in identifying the fatigue driving state of unknown drivers in the prior art. Through this method, the system can extract driving pattern feature knowledge from the model of known drivers and transfer it to new drivers to form a personalized fatigue driving identification model.

[0005] To achieve the above object, the specific scheme of the present invention is as follows:

[0006] Step A, randomly set multiple known subjects KS and multiple unknown subjects US;

[0007] Step B: Establish a pre-trained model library for deep transfer learning;

[0008] Step C: Select the best known subject BKS for deep transfer learning;

[0009] Step D: Retrain the pre-trained model using the data of the unknown subject US, complete deep transfer learning, and establish a fatigue driving recognition model for the unknown subject US.

[0010] In step A:

[0011] The data for known subjects’ KS include: pre-acquired sober driving data set and fatigue driving data set;

[0012] The data for unknown subjects US include: initial stage data set and remaining data set;

[0013] The initial stage data set is the driving data within the initial driving time IPT; the remaining data set is the remaining data after removing the initial driving time IPT, representing the driving data collected after the initial driving time IPT period;

[0014] Initial driving time IPT≤30min.

[0015] In step B, a driving pattern-based fatigue driving identification model DRMD is established for each known subject KS;

[0016] Combining principal component analysis (PCA) with one-dimensional convolutional neural network (1D-CNN), the driving behavior features are automatically extracted and learned, without relying on manually labeled features, so as to identify whether the driver is in a fatigued state.

[0017] For the i-th known subject KS, firstly, the whole sample set is divided into a training set and a test set using 5-fold cross validation. Then, the training set is used to train the fatigue driving identification model DRMD based on the combination model PCA+1DCNN of principal component analysis PCA and 1D-CNN. Finally, the test set is used to test and select the fatigue driving identification model DRMD with the highest identification accuracy as the fatigue driving identification model DRMD for transfer learning of the known subject KS, which is expressed as the fatigue driving identification model KSDRMD. i .

[0018] In step C, the process of selecting the best known subject BKS for deep transfer learning includes: calculating the relationship between the unknown subject US and one of the known subjects KS i The selection factor CF of the driving pattern difference i , select the best known subject BKS with the smallest difference from the unknown subject's US driving pattern for deep transfer learning;

[0019] The initial data set of the unknown subject US is mixed with the test data of each known subject KS, and then the mixed test data is input into the fatigue driving identification model KSDRMD of each known subject KS. i , calculate the driving state recognition accuracy of the mixed test data set one by one; by comparing the fatigue driving recognition model KSDRMD of each known subject KS iThe driving state recognition accuracy of the test set containing the initial stage data of unknown subjects is determined by using the selection factor CF i Calculation method: Determine the best known subject BKS for deep transfer learning based on the selection factor score.

[0020] Using the best known subject BKS results, deep transfer learning is used to establish the fatigue driving identification model USDRMD for the unknown subject US.

[0021] Using deep transfer learning to establish the fatigue driving identification model USDRMD of unknown subjects US includes:

[0022] Step ①: retrieve the best known subject BKS fatigue driving identification model BKSDRMD and use it as a pre-training model for deep transfer learning;

[0023] Step ②: Construct a training data set for deep transfer learning; compensate the unknown subject US’s sober driving pattern knowledge to the unknown subject US’s fatigue driving recognition model USDRMD; supplement the training set of the best known subject BKS to form a training data set;

[0024] Step ③: Use deep transfer learning to process the fatigue driving identification model BKSDRMD of the best known subject BKS, and establish the fatigue driving identification model USDRMD of the unknown subject US; the fatigue driving identification model BKSDRMD is a fatigue driving identification model based on deep learning, and has a strong fatigue feature mining ability; the fatigue driving identification model BKSDRMD of the best known subject BKS is used as a pre-trained model, and the pre-trained model is retrained using the training data set, and the network parameter adjustment coefficient AC is designed. Based on the deep transfer learning method, the fatigue driving identification model BKSDRMD of the best known subject BKS is transferred to the fatigue driving identification model USDRMD of the unknown subject US;

[0025] Step ④: Use the remaining US data of unknown subjects to verify the effect of selective deep transfer learning based on driving pattern differences, and analyze the influence of network parameter adjustment coefficient AC and initial driving time IPT.

[0026] Selection Factor CF i The calculation formula is:

[0027]

[0028] Where, CF i is the selection factor score of the i-th known subject KS, 1≤i≤nk; nk is the number of known subjects KS; KSDRMD i KS is the known subject i The highest precision KSDRMD; mixd iKS is the known subject i The mixed data of the test data and the initial driving data of the unknown subject US; ACCDS (KSDRMD i ,mixd i ) is KSDRMD i About Mixd i driving state recognition accuracy.

[0029] Through the above method, the present invention can significantly improve the generalization ability of the fatigue driving identification model in the absence of unknown driver fatigue data. Experiments show that through selective deep transfer learning, the accuracy, sensitivity, specificity and F1 score of the model are improved by 21.32%, 8.05%, 30.18% and 20.18% respectively compared with before transfer learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the method of the present invention;

[0031] Figure 2 It is a schematic diagram of a fatigue driving identification model based on a combination model of principal component analysis PCA and a one-dimensional convolutional neural network 1D-CNN, PCA+1DCNN, of the present invention;

[0032] Figure 3 The accuracy of the fatigue driving identification model KSDRMD of each known subject KS is shown in the present invention. KS and accuracy ACCDS KSUS Schematic diagram;

[0033] Figure 4 The present invention shows the sensitivity of the fatigue driving identification model KSDRMD for each known subject KS SENDS KS and sensitivity SENDS KSUS Schematic diagram;

[0034] Figure 5 is the specificity SPEDS of the fatigue driving identification model KSDRMD of each known subject KS in the present invention KS and specificity SPEDS KSUS Schematic diagram;

[0035] Figure 6 The present invention shows the F1 score F1DS of each known subject KS's fatigue driving identification model KSDRMD to identify its own driving state KS And the F1 score F1DS for identifying unknown driving status KSUS Schematic diagram of

[0036] Figure 7It is a schematic diagram of the accuracy ACCDS of the fatigue driving identification model KSDRMD before and after the migration of the present invention in identifying the driving state of the unknown subject US;

[0037] Figure 8 It is a schematic diagram of the sensitivity SENDS of the fatigue driving identification model KSDRMD before and after migration of the present invention to identify the driving state of the unknown subject US;

[0038] Fig. 9 It is a schematic diagram of the specificity SPEDS of the fatigue driving identification model KSDRMD before and after migration of the present invention in identifying the driving state of an unknown subject US;

[0039] Fig.10 It is a schematic diagram of the F1 score F1DS of the driving state of an unknown subject US identified by the fatigue driving identification model KSDRMD before and after migration of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the implementation scope of the present invention is not limited thereto.

[0041] Embodiment 1:

[0042] like Figure 1 As shown, the selective deep transfer learning method based on driving pattern difference described in this embodiment is implemented by the following steps:

[0043] Step A: Randomly set multiple known subjects KS (Known Subject) and multiple unknown subjects US (Unknown Subject);

[0044] Step B: Establish a pre-trained model library for deep transfer learning;

[0045] Step C: Select the best known subject (BKS) for deep transfer learning;

[0046] Step D: Retrain the pre-trained model using the data of unknown subjects, complete deep transfer learning, and establish a fatigue driving recognition model for unknown subjects.

[0047] In step A:

[0048] The data for known subjects’ KS include: obtaining sufficient sober driving data sets and fatigue driving data sets in advance;

[0049] The data for unknown subjects US include: initial stage data set and remaining data set;

[0050] The initial stage data set is the driving data within the initial driving period IPT (Initial driving Period Time); the remaining data set is the remaining data after removing the IPT;

[0051] Initial driving time IPT≤30min. For example, driving data within the first 30 minutes. Previous studies have shown that, under the premise of adequate rest before driving, drivers have higher attention and alertness during the initial short driving time (within the first 30 minutes), and are usually in a more awake state.

[0052] It is assumed that the data collected within the initial driving time IPT (0 < initial driving time IPT ≤ 30min) is driving data in a sober state, but the driving data of the unknown subject in a fatigued state is unknown. Therefore, the experimental data of the unknown subject US is divided into two parts: the initial stage data set (driving data within the initial driving time IPT) and the remaining data set (data remaining after removing the initial driving time IPT).

[0053] The initial dataset is an important training data for deep transfer learning DTL (Deep Transfer Learning), and the remaining dataset is used as a test set (DTL_test dataset) to verify the effect of deep transfer learning.

[0054] In step B, a driving pattern-based fatigue driving identification model DRMD is established for each known subject KS;

[0055] Combining principal component analysis PCA with one-dimensional convolutional neural network 1D-CNN, the driving behavior characteristics under fatigue state are automatically extracted and learned, without relying on manually labeled features, and whether the driver is in a fatigue state is identified based on the driving behavior characteristics. For the i-th known subject KS, firstly, the whole sample set is divided into training set and test set using 5-fold cross validation, and then the training set is used to train the fatigue driving identification model DRMD based on the combination model PCA+1DCNN based on principal component analysis PCA and one-dimensional convolutional neural network 1D-CNN. Finally, the test set is used to test and select the fatigue driving identification model DRMD with the highest identification accuracy as the fatigue driving identification model DRMD for transfer learning of the known subject KS, expressed as the fatigue driving identification model KSDRMD i (ith Known Subject's DRMD).

[0056] In view of the advantages of deep learning algorithms, a driving pattern fatigue driving recognition model is established with the deep learning algorithm 1D-CNN as the core. The overall structure of the model is as follows: Figure 2 Using the awake data and fatigue data of the known subjects KS to train Figure 2The driving pattern-based fatigue driving identification model uses deep learning algorithms to mine the mapping relationship between the driving pattern feature matrix DF and the driving state label to achieve driving pattern-based fatigue driving identification.

[0057] In step C, the process of selecting the best known subject BKS for deep transfer learning includes: calculating the selection factor CF representing the difference in driving pattern between the unknown subject US and one of the known subjects KSi i , select the best known subject BKS with the smallest difference from the unknown subject's US driving pattern for deep transfer learning;

[0058] The initial data set of the unknown subject US is mixed with the test data of each known subject KS, and then the mixed test data is input into the fatigue driving identification model KSDRMD of each known subject KS. i , calculate the fatigue driving identification model KSDRMD one by one i The driving state recognition accuracy of the mixed test data set; by comparing the fatigue driving recognition model KSDRMD of each known test subject KS i For the driving state recognition accuracy of the test set containing the initial stage data of the unknown subject, the selection factor representing the difference between the unknown subject US and the driving patterns of each known subject KS is calculated; based on the selection factor score, the best known subject BKS for deep transfer learning is determined.

[0059] The fatigue driving recognition model BKSDRMD of the best known subject BKS is used, and deep transfer learning is used to establish the fatigue driving recognition model USDRMD (Unknown Subject's DRMD) of the unknown subject US.

[0060] Using deep transfer learning to establish the fatigue driving identification model USDRMD for unknown subjects includes:

[0061] Step 1: Retrieve the best known subject's fatigue driving identification model BKSDRMD (Best Known Subject's DRMD) and use it as a pre-training model for deep transfer learning;

[0062] Step ②: Construct a training dataset for deep transfer learning (DTL_training dataset); the initial stage data of the unknown subject US contains rich driving pattern features in the awake state, which is the basis for constructing the training dataset. The awake driving pattern knowledge of the unknown subject US is compensated to the fatigue driving identification model USDRMD of the unknown subject; the training set of the best known subject BKS is used as a supplement to form the training dataset;

[0063] Step ③: Use deep transfer learning to process the fatigue driving identification model BKSDRMD of the best known subject KS, and establish the fatigue driving identification model USDRMD of the unknown subject US; the fatigue driving identification model BKSDRMD is a fatigue driving identification model based on deep learning, which has a strong fatigue feature mining ability; take the fatigue driving identification model BKSDRMD of the best known subject KS as the pre-trained model, use the training data set to retrain the pre-trained model, design the network parameter adjustment coefficient AC (Adjustment Coefficient), and transfer the fatigue driving identification model BKSDRMD of the best known subject KS to the unknown subject US based on the deep transfer learning method, and establish the personalized fatigue driving identification model USDRMD of the unknown subject US;

[0064] Step ④, use the remaining data of the unknown subject US to verify the effect of selective deep transfer learning based on driving pattern differences, and analyze the influence of the network parameter adjustment coefficient AC and the initial driving time IPT.

[0065] Selective deep transfer learning refers to selecting the best known test subject KS for deep transfer learning and establishing the fatigue driving identification model USDRMD, which includes the selection process and the deep transfer learning process. The selection process is the process of determining which known test subject KS to perform deep transfer learning based on the selection factor. The selection factor CF i (Choose Factor) is the key to selective deep transfer learning. The more similar the datasets of the source task and the target task are, the better the effect of transfer learning will be. Therefore, a known subject KS with a small difference in driving pattern from the unknown subject US (similar driving pattern) should be selected for deep transfer learning. The smaller the difference in driving pattern between the known subject KS and the unknown subject US, the better the effect of the fatigue driving identification model DRMD of the known subject KS on the driving state identification of the mixed test set containing the unknown subject US samples, and the easier it will be to perform transfer learning.

[0066] Selection Factor CF i The calculation formula is:

[0067]

[0068] Where, CF i is the selection factor score of the i-th KS, 1≤i≤nk; nk is the number of known KSs; KSDRMD i KS is the known subject i The most accurate fatigue driving identification model KSDRMD; mixd i KS is the known subject i The mixed data of the test data and the initial driving data of the unknown subject US; ACCDS (KSDRMD i,mixd i ) is KSDRMD i About Mixd i Under the condition that only the driving data of the unknown subject US in the initial driving phase can be obtained, it reflects the driving state recognition accuracy of the known subject KS to a certain extent. i Differences in driving patterns with unknown subject US, known subject KS i The larger the value, the better the fatigue driving identification model KSDRMD. i The stronger the generalization ability of unknown subjects US is, the better the generalization ability of known subjects KS is. i The smaller the difference between the driving pattern and the unknown subject US is, the smaller the difference between the driving pattern and the unknown subject US is. Therefore, the known subject KS is divided into i Sort in descending order, select factor CF i The largest known subject KS is used as the known subject BKS for deep transfer learning, and the fatigue driving identification model USDRMD is established.

[0069] Deep transfer learning methods include:

[0070] according to Figure 1 The method flow is as follows: after selecting the known subject BKS, the fatigue driving identification model BKSDRMD is transferred to the unknown subject US using deep transfer learning technology to establish the fatigue driving identification model USDRMD. Deep transfer learning refers to the process of applying the model of the source task based on the deep learning algorithm to the target task; deep transfer learning first uses the data of the source task to train the deep learning model, and uses the deep learning model of the source task as the pre-trained model; then the pre-trained model is fine-tuned using the data of the target task, and the general features and patterns learned by the pre-trained model are used to improve the model's ability to solve the target task.

[0071] Deep transfer learning has many advantages. For example, deep transfer learning transfers data based on models based on deep learning algorithms, has strong nonlinear modeling capabilities, and is more capable of processing large-scale complex data. In addition, deep transfer learning uses large-scale data for pre-training, learns richer general features, requires less data for new tasks, and has higher transfer efficiency. Therefore, deep transfer learning is used to combine a small amount of driving data from the initial driving stage of an unknown subject US to establish a fatigue driving identification model USDRMD.

[0072] The deep transfer learning in this study belongs to network-based deep transfer learning. Its process is as follows Figure 1As shown in Section 4 of . Network-based deep transfer learning refers to using some network layers of the pre-trained model in the source domain and retraining them to migrate them into a deep learning model that solves the target task. Network-based deep transfer learning assumes that the front layer of the deep learning network can be regarded as a relatively general feature extractor, and its feature expression mode is applicable to other related target tasks. The main steps and mathematical expressions for deep transfer learning are as follows:

[0073] Assume that the dataset of the source domain is The indicator vector of the i-th sample in the source domain, Its corresponding driving state label. The dataset of the target domain is The indicator vector of the jth sample in the target domain, is the corresponding driving state label. ns and nt are the sample sizes of the source domain and the target domain, respectively. The dataset of the source domain is the dataset of the known subject BKS, and the dataset of the target domain is the dataset of the unknown subject US.

[0074] First, in D s The model M of the source task based on the combined model PCA+1DCNN is trained on s , M s The modeling process is:

[0075]

[0076] In formula (1-2), argmin is the minimum value; θ s is the model parameter; M s is the parameter space of the model. The meaning of this formula is that in D s Find the average loss between the driving state value output by the model and the actual driving state value, and find the model parameter θ that can minimize the average loss s , establish the best known subject-based fatigue driving identification model M s .

[0077] Then, with M s To pre-train the model, use the dataset D of the target domain t Retrain M s , fine-tune M s The model parameters of fatigue driving identification model M suitable for the target task are established t , M t The formula is:

[0078]

[0079] In formula (1-3), argmin is the minimum value; θ t Fine-tune M on the target domains The model parameters obtained after t Model is M t The parameter space of ; l is the loss function. After fine-tuning the model, in D t Find the average loss between the driving state value output by the model and the actual driving state value, and find the model parameter θ that can minimize the average loss t , establish M t , that is, the fatigue driving identification model USDRMD of the unknown subject US.

[0080] The convolutional layer of the fatigue driving identification model BKSDRMD can learn the underlying, universal fatigue feature expression. These universal fatigue feature knowledge (convolutional layer parameters) should be applied to unknown subjects as much as possible. The fully connected layer of the fatigue driving identification model BKSDRMD usually contains high-level features that are highly correlated with the known subjects BKS, and should be adjusted significantly to adapt to the features of the unknown subjects US as quickly as possible. Therefore, when performing deep transfer learning, by setting a larger learning rate for the fully connected layer and using a smaller learning rate for other layers, freeze the layers except the fully connected layer in the fatigue driving identification model BKSDRMD, and focus on training and adjusting the fully connected layer.

[0081] Example 2: According to Figure 1 In the process shown, subject No. 20 is randomly selected as the unknown subject US, and the remaining 23 subjects are known subjects KS, and this is used as an example for research.

[0082] According to existing research, the fatigue driving identification model KSDRMD has a good identification effect on KS's own driving state. However, due to the differences in driving patterns among subjects, the identification effect of the fatigue driving identification model KSDRMD on the driving state of unknown subjects needs further testing to analyze whether it is necessary to consider the differences in driving patterns for transfer learning. Therefore, the remaining data of unknown subjects are used as test data, and are input into the fatigue driving identification model KSDRMD of each known subject KS for testing, so as to obtain the driving state identification effect of the fatigue driving identification model KSDRMD on the unknown subject US before transfer learning. The accuracy of driving state identification ACCDS (ACCuracy of Driving State), sensitivity SENDS (SENsitivity of Driving State), specificity SPEDS (SPEcificity of Driving State) and F1 score F1DS (F1ofDriving State) are used to comprehensively evaluate the driving state identification effect of the fatigue driving identification model KSDRMD on the unknown subject US.

[0083] Figure 3Display the accuracy of fatigue driving identification model KSDRMD of each known subject KS in identifying its own driving state ACCDS KS and the accuracy of identifying unknown driving states of test subjects ACCDS KSUS .

[0084] like Figure 3 As shown, the accuracy of each known subject KS ACCDS KS All reached 100%, however, the accuracy of each known subject KS ACCDS KSUS Among them, the accuracy of subject No.1 is ACCDS KSUS The minimum is 41.68%, the accuracy of subject No.18 is ACCDS KSUS The maximum is only 62.52%. KS Compared with the known subjects' KS accuracy ACCDS KSUS Among them, the accuracy of subject No.1 was ACCDS KSUS The maximum decrease is 58.32%. The test results show that the fatigue driving identification model KSDRMD of the known subject has a good identification effect on its own driving status. However, due to the difference in driving patterns between the known subject KS and the unknown subject US, the fatigue driving identification model KSDRMD of the known subject KS has a poor overall identification effect on the driving status of the unknown subject US.

[0085] Figure 4 Display the sensitivity of fatigue driving identification model KSDRMD of each known subject KS to identify its own driving state SENDS KS and the sensitivity of identifying unknown driving status SENDS KSUS .

[0086] like Figure 4 As shown, the sensitivity SENDS of all known subjects KS KS All reached 100%, which shows that the fatigue driving recognition model KSDRMD of the known subject KS has a good recognition accuracy for its own fatigue samples. Compared with the sensitivity SENDS KS , the sensitivity SENDS of each known subject KS KSUS Among them, the sensitivity of subject No.1 SENDS KSUS The decline was the largest (51.37%), and the sensitivity of subject No.11 was SENDS KSUS The decrease of is the smallest (14.96%). The results show that the difference in driving patterns makes the fatigue driving identification model KSDRMD of each known subject KS decrease in different degrees in the detection ability of fatigue samples of unknown subjects US, and the missed reporting rate of fatigue samples of unknown subjects US increases.

[0087] Figure 5 Display the specificity SPEDS of KSDRMD for identifying the driving state of each known subject KS and the specificity of identifying unknown subjects' driving status SPEDS KSUS .

[0088] like Figure 5 As shown, the specificity SPEDS of all known subjects' KS KS All reached 100%, which shows that the fatigue driving identification model KSDRMD with known subject KS can accurately identify its own awake samples. However, the specificity SPEDS of each known subject KS KSUS Among them, the specificity of subjects No.5 and No.9 was SPEDS KSUS The smallest value is 13.16%. The specificity of subject No.23 SPEDS BKSUS is the largest, with a value of 64.66%. The specificity of all known subjects KS SPEDS KSUS Average bit anisotropy SPEDS KS Among them, the largest decrease was 86.84% for subjects No.5 and No.9. Figure 4 and Figure 5 The results show that compared with the specificity SENDS KSUS , the specificity of each known subject KS SPEDS KSUS This shows that the fatigue driving identification model KSDRMD of the known subject KS has a worse identification effect on the sober driving state samples of the unknown subject US, and a large number of sober samples are misjudged as fatigue samples. Therefore, it is more necessary to use the sober driving data of the unknown subject US in the initial driving stage to adjust the fatigue driving identification model KSDRMD of the known subject KS, obtain compensation for the sober driving pattern characteristics of the unknown subject US, and improve the recognition ability of the sober samples of the unknown subject US.

[0089] Figure 6 Display the F1 score F1DS of each known subject KS's fatigue driving identification model KSDRMD to identify its own driving state KS And the F1 score F1DS for identifying unknown driving status KSUS .like Figure 6 As shown, the F1DS of each known subject KS KS All of them reached 100%, which shows that the fatigue driving identification model KSDRMD of each known subject KS has a strong comprehensive identification ability for the driving state of its own sample. However, compared with the F1 score F1DS KS , the F1 score F1DS of each known subject KS KSUS The F1 score of subject No.1 is F1DS. KSUSThe smallest is 50.06%, and the F1 score F1DS of the No.1 test subject is KSUS The decline is also the largest, at 49.94%. The F1 score of subject No.18 is F1DS KSUS The maximum is 72.53%, and the F1 score F1DS of the No.18 test subject KSUS The decline was the smallest, at 27.47%. Figure 3 The results are consistent with those in . The fatigue driving identification model KSDRMD of the known subject KS has a poor comprehensive identification effect on the driving status of unknown subject samples, and there are many missed or false positives.

[0090] Depend on Figures 3 to 6 From the comparison results of different model evaluation indicators, it can be seen that the fatigue driving identification model KSDRMD of each known subject KS has a good identification effect on its own driving state, but has a poor identification effect on the driving state of unknown subjects. On the one hand, this phenomenon reflects the personalized characteristics of the fatigue driving identification model KSDRMD. On the other hand, it shows that due to the differences in driving patterns, the fatigue driving identification model KSDRMD of the known subject KS has a poor generalization effect on the unknown subject US and cannot be directly used to identify the driving state of the unknown subject US. Therefore, it is necessary to fine-tune the fatigue driving identification model KSDRMD of the known subject KS through deep transfer learning to improve the identification effect of the driving state of the unknown subject US.

[0091] Calculation results of selection factors: Due to the obvious driving pattern differences between subjects, the fatigue driving identification model KSDRMD has a poor identification effect on the driving status of unknown subjects US. Therefore, the fatigue feature knowledge learned from the fatigue driving identification model KSDRMD should be used, combined with the driving data of the unknown subjects US in the initial stage, and deep transfer learning technology should be applied to establish a fatigue driving identification model suitable for unknown subjects US. Figure 1 When performing selective deep transfer learning, the first step is to calculate the selection factor CF of the known subject KS according to formula (1-1): i , the best known subject BKS for deep transfer learning is determined based on the selection factor score. Table 2-1 shows the selection factor scores of all known subject KSs, and each known subject KS is arranged in ascending order according to the selection factor score.

[0092] Table 2-1 Selective factor scores of known subjects KS

[0093]

[0094]

[0095] ACCDS(KSDRMD i ,ksd i ) is KSi KSDRMD i Accuracy of identifying own driving status, ACCDS (KSDRMD i ,mixd i ) is shown in formula (1-1). As shown in Table 2-1, the ACCDS (KSDRMD i ,ksd i ) all reached 100.00%, however, the ACCDS (KSDRMD i ,mixd i ) are both higher than ACCDS (KSDRMD i ,ksd i ) is low. i ,mixd i ) is the largest, with a value of 86.42%. The ACCDS (KSDRMD i ,mixd i ) is the smallest, and its value is 51.11%. From formula (1-1), we can see that the selection factor CF i With ACCDS (KSDRMD i ,mixd i ) is proportional to the selection factor CF of No.18 subjects. i The maximum value is 1.33, the selection factor CF of subject No.5 i The minimum value is 0.79. Therefore, the selection factor CF is selected i The largest subject No.18 is used as the best known subject BKS for deep transfer learning.

[0096] In order to compare the different selection factors CF i The deep transfer learning effect of the known test subject KS is Figure 1 Deep transfer learning was performed in the fourth part to establish the fatigue driving identification model USDRMD. When performing deep transfer learning, all known subjects used the fatigue driving identification model KSDRMD as their pre-trained model for transfer learning. The pre-trained models before and after the transfer were tested using the remaining data of unknown subjects, and the driving state identification effect of the model was comprehensively evaluated using accuracy ACCDS, sensitivity SENDS, specificity SPEDS, and F1 score F1DS. In order to control variables, the same initial driving time IPT=30min and network adjustment coefficient AC=1 were used for deep transfer learning for each known subject KS.

[0097] Figure 7The accuracy of the fatigue driving identification model KSDRMD with known subject KS before and after migration is shown in ACCDS. KSUS and accuracy ACCDS KSUST They represent the accuracy of the fatigue driving identification model KSDRMD in identifying the unknown driving state of the test subject before and after deep transfer learning, respectively. The left vertical axis is the selection factor CF i , the right vertical axis is the accuracy ACCDS. The horizontal axis is the KS number of the known subjects arranged in ascending order of the selection factor score. The data in the figure are all approximate results with two decimal places.

[0098] like Figure 7 As shown in the figure, for all known subjects’ KS, after deep transfer learning, the ACCDS of the KS accuracy of each known subject is KSUST Both are higher than the accuracy of ACCDS KSUS It is shown that based on the deep transfer learning method, the fatigue driving identification model KSDRMD of the known subject KS is fine-tuned using the initial driving data of the unknown subject US, which can improve the identification accuracy of the unknown subject's driving state. In addition, the accuracy of each known subject KS ACCDS KSUST There is a random selection factor CF i The selection factor CF i The largest one is the selection factor CF of subject No.18 i (1.33), the accuracy of subject No.18 ACCDS KSUST It is also the largest, with a value of 92.80%. i The smallest one is the selection factor CF of subject No.5 i (0.79), Accuracy of Subject No.5 ACCDS KSUST is the smallest, with a value of 68.37%. This shows that the selection factor CF i The fatigue driving identification model USDRMD established by deep transfer learning based on a large known subject KS has a higher identification accuracy for the driving state of the unknown subject US. Therefore, the selection factor CF should be used first. i A larger known subject KS is used for deep transfer learning to establish the fatigue driving identification model USDRMD.

[0099] Figure 8 and Fig. 9 The sensitivity SENDS and sensitivity SPEDS of the fatigue driving identification model KSDRMD of each known subject KS before and after migration are respectively shown. Figure 8 The vertical axis on the right is sensitivity SENDS, sensitivity SENDS KSUS and sensitivity SENDSKSUST They represent the sensitivity SENDS of the fatigue driving identification model KSDRMD to identify the driving state of the unknown subject US before and after deep transfer learning. Fig. 9 The right vertical axis is the specificity SPEDS. KSUS and specificity SPEDS KSUST They respectively represent the specificity SPEDS of the fatigue driving identification model KSDRMD in identifying the driving status of the unknown subject US before and after deep transfer learning. Figure 8 and Fig. 9 The meaning of the left vertical and horizontal axes is the same as Figure 7 same.

[0100] Depend on Figure 8 and Fig. 9 It can be seen that compared with accuracy ACCDS, the effect of deep transfer learning on sensitivity SENDS and specificity SPEDS is more complex. Figure 8 In the figure, after deep transfer learning, the sensitivity SENDS of the fatigue driving identification model KSDRMD of each known subject KS changes in different trends. For example, the sensitivity SENDS of subjects No.6, No.7, No.11, and No.17 KSUST Less than sensitivity SENDS KSUS , the sensitivity of the model decreases after deep transfer learning. The sensitivity of the remaining known subjects KS SENDS KSUST Both are greater than the sensitivity SENDS KSUS . SENSITIVITY SENDS KSUST There is a random selection factor CF i The results show that the selection factor CF i The fatigue driving recognition model KSDRMD of a larger known subject KS usually has enhanced recognition ability for fatigue samples of unknown subjects US after deep transfer learning. However, the one-sided pursuit of high sensitivity will lead to increased misjudgment, so the transfer learning effect needs to be comprehensively analyzed in combination with specificity SPEDS.

[0101] exist Fig. 9 The specificity SPEDS of each known subject KS KSUST Both were higher than SPEDS KSUS This shows that deep transfer learning can increase the specificity SPEDS of the fatigue driving identification model KSDRMD in identifying the driving state of unknown subjects US, and improve the recognition ability of the fatigue driving identification model KSDRMD for the awake samples of unknown subjects US. Specificity SPEDS KSUST There is also a selection factor CF i Increase and approximate growth trend, select factor CF iThe specificity of the larger known subject KS SPEDS KSUST The specificity of subject No. 14 SPEDS is usually higher, and its fatigue driving identification model KSDRMD has a stronger ability to recognize the awake samples of unknown subjects US after deep migration. KSUST The specificity SPEDS reached 100%, which is higher than that of subject No.18. KSUST (98.50%), however, the sensitivity of subject No.14 SENDS KSUST (83.08%) is lower than the sensitivity of subject No.18 SENDS KSUST (89.03%). This shows that compared with subject No. 18, the deep transfer learning of subject No. 14 to establish the fatigue driving identification model USDRMD can identify all the sober samples of the unknown subject US, but some fatigue samples are mistakenly classified as sober samples, with a high false alarm rate. Therefore, the comprehensive sensitivity SENDS KSUST and specificity SPEDS KSUST , selecting subject No.18 as the best known subject BKS, and performing deep transfer learning to establish USDRMD has a better effect.

[0102] In addition, compared Figure 8 and Fig. 9 It can be seen that compared with the model effect before deep transfer, the specificity of all known subjects KS SPEDS KSUST The average improvement (49.97%) is higher than the sensitivity of all known subjects KS SENDS KSUST The average improvement was (10.42%). This result shows that compared with the sensitivity SENDS, deep transfer learning has a better improvement on the specificity SPEDS. After deep transfer learning, the ability of the fatigue driving identification model KSDRMD to identify the sober samples of the unknown subject US is greatly improved, and the false alarm rate is reduced. The reason may be that the driving data of the unknown subject US in the initial driving stage reflects the driving pattern characteristics of the sober state. The sober driving data of the unknown subject US is used to retrain the fatigue driving identification model KSDRMD of the known subject KS, so that the fatigue driving identification model KSDRMD obtains the sober driving pattern feature information of the unknown subject US, which significantly improves the fatigue driving identification model KSDRMD's ability to identify the sober samples of the unknown subject US.

[0103] Fig.10 The F1 score F1DS of the fatigue driving identification model KSDRMD of the known subject KS before and after migration to identify the driving status of the unknown subject US. KSUS F1 Score and F1DS KSUSTThey represent the F1 score F1DS of the fatigue driving identification model KSDRMD before and after deep transfer learning to identify the driving state of the unknown subject US. The vertical axis on the right is the F1 score F1DS, and the meanings of the vertical and horizontal axes on the left are the same as Figure 7 Same. Fig.10 As shown, the F1 score F1DS of each known subject KS KSUST Both are higher than F1 score F1DS KSUS , indicating that deep transfer learning can improve the comprehensive recognition ability of the fatigue driving recognition model KSDRMD of each known subject KS to the driving state of the unknown subject US. In addition, with the selection factor CF i The increase in F1 score F1DS KSUST There is an increasing trend, select factor CF i The largest F1 score F1DS of the No.18 test subject KSUST is the largest, reaching 93.70%. This shows that the overall performance of selecting subject No.18 for deep transfer learning and establishing the fatigue driving identification model USDRMD is better, which can reduce the missed alarm rate and false alarm rate.

[0104] according to Figure 1 The method treats each subject as an unknown subject US for selective deep transfer learning, and repeats it 24 times Figure 1 The best known subject KS is selected for deep transfer learning to establish a fatigue driving recognition model for unknown subject US. The fatigue driving recognition model BKSDRMD of the best known subject BKS is established after deep transfer learning. The accuracy, sensitivity, specificity and F1 score of the fatigue driving recognition model USDRMD for unknown subject US driving state recognition are expressed as accuracy ACCDS BKSUST 、SENSITIZATIONSENDS BKSUST , specificity SPEDS BKSUST 、F1 score F1DS BKSUST The average accuracy of repeated modeling results is ACCDS BKSUST 、SENSITIZATIONSENDS BKSUST , specificity SPEDS BKSUST 、F1 score F1DS BKSUST They are 89.26%, 87.74%, 90.28% and 86.74% respectively, which are 21.32%, 8.05%, 30.18% and 20.18% higher than before deep transfer learning. In summary, the fatigue driving identification model USDRMD established by selecting different known subjects KS for deep transfer learning has different identification effects on the driving status of unknown subjects US. i Deep transfer learning with a larger known subject KS usually achieves better driving state recognition. The reason may be the selection factor CF iThe larger the value is, the smaller the difference in driving patterns between the unknown subject US and the known subject KS is, the more similar the driving pattern characteristics under different driving states are, the easier it is to perform deep transfer learning on the unknown subject US, and the better the recognition effect of the driving state of the unknown subject US is after transfer.

[0105] The above is only a preferred embodiment of the present invention. Therefore, any equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the protection scope of the patent application of the present invention.

Claims

1. A selective deep transfer learning method based on driving pattern difference, characterized in that: The method is implemented by the following steps: Step A, randomly set multiple known subjects KS and multiple unknown subjects US; Step B: Establish a pre-trained model library for deep transfer learning; Step C: Select the best known subject BKS for deep transfer learning; Step D: Retrain the pre-trained model using the data of the unknown subject US, complete deep transfer learning, and establish a fatigue driving recognition model for the unknown subject US.

2. The selective deep transfer learning method based on driving pattern difference according to claim 1, characterized in that: In step A: The data for known subjects’ KS include: pre-acquired sober driving data set and fatigue driving data set; The data for unknown subjects US include: initial stage data set and remaining data set; The initial stage data set is the driving data within the initial driving time IPT; the remaining data set is the remaining data after removing the initial driving time IPT; Initial driving time IPT≤30min.

3. The selective deep transfer learning method based on driving pattern difference according to claim 2, characterized in that: In step B, a driving pattern-based fatigue driving identification model DRMD is established for each known subject KS; Combining principal component analysis (PCA) with one-dimensional convolutional neural network (1D-CNN), the driving behavior features can be automatically extracted and learned to identify whether the driver is in a fatigue state.

4. The selective deep transfer learning method based on driving pattern difference according to claim 3 is characterized in that: For the i-th known subject KS, the entire sample set is first divided into a training set and a test set using 5-fold cross validation. Then, the training set is used to train the fatigue driving identification model DRMD based on the combined model PCA+1DCNN of principal component analysis PCA and one-dimensional convolutional neural network 1D-CNN. Finally, the test set is used to test and select the fatigue driving identification model DRMD with the highest identification accuracy as the fatigue driving identification model DRMD for transfer learning of the known subject KS, which is expressed as the fatigue driving identification model KSDRMD.

5. The selective deep transfer learning method based on driving pattern difference according to claim 4 is characterized in that: The process of selecting the best known subject BKS for deep transfer learning includes: calculating the relationship between the unknown subject US and one of the known subjects KS i The selection factor CF of the driving pattern difference i , select the best known subject BKS with the smallest difference from the unknown subject's US driving pattern for deep transfer learning; The initial data set of the unknown subject US is mixed with the test data of each known subject KS, and then the mixed test data is input into the fatigue driving identification model DRMD of each known subject KS, and the driving state identification accuracy of the mixed test data set is calculated one by one; by comparing the fatigue driving identification model KSDRMD of each known subject KS i The driving state recognition accuracy of the test set containing the initial stage data of unknown subjects is determined by using the selection factor CF i Calculation method: Determine the best known subject BKS for deep transfer learning based on the selection factor score.

6. The selective deep transfer learning method based on driving pattern difference according to claim 5, characterized in that: Using the best known subject BKS results, deep transfer learning is used to establish the fatigue driving identification model USDRMD for the unknown subject US.

7. The selective deep transfer learning method based on driving pattern difference according to claim 6, characterized in that: Using deep transfer learning to establish the fatigue driving identification model USDRMD of unknown subjects US includes: Step ①: retrieve the best known subject BKS fatigue driving identification model BKSDRMD and use it as a pre-training model for deep transfer learning; Step ②: Construct a training data set for deep transfer learning; compensate the unknown subject US’s sober driving pattern knowledge to the unknown subject US’s fatigue driving recognition model USDRMD; supplement the training set of the best known subject BKS to form a training data set; Step ③: Use deep transfer learning to process the fatigue driving identification model BKSDRMD of the best known subject BKS, and establish the fatigue driving identification model USDRMD of the unknown subject US; the fatigue driving identification model BKSDRMD is a fatigue driving identification model based on deep learning; the fatigue driving identification model BKSDRMD of the best known subject BKS is used as a pre-trained model, the pre-trained model is retrained using the training data set, the network parameter adjustment coefficient AC is designed, and the fatigue driving identification model BKSDRMD of the best known subject BKS is transferred to the fatigue driving identification model USDRMD of the unknown subject US based on the deep transfer learning method; Step ④: Use the remaining US data of unknown subjects to verify the effect of selective deep transfer learning based on driving pattern differences, and analyze the influence of network parameter adjustment coefficient AC and initial driving time IPT.

8. The selective deep transfer learning method based on driving pattern difference according to claim 7, characterized in that: Selection Factor CF i The calculation formula is: Where, CF i is the selection factor score of the i-th known subject KS, 1≤i≤nk; nk is the number of known subjects KS; KSDRMD i KS is the known subject i The highest precision KSDRMD; mixd i KS is the known subject i The mixed data of the test data and the initial driving data of the unknown subject US; ACCDS (KSDRMD i ,mixd i ) is KSDRMD i About Mixd i Driving state recognition accuracy.

Images