Construction method, system and equipment of driver fatigue judgment model and medium

Abstract

The invention provides a construction method, system and device of a driver fatigue judgment model and a medium. The construction method comprises the steps that S1, vehicle motion information, a subjective fatigue state and driver behavior information are obtained; s2, calculating mutual information between the vehicle motion information and the subjective fatigue state, and screening the vehicle motion information according to the mutual information to obtain a first fatigue feature; s3, performing deep feature extraction on the driver behavior information to obtain a second fatigue feature; s4, performing feature fusion on the first fatigue feature and the second fatigue feature to obtain a fused feature; and S5, repeating the steps S1-S4 to obtain multiple groups of fusion features, then constructing an initial discrimination model, and training the initial discrimination model by using all the fusion features and the corresponding subjective fatigue states to obtain a driver fatigue determination model. The problem that in the prior art, a detection result has one-sidedness due to the fact that a single type of driver information is collected is solved.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving detection technology, and in particular to a method, system, device and medium for constructing a driver fatigue judgment model. Background Technology

[0002] Fatigue driving is one of the main causes of traffic accidents. When drivers are fatigued, their perception, cognitive processing, and decision-making abilities are severely impaired, leading to a significant reduction in their ability to control the vehicle and thus increasing the risk of traffic accidents. Drivers are usually aware that they are fatigued, but many choose to continue driving out of a sense of complacency. Therefore, driver fatigue detection is crucial for reducing traffic accidents. Existing fatigue detection methods often rely on single-category facial or physiological information of the driver to determine fatigue levels. By extracting fatigue features from the information and comparing the changes in these features before and after fatigue, a simple numerical judgment is made against a preset fatigue threshold to determine the driver's fatigue state. However, obtaining only a single category of driver information and making simple numerical judgments leads to one-sided and incomplete detection results. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method for constructing a driver fatigue judgment model, which solves the problem that the detection results are one-sided due to the collection of driver information from a single category in existing technologies.

[0004] According to an embodiment of the present invention, a method for constructing a driver fatigue judgment model includes: S1: Acquire vehicle motion information, subjective fatigue status, and driver behavior information; S2: Calculate the mutual information between vehicle motion information and subjective fatigue state, and filter the vehicle motion information based on the mutual information to obtain the first fatigue feature; S3: Perform deep feature extraction on driver behavior information to obtain the second fatigue feature; S4: The first fatigue feature and the second fatigue feature are fused to obtain the fused feature; S5. Repeat steps S1-S4 to obtain multiple sets of fused features. Then, construct an initial discrimination model and train the initial discrimination model using all fused features and the corresponding subjective fatigue state to obtain a driver fatigue judgment model.

[0005] Preferably, the vehicle motion information includes driving behavior characteristics and vehicle motion characteristics; Methods for calculating the mutual information between vehicle motion information and subjective fatigue state, and for filtering vehicle motion information based on the mutual information to obtain the first fatigue characteristic, include: Calculate the mutual information between all driving behavior features and vehicle motion features and subjective fatigue state, and then select the driving behavior feature or vehicle motion feature with the largest mutual information as the baseline feature, and the remaining features as features to be determined. Calculate the redundancy between all undetermined features and the baseline feature, and then calculate the mRMR value for each undetermined feature based on the redundancy and mutual information corresponding to the undetermined features. All undetermined features and benchmark features with mRMR values ​​greater than the preset value are combined into the first fatigue feature.

[0006] Preferably, the formula for calculating the redundancy is as follows: in, As the benchmark feature, Let F be the features to be determined, and let F be the set of features to be determined.

[0007] Preferably, the driver behavior information includes driver facial information and driver head movement information; Methods for extracting deep features from driver behavior information to obtain the second fatigue feature include: The driver behavior information is convolved through three convolution kernels of different sizes to obtain the corresponding first-dimensional feature maps. Then, all the first-dimensional feature maps are concatenated to obtain the convolutional features. Perform three different max pooling operations on the first dimension feature to obtain the corresponding second dimension feature map. Then, concatenate all the second dimension feature maps and reduce the dimensionality to obtain multi-scale pooled features. The graph neural network node graph is obtained based on the driver's facial information. Then, deep features are extracted from the multi-scale pooling features based on the graph neural network node graph to obtain deep features. The deep features are mapped to dimensions to obtain the second fatigue feature.

[0008] Preferably, the method for extracting deep features from multi-scale pooling features based on the graph neural network node graph to obtain deep features includes: Based on the graph neural network node graph, the multi-scale pooling features are reduced in dimensionality to obtain graph convolution features; Based on the graph neural network node graph and graph convolution features, the GCN model is used to extract facial spatial features; The Transform model is used to extract attention features from facial spatial features, resulting in deep features.

[0009] Preferably, after obtaining the second-dimensional feature map, the driver behavior information is convolved using a 1x1 convolution kernel to obtain the convolutional dimension feature map, and then the convolutional dimension feature map is concatenated with all the second-dimensional feature maps.

[0010] Preferably, the method for fusing the first fatigue feature and the second fatigue feature to obtain the fused feature includes: After standardizing the first fatigue feature and the second fatigue feature, the Transform model is used to encode the features respectively to obtain the corresponding first encoded feature and second encoded feature; The second coding feature is conditionally deconditioned using the first coding feature to obtain the fused feature.

[0011] On the other hand, according to embodiments of the present invention, a non-invasive driver fatigue assessment system is also provided, which uses the above-described method for constructing a driver fatigue assessment model, including: The data acquisition module is used to acquire vehicle motion information, subjective fatigue state, and driver behavior information; A filtering module is used to calculate the mutual information between vehicle motion information and subjective fatigue state, and to filter the vehicle motion information based on the mutual information to obtain the first fatigue feature. A feature extraction module is used to perform deep feature extraction on driver behavior information to obtain a second fatigue feature. A feature fusion module is used to fuse a first fatigue feature and a second fatigue feature to obtain a fused feature. The model training module is used to construct an initial discrimination model and train the initial discrimination model using all fused features and corresponding subjective fatigue states to obtain a driver fatigue judgment model.

[0012] On the other hand, according to an embodiment of the present invention, a computer is also provided, including at least one processor and a memory, the memory storing a computer program configured to be executed by the processor to implement the above-described method for constructing a driver fatigue judgment model.

[0013] On the other hand, according to an embodiment of the present invention, a storage medium is also provided, which is a computer-readable storage medium, and a computer program is stored on the storage medium. The computer program can be executed by one or more processors to implement the above-described method for constructing a driver fatigue judgment model.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention extracts features from three types of information—vehicle motion information, subjective fatigue state, and driver behavior information—using different information extraction methods. The extracted features are then fused to train a driver fatigue judgment model that can determine whether a driver is fatigued from multiple perspectives, thereby making a comprehensive and objective judgment on whether a driver is fatigued. Attached Figure Description

[0015] Figure 1 This diagram illustrates the method for constructing a driver fatigue assessment model according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] like Figure 1 As shown in the figure, this invention proposes a method for constructing a driver fatigue judgment model, including: S1: Acquire vehicle motion information, subjective fatigue status, and driver behavior information; Before training the model, data preparation is required. This invention selects a vehicle as the experimental vehicle, and the driver operates the experimental vehicle to drive normally on the road. A CAN analyzer is connected to the CAN bus of the experimental vehicle to collect and store various motion data of the vehicle in real time (mainly steering wheel signals and vehicle speed signals). The signal reception frequency is set to 100 Hz. Vehicle motion information is obtained through this method.

[0018] During the driving process, an infrared binocular sensor installed directly in front of the driver's seat collects and stores the driver's behavioral information (mainly facial changes and head movements) in real time, saving it as video for subsequent feature extraction. This method obtains driver behavior information and uses a small, non-contact infrared binocular sensor to collect information, avoiding the subtle influence on human perception caused by wearing corresponding sensors on the body, thus improving the accuracy of the collected information.

[0019] Drivers are required to report their fatigue level every 10 minutes based on the KSS scale during driving, and record it promptly. This report will be used as the driver's subjective fatigue perception for those 10 minutes. A KSS < 6 indicates a state of alertness, while a KSS >= 6 indicates a state of fatigue. Based on these rules, KSS will be used as the data label. This allows for the labeling of vehicle motion information and driver behavior information to obtain subjective fatigue status.

[0020] By acquiring vehicle motion information and driver behavior information simultaneously with subjective fatigue status, and aligning these three types of data in the time dimension, subjective fatigue status can serve as a data status label for vehicle motion information and driver behavior information.

[0021] Four types of data—driving duration, vehicle motion signals, driver facial signals, and driver subjective fatigue level—are aligned along the time dimension. Among them, vehicle motion signals and driver facial signals serve as the main information for judging driver fatigue, while the driver fatigue level based on KSS is used as a data status label, which is then used as a basis for evaluating the detection performance.

[0022] S2: Calculate the mutual information between vehicle motion information and subjective fatigue state, and filter the vehicle motion information based on the mutual information to obtain the first fatigue feature; Based on the steering wheel signal and vehicle speed signal acquired via the CAN bus, feature extraction is performed to obtain vehicle motion information, which can be divided into two categories: driving behavior features and vehicle motion features. Before formal feature extraction, the vehicle motion information is preprocessed through filtering, noise reduction, and other preprocessing modules to improve its signal-to-noise ratio.

[0023] Driving behavior characteristics Fd represent the various performance indicators and their mathematical statistics corresponding to the driver's control of the vehicle during driving, including: steering wheel angle, steering wheel angle angular velocity, steering wheel force feedback, vehicle lateral displacement, vehicle lateral acceleration, vehicle longitudinal velocity, vehicle longitudinal acceleration, etc., as well as their mean, maximum value, standard deviation, extreme value mean, and extreme value mean standard deviation per unit time.

[0024] Vehicle motion characteristics (Fo) refer to the driving characteristics exhibited by a vehicle during driving, including: vehicle lane crossing frequency, vehicle lane crossing time, number of times the steering wheel crosses zero, steering wheel stationary time, steering wheel stationary angle product, vehicle offset area, number of times the vehicle brakes suddenly, and vehicle braking amplitude.

[0025] By performing correlation analysis between the extracted signals and the corresponding fatigue states, preliminary feature selection was achieved, and fatigue characteristics based on vehicle motion were obtained. The process is as follows: (1) Calculate the mutual information between all vehicle motion information and the corresponding subjective fatigue state. The calculation formula is as follows: in, Let represent the i-th feature in the vehicle motion information; y represents the subjective fatigue state; and I is the mutual information value, calculated as follows: Where P(k,j) is the joint probability distribution, representing The probabilities of y taking the value k and y taking the value j; P(k) and P(j) are respectively The marginal probability distributions of y and y respectively represent The probability of y taking the value k and the probability of y taking the value j.

[0026] Then, the feature with the largest mutual information value is used as the baseline feature, and the remaining features are used as features to be determined.

[0027] (2) Calculate the redundancy between all undetermined features and the baseline feature. The calculation formula is as follows: in, As the benchmark feature, Let F be the set of features to be determined; |F| is the size of the set of features to be determined.

[0028] (3) Calculate the mRMR value of each feature to be determined using the following formula: Then, based on the mRMR values, the features to be determined with all mRMR values ​​greater than the preset value and the baseline features are combined to form the first fatigue feature. This allows for the initial screening of features.

[0029] S3: Perform deep feature extraction on driver behavior information to obtain the second fatigue feature; Driver behavior information acquired from binocular infrared cameras includes driver facial information (Ff) and driver head movement information (Fh), both in image format. These are fed in parallel into a multi-scale convolutional neural network (MCCNN) and a G-Trans network for signal processing and feature extraction. The MCCNN primarily focuses on extracting short-term signal features, where "multi-scale" refers to multiple scales at a short time level, such as short-term eye closure or distraction. The G-Trans network, on the other hand, is more focused on long-term features and spatial variations in the driver's actions, such as yawning and head movements. Simultaneously, the Transformer module in the G-Trans network enables deep fusion of the features extracted from the two types of signals by the two networks, thereby obtaining multi-scale spatiotemporal features. This provides a more comprehensive representation of driver fatigue at the behavioral level.

[0030] The fusion process is as follows: The input data image (including driver facial information Ff and driver head movement information Fh) X∈RB×C×T, where B is the batch size, C is the image width, and T is the image length. First, the input signal X∈RB×C×T undergoes multi-scale convolution operations using three convolution kernels of different sizes (1×3, 1×11, 1×19) to extract the first-dimensional feature maps of different receptive fields. , , Subsequently, these features are concatenated along the channel dimension to form a new feature representation MT: Where Concat represents tensor concatenation operation, the superscript indicates the dimension of the feature, the subscript indicates the size of the convolution kernel, and C1=36.

[0031] Next, max pooling operations of three different sizes (7, 19, 31) are performed on MT to obtain the second-dimensional feature map. , , Simultaneously, the driver's behavior information is convolved using a 1x1 kernel to obtain a convolutional dimension feature map. Then, the convolutional dimension feature map and all second-dimensional feature maps are concatenated, and then dimensionality reduction is performed to obtain the final multi-scale pooling features. : A graph neural network node graph is obtained based on the driver's facial information. The graph neural network node graph contains multiple key facial nodes and the relationships between nodes, and multi-scale pooling features are then applied. Dimensionality reduction to d=64: Subsequently, the feature K is combined with the edge index relationship of the neural network node graph, and message passing is performed through GCN to extract the local topological relationship between faces. The facial spatial feature G, which integrates global and local information, is obtained through a two-layer GCN network.

[0032] Subsequently, G is transformed into a dense matrix Gdense and fed into the Transformer encoding module. After attention computation, the global network representation is obtained. It also interacts with previous temporal features to complete multi-scale spatiotemporal joint modeling: In this context, Q, K, and V represent the three core components of information interaction: query, key, and value, respectively; WQ, WK, and WV are the trainable parameter matrices for calculating the query, key, and value, respectively; and Attn represents the attention mechanism.

[0033] Finally, the information is fused and reconstructed through the Transformer decoding layer to obtain the decoded deep features. The second fatigue feature is obtained by mapping its channel dimension back to 2T through the output layer. .

[0034] S4: The first fatigue feature and the second fatigue feature are fused to obtain the fused feature; First, regarding the first fatigue characteristic With the second fatigue characteristic Perform standardized operations: Where μ is the mean of the feature and σ is the standard deviation of the feature.

[0035] Subsequently, the standardized features are fed into the Transformer model for feature encoding: Here, Trans is a trainable mapping in the Transformer model, and the two have independent weights, projecting the two modalities into the same dimension.

[0036] Finally, by changing the conditions, the driver's facial features are deconditioned using vehicle features, and the final fused feature F is obtained.

[0037] S5. Repeat steps S1-S4 to obtain multiple sets of fused features. Then, construct an initial discrimination model and train the initial discrimination model using all fused features and the corresponding subjective fatigue state to obtain a driver fatigue judgment model.

[0038] Throughout the training process, the KSS level is collected every 10 minutes for classification processing. At the same time, vehicle motion information and driver behavior information are collected once. By repeating steps S1-S5, multiple sets of fused features and corresponding subjective fatigue states can be obtained.

[0039] An initial discrimination model is constructed, using the obtained fused features F as input and the sobriety / fatigue fragments labeled with the tag y as the correct classification labels. Since this method only needs to distinguish the driver's driving state (sobriety / fatigue), which is a binary classification, the loss function of the initial discrimination model is cross-entropy. The Adam optimizer is used to backpropagate gradients to update the model parameters. Finally, the output of the fully connected layer is transformed through a softmax layer to obtain the fatigue probability and output the final detection result. The final detection result is compared with the subjective fatigue state to adjust the initial discrimination model until the maximum number of iterations is reached or the final detection result of each set of fused features is completely identical to the subjective fatigue state. At this point, training ends, and the driver fatigue judgment model is obtained.

[0040] This invention extracts features from three types of information—vehicle motion information, subjective fatigue state, and driver behavior information—using different information extraction methods. The extracted features are then fused to train a driver fatigue judgment model that can determine whether a driver is fatigued from multiple perspectives, thereby making a comprehensive and objective judgment on whether a driver is fatigued.

[0041] On the other hand, embodiments of the present invention also provide a non-invasive driver fatigue assessment system, which uses the above-described method for constructing a driver fatigue assessment model, including: The data acquisition module is used to acquire vehicle motion information, subjective fatigue state, and driver behavior information; A filtering module is used to calculate the mutual information between vehicle motion information and subjective fatigue state, and to filter the vehicle motion information based on the mutual information to obtain the first fatigue feature. A feature extraction module is used to perform deep feature extraction on driver behavior information to obtain a second fatigue feature. A feature fusion module is used to fuse a first fatigue feature and a second fatigue feature to obtain a fused feature. The model training module is used to construct an initial discrimination model and train the initial discrimination model using all fused features and corresponding subjective fatigue states to obtain a driver fatigue judgment model.

[0042] On the other hand, embodiments of the present invention also provide a computer, including at least one processor and a memory, wherein the memory stores a computer program configured to be executed by the processor to implement the above-described method for constructing a driver fatigue judgment model.

[0043] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium storing a computer program that can be executed by one or more processors to implement the above-described method for constructing a driver fatigue judgment model.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for constructing a driver fatigue judgment model, characterized in that: include: S1: Acquire vehicle motion information, subjective fatigue status, and driver behavior information; S2: Calculate the mutual information between vehicle motion information and subjective fatigue state, and filter the vehicle motion information based on the mutual information to obtain the first fatigue feature; S3: Perform deep feature extraction on driver behavior information to obtain the second fatigue feature; S4: The first fatigue feature and the second fatigue feature are fused to obtain the fused feature; S5: Repeat steps S1-S4 to obtain multiple sets of fused features, then construct an initial discrimination model, and use all fused features and corresponding subjective fatigue states to train the initial discrimination model to obtain a driver fatigue judgment model.

2. The method for constructing a driver fatigue judgment model as described in claim 1, characterized in that: The vehicle motion information includes driving behavior characteristics and vehicle motion characteristics; Methods for calculating the mutual information between vehicle motion information and subjective fatigue state, and for filtering vehicle motion information based on the mutual information to obtain the first fatigue characteristic, include: Calculate the mutual information between all driving behavior features and vehicle motion features and subjective fatigue state, and then select the driving behavior feature or vehicle motion feature with the largest mutual information as the baseline feature, and the remaining features as features to be determined. Calculate the redundancy between all undetermined features and the baseline feature, and then calculate the mRMR value for each undetermined feature based on the redundancy and mutual information corresponding to the undetermined features. All undetermined features and benchmark features with mRMR values ​​greater than the preset value are combined into the first fatigue feature.

3. The method for constructing a driver fatigue judgment model as described in claim 2, characterized in that: The formula for calculating redundancy is as follows: in, As the benchmark feature, Let F be the features to be determined, and let F be the set of features to be determined.

4. The method for constructing a driver fatigue judgment model as described in claim 1, characterized in that: The driver behavior information includes driver facial information and driver head movement information; Methods for extracting deep features from driver behavior information to obtain the second fatigue feature include: The driver behavior information is convolved through three convolution kernels of different sizes to obtain the corresponding first-dimensional feature maps. Then, all the first-dimensional feature maps are concatenated to obtain the convolutional features. Perform three different max pooling operations on the first-dimensional feature map to obtain the corresponding second-dimensional feature map. Then, concatenate all the second-dimensional feature maps and reduce the dimensionality to obtain multi-scale pooled features. The graph neural network node graph is obtained based on the driver's facial information. Then, deep features are extracted from the multi-scale pooling features based on the graph neural network node graph to obtain deep features. The deep features are mapped to dimensions to obtain the second fatigue feature.

5. The method for constructing a driver fatigue judgment model as described in claim 4, characterized in that: Methods for extracting deep features from multi-scale pooling features based on graph neural network node graphs include: Based on the graph neural network node graph, the multi-scale pooling features are reduced in dimensionality to obtain graph convolution features; Based on the graph neural network node graph and graph convolution features, the GCN model is used to extract facial spatial features; The Transform model is used to extract attention features from facial spatial features, resulting in deep features.

6. The method for constructing a driver fatigue judgment model as described in claim 4, characterized in that: After obtaining the second-dimensional feature map, the driver behavior information is convolved with a 1x1 convolution kernel to obtain the convolutional dimension feature map. Then, the convolutional dimension feature map is concatenated with all the second-dimensional feature maps.

7. The method for constructing a driver fatigue judgment model as described in claim 1, characterized in that: Methods for fusing the first fatigue feature and the second fatigue feature to obtain the fused feature include: After standardizing the first fatigue feature and the second fatigue feature, the Transform model is used to encode the features respectively to obtain the corresponding first encoded feature and second encoded feature; The second coding feature is conditionally deconditioned using the first coding feature to obtain the fused feature.

8. A non-invasive driver fatigue assessment system, characterized in that: The system uses a method for constructing a driver fatigue judgment model as described in any one of claims 1-7, comprising: The data acquisition module is used to acquire vehicle motion information, subjective fatigue state, and driver behavior information; A filtering module is used to calculate the mutual information between vehicle motion information and subjective fatigue state, and to filter the vehicle motion information based on the mutual information to obtain the first fatigue feature. A feature extraction module is used to perform deep feature extraction on driver behavior information to obtain a second fatigue feature. A feature fusion module is used to fuse a first fatigue feature and a second fatigue feature to obtain a fused feature. The model training module is used to construct an initial discrimination model and train the initial discrimination model using all fused features and corresponding subjective fatigue states to obtain a driver fatigue judgment model.

9. A computer, characterized in that: It includes at least one processor and a memory, the memory storing a computer program configured to be executed by the processor to implement a method for constructing a driver fatigue judgment model as described in any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. The computer program can be executed by one or more processors to implement a method for constructing a driver fatigue judgment model as described in any one of claims 1-7.

Images