A method, system, and equipment for detecting fatigue driving by train drivers en route.

Abstract

This invention discloses a method, system, and apparatus for detecting fatigue driving by train drivers en route, relating to the field of fatigue driving detection technology. The method includes: inputting a facial image of a candidate driver into a fatigue driving detection model to obtain the driver's fatigue state; the fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set; the preset facial static and dynamic fatigue feature network includes a teacher network and a student network; the teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the student network includes a second ResNet-18 network and a second LSTM network connected sequentially. This invention ensures the real-time performance of train driver fatigue driving detection en route and improves detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving detection technology, and in particular to a method, system and equipment for detecting fatigue driving of train drivers en route. Background Technology

[0002] Driver fatigue is a significant human factor in train accidents, and proper driver fatigue risk management is a crucial task in preventing such accidents. Although relevant departments have relatively comprehensive measures for managing driver fatigue before they start work, the inherent characteristics of train driving tasks—long working hours, high work pressure, and monotonous tasks—make driver fatigue difficult to avoid.

[0003] Existing fatigue detection methods primarily rely on detecting the appearance of key facial features. Unlike drowsiness, train drivers' fatigue tends to manifest as subtle mental fatigue, which existing methods struggle to detect. Regarding the lighting issue, there are currently two main approaches: 1. Using traditional RGB cameras to capture images and extracting lighting-invariant features through a series of lighting processing algorithms. However, under harsh lighting conditions, the effectiveness of this feature extraction is difficult to guarantee. 2. Using near-infrared cameras to capture images under adverse lighting conditions by supplementing with infrared light sources. However, the image quality of current near-infrared images is generally lower than that of RGB cameras, affecting the accuracy of fatigue detection. Furthermore, train driving mileage is typically long, drivers work extended hours, and the amount of video data is enormous. However, deep learning methods typically employ wide and deep networks, making it difficult to guarantee real-time fatigue detection. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and equipment for detecting fatigue driving of train drivers en route, ensuring the real-time nature of fatigue driving detection and improving detection accuracy.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] In a first aspect, the present invention provides a method for detecting fatigue driving of train drivers en route, comprising:

[0007] Obtain the facial image of the driver to be used;

[0008] The driver's facial image is input into the fatigue driving detection model to obtain the driver's fatigue state; the driver fatigue state includes driver fatigue and driver non-fatigue; wherein, the fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a facial sample image and the corresponding facial fatigue state; the preset facial static and dynamic fatigue feature network includes a teacher network and a student network;

[0009] The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the pre-trained first ResNet-18 network is used to extract static fatigue features of the face; the pre-trained FlowNetS network is used to extract dynamic fatigue features of the face; the attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic and static fatigue feature of the face; the first LSTM network is used to extract features from the first dynamic and static fatigue feature of the face.

[0010] The student network includes a second ResNet-18 network and a second LSTM network connected in sequence; the second ResNet-18 network is used to extract second facial dynamic fatigue features; the second LSTM network is used to extract features from the second facial dynamic fatigue features.

[0011] Optionally, the output of the pre-trained first ResNet-18 network is connected to the first input of the attention feature fusion network through a first fully connected layer, the output of the pre-trained FlownetS network is connected to the second input of the attention feature fusion network, the output of the attention feature fusion network is connected to the input of the first LSTM network, and the output of the first LSTM network is connected to the first softmax layer through a second fully connected layer.

[0012] The output of the second ResNet-18 network is connected to the input of the second LSTM network through a third fully connected layer, and the output of the second LSTM network is connected to the second softmax layer through a fourth fully connected layer.

[0013] The input terminals of the pre-trained first ResNet-18 network and the second ResNet-18 network are both used to input the first training sample set.

[0014] Optionally, the process of training a preset facial static and dynamic fatigue feature network using a first training sample set specifically includes:

[0015] A comprehensive loss function is constructed to use the driver fatigue state prediction results output by the teacher network as the training objective of the student network, thereby guiding the training of the student network.

[0016] The comprehensive loss function is obtained by weighting the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function; the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function are all MSE loss functions.

[0017] The first soft loss function is determined based on the difference between the first facial motion and static fatigue feature output by the attention feature fusion network and the second facial motion and static fatigue feature output by the second ResNet-18 network;

[0018] The second soft loss function is determined based on the difference between the feature vector output by the first LSTM network and the feature vector output by the second LSTM network;

[0019] The first cross-entropy loss function is determined by the difference between the feature vector output by the first LSTM network and the true label; the second cross-entropy loss function is determined by the difference between the feature vector output by the second LSTM network and the true label; the true label is the facial fatigue state corresponding to the facial sample image in the training samples.

[0020] Optionally, the process of determining the first training sample set specifically includes:

[0021] Select a sequence of RGB images of a human face;

[0022] Using MTCNN, face recognition and face cropping are performed on each frame of the face RGB image sequence to obtain multiple face images of the same size;

[0023] Multiple facial images of the same size are sampled at preset time intervals to obtain the first training sample set.

[0024] Optionally, the process of determining the pre-trained FlownetS network specifically includes:

[0025] Based on the Flychairs database, a second training sample set is determined; each sample in the second training sample set includes an NIR sample image sequence and the corresponding optical flow features.

[0026] The second training sample set is input into the FlownetS network for feature extraction, and then trained using an optimizer and a preset endpoint error loss function to obtain the trained FlownetS network; the preset endpoint error loss function is used to measure the difference between the predicted optical flow and the actual optical flow.

[0027] Optionally, the pre-training of the first ResNet-18 network, the pre-training of the FlownetS network, and the training of the preset face static and dynamic fatigue feature network are all implemented using the Adam optimizer.

[0028] Secondly, the present invention provides a train driver fatigue detection system, comprising:

[0029] The standby image acquisition module is used to acquire the facial image of the standby driver;

[0030] A fatigue driving detection module is used to input the facial image of the candidate driver into a fatigue driving detection model to obtain the driver's fatigue state; the driver fatigue state includes driver fatigue and driver non-fatigue; wherein, the fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a facial sample image and the corresponding facial fatigue state; the preset facial static and dynamic fatigue feature network includes a teacher network and a student network;

[0031] The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the pre-trained first ResNet-18 network is used to extract static fatigue features of the face; the pre-trained FlowNetS network is used to extract dynamic fatigue features of the face; the attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic and static fatigue feature of the face; the first LSTM network is used to extract features from the first dynamic and static fatigue feature of the face.

[0032] The student network includes a second ResNet-18 network and a second LSTM network connected in sequence; the second ResNet-18 network is used to extract second facial dynamic fatigue features; the second LSTM network is used to extract features from the second facial dynamic fatigue features.

[0033] Thirdly, the present invention provides an electronic device, characterized in that it includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a method for detecting fatigue driving of train drivers en route.

[0034] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0035] This invention discloses a method, system, and device for detecting fatigue driving by train drivers en route. A fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set, and the driver's facial image is detected in real time to obtain the driver's fatigue state. The pre-defined facial static and dynamic fatigue feature network includes a teacher network and a student network. The teacher network comprises a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network. The pre-trained first ResNet-18 network extracts static facial fatigue features; the pre-trained FlowNetS network extracts dynamic facial fatigue features, i.e., it introduces an optical flow module to improve the network's ability to recognize subtle changes; the attention feature fusion network fuses the static and dynamic facial fatigue features to obtain the first facial static and dynamic fatigue features; and the first LSTM network extracts features from the first facial static and dynamic fatigue features. The student network comprises a second ResNet-18 network and a second LSTM network connected sequentially. The second ResNet-18 network extracts second facial static and dynamic fatigue features; and the second LSTM network extracts features from the second facial static and dynamic fatigue features. The introduction of the teacher and student networks simplifies the network structure by using knowledge distillation to preserve the original network's perceptual capabilities as much as possible. This invention uses the pre-trained static and dynamic facial fatigue feature network, i.e., the fatigue driving detection model, to detect subtle changes in the fatigue state of train drivers, thereby improving detection accuracy. It also has anti-lighting interference capabilities and a certain degree of real-time performance. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the method for detecting fatigue driving of train drivers during the journey according to the present invention.

[0038] Figure 2 This is a schematic diagram of the fatigue driving detection model of the present invention;

[0039] Figure 3 This is a schematic diagram of the train driver fatigue detection system of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention provides a method, system, and device for detecting fatigue driving by train drivers en route. It integrates static fatigue features of the face and optical flow motion features of the face, and uses an LSTM network to predict fatigue state. It introduces a knowledge distillation teacher-student network training method to reduce the complexity of the student network. It inputs RGB image sequences and NIR image sequences into the teacher network and student network respectively to improve the student network's ability to recognize NIR images while ensuring accuracy.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Example 1

[0044] like Figure 1 As shown, the present invention provides a method for detecting fatigue driving of train drivers en route, comprising:

[0045] Step 100: Obtain the facial image of the driver to be used; the facial image of the driver to be used is generally an RGB image.

[0046] Step 200: Input the facial image of the driver to be used into the fatigue driving detection model to obtain the driver's fatigue state; the driver fatigue state includes driver fatigue and driver non-fatigue.

[0047] like Figure 2 As shown, the fatigue driving detection model is obtained by training a preset face static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a face sample image and the corresponding face fatigue state; the preset face static and dynamic fatigue feature network includes a teacher network and a student network.

[0048] The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network. The pre-trained first ResNet-18 network is used to extract static fatigue features of the face. The pre-trained FlowNetS network is used to extract dynamic fatigue features of the face. The attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic fatigue feature of the face. The first LSTM network is used to extract features from the first dynamic fatigue feature of the face. The student network includes a second ResNet-18 network and a second LSTM network connected in sequence. The second ResNet-18 network is used to extract a second dynamic fatigue feature of the face. The second LSTM network is used to extract features from the second dynamic fatigue feature of the face.

[0049] The output of the pre-trained first ResNet-18 network is connected to the first input of the attention feature fusion network through a first fully connected layer. The output of the pre-trained FlownetS network is connected to the second input of the attention feature fusion network. The output of the attention feature fusion network is connected to the input of the first LSTM network. The output of the first LSTM network is connected to the first softmax layer through a second fully connected layer.

[0050] The output of the second ResNet-18 network is connected to the input of the second LSTM network through a third fully connected layer, and the output of the second LSTM network is connected to the second softmax layer through a fourth fully connected layer.

[0051] The input terminals of the pre-trained first ResNet-18 network and the second ResNet-18 network are both used to input the first training sample set.

[0052] To fuse static and dynamic facial features and construct a teacher network, an attention-based feature fusion mechanism is used. This mechanism weights and fuses features at different levels according to their contribution to fatigue status. By introducing an attention mechanism, it automatically learns which features are more important for judging fatigue status and accordingly strengthens or weakens their influence. Feeding the fused features into the teacher network for training yields more accurate predictions of facial fatigue status.

[0053] A second ResNet-18 was selected as the backbone of the student network to extract static and dynamic features from face images. An LSTM network was introduced to process the fused feature sequence for more accurate fatigue prediction. The fused feature sequence contains temporal information of both static and dynamic features, representing the evolution of facial fatigue state over time. By inputting the feature sequence into the LSTM network, the correlation between features at different time steps was learned, capturing dynamic changes over time. LSTM networks with identical structures were applied to both the teacher and student ends to achieve knowledge transfer and distillation.

[0054] (a) Training on fatigue driving detection models.

[0055] (1) The process of determining the first training sample set specifically includes:

[0056] 11) Select a sequence of RGB face images; specifically, use a sequence of 1800 RGB images as input to the teacher network. This image sequence can contain continuous face images to capture dynamic changes in the face.

[0057] 12) Using MTCNN (Multi-task Cascaded Convolutional Networks), face recognition and face cropping are performed on each frame of the face RGB image sequence to obtain multiple face images of the same size. MTCNN can detect and locate faces in the images, reducing the complexity of the input data. Further cropping reduces the size of the input data while retaining key facial information for subsequent feature extraction. The cropped face image size is 224x224, a common image size and the input size accepted by many classic convolutional neural networks.

[0058] 13) Sample multiple facial images of the same size at preset time intervals to obtain the first training sample set. Specifically, extract 60 frames of image data from multiple facial images of the same size at certain time intervals (e.g., every 1 second). This captures the static features of the face for static fatigue feature extraction. For the overall network training, random downsampling is used to handle the data imbalance problem. Furthermore, the input image of the facial static fatigue feature network (the input image of the teacher network) and its next image are used as the input of the student network.

[0059] (2) The process of training the preset face static and dynamic fatigue feature network using the first training sample set specifically includes: constructing a comprehensive loss function to use the driver fatigue state prediction result output by the teacher network as the training target of the student network to guide the training of the student network.

[0060] The teacher-side LSTM network can be trained first using a large-scale, labeled fatigue dataset to learn fatigue state predictions. Then, the fused feature sequences are fed into the teacher-side LSTM network to obtain the teacher's predictions. Using the teacher's predictions as the objective, a new loss function is constructed to guide the training of the student network. The goal of the student-side LSTM network is to approximate the teacher's predictions as closely as possible. In this way, the student network can learn from the teacher network's knowledge and improve its own predictive ability.

[0061] The entire system's training process utilizes the backpropagation algorithm for end-to-end optimization. By continuously iterating the training of the student network, it can gradually learn effective representations of static and dynamic features, and fatigue prediction is performed using an LSTM network.

[0062] To achieve knowledge transfer and distillation between the teacher and student networks, the difference in intermediate outputs is used as a soft loss function. Specifically, the difference between the outputs of the teacher and student networks at the corresponding intermediate layers is calculated and added as an additional loss term to the overall loss function. The difference is measured using the mean squared error (MSE) loss function. By calculating the MSE loss between the intermediate layer outputs of the teacher and student networks, the student network is encouraged to approximate the intermediate layer representation of the teacher network as closely as possible. This allows the student network to better learn the knowledge and feature representations of the teacher network, improving the accuracy of fatigue state prediction.

[0063] In addition to the soft loss, to further train the network model to achieve the ability to recognize fatigue states, the cross-entropy loss function is used as the hard loss. The cross-entropy loss function is commonly used in classification tasks and can measure the difference between the predicted result and the true label.

[0064] The fatigue state can be viewed as a binary classification problem, either classified as fatigued (positive class) or non-fatigue (negative class). By comparing the network's output with the true fatigue state labels, cross-entropy loss can be calculated, and this loss can be used to train the network model.

[0065] Based on the above, this invention constructs a comprehensive loss function. The comprehensive loss function balances the importance of soft loss and hard loss through a weighted sum. Specifically, the comprehensive loss function is obtained by weighting the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function; the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function are all MSE loss functions.

[0066] The first soft loss function is determined by the difference between the first dynamic fatigue feature of the face output by the attention feature fusion network and the second dynamic fatigue feature of the face output by the second ResNet-18 network; the second soft loss function is determined by the difference between the feature vector output by the first LSTM network and the feature vector output by the second LSTM network; the first cross-entropy loss function is determined by the difference between the feature vector output by the first LSTM network and the true label; the second cross-entropy loss function is determined by the difference between the feature vector output by the second LSTM network and the true label; the true label is the facial fatigue state corresponding to the facial sample image in the training samples.

[0067] The MSE loss function formula is as follows:

[0068]

[0069] Where n represents the feature dimension, Y i Indicates the true label, This represents the label predicted by the network (or, the feature vector output by the network).

[0070] Simultaneous training using both soft and hard loss allows the student network to better learn from the teacher network and accurately identify fatigue states. This combined loss function design improves the network model's performance, making it more suitable for face fatigue detection tasks.

[0071] Based on the above settings, in a specific instance, the Adam optimizer with momentum of 0.9 and an initial learning rate of 0.01 is used, and training is performed with a batch size of 128 and 100 epochs at a training set to test set ratio of 7:3.

[0072] (ii) Pre-training of the FlownetS network.

[0073] The process of determining the pre-trained FlownetS network specifically includes:

[0074] (1) Based on the Flychairs database, a second training sample set is determined; each sample in the second training sample set includes a sample image sequence and the corresponding optical flow features. Flychairs is a widely used database for optical flow estimation, containing a large number of inter-frame optical flow image sequences, of which the images used are NIR (Near Infrared) images. These image sequences provide rich dynamic information, which helps to train the dynamic feature extraction network.

[0075] (2) Input the second training sample set into the FlownetS network for feature extraction, and train it through an optimizer and a preset endpoint error loss function to obtain the trained FlownetS network; the preset endpoint error loss function is used to measure the difference between the predicted optical flow and the real optical flow. By minimizing the endpoint error, the dynamic feature extraction network can more accurately predict the high-dimensional features of face motion.

[0076] FlownetS is a network that uses optical flow to estimate motion vectors between two consecutive frames. Taking a sequence of face images as input, the FlownetS network computes the output of its intermediate layers, which are high-dimensional motion features of the face. These features represent temporal changes in the face, such as blinking and head rotation. By learning these motion features, a better understanding of the dynamic fatigue state of the face can be achieved. Specifically, the input to the FlownetS network is two adjacent images, and the output of the FlownetS network is 1024-dimensional depth features.

[0077] In a specific example, the Adam optimizer is chosen to train the dynamic feature extraction network. The momentum is set to 0.9, and the initial learning rate is 0.001. The Adam optimizer's momentum can accelerate convergence, and its learning rate adjustment mechanism can adapt to different data distributions. By adjusting the momentum and learning rate, the stability and speed of the optimization process can be controlled.

[0078] The second training sample set was divided into a training set and a test set in a 7:3 ratio. The batch size was set to 10, representing the number of samples used in each training iteration. An appropriate number of iterations was selected, and training was performed for 500 epochs. During this training process, the network parameters were continuously updated to enable it to extract accurate high-dimensional facial motion features.

[0079] (III) Pre-training of the first ResNet-18 network.

[0080] ResNet-18 is a deep residual network consisting of 18 convolutional layers and fully connected layers. This network can extract static features from face images, including facial expressions, eye states, and facial fatigue levels. By training on a large number of labeled face images (a third training sample set), ResNet-18 can learn representations of static facial features, which can then be used for fatigue detection. The face image is labeled with a value between 0 and 1, where 1 indicates the face is fatigued and 0 indicates it is not fatigued; or, 0 indicates fatigued and 1 indicates it is not fatigued.

[0081] The process of determining the third training sample set specifically includes: preparing a database containing a large number of face image sequences. Each image sequence contains 1800 frames of RGB images. To reduce computational burden, 120 frames of RGB images are extracted at certain intervals. This ensures that key static feature information is preserved in the image sequences. Simultaneously, labels are generated for each image sequence, so that the labels of the entire image sequence are used as the labels for a single frame image for training and evaluation.

[0082] In the constructed database, there may be an imbalance between data categories. To address this issue, a random downsampling method is used. This involves randomly selecting a subset of samples from the categories with the most samples, ensuring a relatively balanced number of samples across each category. This prevents the model from becoming overly biased towards any one category, thereby improving the model's generalization performance.

[0083] In a specific example, the Adam optimizer is chosen to train the static feature extraction network. Momentum is set to 0.9, and the initial learning rate is 0.01. The Adam optimizer combines momentum and learning rate adjustment mechanisms, enabling faster convergence and adaptation to different data distributions. By adjusting momentum and learning rate, the stability and speed of optimization can be controlled.

[0084] Furthermore, the batch size was set to 128, and the dataset was divided into training and test sets in a 7:3 ratio. This ensures that the sample distributions of the training and test sets are similar, which helps in evaluating the model's performance and generalization ability. During training, an appropriate number of epochs was selected, and training was performed for 100 epochs.

[0085] Additionally, cross-entropy loss is chosen to measure the difference between the probability distribution output by the model and the true label. By minimizing the cross-entropy loss, the model can more accurately predict different fatigue states.

[0086] In summary, this invention establishes a deep learning fatigue detection network that integrates RGB and NIR images, combining the advantages of both RGB and NIR cameras. An optical flow detection module is added to the network to implement a dual-channel processing algorithm that combines RGB and NIR images. Finally, a knowledge distillation method is used to simplify the network structure while preserving the original network's perceptual capabilities as much as possible. In practical train driver fatigue detection tasks, this network can better detect subtle changes in the fatigue state of train drivers, while also possessing resistance to light interference and a certain degree of real-time performance.

[0087] Example 2

[0088] like Figure 3As shown, in order to implement the technical solution in Embodiment 1 and achieve the corresponding functions and technical effects, this embodiment also provides a train driver fatigue detection system, characterized in that the system includes:

[0089] The standby image acquisition module 101 is used to acquire the facial image of the standby driver.

[0090] The fatigue driving detection module 201 is used to input the facial image of the driver to be tested into the fatigue driving detection model to obtain the driver's fatigue state; the driver fatigue state includes driver fatigue and driver non-fatigue; wherein, the fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a facial sample image and the corresponding facial fatigue state; the preset facial static and dynamic fatigue feature network includes a teacher network and a student network.

[0091] The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the pre-trained first ResNet-18 network is used to extract static fatigue features of the face; the pre-trained FlowNetS network is used to extract dynamic fatigue features of the face; the attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic fatigue feature of the face; the first LSTM network is used to extract features from the first dynamic fatigue feature of the face.

[0092] The student network includes a second ResNet-18 network and a second LSTM network connected in sequence; the second ResNet-18 network is used to extract second facial dynamic fatigue features; the second LSTM network is used to extract features from the second facial dynamic fatigue features.

[0093] Example 3

[0094] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to execute the train driver fatigue detection method of Embodiment 1. Optionally, the above-mentioned electronic device may be a server.

[0095] In addition, this embodiment of the invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the train driver fatigue detection method of Embodiment 1.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0097] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting fatigue driving by train drivers en route, characterized in that, The methods include: Obtain the facial image of the driver to be used; The driver's facial image is input into the fatigue driving detection model to obtain the driver's fatigue state; The driver fatigue state includes driver fatigue and driver non-fatigue; wherein, the fatigue driving detection model is obtained by training a preset face static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a face sample image and the corresponding face fatigue state; the preset face static and dynamic fatigue feature network includes a teacher network and a student network; The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the pre-trained first ResNet-18 network is used to extract static fatigue features of the face; the pre-trained FlowNetS network is used to extract dynamic fatigue features of the face; the attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic and static fatigue feature of the face; the first LSTM network is used to extract features from the first dynamic and static fatigue feature of the face. The student network includes a second ResNet-18 network and a second LSTM network connected in sequence; the second ResNet-18 network is used to extract second facial dynamic and static fatigue features; the second LSTM network is used to extract features from the second facial dynamic and static fatigue features. The process of training a pre-defined static and dynamic fatigue feature network for faces using the first training sample set specifically includes: A comprehensive loss function is constructed to use the driver fatigue state prediction results output by the teacher network as the training objective of the student network, thereby guiding the training of the student network. The comprehensive loss function is obtained by weighting the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function; the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function are all MSE loss functions; The first soft loss function is determined based on the difference between the first facial motion and static fatigue feature output by the attention feature fusion network and the second facial motion and static fatigue feature output by the second ResNet-18 network; The second soft loss function is determined based on the difference between the feature vector output by the first LSTM network and the feature vector output by the second LSTM network; The first cross-entropy loss function is determined by the difference between the feature vector output by the first LSTM network and the true label; the second cross-entropy loss function is determined by the difference between the feature vector output by the second LSTM network and the true label; the true label is the facial fatigue state corresponding to the facial sample image in the training samples.

2. The method for detecting train driver fatigue en route according to claim 1, characterized in that, The output of the pre-trained first ResNet-18 network is connected to the first input of the attention feature fusion network through a first fully connected layer. The output of the pre-trained FlownetS network is connected to the second input of the attention feature fusion network. The output of the attention feature fusion network is connected to the input of the first LSTM network. The output of the first LSTM network is connected to the first softmax layer through a second fully connected layer. The output of the second ResNet-18 network is connected to the input of the second LSTM network through a third fully connected layer, and the output of the second LSTM network is connected to the second softmax layer through a fourth fully connected layer. The input terminals of the pre-trained first ResNet-18 network and the second ResNet-18 network are both used to input the first training sample set.

3. The method for detecting train driver fatigue en route according to claim 1, characterized in that, The process of determining the first training sample set specifically includes: Select a sequence of RGB images of a human face; Using MTCNN, face recognition and face cropping are performed on each frame of the face RGB image sequence to obtain multiple face images of the same size; Multiple facial images of the same size are sampled at preset time intervals to obtain the first training sample set.

4. The method for detecting train driver fatigue en route according to claim 1, characterized in that, The process of determining the pre-trained FlownetS network specifically includes: Based on the Flychairs database, a second training sample set is determined; each sample in the second training sample set includes an NIR sample image sequence and the corresponding optical flow features. The second training sample set is input into the FlownetS network for feature extraction, and then trained using an optimizer and a preset endpoint error loss function to obtain the trained FlownetS network; the preset endpoint error loss function is used to measure the difference between the predicted optical flow and the actual optical flow.

5. The method for detecting train driver fatigue en route according to claim 1, characterized in that, The pre-training of the first ResNet-18 network, the pre-training of the FlownetS network, and the training of the preset face static and dynamic fatigue feature network are all implemented using the Adam optimizer.

6. A train driver fatigue detection system, characterized in that, The system includes: The standby image acquisition module is used to acquire the facial image of the standby driver; A fatigue driving detection module is used to input the facial image of the candidate driver into a fatigue driving detection model to obtain the driver's fatigue state; the driver fatigue state includes driver fatigue and driver non-fatigue; wherein, the fatigue driving detection model is obtained by training a preset facial static and dynamic fatigue feature network using a first training sample set; each training sample in the first training sample set includes a facial sample image and the corresponding facial fatigue state; the preset facial static and dynamic fatigue feature network includes a teacher network and a student network; The teacher network includes a pre-trained first ResNet-18 network, a pre-trained FlowNetS network, an attention feature fusion network, and a first LSTM network; the pre-trained first ResNet-18 network is used to extract static fatigue features of the face; the pre-trained FlowNetS network is used to extract dynamic fatigue features of the face; the attention feature fusion network is used to fuse the static fatigue features of the face with the dynamic fatigue features of the face to obtain a first dynamic and static fatigue feature of the face; the first LSTM network is used to extract features from the first dynamic and static fatigue feature of the face. The student network includes a second ResNet-18 network and a second LSTM network connected in sequence; the second ResNet-18 network is used to extract second facial dynamic and static fatigue features; the second LSTM network is used to extract features from the second facial dynamic and static fatigue features. The process of training a pre-defined static and dynamic fatigue feature network for faces using the first training sample set specifically includes: A comprehensive loss function is constructed to use the driver fatigue state prediction results output by the teacher network as the training objective of the student network, thereby guiding the training of the student network. The comprehensive loss function is obtained by weighting the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function; the first soft loss function, the second soft loss function, the first cross-entropy loss function, and the second cross-entropy loss function are all MSE loss functions; The first soft loss function is determined based on the difference between the first facial motion and static fatigue feature output by the attention feature fusion network and the second facial motion and static fatigue feature output by the second ResNet-18 network; The second soft loss function is determined based on the difference between the feature vector output by the first LSTM network and the feature vector output by the second LSTM network; The first cross-entropy loss function is determined by the difference between the feature vector output by the first LSTM network and the true label; the second cross-entropy loss function is determined by the difference between the feature vector output by the second LSTM network and the true label; the true label is the facial fatigue state corresponding to the facial sample image in the training samples.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store computer programs, and the processor running the computer programs to cause the electronic device to perform the train driver fatigue detection method according to any one of claims 1-5.

Images