Fatigue detection method and device based on temporal self-attention and auxiliary supervision

Abstract

The embodiment of the present application provides a fatigue detection method and device based on temporal self-attention and auxiliary supervision, which extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and uses the temporal self-attention mechanism to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a fatigue detection method and device based on temporal self-attention and auxiliary supervision. Background Art

[0002] With the continuous improvement of road traffic safety requirements, driver fatigue detection technology has received widespread attention. However, existing fatigue detection technology still has many limitations. The current mainstream detection schemes mostly rely on single feature analysis, such as simply monitoring blinking frequency or head posture, which is difficult to fully reflect the driver's fatigue state. These methods face interference factors such as complex lighting conditions and changes in driver posture in actual driving environments, and detection accuracy and stability are difficult to guarantee.

[0003] At the same time, traditional detection methods often use simple threshold judgments or fixed rule classifications, lacking in-depth analysis of the dynamic evolution of fatigue behavior. This static detection mechanism cannot effectively capture the gradual change characteristics of fatigue status over time, and is prone to misjudgment or detection lag. Especially during long-term driving, the driver's fatigue performance often presents complex time series characteristics and multi-dimensional change patterns, and existing technologies are difficult to accurately identify these subtle state changes.

[0004] In addition, the current fatigue detection system also has deficiencies in model training and optimization. Due to the lack of an effective feature learning supervision mechanism, the model often finds it difficult to accurately extract and integrate multi-dimensional fatigue features, which affects the practicality of the system. In practical applications, these technical limitations not only reduce the reliability of the detection system, but also restrict the further development of intelligent driving safety assurance technology. Therefore, it is of great theoretical value and practical significance to develop more advanced fatigue detection methods to achieve accurate identification of the driver's status and timely warning. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a fatigue detection method and device based on temporal self-attention and auxiliary supervision, which can accurately identify the driver's fatigue state and provide reliable technical support for improving road traffic safety.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a fatigue detection method based on temporal self-attention and auxiliary supervision, comprising:

[0008] Collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating an eyelid aspect ratio value based on the eye region key points, calculating an opening and closing degree value based on the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and composing a feature vector sequence with the aspect ratio value, the opening and closing degree value, and the rotation angle value;

[0009] Constructing a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, inputting the feature vector sequence into the trunk feature extraction branch, extracting facial state temporal features through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receiving the temporal features as input, outputting predicted eyelid opening and closing states, mouth opening and closing states, and head posture states, and constructing an auxiliary loss function based on the difference between the predicted states and the true labels;

[0010] The network structure is trained. First, the weighted sum of the auxiliary loss function and the main branch classification loss function is used as the optimization target, and the parameters of the main feature extraction branch and the auxiliary supervision branch are optimized. After the network converges, the parameters of the main feature extraction branch are fixed, and the auxiliary supervision branch is removed. The main feature extraction branch is used as a fatigue driving detection model, and the fatigue driving detection model is deployed to a vehicle-mounted terminal device. The driver's fatigue state is judged based on the fatigue driving detection model and the feature vector sequence.

[0011] Furthermore, the collecting of the driver's facial image, obtaining the facial region through a face detector, and extracting the eye region key points and the mouth region key points in the facial region based on a key point detection algorithm include:

[0012] A vehicle-mounted camera is used to collect a driver's facial image sequence, the collected image sequence is input into a multi-scale face detector, a target detection algorithm is executed on each frame of the image, a rectangular bounding box of the facial area is located, facial image data within the rectangular bounding box is extracted, and illumination equalization and size standardization processing are performed on the facial image data;

[0013] A key point feature map pyramid is constructed for the face image data, the coordinates of facial key points are extracted through a cascaded deformation network, the extracted key points are divided into eye area key points and mouth area key points according to facial areas, a key point index table is established, and smoothing filtering is performed on the key point coordinate data to eliminate jitter.

[0014] Further, the eyelid aspect ratio value is calculated according to the key points of the eye area, the opening and closing value is calculated according to the key points of the mouth area, the angle value of the facial area is calculated based on the posture estimation algorithm, and the aspect ratio value, the opening and closing value and the angle value are combined into a feature vector sequence, including:

[0015] Read the coordinate data of the key points of the eye area and the mouth area, calculate the longitudinal Euclidean distance of the upper and lower key points of the eyelids and the transverse Euclidean distance of the inner and outer eye corner key points, divide the longitudinal distance by the transverse distance to obtain the eyelid aspect ratio value, and similarly calculate the distance ratio of the upper and lower key points and the left and right key points of the mouth to obtain the opening and closing value;

[0016] According to the three-dimensional spatial distribution of the key points of the facial area, a perspective projection equation group is constructed, the facial posture parameters are obtained through PnP solution, the pitch angle, yaw angle and roll angle values ​​of the facial area are extracted as the angle value, and the aspect ratio value, opening and closing degree value and angle value are organized into a feature vector sequence in time sequence.

[0017] Furthermore, the multi-layer self-attention encoding network structure is constructed, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, the feature vector sequence is input into the trunk feature extraction branch, the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receives the temporal features as input, and outputs the predicted eyelid opening and closing state, mouth opening and closing state and head posture state, including:

[0018] Constructing a self-attention network including multiple encoder stacks, each encoder consisting of a self-attention layer, a feedforward network layer, and a normalization layer, inputting the feature vector sequence into the self-attention layer of the first encoder, obtaining the attention weight by matrix multiplication of the query matrix, the key matrix, and the value matrix, performing weighted aggregation on the time series features based on the attention weight, and outputting them to the next encoder after being processed by the normalization layer and the feedforward network layer;

[0019] The temporal features output by the last encoder are mapped to the main feature extraction branch and the auxiliary supervision branch through a fully connected layer, where the main feature extraction branch outputs a binary classification prediction result of the fatigue state, and the auxiliary supervision branch outputs the eyelid opening and closing state prediction value, the mouth opening and closing state prediction value, and the head posture state prediction value through the regression layer, and the supervision loss is calculated with the corresponding true label.

[0020] Furthermore, constructing an auxiliary loss function based on the difference between the predicted state and the true label includes:

[0021] Read the true labels of facial states in the training data set, including eyelid opening and closing state labels, mouth opening and closing state labels, and head posture state labels, calculate the mean square error between the predicted value output by the auxiliary supervision branch and the corresponding true label, and construct an auxiliary loss function by weighted summing the mean square errors of the three state predictions;

[0022] Based on the prediction results and true labels of the auxiliary supervision branch, a cross entropy loss function is constructed as the classification loss, and the classification loss and the auxiliary loss function are added through a weight coefficient to obtain a total loss function, and the weight coefficient is used to balance the training contribution of the trunk feature extraction branch and the auxiliary supervision branch.

[0023] Furthermore, the training of the network structure firstly takes the weighted sum of the auxiliary loss function and the main branch classification loss function as the optimization target, optimizes the parameters of the main feature extraction branch and the auxiliary supervision branch, fixes the parameters of the main feature extraction branch after the network converges, removes the auxiliary supervision branch, and uses the main feature extraction branch as a fatigue driving detection model, including:

[0024] Construct a back-propagation optimizer, take the total loss function as the optimization target, use the gradient descent method to synchronously update the network parameters of the main feature extraction branch and the auxiliary supervision branch, iterate the training on the training data set until the loss function converges, and monitor the performance indicators on the validation set to ensure that the model does not overfit;

[0025] The network parameters of the trained trunk feature extraction branch are fixed, the connection with the auxiliary supervision branch is released, the network structure and parameters of the auxiliary supervision branch are removed, the calculation graph of the trunk feature extraction branch is pruned and optimized, and the optimized network structure and its parameters are saved as the fatigue driving detection model.

[0026] Furthermore, the step of deploying the fatigue driving detection model to a vehicle-mounted terminal device and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence includes:

[0027] The fatigue driving detection model is quantized and compressed, network parameters are converted from floating point numbers to fixed point numbers, the compressed model is compiled into a calculation format supported by the vehicle terminal device using a model acceleration framework, and the compiled model file is deployed to a processor of the vehicle terminal device;

[0028] The feature vector sequence collected in real time is divided into sliding windows of fixed length, and normalization preprocessing is performed on the feature vector sequence in each time window, and the feature vector sequence is input into the fatigue driving detection model for forward calculation to obtain the predicted probability value of the fatigue state, and the driver's fatigue state is determined based on the predicted probability value.

[0029] In a second aspect, the present application provides a fatigue detection device based on temporal self-attention and auxiliary supervision, comprising:

[0030] A data acquisition module is used to acquire a facial image of the driver, obtain a facial region through a face detector, extract eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculate an eyelid aspect ratio value based on the eye region key points, calculate an opening and closing degree value based on the mouth region key points, calculate a rotation angle value of the facial region based on a posture estimation algorithm, and form a feature vector sequence with the aspect ratio value, the opening and closing degree value, and the rotation angle value;

[0031] A model building module is used to build a multi-layer self-attention encoding network structure, wherein the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, wherein the feature vector sequence is input into the trunk feature extraction branch, and the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism. The auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state, and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0032] The vehicle-mounted monitoring module is used to train the network structure. First, the weighted sum of the auxiliary loss function and the main branch classification loss function is used as the optimization target to optimize the parameters of the main feature extraction branch and the auxiliary supervision branch. After the network converges, the parameters of the main feature extraction branch are fixed, the auxiliary supervision branch is removed, and the main feature extraction branch is used as a fatigue driving detection model. The fatigue driving detection model is deployed to the vehicle-mounted terminal device, and the driver's fatigue state is judged based on the fatigue driving detection model and the feature vector sequence.

[0033] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision are implemented.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision.

[0036] It can be seen from the above technical solution that the present application provides a fatigue detection method and device based on temporal self-attention and auxiliary supervision, which extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and the temporal self-attention mechanism is used to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is one of the flow charts of the fatigue detection method based on temporal self-attention and auxiliary supervision in the embodiment of the present application;

[0039] Figure 2 This is a second flow chart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0040] Figure 3 This is a flowchart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0041] Figure 4 This is a fourth flow chart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0042] Figure 5 This is a fifth flow chart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0043] Figure 6 This is a sixth flow chart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0044] Figure 7 FIG7 is a flowchart of a fatigue detection method based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0045] Figure 8It is a structural diagram of a fatigue detection device based on temporal self-attention and auxiliary supervision in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0047] Reference numerals:

[0048] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0051] Taking into account the problems existing in the prior art, the present application provides a fatigue detection method and device based on temporal self-attention and auxiliary supervision, which extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision. The temporal self-attention mechanism is used to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0052] In order to accurately identify the driver's fatigue state and provide reliable technical support for improving road traffic safety, this application provides an embodiment of a fatigue detection method based on temporal self-attention and auxiliary supervision, see Figure 1 The fatigue detection method based on temporal self-attention and auxiliary supervision specifically includes the following contents:

[0053] Step S101: collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating eyelid aspect ratio values ​​according to the eye region key points, calculating opening and closing values ​​according to the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and combining the aspect ratio value, the opening and closing value, and the rotation angle value into a feature vector sequence;

[0054] Optionally, the embodiment collects the driver's facial image through an on-board camera, and the camera used has 1920x1080 resolution, 60fps sampling rate and 120dB wide dynamic range characteristics. The camera is installed above the steering column, at an angle of 30-45 degrees to the driver's face, to ensure the best shooting angle without affecting the driving line of sight. To cope with different lighting environments, the camera is equipped with an adaptive exposure control module, which automatically reduces the exposure parameters when the sun is strong during the day, increases the sensitivity in low light conditions at night, and cooperates with the infrared fill light system to ensure image quality.

[0055] After acquiring the original image, this embodiment uses an improved YOLOv5 face detector to locate the facial area. The detector constructs a 3-layer image pyramid, scales the input image by 1.0, 0.75, and 0.5 times the original size, and performs face detection independently at each scale. The detection network outputs the coordinates of the rectangular bounding box of the facial area, including the upper left corner point (x1, y1) and the lower right corner point (x2, y2). The detection results are subjected to non-maximum suppression processing, the IoU threshold is set to 0.5, and overlapping detection boxes are removed. The retained bounding box is expanded by 15% in all directions to ensure that the facial area is completely contained.

[0056] This embodiment uses an improved HRNet key point detection network to extract facial feature points. The network adopts a four-stage cascade structure, and each stage contains two branches, high resolution and low resolution. In the first stage, 68 facial key points are roughly located, and the key point positions are gradually refined through multi-scale feature fusion in subsequent stages. Among them, the focus is on 6 pairs of key points in the eye area (including the inner corner of the eye, the outer corner of the eye, the midpoint of the upper eyelid, the midpoint of the lower eyelid, etc.) and 8 key points in the mouth area (including the corner of the mouth, the lip peak, the lip valley, etc.). In order to improve the detection stability, Kalman filtering is applied to the key point coordinate sequence, and the predicted state vector contains position and velocity information.

[0057] When calculating the characteristic parameters, this embodiment processes the left and right eye areas separately. For each eye, the longitudinal Euclidean distance d_v between the midpoints of the upper and lower eyelids and the lateral distance d_h between the inner and outer corners of the eye are calculated, and the aspect ratio EAR = d_v / d_h. Considering the symmetry of the left and right eyes, the average of the aspect ratios of the two eyes is taken as the final eyelid opening index. Similarly, when calculating the mouth opening, the ratio MAR of the longitudinal distance between the midpoints of the upper and lower lips and the lateral distance between the corners of the mouth is measured, which reflects the degree of mouth opening.

[0058] This embodiment innovatively adopts an improved PnP algorithm in head posture estimation. First, a standard three-dimensional face model is established, which contains 3D coordinates corresponding to the detected key points. Based on the perspective projection principle, a mapping relationship between 2D image coordinates and 3D space coordinates is established. The RANSAC method is used to iteratively solve the optimal rotation matrix R and translation vector t, from which the head posture parameters represented by Euler angles are extracted: pitch angle pitch, yaw angle yaw and roll angle roll.

[0059] In order to enhance the temporal correlation of the features, this embodiment performs temporal processing on the extracted parameters. A sliding window with a length of 30 frames (0.5 seconds) is set, and the feature parameters in the window are subjected to exponential weighted averaging, giving higher weights to the data at the most recent moment. Finally, the processed eyelid opening and closing degree, mouth opening and closing degree, and three corner parameters are combined into a 5-dimensional feature vector, which is organized into a feature sequence according to a fixed time window.

[0060] Through the above-mentioned detailed feature extraction process, this embodiment achieves stable acquisition of key indicators of driver fatigue status. The fusion of multimodal features provides a more comprehensive representation of fatigue status, and the timing processing mechanism ensures the continuity and reliability of features. This solution has shown good environmental adaptability and operational stability in actual road tests, laying a solid data foundation for subsequent fatigue status analysis.

[0061] The technical solution of this embodiment effectively solves the influence of interference factors such as lighting changes and head movement in the driving environment through a carefully designed feature extraction process, ensures the accuracy and timeliness of feature data, and has important practical application value for improving driving safety. The innovative head posture estimation method and multi-feature fusion strategy further enhance the system's ability to identify different forms of fatigue behavior.

[0062] Step S102: construct a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, the feature vector sequence is input into the trunk feature extraction branch, the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0063] Optionally, this embodiment constructs a multi-layer self-attention encoding network structure specifically for driver fatigue state analysis. The network adopts a dual-branch architecture, including a trunk feature extraction branch and an auxiliary supervision branch, and achieves accurate modeling of the driver's fatigue state through multi-level feature extraction and multi-task learning strategies.

[0064] In the trunk feature extraction branch, this embodiment designs a 4-layer temporal self-attention encoder. Each layer of the encoder contains a multi-head self-attention module and a feedforward neural network, and residual connections and layer normalization operations are used between layers. The input feature vector sequence is first positionally encoded to embed the temporal position information into the feature representation. In the self-attention calculation process, 8 attention heads are used, and each attention head independently learns feature associations from different perspectives. This multi-head mechanism enables the network to simultaneously focus on state change patterns on multiple time scales.

[0065] In order to capture fatigue characteristics of different time spans, this embodiment innovatively introduces a multi-scale receptive field design in the self-attention layer. The first layer uses a local attention mechanism, and the receptive field is limited to 5 adjacent time steps, focusing mainly on short-term state changes. As the number of layers increases, the calculation scope of attention is gradually expanded, and the highest layer can model long-term dependencies within the entire sequence. This hierarchical attention structure not only ensures sensitivity to instantaneous fatigue manifestations, but also effectively captures the gradual evolution of fatigue states.

[0066] The auxiliary supervision branch receives the time series features of each layer of the main branch as input. This embodiment designs three parallel prediction heads, which are responsible for predicting the eyelid opening and closing state, mouth opening and closing state, and head posture state. Each prediction head contains two fully connected layers, and a dropout layer is added in the middle to prevent overfitting. Eyelid and mouth state prediction uses binary classification output to indicate open or closed state; head posture state prediction uses regression output to predict angle values ​​in three directions.

[0067] This embodiment adopts a multi-task joint optimization strategy during the training process. The loss function of the auxiliary supervision branch consists of three parts: the cross entropy loss of the eyelid state, the cross entropy loss of the mouth state, and the mean square error loss of the head posture. These intermediate supervision signals not only provide additional gradient sources to promote the network to learn more discriminative feature representations, but also play a role in regularization and improve the generalization ability of the model.

[0068] In order to adapt to various interference factors in driving scenarios, this embodiment introduces a variety of transformation strategies when enhancing training data, including randomly blocking some time series features, adding Gaussian noise, and randomly replacing short sequence fragments. These data enhancement methods improve the robustness of the model to feature loss and noise, enabling it to maintain stable performance in actual driving environments.

[0069] Through the above network design, this embodiment successfully solves the shortcomings of traditional methods in time series modeling capabilities. The multi-layer self-attention mechanism can adaptively capture fatigue features at different time scales, and the auxiliary supervision mechanism ensures the semantic clarity of intermediate features. This solution demonstrates excellent time series modeling capabilities in actual driving scenarios and can accurately capture the gradual evolution of the driver's fatigue state.

[0070] The technical solution of this embodiment significantly improves the accuracy and stability of fatigue state modeling through a carefully designed network structure and training strategy. The multi-task learning framework not only provides rich supervision signals, but also realizes the effective fusion of fatigue features of different modalities, providing reliable feature support for subsequent fatigue level assessment.

[0071] Step S103: training the network structure, first taking the weighted sum of the auxiliary loss function and the trunk branch classification loss function as the optimization target, optimizing the parameters of the trunk feature extraction branch and the auxiliary supervision branch, fixing the trunk feature extraction branch parameters after the network converges, removing the auxiliary supervision branch, using the trunk feature extraction branch as a fatigue driving detection model, deploying the fatigue driving detection model to the vehicle-mounted terminal device, and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence.

[0072] Optionally, this embodiment uses a two-stage training strategy to optimize the network. In the first stage, joint training is performed to optimize the main feature extraction branch and the auxiliary supervision branch at the same time. In the second stage, the main branch parameters are fixed to obtain the final fatigue driving detection model. This progressive training scheme fully utilizes the advantages of multi-task learning and improves the feature extraction ability of the model.

[0073] In the first stage of training, this embodiment constructs a composite loss function, which includes the fatigue state classification loss of the main branch and the three subtask losses of the auxiliary supervision branch. The main branch adopts the cross entropy loss function to optimize the final fatigue state classification performance. The auxiliary losses include the cross entropy loss of the eyelid state and mouth state and the mean square error loss of the head posture. The various loss items are balanced by adaptive weights, and the weight coefficients are dynamically adjusted during the training process to ensure that different tasks can be optimized collaboratively.

[0074] This embodiment pays special attention to the coverage of real driving scenarios when constructing training data. Driving data in different time periods and weather conditions are collected, including daytime, nighttime, cloudy, rainy and other environments. At the same time, considering the individual differences of drivers, the training data contains variables such as different ages, genders, and whether or not they wear glasses. This diversified data collection strategy significantly improves the generalization ability of the model.

[0075] The training process uses a small batch stochastic gradient descent optimizer with a batch size of 32. The learning rate is initially set to 0.001 and dynamically adjusted using a cosine annealing strategy. To prevent overfitting, this embodiment introduces a variety of regularization techniques in training, including weight decay, gradient clipping, and early stopping strategies. At the same time, the performance on the validation set is used as the basis for model selection. When the validation set performance does not improve for 5 consecutive cycles, the learning rate is reduced to continue training.

[0076] After the first phase of training is completed, this embodiment fixes the parameters of the main feature extraction branch and removes the auxiliary supervision branch. The purpose of this is to reduce the computational complexity of the model and make it more suitable for deployment in a vehicle environment with limited computing resources. The main branch trained with the auxiliary task has learned a rich feature representation and can accurately capture various manifestations of fatigue driving.

[0077] In the model deployment phase, this embodiment is optimized for the hardware characteristics of the vehicle terminal. The model quantization technology is used to convert the network parameters from 32-bit floating point numbers to 8-bit integers, which significantly reduces the model size and inference time. At the same time, the hardware accelerator is used to accelerate the calculation to ensure that the model can process the input feature sequence in real time.

[0078] In practical applications, the fatigue driving detection model of this embodiment performs a state assessment every 100 milliseconds. The model receives a feature vector sequence within the last 30 seconds as input and outputs the driver's current fatigue level. When fatigue driving behavior is detected, the vehicle system will issue an audible and visual warning to remind the driver to take a rest in time.

[0079] Through this carefully designed training and deployment scheme, this embodiment successfully solves the problems of insufficient accuracy and poor real-time performance of traditional fatigue detection methods. Multi-task assisted training improves the model's feature extraction capabilities, while model compression and hardware acceleration ensure the stable operation of the system in a vehicle environment.

[0080] The technical solution of this embodiment has shown excellent performance in actual road tests, and can accurately identify various forms of fatigue driving behaviors, including drowsiness, yawning, distraction, etc. The timely warning function of the system effectively reduces the safety risks caused by fatigue driving, making an important contribution to improving road traffic safety.

[0081] From the above description, it can be seen that the fatigue detection method based on temporal self-attention and auxiliary supervision provided in the embodiment of the present application can extract multi-dimensional features such as eyelid aspect ratio, mouth opening and closing degree, and head angle of the driver's face through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and the temporal self-attention mechanism is used to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0082] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 2 , and can also include the following:

[0083] Step S201: using a vehicle-mounted camera to collect a driver's facial image sequence, inputting the collected image sequence into a multi-scale face detector, executing a target detection algorithm on each frame of the image, locating a rectangular bounding box of the facial area, extracting facial image data within the rectangular bounding box, and performing illumination equalization and size standardization processing on the facial image data;

[0084] Step S202: construct a key point feature map pyramid for the face image data, extract facial key point coordinates through a cascaded deformation network, divide the extracted key points into eye area key points and mouth area key points according to facial areas, establish a key point index table, and perform smoothing filtering on the key point coordinate data to eliminate jitter.

[0085] Optionally, this embodiment first uses a vehicle-mounted camera to monitor the driver in real time. The camera is installed above the steering column, and a wide-angle lens is used to ensure that the driver's upper body area is fully captured. To cope with complex driving environments, the camera is equipped with adaptive exposure control and infrared fill light system to obtain clear facial images under different lighting conditions. The image acquisition frequency is set to 30 frames per second and the resolution is 1920×1080 to ensure that subtle changes in the driver's facial state are captured.

[0086] In the face detection stage, this embodiment adopts an improved multi-scale detection strategy. First, a three-layer image pyramid is constructed with scale factors of 1.0, 0.75, and 0.5, and a multi-scale transformation is performed on the input image. The face detector is run independently at each scale. The detector adopts an improved SSD architecture and the backbone network uses MobileNetV3 to reduce the amount of calculation while ensuring detection accuracy. The detection result includes the rectangular bounding box coordinates and confidence score of the facial area.

[0087] This embodiment innovatively designs an adaptive non-maximum suppression algorithm, which dynamically adjusts the IoU threshold according to the size of the bounding box. When the detected face is large, a higher IoU threshold (0.6) is used to retain more candidate boxes; when the face is small, the IoU threshold (0.4) is lowered to reduce repeated detection. Finally, the bounding box with the highest confidence is selected as the face area. To ensure the stability of subsequent processing, the bounding box is expanded by 15% in all directions to completely include the facial contour.

[0088] For the extracted face image, this embodiment first performs illumination equalization. Using an improved adaptive histogram equalization algorithm, the image is divided into a 4×4 grid, histogram equalization is performed independently in each sub-region, and then the results of adjacent regions are fused by bilinear interpolation to avoid excessive enhancement of local contrast. For backlit or shadowed areas, gamma correction is additionally applied to enhance details. Finally, the processed image is uniformly scaled to 224×224 size.

[0089] In the key point detection link, this embodiment first constructs a feature map pyramid network. Taking the processed face image as input, it downsamples layer by layer through a 5-layer convolutional network to generate feature maps of different resolutions. The high-level feature map encodes global structural information, and the low-level feature map retains local detail features. Through the feature pyramid network, features are fused layer by layer from top to bottom to achieve effective integration of multi-scale features.

[0090] This embodiment adopts a two-stage cascade strategy to locate key points. In the first stage, 68 facial key points are roughly located, including areas such as eyes, eyebrows, nose, mouth and facial contour. In the second stage, the key point positions of the eye and mouth areas are specifically refined. Among them, the eye area focuses on 6 pairs of key points, and the mouth area focuses on 8 key points. A key point index table is constructed to store the correspondence between these key areas to facilitate subsequent feature calculations.

[0091] In order to improve the stability of key point detection, this embodiment applies Kalman filtering to smooth the detection results. The state vector contains the two-dimensional coordinates and velocity information of the key points, and the detection noise and jitter are suppressed through the prediction-update cycle. At the same time, considering the spatial constraint relationship between facial key points, the deformation consistency constraint is introduced to ensure that the relative position of the key points remains reasonable.

[0092] Through the above carefully designed image processing process, this embodiment successfully solves the challenges brought by lighting changes and posture changes in the driving environment. Multi-scale detection and feature fusion strategies improve the robustness of face detection, and the cascaded deformation network ensures the accuracy of key point positioning. This solution lays a solid foundation for subsequent fatigue feature extraction.

[0093] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 3 , and can also include the following:

[0094] Step S301: Read the coordinate data of the key points of the eye region and the mouth region, calculate the longitudinal Euclidean distance of the upper and lower eyelid key points and the transverse Euclidean distance of the inner and outer eye corner key points, divide the longitudinal distance by the transverse distance to obtain the eyelid aspect ratio value, and similarly calculate the distance ratio of the upper and lower mouth key points and the left and right mouth key points to obtain the opening and closing value;

[0095] Step S302: construct a perspective projection equation group according to the three-dimensional spatial distribution of the key points of the facial area, obtain facial posture parameters through PnP solution, extract the pitch angle, yaw angle and roll angle values ​​of the facial area as the angle value, and organize the aspect ratio value, opening degree value and angle value into a feature vector sequence in time sequence.

[0096] Optionally, this embodiment constructs a complete feature extraction process based on the acquired facial key point coordinates. First, the feature calculation of the eye area is processed, and 6 key point coordinates are extracted for each eye, including the midpoint of the upper eyelid, the midpoint of the lower eyelid, the inner corner of the eye, and the outer corner of the eye. By calculating the longitudinal Euclidean distance of the upper and lower eyelid key points, the degree of eye opening can be effectively characterized; and the lateral distance of the inner and outer corners of the eye is used as a normalized reference to eliminate the scale effect caused by the change in camera distance.

[0097] In actual driving scenarios, this embodiment optimizes the particularity of eyelid detection under different lighting conditions. For example, when driving at night, the eyelid edge is often blurred due to the dim ambient light. At this time, infrared fill light and local contrast enhancement technology are used to improve the clarity of the eyelid contour and ensure the accuracy of key point positioning. At the same time, considering that the driver may wear glasses, this embodiment designs a lens reflection compensation algorithm to reduce reflection interference through local brightness balancing.

[0098] A similar calculation method is used for mouth feature extraction. The midpoints of the upper lip, lower lip, and mouth corners are selected as feature points, and the ratio of the longitudinal opening and closing distance to the horizontal width is calculated. This ratio can effectively reflect fatigue performance such as yawning. In order to cope with the interference of normal behaviors such as speaking, this embodiment introduces a time series sliding window analysis to distinguish yawning from normal speaking behaviors by observing the duration and change pattern of the opening and closing degree.

[0099] In the posture estimation link, this embodiment innovatively adopts the improved PnP algorithm. First, a standard 3D face model is established, including the spatial coordinates of key structural points such as eyebrows, nose and mouth. The detected 2D key points are corresponding to the 3D model points to construct the perspective projection equation group. Considering the imaging characteristics of the vehicle-mounted camera, this embodiment adopts the equivalent focal length model to simplify the calculation process and improve the solution efficiency.

[0100] This embodiment designs a robust RANSAC optimization strategy to find the optimal posture parameters through iterative sampling. In each iteration, some key points are randomly selected to calculate the initial solution, and then the reprojection errors of other points are verified. This method can effectively deal with the influence of key point positioning errors and outliers. The final output posture parameters include the pitch angle, yaw angle and roll angle of the head, which fully describe the spatial information of the driver's head posture.

[0101] In order to capture the temporal characteristics of the fatigue state, this embodiment organizes the extracted features into a sequence form. A sliding window with a length of 30 seconds is used, which is updated every 100 milliseconds. The feature vector contains the aspect ratio of the left and right eyes, the degree of mouth opening, and three angle parameters. By maintaining this dynamically updated feature sequence, the progressive process of fatigue driving can be accurately captured.

[0102] The feature extraction scheme of this embodiment overcomes many difficulties of traditional methods in actual driving environments. By combining geometric features and time series analysis, it not only ensures the real-time nature of feature extraction, but also improves the reliability of fatigue state recognition. Especially in complex scenarios such as night driving and drivers wearing glasses, this scheme can still maintain a stable feature extraction effect, providing a reliable data basis for subsequent fatigue state judgment.

[0103] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 4 , and can also include the following:

[0104] Step S401: construct a self-attention network including multiple encoder stacks, each encoder consisting of a self-attention layer, a feedforward network layer and a normalization layer, input the feature vector sequence into the self-attention layer of the first encoder, obtain the attention weight by matrix multiplication operation of the query matrix, the key matrix and the value matrix, perform weighted aggregation on the time series features based on the attention weight, and output to the next encoder after being processed by the normalization layer and the feedforward network layer;

[0105] Step S402: Map the time series features output by the last encoder to the main feature extraction branch and the auxiliary supervision branch through a fully connected layer, wherein the main feature extraction branch outputs a binary classification prediction result of the fatigue state, and the auxiliary supervision branch outputs the eyelid opening and closing state prediction value, the mouth opening and closing state prediction value, and the head posture state prediction value through the regression layer, and the supervision loss is calculated with the predicted value and the corresponding true label.

[0106] Optionally, this embodiment constructs a deep neural network based on the self-attention mechanism to analyze the temporal characteristics of the driver's fatigue state. The network structure adopts the form of a multi-layer encoder stack, and each layer of the encoder contains three key components: a self-attention layer, a feedforward network layer, and a layer normalization module. This design can effectively capture the long-term dependency of fatigue characteristics in the time dimension.

[0107] In the processing of the self-attention layer, this embodiment first obtains the query matrix Q, key matrix K and value matrix V by three independent linear transformations of the input feature vector sequence. Considering the continuity of fatigue features in driving scenarios, the relative position encoding method is adopted to inject the temporal position information into the attention calculation. The attention score is calculated by matrix multiplication QK^T, and the scaling factor is used to prevent the gradient from disappearing. The obtained attention weight reflects the importance of features at different times.

[0108] This embodiment innovatively designs a multi-head attention mechanism to project the input features into 8 different subspaces in parallel. Each attention head independently calculates weights and performs feature aggregation, which enables the network to focus on feature patterns on multiple time scales at the same time. For example, some attention heads may focus on detecting short-term instantaneous changes (such as blinking), while other heads focus on behavioral patterns with longer time spans (such as the complete process of yawning).

[0109] In each encoder, the output of the self-attention layer is processed by residual connection and layer normalization to ensure the training stability of the deep network. The subsequent feedforward network layer adopts a two-layer structure with a GELU activation function in the middle. This design enhances the nonlinear expression ability of the model. The series connection of multiple encoders enables the network to extract more abstract fatigue feature representations layer by layer.

[0110] This embodiment adopts a multi-task learning framework, and designs a trunk feature extraction branch and an auxiliary supervision branch at the end of the network. The trunk branch outputs the binary classification prediction results of the fatigue state through the fully connected layer and the Sigmoid function. The auxiliary branch contains three independent regressors, which predict the eyelid opening, mouth opening, and head posture angle respectively. This multi-task design can introduce additional supervision signals and provide richer gradient information to guide network training.

[0111] In the loss function design, this embodiment adopts a weighted combination method for different tasks. The main branch uses cross entropy loss, while the auxiliary branch uses smooth L1 loss. By adjusting the weights of each loss, the learning difficulty of different tasks is balanced. At the same time, considering the data imbalance of fatigue driving samples, the focus loss mechanism is introduced to increase the optimization of difficult samples.

[0112] The network structure of this embodiment shows excellent characteristics in practical applications. The self-attention mechanism enables the model to adaptively capture fatigue characteristics on different time scales; the multi-task learning framework provides a more comprehensive fatigue status assessment. Especially in the process of continuous driving, this scheme can timely detect the early signs of fatigue driving and provide a reliable basis for judgment for the early warning system. The prediction results of the auxiliary tasks can also be used to explain the decision-making process of the model, enhancing the interpretability of the system.

[0113] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 5 , and can also include the following:

[0114] Step S501: read the real labels of facial states in the training data set, including eyelid opening and closing state labels, mouth opening and closing state labels, and head posture state labels, calculate the mean square error between the predicted value output by the auxiliary supervision branch and the corresponding real label, and construct an auxiliary loss function by weighted summing the mean square errors of the three state predictions;

[0115] Step S502: Based on the prediction result of the auxiliary supervision branch and the true label, a cross entropy loss function is constructed as the classification loss, and the classification loss and the auxiliary loss function are added through a weight coefficient to obtain a total loss function, and the weight coefficient is used to balance the training contribution of the trunk feature extraction branch and the auxiliary supervision branch.

[0116] Optionally, this embodiment designs a complete multi-task loss function calculation scheme to guide the training process of the fatigue driving detection model. First, for the auxiliary supervision branch, three types of continuous value prediction tasks, namely eyelid opening and closing state, mouth opening and closing state, and head posture state, are processed separately. The true label of each type of state is standardized to eliminate the influence of different feature dimensions.

[0117] When processing the eyelid opening and closing state, this embodiment takes into account the rapid characteristics of the driver's blinking action and adopts an improved mean square error calculation method. By setting dynamic weights, a higher penalty is given to the prediction error in the fast-changing interval, thereby improving the model's sensitivity to blinking actions. For example, when the eyelid closes quickly from an open state, the prediction error in this time period will receive a larger weight.

[0118] For the prediction of the mouth opening and closing state, this embodiment introduces a smooth L1 loss function. This design is more stable when dealing with large changes such as yawning, and can reduce the interference of outliers on model training. At the same time, considering the interference that may be caused by normal behaviors such as speaking, a threshold mechanism is set when calculating the loss to reduce the loss contribution within the normal opening and closing range.

[0119] The loss calculation of the head posture state adopts an improved angle error measurement method. Considering the periodic characteristics of the angle, this embodiment uses a cycle consistency loss to ensure that the loss calculation is still reasonable when the angle is close to the junction of 360 degrees and 0 degrees. At the same time, different weights are set for the prediction errors of the three rotation angles, among which the weights of the yaw angle and the pitch angle are larger because these two angles are more closely related to fatigue driving.

[0120] In the process of constructing the auxiliary loss function, this embodiment adopts an adaptive weight mechanism. By analyzing the changing trends of various losses during the training process, the weight coefficients of the three types of state predictions are dynamically adjusted. This design can focus on easy-to-learn features in the early stage of training, and then gradually increase the weights of difficult-to-learn features to improve the overall performance of the model.

[0121] The classification loss of the trunk branch adopts an improved cross entropy loss function. Considering the uneven distribution of fatigue state samples in the data set, this embodiment introduces a sample weight balancing mechanism. Positive samples of fatigue state are given a higher weight to increase their importance in the training process. At the same time, through the difficult sample mining strategy, the samples with low prediction confidence are optimized.

[0122] In the construction of the total loss function, this embodiment designs a dynamic weight adjustment mechanism. In the early stage of training, the weight of the auxiliary loss is relatively large to help the model learn the basic feature representation; as the training progresses, the weight of the classification loss is gradually increased to make the model more focused on the judgment of the fatigue state. This progressive training strategy significantly improves the convergence speed and generalization performance of the model.

[0123] The loss function design of this embodiment shows good results in practical applications. Through multi-task collaborative optimization, the model can accurately predict facial micro-expression changes and overall fatigue status at the same time. Especially in complex scenarios such as night driving, the supervisory signal of the auxiliary task helps the model establish a more stable feature representation and improves the reliability of fatigue detection. This design also has good scalability and can easily integrate new supervisory signals to further improve model performance.

[0124] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 6 , and can also include the following:

[0125] Step S601: construct a back propagation optimizer, take the total loss function as the optimization target, use the gradient descent method to synchronously update the network parameters of the main feature extraction branch and the auxiliary supervision branch, iterate the training on the training data set until the loss function converges, and monitor the performance indicators on the validation set to ensure that the model does not overfit;

[0126] Step S602: fix the network parameters of the trained trunk feature extraction branch, disconnect it from the auxiliary supervision branch, remove the network structure and parameters of the auxiliary supervision branch, perform pruning optimization on the calculation graph of the trunk feature extraction branch, and save the optimized network structure and parameters as a fatigue driving detection model.

[0127] Optionally, this embodiment describes in detail the complete process of model training and optimization. When constructing the back-propagation optimizer, the Adam optimizer with momentum term is used to accelerate the model convergence through the adaptive learning rate mechanism. The selection of momentum coefficient fully considers the temporal continuity of fatigue driving characteristics, which not only ensures the stability of training, but also can quickly respond to feature changes.

[0128] In order to handle various complex situations in actual driving scenarios, this embodiment adopts a phased training strategy. In the first phase, a larger initial learning rate is used to focus on optimizing feature extraction capabilities; in the second phase, the learning rate is reduced and model parameters are finely adjusted. This strategy is particularly suitable for extracting driver fatigue features at different times and under different lighting conditions.

[0129] During the training process, this embodiment innovatively introduces a dynamic batch balancing mechanism. Considering the relatively rare characteristics of fatigue state samples in real scenarios, each training batch is ensured to contain a sufficient proportion of fatigue samples. At the same time, through the online difficult example mining technology, samples that are difficult to predict are preferentially selected for training, thereby improving the model's ability to recognize boundary conditions.

[0130] To prevent overfitting, this embodiment designs multiple regularization strategies. First, the dropout technology is applied at the network level, especially at the key layer nodes of the self-attention mechanism. Secondly, L2 regularization is introduced to constrain the network weights, and sparse constraints are applied in the attention calculation process to encourage the model to focus on the most relevant time series features. In addition, an early stopping strategy is adopted to terminate the training in time by monitoring the performance indicators on the validation set.

[0131] In the actual driving environment, this embodiment ensures the generalization ability of the model through the performance evaluation of the validation set. The validation set specifically includes various complex scenarios, such as night driving, wearing glasses, lighting changes, etc. By observing the performance indicators in these scenarios, the training strategy is adjusted in time to ensure the stability of the model in actual applications.

[0132] After the model training is completed, this embodiment performs careful model optimization and pruning. First, the importance of each component in the network is analyzed, and the statistical characteristics of attention weights and activation values ​​are calculated to identify redundant structures that contribute less to the final prediction. When removing auxiliary supervision branches, a progressive pruning strategy is adopted to ensure that the model performance does not drop significantly.

[0133] This embodiment innovatively designs a knowledge distillation mechanism to transfer the knowledge of the complete model to the streamlined backbone network. Through temperature parameter adjustment, the form of soft labels guides the streamlined model to learn more detailed feature representations. This approach not only retains the advantages brought by multi-task learning, but also significantly reduces the computational complexity of the model.

[0134] In the final model saving stage, this embodiment performs quantization and compression processing on the network parameters. Through the combination of careful pruning and quantization strategies, the model storage space requirements are greatly reduced while maintaining prediction accuracy. This enables the model to better adapt to the hardware limitations of the vehicle environment and achieve efficient real-time fatigue detection.

[0135] The optimized fatigue driving detection model shows excellent practicality. The model can respond quickly with limited computing resources and accurately identify the driver's fatigue state. Especially in the process of continuous driving, the model has a good tracking ability for the gradual change of fatigue level, which provides a reliable guarantee for timely warning. The optimized model structure is simple and efficient, which is easy to deploy and use in the actual vehicle environment.

[0136] In one embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision of the present application, see Figure 7 , and can also include the following:

[0137] Step S701: quantizing and compressing the fatigue driving detection model, converting network parameters from floating point numbers to fixed point numbers, using a model acceleration framework to compile the compressed model into a calculation format supported by the vehicle terminal device, and deploying the compiled model file to a processor of the vehicle terminal device;

[0138] Step S702: Segment the feature vector sequence collected in real time according to sliding windows of fixed length, perform normalization preprocessing on the feature vector sequence in each time window, input the fatigue driving detection model for forward calculation, obtain the predicted probability value of the fatigue state, and determine the driver's fatigue state based on the predicted probability value.

[0139] Optionally, this embodiment elaborates on the deployment and application process of the fatigue driving detection model in the actual vehicle environment. First, the model is optimized using a mixed precision quantization strategy based on the hardware characteristics of the vehicle terminal device. For key layers such as attention calculation, a high numerical accuracy is maintained, while for the intermediate feature transfer layer, a fixed-point representation with a lower bit width is used, which significantly reduces the computational overhead while ensuring the detection accuracy.

[0140] In the quantization process, this embodiment innovatively designs a hierarchical quantization scheme. For the weight calculation in the self-attention mechanism, 8-bit fixed-point numbers are used; for the activation value, a suitable quantization range is selected according to its distribution characteristics. Especially when processing sensitive information such as facial features, the quantization parameters are optimized through the calibration data set to ensure that subtle fatigue performance can be accurately captured under low precision.

[0141] During the model compilation phase, this embodiment makes full use of the hardware acceleration features of the onboard processor. Through operator fusion technology, continuous convolution operations, normalization, and activation functions are combined into a single computing unit to reduce intermediate data transmission overhead. At the same time, a special memory access mode is designed for dense matrix operations such as self-attention calculations to improve cache utilization efficiency.

[0142] In actual deployment, this embodiment adopts a pipeline parallel processing mechanism. The feature extraction and reasoning process is divided into multiple parallel execution stages, making full use of the parallel computing capabilities of the processor. This design enables the model to achieve near real-time processing speed while maintaining high detection accuracy.

[0143] This embodiment adopts an innovative sliding window strategy in feature sequence processing. The window length is determined according to the time characteristics of fatigue behavior, usually covering an observation period of 3-5 seconds. A certain overlap is maintained between adjacent windows to ensure that continuous fatigue state changes can be captured. For each time window, an adaptive normalization method is used to process the feature sequence to effectively eliminate the influence of external factors such as illumination changes and posture changes.

[0144] In the real-time prediction stage, this embodiment designs a time series smoothing mechanism. By weighted averaging the prediction results of multiple continuous time windows, misjudgments caused by short-term fluctuations are suppressed. The weight coefficient decays exponentially over time, so that the model pays more attention to the fatigue state at the most recent moment. At the same time, state migration constraints are introduced to prevent unreasonable and drastic jumps in the prediction results.

[0145] In order to adapt to the individual differences of different drivers, this embodiment adopts an adaptive mechanism in the prediction threshold setting. The system will collect the driver's normal state data in the initial use stage and establish a personalized baseline model. The prediction threshold will be dynamically adjusted according to factors such as driving time and time period to improve the accuracy of detection.

[0146] The deployment scheme of this embodiment shows excellent performance in the actual driving environment. The quantized and compressed model can run smoothly on the vehicle terminal and continuously monitor the driver's fatigue state. Especially in high-risk scenarios such as long-distance driving at night, the system can promptly detect early signs of fatigue driving, remind the driver through the early warning mechanism, and effectively prevent the occurrence of fatigue driving accidents. At the same time, the optimized model has low power consumption, suitable for long-term continuous work, and provides reliable protection for driving safety.

[0147] In order to accurately identify the driver's fatigue state and provide reliable technical support for improving road traffic safety, the present application provides an embodiment of a fatigue detection device based on temporal self-attention and auxiliary supervision for realizing all or part of the fatigue detection method based on temporal self-attention and auxiliary supervision, see Figure 8 The fatigue detection device based on temporal self-attention and auxiliary supervision specifically includes the following contents:

[0148] The data acquisition module 10 is used to acquire a facial image of the driver, obtain a facial region through a face detector, extract eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculate an eyelid aspect ratio value based on the eye region key points, calculate an opening and closing degree value based on the mouth region key points, calculate a rotation angle value of the facial region based on a posture estimation algorithm, and form a feature vector sequence with the aspect ratio value, the opening and closing degree value, and the rotation angle value;

[0149] A model construction module 20 is used to construct a multi-layer self-attention encoding network structure, wherein the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, wherein the feature vector sequence is input into the trunk feature extraction branch, and the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism. The auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state, and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0150] The vehicle-mounted monitoring module 30 is used to train the network structure. First, the weighted sum of the auxiliary loss function and the main branch classification loss function is used as the optimization target to optimize the parameters of the main feature extraction branch and the auxiliary supervision branch. After the network converges, the parameters of the main feature extraction branch are fixed, the auxiliary supervision branch is removed, and the main feature extraction branch is used as a fatigue driving detection model. The fatigue driving detection model is deployed to the vehicle-mounted terminal device, and the driver's fatigue state is judged based on the fatigue driving detection model and the feature vector sequence.

[0151] From the above description, it can be seen that the fatigue detection device based on temporal self-attention and auxiliary supervision provided in the embodiment of the present application can extract multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and the temporal self-attention mechanism is used to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0152] From the hardware level, in order to accurately identify the driver's fatigue state and provide reliable technical support for improving road traffic safety, the present application provides an embodiment of an electronic device for implementing all or part of the fatigue detection method based on temporal self-attention and auxiliary supervision, and the electronic device specifically includes the following contents:

[0153] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the fatigue detection device based on temporal self-attention and auxiliary supervision and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the fatigue detection method based on temporal self-attention and auxiliary supervision, and the embodiment of the fatigue detection device based on temporal self-attention and auxiliary supervision in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0154] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0155] In practical applications, part of the fatigue detection method based on temporal self-attention and auxiliary supervision can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0156] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0157] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0158] In one embodiment, the fatigue detection method based on temporal self-attention and auxiliary supervision can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0159] Step S101: collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating eyelid aspect ratio values ​​according to the eye region key points, calculating opening and closing values ​​according to the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and combining the aspect ratio value, the opening and closing value, and the rotation angle value into a feature vector sequence;

[0160] Step S102: construct a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, the feature vector sequence is input into the trunk feature extraction branch, the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0161] Step S103: training the network structure, first taking the weighted sum of the auxiliary loss function and the trunk branch classification loss function as the optimization target, optimizing the parameters of the trunk feature extraction branch and the auxiliary supervision branch, fixing the trunk feature extraction branch parameters after the network converges, removing the auxiliary supervision branch, using the trunk feature extraction branch as a fatigue driving detection model, deploying the fatigue driving detection model to the vehicle-mounted terminal device, and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence.

[0162] From the above description, it can be seen that the electronic device provided in the embodiment of the present application extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and uses the temporal self-attention mechanism to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0163] In another embodiment, the fatigue detection device based on temporal self-attention and auxiliary supervision can be configured separately from the central processing unit 9100. For example, the fatigue detection device based on temporal self-attention and auxiliary supervision can be configured as a chip connected to the central processing unit 9100, and the function of the fatigue detection method based on temporal self-attention and auxiliary supervision can be realized through the control of the central processing unit.

[0164] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0165] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0166] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0167] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0168] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0169] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0170] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0171] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0172] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision in the above-mentioned embodiment, in which the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0173] Step S101: collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating eyelid aspect ratio values ​​according to the eye region key points, calculating opening and closing values ​​according to the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and combining the aspect ratio value, the opening and closing value, and the rotation angle value into a feature vector sequence;

[0174] Step S102: construct a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, the feature vector sequence is input into the trunk feature extraction branch, the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0175] Step S103: training the network structure, first taking the weighted sum of the auxiliary loss function and the trunk branch classification loss function as the optimization target, optimizing the parameters of the trunk feature extraction branch and the auxiliary supervision branch, fixing the trunk feature extraction branch parameters after the network converges, removing the auxiliary supervision branch, using the trunk feature extraction branch as a fatigue driving detection model, deploying the fatigue driving detection model to the vehicle-mounted terminal device, and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence.

[0176] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and uses the temporal self-attention mechanism to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0177] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision are implemented. For example, the computer program / instruction implements the following steps:

[0178] Step S101: collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating eyelid aspect ratio values ​​according to the eye region key points, calculating opening and closing values ​​according to the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and combining the aspect ratio value, the opening and closing value, and the rotation angle value into a feature vector sequence;

[0179] Step S102: construct a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, the feature vector sequence is input into the trunk feature extraction branch, the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label;

[0180] Step S103: training the network structure, first taking the weighted sum of the auxiliary loss function and the trunk branch classification loss function as the optimization target, optimizing the parameters of the trunk feature extraction branch and the auxiliary supervision branch, fixing the trunk feature extraction branch parameters after the network converges, removing the auxiliary supervision branch, using the trunk feature extraction branch as a fatigue driving detection model, deploying the fatigue driving detection model to the vehicle-mounted terminal device, and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence.

[0181] From the above description, it can be seen that the computer program product provided in the embodiment of the present application extracts multi-dimensional features of the driver's face such as eyelid aspect ratio, mouth opening and closing degree, and head angle through face detection and key point positioning. An innovative multi-layer self-attention encoding network is constructed, which includes two branches: trunk feature extraction and auxiliary supervision, and uses the temporal self-attention mechanism to capture the dynamic changes of facial states. An auxiliary supervision branch is introduced to predict the states of various facial parameters, and the feature learning process is optimized through a multi-task loss function. This method breaks through the limitations of traditional single feature analysis, realizes the accurate identification of the driver's fatigue state, and provides reliable technical support for improving road traffic safety.

[0182] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A fatigue detection method based on temporal self-attention and auxiliary supervision, characterized in that: The method comprises: Collecting a facial image of the driver, obtaining a facial region through a face detector, extracting eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculating an eyelid aspect ratio value based on the eye region key points, calculating an opening and closing degree value based on the mouth region key points, calculating a rotation angle value of the facial region based on a posture estimation algorithm, and composing a feature vector sequence with the aspect ratio value, the opening and closing degree value, and the rotation angle value; Constructing a multi-layer self-attention encoding network structure, the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, inputting the feature vector sequence into the trunk feature extraction branch, extracting facial state temporal features through a multi-layer temporal self-attention mechanism, the auxiliary supervision branch receiving the temporal features as input, outputting predicted eyelid opening and closing states, mouth opening and closing states, and head posture states, and constructing an auxiliary loss function based on the difference between the predicted states and the true labels; The network structure is trained. First, the weighted sum of the auxiliary loss function and the main branch classification loss function is used as the optimization target, and the parameters of the main feature extraction branch and the auxiliary supervision branch are optimized. After the network converges, the parameters of the main feature extraction branch are fixed, and the auxiliary supervision branch is removed. The main feature extraction branch is used as a fatigue driving detection model, and the fatigue driving detection model is deployed to a vehicle-mounted terminal device. The driver's fatigue state is judged based on the fatigue driving detection model and the feature vector sequence.

2. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The method of collecting the driver's facial image, obtaining the facial region through a face detector, and extracting the eye region key points and the mouth region key points in the facial region based on a key point detection algorithm includes: A vehicle-mounted camera is used to collect a driver's facial image sequence, the collected image sequence is input into a multi-scale face detector, a target detection algorithm is executed on each frame of the image, a rectangular bounding box of the facial area is located, facial image data within the rectangular bounding box is extracted, and illumination equalization and size standardization processing are performed on the facial image data; A key point feature map pyramid is constructed for the face image data, the coordinates of facial key points are extracted through a cascaded deformation network, the extracted key points are divided into eye area key points and mouth area key points according to facial areas, a key point index table is established, and smoothing filtering is performed on the key point coordinate data to eliminate jitter.

3. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The step of calculating the eyelid aspect ratio value according to the eye region key points, calculating the opening and closing value according to the mouth region key points, calculating the rotation angle value of the facial region based on a posture estimation algorithm, and forming a feature vector sequence with the aspect ratio value, the opening and closing value, and the rotation angle value includes: Read the coordinate data of the key points of the eye area and the mouth area, calculate the longitudinal Euclidean distance of the upper and lower key points of the eyelids and the transverse Euclidean distance of the inner and outer eye corner key points, divide the longitudinal distance by the transverse distance to obtain the eyelid aspect ratio value, and similarly calculate the distance ratio of the upper and lower key points and the left and right key points of the mouth to obtain the opening and closing value; According to the three-dimensional spatial distribution of the key points of the facial area, a perspective projection equation group is constructed, the facial posture parameters are obtained through PnP solution, the pitch angle, yaw angle and roll angle values ​​of the facial area are extracted as the angle value, and the aspect ratio value, opening and closing degree value and angle value are organized into a feature vector sequence in time sequence.

4. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The method of constructing a multi-layer self-attention encoding network structure, wherein the network structure comprises a trunk feature extraction branch and an auxiliary supervision branch, inputting the feature vector sequence into the trunk feature extraction branch, extracting facial state temporal features through a multi-layer temporal self-attention mechanism, and the auxiliary supervision branch receiving the temporal features as input and outputting predicted eyelid opening and closing states, mouth opening and closing states, and head posture states, comprises: Constructing a self-attention network including multiple encoder stacks, each encoder consisting of a self-attention layer, a feedforward network layer, and a normalization layer, inputting the feature vector sequence into the self-attention layer of the first encoder, obtaining the attention weight by matrix multiplication of the query matrix, the key matrix, and the value matrix, performing weighted aggregation on the time series features based on the attention weight, and outputting them to the next encoder after being processed by the normalization layer and the feedforward network layer; The temporal features output by the last encoder are mapped to the main feature extraction branch and the auxiliary supervision branch through a fully connected layer, where the main feature extraction branch outputs a binary classification prediction result of the fatigue state, and the auxiliary supervision branch outputs the eyelid opening and closing state prediction value, the mouth opening and closing state prediction value, and the head posture state prediction value through the regression layer, and the supervision loss is calculated with the corresponding true label.

5. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The constructing of an auxiliary loss function based on the difference between the predicted state and the true label includes: Read the true labels of facial states in the training data set, including eyelid opening and closing state labels, mouth opening and closing state labels, and head posture state labels, calculate the mean square error between the predicted value output by the auxiliary supervision branch and the corresponding true label, and construct an auxiliary loss function by weighted summing the mean square errors of the three state predictions; Based on the prediction results and true labels of the auxiliary supervision branch, a cross entropy loss function is constructed as the classification loss, and the classification loss and the auxiliary loss function are added through a weight coefficient to obtain a total loss function, and the weight coefficient is used to balance the training contribution of the trunk feature extraction branch and the auxiliary supervision branch.

6. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The training of the network structure firstly takes the weighted sum of the auxiliary loss function and the main branch classification loss function as the optimization target, optimizes the parameters of the main feature extraction branch and the auxiliary supervision branch, fixes the parameters of the main feature extraction branch after the network converges, removes the auxiliary supervision branch, and uses the main feature extraction branch as a fatigue driving detection model, including: Construct a back-propagation optimizer, take the total loss function as the optimization target, use the gradient descent method to synchronously update the network parameters of the main feature extraction branch and the auxiliary supervision branch, iterate the training on the training data set until the loss function converges, and monitor the performance indicators on the validation set to ensure that the model does not overfit; The network parameters of the trained trunk feature extraction branch are fixed, the connection with the auxiliary supervision branch is released, the network structure and parameters of the auxiliary supervision branch are removed, the calculation graph of the trunk feature extraction branch is pruned and optimized, and the optimized network structure and its parameters are saved as the fatigue driving detection model.

7. The fatigue detection method based on temporal self-attention and auxiliary supervision according to claim 1 is characterized in that: The step of deploying the fatigue driving detection model to a vehicle-mounted terminal device and judging the driver's fatigue state based on the fatigue driving detection model and the feature vector sequence includes: The fatigue driving detection model is quantized and compressed, network parameters are converted from floating point numbers to fixed point numbers, the compressed model is compiled into a calculation format supported by the vehicle terminal device using a model acceleration framework, and the compiled model file is deployed to a processor of the vehicle terminal device; The feature vector sequence collected in real time is divided into sliding windows of fixed length, and normalization preprocessing is performed on the feature vector sequence in each time window, and the feature vector sequence is input into the fatigue driving detection model for forward calculation to obtain the predicted probability value of the fatigue state, and the driver's fatigue state is determined based on the predicted probability value.

8. A fatigue detection device based on temporal self-attention and auxiliary supervision, characterized in that: The device comprises: A data acquisition module is used to acquire a facial image of the driver, obtain a facial region through a face detector, extract eye region key points and mouth region key points in the facial region based on a key point detection algorithm, calculate an eyelid aspect ratio value based on the eye region key points, calculate an opening and closing degree value based on the mouth region key points, calculate a rotation angle value of the facial region based on a posture estimation algorithm, and form a feature vector sequence with the aspect ratio value, the opening and closing degree value, and the rotation angle value; A model building module is used to build a multi-layer self-attention encoding network structure, wherein the network structure includes a trunk feature extraction branch and an auxiliary supervision branch, wherein the feature vector sequence is input into the trunk feature extraction branch, and the facial state temporal features are extracted through a multi-layer temporal self-attention mechanism. The auxiliary supervision branch receives the temporal features as input, outputs the predicted eyelid opening and closing state, mouth opening and closing state, and head posture state, and constructs an auxiliary loss function based on the difference between the predicted state and the true label; The vehicle-mounted monitoring module is used to train the network structure. First, the weighted sum of the auxiliary loss function and the main branch classification loss function is used as the optimization target to optimize the parameters of the main feature extraction branch and the auxiliary supervision branch. After the network converges, the parameters of the main feature extraction branch are fixed, the auxiliary supervision branch is removed, and the main feature extraction branch is used as a fatigue driving detection model. The fatigue driving detection model is deployed to the vehicle-mounted terminal device, and the driver's fatigue state is judged based on the fatigue driving detection model and the feature vector sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fatigue detection method based on temporal self-attention and auxiliary supervision described in any one of claims 1 to 7 are implemented.

Images