A fatigue detection method, apparatus, device, and medium

Abstract

This application relates to the field of detection, and in particular to a fatigue detection method, apparatus, device, and medium. The method includes: determining a personalized fatigue detection model for a driver to be detected. This personalized fatigue detection model incorporates a personalized branch module in addition to a general fatigue detection module. The parameters of the personalized branch module are adjusted according to changes in the driver's identity information, while the parameters of the general fatigue detection module remain fixed. Furthermore, after acquiring an image of the driver to be detected, fatigue detection can be performed based on the personalized fatigue detection model. Compared to related technologies, the fatigue detection method of this application does not require the driver to wear additional equipment, resulting in lower costs. Moreover, the personalized fatigue detection model of this application can more accurately identify the driver's fatigue state, reducing false positives and false negatives, and improving driving safety.

Description

Technical Field

[0001] This application relates to the field of testing, and in particular to a fatigue testing method, apparatus, equipment, and medium. Background Technology

[0002] Currently, various fatigue detection technologies exist in the market, mainly including methods based on physiological signals such as electroencephalography (EEG) and electrocardiogram (ECG), and methods based on behavioral characteristics such as eye closure time and head movements. Physiological signal-based methods require drivers to wear additional equipment, which is inconvenient and costly. Behavioral characteristic-based methods typically use generic models that do not consider individual differences, resulting in limited accuracy. For example, some camera-based fatigue detection systems judge fatigue by monitoring the driver's eye state, but different drivers have different blinking habits, making generic models prone to misjudgment. Current mainstream driver fatigue detection methods mainly rely on single or combined features such as facial expressions, eye movements, and driving operations, using uniform discrimination criteria and failing to fully consider individual driver behavioral differences. Some studies attempt to introduce personalized factors, but the lack of systematic historical behavioral data accumulation and modeling methods makes it difficult to achieve high-precision personalized fatigue detection. Summary of the Invention

[0003] The purpose of this application is to provide a fatigue detection method, apparatus, equipment, and medium that can improve the accuracy of fatigue detection.

[0004] Firstly, a fatigue detection method is provided, including: Obtain the identity information of the driver to be tested; Based on the identity information of the driver to be detected, a personalized fatigue detection model corresponding to the driver to be detected is determined. The personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the identity information of the driver to be detected. Acquire an image of the driver to be detected; and perform fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

[0005] In a preferred embodiment, this application can be further configured to: determine a personalized fatigue detection model corresponding to the driver to be detected based on the driver's identity information, including: Based on the identity information of the driver to be detected, determine from the personalized metadata whether there are personalized branch parameters corresponding to the driver to be detected. If it exists, then determine the personalized branch parameters corresponding to the driver to be detected based on the identity information of the driver to be detected; adjust the parameters of the personalized branch module of the initial personalized fatigue detection model to the personalized branch parameters corresponding to the driver to be detected, and obtain the personalized fatigue detection model corresponding to the driver to be detected. If not, the personalized training data of the driver to be detected is obtained, and the personalized branch module of the initial personalized fatigue detection model is trained based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected.

[0006] In a preferred embodiment, this application may be further configured as follows: after training the personalized branch module of the initial personalized fatigue detection model based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, it further includes: The personalized branch parameters of the personalized fatigue detection model corresponding to the driver to be detected are written into the personalized metadata.

[0007] In a preferred embodiment, this application can be further configured to: train the personalized branch module of the initial personalized fatigue detection model based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, including: An initial personalized fatigue detection model is obtained, which includes a general fatigue detection module and an initial personalized branch module. The general fatigue detection module is trained on a large-scale dataset in the field of fatigued driving. The general fatigue detection module includes an encoding module, an intermediate module, a decoding module, and a detection head. The attention mechanism of the intermediate module is a cross-attention mechanism. Obtain the personalized training data; The personalized training data is input into the encoding module to obtain a first feature; the personalized training data is input into the initial personalized branch module to obtain a second feature; the first feature and the second feature are input into the intermediate module to obtain a third feature; the third feature is passed through the decoding module and the detection head to obtain the fatigue detection result; Based on the fatigue detection results and the second feature, a total loss value is determined; and based on the total loss value, the initial personalized branch module is iteratively trained to obtain a personalized fatigue detection model corresponding to the driver to be detected, wherein the parameters of the general fatigue detection module remain unchanged during the training process.

[0008] In a preferred embodiment, this application can be further configured such that the personalized branch module includes a low-level feature extraction module and a context information extraction module. The personalized training data is input into the initial personalized branch module to obtain the second feature, including: The personalized training data is input into the low-level feature extraction module to extract the driver's low-level features; The underlying features are input into the context information extraction module to mine individual difference features and obtain the second feature.

[0009] In a preferred embodiment, this application can be further configured to: determine the total loss value based on the fatigue detection results and the second feature, including: Based on the fatigue test results, determine the result loss value; The contrastive learning loss value is determined based on the second feature; Based on the resulting loss value and the contrastive learning loss value, the total loss value is determined.

[0010] In a preferred embodiment, this application may be further configured to include: Obtain a pre-trained general fatigue detection module and a large-scale dataset in the field of fatigued driving; Based on a large-scale dataset in the field of fatigued driving, the pre-trained general fatigue detection module is trained to obtain a general fatigue detection module.

[0011] Secondly, a fatigue detection device is provided, comprising: The acquisition module is used to acquire the identity information of the driver to be detected; The model determination module is used to determine the personalized fatigue detection model corresponding to the driver to be detected based on the driver's identity information. The personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the driver's identity information. The fatigue detection module is used to acquire an image of the driver to be detected and to perform fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

[0012] Thirdly, an electronic device is provided, the electronic device including a memory and a processor, the memory storing a computer program, the processor executing the method of any one of the first aspects when running the computer program.

[0013] Fourthly, a computer-readable storage medium is provided, wherein at least one piece of program code is stored therein, the program code being loaded and executed by a processor to implement the method as described in any of the first aspects.

[0014] Fifthly, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the method as described in any of the first aspects.

[0015] In summary, the method provided in this application offers the following beneficial technical effects: It determines a personalized fatigue detection model for the driver to be detected. This personalized fatigue detection model incorporates a personalized branch module in addition to a general fatigue detection module. The parameters of the personalized branch module are adjusted according to changes in the driver's identity information, while the parameters of the general fatigue detection module remain fixed. Furthermore, after acquiring an image of the driver to be detected, fatigue detection can be performed based on the personalized fatigue detection model. Compared to related technologies, the fatigue detection method of this application does not require the driver to wear additional equipment, resulting in lower costs. Moreover, the personalized fatigue detection model of this application can more accurately identify the driver's fatigue state, reducing false positives and false negatives, and improving driving safety.

[0016] In addition, this application also provides an apparatus, device, and medium, all of which have the aforementioned beneficial technical effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a fatigue detection method provided in an embodiment of this application; Figure 2 This is a structural diagram of a general fatigue detection module provided in an embodiment of this application; Figure 3 This is a structural diagram of a personalized fatigue detection model provided in an embodiment of this application; Figure 4 This is a structural diagram of a pre-trained general fatigue detection module provided in the embodiments of this application; Figure 5 This is a structural diagram of a lightweight Transformer module provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a detection head provided in an embodiment of this application; Figure 7 This is a schematic diagram of a personalized branch structure provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0020] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the permission and consent of the object, the permission and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the permission and consent of the object.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0023] In related technologies, fatigue driving detection is mostly based on a single data modality, such as capturing facial expression features solely through a camera, or analyzing only driving behavior data and using general machine learning models for classification. While these methods can initially identify some fatigue states, their detection performance is poor in complex real-world driving scenarios.

[0024] Most related technologies use general models. However, different drivers may behave very differently when driving normally and when fatigued. General models have difficulty accurately identifying the fatigue state of each driver, and the adaptability and robustness of the detection models are poor, resulting in inaccurate detection results.

[0025] To better understand and explain the solutions of the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0026] Image classification is an important task in the field of computer vision. It mainly involves classifying a given image into a predefined category label. In this task, the predefined category label is whether the driver is in a state of fatigue.

[0027] Keypoint Regression: Keypoint regression is a technique for predicting the locations of key points on a target object from an image or video. Keypoints are typically specific locations on an object that are representative of its location. For example, in human pose estimation, keypoints can be joints such as elbows and knees; in facial feature detection, they can be parts such as eyes, nose, and mouth.

[0028] The input image is fed into a pre-trained convolutional neural network, which extracts feature representations of the image. These features include information such as the shape and texture of the target objects in the image, providing a foundation for subsequent keypoint location prediction. Based on the extracted features, a regression algorithm is used to predict the coordinates of each keypoint. The regression model learns the mapping relationship between features and keypoint coordinates, and by adjusting the model's parameters, minimizes the error between the predicted coordinates and the actual coordinates.

[0029] The solution proposed in this application can be applied to identify driver fatigue during dangerous driving and regularly remind drivers to take breaks and avoid driving for extended periods. The vehicles include, but are not limited to, various types of vehicles, including private cars, taxis, buses, etc.

[0030] This application provides a fatigue detection method, such as... Figure 1 As shown, the method provided in this embodiment can be executed by an electronic device, which is a server. This server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The electronic device is a terminal device, which can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The method includes: S101. Obtain the identity information of the driver to be detected; The driver to be tested refers to the person driving the vehicle who is about to be tested or evaluated. The identity information used to identify the driver to be tested includes, but is not limited to, name, facial features, fingerprint features, iris features, etc.

[0031] In some embodiments, the identity information of the driver to be detected can be obtained in various ways, such as scanning the driver's identification document to obtain the driver's identity information; or, for example, using facial recognition technology to capture the driver's facial image; comparing the captured image with facial feature information pre-stored in a database, and if a match is found, the driver's identity information is obtained. It is understood that other biometric identification methods such as fingerprint recognition and iris recognition can also be used to obtain the identity information of the driver to be detected, and this is not limited here.

[0032] S102. Based on the identity information of the driver to be detected, determine the personalized fatigue detection model corresponding to the driver to be detected. The personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the identity information of the driver to be detected. A personalized fatigue detection model is used to detect the fatigue information of a specified driver. In this embodiment, the specified driver is the driver to be detected. The personalized fatigue detection model combines a general fatigue detection module and a personalized branch module for that driver, improving the accuracy and specificity of fatigue detection. The general fatigue detection module is the universally applicable part of the model, and its parameters remain fixed regardless of the driver being detected. The personalized branch module is set according to the driver's individual characteristics. Its personalized branch parameters correspond to the identity information of the driver to be detected. Different drivers will have different parameter settings due to individual differences, thereby achieving more accurate fatigue detection for each driver.

[0033] General fatigue detection modules such as Figure 2As shown, the general fatigue detection module can adopt a classic structure of backbone network and detection head. The general fatigue detection module includes: encoder, intermediate module, decoder, and detection head. The detection head includes: fatigue state classification module and feature keypoint regression module. The backbone network can use a classic encoder and decoder architecture, where the encoder and decoder are symmetrically structured. Each structure consists of a lightweight Transformer module, which, with its multi-head self-attention mechanism, can efficiently capture long-distance dependencies in images. Compared to traditional convolutional networks, it has a stronger ability to extract subtle fatigue features when processing driver images in complex driving scenarios. The detection head adopts an improved structure based on fully convolution, adjusting the original classification and regression tasks for general object detection to fatigue state classification and feature keypoint regression, thereby achieving accurate detection and state classification of fatigued driving behavior. Fatigue state classification can include multiple fatigue levels, such as normal, mild fatigue, and severe fatigue; feature keypoints can include key locations on the eyes, mouth, and head.

[0034] Personalized fatigue detection models, such as Figure 3 As shown, the output of the personalized branch module serves as an input to the intermediate module. The output of the personalized branch module, i.e., the branch of the personalized network structure, is fused with the output of the intermediate module and the encoder, and then enters the decoder for classification and regression.

[0035] In this embodiment, a pre-established correspondence between driver identity information and the parameters of the corresponding personalized branch module is constructed. Once the identity information of the driver to be detected is determined, the corresponding parameters can be retrieved based on the identity information and loaded into the model. If no identity information is detected, initialization and training can be performed to obtain the parameters of the corresponding personalized branch module, which are then further stored.

[0036] S103. Obtain the image of the driver to be detected; and perform fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

[0037] The image of the driver to be detected is set on the vehicle, and the image collected during the driver's driving includes at least the image of the driver's head.

[0038] In this embodiment, a personalized fatigue detection model is trained using the driver's historical behavior data. After determining the personalized fatigue detection model for the driver to be detected, the individual differences of each driver and their impact on fatigue status can be fully considered. Compared with the traditional general fatigue detection model, the personalized fatigue detection model can more accurately, reliably, and specifically identify the driver's fatigue status, avoid misjudgments and omissions caused by individual differences, and promptly remind the driver to rest, effectively improving driving safety.

[0039] Furthermore, the system can continuously collect new driver data and regularly update personalized branch modules, enabling the detection algorithm to evolve with changes in driver status and maintain accuracy, efficiency, and robustness in the long term.

[0040] As can be seen, in this embodiment, a personalized fatigue detection model for the driver to be detected is determined. This personalized fatigue detection model, in addition to a general fatigue detection module, also incorporates a personalized branch module. The parameters of the personalized branch module are adjusted according to changes in the driver's identity information, while the parameters of the general fatigue detection module remain fixed. Therefore, after acquiring the image of the driver to be detected, fatigue detection can be performed based on the personalized fatigue detection model. Compared to related technologies, the fatigue detection method of this application does not require the driver to wear additional equipment, resulting in lower costs. Furthermore, the personalized fatigue detection model of this application can more accurately identify the driver's fatigue state, reducing false positives and false negatives, and improving driving safety.

[0041] In one possible implementation of this application embodiment, S102 determines the personalized fatigue detection model corresponding to the driver to be detected based on the driver's identity information, including: S1021. Based on the identity information of the driver to be detected, determine from the personalized metadata whether there are personalized branch parameters corresponding to the driver to be detected. The personalized metadata stores information such as the driver's historical behavior and personalized branch parameters.

[0042] S1022. If it exists, determine the personalized branch parameters corresponding to the driver to be detected based on the driver's identity information; adjust the parameters of the personalized branch module of the initial personalized fatigue detection model to the personalized branch parameters corresponding to the driver to be detected, and obtain the personalized fatigue detection model corresponding to the driver to be detected. If present, the personalized branch parameters corresponding to the driver to be detected are directly retrieved. The initial personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed for any driver, while the personalized branch parameters differ for different drivers. Then, the parameters of the personalized branch module of the initial personalized fatigue detection model are adjusted to match the personalized branch parameters corresponding to the driver to be detected. At this point, the personalized fatigue detection model corresponding to that driver is obtained.

[0043] This personalized fatigue detection model is also pre-trained; the specific training process can be found in the training process of S1023.

[0044] S1023. If not, obtain the personalized training data of the driver to be detected, and train the personalized branch module of the initial personalized fatigue detection model based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected.

[0045] If no match is found, it means that the personalized fatigue detection model for the driver under test has not yet been generated, i.e., the parameters of the personalized branch module for the driver under test do not exist. Furthermore, a personalized fatigue detection model for the driver under test can be obtained through training, so that the parameters of the personalized branch module for the driver under test can be called subsequently for personalized fatigue detection.

[0046] In this step, after training the personalized fatigue detection model, the personalized fatigue detection model can be used directly. Alternatively, the parameters of its personalized branch modules can be stored before executing step S101 to achieve fatigue detection. The specific means used are not limited in this application embodiment, and users can set them according to their actual needs.

[0047] For a general fatigue detection module, it can be trained directly using a large-scale dataset in the field of fatigued driving, or it can be trained by first building a pre-trained general fatigue detection module and then training it using a large-scale dataset in the field of fatigued driving. The specific method used is not limited in the embodiments of this application, and users can set it according to their actual needs.

[0048] Therefore, in one possible implementation of this application embodiment, the training process of the general fatigue detection module includes: acquiring a pre-trained general fatigue detection module and a large-scale dataset in the field of fatigued driving; and training the pre-trained general fatigue detection module based on the large-scale dataset in the field of fatigued driving to obtain the general fatigue detection module.

[0049] The pre-trained general fatigue detection module includes an encoding module, a decoding module, and an intermediate module.

[0050] The encoder module consists of four stages. The input image is first downsampled and dimension-mapped through a PatchEmbedding layer. Each stage is composed of N lightweight Transformer modules stacked together. As the number of layers increases, the feature map resolution decreases while the number of channels increases. The decoder module is mirror-symmetric to the encoder. It restores the resolution through up-sampling layers. Skip connections, mimicking the U-Net structure, concatenate and fuse the local features output from each stage of the encoder with the corresponding features from the decoder to preserve more spatial detail. See also... Figure 4 , Figure 4 This is a structural diagram of a pre-trained general fatigue detection module provided in an embodiment of this application. See also... Figure 5 , Figure 5 This is a structural diagram of a lightweight Transformer module provided in the embodiments of this application.

[0051] The intermediate modules are lightweight Transformers. Each module contains, in sequence: Layer Normalization: used to stabilize the training of deep networks. Multi-Head Self-Attention (MHSA): to achieve lightweight performance, depthwise separable convolutions can be introduced to reduce the dimensionality of Q, K, and V, or window-based attention can be used to reduce computation. Feed-Forward Network (FFN): consists of two linear layers and one activation layer (such as GELU), with intermediate layers typically expanded by 2-4 times before compression. Residual Connection: runs through MHSA and FFN to prevent gradient vanishing.

[0052] The detection head is crucial for fatigue detection. The input to the detection head is the feature map output from the decoder. Structurally, it consists of two parallel branches: Classification Branch: Structure: 3x3 convolution, global average pooling (GAP), fully connected layer (FC), and softmax layer. Input: High-level semantic features from the decoder. Output: A probability vector of length 3 (corresponding to normal, mild, and severe fatigue).

[0053] Regression Branch: Structure: Two consecutive sets of 3x3 convolutions (preserving spatial resolution) and 1x1 convolutions. Input: Spatially detailed features combined with skip connections. Output: Coordinate offsets at the feature map scale, ultimately mapped to the (x, y) coordinates of K keypoints.

[0054] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a detection head provided in an embodiment of this application.

[0055] First, the input image is fed into a pre-trained convolutional neural network, which extracts feature representations of the image. These features include information such as the shape and texture of the target objects in the image, providing a foundation for subsequent keypoint location prediction. Based on the extracted features, a regression algorithm is used to predict the coordinates of each keypoint. The regression model learns the mapping relationship between features and keypoint coordinates, and by adjusting the model's parameters, minimizes the error between the predicted and actual coordinates.

[0056] In this embodiment, the training image is input into a network model that extracts feature representations of the image. These features include information such as the shape and texture of the target object in the image, providing a foundation for subsequent keypoint location prediction. Based on the extracted features, a regression algorithm is used to predict the coordinates of each keypoint. The regression model learns the mapping relationship between features and keypoint coordinates, and by adjusting the model parameters, minimizes the error between the predicted coordinates and the actual coordinates.

[0057] In one feasible embodiment, pre-training is performed using a publicly available cross-domain dataset to learn general visual features of images, such as edges and textures. The feature location prediction process of the trained convolutional neural network (including encoder and decoder) includes: using the convolutional backbone of the convolutional neural network to extract multi-scale spatial features of the training images to obtain feature map information; inputting the feature map information into the head module (detection head, which can be a fully connected layer) of the convolutional neural network for prediction and calculating the loss value; adjusting the model parameters based on the loss value and performing iterative training to obtain a pre-trained general fatigue detection module.

[0058] Subsequently, the pre-trained general fatigue detection module was fine-tuned on a large-scale dataset in the field of fatigued driving, and the loss function adopted a weighted combination approach: .in Cross-entropy loss for classifying fatigue states. The model employs a smoothed L1 loss for feature keypoint regression. The learning rate is gradually reduced to ensure rapid convergence during training while avoiding getting trapped in local optima, thus achieving a stable and accurate general-purpose fatigue driving detection capability.

[0059] Specifically, pre-training is first performed using a cross-domain dataset. The training tasks are image classification and facial landmark detection, with training labels including category labels and facial landmark location labels. Then, a model with a classic backbone network and detection head structure is trained; the pre-trained parameters are retained. A large-scale dataset in the field of fatigue driving is acquired, and the training tasks are fatigue classification and landmark location detection, with training labels including classification results and facial landmark labels. Then, the model is trained again while retaining the encoder and decoder parameters. The training classification results and landmark location detection results are obtained, and the two results are calculated with their corresponding labels to obtain... and Then, the loss value is obtained; based on the loss value L, the parameters of the model are adjusted, and finally a general fatigue detection module is obtained.

[0060] In one possible implementation, S1023 trains the personalized branch module of the initial personalized fatigue detection model based on personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, including: SA1. Obtain the initial personalized fatigue detection model. The initial personalized fatigue detection model includes a general fatigue detection module and an initial personalized branch module. The general fatigue detection module is trained on a large-scale dataset in the field of fatigued driving. The general fatigue detection module includes an encoding module, an intermediate module, a decoding module, and a detection head. The attention mechanism of the intermediate module is a cross-attention mechanism. It is understandable that when the general fatigue detection module is used as a fatigue detection model on its own, the attention mechanism of its intermediate module is a self-attention mechanism; if the feature layer output by the personalized branch is injected into the intermediate module of the general fatigue driving detection model, the attention mechanism of the intermediate module is transformed into a cross-attention mechanism.

[0061] The personalized branch structure comprises two main sub-modules: the first is a low-level feature extraction module, consisting of multiple depthwise separable residual convolutional blocks, which extracts the driver's low-level feature information; the second is a context information extraction module, consisting of multiple self-attention modules. Through the self-attention mechanism, the model can focus on each driver's unique facial features and driving habits, uncovering individual differences. Simultaneously, a positional encoding mechanism is introduced to encode spatial information in the feature maps, enhancing the model's ability to recognize driver features under different postures.

[0062] Specifically, the design logic of personalized branches is a progressive modeling from local textures to global behaviors.

[0063] See Figure 7 , Figure 7This is a schematic diagram of a personalized branch structure provided in an embodiment of this application, wherein the left side is the low-level feature extraction module and the right side is the context information extraction module.

[0064] The low-level feature extraction module includes multiple depthwise separable residual convolutional blocks; these blocks include layer normalization, depthwise convolution, BN, ReLU, pointwise convolution, BN, ReLU, and fusion summation units. Lightweight convolution is achieved through this low-level feature extraction module, focusing on local facial textures.

[0065] For the underlying feature extraction module, the core structure is a depthwise separable residual convolution. Its function is lightweighting: compared to standard convolution, depthwise separable convolution significantly reduces computational cost, making it suitable for automotive embedded environments.

[0066] The underlying feature extraction module extracts driver features in at least four dimensions: 1. Geometric edges and contours: the core of the underlying features. The model uses convolutional kernels to identify abrupt changes in pixel intensity and capture line information. Specifically, this includes: eye contours, the edge curves of the upper and lower eyelids, which are crucial for subsequent judgment of "eye opening and closing"; mouth shape, the boundary line of the lips, used to extract the initial features of yawning; facial contours, face shape, jawline, used to distinguish the skeletal features of different drivers. 2. Texture and local details: Skin texture: wrinkles, eye bags, or the undulating texture of facial muscles (feature differences among drivers of different ages). Hair features: the density of eyebrows, the direction of eyelashes, and even beards, which are important identifiers for distinguishing individual identities. 3. Gradients and Orientations: describing the direction of the most dramatic change in pixel values. Contrast between the iris and sclera: the brightness gradient between the black part of the eyeball and the white part of the eye, which is the basic underlying data for accurately locating the pupil and calculating the direction of gaze. Nose bridge shadow: Creates a sense of depth through light and shadow gradients, aiding the model in understanding the 3D orientation of the face. 4. Color and Intensity Distributions. Although usually converted to grayscale or normalized, local brightness distribution remains a feature: local lighting variations, and the bright and dark patches on the driver's face caused by light from outside the car window. Feature point contrast, and extreme brightness values ​​near key points (such as highlights on the tip of the nose).

[0067] For the context information extraction module, the core structure consists of self-attention and position encoding. Its function is to address long-distance dependencies by capturing the relative positional relationships between facial features. For example, if a driver habitually squints or looks down, the model identifies this as a personal habit rather than fatigue from a global perspective.

[0068] The self-attention module includes: a feedforward network, a linter-gelu-linear network, an add-layer-norm (ADD), multiple self-attention mechanisms, a layer-norm (LAYER NORM), and an add-layer-norm (ADD). Through the context information extraction module, it globally focuses on driving habits and behavior patterns, adapting to unique posture changes.

[0069] SA2, Obtain personalized training data; For training a personalized fatigue model, data collection can begin by constructing a personalized driver dataset. Specifically, dual cameras are installed inside the vehicle: a front-facing camera captures the driver's overall facial and upper body posture, while a side-facing camera focuses on capturing details of the eyes and mouth. Then, a semi-automatic annotation method is used. The collected data is initially annotated using the aforementioned general fatigue driving detection module, followed by manual review and correction by professional annotators. Annotation includes fatigue state labels and key feature points. Fatigue state labels can include multiple levels of fatigue, such as normal, mild fatigue, and severe fatigue; key feature points include several important key areas, such as the eyes, mouth, and head. Next, the collected data undergoes preprocessing, including image enhancement (adjusting brightness, contrast, and saturation), data cleaning (removing noise and incorrectly labeled data), and data normalization (unifying different types of data to the same range) to improve data quality and consistency.

[0070] Furthermore, the data is categorized and stored according to driver identity, constructing a personalized dataset for each driver. The dataset employs a hierarchical storage structure, including a raw data layer, a preprocessed data layer, and a labeled data layer, facilitating data management and use. Metadata is established for each driver's personalized dataset, recording auxiliary information such as data collection time, driving scenario, and vehicle status. This metadata will aid in subsequent model training and analysis. In deep learning models, metadata is typically used as auxiliary features to provide contextual background for image features. Metadata is usually not used at the beginning of the encoder but is injected between the encoder and decoder.

[0071] SA3. Input the personalized training data into the encoding module to obtain the first feature; input the personalized training data into the initial personalized branch module to obtain the second feature; input the first and second features into the intermediate module to obtain the third feature; and use the third feature to obtain the fatigue detection result through the decoding module and the detection head. One possible implementation of this application embodiment is that the personalized branch module includes a low-level feature extraction module and a context information extraction module. The personalized training data is input into the initial personalized branch module to obtain the second feature, including: inputting the personalized training data into the low-level feature extraction module to extract the driver's low-level features; inputting the low-level features into the context information extraction module to mine individual difference features and obtain the second feature.

[0072] The second feature output from the personalized branch and the first feature output from the encoding module are injected into the intermediate module of the general fatigue driving detection model. The attention mechanism of the intermediate module is then transformed into a cross-attention mechanism. This cross-attention mechanism captures the dependencies between personalized and general features, establishing connections and promoting information exchange and integration, thereby improving detection accuracy.

[0073] SA4. Based on the fatigue detection results and the second feature, determine the total loss value; and iteratively train the initial personalized branch module based on the total loss value to obtain the personalized fatigue detection model corresponding to the driver to be detected. During the training process, the parameters of the general fatigue detection module remain unchanged.

[0074] One possible implementation of this application embodiment is to determine the total loss value based on the fatigue detection result and the second feature, including: determining the result loss value based on the fatigue detection result; determining the contrastive learning loss value based on the second feature; and determining the total loss value based on the result loss value and the contrastive learning loss value.

[0075] In this embodiment, the parameters of the general fatigue driving detection model are frozen, and training is performed only on the personalized network structure branch. The dataset used is the aforementioned personalized dataset of drivers. Since the parameters of the personalized network structure branch are relatively lightweight, a specific branch weight is fine-tuned for each driver to facilitate subsequent online model updates. The loss function is based on the general model loss (i.e., the outcome loss value L), with the addition of a contrastive learning loss value for the personalized branch. By maximizing the similarity of features of the same driver at different times and minimizing the similarity of features of different drivers, the model's ability to learn individual features is enhanced. The total loss function is... .in, The initial value of the balance coefficient is set to 0.2.

[0076] Specifically, a personalized network structure branch is introduced, while the model parameters for the encoder, intermediate module, decoder, fatigue state classification, and feature keypoint regression modules remain unchanged. During training of the personalized network structure branch, the input driver data includes: driver image, metadata information, and labels, with labels including fatigue results and keypoint location labels. During training, the driver image is input to the encoder to obtain the encoded result; the driver image is then input to the personalized network structure branch to obtain personalized features; the encoded result and personalized features are input to the intermediate module for fusion, and then input to the decoder to obtain... ; then calculate ;calculate .

[0077] The calculation method of Lcon (contrastive learning loss) will be further explained. The goal of contrastive learning loss is to make different samples of the same driver as close as possible in the feature space, and to make samples of different drivers as far apart as possible.

[0078] The InfoNCE loss is typically used. The calculation steps are as follows: 1. Input: Randomly sample a batch of data from the dataset, containing samples from multiple drivers. 2. Feature extraction: Input the samples into the model and extract the feature vector f from the output layer of the personalized network branch. 3. Construct positive and negative sample pairs. Positive sample pairs are features from different images or at different times of the same driver. If there are N drivers, and each driver takes K different samples, then the K samples from that driver are positive samples. Negative sample pairs are features from different drivers. 4. Calculation formula: For a batch containing M samples (assuming each driver has 2 different samples, i.e., M = 2 × number of drivers), for a positive sample pair (i, j), where sample j is a sample from the same driver as sample i, the loss is calculated as follows: ;wherein, sim( ): Cosine similarity (usually expressed as a dot product). τ: Temperature coefficient (controls the degree of attention given to difficult samples). 1[k≠i]: Indicator function, excluding comparisons with itself. Numerator: Two features that bring the same driver closer together. Denominator: Features that push the driver away from all other drivers (and other samples in the batch).

[0079] Furthermore, the model is updated and feedback is provided online. 10% of the latest collected data is randomly selected as the validation set, and the remaining data is used as the training set. An incremental learning method is employed, updating parameters only for the customized network structure branches. After the update, performance is evaluated on the validation set. If the updated model's accuracy improves by more than 2%, the new model is deployed in real-world applications; otherwise, the original model is retained, and the updated data is relabeled and added to the training set, awaiting the next update opportunity. Simultaneously, information such as model performance changes and updated data characteristics is fed back to the user, prompting them to pay attention to their driving status.

[0080] One possible implementation of this application embodiment is as follows: after training the personalized branch module of the initial personalized fatigue detection model based on personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, the method further includes: writing the personalized branch parameters of the personalized fatigue detection model corresponding to the driver to be detected into personalized metadata.

[0081] Based on any of the above embodiments, this application provides a personalized fatigue driving detection method, which specifically includes: constructing a general fatigue driving detection model; data collection to construct a personalized driver dataset; personalized network structure branch design and training; and online model updating and feedback.

[0082] If the model needs to serve 5 drivers, it needs to store the weights of 5 sets of personalized branches, meaning each driver corresponds to an independent, lightweight set of network parameters. The general model parts, such as the backbone network and feature extractor, are shared and frozen, while the personalized branches are stored and loaded separately for each driver.

[0083] I. Specific training steps for 5 or more drivers.

[0084] Step 1: Initialization and Cold Start. When a new driver first gets into the car, the personalized branch uses a set of universal initial weights. Data: The system uses the driver's normal driving data from the first 5-10 minutes as an anchor point.

[0085] Step 2: Online Personalized Fine-Tuning. Freeze: Freeze the general backbone network parameters. Optimize: Optimize only for the personalized branch, using the driver's image as a positive sample and other driver features stored in the database as negative samples, and calculate the loss. Calculate the contrastive learning loss to narrow the distance between the driver's real-time features and their own normal baseline features, while widening the distance between them and others.

[0086] Step 3: Weight Storage and Update. After training is complete, the system does not update the general model, but only saves the branch difference weights after this fine-tuning and associates them with the driver's personalized dataset metadata.

[0087] II. Specific reasoning steps.

[0088] When the system starts or a driver is switched, the following steps are performed: 1. Identity Recognition: The current driver ID (n) is determined through facial recognition or the vehicle system ID. 2. Weight Indexing: The model extracts the corresponding personalized branch weights from the driver's personalized metadata. 3. Dynamic Injection: The personalized branch weights are loaded into the personalized branch. 4. Forward Inference: The model combines the general parameters of the result and the loaded personalized weights to perform real-time inference.

[0089] The following describes a device provided by an embodiment of this application. The device described below can be referred to in correspondence with the method described above. The device of this embodiment is installed in an electronic device. Figure 8 , Figure 8 This is a structural block diagram of an apparatus according to one embodiment of this application, including: an acquisition module 210 for acquiring the identity information of a driver to be detected; a model determination module 220 for determining a personalized fatigue detection model corresponding to the driver to be detected based on the identity information of the driver to be detected, wherein the personalized fatigue detection model includes: a general fatigue detection module and a personalized branch module, the parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the identity information of the driver to be detected; and a fatigue detection module 230 for acquiring an image of the driver to be detected and performing fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

[0090] In one feasible approach, the model determination module 220 is configured to: determine, based on the identity information of the driver to be detected, whether there are personalized branch parameters corresponding to the driver to be detected from the personalized metadata; if so, determine the personalized branch parameters corresponding to the driver to be detected based on the identity information of the driver to be detected; adjust the parameters of the personalized branch module of the initial personalized fatigue detection model to the personalized branch parameters corresponding to the driver to be detected, thereby obtaining the personalized fatigue detection model corresponding to the driver to be detected; if not, obtain the personalized training data of the driver to be detected, and train the personalized branch module of the initial personalized fatigue detection model based on the personalized training data, thereby obtaining the personalized fatigue detection model corresponding to the driver to be detected.

[0091] In one possible implementation, it also includes a storage module for writing personalized branch parameters of the personalized fatigue detection model corresponding to the driver to be detected into personalized metadata.

[0092] In one feasible approach, the model determination module 220 is configured to: acquire an initial personalized fatigue detection model, which includes a general fatigue detection module and an initial personalized branch module. The general fatigue detection module is trained on a large-scale dataset in the field of fatigued driving. The general fatigue detection module includes an encoding module, an intermediate module, a decoding module, and a detection head. The attention mechanism of the intermediate module is a cross-attention mechanism. The module acquires personalized training data; inputs the personalized training data into the encoding module to obtain a first feature; inputs the personalized training data into the initial personalized branch module to obtain a second feature; inputs the first and second features into the intermediate module to obtain a third feature; passes the third feature through the decoding module and the detection head to obtain a fatigue detection result; determines a total loss value based on the fatigue detection result and the second feature; and iteratively trains the initial personalized branch module based on the total loss value to obtain a personalized fatigue detection model corresponding to the driver to be detected. During training, the parameters of the general fatigue detection module remain unchanged.

[0093] In one feasible approach, the personalized branch module includes a low-level feature extraction module and a context information extraction module. The model determination module 220 is used to: input personalized training data into the low-level feature extraction module to extract the driver's low-level features; input the low-level features into the context information extraction module to mine individual difference features and obtain the second feature.

[0094] In one feasible approach, the model determination module 220 is used to: determine a result loss value based on the fatigue detection result; determine a contrastive learning loss value based on a second feature; and determine a total loss value based on the result loss value and the contrastive learning loss value.

[0095] In one feasible approach, the model determination module 220 is further configured to: acquire a pre-trained general fatigue detection module and a large-scale dataset in the field of fatigued driving; and train the pre-trained general fatigue detection module based on the large-scale dataset in the field of fatigued driving to obtain a general fatigue detection module.

[0096] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 9 As shown, the electronic device includes: a memory 60 for storing a computer program; and a processor 61 for executing the computer program to implement the steps of the method as described in the above embodiments.

[0097] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0098] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0099] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc.

[0100] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0101] Those skilled in the art will understand that Figure 9 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0102] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, magnetic disks, or optical disks, and other media capable of storing program code.

[0103] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0104] Based on this, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method. It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0105] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A fatigue detection method characterized by, include: Obtain the identity information of the driver to be tested; Based on the identity information of the driver to be detected, a personalized fatigue detection model corresponding to the driver to be detected is determined. The personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the identity information of the driver to be detected. Acquire an image of the driver to be detected; and perform fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

2. The method of claim 1, wherein, Based on the identity information of the driver to be detected, a personalized fatigue detection model corresponding to the driver to be detected is determined, including: Based on the identity information of the driver to be detected, determine from the personalized metadata whether there are personalized branch parameters corresponding to the driver to be detected. If it exists, then determine the personalized branch parameters corresponding to the driver to be detected based on the identity information of the driver to be detected; adjust the parameters of the personalized branch module of the initial personalized fatigue detection model to the personalized branch parameters corresponding to the driver to be detected, and obtain the personalized fatigue detection model corresponding to the driver to be detected. If not, the personalized training data of the driver to be detected is obtained, and the personalized branch module of the initial personalized fatigue detection model is trained based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected.

3. The method of claim 2, wherein, After training the personalized branch module of the initial personalized fatigue detection model based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, the method further includes: The personalized branch parameters of the personalized fatigue detection model corresponding to the driver to be detected are written into the personalized metadata.

4. The method of claim 2, wherein, The personalized branch module of the initial personalized fatigue detection model is trained based on the personalized training data to obtain the personalized fatigue detection model corresponding to the driver to be detected, including: An initial personalized fatigue detection model is obtained, which includes a general fatigue detection module and an initial personalized branch module. The general fatigue detection module is trained on a large-scale dataset in the field of fatigued driving. The general fatigue detection module includes an encoding module, an intermediate module, a decoding module, and a detection head. The attention mechanism of the intermediate module is a cross-attention mechanism. Obtain the personalized training data; The personalized training data is input into the encoding module to obtain a first feature; the personalized training data is input into the initial personalized branch module to obtain a second feature; the first feature and the second feature are input into the intermediate module to obtain a third feature; the third feature is passed through the decoding module and the detection head to obtain the fatigue detection result; Based on the fatigue detection results and the second feature, a total loss value is determined; and based on the total loss value, the initial personalized branch module is iteratively trained to obtain a personalized fatigue detection model corresponding to the driver to be detected, wherein the parameters of the general fatigue detection module remain unchanged during the training process.

5. The method of claim 4, wherein, The personalized branching module includes a low-level feature extraction module and a context information extraction module. The personalized training data is input into the initial personalized branch module to obtain the second feature, including: The personalized training data is input into the low-level feature extraction module to extract the driver's low-level features; The underlying features are input into the context information extraction module to mine individual difference features and obtain the second feature.

6. The method of claim 4, wherein, Based on the fatigue detection results and the second feature, the total loss value is determined, including: Based on the fatigue test results, determine the result loss value; The contrastive learning loss value is determined based on the second feature; Based on the resulting loss value and the contrastive learning loss value, the total loss value is determined.

7. The method of claim 1, wherein, Also includes: Obtain a pre-trained general fatigue detection module and a large-scale dataset in the field of fatigued driving; Based on a large-scale dataset in the field of fatigued driving, the pre-trained general fatigue detection module is trained to obtain a general fatigue detection module.

8. A fatigue detection device characterized by comprising: include: The acquisition module is used to acquire the identity information of the driver to be detected; The model determination module is used to determine the personalized fatigue detection model corresponding to the driver to be detected based on the driver's identity information. The personalized fatigue detection model includes a general fatigue detection module and a personalized branch module. The parameters of the general fatigue detection module are fixed, and the personalized branch parameters of the personalized branch module correspond to the driver's identity information. The fatigue detection module is used to acquire an image of the driver to be detected and to perform fatigue detection on the image according to the personalized fatigue detection model corresponding to the driver to be detected.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the method according to any one of claims 1 to 7 when running the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 7.

Images