Fatigue-state detection method and apparatus

Abstract

A fatigue-state detection method. The method comprises: acquiring facial key points of a user to be subjected to detection; on the basis of the facial key points and preset facial parameters, performing three-dimensional facial reconstruction on said user, so as to determine facial expression coefficients, wherein the preset facial parameters are facial feature information obtained after three-dimensional modeling is performed on a facial image; performing serialization processing on the facial expression coefficients within a set time, so as to obtain a timing sequence of the facial expression coefficients; and on the basis of the timing sequence of the facial expression coefficients, using a pre-trained detection model to detect a fatigue state of said user within the set time, so as to obtain a fatigue detection result.

Description

Methods and apparatus for detecting fatigue state

[0001] This application claims priority to Chinese Patent Application No. 202410489669.5, filed on April 23, 2024, entitled “Method and Apparatus for Detecting Fatigue Condition”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of fatigue detection technology, and in particular to a method and apparatus for detecting fatigue state. Background Technology

[0003] Fatigue has always been a significant safety issue for drivers, whether in traditional manual driving or driver assistance systems in intelligent vehicles. In purely manual driving situations, fatigue affects the driver's reaction speed, takeover speed, and judgment accuracy; in driver assistance modes, fatigue also affects the safety of human-machine co-driving and handover.

[0004] In related technologies, fatigue detection algorithms primarily determine whether a user is fatigued by defining the duration of time their eyes are closed and their mouth is open, as defined by rules in the algorithm. This process relies on the stability of the rules in the detection algorithm; if the rules change, the corresponding detection results will also have errors, making it impossible to accurately detect fatigue.

[0005] Application content

[0006] In view of this, this application provides a method and apparatus for detecting fatigue state, the main purpose of which is to solve the problem that fatigue detection algorithms in the prior art cannot accurately detect fatigue state.

[0007] According to a first aspect of this application, a method for detecting fatigue state is provided, the method comprising:

[0008] Obtain facial key points of the user to be detected;

[0009] Based on the facial key points and preset facial parameters, a three-dimensional facial reconstruction of the user to be detected is performed to determine the facial expression coefficient. The preset facial parameters are the facial feature information obtained after the facial image is three-dimensionally modeled.

[0010] The facial expression coefficients within a set time period are serialized to obtain a temporal sequence of facial expression coefficients;

[0011] Based on the temporal sequence of the facial expression coefficients, a pre-trained detection model is used to detect the fatigue state of the user to be detected within a set time period, and fatigue detection results are obtained.

[0012] Furthermore, the acquisition of facial key points of the user to be detected specifically includes:

[0013] Acquire the image of the user to be detected output by the camera device;

[0014] The face image of the user to be detected is obtained by identifying the face location in the image of the user to be detected through a pre-trained face recognition model.

[0015] The face image is subjected to key point detection by a pre-trained key point detection model to obtain the key points of the face of the user to be detected.

[0016] Furthermore, the step of performing 3D facial reconstruction of the user to be detected based on the facial key points and preset facial parameters, and determining the facial expression coefficients, specifically includes:

[0017] The facial key points are combined with preset facial parameters and then the three-dimensional facial reconstruction is performed using a pre-trained facial statistical model to obtain the three-dimensional facial model of the user to be detected.

[0018] In the process of facial 3D reconstruction, the shape parameters of the 3D facial model are iteratively solved, and the facial expression coefficients are determined based on the shape parameters.

[0019] Furthermore, the iterative solution of the shape parameters of the 3D face model during the 3D face reconstruction process, and the determination of facial expression coefficients based on the shape parameters, specifically includes:

[0020] During the 3D reconstruction of the face, basis functions describing the changes in the face are obtained;

[0021] For the basis functions describing facial changes, the shape parameters of the 3D facial model are solved by parameter estimation.

[0022] Based on the shape parameters of the three-dimensional face model, the basis functions describing face changes are weighted and superimposed to obtain the facial expression coefficients.

[0023] Furthermore, the step of obtaining the basis functions describing facial changes during the 3D facial reconstruction process specifically includes:

[0024] During the three-dimensional reconstruction of the face, the location information of key facial points is obtained;

[0025] Based on the location information of the facial key points, shape modeling and texture modeling are performed on the preset facial parameters to obtain basis functions describing facial changes. The basis functions describing facial changes include identity basis functions with inherent features and expression basis functions with dynamic expression.

[0026] Accordingly, the step of weighting and superimposing the basis functions describing facial changes based on the shape parameters of the three-dimensional face model to obtain facial expression coefficients specifically includes:

[0027] Based on the shape parameters of the three-dimensional face model, the identity basis function and the expression basis function are weighted and superimposed to obtain the facial expression coefficients.

[0028] Furthermore, the step of detecting the fatigue state of the user under test within a set time period using a pre-trained detection model based on the temporal sequence of the facial expression coefficients to obtain fatigue detection results specifically includes:

[0029] The time series of the facial expression coefficients is processed to convert each element in the time series into a discrete form of facial expression coefficient;

[0030] The discrete facial expression coefficients are input into a pre-trained detection model, and the discrete facial expression coefficients are spatiotemporally fused to obtain the deep features of facial expressions.

[0031] The fatigue state of the user under test is detected based on the depth features of the facial expression within a set time period, and the fatigue detection result is determined.

[0032] Furthermore, the step of detecting the fatigue state of the user under test within a set time period based on the depth features of the facial expression and determining the fatigue detection result specifically includes:

[0033] The deep features of the facial expressions are processed using flat pooling and fully connected methods to obtain the global features of the facial expressions.

[0034] Based on the preset fatigue expression, determine the fatigue score of the global features of the facial expression on different fatigue labels;

[0035] Based on the global features of the facial expressions and the fatigue scores on different fatigue tags, the fatigue state of the user to be tested is detected within a set time period, and the fatigue detection result is determined.

[0036] According to a second aspect of this application, a fatigue state detection device is provided, the device comprising:

[0037] The acquisition unit is used to acquire the facial key points of the user to be detected;

[0038] The determining unit is used to perform three-dimensional facial reconstruction of the user to be detected based on the facial key points and preset facial parameters, and to determine the facial expression coefficients. The preset facial parameters are facial feature information obtained after the facial image is three-dimensionally modeled.

[0039] The processing unit is used to serialize the facial expression coefficients within a set time period to obtain a temporal sequence of the facial expression coefficients.

[0040] The detection unit is used to detect the fatigue state of the user to be detected within a set time period based on the temporal sequence of the facial expression coefficients and a pre-trained detection model, and obtain the fatigue detection result.

[0041] Furthermore, the acquisition unit is specifically used to acquire the image of the user to be detected output by the camera; identify the face position in the image of the user to be detected by a pre-trained face recognition model to obtain the face image of the user to be detected; and perform key point detection on the face image by a pre-trained key point detection model to obtain the face key points of the user to be detected.

[0042] Furthermore, the determining unit includes:

[0043] The reconstruction module is used to combine the facial key points with preset facial parameters and then perform three-dimensional facial reconstruction through a pre-trained facial statistical model to obtain a three-dimensional facial model of the user to be detected.

[0044] The determination module is used to iteratively solve the shape parameters of the three-dimensional face model during the three-dimensional reconstruction of the face, and determine the facial expression coefficients based on the shape parameters.

[0045] Furthermore, the determining module is specifically used to obtain basis functions describing facial changes during the three-dimensional reconstruction of the face; to solve for the shape parameters of the three-dimensional face model using parameter estimation methods for the basis functions describing facial changes; and to perform weighted superposition of the basis functions describing facial changes based on the shape parameters of the three-dimensional face model to obtain facial expression coefficients.

[0046] Furthermore, the determining module is specifically used to obtain the position information of facial key points during the three-dimensional reconstruction of the face; to perform shape modeling and texture modeling on the preset facial parameters based on the position information of the facial key points, and to obtain basis functions describing facial changes, wherein the basis functions describing facial changes include identity basis functions with inherent features and expression basis functions with dynamic expression.

[0047] Accordingly, the step of weighting and superimposing the basis functions describing facial changes according to the shape parameters of the three-dimensional face model to obtain facial expression coefficients specifically includes: weighting and superimposing the identity basis function and the expression basis function according to the shape parameters of the three-dimensional face model to obtain facial expression coefficients.

[0048] Furthermore, the detection unit includes:

[0049] The processing module is used to process the time sequence of the facial expression coefficients to convert each element in the time sequence into a discrete form of facial expression coefficients.

[0050] The fusion module is used to input the discrete facial expression coefficients into a pre-trained detection model, and to perform spatiotemporal fusion on the discrete facial expression coefficients to obtain the deep features of facial expressions.

[0051] The detection module is used to detect the fatigue state of the user to be detected within a set time period based on the depth features of the facial expression, and to determine the fatigue detection result.

[0052] Furthermore, the detection module is specifically used to process the deep features of the facial expression using average pooling and a fully connected method to obtain the global features of the facial expression; determine the fatigue score of the global features of the facial expression on different fatigue labels based on the preset fatigue expression; and detect the fatigue state of the user to be detected within a set time period based on the fatigue score of the global features of the facial expression on different fatigue labels to determine the fatigue detection result.

[0053] According to a third aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described fatigue state detection method.

[0054] According to a fourth aspect of this application, a fatigue state detection device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described fatigue state detection method.

[0055] By employing the above technical solution, this application provides a fatigue state detection method and apparatus. Compared with the current method of using rule definitions in fatigue detection algorithms to determine whether a user is in a fatigue state, this application obtains the facial key points of the user to be detected, performs 3D facial reconstruction of the user's face based on the facial key points and preset facial parameters, determines facial expression coefficients, and the preset facial parameters are the facial feature information obtained after 3D modeling of the facial image. Then, the facial expression coefficients within a set time period are serialized to obtain a temporal sequence of facial expression coefficients. Finally, based on the temporal sequence of facial expression coefficients, a pre-trained detection model is used to detect the fatigue state of the user to be detected within the set time period to obtain fatigue detection results. The entire process uses facial expression coefficients obtained from 3D reconstruction to detect the user's fatigue state within a set time period, giving more consideration to the influence of facial expression features on fatigue state. It can also add facial expression features that help with classification to the fatigue state detection process, improving the accuracy of fatigue detection results.

[0056] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0058] Figure 1 is a flowchart illustrating a fatigue state detection method according to an embodiment of this application;

[0059] Figure 2 is a flowchart illustrating a specific implementation of step 101 in Figure 1;

[0060] Figure 3 is a schematic flowchart of a specific implementation of step 102 in Figure 1;

[0061] Figure 4 is a flowchart illustrating a specific implementation of step 104 in Figure 1.

[0062] Figure 5 is a schematic diagram of the structure of a fatigue state detection device in one embodiment of this application;

[0063] Figure 6 is a schematic diagram of the device structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0064] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0065] In related technologies, fatigue detection algorithms primarily determine whether a user is fatigued by defining the duration of time their eyes are closed and their mouth is open, based on rules in the algorithm. This process relies on the stability of the rules in the detection algorithm; if the rules change, the corresponding detection results will also have errors, making it impossible to accurately detect fatigue.

[0066] To address this issue, this embodiment provides a method for detecting fatigue state, as shown in Figure 2. This method can be applied to an in-vehicle server and includes the following steps:

[0067] 101. Obtain the facial key points of the user to be detected.

[0068] The facial key points of the user to be detected can include, but are not limited to, facial features and contours, such as eyebrows, eyes, nose, mouth, and facial outlines. Here, the user's facial key points are equivalent to key points in a two-dimensional facial image. Typically, an in-vehicle facial image acquisition system can be used to capture facial images and real-time video streams. For example, devices with facial capture capabilities, such as video surveillance cameras, transmit the captured facial images and / or real-time video streams to an in-vehicle server. The in-vehicle server then processes the facial images and / or real-time video streams to obtain the facial key points of the user to be detected.

[0069] In the specific process of acquiring facial key points, face detection algorithms can be used to identify the position and size of faces in an image. Deep learning algorithms can be used for face detection. Then, the detected faces are aligned to reduce the impact of pose and scale variations on key point extraction. Common alignment methods include feature point-based alignment and geometric transformation-based alignment. Further, on the aligned face image, feature point localization algorithms are used to locate the key points. Common feature point localization methods include template matching, regression-based, and deep learning-based methods. Finally, the feature point localization results may contain some errors or inaccuracies. Smoothing and shape constraints can be used to improve the accuracy of facial key point extraction.

[0070] In practical applications, considering that the face captured by the device may not be a frontal view, it is necessary to correct the facial pose in the image. This is done by obtaining the coordinates of the key points of the face through facial landmark detection, adjusting the angle of the face based on the coordinates of the key points, and then using affine transformation to correct the face.

[0071] The execution subject of this application embodiment can be a fatigue state detection device. It can automatically detect and locate certain key facial points, such as eyes, nose, and mouth, from a given face image. The location information of these key points contains information in the face image, which is very important for tasks such as face recognition, expression analysis, and pose estimation.

[0072] 102. Based on the facial key points and preset facial parameters, perform three-dimensional facial reconstruction of the user to be detected, and determine the facial expression coefficient.

[0073] Among them, the preset face parameters are the face feature information obtained after the face image is 3D modeled. The face feature information includes feature points or feature descriptors extracted from the 3D model of each face. These feature points can be key points of the face, such as eyes, nose, mouth, or feature descriptors of some local areas, such as texture gradient, shape curvature, etc.

[0074] Specifically, in the process of 3D facial reconstruction, a parametric 3DMM model can be built using facial key points. This model can generate different facial shapes and textures based on the input parameters. The model usually consists of a basic shape and texture model and a set of principal components that vary the shape and texture. Then, an optimization algorithm is used to iteratively match the facial features with preset facial parameters as input with the 3DMM model. During the iterative matching process, the optimal facial expression coefficient is solved. This facial expression coefficient can control the shape and texture changes of the facial model, thereby achieving different expression effects.

[0075] 103. Serialize the facial expression coefficients within a set time period to obtain a temporal sequence of facial expression coefficients.

[0076] It is understandable that facial expression coefficients contain rich label information in the current frame of the face image, such as the opening and closing state of the eyes / mouth, the raising or lowering of the eyebrows / corners of the mouth, etc. By obtaining the facial expression coefficients within a set time period and serializing the facial expression coefficients within the set time period, a temporal sequence of facial expression coefficients can be obtained.

[0077] Considering the continuity of facial feature changes within a preset time period, feature extraction is performed for each window or frame of the facial expression coefficients within the set time period. The feature extraction method can be selected according to the specific task. Here, a convolutional neural network can be used to extract image features, and then the extracted features are connected in chronological order to form a continuous sequence. The connection process can arrange the features of each window or frame in chronological order, or it can splice the features of the entire sequence to obtain the temporal sequence of facial expression coefficients.

[0078] 104. Based on the time sequence of the facial expression coefficients, a pre-trained detection model is used to detect the fatigue state of the user to be detected within a set time period, and fatigue detection results are obtained.

[0079] The pre-trained detection model can be a model trained using a self-attention neural network. Here, a fully connected mapping can be performed using a temporal sequence of facial expression coefficients within a set time period, and a mapping label is added at the beginning of the fully connected mapping. Considering that the set time period only includes fatigue state and is not sensitive to position, it is optional to add position labels. The temporal mapping vector with mapping labels is then input into the self-attention neural network model for training, outputting the probability values ​​of facial expressions on different fatigue state labels. The fatigue state label with the highest probability is used as the fatigue state classification result output by the model. The actual fatigue state label corresponding to the facial expression is compared with the fatigue state classification result output by the model, and the loss value of the model is calculated using a loss function. When the loss value does not meet the model's iteration stopping condition, the above model training process is repeated to adjust the model parameters until the loss value meets the model's iteration stopping condition. The model that meets the iteration stopping condition is then output as the detection model.

[0080] Specifically, in the fatigue state detection process, firstly, a fully connected linear mapping is performed on the temporal sequence of facial expression coefficients to obtain discrete units of facial expression coefficients. Then, a pre-trained detection model is used to extract deep features from the discrete units of facial expression coefficients. Furthermore, the extracted deep features are spatiotemporally fused through a fully connected layer. Finally, the spatiotemporally fused features are classified according to predefined fatigue state labels. The fatigue detection result is determined based on the probability values ​​of the spatiotemporally fused features distributed on different fatigue state labels.

[0081] The fatigue detection method provided in this application differs from current methods that use rule definitions in fatigue detection algorithms to determine whether a user is fatigued. This application obtains facial key points of the user to be detected, performs 3D facial reconstruction based on these key points and preset facial parameters, determines facial expression coefficients (the preset facial parameters are facial feature information obtained after 3D modeling of the facial image), and then serializes the facial expression coefficients within a set time period to obtain a temporal sequence of these coefficients. Finally, based on this temporal sequence, a pre-trained detection model is used to detect the user's fatigue state within the set time period, yielding a fatigue detection result. The entire process uses facial expression coefficients obtained from 3D reconstruction to detect the user's fatigue state within a set time period, giving greater consideration to the influence of facial expression features on fatigue. This allows for the addition of facial expression features that aid in classification to the fatigue detection process, improving the accuracy of the fatigue detection results.

[0082] In practical applications, the accuracy and efficiency of facial landmark detection are crucial for many applications, such as face recognition, expression analysis, and face tracking. Specifically, in the above embodiment, as shown in Figure 2, step 101 includes the following steps:

[0083] 201. Obtain the image of the user to be detected output by the camera device.

[0084] 202. The face location is identified in the image of the user to be detected by a pre-trained face recognition model, and the face image of the user to be detected is obtained.

[0085] 203. Perform key point detection on the face image using a pre-trained key point detection model to obtain the facial key points of the user to be detected.

[0086] Facial expressions can be used as a basis for judging fatigue status. By performing facial recognition on the image of the user to be identified, the face can be accurately located and identified in the image or video. Furthermore, the fatigue status of the user can be detected based on the facial expressions observed, thereby obtaining fatigue detection results.

[0087] Considering the accuracy and reliability of the image to be detected in subsequent analysis and application, the image of the user to be detected output by the camera device can be preprocessed, including image scaling, grayscale conversion, and histogram equalization, to improve the subsequent processing effect. Specifically, in the face recognition process, a pre-trained face recognition model can be used to find image regions that may contain faces. By calculating local features in the image, such as edges and textures, it is determined whether a face exists. Then, the extracted features are input into a classifier for classification to determine whether the region is a face. The classifier can learn the features of a series of positive and negative samples to determine whether a new image region is a face. Then, based on the output of the classifier, the regions in the image that may contain faces are determined, and the face image of the user to be detected is obtained. Rectangular or elliptical boxes are usually used to represent the position and size of the face. After obtaining the face image of the user to be detected, a pre-trained keypoint detection model can be used to locate the key points of the face, and the key points of the face of the user to be detected are obtained.

[0088] Specifically, in the above embodiment, as shown in FIG3, step 102 includes the following steps:

[0089] 301. After combining the facial key points with preset facial parameters, the face is reconstructed in three dimensions using a pre-trained facial statistical model to obtain a three-dimensional facial model of the user to be detected.

[0090] 302. In the process of three-dimensional face reconstruction, the shape parameters of the three-dimensional face model are iteratively solved, and the facial expression coefficients are determined based on the shape parameters.

[0091] Among them, the pre-trained face statistical model can use the 3DMM method. 3DMM is a process of monocular reconstruction of the face, which can parameterize the face, including shape parameters and texture parameters, to simulate the deformation and texture changes that may occur in the face.

[0092] Specifically, in determining facial expression coefficients based on shape parameters, basis functions describing facial changes can be obtained during the 3D facial reconstruction process. For the basis functions describing facial changes, the shape parameters of the 3D facial model are solved by parameter estimation. Then, based on the shape parameters of the 3D facial model, the basis functions describing facial changes are weighted and superimposed to obtain the facial expression coefficients.

[0093] The aforementioned shape parameters can be viewed as a weighted superposition of identity basis functions and expression basis functions, thus capturing comprehensive information about the face, including inherent and dynamic features. Specifically, in the process of obtaining basis functions describing facial changes during 3D face reconstruction, the positional information of facial key points can be obtained during the 3D face reconstruction process. Based on the positional information of facial key points, shape modeling and texture modeling are performed on preset facial parameters to obtain basis functions describing facial changes. The basis functions describing facial changes include identity basis functions with inherent features and expression basis functions with dynamic expression. Correspondingly, based on the shape parameters of the 3D face model, the basis functions describing facial changes are weighted and superimposed to obtain facial expression coefficients. The identity basis functions and expression basis functions can be weighted and superimposed based on the shape parameters of the 3D face model to obtain facial expression coefficients.

[0094] Specifically, in the above embodiment, as shown in FIG4, step 104 includes the following steps:

[0095] 401. Process the time sequence of the facial expression coefficients to convert each element in the time sequence into a discrete form of facial expression coefficient.

[0096] 402. Input the discrete form of facial expression coefficients into a pre-trained detection model, and perform spatiotemporal fusion on the discrete form of facial expression coefficients to obtain the deep features of facial expressions.

[0097] 403. Detect the fatigue state of the user to be detected within a set time period based on the depth features of the facial expression, and determine the fatigue detection result.

[0098] Understandably, the time series of facial expression coefficients is usually continuous data. To convert this continuous data into discrete labels for easier computer processing and analysis, the time series of facial expression coefficients can be processed, transforming the continuous data into discrete labels, each representing a specific meaning or symbol. This makes data representation and processing more convenient. In machine learning and deep learning tasks, discretizing the time series converts it into fixed-length feature vectors, which can then be used as input for training models or performing other analytical tasks.

[0099] Specifically, the fatigue state of the user under test is detected based on the deep features of facial expressions within a set time period. In the process of determining the fatigue detection result, average pooling and fully connected methods can be used to process the deep features of facial expressions to obtain global features of facial expressions. Then, based on preset fatigue expressions, the fatigue score of the global features of facial expressions on different fatigue labels is determined. Finally, the fatigue state of the user under test is detected based on the fatigue scores of the global features of facial expressions on different fatigue labels within a set time period to determine the fatigue detection result.

[0100] Furthermore, as a specific implementation of the method in Figures 1-4, this application embodiment provides a fatigue state detection device, as shown in Figure 5. The device includes: an acquisition unit 51, a determination unit 52, a processing unit 53, and a detection unit 54.

[0101] Acquisition unit 51 is used to acquire facial key points of the user to be detected;

[0102] The determining unit 52 is used to perform three-dimensional facial reconstruction of the user to be detected based on the facial key points and preset facial parameters, and to determine the facial expression coefficients. The preset facial parameters are facial feature information obtained after the facial image is three-dimensionally modeled.

[0103] Processing unit 53 is used to serialize the facial expression coefficients within a set time period to obtain a temporal sequence of facial expression coefficients;

[0104] The detection unit 54 is used to detect the fatigue state of the user to be detected within a set time period based on the time sequence of the facial expression coefficients and a pre-trained detection model, so as to obtain the fatigue detection result.

[0105] The fatigue detection device provided in this application, compared with the current method of using rule definitions in fatigue detection algorithms to determine whether a user is in a fatigue state, obtains the facial key points of the user to be detected, performs three-dimensional facial reconstruction of the user's face based on the facial key points and preset facial parameters, determines the facial expression coefficients, and the preset facial parameters are the facial feature information obtained after three-dimensional modeling of the facial image. Then, the facial expression coefficients within a set time period are serialized to obtain a temporal sequence of facial expression coefficients. Finally, based on the temporal sequence of facial expression coefficients, a pre-trained detection model is used to detect the fatigue state of the user to be detected within the set time period to obtain the fatigue detection result. The entire process uses the facial expression coefficients obtained by solving the three-dimensional reconstruction to detect the user's fatigue state within the set time period, giving more consideration to the influence of facial expression features on fatigue state. It can also add facial expression features that help classify fatigue state to the fatigue detection process, improving the accuracy of fatigue detection results.

[0106] In a specific application scenario, the acquisition unit 51 is specifically used to acquire the image of the user to be detected output by the camera; identify the face position in the image of the user to be detected by a pre-trained face recognition model to obtain the face image of the user to be detected; and perform key point detection on the face image by a pre-trained key point detection model to obtain the face key points of the user to be detected.

[0107] In specific application scenarios, the determining unit 52 includes:

[0108] The reconstruction module is used to combine the facial key points with preset facial parameters and then perform three-dimensional facial reconstruction through a pre-trained facial statistical model to obtain a three-dimensional facial model of the user to be detected.

[0109] The determination module is used to iteratively solve the shape parameters of the three-dimensional face model during the three-dimensional reconstruction of the face, and determine the facial expression coefficients based on the shape parameters.

[0110] In a specific application scenario, the determining module is specifically used to obtain basis functions describing facial changes during the three-dimensional reconstruction of the face; for the basis functions describing facial changes, the shape parameters of the three-dimensional face model are solved by parameter estimation; and based on the shape parameters of the three-dimensional face model, the basis functions describing facial changes are weighted and superimposed to obtain facial expression coefficients.

[0111] In specific application scenarios, the determining module is further used to obtain the position information of facial key points during the three-dimensional reconstruction of the face; and to perform shape modeling and texture modeling on the preset facial parameters based on the position information of the facial key points to obtain basis functions describing facial changes. The basis functions describing facial changes include identity basis functions with inherent features and expression basis functions with dynamic expression.

[0112] Accordingly, the step of weighting and superimposing the basis functions describing facial changes according to the shape parameters of the three-dimensional face model to obtain facial expression coefficients specifically includes: weighting and superimposing the identity basis function and the expression basis function according to the shape parameters of the three-dimensional face model to obtain facial expression coefficients.

[0113] In specific application scenarios, the detection unit 54 includes:

[0114] The processing module is used to process the time sequence of the facial expression coefficients to convert each element in the time sequence into a discrete form of facial expression coefficients.

[0115] The fusion module is used to input the discrete facial expression coefficients into a pre-trained detection model, and to perform spatiotemporal fusion on the discrete facial expression coefficients to obtain the deep features of facial expressions.

[0116] The detection module is used to detect the fatigue state of the user to be detected within a set time period based on the depth features of the facial expression, and to determine the fatigue detection result.

[0117] In specific application scenarios, the detection module is specifically used to process the deep features of the facial expression using average pooling and full connection methods to obtain the global features of the facial expression; determine the fatigue score of the global features of the facial expression on different fatigue labels based on the preset fatigue expression; and detect the fatigue state of the user to be detected within a set time period based on the fatigue score of the global features of the facial expression on different fatigue labels to determine the fatigue detection result.

[0118] [Correction 21.01.2025 according to Rule 91] Based on the methods shown in Figures 1-4 above, the present application also provides a storage medium storing a computer program, which, when executed by a processor, implements the fatigue state detection method shown in Figures 1-4 above.

[0119] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0120] Based on the methods shown in Figures 1-4 and the virtual device embodiment shown in Figure 5, in order to achieve the above objectives, this application embodiment also provides a physical device for detecting fatigue state, which can be a computer, smartphone, tablet computer, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the fatigue state detection method shown in Figures 1-4.

[0121] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0122] In an exemplary embodiment, referring to FIG6, the above-described physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the fatigue state detection method in the above embodiment.

[0123] Those skilled in the art will understand that the physical device structure for detecting fatigue state provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0124] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device for detecting the aforementioned fatigue state, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with the existing methods, this application uses facial expression coefficients obtained from 3D reconstruction to detect the user's fatigue state within a set time, giving more consideration to the influence of facial expression features on fatigue state. This allows for the addition of facial expression features that facilitate classification to the fatigue state detection process, thereby improving the accuracy of fatigue detection results.

[0126] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0127] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for detecting fatigue state, wherein, include: Obtain facial key points of the user to be detected; Based on the facial key points and preset facial parameters, a three-dimensional facial reconstruction of the user to be detected is performed to determine the facial expression coefficient. The preset facial parameters are the facial feature information obtained after the facial image is three-dimensionally modeled. The facial expression coefficients within a set time period are serialized to obtain a temporal sequence of facial expression coefficients; Based on the temporal sequence of the facial expression coefficients, a pre-trained detection model is used to detect the fatigue state of the user to be detected within a set time period, and fatigue detection results are obtained.

2. The method according to claim 1, wherein, The acquisition of facial key points of the user to be detected specifically includes: Acquire the image of the user to be detected output by the camera device; The face image of the user to be detected is obtained by identifying the face location in the image of the user to be detected through a pre-trained face recognition model. The face image is subjected to key point detection by a pre-trained key point detection model to obtain the key points of the face of the user to be detected.

3. The method according to claim 1, wherein, The step of performing 3D facial reconstruction of the user to be detected based on the facial key points and preset facial parameters, and determining facial expression coefficients, specifically includes: The facial key points are combined with preset facial parameters and then the three-dimensional facial reconstruction is performed using a pre-trained facial statistical model to obtain the three-dimensional facial model of the user to be detected. In the process of facial 3D reconstruction, the shape parameters of the 3D facial model are iteratively solved, and the facial expression coefficients are determined based on the shape parameters.

4. The method according to claim 3, wherein, The step of iteratively solving the shape parameters of the 3D face model during the 3D face reconstruction process, and determining facial expression coefficients based on the shape parameters, specifically includes: During the 3D reconstruction of the face, basis functions describing the changes in the face are obtained; For the basis functions describing facial changes, the shape parameters of the 3D facial model are solved by parameter estimation. Based on the shape parameters of the three-dimensional face model, the basis functions describing face changes are weighted and superimposed to obtain the facial expression coefficients.

5. The method according to claim 4, wherein, The step of obtaining the basis functions describing facial changes during the 3D facial reconstruction process specifically includes: During the three-dimensional reconstruction of the face, the location information of key facial points is obtained; Based on the location information of the facial key points, shape modeling and texture modeling are performed on the preset facial parameters to obtain basis functions describing facial changes. The basis functions describing facial changes include identity basis functions with inherent features and expression basis functions with dynamic expression. Accordingly, the step of weighting and superimposing the basis functions describing facial changes based on the shape parameters of the three-dimensional face model to obtain facial expression coefficients specifically includes: Based on the shape parameters of the three-dimensional face model, the identity basis function and the expression basis function are weighted and superimposed to obtain the facial expression coefficients.

6. The method according to any one of claims 1-5, wherein, The step of detecting the fatigue state of the user within a set time period based on the time sequence of the facial expression coefficients, using a pre-trained detection model, and obtaining fatigue detection results, specifically includes: The time series of the facial expression coefficients is processed to convert each element in the time series into a discrete form of facial expression coefficient; The discrete facial expression coefficients are input into a pre-trained detection model, and the discrete facial expression coefficients are spatiotemporally fused to obtain the deep features of facial expressions. The fatigue state of the user under test is detected based on the depth features of the facial expression within a set time period, and the fatigue detection result is determined.

7. The method according to claim 6, wherein, The step of detecting the fatigue state of the user under test within a set time period based on the depth features of the facial expression and determining the fatigue detection result specifically includes: The deep features of the facial expressions are processed using flat pooling and fully connected methods to obtain the global features of the facial expressions. Based on the preset fatigue expression, determine the fatigue score of the global features of the facial expression on different fatigue labels; Based on the global features of the facial expressions and the fatigue scores on different fatigue tags, the fatigue state of the user to be tested is detected within a set time period, and the fatigue detection result is determined.

8. A fatigue state detection device, wherein, include: The acquisition unit is used to acquire the facial key points of the user to be detected; The determining unit is used to perform three-dimensional facial reconstruction of the user to be detected based on the facial key points and preset facial parameters, and to determine the facial expression coefficients. The preset facial parameters are facial feature information obtained after the facial image is three-dimensionally modeled. The processing unit is used to serialize the facial expression coefficients within a set time period to obtain a temporal sequence of the facial expression coefficients. The detection unit is used to detect the fatigue state of the user to be detected within a set time period based on the temporal sequence of the facial expression coefficients and a pre-trained detection model, and obtain the fatigue detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images