Working fatigue condition detection method and system based on sitting posture information and behavior data assisted facial expression recognition

By combining the algorithm of human posture and facial expression recognition, spatial flow and timing flow characteristics are extracted, and behavioral data are combined, and multi-task detection is used to use the DNN model to perform multi-task detection, which solves the problem of low fatigue detection accuracy in the existing technology and achieves high accuracy and high reliability work fatigue detection.

CN118968554AActive Publication Date: 2024-11-15SUZHOU UNIV

Patent Information

Application Number
CN202411036030.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing working fatigue detection methods based on facial expressions have low accuracy and insufficient reliability, especially affected by external factors such as ambient light and camera angle, making it difficult to accurately evaluate individual fatigue status.

Method used

Using an algorithm combining human posture and facial expression recognition, RGB video images are obtained in real time, spatial flow characteristics and timing flow characteristics of sitting posture and face are extracted, and behavioral data are combined to perform multi-task detection using DNN model to achieve high-precision fatigue situation classification. .

Benefits of technology

It improves the accuracy and reliability of work fatigue detection, ensures real-time and cost-effectiveness, and can more accurately evaluate individual fatigue status.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968554A_ABST
    Figure CN118968554A_ABST
Patent Text Reader

Abstract

The invention discloses a working fatigue condition detection method and system based on sitting posture information and behavior data assisted facial expression recognition. The method comprises the following steps: acquiring RGB video image information of a human body sitting posture and a face in real time; segmenting the single-frame image to obtain a sequence sitting posture image and a face image; acquiring a spatial stream feature vector and a time sequence stream feature vector containing the sitting posture image and the facial image features; splicing the spatial flow feature vector and the time sequence flow feature vector to obtain a facial feature vector and a sitting posture feature vector; inputting the sitting posture feature vector into a pre-trained DNN model for sitting posture classification to obtain a sitting posture classification result; inputting the sitting posture classification result into a full connection layer to obtain a sitting posture classification feature vector; and splicing and inputting the facial feature vector, the sitting posture classification feature vector and the behavior data feature vector into a DNN model for fatigue multi-task detection and classification to obtain a fatigue classification result. The method has the characteristics of high precision, high reliability and low cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a method and system for detecting work fatigue using a human body posture and facial expression recognition algorithm to perform facial expression recognition assisted by sitting posture information and behavior data. Background Art

[0002] In modern society, especially in office environments, white-collar workers usually need to sit in front of computers for long periods of time to work. Long periods of sitting work can not only cause physical discomfort, but also may cause work fatigue. Work fatigue is a complex physiological and psychological phenomenon that is difficult to describe with simple quantitative standards, and due to individual differences, fatigue manifests in various forms. A common method of judging work fatigue is to observe changes in the operator's behavioral ability and work efficiency.

[0003] In recent years, with the development of science and technology, more and more methods have been proposed to detect and evaluate the state of work fatigue. Among them, fatigue detection methods based on facial expressions have gradually attracted widespread attention. Facial expression is one of the important external manifestations of human emotions and psychological states. By analyzing facial expressions, we can indirectly understand the degree of fatigue of an individual. However, fatigue detection based solely on facial expressions may have certain limitations. For example, facial expressions may be affected by external factors such as ambient light and camera angle, resulting in inaccurate detection results. Therefore, it is urgent to develop a more accurate fatigue detection method and system. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method and system for detecting work fatigue using human body posture and facial expression recognition algorithms to perform facial expression recognition assisted by sitting posture information and behavioral data, which has the characteristics of high precision, high reliability and low cost.

[0005] Technical solution: A method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition of the present invention comprises the following steps:

[0006] Real-time acquisition of RGB video image information of human sitting posture and face;

[0007] Segment the RGB images of the human body's sitting posture and face to obtain sequential sitting posture images and facial images;

[0008] Recognize the information of the sequential sitting image and facial image respectively, extract the spatiotemporal features of the image, perform convolution feature extraction, and obtain the spatial stream feature vector and temporal stream feature vector containing the sitting image and facial image features;

[0009] The sitting posture spatial stream feature vector and the sitting posture temporal stream feature vector are concatenated to obtain the sitting posture feature vector; the facial spatial stream feature vector and the facial temporal stream feature vector are concatenated to obtain the facial feature vector;

[0010] Input the sitting posture feature vector into the pre-trained DNN model to perform sitting posture classification and obtain the sitting posture classification result; input the sitting posture classification result into the fully connected layer to obtain the sitting posture classification feature vector;

[0011] The facial feature vector, sitting posture classification feature vector and behavioral data feature vector are concatenated and input into the DNN model for fatigue multi-task detection and classification to obtain the fatigue classification result.

[0012] Furthermore, before segmenting the single-frame image, the original single-frame image needs to be preprocessed, including: converting the original color image into a grayscale image, and then using Gaussian filtering to remove noise in the grayscale image.

[0013] Furthermore, the single frame image is segmented to obtain a sequence of sitting images and facial images; specifically:

[0014] Use the MediaPipe framework to detect key points on the preprocessed image, draw key points on a black background image of equal size, and connect the key points to obtain the human skeleton;

[0015] The facial region in the preprocessed image is detected by MTCNN to obtain the facial bounding box, the facial region is cropped from the original image, and the facial key point information is detected.

[0016] Furthermore, the spatial stream feature vector containing the sitting image and facial image features is obtained, specifically:

[0017] In the spatial stream, the black background image of the drawn skeleton and the facial image are input into MobileNetV3-Small respectively to obtain the sitting posture spatial stream feature vector and the facial spatial stream feature vector respectively;

[0018] After the image is input into MobileNetV3-Small, the spatial feature size of the image is first mapped from the high-dimensional space to the low-dimensional space through the Mapping Block. The Mapping Block includes a convolutional layer, a regularization layer, and an activation layer. The mapped low-dimensional feature representation is obtained through the series operation of these three components.

[0019] Furthermore, the time series feature vector containing the sitting image and facial image features is obtained, specifically:

[0020] In the time series stream, the motion vectors are created by selecting continuous bone joint frames and facial image frames respectively, and the vectors are input into LSTM to obtain the sitting posture time series stream feature vector and the facial time series stream feature vector respectively;

[0021] ResLSTM Block is used to enhance the extraction of time series stream features by the LSTM model. ResLSTM Block includes a fully connected layer, a regularization layer, and an activation layer, and residual connections are added between these layers to improve the model's utilization of action information.

[0022] Furthermore, by selecting continuous bone joint frames and facial image frames to create motion vectors, the vectors are input into LSTM to obtain time series flow features, and the following steps are also included:

[0023] The model is regularized by the label smoothing method used for action recognition in video sequence frames, specifically:

[0024] Step 1: Assign a label L = [l1, l2, ..., l t ], where l t is the label of the t-th frame image;

[0025] Step 2: Calculate the number of times c that tag n appears in the given tag list L n (L) = ∑I(L m =L n ), where I is the indicator function, when L m =L n Take 1 when it is, otherwise take 0;

[0026] Step 3: Generate new tags: in For The tag that appears most often in the range, For The tag that appears most often in the range, is the new label of the t-th frame image.

[0027] Step 4: Smooth the label vector by:

[0028]

[0029] in, Represents the probability of the jth label of the i-th frame image.

[0030] Furthermore, the sitting posture classification results are divided into the following classification labels: lying down, head forward, right body tilt, left body tilt, right head tilt, left head tilt, left hand support, right hand support, normal sitting posture;

[0031] The method for making sitting posture classification labels includes:

[0032] Determine the tilt of the head by obtaining the Euler angle of the head;

[0033] The body tilt action is determined by obtaining the body tilt angle, and the angle between the line between the center point of the head and the center point of the shoulder and the vertical line is taken as the body tilt angle. If the tilt angle exceeds 10 degrees, the body is considered to be tilted;

[0034] The support action is determined by whether the hands are in contact with the head;

[0035] Calculate the Pitch value. Pitch represents the head pitch angle. If the Pitch value is less than -30 degrees, it means that the head is tilted too far forward, which is defined as abnormal.

[0036] Calculate the absolute Roll value. Roll represents the head roll angle. If the Roll value is greater than 15 degrees and is positive, the head is tilted to the right, otherwise it is tilted to the left.

[0037] Furthermore, the method for obtaining the behavioral data feature vector is: obtaining work behavior data through keyboard and mouse monitoring software, including the mouse movement speed, click frequency, typing speed and accuracy during work; according to the mouse activity speed value discrete curve, the mouse click frequency discrete curve, the mouse activity typing speed change discrete curve, and the typing error rate change discrete curve, extract the mean value, standard deviation, and peak characteristic value, and combine them to generate the behavioral data feature vector.

[0038] Based on the same inventive concept, in one embodiment of the present invention, a work fatigue detection system based on sitting posture information and behavior data to assist facial expression recognition includes:

[0039] Video image acquisition module, used to acquire RGB video image information of human sitting posture and face in real time;

[0040] An image segmentation module is used to segment the RGB images of the human body sitting posture and face to obtain sequential sitting posture images and facial images;

[0041] A spatial stream feature extraction module is used to extract spatial stream feature vectors of sequential sitting skeleton and facial images;

[0042] A temporal stream feature extraction module is used to extract temporal stream feature vectors of sequential sitting skeletons and facial images;

[0043] A sitting posture spatiotemporal feature concatenation module is used to concatenate the spatial stream feature vector and the temporal stream feature vector of the sitting posture to obtain a sitting posture feature vector;

[0044] A facial spatiotemporal feature concatenation module is used to concatenate the facial spatial stream feature vector and the temporal stream feature vector to obtain a facial feature vector;

[0045] A behavior data feature extraction module is used to manually extract features from behavior data to obtain a behavior feature vector;

[0046] The sitting posture classification module is used to input the sitting posture feature vector into the pre-trained DNN model to perform sitting posture classification and obtain the sitting posture classification result; the sitting posture classification result is input into the fully connected layer to obtain the sitting posture classification feature vector;

[0047] The fatigue classification module is used to splice the facial feature vector, the sitting posture classification feature vector and the behavior data feature vector into the DNN model for fatigue multi-task detection and classification to obtain the fatigue classification result.

[0048] Based on the same inventive concept, in one embodiment of the present invention, an electronic device includes a memory and a processor, wherein:

[0049] A memory for storing computer programs that can be run on the processor;

[0050] The processor is used to execute the steps of the above-mentioned work fatigue detection method based on sitting posture information and behavior data to assist facial expression recognition when running the computer program.

[0051] Beneficial effect: Compared with the prior art, the advantages of the present invention are: combining sitting posture information and behavior data to assist facial expression features in fatigue recognition, and using the advantages of spatial flow: selecting the mobilenetv3 lightweight model, the inference time is very short, and real-time performance can be guaranteed; at the same time, a Mapping Block is designed to weaken the spatial flow characteristics to strengthen the model's extraction of motion changes, and ResLSTM Block is used to enhance the LSTM model's extraction of time series flow features; the overall model fully integrates time flow and space flow features, has the characteristics of real-time, high precision, high reliability and low cost, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the detection method of the present invention;

[0053] Figure 2 This is a block diagram of the detection system of the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 1 As shown, a method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition includes the following steps:

[0056] S1, by shooting RGB video, real-time acquisition of RGB image information of human sitting posture and face;

[0057] S2. Segment the RGB images of the human body sitting posture and face to obtain sequential sitting posture images and facial images; specifically:

[0058] (1) Preprocessing the RGB images of human sitting posture and face: The original images are color images. First, the original color images need to be converted into grayscale images, and then Gaussian filtering is used to remove noise in the grayscale images;

[0059] (2) Detecting the preprocessed image, segmenting the sitting posture and facial image, and obtaining a sequence of sitting posture images and facial images; including:

[0060] First, segment the single frame image to obtain the sitting image and facial image, specifically:

[0061] Use the MediaPipe framework to detect key points of the preprocessed image, including the face, shoulders, elbows, wrists, hips, etc.; draw the key points on a black background image of equal size and connect the key points to obtain the human skeleton, that is, the sitting image; finally, resize the image to 720*1080.

[0062] The facial region in the preprocessed image is detected by MTCNN (Multi-task Cascaded Convolutional Networks) to obtain a facial bounding box, the facial region is cropped from the preprocessed image, and facial key point information is detected to obtain a facial image.

[0063] All frame images in the entire video image are segmented separately to obtain sequential sitting images and facial images.

[0064] S3, identifying the information of the sequential sitting posture images and facial images, performing convolution feature extraction, and obtaining a spatiotemporal feature vector containing the features of the sitting posture images and facial images; including:

[0065] (1) In the spatial stream, the black background image with the skeleton drawn (i.e., the sequence sitting image) and the sequence face image are respectively input into MobileNetV3-Small to obtain the sitting spatial stream feature vector and the face spatial stream feature vector, respectively.

[0066] After using MobileNetV3-Small to extract spatial stream features, the mapping block is added to weaken the spatial stream features to enhance the model's extraction of motion changes. The mapping block contains a convolutional layer, a regularization layer, and an activation layer. The mapped low-dimensional feature representation is obtained through the series operation of these three components. The spatial feature size is mapped from the high-dimensional space to the low-dimensional space through the mapping block.

[0067] (2) In the time series stream, motion vectors are created by selecting continuous skeletal joint frames in the sequence of sitting images and continuous facial image frames in the sequence of facial images. The motion vectors are input into the long short-term memory recurrent neural network (LSTM) to obtain the sitting time series stream feature vector and the facial time series stream feature vector, respectively.

[0068] Motion vectors are created by calculating the changes in each joint or feature point between consecutive frames. Specifically, motion vectors can be created in the following ways: for skeletal joint frames, the position changes of each joint over time are calculated; for facial image frames, the position changes of each facial feature point over time are calculated.

[0069] Motion vectors represent motion by calculating the position change of each joint point in different time frames. Specifically, a motion vector can be the coordinate difference of the corresponding joint points in adjacent frames. For example, if the position of a joint point in time frame t is (x_t, y_t), and the position in time frame t+1 is (x_{t+1}, y_{t+1}), then the motion vector of the joint point is ((x_{t+1}-x_t), (y_{t+1}-y_t)). The motion vectors of multiple joint points can be combined to form a complete motion description. For example, for a skeleton frame with N joint points, N motion vectors can be obtained, which are combined to represent the motion of the entire skeleton.

[0070] ResLSTM Block is used to enhance the LSTM model's extraction of time series features. ResLSTM Block combines fully connected layers, regularization layers, and activation layers, and adds residual connections between these layers to improve the model's utilization of action information. ResLSTM Block is a neural network structure that combines a residual network (Residual Network, ResNet) and a long short-term memory network (Long Short-Term Memory, LSTM).

[0071] S4. Concatenate the spatial stream feature vector and the temporal stream feature vector to obtain a facial feature vector and a sitting posture feature vector, specifically:

[0072] The sitting posture spatial stream feature vector and the sitting posture temporal stream feature vector are concatenated to obtain the sitting posture feature vector; the facial spatial stream feature vector and the facial temporal stream feature vector are concatenated to obtain the facial feature vector.

[0073] S5. Input the sitting posture feature vector into the pre-trained DNN model to perform sitting posture classification and obtain the sitting posture classification result; input the sitting posture classification result into the fully connected layer to obtain the sitting posture classification feature vector;

[0074] The sitting posture classification results are divided into the following classification labels: lying down, head forward, right body tilted, left body tilted, right head tilted, left head tilted, left hand support, right hand support, normal sitting posture.

[0075] Sitting posture classification label making includes:

[0076] Determine the tilt of the head by obtaining the Euler angle of the head;

[0077] The body tilt action is determined by obtaining the body tilt angle, and the angle between the line between the center point of the head and the center point of the shoulder and the vertical line is taken as the body tilt angle. If the tilt angle exceeds 10 degrees, the body is considered to be tilted;

[0078] The support action is determined by whether the hands are in contact with the head;

[0079] Calculate the Pitch value. Pitch represents the head pitch angle. If the Pitch value is less than -30 degrees, it means that the head is tilted too far forward, which is defined as abnormal.

[0080] Calculate the absolute Roll value. Roll represents the head roll angle. If the Roll value is greater than 15 degrees and is positive, the head is tilted to the right, otherwise it is tilted to the left.

[0081] The LSTM model is regularized by a label smoothing method for action recognition in video sequence frames. The specific steps include:

[0082] Step 1: Assign a label to each frame of the image. The label set of the entire video image is L = [l1,l2,…,l t ], where l t is the label of the t-th frame image;

[0083] Step 2: Calculate the label l of the nth image n The number of times c appears in a given tag list L n (L) = ∑I(l m = l n ), where I is the indicator function, when l m = l n 1 when it is, otherwise 0; lm is the label of the m-th image;

[0084] Step 3: Generate new tags: in For The tag that appears most often in the range, For The tag that appears most often in the range, is the new label of the t-th frame image.

[0085] Step 4: Smooth the label vector by:

[0086]

[0087] in, Represents the probability of the jth label of the i-th frame image.

[0088] S6. Concatenate the facial feature vector, the sitting posture classification feature vector, and the behavior data feature vector and input them into the DNN model for fatigue multi-task detection and classification to obtain the fatigue classification result.

[0089] In the last layer of the DNN, the Softmax function is used to convert the output of the network into a probability distribution. Assume that there are four categories for fatigue detection classification: awake, mild fatigue, moderate fatigue, and severe fatigue. Softmax outputs four probability values ​​corresponding to these four categories.

[0090] According to the probability distribution of Softmax output, the fatigue classification result is determined. The following are the specific judgment rules:

[0091] Awake: The probability value of the Softmax output corresponding to the “awake” category is the highest, and this value is greater than the set first threshold (for example, 0.5).

[0092] Mild fatigue: The probability value of the Softmax output corresponding to the “mild fatigue” category is the highest, and this value is greater than the set second threshold (eg, 0.5).

[0093] Moderate fatigue: The probability value of the Softmax output corresponding to the “moderate fatigue” category is the highest, and this value is greater than the set third threshold (eg, 0.5).

[0094] Severe fatigue: The probability value of the Softmax output corresponding to the “severe fatigue” category is the highest, and this value is greater than the set fourth threshold (eg, 0.5).

[0095] Among them, the work behavior data is obtained through keyboard and mouse monitoring software, including the mouse movement speed, click frequency, typing speed and accuracy during work. According to the discrete curve of mouse activity speed value, mouse click frequency, mouse activity typing speed change discrete curve, typing error rate change discrete curve, the corresponding average value, standard deviation, peak value and other characteristic values ​​are extracted respectively, and the behavioral data characteristic vector is generated by combining them.

[0096] like Figure 2 As shown, a work fatigue detection system based on sitting posture information-assisted facial expression recognition comprises:

[0097] RGB camera: located in front of the human body, capturing RGB video including the face and upper body;

[0098] Image segmentation module: used for sequence image segmentation to obtain sitting images and facial images;

[0099] Spatial stream feature extraction module: used to extract spatial stream features of single-frame sitting images and single-frame facial images;

[0100] Time series feature extraction module: used to extract time series features of sequential sitting images and facial images;

[0101] Sitting posture spatiotemporal feature concatenation module: used to concatenate the spatial stream feature vector and temporal stream feature vector of the sitting posture;

[0102] Facial spatiotemporal feature splicing module: used to splice the spatial stream feature vector and temporal stream feature vector of the face;

[0103] Behavior data feature extraction module: used to manually extract features from behavior data to obtain behavior feature vectors;

[0104] Sitting posture classification module: used to obtain the classification results of sitting posture;

[0105] Fatigue classification module: used to combine facial features, behavioral data features and sitting posture classification results to obtain fatigue classification results;

[0106] Behavior feature extraction module: used to extract behavior features from the behavior data and behavior data curve of the person to be detected.

[0107] Based on the same inventive concept, in one embodiment of the present invention, an electronic device includes a memory and a processor, wherein:

[0108] A memory for storing computer programs that can be run on the processor;

[0109] The processor is used to execute the steps of the above-mentioned work fatigue detection method based on sitting posture information and behavior data to assist facial expression recognition when running the computer program.

Claims

1. A method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition, characterized in that: The following steps are involved: Real-time acquisition of RGB video image information of human sitting posture and face; Segment the RGB images of the human body's sitting posture and face to obtain sequential sitting posture images and facial images; Recognize the information of the sequential sitting image and facial image respectively, extract the spatiotemporal features of the image, perform convolution feature extraction, and obtain the spatial stream feature vector and temporal stream feature vector containing the sitting image and facial image features; The sitting posture spatial stream feature vector and the sitting posture temporal stream feature vector are concatenated to obtain the sitting posture feature vector; the facial spatial stream feature vector and the facial temporal stream feature vector are concatenated to obtain the facial feature vector; Input the sitting posture feature vector into the pre-trained DNN model to perform sitting posture classification and obtain the sitting posture classification result; input the sitting posture classification result into the fully connected layer to obtain the sitting posture classification feature vector; The facial feature vector, sitting posture classification feature vector and behavioral data feature vector are concatenated and input into the DNN model for fatigue multi-task detection and classification to obtain the fatigue classification result.

2. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: Before segmenting a single-frame image, the original single-frame image needs to be preprocessed, including: converting the original color image into a grayscale image, and then using Gaussian filtering to remove noise in the grayscale image.

3. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: Segment the single-frame image to obtain sequential sitting images and facial images; specifically: Use the MediaPipe framework to detect key points on the preprocessed image, draw key points on a black background image of equal size, and connect the key points to obtain the human skeleton; The facial region in the preprocessed image is detected by MTCNN to obtain the facial bounding box, the facial region is cropped from the original image, and the facial key point information is detected.

4. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: Get the spatial stream feature vector containing the sitting image and facial image features, specifically: In the spatial stream, the black background image of the drawn skeleton and the facial image are input into MobileNetV3-Small respectively to obtain the sitting posture spatial stream feature vector and the facial spatial stream feature vector respectively; After using MobileNetV3-Small to extract spatial stream features, a mapping block is added to weaken the spatial stream features to enhance the model's extraction of motion changes. The mapping block contains a convolutional layer, a regularization layer, and an activation layer. The low-dimensional feature representation of the mapping is obtained through the series operation of the convolutional layer, the regularization layer, and the activation layer. The spatial feature size is mapped from the high-dimensional space to the low-dimensional space through the mapping block.

5. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: Get the time series feature vector containing the sitting image and facial image features, specifically: In the time series stream, the motion vectors are created by selecting continuous bone joint frames and facial image frames respectively, and the vectors are input into LSTM to obtain the sitting posture time series stream feature vector and the facial time series stream feature vector respectively; ResLSTM Block is used to enhance the extraction of time series stream features by the LSTM model. ResLSTM Block includes a fully connected layer, a regularization layer, and an activation layer, and residual connections are added between these layers to improve the model's utilization of action information.

6. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 5, characterized in that: The model is regularized by the label smoothing method used for action recognition in video sequence frames, specifically: Step 1: Assign a label L = [l1, l2, ..., l t ], where l t is the label of the t-th frame image; Step 2: Calculate the number of times c that tag n appears in the given tag list L n (L) = ∑I(L m =L n ), where I is the indicator function, when L m =L n When it is 1, otherwise it is 0; Step 3: Generate new tags: in For The tag that appears most often in the range, For The tag that appears most often in the range, is the new label of the t-th frame image. Step 4: Smooth the label vector by: in, Represents the probability of the jth label of the i-th frame image.

7. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: The sitting posture classification results are divided into the following classification labels: lying down, head forward, right body tilted, left body tilted, right head tilted, left head tilted, left hand support, right hand support, normal sitting posture; The method for making sitting posture classification labels includes: Determine the tilt of the head by obtaining the Euler angle of the head; The body tilt action is determined by obtaining the body tilt angle, and the angle between the line between the center point of the head and the center point of the shoulder and the vertical line is taken as the body tilt angle. If the tilt angle exceeds 10 degrees, the body is considered to be tilted; The support action is determined by whether the hands are in contact with the head; Calculate the Pitch value. Pitch represents the head pitch angle. If the Pitch value is less than -30 degrees, it means that the head is tilted too far forward, which is defined as abnormal. Calculate the absolute Roll value. Roll represents the head roll angle. If the Roll value is greater than 15 degrees and is positive, the head is tilted to the right, otherwise it is tilted to the left.

8. The method for detecting work fatigue based on sitting posture information and behavior data-assisted facial expression recognition according to claim 1, characterized in that: The method for obtaining the behavioral data feature vector is as follows: obtaining work behavior data through keyboard and mouse monitoring software, including the mouse movement speed, click frequency, typing speed and accuracy during work; according to the discrete curve of mouse activity speed value, the discrete curve of mouse click frequency, the discrete curve of mouse activity typing speed change, and the discrete curve of typing error rate change, extracting the mean value, standard deviation, and peak feature value, and combining them to generate the behavioral data feature vector.

9. A work fatigue detection system based on sitting posture information and behavior data to assist facial expression recognition, characterized in that: include: Video image acquisition module, used to acquire RGB video image information of human sitting posture and face in real time; An image segmentation module is used to segment the RGB images of the human body sitting posture and face to obtain sequential sitting posture images and facial images; A spatial stream feature extraction module is used to extract spatial stream feature vectors of sequential sitting skeleton and facial images; A temporal stream feature extraction module is used to extract temporal stream feature vectors of sequential sitting skeletons and facial images; A sitting posture spatiotemporal feature concatenation module is used to concatenate the spatial stream feature vector and the temporal stream feature vector of the sitting posture to obtain a sitting posture feature vector; A facial spatiotemporal feature concatenation module is used to concatenate the facial spatial stream feature vector and the temporal stream feature vector to obtain a facial feature vector; A behavior data feature extraction module is used to manually extract features from behavior data to obtain a behavior feature vector; The sitting posture classification module is used to input the sitting posture feature vector into the pre-trained DNN model to perform sitting posture classification and obtain the sitting posture classification result; the sitting posture classification result is input into the fully connected layer to obtain the sitting posture classification feature vector; The fatigue classification module is used to splice the facial feature vector, the sitting posture classification feature vector and the behavior data feature vector into the DNN model for fatigue multi-task detection and classification to obtain the fatigue classification result.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor is used to execute the steps of the work fatigue detection method based on sitting posture information and behavior data assisted facial expression recognition as described in any one of claims 1-8 when running the computer program.

Citation Information

Patent Citations

  • Fatigue driving monitoring method and system based on fusion of face recognition and behavior recognition

    CN110096957A

  • Driver fatigue state rapid detection method based on deep learning

    CN110674701A

  • Driver fatigue detection method based on skeleton data behavior recognition

    CN112131981A

  • Visual fatigue detection method based on double-flow convolutional neural network

    CN113537132A

  • Fatigue detection system and method based on facial skeleton model, medium and detection equipment

    CN113920491A

Cited By

  • Intelligent graphical interface operation optimization method based on image processing

    CN120104233A