Fatigue state detection method and system based on key point detection and head posture

Abstract

The present invention discloses a fatigue state detection method and system based on key point detection and head posture, constructs and trains an MMC multi-task prediction model whose main network adopts a deep separable convolutional network, obtains a plurality of frames of face images within a unit time, adopts an MTCNN network to detect the face position of each image and cuts out the head image; inputs the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face; uses a double threshold method to respectively determine the fatigue state of the head, eyes and mouth; sets a correlation coefficient to comprehensively determine the fatigue state of the person, combines the correlation between the key point detection of the face and the head posture, adopts the MMC multi-task prediction model whose main network is a deep separable convolutional network, places the two tasks in the same network and performs them simultaneously, which can greatly reduce the required parameter amount and calculation amount, thereby improving the detection speed of the model and achieving a real-time effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of automated fatigue detection, and in particular relates to a fatigue state detection method and system based on key point detection and head posture. Background Art

[0002] Face detection and head pose estimation in the field of computer vision refer to detecting all faces in an image and estimating that each face can represent three directional angles: yaw, pitch, and roll. Judging people's motivations and intentions based on head pose has wide applications in providing clues and gaze, such as human behavior analysis and gaze estimation. Although face detection and pose estimation have made great progress respectively, it is still a daunting task to implement a multi-task framework with good real-time and robustness in complex environments. At present, convolutional neural networks (CNNs) are generally used to solve face detection and head pose estimation, which can achieve extraordinary success in a series of complex computer vision tasks such as image classification, face recognition, and object detection. In order to solve face detection and head pose estimation at the same time, the face detection network is used to detect all face images in the face, and then the head pose estimation network is used to estimate the pose of each face. The problem is that because the two networks are separate, the inaccuracy of the face detection network will affect the result of head pose estimation. In addition, such a framework is not conducive to the intrinsic correlation between the two tasks to promote each other's performance. At the same time, complex convolutional neural networks will also affect the real-time performance of head pose estimation. Common head pose estimation systems first use a face detection network to detect faces, and then use a head pose estimation network to estimate the pose of each face. Since the two networks are separate and use different data sets, the performance of head pose estimation will be affected by the face detection network. At the same time, common convolutional neural network models are large, making it difficult to achieve real-time results when predicting. Summary of the invention

[0003] The technical problem to be solved by the present invention is how to improve the real-time performance of automated face fatigue state detection. The purpose of the present invention is to provide a fatigue state detection method and system based on key point detection and head posture estimation. In the optimization process, the network trained with the learning objectives of the two tasks of face key point detection and head posture estimation can use the same network due to the correlation between the two. The MMC multi-task prediction model with a backbone network as a deep separable convolutional network is adopted. The two tasks are performed simultaneously in the same network, which can greatly reduce the required number of parameters and the amount of calculation, thereby improving the detection speed of the model and achieving a real-time effect.

[0004] The present invention is achieved through the following technical solutions:

[0005] In one aspect, the present invention provides a fatigue state detection method based on key point detection and head posture, comprising the following steps:

[0006] Construct and train an MMC multi-task prediction model whose backbone network uses a deep separable convolutional network to obtain a trained MMC multi-task prediction model;

[0007] Get several frames of face images within a unit time, treat each frame as an image, use the MTCNN network to detect the face position of each image and crop the head image;

[0008] Input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face;

[0009] According to the head posture angle of each image, the double threshold method is used to determine the fatigue state of the head in several images. At the same time, according to the position information of the eyes and mouth in the facial key points of each image, the double threshold method is used to determine the fatigue state of the eyes and mouth in several images respectively.

[0010] The fatigue state of a person is comprehensively determined based on the fatigue state of the head, eyes and mouth.

[0011] When detecting a person's fatigue state, the person's fatigue state is generally judged based on the state of the face. However, a single method of obtaining the features of the eyes, mouth and head separately is relatively complicated and the effect is general. The common deep convolutional neural network model is difficult to achieve real-time performance and pays less attention to the role of image spatial information. After detecting the features of the eyes, mouth and head, it is necessary to establish a suitable algorithm to determine whether the person is in a fatigue state. The convolutional neural network model thus constructed is relatively large, and it is difficult to achieve real-time performance during prediction. Therefore, the present invention takes into account that both face detection and head posture estimation are tasks related to the face and rely on potential facial features. Therefore, the same network can be used to simultaneously perform face key point detection and head posture estimation tasks during training and prediction. When the same network is used, head posture information can improve the accuracy of locating face key points, which in turn can improve the accuracy of locating face key points. Point positioning can also reflect the information of head posture. There is a strong correlation between the two. This correlation has a positive effect on both tasks. The MTCNN network is first used to detect the face position and crop the head image. The preprocessed head image is input into the MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face such as the eyes and mouth. In the prediction process, the double threshold method is used to determine the fatigue state of each part, and the regressed head posture angle is used to determine the head state. The coordinates of some feature points of the eyes and mouth in the regressed face key points are used to determine the eye and mouth states respectively. Finally, the fatigue state of the person is comprehensively determined. The backbone network of the model adopts the architecture of a deep separable convolutional network, which can realize two tasks in one network at the same time, greatly reducing the number of parameters and the amount of calculation, thereby improving the detection speed of the model and achieving real-time effects.

[0012] Furthermore, the 300W_LP dataset is used for training the MMC multi-task prediction model. The 300W_LP dataset has facial key point coordinates and head posture angle labels. Before using the 300W_LP dataset to train the MMC multi-task prediction model, the images in the dataset are preprocessed, including:

[0013] The redundant background part in the image is cropped according to the coordinates of the facial key points in the data set, the cropped image size is unified to 224×224, and the unified size image is grayscaled and normalized.

[0014] Furthermore, the process of training the MMC multi-task prediction model includes two tasks:

[0015] The facial key point detection task is used to locate the position of facial feature points according to the coordinates of facial key points in the image, using the L2 loss function loss aMeasure the difference between the predicted coordinate value and the actual coordinate value of the feature point, and regress to obtain the position information of the key points of the face.

[0016] The head estimation task is used to predict the angles of the head in the image in the three directions of yaw, pitch, and roll based on the head posture angle label in the image. The loss function is:

[0017]

[0018] in, is the estimated result of the head posture angle in the three directions of yaw, pitch and roll, and (x1,x2,x3) is the head posture angle label in the three directions;

[0019] The total loss of the MMC multi-task prediction model is used as the learning target for training. The total loss is the sum of the losses of the face key point detection task and the head estimation task:

[0020] loss=loss a +ηloss b

[0021] Among them, η is the task assignment weight set to 1.

[0022] Furthermore, the process of obtaining the head posture angle and the position information of the key points of the face is as follows:

[0023] The backbone network of the MMC multi-task prediction model is used to extract and fuse the features of the input head image to obtain a feature map; the backbone network adopts an improved lightweight convolution MobileNet-V2 network structure.

[0024] At the same time, the CA attention module embedded in the backbone network is used to perform pooling operations on the feature map in the horizontal and vertical directions to obtain the position information of the feature map;

[0025] According to the position information of the feature map, the two fully connected layers after the backbone network are used to regress the head posture angle and the position information of the facial key points.

[0026] Furthermore, the improved lightweight convolutional MobileNet-V2 network structure uses 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels to extract features of the input head image, sets the convolution stride to 1, and sets the pad corresponding to each convolution kernel to 0, 1, and 2, respectively. The size of the improved lightweight convolutional MobileNet-V2 network structure is 4M.

[0027] Furthermore, for a plurality of consecutive frames of face images acquired within a unit time, the process of determining the fatigue state of each part using the double threshold method is as follows:

[0028] Determine the fatigue state of the head in each image:

[0029] According to the head posture angle of each image, it is determined whether the pitch posture angle of the head posture angle when the head is lowered is greater than 30°. If it is greater than 30°, it is determined that the head in the image is in a fatigue state; if the proportion of images in which the head is in a fatigue state accounts for more than 30% of all images, it is determined that the head is in a fatigue state;

[0030] Determine the eye fatigue status of each image:

[0031] According to the position information of the eye key points of each image, the eye aspect ratio is calculated to determine whether the eye aspect ratio is less than 0.2. If it is less than 0.2, the eye of the image is judged to be in a fatigue state; if the proportion of images in which the eye is in a fatigue state accounts for more than 40% of all images, the eye is judged to be in a fatigue state;

[0032] Determine the fatigue state of the mouth in each image:

[0033] According to the position information of the mouth key points of each image, the mouth aspect ratio is calculated to determine whether the mouth aspect ratio is greater than 0.3. If it is greater than 0.3, the mouth of the image is judged to be in a fatigued state; if the proportion of images with fatigued mouths in all images exceeds 40%, the mouth is judged to be in a fatigued state.

[0034] Furthermore, according to the influence weight of the fatigue state of the head, eyes and mouth on the fatigue state of the person, a correlation coefficient is set for the fatigue state of each part to comprehensively determine the fatigue state Z of the person:

[0035] Z=αZ eye +βZ mouth +λZ head

[0036] Among them, Z eye Each indicates the state of eye fatigue, Z mouth Indicates the fatigue state of the mouth, Z head To represent the fatigue state of the head, the related systems α, β, and λ are set to 0.2, 0.3, and 0.5, respectively;

[0037] When Z is greater than or equal to 0.5, it is judged that the person is in a fatigue state.

[0038] On the other hand, the present invention provides a fatigue state detection system based on key point detection and head posture, comprising:

[0039] The model training module is used to build and train the MMC multi-task prediction model to obtain a trained MMC multi-task prediction model;

[0040] The face position detection module is used to detect the face position of each image using the MTCNN network and crop the head image based on the acquired face image frames within a unit time, taking each frame as an image;

[0041] The parallel prediction module is used to input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face;

[0042] A local state detection module is used to determine the fatigue state of the head in several images using a double threshold method based on the head posture angle of each image, and to determine the fatigue state of the eyes and mouth in several images using a double threshold method based on the position information of the eyes and mouth in the key points of the face of each image;

[0043] The comprehensive fatigue state detection module is used to comprehensively determine a person's fatigue state based on the fatigue states of the head, eyes and mouth.

[0044] Furthermore, the MMC multi-task prediction model includes a backbone network and two fully connected layers for regressing the head posture angle and the position information of the key points of the face respectively. The backbone network adopts an improved lightweight convolutional MobileNet-V2 network structure, and a CA attention module is embedded in the backbone network.

[0045] The backbone network is used to extract and fuse the features of the input head image to obtain the feature map.

[0046] The CA attention module is used to perform pooling operations on the feature map in the horizontal and vertical directions respectively to obtain the position information of the feature map.

[0047] Furthermore, the improved lightweight convolutional MobileNet-V2 network structure uses 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels to extract features of the input head image, sets the convolution stride to 1, and sets the pad corresponding to each convolution kernel to 0, 1, and 2, respectively. The size of the improved lightweight convolutional MobileNet-V2 network structure is 4M.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] The present invention uses a deep separable convolutional network as the backbone network of the MMC multi-task prediction model, and uses the same network to train and predict the two tasks based on the task relevance of face key point detection and head posture estimation, which can greatly reduce the required parameter amount and calculation amount, thereby improving the detection speed of the model and achieving a real-time effect.

[0050] The present invention uses an improved lightweight convolutional MobileNet-V2 network structure as the backbone network. On the one hand, different scales of convolution are used in the first layer of the backbone network to extract and fuse the features of the image, and different receptive fields are obtained through multiple convolution kernels of different sizes. Considering the influence of the relative positions of the eyes, nose, mouth and other parts of the face on the posture angle, a larger receptive field can better describe this correlation, so that the model can be trained and predicted for the two tasks faster and with high real-time performance. At the same time, a CA attention module is embedded in the backbone network, which can capture accurate position information in space, thereby achieving real-time performance while also paying attention to the role of image spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0052] Figure 1 This is a flow chart of a fatigue state detection method based on facial key points and head posture in Embodiment 1 of the present invention;

[0053] Figure 2 A schematic diagram of the network structure of the MMC multi-task prediction model in an embodiment of the present invention;

[0054] Figure 3 This is a distribution diagram of key points of faces in the 300W_LP dataset in an embodiment of the present invention;

[0055] Figure 4 This is a system structure block diagram in Example 2 of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment 1 provides a fatigue state detection method based on key point detection and head posture, comprising the following steps:

[0059] S1. Build and train the MMC multi-task prediction model whose backbone network adopts a deep separable convolutional network to obtain a trained MMC multi-task prediction model;

[0060] The 300W_LP dataset is used to train the MMC multi-task prediction model. The 300W_LP dataset is widely used for facial feature recognition and head posture analysis. It is a commonly used field 2D landmark dataset. It consists of 61,225 head posture images and is expanded to 122,450 images by flipping. The 300W_LP dataset has facial key point coordinates and head posture angle labels. Before using the 300W_LP dataset to train the MMC multi-task prediction model, the images in the dataset are preprocessed, including:

[0061] Cropping the redundant background part in the image according to the coordinates of the facial key points in the data set is beneficial to improving the training effect. The cropped image size is unified to 224×224, and the unified size image is grayscaled and normalized.

[0062] Specifically, the backbone network adopts the improved lightweight convolutional MobileNet-V2 network structure. The overall network structure is as follows: Figure 2 As shown in the figure, different scale convolutions are used in the first layer of the backbone network to extract image features and fuse them. Different receptive fields are obtained through multiple convolution kernels of different sizes. Specifically, 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels are used to replace the original single 3ⅹ3 convolution kernel to extract features from the input head image, and the convolution stride is set to 1, and the pad corresponding to each convolution kernel is set to 0, 1, and 2, respectively, so that the convolution image can obtain features of the same dimension and can be directly spliced ​​together. Increasing the network width in this way can increase the performance of the network. In order to reduce the amount of calculation, 1ⅹ1 convolution kernels are used for dimensionality reduction processing, which reduces the number of parameters while ensuring the performance of the network, introduces more nonlinearity, and improves the generalization ability. A 5ⅹ5 convolution kernel is used to obtain a larger receptive field. Considering the influence of the relative positions of the eyes, nose, mouth and other parts of the face on the posture angle, obtaining a larger receptive field can better describe this correlation. At the same time, the CA attention module is embedded in the backbone network. The CA attention module can capture accurate location information in space. The CA method decomposes the two-dimensional global pooling operation into two one-dimensional encoding processes, that is, decomposing the global pooling operation into pooling operations along the horizontal and vertical directions of the input feature map, thereby obtaining the position information related to the x and y axes of the input feature map. Two fully connected layers are added after the fully connected layer of the improved MobileNet-V2. The FC1 fully connected layer is used for face key point detection and regression to obtain the coordinates of 68 feature points. The FC2 fully connected layer is used for head posture estimation and regression of posture angles in three directions. The size of the improved lightweight convolutional MobileNet-V2 network structure is 4M. While ensuring the prediction accuracy, the model is smaller and can achieve real-time performance during prediction.

[0063] More specifically, the process of training the MMC multi-task prediction model includes two tasks:

[0064] The facial key point detection task is used to locate the position of facial feature points according to the coordinates of facial key points in the image, using the L2 loss function loss a Measure the difference between the predicted coordinate value and the actual coordinate value of the feature point, and regress to obtain the position information of the key points of the face;

[0065] The head estimation task is used to predict the angles of the head in the image in the three directions of yaw, pitch, and roll based on the head posture angle label in the image. The learning goal is to regress the three angles of yaw, pitch, and roll to describe the position of the head. The loss function is:

[0066]

[0067] in, is the estimated result of the head posture angle in the three directions of yaw, pitch and roll, and (x1,x2,x3) is the head posture angle label in the three directions;

[0068] The total loss of the MMC multi-task prediction model is used as the learning target for training. The total loss is the sum of the losses of the face key point detection task and the head estimation task:

[0069] loss=loss a +ηloss b (2)

[0070] Among them, since facial key point detection and head posture estimation are both regression tasks, the task assignment weight η is set to 1.

[0071] S2, obtain several frames of face images within a unit time, take each frame as an image, use MTCNN network to detect the face position of each image and crop the head image;

[0072] S3, input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face;

[0073] Specifically, the process of obtaining the head posture angle and the position information of the key points of the face is:

[0074] The backbone network of the MMC multi-task prediction model is used to extract and fuse the features of the input head image to obtain a feature map; the backbone network adopts an improved lightweight convolution MobileNet-V2 network structure.

[0075] At the same time, the CA attention module embedded in the backbone network is used to perform pooling operations on the feature map in the horizontal and vertical directions to obtain the position information of the feature map;

[0076] According to the position information of the feature map, the two fully connected layers after the backbone network are used to regress the head posture angle and the position information of the facial key points.

[0077] S4. According to the head posture angle of each image, the double threshold method is used to determine the fatigue state of the head in several images. At the same time, according to the position information of the eyes and mouth in the key points of the face of each image, the double threshold method is used to determine the fatigue state of the eyes and mouth in several images. The double threshold method is used to determine the fatigue state of each part. The unit time is set to 30 seconds, and multiple continuous frames of images within 30 seconds are obtained. Each frame is used as an image. By calculating the proportion of the number of frames in the state of each part to the total number of frames per unit time, it is determined whether each part is in a fatigue state. For example, it can be set that the number of video frames with eyes closed and mouth yawning accounts for more than 40% of the number of frames per unit time, and it can be set that the number of video frames with a pitch posture angle greater than 30° in the case of lowering the head accounts for more than 30% of the number of frames per unit time, and it is determined to be a fatigue state. The specific judgment process is as follows:

[0078] 1. Determine the fatigue state of the head in each image:

[0079] The three Euler angles (pitch, yaw, roll) of the head are directly regressed by the MMC multi-task prediction model. Since the pitch angle changes greatly when a person is in a fatigued state, in order to reduce the amount of calculation, we can focus on the change of the pitch posture angle and set a certain threshold. The pitch posture angle of the detected continuous multi-frame images when the head is lowered is greater than 30°, which is considered to be a fatigued state. Then, according to the head posture angle of each image, it is determined whether the pitch posture angle of the head posture angle is greater than 30°. If it is greater than 30°, the head of the image is judged to be in a fatigued state; if the proportion of images with fatigued heads accounts for more than 30% of all images, the head is judged to be in a fatigued state.

[0080] 2. Determine the eye fatigue status of each image:

[0081] According to the position information of the key points of the eyes in each image, the position coordinates of the key points of the eyes can reflect the degree of opening and closing of the eyes. If the frequency of eye closing is too high within a certain period of time, it is judged as fatigue. The open state of the eyes is judged by calculating the eye aspect ratio EAR, and whether the eye aspect ratio is less than 0.2. If it is less than, the eye of the image is judged to be in a fatigue state; if the proportion of images with fatigued eyes accounts for more than 40% of all images, the eyes are judged to be in a fatigued state; according to Figure 3The facial key point numbers are shown, using the EAR calculation formula as follows:

[0082]

[0083]

[0084]

[0085] Calculate the coordinates of the left eye and the right eye respectively to get the left eye aspect ratio EAR l , right eye aspect ratio EAR r Finally, the eye aspect ratio EAR is comprehensively determined, where the width and height of the eye are calculated using the Euclidean distance formula. When the eyes are closed, the EAR value is 0. The initial threshold is set to 0.2. A value less than this is a closed eye state, and a value greater than this is an open eye state. Different people have different eye opening sizes, so different thresholds can be set according to specific circumstances.

[0086] 3. Determine the fatigue state of the mouth in each image:

[0087] Similarly, based on the position information of the mouth key points of each image, the mouth aspect ratio MAR is used to distinguish the state of the mouth. By detecting the distance between the upper lip and the lower lip and the time of opening the mouth, it is determined whether the current state is yawning. The coordinates of the highest and lowest points of the inner lip and the corners of the mouth are used to calculate the MAR value. Figure 3 The facial key point labels are shown, and the formula for calculating the MAR value is:

[0088]

[0089] Set the MAR threshold to 0.3, and determine whether the aspect ratio of the mouth is greater than 0.3. If so, the mouth of the image is determined to be in a fatigue state; if the proportion of images with fatigued mouths in all images exceeds 40%, the mouth is determined to be in a fatigue state. S5. Comprehensively determine the fatigue state of the person based on the fatigue states of the head, eyes, and mouth.

[0090] Specifically, considering that a person is easily considered to be in a fatigue state when bowing his head for a long time, and that the eyes are often in a nearly closed state when yawning, a correlation coefficient is set for each part to comprehensively determine the fatigue state of the person. According to the weight of the influence of the fatigue state of the head, eyes and mouth on the fatigue state of the person, a correlation coefficient is set for the fatigue state of each part to comprehensively determine the fatigue state Z of the person:

[0091] Z=αZ eye +βZ mouth +λZ head (7)

[0092] Among them, Z eye Each indicates the state of eye fatigue, Z mouth Indicates the fatigue state of the mouth, Z head Indicates the fatigue state of the head, and sets the relevant systems α, β, and λ to 0.2, 0.3, and 0.5 respectively; when Z is greater than or equal to 0.5, it is judged that the person is in a fatigue state.

[0093] Example 2

[0094] like Figure 4 As shown, this embodiment provides a fatigue state detection system based on key point detection and head posture, including:

[0095] The model training module is used to build and train the MMC multi-task prediction model to obtain a trained MMC multi-task prediction model;

[0096] The 300W_LP dataset is used to train the MMC multi-task prediction model. The 300W_LP dataset is widely used for facial feature recognition and head posture analysis. It is a commonly used field 2D landmark dataset. It consists of 61,225 head posture images and is expanded to 122,450 images by flipping. The 300W_LP dataset has facial key point coordinates and head posture angle labels. Before using the 300W_LP dataset to train the MMC multi-task prediction model, the images in the dataset are preprocessed, including:

[0097] Cropping the redundant background part in the image according to the coordinates of the facial key points in the data set is beneficial to improving the training effect. The cropped image size is unified to 224×224, and the unified size image is grayscaled and normalized.

[0098] The MMC multi-task prediction model includes a backbone network and two fully connected layers for regressing the head posture angle and the position information of the facial key points respectively. The FC1 fully connected layer is used for facial key point detection and regresses the coordinates of 68 feature points. The FC2 fully connected layer is used for head posture estimation and regresses the posture angles in three directions. The backbone network adopts the improved lightweight convolution MobileNet-V2 network structure. In the first layer of the backbone network, different scale convolutions are used to extract image features and fuse them. Different receptive fields are obtained through multiple convolution kernels of different sizes. Specifically, 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels are used to replace the original single 3ⅹ3 convolution kernel to extract features from the input head image, and the convolution stride is set to 1, and the pad corresponding to each convolution kernel is set to 0, 1, and 2, respectively, so that the convolution image can obtain features of the same dimension and can be directly spliced ​​together. Increasing the network width in this way can increase the performance of the network. In order to reduce the amount of calculation, 1ⅹ1 convolution kernels are used for dimensionality reduction processing, which reduces the number of parameters while ensuring the performance of the network, introduces more nonlinearity, improves the generalization ability, and uses 5ⅹ5 convolution kernels to obtain a larger receptive field. Considering the influence of the relative positions of the eyes, nose, mouth and other parts of the face on the posture angle, obtaining a larger receptive field can better describe this correlation. At the same time, the CA attention module is embedded in the backbone network. The CA attention module can capture accurate location information in space. The CA method decomposes the two-dimensional global pooling operation into two one-dimensional encoding processes, that is, decomposing the global pooling operation into pooling operations along the horizontal and vertical directions of the input feature map, thereby obtaining the position information related to the x and y axes of the input feature map. The size of the improved lightweight convolution MobileNet-V2 network structure is 4M. While ensuring the prediction accuracy, the model is smaller and can achieve real-time prediction.

[0099] The face position detection module is used to detect the face position of each image using the MTCNN network and crop the head image based on the acquired face image frames within a unit time, taking each frame as an image;

[0100] The parallel prediction module is used to input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face;

[0101] A local state detection module is used to determine the fatigue state of the head in several images using a double threshold method based on the head posture angle of each image, and to determine the fatigue state of the eyes and mouth in several images using a double threshold method based on the position information of the eyes and mouth in the key points of the face of each image;

[0102] The double threshold method is used to determine the fatigue state of each part. The unit time is set to 30 seconds, and multiple continuous frames of images within 30 seconds are obtained. Each frame is taken as an image. Whether each part is in a fatigue state is determined by calculating the proportion of the number of frames of each part's state to the total number of frames per unit time. For example, it can be set that the number of video frames with eyes closed and mouth yawning accounts for more than 40% of the number of frames per unit time to be judged as a fatigue state; and the number of video frames with a pitch posture angle greater than 30° in the case of lowering the head accounts for more than 30% of the number of frames per unit time to be judged as a fatigue state.

[0103] The comprehensive fatigue state detection module is used to comprehensively determine the fatigue state of a person based on the fatigue states of the head, eyes and mouth. Considering that a person is easily considered to be in a fatigue state when he bows his head for a long time, and the eyes are often in a nearly closed state when yawning, a correlation coefficient is set for each part to comprehensively determine the fatigue state of a person. According to the weight of the fatigue state of the head, eyes and mouth on the fatigue state of a person, similarly, a correlation coefficient is set for the fatigue state of each part to comprehensively determine the fatigue state Z of a person using formula (7).

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] A person of ordinary skill in the art can understand that all or part of the steps in realizing the above-mentioned facts and methods can be completed by instructing the relevant hardware through a program, and the program involved or the program described can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: At this time, the corresponding method steps are derived, and the storage medium can be ROM / RAM, a disk, an optical disk, etc.

[0109] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fatigue state detection method based on key point detection and head posture, characterized in that: The following steps are involved: Construct and train the MMC multi-task prediction model whose backbone network adopts a deep separable convolutional network to obtain a trained MMC multi-task prediction model; Get several frames of face images within a unit time, treat each frame as an image, use the MTCNN network to detect the face position of each image and crop the head image; Input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face; According to the head posture angle of each image, the double threshold method is used to determine the fatigue state of the head in several images. At the same time, according to the position information of the eyes and mouth in the facial key points of each image, the double threshold method is used to determine the fatigue state of the eyes and mouth in several images respectively. Comprehensively judge a person's fatigue status based on the fatigue status of the head, eyes and mouth; The process of obtaining the head posture angle and the position information of the key points of the face is as follows: The backbone network of the MMC multi-task prediction model is used to extract and fuse the features of the input head image to obtain a feature map; the backbone network adopts an improved lightweight convolution MobileNet-V2 network structure. At the same time, the CA attention module embedded in the backbone network is used to perform pooling operations on the feature map in the horizontal and vertical directions to obtain the position information of the feature map; According to the position information of the feature map, the two fully connected layers after the backbone network are used to regress the head posture angle and the position information of the key points of the face; Among them, the improved lightweight convolution MobileNet-V2 network structure uses 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels to extract features of the input head image, sets the convolution stride to 1, and sets the pad corresponding to each convolution kernel to 0, 1, and 2 respectively. The size of the improved lightweight convolution MobileNet-V2 network structure is 4M.

2. The fatigue state detection method based on key point detection and head posture according to claim 1, characterized in that: The 300W_LP dataset is used to train the MMC multi-task prediction model. The 300W_LP dataset has the facial key point coordinates and head posture angle labels. Before using the 300W_LP dataset to train the MMC multi-task prediction model, the images in the dataset are preprocessed, including: The redundant background part in the image is cropped according to the coordinates of the facial key points in the data set, the cropped image size is unified to 224×224, and the unified size image is grayscaled and normalized.

3. The fatigue state detection method based on key point detection and head posture according to claim 2, characterized in that: The process of training the MMC multi-task prediction model consists of two tasks: The facial key point detection task is used to locate the position of facial feature points according to the coordinates of facial key points in the image, using the L2 loss function loss a Measure the difference between the predicted coordinate value and the actual coordinate value of the feature point, and regress to obtain the position information of the key points of the face. The head estimation task is used to predict the angles of the head in the image in the three directions of yaw, pitch, and roll based on the head posture angle label in the image. The loss function is: in, is the estimated result of the head posture angle in the three directions of yaw, pitch and roll, and (x1,x2,x3) is the head posture angle label in the three directions; The total loss of the MMC multi-task prediction model is used as the learning target for training. The total loss is the sum of the losses of the face key point detection task and the head estimation task: loss=loss a +ηloss b Among them, η is the task assignment weight set to 1.

4. The fatigue state detection method based on key point detection and head posture according to claim 1, characterized in that: For several frames of continuous face images acquired within a unit time, the process of determining the fatigue state of each part using the double threshold method is as follows: Determine the fatigue state of the head in each image: According to the head posture angle of each image, it is determined whether the pitch posture angle of the head posture angle is greater than 30°. If it is greater than 30°, it is determined that the head in the image is in a fatigue state; if the proportion of images in which the head is in a fatigue state accounts for more than 30% of all images, it is determined that the head is in a fatigue state; Determine the eye fatigue status of each image: According to the position information of the eye key points of each image, the eye aspect ratio is calculated to determine whether the eye aspect ratio is less than 0.

2. If it is less than 0.2, the eye of the image is judged to be in a fatigue state; if the proportion of images in which the eye is in a fatigue state accounts for more than 40% of all images, the eye is judged to be in a fatigue state; Determine the fatigue state of the mouth in each image: According to the position information of the mouth key points of each image, the mouth aspect ratio is calculated to determine whether the mouth aspect ratio is less than 0.

3. If it is less than 0.3, the mouth of the image is judged to be in a fatigued state; if the proportion of images with fatigued mouths in all images exceeds 40%, the mouth is judged to be in a fatigued state.

5. The fatigue state detection method based on key point detection and head posture according to claim 1, characterized in that: According to the weight of the fatigue status of the head, eyes and mouth on the fatigue status of the person, a correlation coefficient is set for the fatigue status of each part to comprehensively determine the fatigue status Z of the person: Z=αZ eye +βZ mouth +λZ head Among them, Z eye Each indicates the state of eye fatigue, Z mouth Indicates the fatigue state of the mouth, Z head To represent the fatigue state of the head, the related systems α, β, and λ are set to 0.2, 0.3, and 0.5, respectively; When Z is greater than or equal to 0.5, it is judged that the person is in a fatigue state.

6. A fatigue state detection system based on key point detection and head posture, characterized in that: The method according to any one of claims 1 to 5 comprises: The model training module is used to build and train the MMC multi-task prediction model to obtain a trained MMC multi-task prediction model; The face position detection module is used to detect the face position of each image using the MTCNN network and crop the head image based on the acquired face image frames within a unit time, taking each frame as an image; The parallel prediction module is used to input the head image into the trained MMC multi-task prediction model to obtain the head posture angle and the position information of the key points of the face; A local state detection module is used to determine the fatigue state of the head in several images using a double threshold method based on the head posture angle of each image, and to determine the fatigue state of the eyes and mouth in several images using a double threshold method based on the position information of the eyes and mouth in the key points of the face of each image; The comprehensive fatigue state detection module is used to comprehensively determine a person's fatigue state based on the fatigue states of the head, eyes and mouth.

7. A fatigue state detection system based on key point detection and head posture according to claim 6, characterized in that: The MMC multi-task prediction model includes a backbone network and two fully connected layers for regressing the head posture angle and the position information of the key points of the face. The backbone network adopts an improved lightweight convolutional MobileNet-V2 network structure, and a CA attention module is embedded in the backbone network. The backbone network is used to extract and fuse the features of the input head image to obtain the feature map. The CA attention module is used to perform pooling operations on the feature map in the horizontal and vertical directions respectively to obtain the position information of the feature map.

8. The fatigue state detection system based on key point detection and head posture according to claim 7, characterized in that: The improved lightweight convolution MobileNet-V2 network structure uses 1ⅹ1, 3ⅹ3 and 5ⅹ5 convolution kernels to extract features of the input head image, sets the convolution stride to 1, and sets the pad corresponding to each convolution kernel to 0, 1, and 2 respectively. The size of the improved lightweight convolution MobileNet-V2 network structure is 4M.

Images