Fatigue detection method and device, electronic equipment and storage medium

Abstract

The invention provides a fatigue detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a face image, carrying out the eye key point detection of an eye region of the face image, obtaining the key point position information of the eye region, and determining the eye opening degree based on the key point position information; performing head posture estimation on the face image to obtain head posture information of the face image, and correcting the eye opening degree based on the head posture information to obtain a corrected opening degree; and determining a fatigue detection result of the face image based on the correction opening degree. According to the method and device, the electronic equipment, the storage medium and the application of the head posture information in eye opening degree correction, the influence of head posture changes on the eye opening degree is eliminated, the reliability of the eye opening degree obtained based on key point detection can be effectively improved, and then the reliability and accuracy of fatigue detection are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a fatigue detection method, device, electronic equipment and storage medium. Background Art

[0002] Driver fatigue detection technology mainly relies on machine vision technology, which assesses the driver's fatigue level by capturing and analyzing the physiological reaction characteristics of the driver's face, such as the opening and closing status of the eyes.

[0003] For example, key point detection can be performed on eye images to calculate the degree of eye openness, and then fatigue detection can be performed based on the degree of eye openness.

[0004] However, the real car scene is very complex, and the driver's movements in the car will directly affect the calculation of eye opening degree, which leads to low reliability and accuracy of fatigue detection based on eye opening degree. Summary of the invention

[0005] The present invention provides a fatigue detection method, device, electronic device and storage medium, which are used to solve the defects of low reliability and accuracy of fatigue detection based on eye opening degree in related technologies.

[0006] The present invention provides a fatigue detection method, which comprises the following steps.

[0007] Acquire a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; Performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree; Based on the corrected opening and closing degree, a fatigue detection result of the face image is determined.

[0008] According to a fatigue detection method provided by the present invention, the eye opening degree is corrected based on the head posture information to obtain the corrected eye opening degree, including: Based on the mapping function and the head posture information, the eye opening degree is corrected to obtain the corrected eye opening degree; The mapping function is determined based on the eye opening degree of the sample person in the sample image taken under the same sample eye opening degree and different sample head posture information.

[0009] According to a fatigue detection method provided by the present invention, determining the eye opening degree based on the key point position information includes: Based on the key point position information, calculating the initial opening and closing degree; Based on the eye opening degree distribution information of the person corresponding to the face image, the initial opening degree is individually standardized to obtain the eye opening degree.

[0010] According to a fatigue detection method provided by the present invention, the initial eye opening degree is individually standardized based on the eye opening degree distribution information of the person corresponding to the face image to obtain the eye opening degree, including: Based on the eye opening degree distribution information of the person corresponding to the face image, determining the maximum individual opening degree and the minimum individual opening degree of the person corresponding to the face image; Based on the individual maximum opening degree and the individual minimum opening degree, the initial opening degree is individually standardized to obtain the eye opening degree.

[0011] According to a fatigue detection method provided by the present invention, the eye opening degree distribution information is obtained based on a driving face image of a person corresponding to the face image in an early stage of driving.

[0012] According to a fatigue detection method provided by the present invention, the fatigue detection result of the face image is determined based on the corrected opening and closing degree, including: Performing open-eye and closed-eye classification on the eye region of the face image to obtain an open-eye and closed-eye classification result; Based on the corrected opening and closing degree and the open and closed eyes classification result, a fatigue detection result of the face image is determined.

[0013] According to a fatigue detection method provided by the present invention, the eye-opening and eye-closing classification is performed based on a classification model, and the eye key point detection is performed based on an eye key point detection model; The classification model and the eye key point detection model share a backbone network.

[0014] According to a fatigue detection method provided by the present invention, the eye area of ​​the face image is obtained by cropping the face image based on a key point detection result of the face image.

[0015] The present invention also provides a fatigue detection device, comprising the following modules: A computing unit, configured to acquire a face image, perform eye key point detection on an eye region of the face image, obtain key point position information of the eye region, and determine an eye opening degree based on the key point position information; a correction unit, configured to estimate the head posture of the face image to obtain head posture information of the face image, and correct the eye opening degree based on the head posture information to obtain a corrected eye opening degree; A detection unit is used to determine a fatigue detection result of the face image based on the corrected opening and closing degree.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the fatigue detection methods described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the fatigue detection method described in any one of the above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the fatigue detection method described above is implemented.

[0019] The fatigue detection method, device, electronic device and storage medium provided by the present invention correct the eye opening and closing degree obtained based on the eye key point detection based on the head posture information of the face image, thereby obtaining the corrected opening and closing degree and using it for fatigue detection. The application of head posture information in the correction of eye opening and closing degree eliminates the influence of head posture changes on eye opening and closing degree, and can effectively improve the reliability of the eye opening and closing degree obtained based on the key point detection, thereby ensuring the reliability and accuracy of fatigue detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is one of the flow charts of the fatigue detection method provided by the present invention.

[0022] Figure 2 It is a schematic diagram of the key points of the eye provided by the present invention.

[0023] Figure 3 It is a schematic diagram of eye opening and closing of different persons provided by the present invention.

[0024] Figure 4 It is a schematic diagram of key points of a face provided by the present invention.

[0025] Figure 5 This is the second flow chart of the fatigue detection method provided by the present invention.

[0026] Figure 6 It is a structural schematic diagram of the fatigue detection device provided by the present invention.

[0027] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] With the continuous development of science and technology, smart cockpits have gradually become a key component of modern automotive technology. Driver fatigue detection, as an important means to improve driving safety in the field of smart cockpits, has received widespread attention. The principles of driver fatigue detection mainly include detection based on the driver's own characteristics and detection methods based on the vehicle's driving status.

[0030] Among them, the detection method based on the driver's own characteristics can be divided into a contact detection method based on the driver's physiological indicators and a non-contact detection method based on the driver's physiological reaction characteristics.

[0031] The driver's physiological indicators may include brain, electricity, heart, electricity, pulse, breathing, etc. The driver's physiological indicators in a fatigued state will deviate from the normal state, so physiological sensors can be used to detect the driver's physiological indicators to determine whether the driver is in a fatigued state. However, since the physiological indicators of different people are different, and these physiological indicators are closely related to psychological activities, the detection method based on physiological indicators faces the problem of individual differences and has great limitations. In addition, the stability of physiological sensors during driving needs to be improved. Therefore, the detection method based on the driver's physiological indicators has not been widely used.

[0032] The physiological reaction characteristics of the driver include eye characteristics, mouth state, head position, etc. Machine vision technology can be used to detect the physiological reaction characteristics of the driver's face and then judge the driver's fatigue state. The advantage of this detection method is that the physiological reaction characteristics that characterize fatigue are intuitive and obvious, and non-contact measurement can be achieved. It is now the mainstream detection method in the industry. However, the disadvantage of this method is that the detection and recognition algorithm is complex, fatigue feature extraction is difficult, and the detection results are affected by light changes, individual differences, head posture angles, and camera positions, which requires high technology.

[0033] At present, in the implementation of detection methods based on physiological response characteristics, single-frame eye closure detection methods are often used, combined with post-processing strategies to perform fatigue detection. Here, the single-frame eye closure detection method can be to perform key point detection on a single-frame eye image to calculate the eye opening degree, and then perform fatigue detection based on the eye opening degree.

[0034] However, the complexity of real car scenes is very high. Changes in the driver's head posture in the car will directly affect the calculation of the eye opening degree. For example, under the same eye opening degree, as the head moves left and right, the yaw angle of the head becomes larger and larger. Due to the shortening of the major axis distance, the value of the eye opening degree obtained based on the key point detection becomes larger and larger; and under the same eye opening degree, as the head moves up and down, the pitch angle of the head becomes larger and larger. Due to the shortening of the minor axis distance, the value of the eye opening degree obtained based on the key point detection becomes smaller and smaller, which leads to low reliability and accuracy of fatigue detection based on eye opening degree.

[0035] In view of the above problems, the present invention provides a fatigue detection method. Figure 1 is one of the flow charts of the fatigue detection method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, acquiring a face image, performing eye key point detection on the eye region of the face image, obtaining key point position information of the eye region, and determining the eye opening degree based on the key point position information.

[0036] Here, the face image is an image containing a face taken for a person who needs to undergo fatigue detection. Here, the person who needs to undergo fatigue detection may be a driver or other person, and the embodiment of the present invention does not specifically limit this.

[0037] After acquiring the face image, the eye key point detection can be performed on the eye area of ​​the face image. Here, the eye area of ​​the face image is the area image where the eyes are located in the face image. The eye key point detection can be implemented by a pre-trained face key point detection model or a pre-trained eye key point detection model, thereby obtaining the key point position information of the eye area. For example, 6 key points can be set in the eye area, namely Figure 2 P 1 To P 6 There are 6 key points in total, and the key point position information of the eye area is Figure 2 The coordinate positions of the six key points in the eye area are shown.

[0038] After completing the eye key point detection, the eye opening degree in the face image can be calculated based on the key point position information of the eye area detected. Here, the eye opening degree refers to the degree of opening and closing of the eyelids, that is, the aspect ratio of the gap between the eyelids, which is an important indicator to measure the open or closed state of the eyes.

[0039] For example, in Figure 2 In the case of the key points shown, the eye opening degree can be calculated by the following formula:

[0040] In the formula, For the eye opening, to for Figure 2 The coordinates of each key point in .

[0041] Step 120, performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree.

[0042] Specifically, after obtaining the face image, head pose estimation can also be performed on the face image. Here, head pose estimation is used to estimate the head pose of the person in the face image, and the head pose information obtained can be expressed by pitch angle (Pitch), yaw angle (Yaw), etc. The pitch angle is used to describe the angle of the head tilting up and down, that is, the angle of raising or lowering the head, and the yaw angle is used to describe the angle of the head rotating left and right, that is, the angle of turning left or right.

[0043] Taking into account that changes in head posture will directly affect the calculation of eye opening, for example, in the case of the same eye opening, as the head moves left and right, the yaw angle yaw of the head becomes larger and larger. Due to the shortening of the major axis distance, the value of the eye opening obtained based on the key point detection becomes larger and larger; and in the case of the same eye opening, as the head moves up and down, the pitch angle pitch of the head becomes larger and larger. Due to the shortening of the minor axis distance, the value of the eye opening obtained based on the key point detection becomes smaller and smaller. Therefore, after obtaining the eye opening calculated based on the key point position information, the eye opening can be corrected based on the head posture information of the face image, thereby obtaining the eye opening that is not affected by the head posture information after correction of the head posture information, which is recorded here as the corrected opening.

[0044] Here, based on the head posture information, the eye opening is corrected. Specifically, the eye opening may be corrected based on the relationship between the pre-set head posture information, the eye opening obtained based on the key point detection and the actual eye opening. The relationship referred to here may be a relationship obtained by theoretical calculation, or may be a relationship obtained by fitting the actual eye opening under various head posture information collected historically and the eye opening based on the key point detection. The embodiment of the present invention does not make any specific limitation to this.

[0045] It can be understood that the corrected eye opening obtained by correcting the eye opening based on the head posture information eliminates the influence of head posture changes on the eye opening, and can effectively improve the reliability of the eye opening obtained based on key point detection.

[0046] Step 130: Determine a fatigue detection result of the face image based on the corrected opening and closing degree.

[0047] Specifically, after obtaining the corrected opening and closing degree, fatigue detection can be performed on the face image based on the corrected opening and closing degree. For example, the corrected opening and closing degree can be compared with a preset opening and closing degree threshold, and the comparison result can be used to determine whether the person in the face image is in a fatigued state.

[0048] In the method provided in an embodiment of the present invention, the eye openness obtained based on the eye key point detection is corrected based on the head posture information of the face image, so as to obtain the corrected openness and use it for fatigue detection. The application of head posture information in the correction of eye openness eliminates the influence of head posture changes on eye openness, and can effectively improve the reliability of the eye openness obtained based on the key point detection, thereby ensuring the reliability and accuracy of fatigue detection.

[0049] Based on the above embodiment, in step 120, the eye opening degree is corrected based on the head posture information to obtain the corrected eye opening degree, including: Based on the mapping function and the head posture information, the eye opening degree is corrected to obtain the corrected eye opening degree; The mapping function is determined based on the eye opening degree of the sample person in the sample image taken under the same sample eye opening degree and different sample head posture information.

[0050] Specifically, in order to achieve correction for the eye opening, a mapping function may be pre-fitted. The mapping function here may reflect the mapping relationship between the head posture information, the eye opening obtained based on key point detection, and the corrected eye opening.

[0051] In order to determine the mapping function, the eye openings of the sample persons in the sample images taken under the same sample eye openings and different sample head pose information can be summarized in advance. Among them, the sample persons are the persons participating in the test, the sample eye openings refer to the real eye openings of the sample persons, and the sample head pose information refers to the head pose information of the sample persons. The eye openings in the sample images thus taken are the eye openings obtained by performing eye key point detection on the sample images. That is, for the sample persons, sample images of the sample persons under the same sample eye openings and different sample head pose information can be collected, and eye key point detection can be performed on the sample images, and then the eye openings in the sample images can be calculated, and the sample head pose information and the eye openings in the sample images are used as the head pose information and the eye openings obtained based on the key point detection, and the sample eye openings are used as the corrected openings, so as to fit the mapping relationship among the head pose information, the eye openings obtained based on the key point detection, and the corrected openings, and then the mapping function can be obtained.

[0052] For example, sample images of 20 sample persons with the same sample eye opening and closing degrees and different sample head pose information can be collected, and the eye opening and closing degrees in the sample images can be obtained by detecting eye key points, and the sample head pose information in the sample images can be obtained by head pose estimation. Thus, a mapping function that reflects the mapping relationship among the head pose information, the eye opening and closing degrees obtained based on the key point detection, and the corrected opening and closing degrees can be fitted by a linear function, which can be specifically expressed as the following formula:

[0053] In the formula, is the eye opening degree obtained based on key point detection, To correct the opening and closing, and are the yaw angle and pitch angle in the head posture information, To avoid the minimum value with denominator equal to 0, and is the hyperparameter of the mapping function, Functions are used to limit The value is between 0 and 1.

[0054] The method provided by the embodiment of the present invention fits a mapping function through the eye opening degrees of sample persons in sample images taken under the same sample eye opening degrees and different sample head posture information, and then eliminates the influence of head posture information on the eye opening degrees based on the mapping function, which can effectively improve the reliability of the eye opening degrees obtained based on key point detection, thereby ensuring the reliability and accuracy of fatigue detection.

[0055] Based on any of the above embodiments, in step 110, determining the eye opening degree based on the key point position information includes: Based on the key point position information, calculating the initial opening and closing degree; Based on the eye opening degree distribution information of the person corresponding to the face image, the initial opening degree is individually standardized to obtain the eye opening degree.

[0056] Specifically, after obtaining the key point position information of the eye area, the eye opening degree can be calculated based on the key point position information. The eye opening degree directly calculated here is recorded as the initial opening degree. It can be understood that the initial opening degree obtained here is the eye opening degree calculated completely from the information displayed by the face image itself, and does not take into account the personalized differences in the eye opening degrees of different people.

[0057] In fact, the degree of eye opening and closing varies greatly among different people. Figure 3 In the figure, the eye opening degree of the person in the upper half of the picture is obviously greater than that of the person in the lower half of the picture. It is understandable that the difference in eye opening degree of different individuals makes it difficult to implement the solution of fatigue detection using a unified threshold.

[0058] In order to alleviate the problem of large differences in eye openness among different individuals, after obtaining the initial openness, the initial openness can be individually standardized based on the eye openness distribution information of the person corresponding to the face image, thereby obtaining the openness after individual standardization, that is, the eye openness.

[0059] Here, the person corresponding to the facial image, that is, the person corresponding to the initial eye opening and closing degree, and the eye opening and closing degree distribution information of the person can reflect the distribution of the eye opening and closing degrees when the person is in a normal state over a period of time, that is, it can reflect the normal range of the eye opening and closing degrees of the person.

[0060] Based on the eye opening degree distribution information of the person, the process of individual standardization of the initial opening degree can be understood as mapping the initial opening degree to the normal range of the eye opening degree represented by the eye opening degree distribution information, and then standardizing the initial opening degree itself based on the position of the initial opening degree in the normal range. The eye opening degree obtained in this way eliminates the influence of individual differences in the eye opening degrees of different persons and is an opening degree value that can objectively reflect the eye opening degree of the person in the face image.

[0061] In the method provided in an embodiment of the present invention, the initial eye opening degree is individually standardized by applying the eye opening degree distribution information of the person corresponding to the facial image, thereby obtaining the eye opening degree that eliminates the influence of individual differences in the eye opening degrees of different persons. This can effectively improve the reliability of the eye opening degree obtained based on key point detection, thereby ensuring the reliability and accuracy of fatigue detection.

[0062] Based on any of the above embodiments, in step 110, the initial eye opening degree is individually standardized based on the eye opening degree distribution information of the person corresponding to the face image to obtain the eye opening degree, including: Based on the eye opening degree distribution information of the person corresponding to the face image, determining the maximum individual opening degree and the minimum individual opening degree of the person corresponding to the face image; Based on the individual maximum opening degree and the individual minimum opening degree, the initial opening degree is individually standardized to obtain the eye opening degree.

[0063] Specifically, after obtaining the eye opening degree distribution information of the person corresponding to the face image, the individual maximum opening degree and the individual minimum opening degree of the person corresponding to the face image can be determined therefrom.

[0064] Among them, the individual maximum opening degree can be understood as the maximum eye opening degree in the eye opening degree distribution information, and can also be understood as the maximum eye opening degree of the person corresponding to the face image in a normal state. The individual maximum opening degree can be the maximum eye opening degree obtained from the eye opening degree distribution information, or it can be the value obtained by statistically analyzing the top 5%-top 20% of the values ​​after removing the top 5% of the data in the eye opening degree distribution information in order to screen out abnormal situations such as staring. The embodiment of the present invention does not make any specific limitation on this.

[0065] The maximum individual eye opening degree can be understood as the minimum eye opening degree in the eye opening degree distribution information, and can also be understood as the minimum eye opening degree of the person corresponding to the face image in a normal state. The minimum individual opening degree can be the minimum eye opening degree obtained from the eye opening degree distribution information.

[0066] After obtaining the maximum individual opening and closing degree and the minimum individual opening and closing degree of the person corresponding to the face image, the initial opening and closing degree can be individually standardized by combining the maximum individual opening and closing degree and the minimum individual opening and closing degree. The individual standardization here can be achieved by the following formula:

[0067] In the formula, is the initial opening degree, is the eye opening degree obtained through individual standardization. and are the maximum and minimum individual opening and closing degrees, Functions are used to limit The value is between 0 and 1.

[0068] Based on any of the above embodiments, the eye opening degree distribution information is obtained based on a driving face image of the person corresponding to the face image in the early stage of driving.

[0069] Specifically, in a driving scenario, for the case where the person to be fatigued is a driver, in order to obtain the distribution of eye openness when the person corresponding to the facial image is in a normal state, that is, to obtain eye openness distribution information, a facial image of the person corresponding to the facial image at the initial stage of driving, that is, the driving facial image, can be collected, and then the eye openness in the driver's image can be detected to obtain statistical information on the eye openness distribution.

[0070] It is understandable that in the initial stage of driving, such as within the first minute of starting the car, the driver is usually in a normal state, during which the driver's eye opening degree is also normal for the individual driver. The eye opening degree obtained based on the driver's facial image collected during this period can reflect the distribution of the eye opening degree when the driver is in a normal state. Based on this statistical eye opening degree distribution information, it can provide conditions for individual standardization of eye opening degree in subsequent fatigue detection, thereby eliminating the influence of individual differences in eye opening degree of different people and ensuring the reliability of fatigue detection.

[0071] Furthermore, in order to prevent the statistics of eye opening degree distribution information from being affected by head posture, after obtaining the driver's face image at the initial stage of driving, head posture detection can be performed on each driver's face image, thereby obtaining the head posture information of each driver image, and then removing the driver's image whose head posture information indicates an extreme change in head posture, that is, removing the large head posture data. It can be understood that the large head posture data referred to here is the driver's image whose head posture information indicates that the head is tilted or rotated significantly. By removing the large head posture data, it can be ensured that the remaining driver's face images are images that are less affected by head posture, and the eye opening degree distribution information obtained by statistics can also more accurately reflect the distribution of the eye opening degree when the driver is in a normal state.

[0072] Based on any of the above embodiments, in step 130, determining the fatigue detection result of the face image based on the corrected opening and closing degree includes: Performing open-eye and closed-eye classification on the eye region of the face image to obtain an open-eye and closed-eye classification result; Based on the corrected opening and closing degree and the open and closed eyes classification result, a fatigue detection result of the face image is determined.

[0073] Specifically, the complexity of the real car scene itself, that is, different lighting, accessories, weather, etc., will bring uncertainty to the key point detection based on the model, and the training data for the model is difficult to cover all scenes, which leads to the weak generalization of the model for abnormal scenes. Once the key point prediction is inaccurate, it will directly affect the reliability of fatigue detection based on key point detection.

[0074] To address this problem, in an embodiment of the present invention, in addition to applying the corrected opening and closing degree obtained based on key point detection to fatigue detection, open and closed eyes classification is also performed on the eye area of ​​the face image, and the open and closed eyes classification result is used together with the corrected opening and closing degree for fatigue detection of the face image.

[0075] Here, the eye area of ​​the face image is classified as open or closed eyes. That is, the face image or the eye area of ​​the face image can be input into a pre-trained eye-open or closed eyes classification model, and the eye-open or closed state is classified by the eye-open or closed eyes classification model to obtain the eye-open or closed eyes classification result.

[0076] After obtaining the eye-opening and eye-closing classification results, fatigue detection can be performed by combining the eye-opening and eye-closing classification results and the corrected opening and closing degree to obtain fatigue detection results of face images. For example, the eye-opening and eye-closing classification results and the corrected opening and closing degree can be weighted fused, which can be specifically expressed as the following formula:

[0077] In the formula, To correct the opening and closing, is the probability value of the closed eyes state in the open and closed eyes classification results, is the weight, The score obtained by weighted fusion of the open and closed eyes classification result and the corrected opening and closing degree can be used to determine whether the fatigue detection result is fatigue or not based on the score.

[0078] In the method provided in an embodiment of the present invention, a classification scheme based on eye opening and closing is combined with a detection scheme based on key points. By introducing multi-cue information including classification and key point estimation, the generalization problem of a single model is alleviated, which helps to further improve the reliability and accuracy of fatigue detection.

[0079] Based on any of the above embodiments, the open and closed eyes classification is performed based on a classification model, and the eye key point detection is performed based on an eye key point detection model; The classification model and the eye key point detection model share a backbone network.

[0080] Specifically, the eye-opening and eye-closing classification and the eye key point detection can be realized by their respective corresponding models. In addition, considering the realization efficiency of fatigue detection and the complementarity of the two tasks of eye-opening and eye-closing classification and eye key point detection, in the embodiment of the present invention, a multi-task approach can be adopted for modeling, that is, a backbone network is shared, and a classification layer is connected after the backbone network to realize the eye-opening and eye-closing classification task, and a key point detection layer is connected to realize the eye key point detection task.

[0081] It can be understood that the backbone network referred to here is used to extract features from the input eye area. Subsequently, the features extracted based on the backbone network can be input into the classification layer for open and closed eyes classification, and also input into the key point detection layer for eye key point detection, that is, the two tasks of open and closed eyes classification and eye key point detection can share the features output by the backbone network.

[0082] In some implementations, mobilenet may be used as the backbone network, the key point detection layer may be implemented using residual log-likelihood estimation (RLE), and the classification layer may be implemented using a full connection (FC) layer.

[0083] Based on any of the above embodiments, the eye area of ​​the facial image is obtained by cropping the facial image based on a key point detection result of the facial image.

[0084] Specifically, before performing eye key point detection on the eye region of the face image, key point detection may be performed on the face image to obtain a key point detection result for the face image. Figure 4 is a schematic diagram of key points of a face provided by the present invention, such as Figure 4 As shown, a human face can contain 68 key points, of which the left and right eyes can contain 6 key points respectively.

[0085] After obtaining the key point detection results for the face image, the position information of the key points of the left and right eyes can be determined therefrom, and based on this, the face image can be cropped to obtain the eye area of ​​the face image. For example, for the right eye, the positions of key points 37-42 can be determined from the key point detection results for the face image, and the coordinate mean of the above key points can be calculated as the center point of the right eye. In addition, according to the positions of key points 37 and 40, the long axis distance of the right eye is calculated, and 1.5 times the long axis distance is used as the side length, and the center point is used as the center. A square containing the eye area is cropped from the face image and scaled to 128*128 as the eye area of ​​the right eye. The left eye can be processed in the same way as above.

[0086] In addition, although the key point detection results of the face image have been applied when cropping the face image, and the key point detection results of the face image also include the positions of the eye key points, since the input of the key point detection is the entire face image, the eye resolution is relatively low. In an embodiment of the present invention, eye key point detection can be performed on the eye area to obtain more precise key point position information of the eye area, thereby improving the reliability and accuracy of fatigue detection.

[0087] Based on any of the above embodiments, Figure 5 FIG. 2 is a flow chart of the fatigue detection method provided by the present invention. Figure 5 As shown, the method includes: First, obtain the face image.

[0088] For example, in a cockpit scenario, real vehicle data can be collected through a cockpit camera, such as a driver monitoring system (DMS) camera, and the driver's facial image can be obtained from the real vehicle data through a face detection model.

[0089] Secondly, the eye area in the face image can be located through the facial key point model, and then the eye area can be captured on the face image, and the eye area can be used as the input of the subsequent eye key point model and the open and closed eyes classification model.

[0090] At the same time, the head pose information of the face image can be obtained through the head pose estimation model, which can be specifically reflected in the pitch angle pitch and yaw angle yaw.

[0091] Subsequently, after the eye area is acquired, the eye area can be input into the eye key point model to realize the eye key point detection, thereby obtaining the key point position information of the eye area. Next, the eye opening degree is calculated based on the key point position information of the eye area, and the calculated opening degree is used as the initial opening degree. In addition, considering the differences in eye opening degree among different individuals, the calculated initial opening degree can be individually standardized based on the individual maximum opening degree and the individual minimum opening degree of the main driver, thereby obtaining the eye opening degree. In addition, considering the influence of different head postures on the eye opening degree, the eye opening degree is corrected based on the head posture information of the facial image calculated above, and the corrected opening degree after the head posture adaptive correction, that is, the regularized opening degree score, can be obtained.

[0092] In addition, the eye region can be input into the classification model to achieve open and closed eyes classification, thereby obtaining the open and closed eyes classification result, that is, the open and closed eyes score.

[0093] Finally, the eye opening and closing scores and the regularized opening and closing scores can be fused, and the fused results can be used as fatigue detection results.

[0094] The method provided by the embodiment of the present invention introduces multi-clue information including key points, head posture information, eye opening distribution information, etc., thereby alleviating the problem of poor generalization of fatigue detection using a classification model alone; at the same time, individual standardization and an adaptive algorithm based on head posture are used to optimize the problem that the opening degrees of different samples are very different, and the opening degrees of the same sample are greatly affected by head posture. The method provided by the embodiment of the present invention has a strong migration capability for complex scenes, extreme samples such as squinting, scenes with different camera installation positions, and the like.

[0095] The fatigue detection device provided by the present invention is described below. The fatigue detection device described below and the fatigue detection method described above can be referenced to each other.

[0096] Figure 6 is a schematic diagram of the structure of the fatigue detection device provided by the present invention, such as Figure 6 As shown, the device comprises: The calculation unit 610 is used to obtain a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; The correction unit 620 is used to estimate the head posture of the face image to obtain the head posture information of the face image, and correct the eye opening degree based on the head posture information to obtain the corrected eye opening degree; The detection unit 630 is used to determine a fatigue detection result of the face image based on the corrected opening and closing degree.

[0097] The device provided in the embodiment of the present invention corrects the eye opening degree obtained based on the eye key point detection based on the head posture information of the face image, thereby obtaining the corrected opening degree and using it for fatigue detection. The application of the head posture information in the correction of the eye opening degree eliminates the influence of the head posture change on the eye opening degree, and can effectively improve the reliability of the eye opening degree obtained based on the key point detection, thereby ensuring the reliability and accuracy of fatigue detection.

[0098] Based on any of the above embodiments, the correction unit is specifically used for: Based on the mapping function and the head posture information, the eye opening degree is corrected to obtain the corrected eye opening degree; The mapping function is determined based on the eye opening degree of the sample person in the sample image taken under the same sample eye opening degree and different sample head posture information.

[0099] Based on any of the above embodiments, the computing unit is specifically used for: Based on the key point position information, calculating the initial opening and closing degree; Based on the eye opening degree distribution information of the person corresponding to the face image, the initial opening degree is individually standardized to obtain the eye opening degree.

[0100] Based on any of the above embodiments, the computing unit is specifically used for: Based on the eye opening degree distribution information of the person corresponding to the face image, determining the maximum individual opening degree and the minimum individual opening degree of the person corresponding to the face image; Based on the individual maximum opening degree and the individual minimum opening degree, the initial opening degree is individually standardized to obtain the eye opening degree.

[0101] Based on any of the above embodiments, the eye opening degree distribution information is obtained based on a driving face image of the person corresponding to the face image in the early stage of driving.

[0102] Based on any of the above embodiments, the detection unit is specifically used for: Performing open-eye and closed-eye classification on the eye region of the face image to obtain an open-eye and closed-eye classification result; Based on the corrected opening and closing degree and the open and closed eyes classification result, a fatigue detection result of the face image is determined.

[0103] Based on any of the above embodiments, the open and closed eyes classification is performed based on a classification model, and the eye key point detection is performed based on an eye key point detection model; The classification model and the eye key point detection model share a backbone network.

[0104] Based on any of the above embodiments, the eye area of ​​the facial image is obtained by cropping the facial image based on a key point detection result of the facial image.

[0105] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the fatigue detection method, which includes: Acquire a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; Performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree; Based on the corrected opening and closing degree, a fatigue detection result of the face image is determined.

[0106] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0107] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the fatigue detection method provided by the above methods, the method includes: Acquire a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; Performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree; Based on the corrected opening and closing degree, a fatigue detection result of the face image is determined.

[0108] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the fatigue detection method provided by the above methods, the method comprising: Acquire a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; Performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree; Based on the corrected opening and closing degree, a fatigue detection result of the face image is determined.

[0109] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fatigue detection method, characterized in that: include: Acquire a face image, perform eye key point detection on the eye region of the face image, obtain key point position information of the eye region, and determine the eye opening degree based on the key point position information; Performing head pose estimation on the face image to obtain head pose information of the face image, and correcting the eye opening degree based on the head pose information to obtain a corrected eye opening degree; Based on the corrected opening and closing degree, a fatigue detection result of the face image is determined.

2. The fatigue detection method according to claim 1, characterized in that: The correcting the eye opening degree based on the head posture information to obtain the corrected eye opening degree includes: Based on the mapping function and the head posture information, the eye opening degree is corrected to obtain the corrected eye opening degree; The mapping function is determined based on the eye opening degree of the sample person in the sample image taken under the same sample eye opening degree and different sample head posture information.

3. The fatigue detection method according to claim 1, characterized in that: The step of determining the eye opening degree based on the key point position information includes: Based on the key point position information, calculating the initial opening and closing degree; Based on the eye opening degree distribution information of the person corresponding to the face image, the initial opening degree is individually standardized to obtain the eye opening degree.

4. The fatigue detection method according to claim 3, characterized in that: The step of performing individual standardization on the initial eye opening degree based on the eye opening degree distribution information of the person corresponding to the face image to obtain the eye opening degree includes: Based on the eye opening degree distribution information of the person corresponding to the face image, determining the maximum individual opening degree and the minimum individual opening degree of the person corresponding to the face image; Based on the individual maximum opening degree and the individual minimum opening degree, the initial opening degree is individually standardized to obtain the eye opening degree.

5. The fatigue detection method according to claim 3, characterized in that: The eye opening degree distribution information is obtained based on a driving face image of a person corresponding to the face image at an early stage of driving.

6. The fatigue detection method according to any one of claims 1 to 5, characterized in that: The step of determining the fatigue detection result of the face image based on the corrected opening and closing degree includes: Performing open-eye and closed-eye classification on the eye region of the face image to obtain an open-eye and closed-eye classification result; Based on the corrected opening and closing degree and the open and closed eyes classification result, a fatigue detection result of the face image is determined.

7. The fatigue detection method according to claim 6, characterized in that: The open and closed eyes classification is performed based on a classification model, and the eye key point detection is performed based on an eye key point detection model; The classification model and the eye key point detection model share a backbone network.

8. The fatigue detection method according to any one of claims 1 to 5, characterized in that: The eye area of ​​the face image is obtained by cropping the face image based on a key point detection result of the face image.

9. A fatigue detection device, characterized in that: include: A computing unit, configured to acquire a face image, perform eye key point detection on an eye region of the face image, obtain key point position information of the eye region, and determine an eye opening degree based on the key point position information; a correction unit, configured to estimate the head posture of the face image to obtain head posture information of the face image, and correct the eye opening degree based on the head posture information to obtain a corrected eye opening degree; A detection unit is used to determine a fatigue detection result of the face image based on the corrected opening and closing degree.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the fatigue detection method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fatigue detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the fatigue detection method according to any one of claims 1 to 8 is implemented.

Images