Multi-feature perception driving fatigue state detection method, device, medium and product

Abstract

The invention discloses a multi-feature perception driving fatigue state detection method and device, a medium and a product, and relates to the technical field of driver fatigue driving detection, and the method comprises the steps: obtaining a face image of a driver; key point extraction is carried out on the face image to obtain face key point coordinates of the driver; based on the facial key point coordinates, performing facial feature analysis on the facial image, and determining facial key features of the facial image; the face key features comprise state information of each part in the face; and determining whether the driver has a fatigue driving behavior or not based on the comprehensive fatigue index according to the facial key features. Whether the driver has the fatigue driving behavior or not can be accurately determined based on the facial key point features.

Description

Technical Field

[0001] The present application relates to the technical field of driver fatigue driving detection, and in particular to a multi-feature perception driving fatigue state detection method, equipment, medium and product. Background Art

[0002] With the rapid development of transportation and the increasing popularity of vehicles, road traffic accidents occur frequently. Among them, fatigue driving behavior is an important factor affecting vehicle driving safety. Fatigue driving behavior refers to the phenomenon that drivers have inattention, fatigue, sleepiness, and operating errors after driving for a long time, which can easily cause traffic accidents. Therefore, fatigue driving detection of drivers to prevent fatigue driving is of great significance to preventing traffic accidents. Summary of the invention

[0003] The purpose of this application is to provide a multi-feature perception driving fatigue state detection method, equipment, medium and product, which can accurately determine whether the driver has fatigue driving behavior based on facial key point features.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a multi-feature perception driving fatigue state detection method, comprising:

[0006] Get the driver's facial image.

[0007] Key points are extracted from the facial image to obtain the coordinates of the driver's facial key points.

[0008] Based on the coordinates of facial key points, facial feature analysis is performed on the facial image to determine the facial key features of the facial image; the facial key features include status information of various parts of the face.

[0009] According to the key facial features and based on the comprehensive fatigue index, it is determined whether the driver has fatigue driving behavior.

[0010] Optionally, before obtaining the driver's facial image, the following steps are further included:

[0011] Get multiple frames of the driver's face images to be detected.

[0012] According to a first frame of face images to be detected among multiple frames of face images to be detected, face information of the first frame of face images to be detected is obtained.

[0013] For each frame of face picture to be detected except the first frame of face picture to be detected, the face position of the current frame of face picture to be detected is generated according to the face information of the previous frame of face picture to be detected corresponding to the current frame of face picture to be detected.

[0014] By adopting the image processing method of contrast enhancement and grayscale interception, each frame of the face image to be detected is processed to obtain the facial image of the driver.

[0015] Optionally, extracting key points from the facial image to obtain the coordinates of the driver's facial key points specifically includes:

[0016] Based on a facial feature point detection algorithm, facial key points in the facial image are located.

[0017] Based on computer vision technology and image processing algorithms, key feature points among the located facial key points are identified; the key feature points include facial key points and contour key points; the facial key points include eyes and mouth.

[0018] The coordinates of the key feature points are determined as the coordinates of the key points on the driver's face.

[0019] Optionally, based on the facial key point coordinates, performing facial feature analysis on the facial image to determine the facial key features of the facial image specifically includes:

[0020] Based on OpenCV, the facial key point coordinates and head rectangle frame in the facial image are annotated.

[0021] The convex hull position is obtained based on cv2.convexHull, and the face contour position is drawn using drawContours; the convex hull position is used to identify the direction of the face in the facial image.

[0022] According to the facial key points, the direction of the face in the facial image and the facial contour, based on the facial key feature measurement index, the facial key features of the facial image are determined.

[0023] Optionally, the key facial features include eye status, mouth opening status and head posture.

[0024] Optionally, the key facial feature measurement indicators include: degree of eye opening, degree of mouth opening, degree of head tilt, frequency of blinking and yawning, and frequency of nodding.

[0025] Optionally, determining whether the driver has fatigue driving behavior based on the comprehensive fatigue index according to the key facial features specifically includes:

[0026] When the comprehensive fatigue index self.score is greater than 30 and less than or equal to 55, the driver is in a mild fatigue state.

[0027] When the comprehensive fatigue index self.score is greater than 55 and less than or equal to 75, the driver is in moderate fatigue state.

[0028] When the comprehensive fatigue index self.score is greater than 75, the driver is in severe fatigue state.

[0029] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-feature perception driving fatigue state detection method.

[0030] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-feature perception driving fatigue state detection method.

[0031] In a fourth aspect, a computer program product includes a computer program, which, when executed by a processor, implements the multi-feature perception driving fatigue state detection method.

[0032] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0033] The present application provides a multi-feature sensing driving fatigue state detection method, equipment, medium and product, the method includes first, collecting the driver's facial image data; then, using advanced algorithms to perform fine key point extraction processing on the facial image to obtain accurate coordinate information of the driver's facial key points; based on these coordinate information, in-depth facial feature analysis is performed on the facial image to clearly identify multiple key facial features of the facial image, which cover the specific status information of various parts of the face; further, based on the identified facial key features, combined with the comprehensive fatigue index as an evaluation standard, it is scientifically determined whether the driver has fatigue driving behavior. The method proposed in this application, by virtue of its precise capture and analysis of facial key point features, can accurately identify the driver's fatigue driving state. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 An application environment diagram of a multi-feature perception driving fatigue state detection method provided in one embodiment of the present application.

[0036] Figure 2A flowchart of a multi-feature perception driving fatigue state detection method provided in one embodiment of the present application.

[0037] Figure 3 A logical architecture diagram of a multi-feature perception driving fatigue state detection method provided in one embodiment of the present application.

[0038] Figure 4 A schematic diagram of the coordinates and information of key points of a 68-face prediction model provided in one embodiment of the present application.

[0039] Figure 5 A schematic diagram of a head pitch angle provided in an embodiment of the present application.

[0040] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] Existing fatigue detection methods mainly include three methods: based on physiological characteristics information, based on vehicle status and based on driver facial features. Among them:

[0043] The fatigue detection method based on physiological characteristics detects the driver's various physiological indicators as the basis for fatigue detection. When the driver is in a fatigued state, the physiological indicators will deviate from the normal state. The detection method based on physiological characteristics is one of the most objective and accurate methods of fatigue detection. Generally, intelligent sensors are used to detect the driver's physiological indicators and judge fatigue by real-time detection of the driver's physiological signals. However, since the collection of physiological characteristics requires the driver to wear a variety of sensors to assist, the collection device is costly and inefficient. And the sensor inevitably needs to come into contact with the driver, so the detection method based on physiological characteristics has a certain degree of safety hazards.

[0044] The fatigue detection method based on vehicle status characteristics mainly uses steering wheel rotation information and road deviation values ​​as fatigue features for detection. Although this method does not require contact with the driver's body, the robustness of the detection algorithm will be greatly affected due to differences in objective factors such as driving habits, road conditions, and vehicle conditions.

[0045] The fatigue detection method based on the driver's facial features mainly detects the external changes of the driver's face during driving, uses machine vision methods to extract the driver's facial change features, and extracts multiple features of the driver's face through algorithms based on the relationship between facial features and fatigue driving: such as blinking, yawning, or changes in head posture, to detect the driver's fatigue state. The results of analyzing the driver's state through this method are more accurate and robust, with the advantages of low cost, no contact, convenient detection, and relatively high accuracy. It is currently the main method for fatigue detection.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The multi-feature sensing driving fatigue state detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the facial image to be processed to the server 104. After the server 104 receives the facial image to be processed, for the facial image to be processed, the server 104 obtains the facial image of the driver; extracts key points from the facial image to obtain the coordinates of the facial key points of the driver; based on the facial key point coordinates, performs facial feature analysis on the facial image to determine the facial key features of the facial image; the facial key features include status information of each part of the face; based on the facial key features, based on the comprehensive fatigue index, determine whether the driver has fatigue driving behavior. The server 104 can feedback the fatigue driving behavior judgment result obtained to the terminal 102. In addition, in some embodiments, the multi-feature perception driving fatigue state detection method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform image processing on the facial image to be processed, or the server 104 can obtain the facial image to be processed from the data storage system and perform image processing on the facial image to be processed.

[0048] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0049] In an exemplary embodiment, Figure 2 As shown, a multi-feature sensing driving fatigue state detection method is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 204. Among them:

[0050] Step 201, obtaining a facial image of the driver.

[0051] Step 202: extract key points from the facial image to obtain the coordinates of the driver's facial key points.

[0052] Step 203, based on the facial key point coordinates, perform facial feature analysis on the facial image to determine the facial key features of the facial image; the facial key features include status information of various parts of the face.

[0053] Step 204: Determine whether the driver has fatigue driving behavior based on the comprehensive fatigue index according to the key facial features.

[0054] In some implementations, before executing step 201, the method further includes:

[0055] Get multiple frames of the driver's face images to be detected.

[0056] According to a first frame of face images to be detected among multiple frames of face images to be detected, face information of the first frame of face images to be detected is obtained.

[0057] For each frame of face picture to be detected except the first frame of face picture to be detected, the face position of the current frame of face picture to be detected is generated according to the face information of the previous frame of face picture to be detected corresponding to the current frame of face picture to be detected.

[0058] By adopting the image processing method of contrast enhancement and grayscale interception, each frame of the face image to be detected is processed to obtain the facial image of the driver.

[0059] Specifically, a 1080p resolution camera is generally used and placed in front of the driver to obtain a clear facial image. The facial behavior image includes a blinking facial behavior image, a yawning facial behavior image, and a head tilt image.

[0060] First, multiple frames of face images to be detected of the target driver are obtained; based on the first frame of face image to be detected in the multiple frames of face images to be detected, the face information of the first frame of face image to be detected is obtained; then for each frame of face image to be detected except the first frame of face image to be detected, the face position of the current frame of face image to be detected is generated according to the face information of the previous frame of face image to be detected corresponding to the current frame of face image to be detected.

[0061] Finally, contrast enhancement and grayscale capture are used to improve image quality, optimize the accuracy of facial analysis, and complete the preliminary processing of the driver's facial image.

[0062] It should be noted that in the face detection task, the maximum margin object detector (MMOD) based on CNN functions is used for face detection in this embodiment. The detection is based on deep learning and can quickly and effectively complete face detection and face alignment. At the same time, MMOD can detect five key points of the face: left eye, right eye, nose tip, left corner of the mouth, and right corner of the mouth. It is suitable for different facial directions, robust in occasional scenes, works very fast on the GPU, and uses a very simple training process. However, small faces will not be detected because the minimum face size it is trained for is 80×80. Therefore, it is necessary to ensure that the face size should be larger than the face size in the application.

[0063] Wherein, in some embodiments, Figure 3 As shown, when executing step 202, the specific steps may be as follows:

[0064] Based on a facial feature point detection algorithm, facial key points in the facial image are located.

[0065] Based on computer vision technology and image processing algorithms, key feature points among the located facial key points are identified; the key feature points include facial key points and contour key points; the facial key points include eyes and mouth.

[0066] The coordinates of the key feature points are determined as the coordinates of the key points on the driver's face.

[0067] Specifically, the key points of the facial image are extracted and the coordinates of the extracted facial key points are output: first, the facial key points of the first frame of the face to be detected are detected and located using the 68 facial feature point detection algorithm encapsulated in the Dlib library.

[0068] Then, through computer vision technology and image processing algorithms, some key feature points in the face image are automatically identified and located, such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc. Through key point detection, the key point detection is performed on the face position of each frame of the face image to be detected, and the key point information of each face image to be detected is obtained.

[0069] Finally, through key point detection, feature information in the face image is extracted and used for subsequent fatigue detection tasks. Specifically, key point detection can help determine the facial features of a face.

[0070] It should be noted that this implementation uses Dlib to extract 68 key points of the face. It is known that face 68 key point detection is an important function in the Dlib library, which can locate key points of a face image and mark important feature points of the face. Key points usually include eyes, eyebrows, nose, mouth and other positions, a total of 68 key points, such as Figure 4 shown.

[0071] In this embodiment, the key point information refers to the information of the key feature points in the face image. These key feature points include the corners of the eyes, the corners of the mouth, the tip of the nose, etc. The key feature points of the face usually include 5 points, 15 points, 68 points, 96 points, 98 points, 106 points, 186 points, etc. The general 68-point detection divides the key points of the face into facial key points and contour key points. The facial key points include eyebrows, eyes, nose, mouth, a total of 51 key points, and the contour key points include 17 key points. The information of the key feature points can be used to determine whether the eyes are closed, the mouth is open, and the degree of eye closure.

[0072] Wherein, in some embodiments, Figure 3 As shown, when executing step 203, the specific steps may be as follows:

[0073] Based on OpenCV, the facial key point coordinates and head rectangle frame in the facial image are annotated.

[0074] The convex hull position is obtained based on cv2.convexHull, and the face contour position is drawn using drawContours; the convex hull position is used to identify the direction of the face in the facial image.

[0075] According to the facial key points, the direction of the face in the facial image and the facial contour, based on the facial key feature measurement index, the facial key features of the facial image are determined.

[0076] Specifically, OpenCV is used to perform image processing first, mark the positions of 68 key points and the head rectangle on the face, use cv2.convexHull to obtain the convex hull position, use drawContours to draw the face contour position, and display the key feature measurement indicator count and face contour in the video display window.

[0077] The key facial features include the state of the eyes, the state of the mouth opening, and the head posture. The key facial feature measurement indicators include: the degree of eye opening and closing, the degree of mouth opening and closing, the degree of head tilt, the frequency of blinking and yawning, and the frequency of nodding. The degree of eye opening and closing, mouth opening and closing, and head tilt of the driver can be detected based on the key features.

[0078] Then, the number of blinks, yawns, and nods are initialized to 0. Based on the positions of the different feature points in the current frame of the face video image and the positions of the different feature points in the N-1 frames of the face video image before the current frame, the following are calculated: the degree of eye opening and closing, the degree of mouth opening and closing, and the degree of head tilt.

[0079] The formula for calculating the degree of eye opening and closing is:

[0080]

[0081] Wherein, EAR is the degree of eye opening and closing, P1 and P4 are the positions of the feature points of the corners of the eyes, P2 and P3 are the positions of the two feature points of the upper eyelids, and P5 and P6 are the positions of the two feature points of the lower eyelids.

[0082] The calculation formula for calculating the degree of mouth opening and closing is:

[0083]

[0084] Wherein, MAR is the degree of opening and closing of the mouth, M1 and M4 are the positions of the feature points of the corners of the mouth at both ends, M2 and M3 are the positions of the two feature points of the upper lip, and M5 and M6 are the positions of the two feature points of the lower lip.

[0085] Among them, the head tilt angle (Euler angle) is solved by the Head Pose Estimation algorithm. The steps are generally as follows: 2D face key point detection; 3D face model matching; solving the conversion relationship between 3D points and corresponding 2D points; solving the Euler angle according to the rotation matrix.

[0086] Specifically, the degree of head tilt is represented by the posture of an object relative to the camera, which can be solved using the rotation matrix and translation matrix:

[0087] Translation matrix: the spatial position relationship matrix of the object relative to the camera.

[0088] Rotation matrix: the spatial attitude relationship matrix of the object relative to the camera.

[0089] The rotation and translation matrices are calculated using the solvePnp() function provided by OpenCV through the position of the point in the world coordinate system, the pixel coordinate position and the camera parameters. Finally, the Euler angles are solved through the Rodriguez formula (converting the rotation matrix R into a rotation vector), the hconcat() function and the decomposeProjectionMatrix() function. The specific formula is:

[0090]

[0091] θ x =atan2(r 32 , r 33 ).

[0092]

[0093] Where: θ x ,θ y and θ z Represents the rotation angle around the x-axis, y-axis and z-axis respectively. The corresponding sine and cosine values ​​are S x , C x , S y , C y , S z , C z The r in the formula 11 to r 33 is the rotation matrix R 11 The atan2 function is a variation of the inverse tangent function that takes into account the signs of the two arguments to determine the correct quadrant for the angle.

[0094] Before executing step 204, the method further includes:

[0095] Determining whether the eye opening degree, mouth opening degree, and head pitch angle meet set thresholds respectively includes:

[0096] like Figure 5 As shown, the EAR is used to judge the degree of eye opening and closing. Under normal circumstances, the EAR (eye aspect ratio) remains near a stable value. When the eyes are closed, the EAR drops rapidly, and when the eyes are open, the EAR rises rapidly. Therefore, the driver's blinking process can be simulated using the EAR eye aspect ratio value, and the blink threshold is set to 0.2.

[0097] Determine the driver's mouth opening and closing state MAR. If the mouth dynamics are greater than the mouth threshold of 0.8, it is determined to be yawning.

[0098] Determine the head pitch angle HAR. If the head pitch angle is greater than the pitch angle threshold, it is determined as a nod.

[0099] Specifically, head posture judgment involves rotation angles in three directions: θx, θy, and θz, as well as Yaw, Pitch, and Roll.

[0100] When the driver is dozing off, his head will obviously make movements such as nodding and tilting. According to the head posture of ordinary people when dozing off, there are obviously few movements in Yaw, but mainly in Pitch. Therefore, the parameter is set to take the pitch rotation angle threshold as 0.3.

[0101] In summary, the threshold standards are set as follows:

[0102] EAR≥20%; MAR≥80%; HAR≥30%, wherein MAR is the degree of mouth opening and closing, EAR is the degree of eye opening and closing, and HAR is the degree of head pitch.

[0103] Determine whether the eye opening degree, mouth opening degree and head pitch angle meet the set thresholds. If the determination result is yes, the corresponding feature count is increased by 1 (number of blinks, number of yawns, number of nods)

[0104] Initialize the frequency of blinking, yawning, and nodding per unit time, with the initial value being 0. Intercept the number of blinks, yawns, and nods within the set time, divide by the set time, calculate the frequency of blinking, yawning, and nodding per unit time, and assign a value.

[0105] The preset cycle value is 10 minutes. The average value of blinking, yawning and nodding frequency within 10 minutes is recorded. This is used as a benchmark to compare with the blinking, yawning and nodding frequency values ​​in the next cycle value. The periodic change amplitude of blinking, yawning and nodding frequency between cycles is recorded.

[0106] It should be noted that the above-mentioned facial feature setting threshold is not a fixed value. This example only gives an example value, and its value can be determined according to the driver's own condition.

[0107] In some embodiments, when executing step 204, the specific steps may be as follows:

[0108] The parameters of the key features are integrated to form a comprehensive fatigue index. Based on the facial features, it is determined whether the driver has fatigue driving behavior.

[0109] Specifically, rules are formulated based on prior knowledge of the phenomenon and mechanism of driver fatigue driving. The fatigue levels are divided into no fatigue, mild fatigue, moderate fatigue and severe fatigue. Facial features are analyzed to determine the driver's fatigue level. When the driver is detected to be in a fatigued state, voice alarms are issued for different fatigue states of the driver.

[0110] This embodiment provides a comprehensive fatigue index algorithm based on the weighted sum of multiple features, and self.score is the comprehensive fatigue index. Specifically: the three facial features of the eyes, mouth and head are multiplied by their weight values ​​and the periodic variation amplitude, and the sum is calculated. It should be noted that the weight values ​​of the three facial features of the eyes, mouth and head are not fixed values, and the weights can be set according to the driver's own discretion.

[0111] Finally, the parameters of the key features are integrated to form a comprehensive fatigue index. When the comprehensive fatigue index self.score is greater than 30 and less than or equal to 55, the driver is in a mild fatigue state. When the comprehensive fatigue index self.score is greater than 55 and less than or equal to 75, the driver is in a moderate fatigue state. When the comprehensive fatigue index self.score is greater than 75, the driver is in a severe fatigue state. When it is detected that the driver is in a corresponding fatigue state, a voice alarm of the corresponding fatigue state is issued to the driver.

[0112] It should be noted that the threshold value for setting the above-mentioned comprehensive fatigue index self.score is not a fixed value. The table only gives example values, and the value can be determined according to the driver's own condition.

[0113] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store fatigue degree judgment results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-feature perception driving fatigue state detection method is implemented.

[0114] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0116] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0117] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0120] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0121] In summary, this application has the following technical effects:

[0122] 1) The facial feature fatigue state detection provided in this application mainly detects fatigue state through facial features, uses a camera for image input, uses facial key points to analyze the eyes, mouth information and head tilt angle, extracts personal facial information, and classifies the driver's facial fatigue state.

[0123] 2) The fatigue state detection process provided by this application will not affect the driver's original fatigue state, and is not restricted by factors such as vehicle properties, road conditions and driver habits. It can effectively prevent the driver from driving while fatigued, thereby significantly reducing traffic accidents caused by fatigue driving.

[0124] 3) In this application, facial image acquisition uses a CNN-based maximum margin object detector (MMOD) for face detection, which is based on deep learning and can quickly and effectively complete face detection and face alignment, is applicable to different face directions, is robust in occasional scenes, works very fast on GPU, and has a very simple training process.

[0125] 4) This application calculates the current eye aspect ratio, the current mouth aspect ratio and the current head pitch angle based on the coordinates of the facial key points, and can accurately determine the facial key features of the facial image.

[0126] 5) This application is based on facial feature fatigue detection. By presetting the cycle value, the periodic variation amplitude of the frequency values ​​of blinking, yawning and nodding in each cycle is recorded, and the three facial features of the eyes, mouth and head are multiplied by their weight values ​​and the periodic variation amplitude, and the sum is calculated. It can effectively reduce the detection error caused by the facial differences of drivers, effectively improve the detection accuracy of the comprehensive driving fatigue index, and improve the accuracy of identifying the driving fatigue state.

[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-feature perception driving fatigue state detection method, characterized in that: include: Obtaining a facial image of the driver; Extracting key points from the facial image to obtain the coordinates of the driver's facial key points; Based on the coordinates of the facial key points, the facial image is subjected to facial feature analysis to determine the facial key features of the facial image; the facial key features include status information of various parts of the face; According to the key facial features and based on the comprehensive fatigue index, it is determined whether the driver has fatigue driving behavior.

2. The multi-feature perception driving fatigue state detection method according to claim 1 is characterized in that: Before acquiring the driver's facial image, it also includes: Obtain multiple frames of the driver's face images to be detected; According to a first frame of face images to be detected among multiple frames of face images to be detected, obtaining face information of the first frame of face images to be detected; For each frame of the face picture to be detected except the first frame of the face picture to be detected, generating the face position of the current frame of the face picture to be detected according to the face information of the previous frame of the face picture to be detected corresponding to the current frame of the face picture to be detected; By adopting the image processing method of contrast enhancement and grayscale interception, each frame of the face image to be detected is processed to obtain the facial image of the driver.

3. The multi-feature sensing driving fatigue state detection method according to claim 1 is characterized in that: Extracting key points from the facial image to obtain the coordinates of the driver's facial key points specifically includes: Based on a facial feature point detection algorithm, locating facial key points in the facial image; Based on computer vision technology and image processing algorithms, identify key feature points among the located facial key points; the key feature points include facial key points and contour key points; the facial key points include eyes and mouth; The coordinates of the key feature points are determined as the coordinates of the key points on the driver's face.

4. The multi-feature sensing driving fatigue state detection method according to claim 1, characterized in that: Based on the facial key point coordinates, facial feature analysis is performed on the facial image to determine the facial key features of the facial image, specifically including: Based on OpenCV, the facial key point coordinates and head rectangle frame in the facial image are annotated; The convex hull position is obtained based on cv2.convexHull, and the face contour position is drawn using drawContours; the convex hull position is used to identify the direction of the face in the facial image; According to the facial key points, the direction of the face in the facial image and the facial contour, based on the facial key feature measurement index, the facial key features of the facial image are determined.

5. The multi-feature sensing driving fatigue state detection method according to claim 1 is characterized in that: The key facial features include eye state, mouth opening state and head posture.

6. The multi-feature sensing driving fatigue state detection method according to claim 1, characterized in that: The key facial feature measurements include: degree of eye openness, degree of mouth openness, degree of head tilt, frequency of blinking and yawning, and frequency of nodding.

7. The multi-feature sensing driving fatigue state detection method according to claim 1 is characterized in that: According to the key facial features and based on the comprehensive fatigue index, determining whether the driver has fatigue driving behavior specifically includes: When the comprehensive fatigue index self.score is greater than 30 and less than or equal to 55, the driver is in a mild fatigue state; When the comprehensive fatigue index self.score is greater than 55 and less than or equal to 75, the driver is in moderate fatigue state; When the comprehensive fatigue index self.score is greater than 75, the driver is in severe fatigue state.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-feature perception driving fatigue state detection method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a multi-feature perception driving fatigue state detection method as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a multi-feature perception driving fatigue state detection method as described in any one of claims 1 to 7 is implemented.

Images