Driving state monitoring method and device, equipment and storage medium

Abstract

The invention relates to the technical field of state monitoring, and discloses a driving state monitoring method, device and equipment and a storage medium, and the method comprises the steps: collecting a driver image, and carrying out the face detection based on the driver image, and obtaining a face image; performing face key point detection based on the face image to obtain face key point coordinates; determining a face opening degree and a head attitude angle according to the face key point coordinates; and determining the current driving state of the driver according to the face opening degree and the head attitude angle, and performing early warning according to the current driving state of the driver. According to the invention, the driving state of the driver can be accurately identified, safety early warning is timely carried out, and the driving safety is effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of state monitoring, and particularly to a driving state monitoring method, device, equipment, and storage medium. Background Art

[0002] The existing technology uses the data obtained by a heartbeat detection sensor and a steering wheel off-hand detection sensor as input, with a relatively high hardware cost and is not easy to be installed on mid- to low-end vehicle models. Moreover, the existing technology obtains the signal changes of the steering wheel off-hand and heartbeat through steering wheel off-hand detection and heartbeat detection. However, due to the relatively small signal changes, there may be false detections or missed detections. In addition, the existing technology cannot effectively judge the driver's distraction situation, further affecting driving safety.

[0003] Therefore, how to accurately identify the driver's driving state and give timely safety warnings, thereby improving driving safety is an urgent problem to be solved at present.

[0004] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a driving state monitoring method, device, equipment, and storage medium, aiming to solve the technical problem of poor accuracy in determining the fatigue state through steering wheel off-hand detection and heartbeat detection, and being unable to effectively judge the driver's distraction situation, resulting in low driving safety.

[0006] To achieve the above purpose, this application proposes a driving state monitoring method, and the method includes:

[0007] Collect a driver image, and perform face detection based on the driver image to obtain a face image;

[0008] Perform face key point detection based on the face image to obtain face key point coordinates;

[0009] Determine the facial opening degree and head pose angle according to the face key point coordinates;

[0010] Determine the driver's current driving state according to the facial opening degree and head pose angle, and give a warning according to the driver's current driving state.

[0011] In one embodiment, the determining the driver's current driving state according to the facial opening degree and head pose angle, and giving a warning according to the driver's current driving state includes:

[0012] Determine the driver's fatigue state according to the facial opening degree;

[0013] Determine the driver's distraction state based on the head pose angle;

[0014] Determine the driver's current driving state based on the driver's fatigue state and the driver's distraction state, and give an alarm according to the driver's current driving state.

[0015] In one embodiment, the determining the driver's fatigue state according to the facial opening degree includes:

[0016] Count the number of eye closures according to the eye opening degree in the facial opening degree to obtain eye closure frequency information;

[0017] Count the number of yawns according to the mouth opening degree in the facial opening degree to obtain yawn frequency information;

[0018] Determine the driver's fatigue state based on the eye closure frequency information and the yawn frequency information.

[0019] In one embodiment, the determining the driver's fatigue state based on the eye closure frequency information and the yawn frequency information includes:

[0020] Determine eye closure fatigue based on the eye closure frequency information;

[0021] Determine yawn fatigue based on the yawn frequency information;

[0022] Determine the driver's fatigue state based on the eye closure fatigue and / or yawn fatigue.

[0023] In one embodiment, the determining the driver's distraction state based on the head pose angle includes:

[0024] Determine the driver's distraction level based on the head pose angle;

[0025] Determine the driver's distraction state based on the driver's distraction level.

[0026] In one embodiment, the determining the facial opening degree and the head pose angle according to the facial key point coordinates includes:

[0027] Determine the eye key point coordinates, the mouth key point coordinates and the target key point coordinates based on the facial key point coordinates;

[0028] Determine the facial opening degree and the head pose angle according to the eye key point coordinates, the mouth key point coordinates and the target key point coordinates.

[0029] In one embodiment, the determining the facial opening degree and the head pose angle according to the eye key point coordinates, the mouth key point coordinates and the target key point coordinates includes:

[0030] Determine the eye opening degree according to the coordinates of the eye key points, and determine the mouth opening degree according to the coordinates of the mouth key points;

[0031] Determine the facial opening degree according to the eye opening degree and the mouth opening degree;

[0032] Determine the rotation matrix according to the coordinates of the target key points, and decompose the rotation matrix to obtain the head pose angle.

[0033] In one embodiment, collecting the driver image and performing face detection based on the driver image to obtain a face image includes:

[0034] Collect the driver image, preprocess the driver image to obtain a preprocessed driver image, where the preprocessing at least includes picture format conversion and picture size scaling;

[0035] Perform face detection based on the preprocessed driver image to obtain the rectangular coordinate frame of the face outline and the face confidence;

[0036] Determine the face image based on the rectangular coordinate frame of the face outline and the face confidence.

[0037] In one embodiment, determining the face image based on the rectangular coordinate frame of the face outline and the face confidence includes:

[0038] Determine whether the driver image is a valid image based on the face confidence;

[0039] When the driver image is a valid image, crop the driver image according to the rectangular coordinate frame of the face outline to obtain a face image.

[0040] In addition, to achieve the above object, the present application also proposes a driving state monitoring device, and the driving state monitoring device includes:

[0041] A detection module for collecting a driver image and performing face detection based on the driver image to obtain a face image;

[0042] The detection module is further configured to perform face key point detection based on the face image to obtain face key point coordinates;

[0043] A determination module for determining the facial opening degree and the head pose angle according to the face key point coordinates;

[0044] An early warning module for determining the current driving state of the driver according to the facial opening degree and the head pose angle, and performing an early warning according to the current driving state of the driver.

[0045] In addition, to achieve the above object, the present application further provides a driving state monitoring device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the driving state monitoring method as described above.

[0046] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the driving state monitoring method as described above.

[0047] In addition, to achieve the above object, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the driving state monitoring method as described above.

[0048] The present application provides a driving state monitoring method. The present application first collects a driver image, performs face detection based on the driver image to obtain a face image; performs face key point detection based on the face image to obtain face key point coordinates; determines the facial opening degree and head pose angle according to the face key point coordinates; determines the current driving state of the driver according to the facial opening degree and head pose angle, and issues a warning according to the current driving state of the driver, which can accurately identify the driving state of the driver and issue a safety warning in a timely manner, effectively improving driving safety.

[0049] In summary, by collecting the driver image and performing face detection, the present application can monitor the facial state of the driver in real time, and then accurately identify the facial area by performing face key point detection on the face image, quickly and accurately determine the facial opening degree and head pose angle according to the face key point coordinates, so as to accurately identify the current driving state of the driver according to the facial opening degree and head pose angle and issue a warning in a timely manner, which can effectively reduce false state reports, effectively improve driving safety, overcome the technical defects of poor accuracy in determining the fatigue state by detecting the departure of the steering wheel from the hand and detecting the heartbeat, and being unable to effectively judge the distraction of the driver, resulting in low driving safety, and can accurately identify the driving state of the driver and issue a safety warning in a timely manner, effectively improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0051] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 It is a schematic flowchart provided for the first embodiment of the driving state monitoring method of the present application;

[0053] Figure 2 It is a schematic flowchart provided for the second embodiment of the driving state monitoring method of the present application;

[0054] Figure 3 It is the overall flowchart of the driving state monitoring provided for an embodiment of the driving state monitoring method of the present application;

[0055] Figure 4 It is a schematic flowchart provided for the third embodiment of the driving state monitoring method of the present application;

[0056] Figure 5 It is a schematic diagram of the module structure of the driving state monitoring device in the embodiment of the present application;

[0057] Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the driving state monitoring method in the embodiment of the present application.

[0058] The realization of the purpose, functional features and advantages of the present application will be further described with reference to the accompanying drawings in combination with the embodiments. Specific Embodiments

[0059] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0060] To better understand the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings of the specification and specific embodiments.

[0061] The main solution of the embodiment of the present application is: collect the driver's image, perform face detection based on the driver's image to obtain a face image; perform face key point detection based on the face image to obtain face key point coordinates; determine the facial opening degree and head pose angle according to the face key point coordinates; determine the current driving state of the driver according to the facial opening degree and head pose angle, and give an alarm according to the current driving state of the driver.

[0062] The existing technology uses the data obtained by a heartbeat detection sensor and a steering wheel departure detection sensor as input, with relatively high hardware costs and is not easy to be installed on mid - to - low - end vehicle models. Moreover, the existing technology obtains the signal changes of the steering wheel departure and heartbeat through steering wheel departure detection and heartbeat detection. However, due to the relatively small signal changes, there may be false detections or missed detections. In addition, the existing technology cannot effectively judge the driver's distracted situation, further affecting driving safety. Therefore, how to accurately identify the driver's driving state and give timely safety warnings, thereby improving driving safety is an urgent problem to be solved currently.

[0063] In this application, by collecting the driver's image and performing face detection, the facial state of the driver can be monitored in real - time. Then, by performing face key - point detection on the face image, the facial area can be accurately identified. According to the face key - point coordinates, the facial opening degree and head pose angle can be quickly and accurately determined. Thus, based on the facial opening degree and head pose angle, the current driving state of the driver can be accurately identified and timely warnings can be given, effectively reducing false alarms and improving driving safety. It overcomes the technical defects that the accuracy of determining the fatigue state by steering wheel departure detection and heartbeat detection is poor and the driver's distracted situation cannot be effectively judged, resulting in low driving safety. It can accurately identify the driver's driving state and give timely safety warnings, effectively improving driving safety.

[0064] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a driving state monitoring device, etc. that can implement the above functions. Hereinafter, taking the driving state monitoring device as an example, this embodiment and the following embodiments will be described.

[0065] Based on this, the embodiment of this application provides a driving state monitoring method, referring to Figure 1 , Figure 1 which is the flowchart of the first embodiment of the driving state monitoring method of this application.

[0066] In this embodiment, the driving state monitoring method includes steps S10 - S40:

[0067] Step S10, collect the driver's image and perform face detection based on the driver's image to obtain a face image.

[0068] It should be noted that the driver image is collected by the IR camera associated with the driver monitoring system and can be a frame image in the video. The IR camera can capture the infrared image of the driver and clearly obtain the facial features of the driver even in a dim environment. The collected driver image is transmitted to the vehicle-mounted controller for processing. The installation of the camera needs to meet the condition that the field of view can cover the entire driving position area, and at the same time, the camera needs to be firmly installed.

[0069] It can be understood that the collected driver image may be the overall image of the driver. Therefore, it is necessary to detect the driver's facial area in the driver image to improve the accuracy of driving state judgment.

[0070] In a feasible implementation manner, the step S10 may include: collecting a driver image, preprocessing the driver image to obtain a preprocessed driver image, where the preprocessing at least includes picture format conversion and picture size scaling; performing face detection based on the preprocessed driver image to obtain the rectangular coordinate frame of the face outline and the face confidence; and determining the face image based on the rectangular coordinate frame of the face outline and the face confidence.

[0071] It should be noted that image preprocessing is to ensure that the image format meets the requirements of the face detection algorithm model, thereby improving the accuracy and efficiency of face detection. Image preprocessing at least includes picture format conversion, picture size scaling, etc., and this embodiment does not make specific limitations on this.

[0072] It can be understood that picture format conversion refers to converting the collected image from the original format to the format that the model can process, such as converting the JPEG format to the RGB format. Picture size scaling is to adapt to cameras with different resolutions and different-sized screen displays. Usually, the image is scaled to a fixed size, such as scaling the image to 224x224 pixels to meet the requirements of the input layer of the deep learning model.

[0073] It is worth noting that the preprocessed driver image is input into the face detection algorithm model for face detection. Among them, the face detection algorithm model is a pre-trained lightweight deep learning model. The trained lightweight deep learning model is processed through model format conversion, model quantization, etc. and deployed into the vehicle-mounted controller and the hardware inference acceleration unit of the vehicle-mounted controller is called to complete the board-side inference of the model. The preprocessed driver image is input into the main driver face detection model, and the inference result is decoded to obtain the face information of the main driver's seat in the picture, including the rectangular coordinate frame of the face outline and the face confidence.

[0074] In a feasible implementation, determining the face image based on the rectangular coordinate frame of the outer contour of the face and the face confidence includes: determining whether the driver image is a valid image based on the face confidence; when the driver image is a valid image, cropping the driver image according to the rectangular coordinate frame of the outer contour of the face to obtain a face image.

[0075] It should be noted that the face confidence is used to evaluate whether the detected face is a valid face, that is, whether the face is occluded. The rectangular coordinate frame of the outer contour of the face is used to determine the position of the face in the image. If the face confidence is lower than the threshold of the preset value, it indicates that there is facial occlusion in this image frame and it is an invalid image, and the next image frame is directly processed; if the face confidence is greater than or equal to the threshold of the preset value, it indicates that this image frame is a valid image, and the driver image is cropped according to the rectangular coordinate frame of the outer contour of the face to obtain the face image of the driver.

[0076] Step S20: Perform face key point detection based on the face image to obtain face key point coordinates.

[0077] It should be noted that the face image of the driver is input into the face key point detection model for face key point detection. Among them, the face key point detection model is a pre-trained lightweight deep learning model used to identify and locate the key points on the face, such as the coordinates of parts such as eyes, nose, and mouth.

[0078] It can be understood that the face key point detection model also undergoes model format conversion and quantization processing to adapt to the hardware inference acceleration unit of the vehicle-mounted controller and complete the board-side inference of the model. The face image of the driver is input into the face key point detection model, and the inference result is decoded to obtain a total of 19 driver face key points including eyes, mouth, nose tip, nose corner points, etc. Each key point includes point position coordinates and point confidence.

[0079] Step S30: Determine the facial opening degree and head pose angle according to the face key point coordinates.

[0080] It should be noted that the facial opening degree includes the eye opening degree and the mouth opening degree. By analyzing the position coordinates of the key points of the eyes and mouth in the face key point coordinates, the eye opening degree and the mouth opening degree of the driver can be accurately calculated. The head pose angle is determined by analyzing the position coordinates of multiple pre-set key points in the face key point coordinates.

[0081] It can be understood that the facial opening degree is used to evaluate the fatigue condition of the driver, while the head posture angle reflects the driver's attention concentration. By accurately calculating the facial opening degree and the head posture angle, it is possible to effectively determine whether the driver is in a state of fatigue driving or distracted driving, thereby providing real-time warnings for driving safety, reminding the driver to take a rest or adjust the driving state to prevent potential dangers caused by fatigue driving or distracted driving.

[0082] Step S40, determine the current driving state of the driver according to the facial opening degree and the head posture angle, and issue a warning according to the current driving state of the driver.

[0083] It should be noted that the facial opening degree, that is, the eye opening degree and the mouth opening degree, is used to evaluate the fatigue degree of the driver. When the eye opening degree is lower than a certain threshold, it indicates that the driver may be in a fatigued state and needs to rest; while the mouth opening degree can assist in judging whether the driver is yawning and further confirm the fatigued state. The head posture angle is used to judge whether the driver is distracted. For example, if the head posture angle shows that the driver's head is overly tilted or turns frequently, it may mean that the driver's attention is not concentrated. Based on these parameters, the driving state of the driver can be monitored in real time, and a warning is issued when fatigue or distracted driving is detected, reminding the driver to take measures such as resting or adjusting the sitting posture to ensure driving safety.

[0084] It is worth noting that in low-speed driving scenarios such as reversing and idling, the driver needs to comprehensively understand the situation around the vehicle body, so it is necessary to constantly observe left and right. At this time, the driving state monitoring function needs to be inhibited. Therefore, vehicle speed information is introduced. When the vehicle speed is higher than 25 km / h, the driving state monitoring function is activated. In addition, when turning, if the turn signal is on, the alarm prompt of the driving state monitoring function is also inhibited, and when the turn signal is off, the alarm prompt returns to normal.

[0085] This embodiment provides a driving state monitoring method. In this embodiment, first, a driver image is collected, and face detection is performed based on the driver image to obtain a face image; face key point detection is performed based on the face image to obtain face key point coordinates; the facial opening degree and the head posture angle are determined according to the face key point coordinates; the current driving state of the driver is determined according to the facial opening degree and the head posture angle, and a warning is issued according to the current driving state of the driver, which can accurately identify the driving state of the driver and timely issue a safety warning, effectively improving driving safety.

[0086] In summary, in this embodiment, by collecting the driver's image and performing face detection, the facial state of the driver can be monitored in real time. Furthermore, by performing face key point detection on the face image, the facial area can be accurately identified. According to the coordinates of the face key points, the facial opening degree and the head pose angle can be quickly and accurately determined. Thus, according to the facial opening degree and the head pose angle, the current driving state of the driver can be accurately identified and early warnings can be given in a timely manner, which can effectively reduce false alarms and improve driving safety. It overcomes the technical defects that the accuracy of determining the fatigue state by detecting the departure of the hand from the steering wheel and detecting the heartbeat is poor, and it is impossible to effectively judge the distraction of the driver, resulting in low driving safety. It can accurately identify the driving state of the driver and give safety warnings in a timely manner, effectively improving driving safety.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , the step S40 further includes steps S401-S403:

[0088] Step S401, determining the fatigue state of the driver according to the facial opening degree.

[0089] It should be noted that the fatigue state of the driver includes fatigue and non-fatigue. The fatigue state of the driver can be specifically judged according to the facial features and behavior patterns of the driver. For example, when the eye opening degree of the driver is lower than a preset threshold and the duration exceeds a certain time, it can be considered that the driver is in a fatigue state. In addition, if the driver yawns frequently, that is, the mouth opening degree is large and the duration is long, it can also be used as a judgment basis for the fatigue state. However, an occasional yawn or a brief eye closure should not be directly judged as a fatigue state to avoid misjudgment. Therefore, in this embodiment, the determination of the fatigue state needs to comprehensively evaluate the duration and frequency of the eye opening degree and the mouth opening degree.

[0090] In a feasible implementation manner, step S401 specifically includes: counting the eye closing frequency according to the eye opening degree in the facial opening degree to obtain the eye closing frequency information; counting the yawning frequency according to the mouth opening degree in the facial opening degree to obtain the yawning frequency information; and determining the fatigue state of the driver according to the eye closing frequency information and the yawning frequency information.

[0091] It should be noted that the eye closing frequency can be counted by setting an eye opening degree threshold v1. If the opening degree of both the left and right eyes in each frame is lower than v1, it is considered that the eyes are in a closed state and the eye closing cumulative count is added. Similarly, the yawning frequency is counted by setting a mouth opening degree threshold v2. If the mouth opening degree is greater than v2, it is considered that the driver is yawning and the yawning cumulative count is added.

[0092] It can be understood that the fatigue state of the driver can be determined based on frequency information such as the cumulative count of eyes closed and the cumulative count of yawning, including fatigue from eyes closed and fatigue from yawning. This embodiment does not make specific restrictions on this.

[0093] In a feasible implementation manner, determining the fatigue state of the driver according to the eyes-closed frequency information and the yawning frequency information includes: determining eyes-closed fatigue according to the eyes-closed frequency information; determining yawning fatigue according to the yawning frequency information; and determining the fatigue state of the driver according to the eyes-closed fatigue and / or yawning fatigue.

[0094] It is worth noting that a statistical sliding window L is set. When the proportion of the cumulative number of eyes closed or yawning within the sliding window L reaches a preset proportion threshold, it is determined that the driver is in a state of eyes-closed fatigue or yawning fatigue. Among them, the preset proportion threshold can be 60%, and this embodiment does not make specific restrictions on this.

[0095] Specifically, in order to improve the alarm accuracy of fatigue, an eye opening / closing degree threshold v3 (satisfying v3 < v1) is set. If there are number_1 values less than v3 in the cumulative count of eyes closed, it is considered eyes-closed fatigue. A mouth opening / closing degree threshold is set to v4 (satisfying v4 > v2). If there are number_2 values greater than v6 in the cumulative count of yawning, it is considered yawning fatigue.

[0096] It is worth noting that the fatigue state of the driver can be eyes-closed fatigue, or yawning fatigue, or a combination of eyes-closed fatigue and yawning fatigue at the same time. The driver may show the behaviors of eyes closed and yawning simultaneously. Therefore, comprehensively evaluating these two fatigue states is crucial for accurately judging the fatigue degree of the driver. By setting reasonable thresholds and statistical windows, misjudgments caused by accidental actions can be effectively avoided, thereby improving the accuracy of fatigue detection.

[0097] Step S402, determining the distracted state of the driver according to the head pose angle.

[0098] It should be noted that the distracted state of the driver includes distracted and non-distracted. The distracted state of the driver is determined according to the head pose angle, and the head pose angle can reflect the movement of the driver's head. For example, when the head pose angle shows that the driver's head is overly tilted or frequently rotated, it may mean that the driver's attention is not concentrated. In order to accurately judge the distracted state, a head pose angle threshold v5 can be set to determine whether the driver is in a distracted state.

[0099] In a feasible implementation manner, step S402 specifically includes: determining the degree of distraction of the driver according to the head pose angle; and determining the distracted state of the driver according to the degree of distraction of the driver.

[0100] It should be noted that the distraction level can also be determined by counting the frequency of re-distraction, that is, by setting the angular thresholds v7 and v8 of the pitch angle. If the pitch angle in the head pose angle is greater than v7 or less than v8, it is considered that the driver is distracted, and the distraction cumulative count is added.

[0101] It can be understood that the statistical sliding window L is also set, and the proportion of the distraction cumulative count within the sliding window L is used as the distraction level of the driver. If the distraction level of the driver reaches or exceeds the preset threshold, it can be determined that the driver is in a distracted state. The preset threshold can also be 60%.

[0102] Step S403, determine the current driving state of the driver according to the fatigue state and the distraction state of the driver, and issue a warning according to the current driving state of the driver.

[0103] It should be noted that the current driving state of the driver can be a fatigue state, a distraction state, a combination of both fatigue and distraction states, or a normal driving state of neither fatigue nor distraction. After determining the current driving state of the driver, a corresponding warning will be issued according to the state. For example, when the driver is in a fatigue state, the system will issue an audible or visual warning to prompt the driver to rest or take measures to relieve fatigue. If the driver is in a distracted state, the system will also issue a warning, advising the driver to concentrate or adjust driving habits. When the driver is in both a fatigue and a distracted state, the system will issue a more urgent warning to ensure that the driver can take timely action to avoid potential dangers. If the driver is in a normal driving state, the system will not issue a warning to ensure the normal progress of the driving process. In this way, this embodiment can provide comprehensive driving state monitoring and warning for the driver, thereby effectively improving driving safety.

[0104] As Figure 3 shown, Figure 3 is the overall flowchart of driving state monitoring. The video frame is input into the vehicle-mounted controller for activation condition detection. When the activation condition is met, image preprocessing is performed. Face detection is performed based on the preprocessed image, and then the detected face is judged for face validity. If it is judged as a valid face, face key point detection is performed. Occlusion judgment is performed according to the face key point detection result, and then eye closure degree calculation and mouth opening degree calculation are performed. Then, the frequency of eye closure and the frequency of yawning are counted respectively. Fatigue threshold discrimination is performed according to the statistical results. If the fatigue threshold is exceeded, a fatigue alarm is issued; Head pose calculation can also be performed according to the face key point detection result, and then distraction threshold discrimination is performed according to the head pose. If the distraction threshold is exceeded, a distraction alarm is issued.

[0105] In this embodiment, the fatigue state and distraction state of the driver are respectively determined by the facial opening degree and the head posture angle, and then the current driving state of the driver is comprehensively determined, and early warnings are given in a timely and accurate manner, effectively improving the accuracy of driver monitoring and early warning.

[0106] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the driving state monitoring method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0107] Based on the first embodiment of this application, in the third embodiment of this application, the same or similar content as in the above-mentioned embodiment one can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 4 , the step S30 further includes steps S301 - S302:

[0108] Step S301, determine the eye key point coordinates, mouth key point coordinates, and target key point coordinates based on the face key point coordinates.

[0109] It should be noted that the eye key point coordinates are determined based on the face key point coordinates, and specifically may include the key point coordinates of the left eye and the right eye, which are used for subsequent calculation of the eye opening degree. The mouth key point coordinates are also determined based on the face key point coordinates, including the key point coordinates of the left and right mouth corners, which are used for calculating the mouth opening degree.

[0110] It can be understood that the target key point coordinates are the 9 key points preset in the face key point coordinates. For example, key points such as the tip of the nose, the outer corner of the left eye, the outer corner of the right eye, the left mouth corner, and the right mouth corner are not specifically limited in this embodiment.

[0111] Step S302, determine the facial opening degree and the head posture angle according to the eye key point coordinates, the mouth key point coordinates, and the target key point coordinates.

[0112] It should be noted that the eye key point coordinates and the mouth key point coordinates are used to determine the facial opening degree, and the target key point coordinates are used to determine the head posture angle. Through the coordinates of these key points, the tilt angle and rotation angle of the head can be calculated using geometric relationships and trigonometric functions, which are not specifically limited in this embodiment.

[0113] In a feasible implementation manner, step S302 specifically includes: determining the eye opening degree according to the eye key point coordinates, and determining the mouth opening degree according to the mouth key point coordinates; determining the facial opening degree according to the eye opening degree and the mouth opening degree; determining the rotation matrix according to the target key point coordinates, and decomposing the rotation matrix to obtain the head posture angle.

[0114] It should be noted that calculating the eye opening degree based on the eye key point coordinates, including the left eye opening degree and the right eye opening degree, can be achieved by measuring the vertical distance between the upper eyelid and the lower eyelid. For example, for the left eye, take the highest point of the left upper eyelid and the lowest point of the lower eyelid, and calculate the vertical distance between these two points; for the right eye, similarly take the highest point of the right upper eyelid and the lowest point of the lower eyelid, and calculate the vertical distance between these two points. These two distances are respectively used as the opening degree values of the left eye and the right eye.

[0115] It can be understood that calculating the mouth opening degree based on the mouth key point coordinates can be achieved by measuring the horizontal distance between the corners of the mouth. For example, take the coordinate points of the left corner of the mouth and the right corner of the mouth, and calculate the horizontal distance between these two points, and this distance is the mouth opening degree. This embodiment does not make specific limitations on this.

[0116] It is worth noting that according to 9 key points preset in the face key point coordinates, that is, the target key points, calculate the rotation matrix from the camera to the 3D coordinate system, and decompose the rotation matrix to calculate the Euler angles, that is, the head pose angles. The Euler angles include the pitch angle, the yaw angle, and the roll angle, which respectively correspond to the three main rotation directions of the head in three-dimensional space. By calculating these angles, the tilt and rotation states of the driver's head can be accurately judged, and then the driver's attention concentration can be evaluated. For example, if the pitch angle is too large, it may indicate that the driver is looking down at the phone or performing other distracting behaviors; if the yaw angle or the roll angle is too large, it may indicate that the driver's head is turned to one side and the attention is not on the road ahead.

[0117] In this embodiment, by accurately determining the facial opening degree according to the eye key point coordinates and the mouth key point coordinates, and accurately determining the head pose angle according to the target key point coordinates, and then combining them to accurately identify the current driving state of the driver, the driving safety is effectively improved.

[0118] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the driving state monitoring method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0119] This application also provides a driving state monitoring device. Please refer to Figure 5 , the driving state monitoring device includes:

[0120] The detection module 10 is used to collect the driver's image and perform face detection based on the driver's image to obtain a face image.

[0121] The detection module 10 is also used to perform face key point detection based on the face image to obtain face key point coordinates.

[0122] A determination module 20, configured to determine the facial opening degree and the head pose angle according to the facial key point coordinates;

[0123] An early warning module 30, configured to determine the current driving state of the driver according to the facial opening degree and the head pose angle, and perform an early warning according to the current driving state of the driver.

[0124] This embodiment provides a driving state monitoring device. This embodiment collects a driver image, performs face detection based on the driver image to obtain a face image; performs facial key point detection on the face image to obtain facial key point coordinates; determines the facial opening degree and the head pose angle according to the facial key point coordinates; determines the current driving state of the driver according to the facial opening degree and the head pose angle, and performs an early warning according to the current driving state of the driver, which can accurately identify the driving state of the driver and perform a safety early warning in time, effectively improving driving safety.

[0125] In summary, in this embodiment, by collecting a driver image and performing face detection, the facial state of the driver can be monitored in real time. Furthermore, by performing facial key point detection on the face image, the facial area can be accurately identified, and the facial opening degree and the head pose angle can be quickly and accurately determined according to the facial key point coordinates. Thus, the current driving state of the driver can be accurately identified according to the facial opening degree and the head pose angle and an early warning can be performed in time, which can effectively reduce false state reports, effectively improve driving safety, overcome the technical defects of poor accuracy in determining the fatigue state by detecting whether the hand leaves the steering wheel and detecting the heartbeat, and being unable to effectively judge the distracted situation of the driver, resulting in low driving safety, and can accurately identify the driving state of the driver and perform a safety early warning in time, effectively improving driving safety.

[0126] Optionally, the early warning module 30 is further configured to determine the fatigue state of the driver according to the facial opening degree; determine the distracted state of the driver according to the head pose angle; determine the current driving state of the driver according to the fatigue state and the distracted state of the driver, and perform an early warning according to the current driving state of the driver.

[0127] Optionally, the early warning module 30 is further configured to count the closing frequency of eyes according to the eye opening degree in the facial opening degree to obtain closing frequency information; count the yawning frequency according to the mouth opening degree in the facial opening degree to obtain yawning frequency information; determine the fatigue state of the driver according to the closing frequency information and the yawning frequency information.

[0128] Optionally, the early warning module 30 is further configured to determine the closing fatigue according to the closing frequency information; determine the yawning fatigue according to the yawning frequency information; determine the fatigue state of the driver according to the closing fatigue and / or the yawning fatigue.

[0129] Optionally, the warning module 30 is further configured to determine the distraction level of the driver according to the head posture angle; and determine the distraction state of the driver according to the distraction level of the driver.

[0130] Optionally, the determination module 20 is further configured to determine the eye key point coordinates, the mouth key point coordinates, and the target key point coordinates based on the face key point coordinates; and determine the facial opening degree and the head posture angle according to the eye key point coordinates, the mouth key point coordinates, and the target key point coordinates.

[0131] Optionally, the determination module 20 is further configured to determine the eye opening degree according to the eye key point coordinates, and determine the mouth opening degree according to the mouth key point coordinates; determine the facial opening degree according to the eye opening degree and the mouth opening degree; determine the rotation matrix according to the target key point coordinates, and decompose the rotation matrix to obtain the head posture angle.

[0132] Optionally, the detection module 10 is further configured to collect a driver image, preprocess the driver image to obtain a preprocessed driver image, where the preprocessing at least includes picture format conversion and picture size scaling; perform face detection based on the preprocessed driver image to obtain a face outer contour rectangular coordinate frame and a face confidence level; and determine a face image based on the face outer contour rectangular coordinate frame and the face confidence level.

[0133] Optionally, the detection module 10 is further configured to determine whether the driver image is a valid image based on the face confidence level; and when the driver image is a valid image, crop the driver image according to the face outer contour rectangular coordinate frame to obtain a face image.

[0134] The driving state monitoring device provided by this application adopts the driving state monitoring method in the above embodiment, and can solve the technical problems that the accuracy of determining the fatigue state by detecting the departure of the hand from the steering wheel and detecting the heartbeat is poor, and the distraction situation of the driver cannot be effectively judged, resulting in low driving safety. Compared with the prior art, the beneficial effects of the driving state monitoring device provided by this application are the same as those of the driving state monitoring method provided by the above embodiment, and other technical features in the driving state monitoring device are the same as the features disclosed in the method of the above embodiment, and will not be elaborated here.

[0135] This application provides a driving state monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the driving state monitoring method in the first embodiment above.

[0136] Refer to the following Figure 6 , which shows a schematic structural diagram of a driving state monitoring device suitable for implementing the embodiments of the present application. The driving state monitoring device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The driving state monitoring device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0137] As Figure 6As shown, the driving state monitoring device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the driving state monitoring device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the driving state monitoring device to communicate with other devices wirelessly or wiredly to exchange data. Although the driving state monitoring device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems can be implemented or had.

[0138] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0139] The driving state monitoring device provided by the present application adopts the driving state monitoring method in the above-mentioned embodiment, which can solve the technical problems of poor accuracy in determining the fatigue state through steering wheel hand-off detection and heartbeat detection, and the inability to effectively judge the driver's distraction situation, resulting in low driving safety. Compared with the prior art, the beneficial effects of the driving state monitoring device provided by the present application are the same as those of the driving state monitoring method provided by the above-mentioned embodiment, and other technical features in the driving state monitoring device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0140] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0141] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0142] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the driving state monitoring method in the above embodiments.

[0143] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] The above computer-readable storage medium can be included in the driving state monitoring device; it can also exist separately without being assembled into the driving state monitoring device.

[0145] The above computer-readable storage medium stores one or more programs, which, when executed by a driving state monitoring device, cause the driving state monitoring device to: collect a driver image, perform face detection based on the driver image to obtain a face image; perform face key point detection based on the face image to obtain face key point coordinates; determine a facial opening degree and a head pose angle according to the face key point coordinates; determine the current driving state of the driver according to the facial opening degree and the head pose angle, and issue a warning according to the current driving state of the driver.

[0146] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through an Internet service provider using the Internet).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0148] The modules described in the embodiments of this application may be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0149] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned driving state monitoring method, which can solve the technical problems that the accuracy of determining the fatigue state through steering wheel off-hand detection and heartbeat detection is poor, and the distraction situation of the driver cannot be effectively judged, resulting in low driving safety. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the driving state monitoring method provided by the above embodiments, and will not be elaborated here.

[0150] This application also provides a computer program product, including a computer program, and the steps of the above-mentioned driving state monitoring method are implemented when the computer program is executed by a processor.

[0151] The computer program product provided by this application can solve the technical problems that the accuracy of determining the fatigue state through steering wheel off-hand detection and heartbeat detection is poor, and the distraction situation of the driver cannot be effectively judged, resulting in low driving safety. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the driving state monitoring method provided by the above embodiments, and will not be elaborated here.

[0152] The above are only some embodiments of this application, and thus do not limit the patent scope of this application. Any equivalent structural transformation made by using the content of the specification and drawings of this application under the technical concept of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. A driving state monitoring method, characterized in that: The method comprises: Collecting a driver image, and performing face detection based on the driver image to obtain a face image; Perform facial key point detection based on the facial image to obtain facial key point coordinates; Determine the facial opening and closing degree and the head posture angle according to the coordinates of the facial key points; The current driving state of the driver is determined according to the facial opening and closing degree and the head posture angle, and an early warning is issued according to the current driving state of the driver.

2. The method according to claim 1, characterized in that The determining the current driving state of the driver according to the facial opening and closing degree and the head posture angle, and issuing an early warning according to the current driving state of the driver, comprises: determining the driver's fatigue state according to the facial opening and closing degree; determining a driver's distraction state based on the head posture angle; The current driving state of the driver is determined according to the fatigue state of the driver and the distraction state of the driver, and an early warning is issued according to the current driving state of the driver.

3. The method according to claim 2, characterized in that The determining the driver's fatigue state according to the facial opening and closing degree includes: Performing eye closing frequency statistics according to the eye opening degree in the facial opening degree to obtain eye closing frequency information; Counting the yawning frequency according to the mouth opening degree in the facial opening degree to obtain yawning frequency information; The driver's fatigue state is determined according to the eye closing frequency information and the yawning frequency information.

4. The method according to claim 3, characterized in that The determining the driver's fatigue state according to the eye closing frequency information and the yawning frequency information includes: determining eye closing fatigue according to the eye closing frequency information; determining yawning fatigue according to the yawning frequency information; The driver's fatigue state is determined according to the eye-closing fatigue and / or yawning fatigue.

5. The method according to claim 2, characterized in that The determining the driver's distraction state according to the head posture angle comprises: determining a driver's distraction level based on the head posture angle; The driver's distraction state is determined based on the driver's distraction level.

6. The method according to claim 1, characterized in that The determining of the facial opening and closing degree and the head posture angle according to the coordinates of the facial key points includes: Determine eye key point coordinates, mouth key point coordinates and target key point coordinates based on the face key point coordinates; The facial opening degree and the head posture angle are determined according to the eye key point coordinates, the mouth key point coordinates and the target key point coordinates.

7. The method according to claim 6, characterized in that The determining of the facial opening and closing degree and the head posture angle according to the eye key point coordinates, the mouth key point coordinates and the target key point coordinates includes: Determine the degree of eye opening and closing according to the eye key point coordinates, and determine the degree of mouth opening and closing according to the mouth key point coordinates; Determining the facial opening degree according to the eye opening degree and the mouth opening degree; A rotation matrix is ​​determined according to the target key point coordinates, and the rotation matrix is ​​decomposed to obtain the head posture angle.

8. The method according to claim 1, characterized in that The collecting of the driver image and performing face detection based on the driver image to obtain the face image includes: Acquire a driver image, and preprocess the driver image to obtain a preprocessed driver image, wherein the preprocessing at least includes image format conversion and image size scaling; Performing face detection based on the preprocessed driver image to obtain a rectangular coordinate frame of the face outer contour and a face confidence level; A face image is determined based on the face outer contour rectangular coordinate frame and the face confidence.

9. The method according to claim 8, characterized in that The determining of the face image based on the face outer contour rectangular coordinate frame and the face confidence level includes: Determining whether the driver image is a valid image based on the face confidence; When the driver image is a valid image, the driver image is cropped according to the facial outer contour rectangular coordinate frame to obtain a facial image.

10. A driving status monitoring device, characterized in that: The driving state monitoring device comprises: A detection module, used for collecting a driver image and performing face detection based on the driver image to obtain a face image; The detection module is further used to detect facial key points based on the facial image to obtain the coordinates of the facial key points; A determination module, used to determine the facial opening and closing degree and the head posture angle according to the coordinates of the facial key points; The early warning module is used to determine the current driving state of the driver according to the facial opening and closing degree and the head posture angle, and to issue an early warning according to the current driving state of the driver.

11. A driving status monitoring device, characterized in that: The driving state monitoring device comprises: a memory, a processor, and a driving state monitoring program stored in the memory and executable on the processor, wherein the driving state monitoring program is configured to implement the driving state monitoring method according to any one of claims 1 to 9.

12. A storage medium, characterized in that: The storage medium stores a driving state monitoring program, which, when executed by a processor, implements the driving state monitoring method according to any one of claims 1 to 9.

Images