A driving state monitoring method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a driving state monitoring method and device, electronic equipment and storage medium, and relate to the technical field of intelligent networking. The method comprises: obtaining and preprocessing a face image of a driver to obtain a face contour image; performing feature point detection on the face contour image to obtain face feature points; calculating the degree of closure of the driver's mouth using mouth feature points and performing fatigue state judgment; calculating the degree of closure of the driver's eyes using eye feature points and performing fatigue state judgment; calculating the eyeball movement rate using the face image and performing fatigue state judgment using the eyeball movement rate; and triggering a warning to the driver based on the judgment result. The method evaluates the degree of driver fatigue in multiple dimensions such as the degree of closure of the mouth, the degree of closure of the eyes and the eyeball movement rate, improves the accuracy and reliability of the judgment result, and solves the problem of existing methods that have usage restrictions and inaccurate judgment results.

Description

Technical Field

[0001] The present application relates to the field of intelligent network technology, and specifically to a driving status monitoring method, device, electronic device and storage medium. Background Art

[0002] Existing driver status detection includes recognition based on driving behavior: judging the driver's fatigue level by monitoring their behavioral characteristics. For example, vehicle sensors record driving behavior data, including vehicle speed, steering wheel operation, acceleration, etc., to determine driving behaviors such as frequent lane changes and shaking; recognition based on physiological signals, such as heart rate, electroencephalogram, eye movement, electromyography, etc. to determine the driver's fatigue level; and recognition based on sound analysis to identify phenomena such as yawning, changes in speaking voice, and making phone calls.

[0003] Because everyone's physiological characteristics, behavioral habits, and expressions vary, recognition solutions based on driving behavior, physiological signals, and sound analysis have limited adaptability to different drivers. These recognition solutions may misjudge fatigue driving. For example, a recognition solution based on driving behavior may falsely report fatigue driving due to temporary circumstances (such as avoiding obstacles). Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a driving status monitoring method, device, electronic device and storage medium, which evaluate the driver's fatigue level through multiple dimensions such as the degree of mouth closure, the degree of eye closure and eye movement rate, improve the accuracy and reliability of the judgment results, and solve the problem that the existing methods have usage limitations and inaccurate judgment results.

[0005] An embodiment of the present application provides a driving state monitoring method, the method comprising:

[0006] Acquire and pre-process the driver's facial image to obtain a facial contour image;

[0007] Performing feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points;

[0008] Use mouth feature points to calculate the degree of closure of the driver's mouth and determine fatigue status;

[0009] Use eye feature points to calculate the degree of driver's eye closure and determine fatigue status;

[0010] Calculating an eye movement rate using the facial image, and using the eye movement rate to determine a fatigue state;

[0011] Based on the judgment results, a warning is triggered to the driver.

[0012] In the above implementation process, feature point detection is performed on the facial contour image to identify the mouth feature points and eye feature points. The driver's fatigue level is assessed through multiple dimensions such as the degree of mouth closure, the degree of eye closure, and the eye movement rate, thereby improving the accuracy and reliability of the judgment results, realizing real-time detection and early warning of the driver's safe driving, and solving the problem that the existing methods have usage limitations and inaccurate judgment results.

[0013] Furthermore, the calculation of the degree of closure of the driver's mouth using the mouth feature points and the determination of fatigue status include:

[0014] Calculate the driver's mouth aspect ratio using mouth feature points:

[0015]

[0016] Among them, p1 and p4 represent the feature points of the corners of the mouth, p2 and p3 represent the feature points of the outer contour of the upper lip, and p5 and p6 represent the feature points of the outer contour of the lower lip;

[0017] If the mouth aspect ratio is greater than a first set value and the duration is greater than a first time threshold, the state is fatigued.

[0018] In the above implementation process, if the mouth aspect ratio value is large and lasts for a long time, the driver may be yawning, and the driver's fatigue state can be assessed by using the mouth aspect ratio.

[0019] Furthermore, the method of calculating the degree of closure of the driver's eyes using eye feature points and determining the fatigue state includes:

[0020] Calculate the driver's eye aspect ratio using eye feature points:

[0021]

[0022] Among them, p7 and p10 represent the feature points of the corners of the eyes, p8 and p9 represent the feature points of the outer contour of the upper eyelid, and p11 and p12 represent the feature points of the outer contour of the lower eyelid;

[0023] If the eye aspect ratio is less than the second set value and the duration is greater than the second time threshold, the eye is in a fatigue state.

[0024] In the above implementation process, a smaller eye aspect ratio indicates that the eyes are in a closed state. If the eye closure lasts for a long time, it indicates that the eyes are currently in a fatigue state, thereby realizing the assessment of the driver's fatigue state by using the eye aspect ratio.

[0025] Furthermore, the calculating of the eye movement rate using the facial image and the determination of fatigue status using the eye movement rate include:

[0026] Segmenting an eye region image from the facial image and performing morphological operations to obtain a binary eye image;

[0027] Using the eye binary image to locate the iris center and the eye corner, respectively obtaining the iris center coordinates and the eye corner coordinates;

[0028] The eye movement distance L and eye movement rate v of two adjacent frames of images are obtained based on the iris center coordinates and eye corner coordinates:

[0029] v=L / t;

[0030] Where, L = |L1-L2|, Where L1 represents the distance from the eyeball to the reference eye corner at the current moment, (x1, y1) represents the coordinates of the iris center at the current moment, (x2, y2) represents the coordinates of the reference eye corner at the current moment, L2 represents the distance from the eyeball to the reference eye corner at the next moment, (x3, y3) represents the coordinates of the iris center at the next moment, (x4, y4) represents the coordinates of the reference eye corner at the next moment, and t represents the time interval between two adjacent frames of images;

[0031] If the eye movement rate is less than the set rate, it is in a fatigue state.

[0032] In the above implementation process, when a person is in a fatigue state, the eye movement speed will slow down, so the fatigue state can be judged based on the eye movement rate.

[0033] Furthermore, the method further comprises:

[0034] Obtaining the coordinates of the highest point and the lowest point of the eyeball using the eye binary image;

[0035] The eye gaze score is calculated using the coordinates of the highest point of the eyeball and the coordinates of the lowest point of the eyeball, and the concentration is judged using the gaze score.

[0036] In the above implementation process, the gaze score is used to judge the driver's concentration, which increases the evaluation dimension of the driver's status and improves driving safety.

[0037] Furthermore, the calculating of the eye gaze score using the coordinates of the highest point of the eyeball and the coordinates of the lowest point of the eyeball, and using the gaze score to judge concentration, includes:

[0038] The gaze score is expressed as:

[0039]

[0040] Among them, (x5, y5) represents the coordinates of the left corner of the eye, (x7, y7) represents the coordinates of the right corner of the eye, (x6, y6) represents the coordinates of the highest point of the eyeball, (x8, y8) represents the coordinates of the lowest point of the eyeball, (x9, y9) represents the coordinates of the center of the eyeball, and (x10, y10) represents the coordinates of the center of the iris;

[0041] If the gaze score is less than the gaze score threshold, it is determined that the driver's concentration is low.

[0042] In the above implementation process, the gaze score can be used to represent a person's attention and alertness level. Therefore, the gaze score can be used to evaluate the driver's concentration, thereby realizing the judgment of the driver's concentration.

[0043] The present application also provides a driving state monitoring device, comprising:

[0044] An image acquisition module is used to acquire and pre-process the driver's facial image to obtain a facial contour image;

[0045] A feature point detection module, configured to perform feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points;

[0046] A first fatigue judgment module is used to calculate the degree of closure of the driver's mouth using mouth feature points and to judge the fatigue state;

[0047] The second fatigue judgment module is used to calculate the degree of closure of the driver's eyes using eye feature points and to judge the fatigue state;

[0048] a third fatigue judgment module, configured to calculate an eye movement rate using the facial image, and use the eye movement rate to judge a fatigue state;

[0049] The warning module is used to trigger a warning to the driver based on the judgment result.

[0050] In the above implementation process, feature point detection is performed on the facial contour image to identify the mouth feature points and eye feature points. The driver's fatigue level is assessed through multiple dimensions such as the degree of mouth closure, the degree of eye closure, and the eye movement rate, thereby improving the accuracy and reliability of the judgment results, realizing real-time detection and early warning of the driver's safe driving, and solving the problem that the existing methods have usage limitations and inaccurate judgment results.

[0051] Furthermore, the first fatigue judgment module includes:

[0052] Aspect ratio calculation module, used to calculate the driver's mouth aspect ratio using mouth feature points:

[0053]

[0054] Among them, p1 and p4 represent the feature points of the corners of the mouth, p2 and p3 represent the feature points of the outer contour of the upper lip, and p5 and p6 represent the feature points of the outer contour of the lower lip;

[0055] The fatigue determination module is configured to determine that the user is in a fatigue state if the aspect ratio of the mouth is greater than a first set value and the duration is greater than a first time threshold.

[0056] In the above implementation process, if the mouth aspect ratio value is large and lasts for a long time, the driver may be yawning, and the driver's fatigue state can be assessed by using the mouth aspect ratio.

[0057] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute any of the above-mentioned driving status monitoring methods.

[0058] An embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, any one of the above-mentioned driving state monitoring methods is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A flowchart of a driving status monitoring method provided in an embodiment of the present application;

[0061] Figure 2 A specific flow chart for fatigue driving and concentration recognition provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a facial contour image provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of facial feature points provided in an embodiment of the present application;

[0064] Figure 5 A flowchart of fatigue state determination based on mouth closure degree provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of mouth feature points provided in an embodiment of the present application;

[0066] Figure 7 A flowchart of fatigue state determination based on eye closure degree provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of eye feature points provided in an embodiment of the present application;

[0068] Figure 9 A flowchart of fatigue state determination based on eye movement rate provided in an embodiment of the present application;

[0069] Figure 10 A schematic diagram of a binarized eye image provided in an embodiment of the present application;

[0070] Figure 11 A schematic diagram of locating the center point of the iris of an eye provided in an embodiment of the present application;

[0071] Figure 12 A schematic diagram of eye corner positioning provided in an embodiment of the present application;

[0072] Figure 13 A schematic diagram of the motion state of a single eye in two consecutive frames provided in an embodiment of the present application;

[0073] Figure 14 A flow chart for determining concentration provided in an embodiment of the present application;

[0074] Figure 15 A schematic diagram of eye feature point coordinates provided in an embodiment of the present application;

[0075] Figure 16 A structural block diagram of a driving status monitoring device provided in an embodiment of the present application;

[0076] Figure 17 This is a structural block diagram of another driving status monitoring device provided in an embodiment of the present application.

[0077] icon:

[0078] 100-image acquisition module; 200-feature point detection module; 300-first fatigue judgment module; 301-aspect ratio calculation module; 302-fatigue judgment module; 400-second fatigue judgment module; 500-third fatigue judgment module; 600-early warning module; 700-concentration judgment module. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0080] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0081] Example 1

[0082] Please see Figure 1 , Figure 1 A flow chart of a driving status monitoring method provided in an embodiment of the present application. The present application identifies mouth and eye features based on facial feature detection, and then calculates the eye movement rate based on the eye feature points, and combines the training and judgment of the fatigue driving classification model and the concentration classification model to achieve real-time detection and early warning of driver safe driving. This method can achieve non-contact monitoring by analyzing facial feature points in facial images, and calculates the mouth aspect ratio, eye aspect ratio, eye movement rate and gaze score through real-time video stream analysis, so as to obtain fatigue and concentration recognition results more quickly and comprehensively, evaluate the driver's fatigue and concentration level from multiple dimensions, and improve accuracy and reliability.

[0083] The method specifically comprises the following steps:

[0084] Step S100: Acquire and pre-process the driver's facial image to obtain a facial contour image;

[0085] like Figure 2 The figure shows a specific flow chart for fatigue driving and concentration recognition. First, a camera facing the driver's face is installed inside the vehicle to capture the driver's facial image in real time. The collected video stream or image is preprocessed, including but not limited to: illumination compensation, grayscale, smoothing and denoising, contrast enhancement, etc., in order to more accurately identify various facial areas and obtain a facial contour image, such as Figure 3 The figure shows a schematic diagram of a facial contour image.

[0086] Step S200: performing feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points;

[0087] For example, the feature point detection algorithm in Dlib can be used to detect the key feature points of the driver's face, thereby identifying the position of the feature points in the face, where the key feature points may include eyes, mouth, etc. Figure 4 The following is a schematic diagram of facial feature points.

[0088] Step S300: Calculating the degree of closure of the driver's mouth using mouth feature points and determining fatigue status;

[0089] Mouth feature point analysis: Using the feature points of the mouth area, the aspect ratio of the mouth is calculated, that is, the ratio of the vertical distance of the mouth to the horizontal distance of the mouth, thereby representing the degree of closure of the mouth.

[0090] Step S400: Calculating the degree of closure of the driver's eyes using eye feature points and determining fatigue status;

[0091] Eye feature point analysis: After detecting a face, the eye region can be further extracted to reduce the complexity and computational effort of subsequent processing. For example, a specific eye feature point location algorithm (such as Dlib's 68-point landmarks or a model specifically designed for eye features) can be used to determine the coordinates of eye feature points, such as the eye contour and the upper and lower eyelid edges. This can then be used to calculate the degree of eye closure. Specifically, the eye aspect ratio (the ratio of the vertical distance to the horizontal distance of the eye) is calculated to determine the eye's open or closed state.

[0092] Step S500: calculating the eye movement rate using the facial image, and using the eye movement rate to determine the fatigue state;

[0093] Pupil feature point analysis: The eye region is cut out from the facial image and processed by MediaPipe Iris to determine pupil feature points. The rate of eye movement is calculated by locating two feature points: the iris center and the corner of the eye.

[0094] Step S600: triggering a warning to the driver based on the judgment result.

[0095] Facial feature motion data is analyzed to identify fatigue-related indicators, including the degree of mouth closure, eye closure, and eye movement rate. These indicators are evaluated based on research or preset thresholds. When they exceed a certain threshold, the driver may be considered fatigued. Based on these analysis results, the system comprehensively determines the driver's condition. If fatigue or low concentration is detected, an alarm can be triggered or the driver can be prompted to take action, achieving real-time detection and early warning of the driver's condition.

[0096] like Figure 5 FIG. 1 is a flowchart of fatigue state determination based on the degree of mouth closure, wherein step S300 specifically includes the following steps:

[0097] Step S301: Calculate the driver's mouth aspect ratio using mouth feature points:

[0098]

[0099] Among them, p1 and p4 represent the feature points of the corners of the mouth, p2 and p3 represent the feature points of the outer contour of the upper lip, and p5 and p6 represent the feature points of the outer contour of the lower lip;

[0100] Step S302: If the mouth aspect ratio is greater than a first set value and the duration is greater than a first time threshold, the state is fatigued.

[0101] like Figure 6 The figure shows a schematic diagram of the mouth feature points. Generally, the aspect ratio of the mouth corresponds to different values ​​when a person is in different states, such as quiet, talking, singing, etc. For example, the mouth is closed when quiet; the mouth is half-open when speaking and talking normally; and when yawning, the mouth is opened wide and lasts for a certain period of time. In order to reduce false positives and improve the accuracy of model judgment, a distinction is made between normal speaking and yawning behaviors when the mouth is open when tired. When a person speaks normally, his mouth opens and closes very frequently, while when yawning, the aspect ratio of the mouth is large and the duration is long (greater than 4s). For example, the duration can be counted by a counter. When the aspect ratio of the mouth is greater than 0.3, the current frame is considered to be in the open mouth state, and the counter is incremented by 1. Once the value of the counter is greater than the first set value, that is, the mouth is open for a long time, it is determined that the current mouth is in a fatigued state.

[0102] like Figure 7 FIG. 4 is a flow chart showing fatigue state determination based on eye closure degree, wherein step S400 specifically includes the following steps:

[0103] Step S401: Calculate the driver's eye aspect ratio using eye feature points:

[0104]

[0105] Among them, p7 and p10 represent the feature points of the corners of the eyes, p8 and p9 represent the feature points of the outer contour of the upper eyelid, and p11 and p12 represent the feature points of the outer contour of the lower eyelid;

[0106] Step S402: If the eye aspect ratio is smaller than the second set value and the duration is greater than the second time threshold, the eye is in a fatigue state.

[0107] like Figure 8The figure shows a schematic diagram of eye feature points. To reduce false positives and improve the accuracy of model judgment, it is necessary to distinguish between blinking in normal conditions and closing the eyes during fatigue. The closing time of blinking in normal conditions is extremely short, while the closing time during fatigue is relatively long. When blinking in normal conditions, the eye aspect ratio is only at 0 for a moment (such as 1 to 3 frames). Therefore, in addition to determining the eye aspect ratio, the duration of eye closure must also be detected. The duration can be counted by a counter. When the eye aspect ratio is less than 0.2, the current frame is considered to be in the closed eye state, and the counter is incremented by 1. Once the counter value is greater than the second set value, that is, the eye closure time is longer, the current eye is judged to be in the fatigue state.

[0108] like Figure 9 FIG. 5 is a flow chart showing fatigue state determination based on eye movement rate, wherein step S500 specifically includes the following steps:

[0109] Step S501: segmenting the eye region image from the facial image and performing morphological operations to obtain a binary eye image;

[0110] Specifically, the eyeball is first located and segmented: the approximate position of the eye is quickly found using the Haar cascade classifier, and the eye area is further refined through morphological operations (such as dilation and erosion) and threshold segmentation to obtain a binary image of the eye, such as Figure 10 The figure shows a schematic diagram of a binarized eye image.

[0111] Step S502: locating the iris center and eye corner using the binarized eye image to obtain the coordinates of the iris center and eye corner respectively;

[0112] Iris center location: The eyelid contour is tracked to find the center of the eye to locate the center of the eye iris. That is, the iris center location is to project the binary image of the eye onto the X-axis and Y-axis respectively, and take the minimum cumulative value. The outer edge of the iris is located by first using the Canny operator for edge detection and then using the Hough transform to achieve positioning, such as Figure 11 The figure shows a schematic diagram of positioning the center point of the iris of the eye.

[0113] Corner of eye location: Use Harris algorithm to detect corners and locate the corners of the eye. Find the contours in the binary image. Since the detected corners may contain some pseudo corners, Harris corner detection is performed by combining the contour information to remove these pseudo corners. Then the obtained corners are calibrated, such as Figure 12 The figure shows a schematic diagram of eye corner positioning.

[0114] Step S503: Obtain the eye movement distance L and eye movement velocity v of two adjacent frames of images based on the iris center coordinates and eye corner coordinates:

[0115] v = L / t;

[0116] wherein, L = |L1-L2|, wherein, L1 represents the distance from eyeball to reference eye corner (optional to take left eye corner or right eye corner as reference eye corner) at current time, (x1, y1) represents the iris center point coordinate at current time, (x2, y2) represents the reference eye corner coordinate at current time, L2 represents the distance from eyeball to reference eye corner at next time, (x3, y3) represents the iris center point coordinate at next time, (x4, y4) represents the reference eye corner coordinate at next time, t represents the time interval of adjacent two frames of images;

[0117] Step S504: if the eyeball movement rate is less than the rate setting value, then in the fatigue state.

[0118] After detecting the iris center and eye corner, the eyeball rate needs to be calculated. Since the time interval t between the camera collecting each frame of image and processing is almost the same, only the distance of the movement of the eyeball of adjacent two frames needs to be calculated. As shown in FIG. 6, it is a schematic diagram of the movement state of single eye of two consecutive frames. Under normal circumstances, when the driver drives the vehicle, the gaze will look at the rearview mirror and the side mirror from time to time, and the eyeball will rotate non-uniformly. When the person is in the fatigue state, the eyeball movement speed slows down (v < standard value). According to the eyeball movement rate, the fatigue state can be judged. Figure 13

[0119] As shown in FIG. 7, it is a flow chart for concentration judgment. The method further comprises: Figure 14

[0120] Step S701: obtaining the highest point coordinate of eyeball and the lowest point coordinate of eyeball by using the eye binary image;

[0121] Step S702: calculating the gaze score of eye by using the highest point coordinate of eyeball and the lowest point coordinate of eyeball, and judging the concentration by using the gaze score.

[0122] The line-of-sight attention estimation includes the gaze region estimation and the pupil gaze direction estimation, and the concentration target and the concentration degree are judged by estimating the line-of-sight direction or the gaze point position of the observer. The gaze score (GS) is an index for evaluating the attention and the alertness level, and provides information about the driver moving away from the line-of-sight in the non- turning head condition. When the GS is measured, the system records the movement characteristics of the eyes of the person, and calculates a gaze score according to a certain algorithm. The gaze score can be used to represent the attention and the alertness level of the person, and the higher the value is, the higher the attention and the alertness level are.

[0123] Specifically, as shown in FIG. 8, it is a schematic diagram of eye feature point coordinate: Figure 15 ​​​

[0124] The gaze score is expressed as:

[0125]

[0126] Among them, (x5, y5) represents the coordinates of the left corner of the eye, (x7, y7) represents the coordinates of the right corner of the eye, (x6, y6) represents the coordinates of the highest point of the eyeball, (x8, y8) represents the coordinates of the lowest point of the eyeball, (x9, y9) represents the coordinates of the center of the eyeball, and (x10, y10) represents the coordinates of the center of the iris;

[0127] If the gaze score is less than the gaze score threshold, it is determined that the driver's concentration is low.

[0128] For example, the gaze score threshold for concentration is set to 0.3. When the GS is less than 0.3, it is determined that the driver's concentration is low.

[0129] This application not only takes into account the characteristics of fatigue driving, but also uses eye characteristics as the basis for judging concentration, and uses multi-feature fusion to judge the driver's facial feature attention state.

[0130] The gaze score and eye movement rate are used to assess driver focus and fatigue. The gaze score reflects the driver's level of attention on the road ahead; a high gaze score indicates the driver's attention is focused on the right place. When fatigued, the driver's gaze deviates from the normal direction of travel, and their gaze behavior becomes erratic, resulting in a lower score. Combining eye tracking technology with advanced image processing algorithms, this technology can accurately analyze subtle changes in eye expression, providing a more comprehensive and accurate basis for judgment than relying solely on single parameters such as eyelid distance. This technology is not only suitable for detecting fatigue, but can also be used to assess driver attention allocation in complex driving environments, helping to improve overall driving safety.

[0131] Example 2

[0132] The embodiment of the present application provides a driving state monitoring device, such as Figure 16 FIG. 1 is a structural block diagram of a driving state monitoring device, which includes but is not limited to:

[0133] The image acquisition module 100 is used to acquire and pre-process the driver's facial image to obtain a facial contour image;

[0134] A feature point detection module 200 is used to perform feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points;

[0135] A first fatigue judgment module 300 is used to calculate the degree of closure of the driver's mouth using mouth feature points and to judge the fatigue state;

[0136] The second fatigue judgment module 400 is used to calculate the degree of closure of the driver's eyes using eye feature points and to judge the fatigue state;

[0137] a third fatigue judgment module 500, configured to calculate an eye movement rate using the facial image, and to judge a fatigue state using the eye movement rate;

[0138] The warning module 600 is used to trigger a warning to the driver based on the judgment result.

[0139] like Figure 17 FIG. 1 is a structural block diagram of another driving state monitoring device, wherein the first fatigue judgment module 300 includes:

[0140] The aspect ratio calculation module 301 is used to calculate the aspect ratio of the driver's mouth using the mouth feature points:

[0141]

[0142] Among them, p7 and p10 represent the feature points of the corners of the eyes, p8 and p9 represent the feature points of the outer contour of the upper eyelid, and p11 and p12 represent the feature points of the outer contour of the lower eyelid;

[0143] The fatigue determination module 302 is configured to determine that the user is in a fatigue state if the aspect ratio of the mouth is greater than a first set value and the duration is greater than a first time threshold.

[0144] The specific judgment process of the second fatigue judgment module 400 and the third fatigue judgment module 500 has been specifically described in Example 1 and will not be repeated here.

[0145] The device also includes a concentration judgment module 700, which is used to obtain the coordinates of the highest point of the eye and the lowest point of the eye using the binarized eye image; calculate the eye gaze score using the coordinates of the highest point of the eye and the coordinates of the lowest point of the eye, and use the gaze score to judge concentration.

[0146] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the driving state monitoring method described in Example 1.

[0147] An embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the driving state monitoring method described in Example 1 is executed.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0149] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0150] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0151] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0152] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0153] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A driving state monitoring method, characterized in that: The method comprises: Acquire and pre-process the driver's facial image to obtain a facial contour image; Performing feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points; Use mouth feature points to calculate the degree of closure of the driver's mouth and determine fatigue status; The eye feature points are used to calculate the degree of closure of the driver's eyes and to judge the fatigue state. Specifically, the eye feature points are used to calculate the aspect ratio of the driver's eyes: ;in, p 7 and p 10 represents the feature point of the corner of the eye, p 8 and p 9 represents the feature point of the upper eyelid outer contour, p 11 and p 12 represents the feature point of the lower eyelid outer contour; if the eye aspect ratio is less than the second set value and the duration is greater than the second time threshold, it is in a fatigue state; The eye movement rate is calculated using the facial image, and the eye movement rate is used to judge the fatigue state. Specifically, the eye area image is segmented from the facial image, and morphological operations are performed to obtain a binary eye image; the iris center and eye corner are located using the binary eye image to obtain the iris center coordinates and eye corner coordinates respectively; the eye movement distance between two adjacent frames of images is obtained based on the iris center coordinates and eye corner coordinates. L and eye movement rate v : ;in, , , ,in, L 1 represents the distance from the current eyeball to the reference eye corner, Indicates the coordinates of the iris center point at the current moment, Indicates the reference eye corner coordinates at the current moment, L 2 represents the distance from the eyeball to the reference eye corner at the next moment, Indicates the coordinates of the iris center point at the next moment, Indicates the reference eye corner coordinates at the next moment, t Indicates the time interval between two adjacent frames of images; if the eye movement rate is less than the set rate value, it is in a fatigue state; Based on the judgment results, a warning is triggered to the driver.

2. The driving state monitoring method according to claim 1, characterized in that: The method of calculating the degree of closure of the driver's mouth by using the mouth feature points and judging the fatigue state includes: Calculate the driver's mouth aspect ratio using mouth feature points: ； in, p 1 and p 4 represents the feature point of the corner of the mouth, p 2 and p 3 represents the feature point of the upper lip outer contour, p 5 and p 6 represents the feature point of the lower lip outer contour; If the mouth aspect ratio is greater than a first set value and the duration is greater than a first time threshold, the state is fatigued.

3. The driving state monitoring method according to claim 1, characterized in that: The method further comprises: Obtaining the coordinates of the highest point and the lowest point of the eyeball using the eye binary image; The eye gaze score is calculated using the coordinates of the highest point of the eyeball and the coordinates of the lowest point of the eyeball, and the concentration is judged using the gaze score.

4. The driving state monitoring method according to claim 3, characterized in that: The calculating of the eye gaze score by using the coordinates of the highest point of the eyeball and the coordinates of the lowest point of the eyeball, and using the gaze score to judge the concentration, includes: The gaze score is expressed as: ； in, Indicates the coordinates of the left eye corner, Indicates the coordinates of the right eye corner, Indicates the coordinates of the highest point of the eyeball, Indicates the coordinates of the lowest point of the eyeball, Represents the coordinates of the center point of the eyeball, Indicates the coordinates of the iris center point; If the gaze score is less than the gaze score threshold, it is determined that the driver's concentration is low.

5. A driving status monitoring device, characterized in that: The device comprises: An image acquisition module is used to acquire and pre-process the driver's facial image to obtain a facial contour image; A feature point detection module, configured to perform feature point detection on the facial contour image to obtain facial feature points, wherein the facial feature points include mouth feature points and eye feature points; A first fatigue judgment module is used to calculate the degree of closure of the driver's mouth using mouth feature points and to judge the fatigue state; The second fatigue judgment module is used to calculate the degree of closure of the driver's eyes using eye feature points and perform fatigue state judgment. Specifically, the aspect ratio of the driver's eyes is calculated using eye feature points: ;in, p 7 and p 10 represents the feature point of the corner of the eye, p 8 and p 9 represents the feature point of the upper eyelid outer contour, p 11 and p 12 represents the feature point of the lower eyelid outer contour; if the eye aspect ratio is less than the second set value and the duration is greater than the second time threshold, it is in a fatigue state; The third fatigue judgment module is used to calculate the eye movement rate using the facial image and use the eye movement rate to judge the fatigue state. Specifically, the module segments the eye area image from the facial image and performs morphological operations to obtain a binary eye image; uses the binary eye image to locate the iris center and the eye corner to obtain the iris center coordinates and the eye corner coordinates respectively; obtains the eye movement distance between two adjacent frames of images based on the iris center coordinates and the eye corner coordinates. L and eye movement rate v : ;in, , , ,in, L 1 represents the distance from the current eyeball to the reference eye corner, Indicates the coordinates of the iris center point at the current moment, Indicates the reference eye corner coordinates at the current moment, L 2 represents the distance from the eyeball to the reference eye corner at the next moment, Indicates the coordinates of the iris center point at the next moment, Indicates the reference eye corner coordinates at the next moment, t Indicates the time interval between two adjacent frames of images; if the eye movement rate is less than the set rate value, it is in a fatigue state; The warning module is used to trigger a warning to the driver based on the judgment result.

6. The driving state monitoring device according to claim 5, characterized in that: The first fatigue judgment module includes: Aspect ratio calculation module, used to calculate the driver's mouth aspect ratio using mouth feature points: ； in, p 1 and p 4 represents the feature point of the corner of the mouth, p 2 and p 3 represents the feature point of the upper lip outer contour, p 5 and p 6 represents the feature point of the lower lip outer contour; The fatigue determination module is configured to determine that the user is in a fatigue state if the aspect ratio of the mouth is greater than a first set value and the duration is greater than a first time threshold.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the driving state monitoring method according to any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the driving state monitoring method according to any one of claims 1 to 4 is executed.

Images