A driver status monitoring method based on machine vision

Abstract

The present invention discloses a driver status monitoring method based on machine vision, which relates to the field of machine vision technology and includes the following steps: capturing a facial image of the driver and extracting the facial contour; obtaining the center points of the upper and lower lips, the contours of the eyes, the pupils of the eyes, and the tip of the nose; establishing a reference line based on the coordinates of the center point of the lower lip and the tip of the nose, and calculating the coordinates of the center of the eyes based on the contours of the eyes; and monitoring the driver's head orientation, eye closure, and yawning. Based on image capture and processing, the present invention obtains facial key points through facial contours and monitors the driver's status by relying on the changing patterns of facial key points under biological posture. The method has high real-time, accuracy, and robustness, and can not only detect fatigue driving but also identify key behaviors such as distraction, thereby achieving comprehensive monitoring of the driver's multi-dimensional status and significantly improving driving safety.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a driver status monitoring method based on machine vision. Background Art

[0002] Currently, traditional driver status monitoring methods rely on physiological sensors such as EEG and heart rate, but these are highly invasive and uncomfortable, making them difficult to deploy widely. In recent years, the development of machine vision technology has provided a feasible path for non-contact, low-cost driver status monitoring.

[0003] However, existing vision-based methods suffer from poor real-time performance, low accuracy, and poor robustness, making them difficult to operate stably in various driving environments. Therefore, there is an urgent need for a highly robust, accurate, and real-time driver status monitoring method based on machine vision. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a driver status monitoring method based on machine vision, which solves the problems of insufficient real-time performance, accuracy and robustness in the prior art.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A driver status monitoring method based on machine vision comprises the following steps:

[0007] S1, capture the driver's facial image and extract the facial contour;

[0008] S2. Based on the facial contour, obtain the center points of the upper and lower lips, the outlines of the eyes, the pupils of both eyes, and the tip of the nose;

[0009] S3. Establish a reference line based on the coordinates of the lower lip center and the nose tip, and calculate the coordinates of the eye centers based on the eye contours.

[0010] S4. Monitor the driver's head orientation, eye closure, and yawning based on the reference line, eye center coordinates, nose tip coordinates, upper and lower lip center coordinates, and eye pupils.

[0011] Furthermore, the S1 includes the following sub-steps:

[0012] S11. Install an infrared camera and an RGB camera in front of the driver's seat of the vehicle to capture infrared and RGB images of the driver's face, respectively;

[0013] S12, performing grayscale conversion on the RGB image, and performing weighted synthesis of the infrared image and the grayscale converted RGB image based on the light intensity to obtain a facial enhanced grayscale image;

[0014] S13, using a 3×3 Laplacian operator as a convolution kernel, performing a convolution operation on the facial enhancement grayscale image to obtain a facial contour grayscale image;

[0015] S14, adaptively adjusting the binarization threshold, converting the facial contour grayscale image into a binary image with only facial contour lines, and obtaining the facial contour.

[0016] Furthermore, in said S12, before weighted synthesis of the infrared image and the grayscale transformed RGB image, they are scaled to the same size by upsampling or downsampling respectively;

[0017] The light intensity is calculated by the following formula:

[0018] ,

[0019] Among them, I avg is the light intensity, I RGB (i, j) is the grayscale value of the i-th row and j-th column of the RGB image after grayscale conversion and scaling, i is the row number, j is the column number, M is the number of row pixels of the RGB image after grayscale conversion and scaling, and N is the number of column pixels of the RGB image after grayscale conversion and scaling;

[0020] The expression of the weighted synthesis is:

[0021] ,

[0022] Among them, I syn (i, j) is the grayscale value of the i-th row and j-th column of the facial enhancement grayscale image, I RGB is the RGB image after grayscale transformation and scaling, I RE (i, j) is the grayscale value of the i-th row and j-th column of the scaled infrared image, and MAX() is the function that takes the maximum value.

[0023] Furthermore, the Laplace operator used in S13 is:

[0024] .

[0025] Furthermore, the method of adaptively adjusting the binarization threshold in S14 is:

[0026] Increase or decrease the binarization threshold so that the contours of both eyes in the converted binary image are exactly closed curves.

[0027] Furthermore, in S3, the expression of the reference straight line established by the coordinates of the center point of the lower lip and the tip of the nose is:

[0028] ,

[0029] ,

[0030] ,

[0031] ,

[0032] Among them, ξ1 is the first parameter of the reference line, x rf is the horizontal coordinate variable of the reference line, ξ2 is the second parameter of the reference line, y rf is the vertical coordinate variable of the reference line, ξ3 is the third parameter of the reference line, x nose is the horizontal coordinate of the nose tip, y nose is the vertical coordinate of the nose tip, x lip,2 is the horizontal coordinate of the center point of the lower lip, y lip,2 is the vertical coordinate of the center point of the lower lip.

[0033] Furthermore, in S3, the left eye center coordinates and the right eye center coordinates are calculated according to the pixel coordinates of the left eye contour and the pixel coordinates of the right eye contour respectively by the following formula:

[0034] ,

[0035] ,

[0036] Among them, x ec is the horizontal coordinate of the left or right eye center, y ec is the vertical coordinate of the left or right eye center, x n is the horizontal coordinate of the nth pixel of the left or right eye contour, y n is the vertical coordinate of the nth pixel of the left or right eye contour, n is the left or right eye contour pixel number, and K is the number of left or right eye contour pixels.

[0037] Furthermore, the step S4 uses the change in the distance from the left eye or right eye center coordinate to the reference straight line as a criterion for determining whether the driver turns his head left or right;

[0038] The change in the difference between the left and right eye center vertical coordinates is used as the criterion for the driver's left or right head tilt;

[0039] The change in the difference between the left or right eye center ordinate and the nose tip ordinate is used as the criterion for whether the driver is looking up or down.

[0040] The distance between the left or right eye center coordinates and the reference straight line is calculated using the following formula:

[0041] ,

[0042] Among them, d ec is the distance from the left or right eye center coordinate to the reference straight line, ξ1 is the first parameter of the reference straight line, ξ2 is the second parameter of the reference straight line, ξ3 is the third parameter of the reference straight line, xec is the horizontal coordinate of the left or right eye center, y ec The vertical coordinate of the center of the left or right eye.

[0043] Furthermore, the S4 monitors the closure of the driver's eyes by taking whether the pupils of both eyes fail to be extracted from the driver's facial image as a criterion for determining whether the driver's eyes are closed.

[0044] Furthermore, the S4 uses the change in the difference between the vertical coordinates of the center point of the upper lip and the center point of the lower lip as a criterion for judging whether the driver's mouth is open or closed, and monitors the driver's yawning condition.

[0045] The beneficial effects of the present invention are:

[0046] 1) Based on image acquisition and processing, this invention obtains facial key points through facial contours and monitors the driver's status by observing the changing patterns of these key points under biological posture. This system features high real-time performance, accuracy, and robustness, eliminates the need for wearing any equipment, and avoids the discomfort associated with traditional physiological sensors. It can not only detect driver fatigue but also identify key behaviors such as distraction, achieving comprehensive monitoring of the driver's multi-dimensional status and significantly improving driving safety.

[0047] 2) In low-light environments, the driver's profile information is enhanced by increasing the weight of the infrared image, allowing the system to take into account both strong and low light conditions, further improving robustness;

[0048] 3) When extracting facial contours, the binarization threshold is adaptively adjusted based on whether the closed eye contours are obtained, so that the facial contours are complete and clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a driver status monitoring method based on machine vision provided by an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of extracting the center points of the upper and lower lips, eye contours, eye pupils, and nose tip from a facial enhanced grayscale image according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of a driver turning his head left and right according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of a driver tilting his head left and right according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a driver looking up or down according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the opening and closing of a driver's mouth during yawning according to an embodiment of the present invention;

[0055] Among them, 1. Right eye contour, 2. Right eye pupil, 3. Left eye pupil, 4. Left eye contour, 5. Nose tip, 6. Upper lip center, 7. Lower lip center, 8. Right eye center before turning the head left or right, 9. Left eye center before turning the head left or right, 10. Right eye center after turning the head left or right, 11. Left eye center after turning the head left or right, 12. Reference line, 13. Right eye center before tilting the head left or right, 14. Left eye center before tilting the head left or right, 15. Right eye center after tilting the head left or right , 16. The center of the left eye after tilting the head left or right, 17. The center of the right eye before looking up or down, 18. The center of the left eye before looking up or down, 19. The tip of the nose before looking up or down, 20. The center of the right eye after looking up or down, 21. The center of the left eye after looking up or down, 22. The tip of the nose after looking up or down, 23. The center of the upper lip when the mouth is closed, 24. The center of the lower lip when the mouth is closed, 25. The center of the upper lip when the mouth is open, 26. The center of the lower lip when the mouth is open. DETAILED DESCRIPTION

[0056] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0057] like Figure 1 As shown, in one embodiment of the present invention, a driver status monitoring method based on machine vision includes the following steps:

[0058] S1, capturing the driver's facial image and extracting the facial contour, including the following steps:

[0059] S11. Install an infrared camera and an RGB camera in front of the driver's seat of the vehicle to capture infrared and RGB images of the driver's face, respectively.

[0060] S12. Perform grayscale transformation on the RGB image, and perform weighted synthesis of the infrared image and the grayscale transformed RGB image based on the light intensity to obtain a facial enhanced grayscale image.

[0061] In order to enable weighted synthesis of different images taken by different cameras, they need to be scaled to the same size by upsampling or downsampling before weighted synthesis.

[0062] The light intensity is calculated by the following formula:

[0063] ,

[0064] Among them, I avgis the light intensity, I RGB (i, j) is the grayscale value of the i-th row and j-th column of the RGB image after grayscale conversion and scaling, i is the row number, j is the column number, M is the number of row pixels of the RGB image after grayscale conversion and scaling, and N is the number of column pixels of the RGB image after grayscale conversion and scaling;

[0065] The expression of weighted synthesis is:

[0066] ,

[0067] Among them, I syn (i, j) is the grayscale value of the i-th row and j-th column of the facial enhancement grayscale image, I RGB is the RGB image after grayscale transformation and scaling, I RE (i, j) is the grayscale value of the i-th row and j-th column of the scaled infrared image, and MAX() is the function that takes the maximum value.

[0068] In low-light environments, the driver's profile information is enhanced by increasing the weight of the infrared image, allowing the system to take into account both strong and low light conditions, further improving robustness.

[0069] S13. Using a 3×3 Laplacian operator as a convolution kernel, perform a convolution operation on the facial enhancement grayscale image to obtain a facial contour grayscale image.

[0070] The Laplace operator used in this embodiment is:

[0071] .

[0072] This operator is an eight-neighborhood template. In addition to considering the four adjacent pixels in the upper, lower, left, and right directions, it also considers the adjacent pixels in the four diagonal directions. It has a good response to edges in all directions and can detect grayscale changes in a wider range of directions.

[0073] S14. Adaptively adjust the binarization threshold to convert the facial contour grayscale image into a binary image containing only the facial contour lines, thereby obtaining the facial contour. The method for adaptively adjusting the binarization threshold is to increase or decrease the binarization threshold so that the contours of both eyes in the converted binary image are exactly closed curves.

[0074] When extracting facial contours, the binarization threshold is adaptively adjusted based on whether the closed eye contours are obtained, making the facial contours complete and clear, and also suppressing the false edge contours that may appear when using the eight-neighborhood template Laplacian operator.

[0075] S2. According to the facial contour, the center points of the upper and lower lips, the outlines of the eyes, the pupils of the eyes and the tip of the nose are obtained. The schematic diagram is as follows Figure 2As shown in the figure, there are relatively mature tools and methods for obtaining the center points of the upper and lower lips, eye contours, eye pupils, and nose tip from the facial contour, including but not limited to Dlib, OpenCV, MTCNN, etc., which will not be described in detail in this invention.

[0076] After obtaining the center points of the upper and lower lips, the outlines of the eyes, the pupils of both eyes, and the tip of the nose, the row numbers and column numbers of these pixel points are used as the horizontal and vertical coordinates respectively.

[0077] S3. Create a reference line using the coordinates of the lower lip center point and the nose tip. The expression is as follows:

[0078] ,

[0079] ,

[0080] ,

[0081] ,

[0082] Among them, ξ1 is the first parameter of the reference line, x rf is the horizontal coordinate variable of the reference line, ξ2 is the second parameter of the reference line, y rf is the vertical coordinate variable of the reference line, ξ3 is the third parameter of the reference line, x nose is the horizontal coordinate of the nose tip, y nose is the vertical coordinate of the nose tip, x lip,2 is the horizontal coordinate of the center point of the lower lip, y lip,2 is the vertical coordinate of the center point of the lower lip.

[0083] And according to the coordinates of each pixel of the left eye contour and the coordinates of each pixel of the right eye contour, the left eye center coordinates and the right eye center coordinates are calculated by the following formula:

[0084] ,

[0085] ,

[0086] Among them, x ec is the horizontal coordinate of the left or right eye center, y ec is the vertical coordinate of the left or right eye center, x n is the horizontal coordinate of the nth pixel of the left or right eye contour, y n is the vertical coordinate of the nth pixel of the left or right eye contour, n is the left or right eye contour pixel number, and K is the number of left or right eye contour pixels.

[0087] S4. Monitor the driver's head orientation, eye closure, and yawning based on the reference line, eye center coordinates, nose tip coordinates, upper and lower lip center coordinates, and eye pupils.

[0088] like Figure 3 As shown, when the driver turns his head, the distance between his eyes and the reference straight line will be shortened due to the effect of the side face projection of the left and right rotation. Therefore, the present invention uses the change in the distance from the left or right eye center coordinate to the reference straight line as the criterion for the driver to turn his head left or right.

[0089] If the driver is detected to have a significant head turn for a long period of time, an alarm will be issued.

[0090] The distance between the left or right eye center coordinates and the reference straight line is calculated using the following formula:

[0091] ,

[0092] Among them, d ec is the distance from the left or right eye center coordinate to the reference straight line, ξ1 is the first parameter of the reference straight line, ξ2 is the second parameter of the reference straight line, ξ3 is the third parameter of the reference straight line, x ec is the horizontal coordinate of the left or right eye center, y ec The vertical coordinate of the center of the left or right eye.

[0093] like Figure 4 As shown, when the driver does not tilt their head left or right, the centers of both eyes should be at the same height. That is, the difference in the ordinates of the left and right eye centers in the image should be small. If the difference in the ordinates of the left and right eye centers increases, it indicates that the driver is tilting his head. Therefore, the present invention uses the change in the difference in the ordinates of the left and right eye centers as a criterion for determining whether the driver has tilted his head left or right.

[0094] An alarm can also be issued if the driver is detected to have a large head tilt for a long time.

[0095] like Figure 5 As shown in FIG. 1 , when the driver raises or lowers his head, the difference between the vertical coordinates of the center of his eye and the tip of his nose decreases due to the projection effect of the up-and-down rotation. Therefore, the present invention uses the change in the difference between the vertical coordinates of the center of the left or right eye and the vertical coordinate of the tip of the nose as the criterion for determining whether the driver raises or lowers his head.

[0096] An alarm can also be issued if the driver is detected to have a significant head-up or head-down movement for a long period of time.

[0097] The present invention also uses the failure of extracting the pupils of both eyes from the driver's facial image as a criterion for determining whether the driver's eyes are closed, and monitors the driver's eye closure. This is because when the driver's eyes are closed, the pupils in the outline of both eyes cannot be extracted.

[0098] When the system detects that the driver's blinking frequency increases or the time his eyes are closed becomes longer, it determines that the driver is in a fatigue driving state and issues an alarm.

[0099] like Figure 6As shown, the vertical coordinate difference between the center points of the upper and lower lips changes when the driver's mouth opens and closes. Therefore, the present invention uses the change in the vertical coordinate difference between the center points of the upper and lower lips as a criterion for determining whether the driver's mouth is open or closed. By presetting a threshold for the vertical coordinate difference between the center points of the upper and lower lips during yawning, the present invention monitors the driver's yawning.

[0100] In summary, the present invention is based on image capture and processing, obtains facial key points through facial contours, and monitors the driver's status by relying on the changing patterns of facial key points under biological posture. It has high real-time, accuracy and robustness, does not require wearing any equipment, avoids the discomfort caused by traditional physiological sensors, can not only detect fatigue driving, but also identify key behaviors such as distraction, realize comprehensive monitoring of the driver's multi-dimensional status, and significantly improve driving safety.

[0101] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A driver status monitoring method based on machine vision, characterized in that: The following steps are involved: S1, capture the driver's facial image and extract the facial contour; S2. Based on the facial contour, obtain the center points of the upper and lower lips, the outlines of the eyes, the pupils of both eyes, and the tip of the nose; S3. Establish a reference line based on the coordinates of the lower lip center and the nose tip, and calculate the coordinates of the eye centers based on the eye contours. S4. Monitor the driver's head orientation, eye closure, and yawning based on the reference line, eye center coordinates, nose tip coordinates, upper and lower lip center coordinates, and eye pupils; S1 The following steps are included: S11. Install an infrared camera and an RGB camera in front of the driver's seat of the vehicle to capture infrared and RGB images of the driver's face, respectively; S12, performing grayscale conversion on the RGB image, and performing weighted synthesis of the infrared image and the grayscale converted RGB image based on the light intensity to obtain a facial enhanced grayscale image; S13, using a 3×3 Laplacian operator as a convolution kernel, performing a convolution operation on the facial enhancement grayscale image to obtain a facial contour grayscale image; S14, adaptively adjusting the binarization threshold to convert the facial contour grayscale image into a binary image with only facial contour lines to obtain the facial contour; In S12, before the infrared image and the grayscale-converted RGB image are weighted and synthesized, they are scaled to the same size by upsampling or downsampling respectively; The light intensity is calculated by the following formula: ， Among them, I avg is the light intensity, I RGB (i, j) is the grayscale value of the i-th row and j-th column of the RGB image after grayscale conversion and scaling, i is the row number, j is the column number, M is the number of row pixels of the RGB image after grayscale conversion and scaling, and N is the number of column pixels of the RGB image after grayscale conversion and scaling; The expression of the weighted synthesis is: , Among them, I syn (i, j) is the grayscale value of the i-th row and j-th column of the facial enhancement grayscale image, I RGB is the RGB image after grayscale transformation and scaling, I RE (i, j) is the grayscale value of the i-th row and j-th column of the scaled infrared image, and MAX() is the function that takes the maximum value.

2. The driver status monitoring method based on machine vision according to claim 1, characterized in that: The method of adaptively adjusting the binarization threshold in S14 is: Increase or decrease the binarization threshold so that the contours of both eyes in the converted binary image are exactly closed curves.

3. The driver status monitoring method based on machine vision according to claim 1, characterized in that: In S3, the expression of the reference straight line established by the coordinates of the center point of the lower lip and the tip of the nose is: ， ， ， ， Among them, ξ1 is the first parameter of the reference line, x rf is the horizontal coordinate variable of the reference line, ξ2 is the second parameter of the reference line, y rf is the vertical coordinate variable of the reference line, ξ3 is the third parameter of the reference line, x nose is the horizontal coordinate of the nose tip, y nose is the vertical coordinate of the nose tip, x lip,2 is the horizontal coordinate of the center point of the lower lip, y lip,2 is the vertical coordinate of the center point of the lower lip.

4. The driver status monitoring method based on machine vision according to claim 1, characterized in that: In S3, the left eye center coordinates and the right eye center coordinates are calculated according to the coordinates of each pixel of the left eye contour and the coordinates of each pixel of the right eye contour by the following formula: ， ， Among them, x ec is the horizontal coordinate of the left or right eye center, y ec is the vertical coordinate of the left or right eye center, x n is the horizontal coordinate of the nth pixel of the left or right eye contour, y n is the vertical coordinate of the nth pixel of the left or right eye contour, n is the left or right eye contour pixel number, and K is the number of left or right eye contour pixels.

5. The driver status monitoring method based on machine vision according to claim 1, characterized in that: S4 uses the change in the distance from the left or right eye center coordinate to the reference straight line as a criterion for the driver to turn his head left or right; The change in the difference between the left and right eye center vertical coordinates is used as the criterion for the driver's left or right head tilt; The change in the difference between the left or right eye center ordinate and the nose tip ordinate is used as the criterion for whether the driver is looking up or down. The distance between the left or right eye center coordinates and the reference straight line is calculated using the following formula: ， Among them, d ec is the distance from the left or right eye center coordinate to the reference straight line, ξ1 is the first parameter of the reference straight line, ξ2 is the second parameter of the reference straight line, ξ3 is the third parameter of the reference straight line, x ec is the horizontal coordinate of the left or right eye center, y ec The vertical coordinate of the center of the left or right eye.

6. The driver status monitoring method based on machine vision according to claim 1, characterized in that: The S4 monitors the closure of the driver's eyes by taking whether the pupils of both eyes fail to be extracted from the driver's facial image as a criterion for determining whether the driver's eyes are closed.

7. The driver status monitoring method based on machine vision according to claim 1, characterized in that: The S4 uses the change of the vertical coordinate difference between the center point of the upper lip and the center point of the lower lip as a criterion for judging whether the driver's mouth is open or closed, and monitors the driver's yawning situation.

Images