An eye pose recognition method based on close-range photogrammetry

Abstract

This invention discloses an eye pose recognition method based on close-range photogrammetry, comprising: step 101, locating facial feature points in two images from two binocular cameras; step 102, using the facial feature points from the two images as corresponding points, recovering the three-dimensional coordinates of the facial feature points using close-range photogrammetry and the least squares algorithm; step 103, extracting the three-dimensional coordinates of the eye region feature points and reprojecting them onto the XOY plane; step 104, calculating the head pose angle using the three-dimensional coordinates of the facial feature points, and using the head pose angle to correct the EAR value. Beneficial effects: The EAR value calculated based on the projection of the three-dimensional coordinates onto the XOY plane does not need to consider the error caused by the yaw angle. Furthermore, by using the three-dimensional coordinates of the facial feature points to calculate the head pose angle to correct the EAR value, the EAR value does not change significantly regardless of the change in the pitch angle γ between two consecutive frames, thus providing a good indication of the eye state.

Description

Technical Field

[0001] This invention relates to the field of facial eye pose recognition and prediction technology, and in particular to an eye pose recognition method based on close-range photogrammetry. Background Technology

[0002] Currently, the most direct basis for detecting physical signals of fatigued driving is based on changes in eye features, such as eye closure time, blink frequency, and PERCLOS. These feature changes are all related to eye posture recognition, and eye posture is used to supervise or predict the detection of physical signals of fatigued driving. Current eye posture recognition methods can be divided into machine learning-based methods and feature point-based methods.

[0003] Machine learning-based methods rely excessively on the size and accuracy of the dataset; a larger dataset and a correctly trained classifier will result in stronger generalization ability. Meanwhile, most feature-point-based methods currently rely on monocular cameras, typically using EAR values ​​to determine eye state.

[0004] Existing The EAR value is a ratio, which can reduce the error caused by the distance of the monocular camera. However, this method of judging the eye status based on the EAR value is easily affected by the head attitude angle. When the head yaw angle changes, the EAR value will be larger; when the head pitch angle changes, the EAR value will be smaller. In this case, using the threshold method to judge the eye status will result in over-detection or under-detection. Summary of the Invention

[0005] The purpose of this invention is to provide an eye pose recognition method based on close-range photogrammetry. The EAR value is calculated based on the projection of three-dimensional coordinates onto the XOY plane, which does not need to consider the error caused by the yaw angle. The head pose angle is then calculated using the three-dimensional coordinates of facial feature points to correct the EAR value. This ensures that the EAR value does not change significantly regardless of the change in pitch angle γ between two consecutive frames. This threshold method can effectively determine the state of the eyes and extract eye fatigue features with high accuracy.

[0006] The technical solution of this invention is implemented as follows:

[0007] An eye pose recognition method based on close-range photogrammetry, comprising:

[0008] Step 101: Locate facial feature points in the left and right images using a binocular camera;

[0009] Step 102: Based on the facial feature points in the left and right images as corresponding points, the three-dimensional coordinates of the facial feature points are recovered using close-range photogrammetry and the least squares algorithm.

[0010] Step 103: Extract the three-dimensional coordinates of the feature points in the eye region and reproject them onto the XOY plane;

[0011] Step 104: Calculate the head pose angle using the three-dimensional coordinates of facial feature points, correct the EAR value using the head pose angle, and then determine the state of the eyes using a threshold method.

[0012] Furthermore, step 102 specifically includes:

[0013] The pixel coordinates (u, v) of the facial feature points in the image are obtained by using a stereo camera; the pixel coordinates of the facial feature points are transformed into coordinates (Xc, Yc, Zc) in the camera coordinate system using interior orientation elements; and the three-dimensional coordinates (Xw, Yw, Zw) of the facial feature points in the image are obtained by combining the exterior orientation elements with the least squares algorithm from the coordinates (Xc, Yc, Zc) in the camera coordinate system.

[0014] Furthermore, step 104 specifically includes:

[0015] Calculate the rotation matrix R and translation matrix T between feature points of two consecutive image frames, using the face feature points p from the previous frame. i As a template, use the facial feature points q of the current frame. i As the target, then p i q i The centers of gravity are respectively

[0016] Furthermore, let q i '=q i -q and p i '=p i -p, to minimize the objective function E to obtain a rotation matrix R and a translation matrix T;

[0017]

[0018] Let N = U∑V T To minimize E, we need to maximize tr(RN). We then perform singular value decomposition on N to obtain N = U∑V T When R = VU T tr(RN) reaches its maximum value, so R is the rotation matrix between feature points in two consecutive frames.

[0019] Furthermore, the rotation matrix Where γ is the change in pitch angle between two consecutive frames.

[0020] Furthermore, the aforementioned

[0021] Among them, (X)p2 Y p2 ), (X p6 Y p6 ), (X p3 Y p3 ), (X p5 Y p5 The coordinates of the feature points are reprojected onto the XOY plane from their three-dimensional coordinates.

[0022] First, the mean EAR value is calculated from the images of the open eyes and recorded as EAR1. Then, the mean EAR value is calculated from the images of the closed eyes and recorded as EAR2. The threshold can be set as EAR1-|EAR1-EAR2|×0.8. If it is less than the threshold, the eye state is closed; if it is greater than the threshold, the eye state is open.

[0023] The beneficial effects of this invention are as follows: This invention uses a binocular camera to locate facial feature points in two images, and obtains the facial feature points in the two images as corresponding points. Then, it recovers the three-dimensional coordinates of the facial feature points using close-range photogrammetry and the least squares algorithm, and reprojects them onto the XOY plane. The EAR value calculated based on the three-dimensional coordinates does not need to consider the error caused by the yaw angle. Furthermore, the EAR value is corrected by calculating the head attitude angle using the three-dimensional coordinates of the facial feature points. This ensures that the EAR value does not change significantly regardless of the change in the pitch angle γ between two consecutive frames. This threshold method can effectively determine the state of the eyes, thereby extracting eye fatigue features with high accuracy. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram showing the distribution of feature points in the eye.

[0026] Figure 2 A frame diagram showing continuous pitch angle changes;

[0027] Figure 3 This is a frame diagram showing the continuous yaw angle variation. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0030] According to an embodiment of the present invention, an eye pose recognition method based on close-range photogrammetry is provided.

[0031] The eye pose recognition method based on close-range photogrammetry according to an embodiment of the present invention includes:

[0032] Step 101: Locate facial feature points in the left and right images using a binocular camera;

[0033] Step 102: Based on the facial feature points in the left and right images as corresponding points, the three-dimensional coordinates of the facial feature points are recovered using close-range photogrammetry and the least squares algorithm.

[0034] Step 103: Extract the three-dimensional coordinates of the feature points in the eye region and reproject them onto the XOY plane;

[0035] Step 104: Calculate the head pose angle using the three-dimensional coordinates of facial feature points, correct the EAR value using the head pose angle, and then determine the state of the eyes using a threshold method.

[0036] Specifically, in step 101, the facial feature points of the left and right images are located using a binocular camera. The facial feature points are located using an end-to-end facial key point detection and facial feature point localization model based on residual neural networks, which is open sourced from Geogle. This model has high accuracy and strong stability in locating facial feature points.

[0037] This model was trained using 30,000 real-world facial photographs under varying lighting conditions. Each photograph contains 478 keypoint annotations. The annotation process employed an iterative training method: first, a subset of photographs were manually annotated as seed images to train an initial model. This initial model was then used to predict keypoints in 30% of the photographs in the training set. The keypoint information was fine-tuned and then fed back into the model, continuously iterating until the final model performance was achieved. This model can detect 478 facial keypoints, with particularly high accuracy in detecting eye features.

[0038] Step 102 specifically includes: acquiring the pixel coordinates (u, v) of the face feature points in the image through a binocular camera; converting the pixel coordinates of the face feature points into coordinates (Xc, Yc, Zc) in the camera coordinate system through interior orientation elements; and then obtaining the three-dimensional coordinates (Xw, Yw, Zw) of the face feature points in the image by combining the exterior orientation elements with the least squares algorithm from the coordinates (Xc, Yc, Zc) in the camera coordinate system.

[0039] Let the pixel coordinates of the facial feature points acquired by the binocular camera be (u1, v1) and (u2, v2). Then,

[0040] Specifically, based on the orientation elements within the camera, there are The pixel coordinates of facial feature points are transformed into camera coordinates (Xc, Yc, Zc) using interior orientation elements, where R is the camera's interior orientation element. Then, the camera coordinates are transformed using... Transform it into the world coordinate system, that is, the three-dimensional coordinates (Xw, Yw, Zw) of the facial feature points in the image.

[0041] Then there is

[0042] It can be concluded that in,

[0043] (u1-m 31 1 -m 11 1 )X W +(u1m 32 1 -m 12 1 )Y W +(u1m 33 1 -m 13 1 )Z W =m 14 1 -u1m 34 1

[0044] (v1-m 31 1 -m 11 1 )X W +(v1m 32 1 -m 12 1 )Y W +(v1m 33 1 -m 131 )Z W =m 14 1 -v1m 34 1

[0045] (u2-m 31 2 -m 11 2 )X W +(u2m 32 2 -m 12 2 )Y W +(u2m 33 2 -m 13 2 )Z W =m 14 2 -u2m 34 2

[0046] (v2-m 31 2 -m 11 2 )X W +(v2m 32 2 -m 12 2 )Y W +(v2m 33 2 -m 13 2 )Z W =m 14 2 -v2m 34 2

[0047] The three-dimensional coordinates (Xw, Yw, Zw) of the facial feature points in the image are obtained by combining interior orientation elements with a least squares algorithm, where R 3×3 Let t be the rotation matrix of the right camera relative to the left camera. 3×1 Let be the translation matrix of the right camera relative to the left camera. Then, extract the 3D coordinates of the feature points in the eye region and reproject them onto the XOY plane, so that the error caused by the yaw angle can be ignored.

[0048] Step 104 specifically includes: calculating the rotation matrix R and translation matrix T between feature points of two consecutive frames of images. The head pose calculation adopts a head tracking-based method, calculating the three-dimensional coordinates of the feature points of the face in each frame, and calculating the rotation matrix and translation matrix using the three-dimensional coordinates of the feature points of the face in two consecutive frames. This method is used to track the face and calculate the head pose.

[0049] The steps of this method are as follows: Use the facial feature points p from the previous frame i As a template, use the facial feature points q of the current frame. i As the target, then p i q i The centers of gravity are respectively

[0050] set up and To obtain a rotation matrix R and a translation matrix T, we minimize the objective function E.

[0051]

[0052] Combining formulas (1) and (2), it can be rewritten as

[0053]

[0054] To minimize the objective function E, the translation matrix T should move the centroid of the feature points to the centroid, so that the objective function can be rewritten as:

[0055]

[0056] Let N = U∑V T To minimize E, we need to maximize tr(RN). We then perform singular value decomposition on N to obtain N = U∑V T When R = VU T tr(RN) reaches its maximum value, so R is the rotation matrix between feature points in two consecutive frames.

[0057] And the rotation matrix Where γ is the change in pitch angle between two consecutive frames.

[0058] Then there is

[0059] Among them, (X) p2 Y p2 ), (X p6 Y p6 ), (X p3 Y p3 ), (X p5 Y p5The EAR value is the coordinate of the feature point's 3D coordinates reprojected onto the XOY plane. Regardless of the change in pitch angle γ between two consecutive frames, the EAR value will not change significantly. This thresholding method can effectively determine the eye's state, thereby extracting eye fatigue features.

[0060] The method for determining eye status using a threshold is as follows: First, calculate the mean EAR value from the part of the image where the eyes are open and record it as EAR1. Then, calculate the mean EAR value from the part of the image where the eyes are closed and record it as EAR2. The threshold can be set as EAR1-|EAR1-EAR2|×0.8. If the value is less than the threshold, the eye status is closed; if the value is greater than the threshold, the eye status is open.

[0061] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for eye pose recognition based on close-range photogrammetry, characterized in that, include: Step 101: Locate facial feature points in the left and right images using a binocular camera; Step 102: Based on the facial feature points in the left and right images as corresponding points, the three-dimensional coordinates of the facial feature points are recovered using close-range photogrammetry and the least squares algorithm. Step 103: Extract the three-dimensional coordinates of the feature points in the eye region and reproject them onto the XOY plane; Step 104: Calculate the head pose angle using the three-dimensional coordinates of facial feature points, correct the EAR value using the head pose angle, and then determine the state of the eyes using a threshold method. Among them, the , in,( , ), ( , ), ( , ), ( , The coordinates of the feature points are reprojected onto the XOY plane from their 3D coordinates. This represents the change in pitch angle between two consecutive frames. First, calculations are performed using the portion of the image where the eyes are open. The mean of the values ​​is denoted as Then, calculations are performed using images of the closed eye area. The mean of the values ​​is denoted as Its threshold is set to If the value is less than the threshold, the eye state is closed; if the value is greater than the threshold, the eye state is open. Specifically, step 102 includes: The pixel coordinates (u, v) of the facial feature points in the image are obtained through a binocular camera; the pixel coordinates of the facial feature points are transformed into coordinates (Xc, Yc, Zc) in the camera coordinate system through interior orientation elements; and the three-dimensional coordinates (Xw, Yw, Zw) of the facial feature points in the image are obtained by combining the exterior orientation elements with the least squares algorithm from the coordinates (Xc, Yc, Zc) in the camera coordinate system. Step 104 specifically includes: Calculate the rotation matrix between feature points in two consecutive frames of an image. Translation matrix Using facial feature points from the previous frame Use the facial feature points of the current frame as a template. As a goal, , The centers of gravity are respectively , ; set up and To obtain a rotation matrix R and a translation matrix T by minimizing the objective function E; ； set up To minimize E, we need to make Maximize by performing singular value decomposition on N. ,when , To obtain the maximum, therefore That is, the rotation matrix between feature points in two consecutive frames; the rotation matrix , .

Images