Driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze

Abstract

The application discloses a driver gaze direction detection method based on two-dimensional and three-dimensional face consistent gaze, uses a two-dimensional convolution network face feature to estimate the gaze direction of a two-dimensional face and the gaze cone of the two-dimensional face, uses a three-dimensional face reconstruction depth feature to estimate the gaze direction of a three-dimensional face and the gaze cone of the three-dimensional face, calculates the consistency loss of the gaze direction of the three-dimensional face and the gaze direction of the two-dimensional face to verify the accuracy of the gaze direction of the three-dimensional face, and calculates the consistency loss of the gaze cone of the three-dimensional face and the gaze cone of the two-dimensional face to verify the accuracy of the gaze cone of the three-dimensional face. The application combines two-dimensional and three-dimensional consistent gaze directions and two-dimensional and three-dimensional consistent gaze cones, can guarantee the reliability of two-dimensional face feature extraction and three-dimensional face feature extraction, and realizes accurate gaze direction detection.

Description

Technical Field

[0001] This invention relates to the field of driving safety, specifically a method for detecting driver gaze direction based on two-dimensional and three-dimensional facial gaze consistency. Background Technology

[0002] With the development of autonomous driving technology, driver attention monitoring has become particularly important. To ensure that drivers maintain appropriate attention during driving, gaze direction detection technology has gradually become an important research area. Existing gaze direction detection methods are mainly divided into two categories: two-dimensional face feature extraction and three-dimensional face reconstruction.

[0003] Two-dimensional face feature extraction methods typically use convolutional neural networks (CNNs) to process face images and estimate gaze direction. However, due to the lack of depth information, these methods have limitations in accuracy and robustness, especially when faced with changes in lighting, occlusion, and head pose, where detection accuracy drops significantly. Furthermore, relying solely on two-dimensional information cannot fully reflect the three-dimensional structural features of the face, affecting the determination of gaze direction.

[0004] On the other hand, 3D face reconstruction technology can more accurately model the spatial structure of the face by capturing its depth information. However, existing 3D reconstruction methods are often complex and computationally intensive, and are easily limited by hardware performance in practical applications. In addition, gaze direction detection methods that rely solely on 3D information may suffer from insufficient stability in certain scenarios.

[0005] Currently, although some studies have attempted to combine two-dimensional and three-dimensional information to improve the accuracy and robustness of gaze direction detection, these methods generally suffer from the following problems: First, the fusion of two-dimensional and three-dimensional information often lacks an effective mechanism to verify their consistency, leading to inaccurate detection results; second, existing methods have high computational complexity in practical applications, making it difficult to meet real-time requirements. Therefore, how to effectively combine two-dimensional and three-dimensional facial information and ensure consistent gaze direction between them has become an urgent problem to be solved.

[0006] Chinese patent application publication number CN117935334A discloses a method and system for estimating driver gaze based on 3D face reconstruction. This method and system recover the 3D shape of a face from a single ordinary face image, and estimate the gaze direction based on the 3D shape, as well as the anatomical structure and movement patterns of the human eye. Compared to methods that solely rely on 3D face reconstruction, this invention effectively compensates for the shortcomings of 3D reconstruction, such as its dependence on image quality and high computational complexity, by combining a 2D convolutional network with 3D depth features for gaze direction and a consistent gaze cone. Through mutual verification of 2D and 3D information, it improves adaptability and detection accuracy in dynamic scenes, while mitigating the poor adaptability of purely 3D methods to individual facial differences, ensuring more stable and accurate gaze direction detection.

[0007] Chinese patent application publication number CN115376112A discloses a method and apparatus for detecting driver gaze. This method determines the driver's gaze passes through each target area. However, this invention relies on precise calibration of multiple cameras, making it susceptible to errors and lacking three-dimensional depth information, which may lead to misjudgments of gaze in complex scenes. Furthermore, multi-camera systems are susceptible to occlusion at certain angles, reducing the accuracy of gaze detection. In contrast, a combined two-dimensional and three-dimensional method is more robust. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for detecting driver gaze direction based on two-dimensional and three-dimensional facial gaze consistency.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A driver gaze direction detection method based on 2D and 3D face consistent gaze includes the following steps:

[0011] Step S1: Define the 3D face coordinate system Two-dimensional face coordinate system and driver's 3D face position image dataset {position 3D Driver's 2D face dataset {face} 2D}, The 3D gaze direction dataset is Two-dimensional gaze direction dataset is The 3D gaze cone dataset is Two-dimensional gaze cone dataset is

[0012] Step S2: Based on the driver's two-dimensional face dataset {face 2D The driver's two-dimensional face image in} 2DA two-dimensional facial feature point set is obtained. Based on this set, the driver's left eye region image (Image_left) and right eye region image (Image_right) are obtained. Then, based on the driver's left eye region image (Image_left) and right eye region image (Image_right), the driver's two-dimensional face image (face) is calculated. 2D Two-dimensional gaze direction Based on the two-dimensional gaze direction Calculate the two-dimensional gaze cone Cone 2D ;

[0013] Step S3: Based on the driver's two-dimensional face image 2D Obtain the two-dimensional nose point set P nose Based on the driver's 3D face position image dataset {position 3D The driver's 3D face position image in} 3D Obtain a set of key feature points for a 3D face. Then, based on the three-dimensional facial key feature point set Calculate the driver's three-dimensional gaze direction Based on the driver's three-dimensional gaze direction Calculate the three-dimensional gaze cone Cone 3D ;

[0014] Step S4: Calculate the two-dimensional gaze direction loss 3D gaze direction loss 2D-3D Consistent Gaze Direction Loss Combined with two-dimensional gaze direction loss 3D gaze direction loss 2D-3D Consistent Gaze Direction Loss The total gaze direction loss was calculated.

[0015] Step S5: Calculate the two-dimensional gaze cone loss. 3D staring cone loss Two-dimensional-three-dimensional consistent gaze cone loss Combining two-dimensional gaze cone loss 3D gaze cone loss Two-dimensional-three-dimensional consistent gaze cone loss The total gaze cone loss of the driver was calculated.

[0016] Then, combined with total gaze cone loss and the total gaze direction loss obtained in step S4 The total gaze loss was calculated. Total ;

[0017] Finally, the method is trained using the total gaze loss to obtain the optimal parameters.

[0018] Step S6: Input the two-dimensional face image to be detected, and process the two-dimensional face image to be detected using the method under the optimal parameters to obtain the driver's three-dimensional face position image, the driver's three-dimensional gaze direction, and the driver's three-dimensional gaze cone in the two-dimensional face image to be detected.

[0019] Furthermore, in step S2, the driver's left eye region image Image_left is subjected to convolutional pooling to obtain the left eye gaze direction. The driver's right eye region image (Image_right) is processed by convolutional pooling to obtain the right eye gaze direction. Finally, combine this with the left eye's gaze direction. Right eye gaze direction Calculation of two-dimensional gaze direction

[0020] Furthermore, in step S3, based on the driver's three-dimensional face position image... 3D Obtain the 3D left eye point set P' left_eye 3D right eye point set P' right_eye 3D nose point set P' nose This constitutes a set of key feature points for a three-dimensional face. Then, the set of key feature points of the three-dimensional face is analyzed. By performing fully connected layer processing, the driver's three-dimensional gaze direction can be obtained.

[0021] Furthermore, in step S4, the two-dimensional gaze direction calculated in step S2 is... Two-dimensional gaze direction dataset in step 1 Two-dimensional gaze direction Calculate the two-dimensional gaze direction loss

[0022] Furthermore, in step S4, the three-dimensional gaze direction calculated in step S3 is... The 3D gaze direction dataset in Step 1 Three-dimensional gaze direction Calculate the loss in three-dimensional gaze direction

[0023] Furthermore, in step S4, the three-dimensional gaze direction is... From the three-dimensional face coordinate system Projected onto a two-dimensional face coordinate system This gives the direction of the projected gaze. Then, combined with the direction of the projection gaze Two-dimensional gaze direction The two-dimensional-three-dimensional consistent gaze direction loss was calculated.

[0024] Furthermore, in step S5, based on the two-dimensional gaze cone Cone obtained in step S2... 2D vertex angle θ 2D Step S1: Two-dimensional gaze cone dataset Two-dimensional staring cone in apex Calculate the loss of the apex angle of the two-dimensional gaze cone.

[0025] Calculate the two-dimensional gaze cone Cone 2D With 2D gaze cone dataset Two-dimensional staring cone in Area of ​​the union region union And the two-dimensional gaze cone Cone 2D With 2D gaze cone dataset Two-dimensional staring cone in Area of ​​intersection int Then, based on the area of ​​the union region, Area union Area of ​​intersection int Calculate the crossover ratio loss of a two-dimensional staring cone.

[0026] Combined with two-dimensional gaze cone apex angle difference loss Two-dimensional gaze cone intersection ratio loss The two-dimensional gaze cone loss was calculated.

[0027] Furthermore, in step S5, based on the three-dimensional gaze cone obtained in step S3... 3D The apex angle, the 3D gaze cone dataset in step S1 Three-dimensional gaze cone Calculate the apex angle difference loss of the three-dimensional staring cone.

[0028] Calculate the three-dimensional gaze cone Cone 3D With 3D gaze cone dataset Three-dimensional gaze cone Union region volume Union And the three-dimensional gaze cone Cone 3D With 3D gaze cone dataset Three-dimensional gaze cone Volume of the intersection regionint Then, based on the union region volume Union Volume of the intersection region int Calculate the crossover ratio loss of a 3D staring cone.

[0029] Combined with 3D staring cone apex angle difference loss 3D staring cone intersection ratio loss The calculated loss of the three-dimensional gaze cone

[0030] Furthermore, in step S5, the three-dimensional gaze cone Cone 3D From the three-dimensional face coordinate system Projected onto a two-dimensional face coordinate system Obtain the projection gaze cone Cone proj And calculate the projected gaze cone Cone proj vertex angle θ proj ;

[0031] According to the vertex angle θ proj and two-dimensional gaze cone Cone 2D vertex angle θ 2D The difference in apex angle of the two-dimensional-three-dimensional consistent gaze cone was calculated.

[0032] Calculate the projective gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' of the union region Union And the projected gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' int Combined with the area of ​​the union region Area' Union Area' of intersection int Calculate the crossover ratio loss of 2D-3D consistent gaze cones.

[0033] Then, combining the two-dimensional-three-dimensional consistent gaze cone apex angle difference loss Two-dimensional-three-dimensional consistent gaze cone intersection-to-union ratio loss The calculation yielded the two-dimensional-three-dimensional consistent gaze cone loss.

[0034] This invention calculates two-dimensional gaze direction loss and two-dimensional gaze cone loss by extracting two-dimensional features, calculates three-dimensional gaze direction loss and three-dimensional gaze cone loss by extracting three-dimensional features, projects the three-dimensional features onto a two-dimensional plane to calculate consistent gaze direction loss and consistent gaze cone loss, and trains and optimizes the parameters of the method using the total gaze direction loss and total gaze cone loss to accurately obtain the driver's three-dimensional gaze direction and three-dimensional gaze cone.

[0035] The two-dimensional and three-dimensional face consistent gaze detection method of this invention has significant advantages. First, by combining two-dimensional convolutional networks and three-dimensional face reconstruction technology, it fully utilizes the detailed information of two-dimensional images and the three-dimensional depth features, overcoming the limitations of single methods. The two-dimensional method exhibits strong robustness in handling complex lighting conditions and occlusion, while the three-dimensional method provides accurate spatial structure information. By calculating the consistency loss of two-dimensional and three-dimensional gaze directions and gaze cones, this method can mutually verify and correct errors, thereby improving the accuracy and reliability of gaze direction detection. Furthermore, the combination of two-dimensional and three-dimensional information effectively enhances the system's adaptability in dynamic scenes, ensuring accurate gaze detection in various driving environments. In summary, this method, through multi-dimensional information fusion, provides a high-precision and highly robust driver gaze direction detection scheme, suitable for the complex needs of real-world driving scenarios. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the definition of the coordinate system and dataset mentioned in this invention.

[0037] Figure 2 This is a flowchart for calculating the two-dimensional staring cone mentioned in this invention.

[0038] Figure 3 This is a flowchart for calculating the three-dimensional staring cone mentioned in this invention.

[0039] Figure 4 This is a flowchart for calculating the total gaze direction loss mentioned in this invention.

[0040] Figure 5 This is a flowchart for calculating the total gaze cone loss mentioned in this invention.

[0041] Figure 6 This is a flowchart of the process for obtaining a three-dimensional gaze cone as mentioned in this invention.

[0042] Figure 7 This is a flowchart illustrating the implementation of an embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] like Figure 7 As shown, this embodiment discloses a method for detecting driver gaze direction based on two-dimensional and three-dimensional facial consistency gaze, including the following steps:

[0045] Step S1: Define the required coordinate system and dataset. For example... Figure 1 As shown, the coordinate system and dataset defined in this embodiment include the following parts:

[0046] (S1A) Defines a three-dimensional face coordinate system in O of the coordinate system Face The tip of the nose on a person's face; coordinate system X-axis Face The horizontal axis of the image is from left to right; coordinate system Y-axis Face The vertical axis of the image is from top to bottom; coordinate system Z-axis Face The depth axis of the image is oriented from far to near.

[0047] (S1B) Define a two-dimensional face coordinate system in O of the coordinate system Face The tip of the nose on a person's face; coordinate system X-axis Face The horizontal axis of the image is from left to right; coordinate system Y-axis Face The vertical axis of the image is from top to bottom.

[0048] (S1C) defines a dataset of driver's 3D face position images {position} 3D Specifically, input the driver's two-dimensional facial data (face). 2D Using PRNET (joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network. In ECCV 2018.), the driver's 3D face position image was obtained. 3D (256×256×3). Among them, face... 2D A 2D image of the driver's face, measuring 256×256×3; and a 3D image showing the driver's face position. 3D The three channels are respectively located in the three-dimensional face coordinate system. The x, y, and z coordinates in the diagram.

[0049] (S1D) defines a driver's two-dimensional face dataset {face} 2D}

[0050] (S1E) defines the three-dimensional gaze direction dataset as follows: Where x gt ,y gt ,z gt These are the three-dimensional gaze directions in X Face Y Face Z Face The portion on top.

[0051] (S1F) defines the two-dimensional gaze direction dataset as follows: Where x gt ,y gt These are the two-dimensional gaze directions in X Face Y Face The portion on top.

[0052] (S1G) defines the 3D gaze cone dataset as follows: The process is as follows:

[0053] Set a 3D gaze cone The vertex is O Face ;

[0054] Set a 3D gaze cone The direction is

[0055] Set a 3D gaze cone The vertex is

[0056] Set a 3D gaze cone The maximum gaze distance is h gt This indicates the effective range of the driver's line of sight;

[0057] Calculate the three-dimensional gaze cone The gaze range is shown in the following formula:

[0058]

[0059] Where (x,y,z) represents the position of the 3D face image. 3D The coordinates of any point in the array;

[0060] Calculate the distance from point (x, y, z) to vertex O. Face vector sum The cosine of the angle between the two points is used to determine whether the point lies within the triangle. Inside.

[0061] (S1H) defines a two-dimensional gaze cone dataset as follows: The process is as follows:

[0062] Set a two-dimensional gaze cone The vertex is O Face ;

[0063] Set a two-dimensional gaze cone The direction is

[0064] Set a two-dimensional gaze cone The vertex is

[0065] Calculate the two-dimensional gaze cone The gaze range is shown in the following formula:

[0066]

[0067] Where (x,y) represents the driver's two-dimensional face image. 2D Any point in it;

[0068] Calculate the distance from point (x, y) to vertex O. Face vector and The cosine of the angle between the two points is used to determine whether the point lies within the triangle. Inside.

[0069] Step S2: Calculate the two-dimensional gaze cone Cone 2D ,like Figure 2 As shown, the process is as follows:

[0070] (S2A) Based on driver 2D face dataset {face 2D The driver's two-dimensional face image in} 2D Using Dlib's 68-point facial landmark detector model, a two-dimensional set of facial landmark points P is obtained. face ={(x i ,y i )},x i ,y i These are two-dimensional facial feature points at X. Face Y Face The position on the face is i, where i is the number of the facial feature point, i = 1, 2, 3, ... 68.

[0071] From feature point set P face ={(x i ,y i Obtain the two-dimensional left eye point set P from )} left_eye ={(x i ,y i)}, where i is the feature point number of the left eye, i = 36, 37, 38, 39, 40, 41.

[0072] From feature point set P face ={(x i ,y i Obtain the two-dimensional right eye point set P from )} right_eye ={(x i ,y i )}, where i is the right eye feature point number, i = 42, 43, 44, 45, 46, 47.

[0073] (S2B) Based on the two-dimensional facial feature point set P face The two-dimensional left eye point set P left_eye Two-dimensional right eye point set P right_eye Two-dimensional facial images of the driver were obtained respectively. 2D The process of creating the left-eye region image (Image_left) and the right-eye region image (Image_right) is as follows:

[0074] Let P be a two-dimensional left eye point set. left_eye The top left corner point P in left_up =(x min ,y max (), the bottom right corner point P right_down =(x max ,y min ),in:

[0075] x min =min(x 36 ,x 37 ,x 38 ,x 39 ,x 40 ,x 41 )

[0076] y max =max(y 36 ,y 37 ,y 38 ,y 39 ,y 40 ,y 41 )

[0077] x max =max(x 36 ,x 37 ,x 38 ,x 39 ,x 40 ,x 41 )

[0078] y min =min(y 36 ,y37 ,y 38 ,y 39 ,y 40 ,y 41 )

[0079] Based on the two-dimensional left eye point set P left_eye The top left corner point P left_up and the bottom right corner point P right_diwn Determine the bounding box of the left eye region image and obtain the left eye region image Image_left.

[0080] Similarly, let P be the two-dimensional right eye point set. right_eye The top left corner point P' left_up =(x' min ,y' max (), the bottom right corner point P' right_down =(x' max ,y' min ),in:

[0081] x' min =min(x' 42 ,x' 43 ,x' 44 ,x' 45 ,x' 46 ,x' 47 )

[0082] y' max =max(y' 42 ,y' 43 ,y' 44 ,y' 45 ,y' 46 ,y' 47 )

[0083] x' max =max(x' 42 ,x' 43 ,x' 44 ,x' 45 ,x' 46 ,x' 47 )

[0084] y' min =min(y' 42 ,y' 43 ,y' 44 ,y' 45 ,y' 46 ,y' 47 )

[0085] Based on the two-dimensional right eye point set P right_eye The top left corner point P' left_upand the bottom right corner point P' right_down Determine the bounding box of the right eye region image and obtain the right eye region image Image_right.

[0086] (S2C) Calculates the driver's two-dimensional face image (face) based on the driver's left-eye region image (Image_left) and right-eye region image (Image_right). 2D Two-dimensional gaze direction

[0087] Specifically, first input the left eye region image Image_left (64×64×3), then process the left eye region image Image... left The convolutional pooling process is as follows: A convolutional layer is used to obtain a feature map of size 64×64×32; a pooling layer is then used to obtain a feature map of size 32×32×32; another convolutional layer is used to obtain a feature map of size 32×32×64; a pooling layer is then used to obtain a feature map of size 16×16×64; a convolutional layer is then used to obtain a feature map of size 16×16×128; a pooling layer is then used to obtain a feature map of size 8×8×128; this feature map is then flattened into a vector of size 1×8192, and a fully connected layer is used to obtain a vector of size 1×512; finally, a fully connected layer is used to obtain the left eye gaze direction. in The left eye's gaze direction is in X Face Y Face The portion on top.

[0088] Similarly, inputting the right eye region image Image_right, performing the same convolutional pooling process as the left eye region image Image_left on the right eye region image Image_right, obtains the right eye gaze direction. in The direction of right eye gaze is X Face Y Face The portion on top.

[0089] Finally, calculate the two-dimensional gaze direction. As shown in the formula below:

[0090]

[0091] (S2D) Based on two-dimensional gaze direction Calculate the two-dimensional gaze cone Cone 2D The process is as follows:

[0092] Set a two-dimensional gaze cone Cone 2D Vertex O Face ;

[0093] Set a two-dimensional gaze cone Cone 2D The direction is obtained from the above calculation.

[0094] Set a two-dimensional gaze cone Cone 2D The vertex angle is θ 2D (0<θ 2D <π);

[0095] Calculate the two-dimensional gaze cone Cone 2D The gaze range is shown in the following formula:

[0096]

[0097] Where (x,y) is the face 2D Any point in it;

[0098] Calculate the distance from point (x, y) to vertex O. Face vector and The cosine of the angle between the two points is used to determine whether the point is in Cone. 2D Inside.

[0099] Step S3: Calculate the three-dimensional gaze cone Cone 3D ,like Figure 3 As shown, the process is as follows:

[0100] (S3A) Based on the driver's two-dimensional face image 2D Obtain the two-dimensional nose point set P nose Two-dimensional nose point set P nose ={(x i ,y i )},x i ,y i The nose feature points are located at X. Face Y Face The position on the nose. Where i is the number of the nose feature point, i = 28, 29, 30, 31, 32, 33, 34, 35, 36.

[0101] (S3B) Based on the driver's 3D face position image dataset {position 3D The driver's 3D face position image in} 3D Obtain a set of key feature points for a 3D face.

[0102] Specifically, input the driver's 3D face position image. 3D Obtain the depth z of 2D facial feature points in 3D facial coordinate system. i =position3D (x i ,y i ,3), where i is the facial feature point number, i=1,2,3,.....68.

[0103] Then obtain the 3D left eye point set P' left_eye ={(x i ,y i ,z i )},x i ,y i ,z i The three-dimensional left eye feature points are located at X. Face Y Face Z Face The position of the feature point on the left eye. Where i is the feature point number of the left eye, i = 36, 37, 38, 39, 40, 41;

[0104] Furthermore, obtain the three-dimensional left-eye right set P' right_eye ={(x i ,y i ,z i )},x i ,y i ,z i The three-dimensional right eye feature points are located in X. Face Y Face Z Face The position of the feature point on the right eye is i, where i is the feature point number of the right eye, i = 42, 43, 44, 45, 46, 47.

[0105] Finally, obtain the 3D nose point set P' nose ={(x i ,y i ,z i )},x i ,y i ,z i The three-dimensional nose feature points are located at X. Face Y Face Z Face The position of the nose feature point is i, where i = 28, 29, 30, 31, 32, 33, 34, 35, 36.

[0106] Thus, a set of key feature points for a three-dimensional face is obtained.

[0107] (S3C) Based on a set of 3D facial key feature points Set of key feature points of a 3D face By performing fully connected layer processing, the driver's three-dimensional gaze direction can be obtained. The fully connected layer processing procedure is as follows:

[0108] A fully connected layer is used to obtain a vector of size 1×256; then a fully connected layer is used to obtain a vector of size 1×128; then a fully connected layer is used to obtain a vector of size 1×64; then a fully connected layer is used to obtain a vector of size 1×32; finally, a fully connected layer is used to obtain the 3D gaze direction. x 3D ,y 3D ,z 3D These are the three-dimensional gaze directions in X Face Y Face Z Face The weight on top.

[0109] (S3D) Based on the driver's three-dimensional gaze direction Calculate the three-dimensional gaze cone Cone 3D The process is as follows:

[0110] Set the 3D gaze cone Cone 3D Vertex O Face ;

[0111] Set the 3D gaze cone Cone 3D The direction is the calculated three-dimensional gaze direction. Set the 3D gaze cone Cone 3D The vertex angle is θ 3D (0<θ 3D <π)

[0112] The maximum gaze distance of the three-dimensional gaze cone is set to h, which represents the effective range of the driver's line of sight;

[0113] Calculate the three-dimensional gaze cone Cone 3D The gaze range is shown in the following formula:

[0114]

[0115] Where (x,y,z) is the position 3D any point in it;

[0116] Calculate the distance from point (x, y, z) to vertex O. Face vector sum The cosine of the angle between the two points is used to determine whether the point is in Cone. 3D Inside.

[0117] Step S4: Calculate the total gaze direction loss like Figure 4 As shown, the specific process is as follows:

[0118] (S4A) Two-dimensional gaze direction calculated based on step S2 Two-dimensional gaze direction dataset in step 1 Two-dimensional gaze direction Calculate the two-dimensional gaze direction loss The calculation formula is as follows:

[0119]

[0120] in, The calculated two-dimensional gaze direction, From 2D gaze direction dataset

[0121] pass calculate and The angle between them.

[0122] (S4B) Based on the three-dimensional gaze direction calculated in step S3 The 3D gaze direction dataset in Step 1 Three-dimensional gaze direction Calculate the loss in three-dimensional gaze direction The calculation formula is as follows:

[0123]

[0124] in, The calculated three-dimensional gaze direction, From 3D gaze direction dataset

[0125] pass calculate and The angle between them.

[0126] (S4C) In step S4, the three-dimensional gaze direction From the three-dimensional face coordinate system Projected onto a two-dimensional face coordinate system This gives the direction of the projected gaze.

[0127] Then, combined with the direction of the projection gaze Two-dimensional gaze direction The two-dimensional-three-dimensional consistent gaze direction loss was calculated. The calculation formula is as follows:

[0128]

[0129] in, The calculated projection gaze direction, The calculated two-dimensional gaze direction;

[0130] pass calculate and The angle between them.

[0131] (S4D) combined with 2D gaze direction loss 3D gaze direction loss 2D-3D consistent gaze direction loss The total gaze direction loss was calculated. The calculation formula is as follows:

[0132]

[0133] Where, λ 2D , λ 3D , λ con These are the learnable weights.

[0134] Step S5: Calculate the total gaze cone loss Combined with total gaze cone loss and the total gaze direction loss obtained in step S4 The total gaze loss was calculated. Total Using total gaze loss Total Training is performed to obtain the optimal parameters of the method.

[0135] In this specific embodiment, for example Figure 5 As shown, total gaze cone loss The calculation process is as follows:

[0136] (S5A) Calculate the loss of a two-dimensional gaze cone The specific process is as follows:

[0137] Calculate the loss of the apex angle of the two-dimensional gaze cone. As shown in the following formula:

[0138]

[0139] Where, θ 2D For two-dimensional staring cone Cone 2D The apex angle; For a two-dimensional gaze cone dataset middle The apex.

[0140] Calculate the two-dimensional gaze cone Cone 2D With 2D gaze cone dataset Two-dimensional staring cone in Area of ​​the union region union As shown in the following formula:

[0141]

[0142] in, For two-dimensional staring cone Cone 2D The range of the gaze; For two-dimensional gaze cone dataset The range of the gaze.

[0143] Calculate the two-dimensional gaze cone Cone 2D With 2D gaze cone dataset Two-dimensional staring cone in Area of ​​intersection int As shown in the following formula:

[0144]

[0145] Based on the area of ​​the union region Area union Area of ​​intersection int Calculate the crossover ratio loss of a two-dimensional staring cone. As shown in the following formula:

[0146]

[0147] Combined with two-dimensional gaze cone apex angle difference loss Two-dimensional gaze cone intersection ratio loss The two-dimensional gaze cone loss was calculated. As shown in the following formula:

[0148]

[0149] (S5B) Calculate the loss of the three-dimensional gaze cone The specific process is as follows:

[0150] Based on the three-dimensional gaze cone obtained in step S3 The apex angle, the 3D gaze cone dataset in step S1 Three-dimensional gaze cone Calculate the apex angle difference loss of the three-dimensional staring cone. As shown in the formula below:

[0151]

[0152] Where, θ 3D For the three-dimensional gaze cone Cone 3D The apex angle; For 3D gaze cone dataset middle The apex.

[0153] Calculate the three-dimensional gaze cone Cone 3D With 3D gaze cone dataset Three-dimensional gaze cone in Union region volume Union The calculation formula is as follows:

[0154]

[0155] in, For the three-dimensional gaze cone Cone 3D The range of the gaze; For the 3D gaze cone dataset The range of the gaze.

[0156] Calculate the three-dimensional gaze cone Cone 3D With 3D gaze cone dataset Three-dimensional gaze cone Volume of the intersection region int The calculation formula is as follows:

[0157]

[0158] Based on the volume of the union region Union Volume of the intersection region int Calculate the crossover ratio loss of a 3D staring cone. The calculation formula is as follows:

[0159]

[0160] Combined with 3D staring cone apex angle difference loss 3D staring cone intersection ratio loss The calculated loss of the three-dimensional gaze cone The calculation formula is as follows:

[0161]

[0162] (S5C) Two-dimensional-three-dimensional consistent gaze cone loss The specific process is as follows:

[0163] Cone with three-dimensional gaze 3D From the three-dimensional face coordinate system Projected onto a two-dimensional face coordinate system Obtain the projection gaze cone Cone proj Cone proj The vertex is O Face Cone proj The direction is in, The projection gaze direction is calculated in step (S4C).

[0164] Calculate the projective gaze cone Cone proj vertex angle θ proj As shown in the following formula:

[0165]

[0166] According to the vertex angle θ proj and two-dimensional gaze cone Cone 2D vertex angle θ 2D The difference in apex angle of the two-dimensional-three-dimensional consistent gaze cone was calculated.

[0167] Calculate Cone proj The gaze range is shown in the following formula:

[0168]

[0169] Where (x,y) is the position 3D Any point projected onto the xoy plane;

[0170] Calculate the distance from point (x, y) to vertex O. Face vector and The cosine of the angle between the two points is used to determine whether the point is in Cone. proj Inside.

[0171] Calculate the projective gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' of the union region Union As shown in the following formula:

[0172]

[0173] Calculate the projective gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' int As shown in the following formula:

[0174]

[0175] Combined Union Area' Union Area' of intersection int Calculate the crossover ratio loss of 2D-3D consistent gaze cones. As shown in the following formula:

[0176]

[0177] Finally, combining the two-dimensional-to-three-dimensional consistent gaze cone apex angle difference loss Two-dimensional-three-dimensional consistent gaze cone intersection-to-union ratio loss The calculation yielded the two-dimensional-three-dimensional consistent gaze cone loss. As shown in the following formula:

[0178]

[0179] (S5D) combined with two-dimensional gaze cone loss 3D staring cone loss Two-dimensional-three-dimensional consistent gaze cone loss Calculate the driver's total gaze cone loss As shown in the following formula:

[0180]

[0181] Where, λ 2D , λ 3D , λ con These are the learnable weights.

[0182] Ultimately, the total gaze loss is... Total The calculation formula is as follows:

[0183]

[0184] Total gaze loss Total Then, using the total gaze loss (Loss) Total Training is performed to obtain the optimal parameters of the method, including the optimal parameters of the fully connected layers in S3C.

[0185] Step S6, as follows Figure 6 As shown, the input is a 2D face image to be detected. The image is processed using a method with optimal parameters. Using PRNET in step S1C, the position image of the driver's 3D face in the 2D face image to be detected is obtained. 3D Using a fully connected layer, the driver's three-dimensional gaze direction in the detected two-dimensional face image is obtained. Using step S3D, the driver's three-dimensional gaze cone (Cone) is obtained from the two-dimensional face image to be detected. 3D .

[0186] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0187] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze, characterized in that, The method comprises the following steps: Step S1, defining a three-dimensional face coordinate system two-dimensional face coordinate system and a driver three-dimensional face position image dataset {position 3D}, a driver two-dimensional face dataset {face 2D}, a three-dimensional gaze direction dataset a two-dimensional gaze direction dataset a three-dimensional gaze cone dataset a two-dimensional gaze cone dataset Step S2, based on the driver two-dimensional face dataset {face 2D} in the driver two-dimensional face image face 2D Obtain a two-dimensional face feature point set, and obtain a left eye region image Image_left and a right eye region image Image_right of the driver based on the two-dimensional face feature point set; then, based on the left eye region image Image_left and the right eye region image Image_right of the driver, calculate the two-dimensional gaze direction 2D Based on the two-dimensional gaze direction Calculate a two-dimensional gaze cone Cone 2D ;​ In step S2, the left eye region image Image_left of the driver is subjected to convolution pooling processing to obtain the left eye gaze direction The right eye region image Image_right of the driver is subjected to convolution pooling processing to obtain the right eye gaze direction Finally, the left eye gaze direction The right eye gaze direction The two-dimensional gaze direction is calculated Based on two-dimensional gaze direction Computing a two-dimensional gaze cone Cone 2d The process is as follows: Setting a two-dimensional gaze cone C 2d with apex O Face ; Setting a two-dimensional gaze cone C 2D in the direction of the above calculated Setting a two-dimensional gaze cone C 2D with an apex angle θ 2D (0 < θ 2D < π); Computing a two-dimensional gaze cone C 2D The gaze range of the gaze cone C is given by the following equation: where (x, y) is any point in face 2D . the cosine of the angle between the vector from the point (x, y) to the vertex O Face and the vector between the vertex O and the point (x, y) is used to determine if the point is inside the Cone 2D ; Step S3, obtaining a two-dimensional face image face of the driver based on the two-dimensional face image face 2D obtaining a two-dimensional nose point set P nose ; obtaining a three-dimensional face position image position of the driver based on a three-dimensional face position image data set {position 3D} of the driver 3D obtaining a three-dimensional face key feature point set then calculating a three-dimensional gaze direction of the driver based on the three-dimensional face key feature point set then calculating a three-dimensional gaze cone Cone based on the three-dimensional gaze direction of the driver 3D ;​​ In step S3, the three-dimensional face position image position of the driver is obtained based on the three-dimensional face position image position of the driver 3D The three-dimensional left eye point set P' is obtained left_eye The three-dimensional right eye point set P' is obtained right_eye The three-dimensional nose point set P' is obtained nose Thus, the three-dimensional face key feature point set is formed Then, the three-dimensional face key feature point set is processed by a full connection layer, and thus the three-dimensional gaze direction of the driver is obtained ​ Based on the driver's three-dimensional gaze direction Computing a three-dimensional gaze cone C 3D The process is as follows: Setting a three-dimensional gaze cone C 3D The apex is O Face ; The direction of the three-dimensional gaze cone Cone is the calculated three-dimensional gaze direction 3D The direction of the three-dimensional gaze cone Cone is the calculated three-dimensional gaze direction The direction of the three-dimensional gaze cone Cone is the calculated three-dimensional gaze direction 3D The apex angle of the three-dimensional gaze cone Cone is θ 3D (0 < θ < π) 3D < π) Setting the maximum gaze distance of the three-dimensional gaze cone as h, representing the effective range of the driver's line of sight; Computing a three-dimensional gaze cone C 3D The gaze range of the gaze cone C is given by the following equation: where (x, y, z) is any one of the points of position 3D of the set of points the cosine of the angle between the vector from the point (x, y, z) to the vertex O Face and the vector from the vertex O to the vertex C is used to determine if the point is inside the Cone 3D ; Step S4, computing a two-dimensional gaze direction loss three-dimensional gaze direction loss two-dimensional-three-dimensional consistency gaze direction loss combining the two-dimensional gaze direction loss three-dimensional gaze direction loss two-dimensional-three-dimensional consistency gaze direction loss computing a total gaze direction loss Step S5, calculating two-dimensional gaze cone loss three-dimensional gaze cone loss two-dimensional-three-dimensional consistent gaze cone loss combined two-dimensional gaze cone loss three-dimensional gaze cone loss two-dimensional-three-dimensional consistent gaze cone loss calculating total gaze cone loss for the driver Then, the total gaze cone loss Losscone is calculated by combining the total gaze cone loss LossconeS4 obtained in step S4 and the total gaze direction loss Lossdir obtained in step S3 The total gaze loss Loss is calculated Total ; Finally, the total gaze loss is used for training to obtain the optimal parameters of the method; In step S6, a two-dimensional face image to be detected is input, and the method under the optimal parameters is used to process the two-dimensional face image to be detected, so as to obtain a driver three-dimensional face position image, a driver three-dimensional gaze direction and a driver three-dimensional gaze cone in the two-dimensional face image to be detected.

2. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze according to claim 1, characterized in that, In step S2, the left eye region image Image_left of the driver is subjected to convolution pooling processing to obtain the left eye gaze direction The right eye region image Image_right of the driver is subjected to convolution pooling processing to obtain the right eye gaze direction Finally, the left eye gaze direction The right eye gaze direction The two-dimensional gaze direction is calculated 3. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S3, the three-dimensional face position image position of the driver is obtained based on the three-dimensional face position image position of the driver 3D The three-dimensional left eye point set P' is obtained left_eye The three-dimensional right eye point set P' is obtained right_eye The three-dimensional nose point set P' is obtained nose Thus, the three-dimensional face key feature point set is formed Then, the three-dimensional face key feature point set is processed by a full connection layer, and thus the three-dimensional gaze direction of the driver is obtained ​ 4. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S4, the two-dimensional gaze direction is calculated based on the two-dimensional gaze direction calculated in step S2 The two-dimensional gaze direction data set in step 1 The two-dimensional gaze direction in step S2 Calculating the two-dimensional gaze direction loss 5. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S4, the three-dimensional gaze direction is calculated based on the three-dimensional gaze direction calculated in step S3 The three-dimensional gaze direction data set in step 1 The three-dimensional gaze direction in Calculating the three-dimensional gaze direction loss 6. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S4, the three-dimensional gaze direction from the three-dimensional face coordinate system is projected to the two-dimensional face coordinate system Thus, the projected gaze direction is obtained. Then, the projected gaze direction is combined with the two-dimensional gaze direction to obtain the two-dimensional-three-dimensional consistent gaze direction loss 7. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S5, based on the two-dimensional gaze cone Cone obtained in step S2... 2D vertex angle θ 2D Step S1: Two-dimensional gaze cone dataset Two-dimensional staring cone in apex Calculate the loss of the apex angle of the two-dimensional gaze cone. calculating a two-dimensional gaze cone cone 2D with a two-dimensional gaze cone data set a union area area of two-dimensional gaze cones in the two-dimensional gaze cone data set union and a two-dimensional gaze cone cone 2D with a two-dimensional gaze cone data set a intersection area area of two-dimensional gaze cones in the two-dimensional gaze cone data set int then calculating a two-dimensional gaze cone intersection-union ratio loss union from the union area area int the intersection area area 3D Combining two-dimensional gaze cone apex angle loss Two-dimensional gaze cone foveal loss Computing two-dimensional gaze cone loss 8. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S5, a three-dimensional gaze cone angle difference loss is calculated based on the three-dimensional gaze cone angle obtained in step S3, the three-dimensional gaze cone angle in the three-dimensional gaze cone data set in step S1, and the three-dimensional gaze cone angle in the three-dimensional gaze cone in step S4. 3D ​​​ computing a three-dimensional gaze cone volume 3D with three-dimensional gaze cones in a three-dimensional gaze cone dataset of three-dimensional gaze cones in the three-dimensional gaze cone dataset Union and three-dimensional gaze cone volume 3D with three-dimensional gaze cones in a three-dimensional gaze cone dataset of three-dimensional gaze cones in the three-dimensional gaze cone dataset int then computing a three-dimensional gaze cone intersection-over-union loss Union int ​​​​ Combining three-dimensional gaze cone vertex angle loss Three-dimensional gaze cone foveal loss Computing three-dimensional gaze cone loss 9. The driver gaze direction detection method based on two-dimensional three-dimensional face consistency gaze of claim 1, wherein, In step S5, the three-dimensional gaze cone Cone 3D is projected to the two-dimensional face coordinate system , resulting in a projected gaze cone Cone . proj The apex angle Θ proj of the projected gaze cone Cone proj is calculated. According to the apex angle θ proj and the apex angle θ 2D of the two-dimensional gaze cone Cone 2D , the two-dimensional-three-dimensional consistency gaze cone apex angle difference loss is calculated Calculate the projective gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' of the union region Union And the projected gaze cone Cone proj With two-dimensional gaze cone Cone 2D Area' int Combined with the area of ​​the union region Area′ Union Area′ of the intersection region int Calculate the crossover ratio loss of 2D-3D consistent gaze cones. Then, combining the two-dimensional-three-dimensional consistency gaze pyramid apex angle difference loss Two-dimensional-three-dimensional consistency gaze pyramid intersection over union loss The calculated two-dimensional-three-dimensional consistency gaze pyramid loss is calculated

Images