Driver staring direction detection method based on two-dimensional and three-dimensional face consistent staring

Abstract

The invention discloses a driver staring direction detection method based on two-dimensional and three-dimensional face consistent staring, and the method comprises the steps: estimating the staring direction of a two-dimensional face and the staring cone of the two-dimensional face through employing the face features of a two-dimensional convolutional network; according to the invention, depth features of three-dimensional face reconstruction are used to estimate the staring direction and the staring cone of the three-dimensional face. In order to verify the accuracy of the staring direction of the three-dimensional face, the consistency loss of the staring direction of the three-dimensional face and the staring direction of the two-dimensional face is calculated In order to verify the accuracy of the staring cone of the three-dimensional face, the consistency loss of the staring cone of the three-dimensional face and the staring cone of the two-dimensional face is calculated According to the method, the two-dimensional-three-dimensional consistent staring direction and the two-dimensional-three-dimensional staring cone consistency are combined, so that the reliability of two-dimensional face feature extraction and three-dimensional face feature extraction can be ensured, and accurate staring direction detection is realized.

Description

Technical Field

[0001] The invention relates to the field of driving safety, and in particular to a method for detecting a driver's gaze direction based on consistent gaze of two-dimensional and three-dimensional faces. Background Art

[0002] With the development of autonomous driving technology, driver attention monitoring has become particularly important. In order to ensure that the driver maintains proper attention during driving, gaze direction detection technology has gradually become an important research field. Existing gaze direction detection methods are mainly divided into two categories: 2D face feature extraction and 3D face reconstruction.

[0003] Two-dimensional facial feature extraction methods usually use convolutional neural networks (CNNs) to process facial images and estimate gaze direction. However, due to the lack of depth information, these methods have certain limitations in accuracy and robustness, especially when facing lighting changes, occlusions, and changes in head posture, the detection accuracy will drop significantly. In addition, relying solely on two-dimensional information cannot fully reflect the three-dimensional structural characteristics of the face, which affects the judgment of gaze direction.

[0004] On the other hand, 3D face reconstruction technology can more accurately model the spatial structure of the face by capturing the depth information of the face. However, existing 3D reconstruction methods are often complex and computationally intensive, and are easily limited by hardware performance in practical applications. In addition, gaze direction detection methods that rely solely on 3D information may have insufficient stability in some scenarios.

[0005] At present, although some studies have tried to combine two-dimensional and three-dimensional information to improve the accuracy and robustness of gaze direction detection, these methods generally have the following problems: First, the fusion of two-dimensional and three-dimensional information often lacks an effective mechanism to verify the consistency between the two, which leads to inaccurate detection results; second, the existing methods have high computational complexity in practical applications and are difficult to meet real-time requirements. Therefore, how to effectively combine two-dimensional and three-dimensional face information and ensure consistent gaze between the two has become a problem that needs to be solved urgently.

[0006] Chinese patent application publication number CN117935334A discloses a driver line of sight estimation method and system based on three-dimensional reconstruction of the face. The method and system recover the three-dimensional shape of the face from a single ordinary face image, and estimate the line of sight direction of the person based on the three-dimensional shape, the anatomical structure and movement law of the human eye. Compared with the method that relies solely on three-dimensional face reconstruction, the present invention effectively makes up for the shortcomings of three-dimensional reconstruction's dependence on image quality and high computational complexity by combining the gaze direction and consistent gaze cone of a two-dimensional convolutional network and three-dimensional depth features. It improves the adaptability and detection accuracy in dynamic scenes through the mutual verification of two-dimensional and three-dimensional information, while alleviating the problem of poor adaptability of the pure three-dimensional method to individual facial differences, ensuring more stable and accurate gaze direction detection.

[0007] Chinese patent application publication number CN115376112A provides a method and device for detecting a driver's line of sight. The method determines, for each target area, that the driver's line of sight passes through the target area. However, the invention relies on the precise calibration of multiple cameras, is susceptible to errors, and lacks three-dimensional depth information, which may lead to misjudgment of line of sight in complex scenes. In addition, the multi-camera system is susceptible to occlusion at certain angles, reducing the accuracy of line of sight detection. In contrast, the two-dimensional-three-dimensional combined method is more robust. Summary of the invention

[0008] The purpose of the present invention is to remedy the defects of the prior art and provide a method for detecting the gaze direction of a driver based on consistent gaze of two-dimensional and three-dimensional faces.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze includes the following steps:

[0011] Step S1: define a 3D face coordinate system 2D face coordinate system And the driver's 3D face position image dataset {position 3D}、Driver 2D face dataset {face 2D}, the three-dimensional gaze direction dataset is The two-dimensional gaze direction dataset is The 3D gaze cone dataset is The two-dimensional gaze cone dataset is

[0012] Step S2: Based on the driver's two-dimensional face dataset {face 2D}The driver's two-dimensional face image face 2DGet a set of two-dimensional facial feature points, and obtain the driver's left eye area image Image_left and right eye area image Image_right based on the two-dimensional facial feature point set; then calculate the driver's two-dimensional facial image face based on the driver's left eye area image Image_left and right eye area image Image_right 2D The two-dimensional gaze direction in Based on the two-dimensional gaze direction Calculate the 2D gaze cone 2D ;

[0013] Step S3: based on the driver's two-dimensional face image face 2D Get the two-dimensional nose point set P nose ; Based on the driver's 3D face position image dataset {position 3D}The driver's three-dimensional face position image position 3D , get the set of key feature points of 3D face Then based on the set of key feature points of the 3D face Calculate the driver's 3D gaze direction Based on the driver's three-dimensional gaze direction Calculate the 3D gaze cone 3D ;

[0014] Step S4: Calculate the two-dimensional gaze direction loss 3D gaze direction loss 2D-3D consistent gaze direction loss Combining 2D gaze direction loss 3D gaze direction loss 2D-3D consistent gaze direction loss The total gaze direction loss is calculated

[0015] Step S5: Calculate the two-dimensional gaze cone loss 3D gaze cone loss 2D-3D consistent gaze cone loss Combining 2D gaze cone loss , 3D gaze cone loss 2D-3D consistent gaze cone loss Calculate the driver's total gaze cone loss

[0016] Then, combined with the total gaze cone loss And the total gaze direction loss obtained in step S4 Calculate the total gaze loss Loss Total ;

[0017] Finally, the total gaze loss is used for training to obtain the optimal parameters of the method;

[0018] Step S6, input the two-dimensional face image to be detected, and process the two-dimensional face image to be detected by the method under the optimal parameters to obtain the three-dimensional face position image of the driver, the three-dimensional gaze direction of the driver, and the three-dimensional gaze cone of the driver in the two-dimensional face image to be detected.

[0019] Furthermore, in step S2, the driver's left eye area image Image_left is subjected to convolution pooling to obtain the left eye gaze direction Perform convolution pooling on the driver’s right eye area image Image_right to obtain the right eye gaze direction Finally, combined with the left eye gaze direction Right eye gaze direction Calculate the 2D gaze direction

[0020] Further, in step S3, based on the driver's three-dimensional face position image position 3D Get the 3D left eye point set P' left_eye , 3D right eye point set P' right_eye , 3D nose point set P' nose , thus forming a set of key feature points of a three-dimensional face Then the key feature point set of the 3D face Perform full connection layer processing to obtain the driver's three-dimensional gaze direction

[0021] Further, in step S4, based on the two-dimensional gaze direction calculated in step S2 The 2D gaze direction dataset from step 1 The two-dimensional gaze direction in Compute 2D gaze direction loss

[0022] Further, in step S4, based on the three-dimensional gaze direction calculated in step S3 The 3D gaze direction dataset from step 1 The three-dimensional gaze direction in Compute 3D gaze direction loss

[0023] Furthermore, in step S4, the three-dimensional gaze direction From the 3D face coordinate system Projection to 2D face coordinate system This gives the projected gaze direction Then, combined with the projected gaze direction 2D gaze direction Calculate the 2D-3D consistent gaze direction loss

[0024] Further, in step S5, based on the two-dimensional gaze cone Cone obtained in step S2 2D The vertex angle θ 2D , the two-dimensional gaze cone dataset in step S1 The two-dimensional gaze cone in The top angle Calculate the 2D gaze cone vertex angle difference loss

[0025] Calculate the 2D gaze cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The area of ​​the union of union , and the two-dimensional gaze cone Cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The intersection area is Area int , and then according to the area of ​​the union area union 、Intersection area Area int Compute the 2D gaze cone intersection-over-union loss

[0026] Combined with 2D gaze cone vertex angle difference loss 2D gaze cone intersection-over-union loss Calculate the two-dimensional gaze cone loss

[0027] Further, in step S5, based on the three-dimensional gaze cone Cone obtained in step S3 3D The vertex angle of the 3D gaze cone dataset in step S1 The three-dimensional gaze cone in The vertex angle of the three-dimensional gaze cone is calculated

[0028] Calculate the 3D gaze cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The union area volume Volume Union , and the three-dimensional gaze cone Cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The intersection area volume Volumeint , and then according to the volume of the union area Volume Union , intersection area volume Volume int Compute the 3D gaze cone intersection-over-union loss

[0029] Combined with 3D gaze cone vertex angle difference loss 3D gaze cone intersection-over-union loss Calculate the three-dimensional gaze cone loss

[0030] Furthermore, in step S5, the three-dimensional gaze cone Cone 3D From the 3D face coordinate system Projection to 2D face coordinate system Get the projected gaze cone Cone proj , and calculate the projected gaze cone Cone proj The vertex angle θ proj ;

[0031] According to the vertex angle θ proj and the two-dimensional gaze cone 2D The vertex angle θ 2D , the 2D-3D consistent gaze cone vertex difference loss is calculated

[0032] Calculate the projected gaze cone proj and the 2D gaze cone 2D The area of ​​the union of Area' Union , and the projected gaze cone Cone proj and the 2D gaze cone 2D The intersection area is Area' int , combined with the area of ​​the union area Area' Union , intersection area Area' int Compute 2D-3D consistent gaze cone intersection loss

[0033] Then, we combine the 2D-3D consistent gaze cone vertex difference loss 2D-3D consistent gaze cone intersection loss Calculate the 2D-3D consistent gaze cone loss

[0034] The present invention calculates the two-dimensional gaze direction loss and the two-dimensional gaze cone loss by extracting two-dimensional features, calculates the three-dimensional gaze direction loss and the three-dimensional gaze cone loss by extracting three-dimensional features, and projects the three-dimensional features onto a two-dimensional plane to calculate the consistent gaze direction loss and the consistent gaze cone loss. The three-dimensional gaze direction and the three-dimensional gaze cone of the driver are accurately obtained by training the parameters of the total gaze direction loss and the total gaze cone loss optimization method.

[0035] The two-dimensional and three-dimensional face consistent gaze detection method of the present invention has significant advantages. First, by combining the two-dimensional convolutional network and the three-dimensional face reconstruction technology, it makes full use of the detail information and three-dimensional depth features of the two-dimensional image, and makes up for the limitations of a single method. The two-dimensional method has strong robustness in dealing with complex lighting conditions and occlusions, while the three-dimensional method provides accurate spatial structure information. By calculating the consistency loss of the two-dimensional and three-dimensional gaze directions and gaze cones, the method can verify and correct errors with each other, thereby improving the accuracy and reliability of gaze direction detection. In addition, the combination of two-dimensional and three-dimensional information effectively enhances the adaptability of the system in dynamic scenes, ensuring accurate line of sight detection in different driving environments. In short, the method provides a high-precision and high-robust driver gaze direction detection solution through multi-dimensional information fusion, which is suitable for complex needs in actual driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart for defining a coordinate system and a data set mentioned in the present invention.

[0037] Figure 2 This is a flow chart of calculating the two-dimensional gaze cone mentioned in the present invention.

[0038] Figure 3 This is a flow chart of calculating the three-dimensional gaze cone mentioned in the present invention.

[0039] Figure 4 This is a flow chart of calculating the total gaze direction loss mentioned in the present invention.

[0040] Figure 5 This is a flow chart for calculating the total gaze cone loss mentioned in the present invention.

[0041] Figure 6 This is a flow chart of obtaining a three-dimensional gaze cone according to the present invention.

[0042] Figure 7 The figure is a flowchart of an implementation of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0044] like Figure 7 As shown, this embodiment discloses a method for detecting a driver's gaze direction based on consistent gaze of two-dimensional and three-dimensional faces, comprising the following steps:

[0045] Step S1: define the required coordinate system and data set. Figure 1 As shown, the coordinate system and data set defined in this embodiment include the following parts:

[0046] (S1A) Define the 3D face coordinate system in The coordinate origin O Face The tip of the nose on the face; coordinate system X axis Face is the horizontal axis of the image, from left to right; coordinate system Y axis Face is the vertical axis of the image, from top to bottom; coordinate system Z axis Face It is the image depth axis, from far to near.

[0047] (S1B) Define the 2D face coordinate system in The coordinate origin O Face The tip of the nose on the face; coordinate system X axis Face is the horizontal axis of the image, from left to right; coordinate system Y axis Face It is the vertical axis of the image, running from top to bottom.

[0048] (S1C) defines the driver's 3D face position image dataset {position 3D Specifically, input the driver's two-dimensional face data face 2D , use PRNET (joint 3D Face Reconstruction and Dense Alignmentwith Position Map Regression Network. In ECCV 2018.) to obtain the driver's 3D face position image position 3D (256×256×3). Among them, face 2D is a 2D face image of the driver, with a size of 256×256×3; a 3D face position image position 3D The three channels are respectively The three coordinates x, y, and z in .

[0049] (S1D) defines the driver 2D face dataset {face 2D}

[0050] (S1E) The three-dimensional gaze direction dataset is defined as where x gt ,y gt ,z gt The three-dimensional gaze direction is in X Face , Y Face , Z Face The amount on.

[0051] (S1F) defines a two-dimensional gaze direction dataset as where x gt ,y gt The two-dimensional gaze direction is in X Face , Y Face The amount on.

[0052] (S1G) Define the 3D gaze cone dataset as The process is as follows:

[0053] Set up a 3D gaze cone The vertex is O Face ;

[0054] Set up a 3D gaze cone The direction is

[0055] Set up a 3D gaze cone The vertex is

[0056] Set up a 3D gaze cone The maximum gaze distance is h gt , indicating the effective range of the driver's line of sight;

[0057] Computing the 3D gaze cone The gaze range is given by:

[0058]

[0059] Among them, (x, y, z) is the three-dimensional face position image position 3D The coordinates of any point in ;

[0060] Calculate the distance from point (x, y, z) to vertex O Face The vector and The cosine value of the angle between is used to determine whether the point is Inside.

[0061] (S1H) Define the two-dimensional gaze cone dataset as The process is as follows:

[0062] Setting up the 2D gaze cone The vertex is O Face ;

[0063] Setting up the 2D gaze cone The direction is

[0064] Setting up the 2D gaze cone The vertex angle is

[0065] Computing the 2D gaze cone The gaze range is given by:

[0066]

[0067] Among them, (x, y) is the driver’s two-dimensional face image face 2D Any point in ;

[0068] Calculate the distance from point (x, y) to vertex O Face The vector and The cosine value of the angle between is used to determine whether the point is Inside.

[0069] Step S2: Calculate the two-dimensional gaze cone Cone 2D ,like Figure 2 As shown, the process is as follows:

[0070] (S2A) based on the driver 2D face dataset {face 2D}The driver's two-dimensional face image face 2D , use the model Dlib's 68-point facial landmark detector to obtain the two-dimensional facial feature point set P face ={(x i ,y i )},x i ,y i They are the two-dimensional facial feature points in X Face , Y Face The position on the face, where i is the facial feature point number, i = 1, 2, 3, ..... 68.

[0071] From the feature point set P face ={(x i ,y i )} to obtain the two-dimensional left eye point set P left_eye ={(x i ,y i)}, where i is the left eye feature point number, i = 36, 37, 38, 39, 40, 41.

[0072] From the feature point set P face ={(x i ,y i )} to obtain the two-dimensional right eye point set P right_eye ={(x i ,y i )}, where i is the feature point number of the right eye, i=42,43,44,45,46,47.

[0073] (S2B) Based on the two-dimensional face feature point set P face The two-dimensional left eye point set P in left_eye , 2D right eye point set P right_eye , respectively obtain the driver's two-dimensional face image face 2D The left eye area image Image_left and the right eye area image Image_right in the image are as follows:

[0074] Assume that the two-dimensional left eye point set P left_eye The upper left corner point P left_up =(x min ,y max ), the bottom right point P right_down =(x max ,y min ),in:

[0075] x min =min(x 36 ,x 37 ,x 38 ,x 39 ,x 40 ,x 41 )

[0076] y max =max(y 36 ,y 37 ,y 38 ,y 39 ,y 40 ,y 41 )

[0077] x max =max(x 36 ,x 37 ,x 38 ,x 39 ,x 40 ,x 41 )

[0078] y min =min(y 36 ,y37 ,y 38 ,y 39 ,y 40 ,y 41 )

[0079] According to the two-dimensional left eye point set P left_eye The upper left corner point P left_up and the lower right point P right_diwn , determine the bounding box of the left eye area image, and obtain the left eye area image Image_left.

[0080] Similarly, let the two-dimensional right eye point set P right_eye The upper left corner point P' left_up =(x' min ,y' max ), the lower right point P' right_down =(x' max ,y' min ),in:

[0081] x' min =min(x' 42 ,x' 43 ,x' 44 ,x' 45 ,x' 46 ,x' 47 )

[0082] y' max =max(y' 42 ,y' 43 ,y' 44 ,y' 45 ,y' 46 ,y' 47 )

[0083] x' max =max(x' 42 ,x' 43 ,x' 44 ,x' 45 ,x' 46 ,x' 47 )

[0084] y' min =min(y' 42 ,y' 43 ,y' 44 ,y' 45 ,y' 46 ,y' 47 )

[0085] According to the two-dimensional right eye point set P right_eye The upper left corner point P' left_upand the lower right point P' right_down , determine the bounding box of the right eye area image, and obtain the right eye area image Image_right.

[0086] (S2C) Calculate the driver's two-dimensional face image face based on the driver's left eye area image Image_left and right eye area image Image_right 2D The two-dimensional gaze direction in

[0087] Specifically, first input the left eye area image Image_left (64×64×3), and left Convolution and pooling processing is performed, and the process is as follows: use the convolution layer to obtain a feature map of size 64×64×32; then use the pooling layer to obtain a feature map of size 32×32×32; then use the convolution layer to obtain a feature map of size 32×32×64; then use the pooling layer to obtain a feature map of size 16×16×64; then use the convolution layer to obtain a feature map of size 16×16×128; then use the pooling layer to obtain a feature map of size 8×8×128; then flatten it into a vector of size 1×8192, use the fully connected layer to obtain a vector of size 1×512; finally use the fully connected layer to obtain the left eye gaze direction in The left eye gaze direction is in X Face , Y Face The amount on.

[0088] Similarly, input the right eye area image Image_right, perform the same convolution pooling process on the right eye area image Image_right as the left eye area image Image_left, and obtain the right eye gaze direction in The gaze direction of the right eye is in X Face , Y Face The amount on.

[0089] Finally, the 2D gaze direction is calculated As shown in the following formula:

[0090]

[0091] (S2D) based on two-dimensional gaze direction Calculate the 2D gaze cone 2D , the process is as follows:

[0092] Set the 2D gaze cone 2D The vertex is O Face ;

[0093] Set the 2D gaze cone 2D The direction is calculated above.

[0094] Set the 2D gaze cone 2D The vertex angle is θ 2D (0<θ 2D <π);

[0095] Calculate the 2D gaze cone 2D The gaze range is given by:

[0096]

[0097] Among them, (x, y) is face 2D Any point in ;

[0098] Calculate the distance from point (x, y) to vertex O Face The vector and The cosine value of the angle between the points is used to determine whether the point is in the Cone 2D Inside.

[0099] Step S3: Calculate the three-dimensional gaze cone Cone 3D ,like Figure 3 As shown, the process is as follows:

[0100] (S3A) Based on the driver's two-dimensional face image face 2D Get the two-dimensional nose point set P nose , two-dimensional nose point set P nose ={(x i ,y i )},x i ,y i The nose feature points are respectively Face , Y Face Where i is the number of the nose feature point, i = 28, 29, 30, 31, 32, 33, 34, 35, 36.

[0101] (S3B) Based on the driver's 3D face position image dataset {position 3D}The driver's three-dimensional face position image position 3D , get the set of key feature points of 3D face

[0102] Specifically, input the driver's 3D face position image position 3D , get the depth z of the two-dimensional face feature points in the three-dimensional face coordinate system i =position3D (x i ,y i ,3), where i is the facial feature point number, i=1,2,3,.....68.

[0103] Then get the three-dimensional left eye point set P' left_eye ={(x i ,y i ,z i )},x i ,y i ,z i They are the three-dimensional left eye feature points at X Face , Y Face , Z Face , where i is the left eye feature point number, i = 36, 37, 38, 39, 40, 41;

[0104] And, obtain the three-dimensional left eye right set P' right_eye ={(x i ,y i ,z i )},x i ,y i ,z i They are the three-dimensional right eye feature points at X Face , Y Face , Z Face , where i is the feature point number of the right eye, i=42,43,44,45,46,47.

[0105] Finally, obtain the three-dimensional nose point set P' nose ={(x i ,y i ,z i )},x i ,y i ,z i The three-dimensional nose feature points are respectively Face , Y Face , Z Face , where i is the nose feature point number, i = 28, 29, 30, 31, 32, 33, 34, 35, 36.

[0106] Thus, we get the key feature point set of the 3D face

[0107] (S3C) based on a set of key feature points of a 3D face The key feature point set of 3D face Perform full connection layer processing to obtain the driver's three-dimensional gaze direction The processing process of the fully connected layer is:

[0108] Use a fully connected layer to get a vector of size 1×256; then use a fully connected layer to get a vector of size 1×128; then use a fully connected layer to get a vector of size 1×64; then use a fully connected layer to get a vector of size 1×32; finally use a fully connected layer to get the 3D gaze direction x 3D ,y 3D ,z 3D The three-dimensional gaze direction is in X Face , Y Face , Z Face , the weight on it.

[0109] (S3D) based on the driver's three-dimensional gaze direction Calculate the 3D gaze cone 3D , the process is as follows:

[0110] Set the 3D gaze cone 3D The vertex is O Face ;

[0111] Set the 3D gaze cone 3D The direction is the calculated three-dimensional gaze direction Set the 3D gaze cone 3D The vertex angle is θ 3D (0<θ 3D <π)

[0112] Set the maximum gaze distance of the three-dimensional gaze cone to h, which represents the effective range of the driver's line of sight;

[0113] Calculate the 3D gaze cone 3D The gaze range is given by:

[0114]

[0115] Among them, (x, y, z) is position 3D Any point in ;

[0116] Calculate the distance from point (x, y, z) to vertex O Face The vector and The cosine value of the angle between the points is used to determine whether the point is in the Cone 3D Inside.

[0117] Step S4: Calculate the total gaze direction loss like Figure 4 As shown, the specific process is as follows:

[0118] (S4A) Based on the two-dimensional gaze direction calculated in step S2 The 2D gaze direction dataset from step 1 The two-dimensional gaze direction in Compute 2D gaze direction loss The calculation formula is as follows:

[0119]

[0120] in, is the calculated 2D gaze direction, From the 2D gaze direction dataset

[0121] pass calculate and The angle between.

[0122] (S4B) Based on the three-dimensional gaze direction calculated in step S3 The 3D gaze direction dataset from step 1 The three-dimensional gaze direction in Compute 3D gaze direction loss The calculation formula is as follows:

[0123]

[0124] in, is the calculated 3D gaze direction, From the 3D Gaze Direction Dataset

[0125] pass calculate and The angle between.

[0126] (S4C) In step S4, the three-dimensional gaze direction From the 3D face coordinate system Projection to 2D face coordinate system This gives the projected gaze direction

[0127] Then, combined with the projected gaze direction 2D gaze direction Calculate the 2D-3D consistent gaze direction loss The calculation formula is as follows:

[0128]

[0129] in, is the calculated projected gaze direction, is the calculated two-dimensional gaze direction;

[0130] pass calculate and The angle between.

[0131] (S4D) Combined with 2D gaze direction loss 3D gaze direction loss , 2D-3D consistent gaze direction loss The total gaze direction loss is calculated The calculation formula is as follows:

[0132]

[0133] Among them, 2D ,λ 3D ,λ con are the learnable weights respectively.

[0134] Step S5: Calculate the total gaze cone loss Combined total gaze cone loss And the total gaze direction loss obtained in step S4 Calculate the total gaze loss Loss Total Using the total gaze loss Loss Total Training is performed to obtain the optimal parameters of the method.

[0135] Specifically in this embodiment, Figure 5 As shown, the total gaze cone loss The calculation process is as follows:

[0136] (S5A) Computing 2D gaze cone loss The specific process is as follows:

[0137] Calculate the 2D gaze cone vertex angle difference loss As shown below:

[0138]

[0139] Among them, θ 2D Cone is a two-dimensional gaze cone 2D The top angle of A two-dimensional gaze cone dataset middle The top angle.

[0140] Calculate the 2D gaze cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The area of ​​the union of union , as shown below:

[0141]

[0142] in, Cone is a two-dimensional gaze cone 2D gaze range; For the two-dimensional gaze cone data set gaze range.

[0143] Calculate the 2D gaze cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The intersection area is Area int , as shown below:

[0144]

[0145] According to the area of ​​the union area union 、Intersection area Area int Compute the 2D gaze cone intersection-over-union loss As shown below:

[0146]

[0147] Combined with 2D gaze cone vertex angle difference loss 2D gaze cone intersection-over-union loss Calculate the two-dimensional gaze cone loss As shown below:

[0148]

[0149] (S5B) Calculation of 3D gaze cone loss The specific process is as follows:

[0150] Based on the three-dimensional gaze cone obtained in step S3 The vertex angle of the 3D gaze cone dataset in step S1 The three-dimensional gaze cone in The vertex angle of the three-dimensional gaze cone is calculated As shown in the following formula:

[0151]

[0152] Among them, θ 3D Cone is a three-dimensional gaze cone 3D The top angle of A 3D gaze cone dataset middle The top angle.

[0153] Calculate the 3D gaze cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The union area volume Volume Union , the calculation formula is as follows:

[0154]

[0155] in, Cone is a three-dimensional gaze cone 3D gaze range; Centralize the 3D gaze cone dataset gaze range.

[0156] Calculate the 3D gaze cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The intersection area volume Volume int , the calculation formula is as follows:

[0157]

[0158] Based on the volume of the union area Union , intersection area volume Volume int Compute the 3D gaze cone intersection-over-union loss The calculation formula is as follows:

[0159]

[0160] Combined with 3D gaze cone vertex angle difference loss 3D gaze cone intersection-over-union loss Calculate the three-dimensional gaze cone loss The calculation formula is as follows:

[0161]

[0162] (S5C) 2D-3D Consistency Gaze Cone Loss The specific process is as follows:

[0163] The three-dimensional gaze cone 3D From the 3D face coordinate system Projection to 2D face coordinate system Get the projected gaze cone Cone proj , Cone proj The vertex is O Face , Cone proj The direction is in, It is the projected gaze direction calculated in step (S4C).

[0164] Calculate the projected gaze cone proj The vertex angle θ proj As shown below:

[0165]

[0166] According to the vertex angle θ proj and the two-dimensional gaze cone 2D The vertex angle θ 2D , the 2D-3D consistent gaze cone vertex difference loss is calculated

[0167] Calculate Cone proj The gaze range is given by the following formula:

[0168]

[0169] Among them, (x, y) is position 3D Any point in the projection on the xoy plane;

[0170] Calculate the distance from point (x, y) to vertex O Face The vector and The cosine value of the angle between the points is used to determine whether the point is in the Cone proj Inside.

[0171] Calculate the projected gaze cone proj and the 2D gaze cone 2D The area of ​​the union of Area' Union , as shown below:

[0172]

[0173] Calculate the projected gaze cone proj and the 2D gaze cone 2D The intersection area is Area' int , as shown below:

[0174]

[0175] Combined area Area' Union , intersection area Area' int Compute 2D-3D consistent gaze cone intersection loss As shown below:

[0176]

[0177] Finally, we combine the 2D-3D consistent gaze cone vertex difference loss 2D-3D consistent gaze cone intersection loss Calculate the 2D-3D consistent gaze cone loss As shown below:

[0178]

[0179] (S5D) Combined with 2D gaze cone loss 3D gaze cone loss 2D-3D consistent gaze cone loss Calculating the driver's total gaze cone loss As shown below:

[0180]

[0181] Among them, λ 2D ,λ 3D ,λ con are the learnable weights respectively.

[0182] Finally, the total gaze loss Loss Total The calculation formula is as follows:

[0183]

[0184] Get the total gaze loss Loss Total Then, using the total gaze loss Loss Total Training is performed to obtain the optimal parameters of the method, including the optimal parameters of the fully connected layer in S3C.

[0185] Step S6: Figure 6 As shown, the two-dimensional face image to be detected is input, the two-dimensional face image to be detected is processed using the method under the optimal parameters, and the PRNET in step S1C is used to obtain the three-dimensional face position image position of the driver in the two-dimensional face image to be detected 3D ; Use the fully connected layer to obtain the 3D gaze direction of the driver in the 2D face image to be detected Using step S3D, obtain the driver's three-dimensional gaze cone Cone in the two-dimensional face image to be detected 3D .

[0186] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction, and such combinations should also be regarded as the contents disclosed in the present disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0187] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by technical personnel in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A driver gaze direction detection method based on two-dimensional and three-dimensional face consistent gaze, characterized in that: The following steps are involved: Step S1: define a 3D face coordinate system 2D face coordinate system And the driver's 3D face position image dataset {position 3D }、Driver 2D face dataset {face 2D }, the three-dimensional gaze direction dataset is The two-dimensional gaze direction dataset is The 3D gaze cone dataset is The two-dimensional gaze cone dataset is Step S2: Based on the driver's two-dimensional face dataset {face 2D }The driver's two-dimensional face image face 2D Get a set of two-dimensional facial feature points, and obtain the driver's left eye area image Image_left and right eye area image Image_right based on the two-dimensional facial feature point set; then calculate the driver's two-dimensional facial image face based on the driver's left eye area image Image_left and right eye area image Image_right 2D The two-dimensional gaze direction in Based on the two-dimensional gaze direction Calculate the 2D gaze cone 2D ; Step S3: based on the driver's two-dimensional face image face 2D Get the two-dimensional nose point set P nose ; Based on the driver's 3D face position image dataset {position 3D }The driver's three-dimensional face position image position 3D , get the set of key feature points of 3D face Then based on the set of key feature points of the 3D face Calculate the driver's 3D gaze direction Based on the driver's three-dimensional gaze direction Calculate the 3D gaze cone 3D ; Step S4: Calculate the two-dimensional gaze direction loss 3D gaze direction loss 2D-3D consistent gaze direction loss Combining 2D gaze direction loss 3D gaze direction loss 2D-3D consistent gaze direction loss The total gaze direction loss is calculated Step S5: Calculate the two-dimensional gaze cone loss 3D gaze cone loss 2D-3D consistent gaze cone loss Combining 2D gaze cone loss , 3D gaze cone loss 2D-3D consistent gaze cone loss Calculate the driver's total gaze cone loss Then, combined with the total gaze cone loss And the total gaze direction loss obtained in step S4 Calculate the total gaze loss Loss Total ; Finally, the total gaze loss is used for training to obtain the optimal parameters of the method; Step S6, input the two-dimensional face image to be detected, and process the two-dimensional face image to be detected by the method under the optimal parameters to obtain the three-dimensional face position image of the driver, the three-dimensional gaze direction of the driver, and the three-dimensional gaze cone of the driver in the two-dimensional face image to be detected.

2. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S2, the driver's left eye area image Image_left is subjected to convolution pooling to obtain the left eye gaze direction Perform convolution pooling on the driver’s right eye area image Image_right to obtain the right eye gaze direction Finally, combined with the left eye gaze direction Right eye gaze direction Calculate the 2D gaze direction 3. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S3, based on the driver's three-dimensional face position image position 3D Get the 3D left eye point set P' left_eye , 3D right eye point set P' right_eye , 3D nose point set P' nose , thus forming a set of key feature points of a three-dimensional face Then the key feature point set of the 3D face Perform full connection layer processing to obtain the driver's three-dimensional gaze direction 4. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S4, based on the two-dimensional gaze direction calculated in step S2 The 2D gaze direction dataset from step 1 The two-dimensional gaze direction in Compute 2D gaze direction loss 5. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S4, based on the three-dimensional gaze direction calculated in step S3 The 3D gaze direction dataset from step 1 The three-dimensional gaze direction in Compute 3D gaze direction loss 6. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S4, the three-dimensional gaze direction From the 3D face coordinate system Projection to 2D face coordinate system This gives the projected gaze direction Then, combined with the projected gaze direction 2D gaze direction Calculate the 2D-3D consistent gaze direction loss 7. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S5, based on the two-dimensional gaze cone Cone obtained in step S2 2D The vertex angle θ 2D , the two-dimensional gaze cone dataset in step S1 The two-dimensional gaze cone in The top angle Calculate the 2D gaze cone vertex angle difference loss Calculate the 2D gaze cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The area of ​​the union of union , and the two-dimensional gaze cone Cone 2D With the 2D gaze cone dataset The two-dimensional gaze cone in The intersection area is Area int , and then according to the area of ​​the union area union 、Intersection area Area int Compute the 2D gaze cone intersection-over-union loss Combined with 2D gaze cone vertex angle difference loss 2D gaze cone intersection-over-union loss Calculate the two-dimensional gaze cone loss 8. The method for detecting the driver's gaze direction based on consistent gaze of two-dimensional and three-dimensional faces according to claim 1, characterized in that: In step S5, based on the three-dimensional gaze cone Cone obtained in step S3 3D The vertex angle of the 3D gaze cone dataset in step S1 The three-dimensional gaze cone in The vertex angle of the three-dimensional gaze cone is calculated Calculate the 3D gaze cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The union area volume Volume Union , and the three-dimensional gaze cone Cone 3D With the 3D gaze cone dataset The three-dimensional gaze cone in The intersection area volume Volume int , and then according to the volume of the union area Volume Union , intersection area volume Volume int Compute the 3D gaze cone intersection-over-union loss Combined with 3D gaze cone vertex angle difference loss 3D gaze cone intersection-over-union loss Calculate the three-dimensional gaze cone loss 9. The method for detecting the driver's gaze direction based on two-dimensional and three-dimensional face consistent gaze according to claim 1, characterized in that: In step S5, the three-dimensional gaze cone Cone 3D From the 3D face coordinate system Projection to 2D face coordinate system Get the projected gaze cone Cone proj , and calculate the projected gaze cone Cone proj The vertex angle θ proj ; According to the vertex angle θ proj and the two-dimensional gaze cone 2D The vertex angle θ 2D , the 2D-3D consistent gaze cone vertex difference loss is calculated Calculate the projected gaze cone proj and the 2D gaze cone 2D The area of ​​the union of Unoion , and the projected gaze cone Cone proj and the 2D gaze cone 2D The intersection area is Area' int , combined with the area of ​​the union area Area' Union , intersection area Area' int Compute 2D-3D consistent gaze cone intersection-over-union loss Then, we combine the 2D-3D consistent gaze cone vertex difference loss 2D-3D consistent gaze cone intersection loss Calculate the 2D-3D consistent gaze cone loss

Images