A method and system for estimating the line of sight of a multi-task driver for assisting safe driving

Abstract

The application provides a multi-task driver line-of-sight estimation method and system for assisting safe driving. A plurality of driver images are obtained from a video sequence, face key feature points and a face region image of each driver image are extracted by a Dlib CNN method, eye region cropping and head posture extraction are performed on the face key feature points of each driver image, left eye region images, right eye region images and head postures of each driver image are obtained, a driver line-of-sight estimation network is constructed, the left eye region images, the right eye region images, the face region images and the head postures of each driver image are taken as inputs of the driver line-of-sight estimation network, estimated line-of-sight directions and estimated gaze region types of each driver image are obtained, real-time driver images are obtained and input into the driver line-of-sight estimation network, and estimated line-of-sight directions and estimated gaze region types of the real-time driver are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent transportation, and in particular to a multi-task driver line-of-sight estimation method and system for assisting safe driving. BACKGROUND

[0002] Driver attention monitoring is a research hotspot in the field of intelligent transportation. Analyzing the gaze behavior of drivers through their line of sight is of great significance for driver visual blind area judgment, driving behavior analysis, and improving safe driving. Existing driver gaze region estimation methods based on vision can be divided into three categories according to the feature information used, including head pose-based methods [1] , face modal-based methods [2] , and methods combining head pose and face modal [3] .

[0003] Head pose feature-based methods: Lee et al. [4] proposed a real-time driver gaze region estimation model based on driver head pose Euler angles (including yaw angle and pitch angle). This model can calculate the driver's gaze region by inputting only simple facial features, and can adapt to different lighting conditions during the day and at night. Lundgren et al. [5] used head pose to represent the driver's gaze direction and proposed using a Bayesian classifier to realize the probability mapping of the driver in each line-of-sight region. Experimental results show that this method has certain advantages in estimating partial gaze regions, but the estimation accuracy is low in some specific gaze regions (such as the instrument panel region). Fridman et al. [6] proposed an algorithm combining face key point information and face Delaunay triangle position information. However, when the driver's head deflection angle is too large, causing the loss of facial information, this type of method relying on face key point information will produce errors. Tawari et al. [7] proposed a distributed camera structure that uses head pose dynamics to estimate the driver's line-of-sight region, and the framework can run stably and continuously even if the head pose changes greatly. Since the relationship between head pose and line-of-sight direction is not one-to-one, using only head pose to represent the driver's line-of-sight direction has a large error in some scenarios.

[0004] Face feature extraction-based methods: Vora et al. [8,9] used CNN structure to train driver facial features. Their experimental results show that using half-face region images can achieve better estimation results. On this basis, Akshay et al.

[10] further proposed a driver glasses removal model based on GAN network, which uses GAN network to handle the occlusion problem of glasses before line-of-sight region estimation. Choi et al.[2] The gaze region is classified using a given face image, and the fatigue, concentration or distraction state of the driver is determined. Ghosh et al.

[11] A CNN model combined with a light robust layer is proposed to solve the problem of face image affected by light. Lollett et al.

[12] A single camera framework gaze region classifier is proposed, which can also run stably under non-uniform lighting, non-frontal face posture, and face being temporarily or permanently occluded. In addition, Lollett et al.

[13] A three-dimensional convolutional neural network model is used to extract the visual features of the driver in multiple adjacent frames, which can express motion features and help to alleviate the defects of each frame recognition system due to the lack of context information.

[0005] Head pose feature combined with face feature method: since the head pose and gaze are not one-to-one corresponding, and the head pose has a greater impact on the driver's gaze region, the method of combining head pose feature and eye feature helps to improve the driver's gaze. For example, Tawari et al.

[14] By using eye image features (pupil position information) and head pose simultaneously for driver gaze region estimation, and comparing the accuracy of using head pose alone, eye features alone, and both combined, the research results show that combining head pose features and gaze estimation tasks helps to improve the accuracy of driver gaze region estimation. Yu et al.

[15] Research shows that the head-eye feature fusion method can improve the accuracy of driver gaze region classification by extracting head pose, eye region, face feature, and key point position information. Wang et al.

[16] A separate CNN network branch is proposed to detect head pose hidden features, and then the hidden features are combined with eye features, and finally the driver gaze region estimation task is realized through the classifier. Compared with other end-to-end methods, their method also achieves good performance. Dari et al.

[17] A driver gaze estimation method is proposed, which represents the gaze direction by combining eye key point position and head pose features. However, the lack of eye key point position information due to the excessive deflection of the driver's head will affect the subsequent processing of the algorithm.

[0006] In addition, experimental results show that the eye gaze is not only related to the appearance features of the eyes, but also has a great relationship with other facial features [18,19] Therefore, some scholars explore the combination of full face features and head pose information to estimate the driver's gaze region. Vicente et al. [3]A vision-based model is proposed to combine head pose features and gaze estimation tasks to detect the situation that the driver's gaze is distributed in the area outside the road. Wang et al.

[20] A driver head pose detection method based on ICP point cloud iterative template matching is proposed. They use depth map to obtain head pose information, and use RGB image to study the driver's gaze information, then take the initial head pose of the driver in each gaze area as the template of point cloud, and further fuse the estimated head pose with the gaze to determine the driver's gaze region. In general, the driver's gaze and gaze region have great relevance with the driver's head pose, eye region features and full face image features. Integrating multiple semantic information to track and monitor the driver's gaze can improve the robustness and accuracy of the model, and has important significance for assisting safe driving.

[0007] References

[0008] [1] Borghi G, Gasparini R, Vezzani R, et al. Embedded recurrent network for head pose estimation in car[Z]. IEEE, 20171503-1508.

[0009] [2] Choi I, Hong S K, Kim Y. Real-time categorization of driver's gaze zone using the deep learning techniques[Z]. 2016143-148.

[0010] [3] Vicente F, Huang Z, Xiong X, et al. Driver Gaze Tracking and Eyes Off the Road Detection System[J]. IEEE Transactions on Intelligent Transportation Systems. 2015, 16(4): 2014-2027.

[0011] [4] Lee S J, Jo J, Jung H G, et al. Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System[J]. IEEE Transactions on Intelligent Transportation Systems. 2011, 12(1): 254-267.

[0012] [5] Lundgren M, Hammarstrand L, Mckelvey T. Driver-Gaze Zone Estimation Using Bayesian Filtering and Gaussian Processes[J]. IEEE Transactions on Intelligent Transportation Systems. 2016, 17(10): 2739-2750.

[0013] [6] Fridman L, Langhans P, Lee J, et al. Driver Gaze Region Estimation without Use of Eye Movement[J]. IEEE Intelligent Systems. 2016, 31(3): 49-56.

[0014] [7] Tawari A, Trivedi M M. Robust and continuous estimation of driver gaze zone by dynamic analysis of multiple face videos[Z]. 2014344-349.

[0015] [8] Vora S, Rangesh A, Trivedi M M. Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis[J]. IEEE Transactions on Intelligent Vehicles. 2018, 3(3): 254-265.

[0016] [9] Vora S, Rangesh A, Trivedi M M. On generalizing driver gaze zone estimation using convolutional neural networks[Z]. IEEE, 2017849-854.

[0017]

[10] Rangesh A, Zhang B, Trivedi M M. Gaze Preserving CycleGANs for Eyeglass Removal and Persistent Gaze Estimation[J]. IEEE Transactions on Intelligent Vehicles. 2022, 7(2): 377-386.

[0018]

[11] Ghosh S, Dhall A, Sharma G, et al. Speak2Label: Using Domain Knowledge for Creating a Large Scale Driver Gaze Zone Estimation Dataset[Z]. Montreal, BC, Canada: 2021.

[0019]

[12] Lollett C, Kamezaki M, Sugano S. Towards a Driver's Gaze Zone Classifier using a Single Camera Robust to Temporal and Permanent Face Occlusions[C]. 2021. IEEE, 2021: 578-585.

[0020]

[13] Lollett C, Kamezaki M, Sugano S. Single Camera Face Position-Invariant Driver's Gaze Zone Classifier Based on Frame-Sequence Recognition Using 3D Convolutional Neural Networks[J]. Sensors. 2022, 22(15): 5857.

[0021]

[14] Tawari A, Chen K H, Trivedi M M. Where is the driver looking:Analysis of head, eye and iris for robust gaze zone estimation[C]. 2014. IEEE, 2014: 988-994.

[0022]

[15] Yu Z, Huang X, Zhang X, et al. A Multi-Modal Approach for DriverGaze Prediction to Remove Identity Bias[Z]. New York, NY, USA: 2020768-776.

[0023]

[16] Wang Z, Zhao J, Lu C, et al. Learning to Detect Head Movement inUnconstrained Remote Gaze Estimation in the Wild[Z]. 20203432-3441.

[0024]

[17] Dari S, Kadrileev N, Hüllermeier E. A Neural Network-BasedDriver Gaze Classification System with Vehicle Signals[Z]. 20201-7.

[0025]

[18] Krafka K, Khosla A, Kellnhofer P, et al. Eye Tracking forEveryone[Z]. 20162176-2184.

[0026]

[19] Xucong Z, Yusuke S, Mario F, et al. It's Written All Over YourFace: Full-Face Appearance-Based Gaze Estimation[Z]. 2017.

[0027]

[20] Wang Y, Yuan G, Mi Z, et al. Continuous Driver's Gaze ZoneEstimation Using RGB-D Camera[J]. Sensors. 2019, 19(6). SUMMARY

[0028] In order to solve the above technical problems, the application provides a multi-task driver gaze estimation method for assisting safe driving and a system thereof, which uses a network based on an attention mechanism to extract driver full-face image features and left and right eye image features, can help to extract global and local finer-grained image feature information, and realizes driver gaze and visual field estimation two tasks.

[0029] The technical scheme of the method of the application is a multi-task driver gaze estimation method for assisting safe driving, which comprises the following steps:

[0030] Step 1: obtaining multiple frames of driver images from a video sequence, marking the gaze direction and gaze area type of each frame of driver image, extracting multiple facial key feature points of each frame of driver image and a facial region image of each frame of driver image by using a Dlib CNN method, performing eye region cropping and driver head posture extraction in combination with the multiple facial key feature points of each frame of driver image, obtaining a left eye region image of each frame of driver image, a right eye region image of each frame of driver image and a head posture of each frame of driver image;

[0031] Step 2: constructing a driving gaze estimation network, taking the left eye region image of each frame of driver image, the right eye region image of each frame of driver image, the facial region image of each frame of driver image and the head posture of each frame of driver image as input to the driver gaze estimation network for gaze estimation, obtaining the estimated gaze direction and estimated gaze area type of each frame of driver image, constructing a driver gaze estimation network loss function in combination with the gaze direction and gaze area type of each frame of driver image, and obtaining an optimized driving gaze estimation network through SGD algorithm optimization training;

[0032] Step 3: Obtain a real-time driver image, input into an optimized driver line-of-sight estimation network for line-of-sight estimation to obtain an estimated line-of-sight direction of the real-time driver image and an estimated gaze region type.

[0033] As preferred, the eye region cropping in step 1 is performed in combination with the multiple facial key feature points of each frame of driver image to obtain a left eye region image of each frame of driver image and a right eye region image of each frame of driver image, specifically as follows:

[0034] The left eye center point position and the distance between the left eye corners are calculated according to the left eye left corner key point and the left eye right key point in the multiple facial key feature points of each frame of driver image in step 1, the length of the left eye cropping region is a certain multiple of the distance between the left eye corners, and the left eye cropping rectangular frame is determined according to the center and the length, and then the left eye region image of each frame of driver image is obtained.

[0035] The right eye center point position and the distance between the right eye corners are calculated according to the right eye left corner key point and the right eye right key point in the multiple facial key feature points of each frame of driver image in step 1, the length of the right eye cropping region is a certain multiple of the distance between the right eye corners, and the right eye cropping rectangular frame is determined according to the center and the length, and then the right eye region image of each frame of driver image is obtained.

[0036] As preferred, the driver head pose extraction in step 1 is performed in combination with the multiple facial key feature points of each frame of driver image, and the specific extraction process is as follows:

[0037] The multiple facial key feature point pixel coordinates of each frame of driver image are extracted by a facial key point detection algorithm.

[0038] According to the multiple facial key feature point pixel coordinates of each frame of driver image and the corresponding key point coordinate values in the standard 3D face model, a classic Perspective-n-Point algorithm is used to calculate the rotation matrix and translation matrix between the 3D face coordinate system and the camera coordinate system.

[0039] The rotation matrix is used as a head pose vector, and the head pose vector includes a pitch angle of the head pose, a yaw angle of the head pose, and a roll angle of the head pose.

[0040] The head pose vector is constructed by the pitch angle of the head pose, the yaw angle of the head pose, and the roll angle of the head pose.

[0041] As preferred, the driver line-of-sight estimation network in step 2 includes:

[0042] left eye EE-Net network model, right eye EE-Net network model, global face feature convolutional neural network, first visual line regression fully connected layer, second visual line regression fully connected layer, gaze region 1 1 convolutional layer, gaze region average pooling layer, gaze region linear regression layer, gaze region LogSoftmax activation layer

[0043] The left eye EE-Net network model is used for inputting the left eye region image of each frame of driver image, and extracting left eye features of each frame of driver image.

[0044] The left eye EE-Net network model is composed of a first left eye convolutional layer, a first left eye activation layer, a first left eye expansion convolutional layer, a first left eye expansion activation layer, a first left eye maximum pooling layer, a second left eye convolutional layer, a second left eye activation layer, a second left eye expansion convolutional layer, a second left eye expansion activation layer, a second left eye maximum pooling layer, a third left eye convolutional layer, a third left eye activation layer, a third left eye expansion convolutional layer, a third left eye expansion activation layer, a fourth left eye convolutional layer, a fourth left eye activation layer, a fourth left eye expansion convolutional layer, and a fourth left eye expansion activation layer.

[0045] The left eye feature extraction is specifically as follows:

[0046] The left eye region image of each frame of driver image is sequentially subjected to convolution feature extraction by the first left eye convolutional layer, feature activation by the first left eye activation layer, model channel expansion by the first left eye expansion convolutional layer, feature activation by the first left eye expansion activation layer, first left eye maximum pooling layer processing, convolution feature extraction by the second left eye convolutional layer, feature activation by the second left eye activation layer, model channel expansion by the second left eye expansion convolutional layer, feature activation by the second left eye expansion activation layer, second left eye maximum pooling layer processing, convolution feature extraction by the third left eye convolutional layer, feature activation by the third left eye activation layer, model channel expansion by the third left eye expansion convolutional layer, feature activation by the third left eye expansion activation layer, convolution feature extraction by the fourth left eye convolutional layer, feature activation by the fourth left eye activation layer, model channel expansion by the fourth left eye expansion convolutional layer, and feature activation by the fourth left eye expansion activation layer, to obtain the left eye features of each frame of driver image.

[0047] The right eye EE-Net network model is used for inputting the right eye region image of each frame of driver image, and extracting right eye features of each frame of driver image.

[0048] The right eye EE-Net network model is sequentially connected by a first right eye convolutional layer, a first right eye activation layer, a first right eye extended convolutional layer, a first right eye extended activation layer, a first right eye maximum pooling layer, a second right eye convolutional layer, a second right eye activation layer, a second right eye extended convolutional layer, a second right eye extended activation layer, a second right eye maximum pooling layer, a third right eye convolutional layer, a third right eye activation layer, a third right eye extended convolutional layer, a third right eye extended activation layer, a fourth right eye convolutional layer, a fourth right eye activation layer, a fourth right eye extended convolutional layer, and a fourth right eye extended activation layer.

[0049] The right eye feature extraction is performed to obtain the right eye feature of each frame of driver image, and the specific process is as follows:

[0050] The right eye region image of each frame of driver image sequentially passes through the first right eye convolutional layer for convolution feature extraction, the first right eye activation layer for feature activation, the first right eye extended convolutional layer for model channel expansion, the first right eye extended activation layer for feature activation, the first right eye convolution maximum pooling layer for processing, the second right eye convolutional layer for convolution feature extraction, the second right eye activation layer for feature activation, the second right eye extended convolutional layer for model channel expansion, the second right eye extended activation layer for feature activation, the second right eye convolution maximum pooling layer for processing, the third right eye convolutional layer for convolution feature extraction, the third right eye activation layer for feature activation, the third right eye extended convolutional layer for model channel expansion, the third right eye extended activation layer for feature activation, the fourth right eye convolutional layer for convolution feature extraction, the fourth right eye activation layer, the fourth right eye extended convolutional layer for model channel expansion, the fourth right eye extended activation layer for feature activation, and the feature activation, to obtain the right eye feature of each frame of driver image.

[0051] The global face feature convolutional neural network is used for inputting the face region image of each driver image, and performing face feature extraction to obtain the face feature of each driver image.

[0052] The global face feature convolutional neural network comprises:

[0053] The first-stage global face convolution module, the second-stage global face convolution module, the third-stage global face convolution module, the fourth-stage global face convolution module, the fifth-stage global face convolution module, the sixth-stage global face convolution module, the seventh-stage global face convolution module, the eighth-stage global face convolution module, and the ninth-stage global face convolution module.

[0054] The first-stage global face convolution module comprises a first global convolutional layer.

[0055] The second-stage global face convolution module comprises a second-stage first global mobile flip bottleneck convolutional layer and a second-stage first global channel attention module.

[0056] The third stage global face convolution module comprises: a third stage first global mobile flip bottleneck convolution layer, a third stage first global channel attention module, a third stage second global mobile flip bottleneck convolution, a third stage second global channel attention module;

[0057] The fourth stage global face convolution module comprises: a fourth stage first global mobile flip bottleneck convolution layer, a fourth stage first global channel attention module, a fourth stage first global spatial attention module, a fourth stage second global mobile flip bottleneck convolution layer, a fourth stage second global channel attention module, a fourth stage second global spatial attention module;

[0058] The fifth stage global face convolution module comprises: a fifth stage first global mobile flip bottleneck convolution layer, a fifth stage first global channel attention module, a fifth stage first global spatial attention module, a fifth stage second global mobile flip bottleneck convolution layer, a fifth stage second global channel attention module, a fifth stage second global spatial attention module, a fifth stage third global mobile flip bottleneck convolution layer, a fifth stage third global channel attention module, a fifth stage third global spatial attention module;

[0059] The sixth stage global face convolution module comprises: a sixth stage first global mobile flip bottleneck convolution layer, a sixth stage first global channel attention module, a sixth stage first global spatial attention module, a sixth stage second global mobile flip bottleneck convolution layer, a sixth stage second global channel attention module, a sixth stage second global spatial attention module, a sixth stage third global mobile flip bottleneck convolution layer, a sixth stage third global channel attention module, a sixth stage third global spatial attention module;

[0060] The seventh stage global face convolution module comprises: a seventh stage first global mobile flip bottleneck convolution layer, a seventh stage first global channel attention module, a seventh stage first global spatial attention module, a seventh stage second global mobile flip bottleneck convolution layer, a seventh stage second global channel attention module, a seventh stage second global spatial attention module, a seventh stage third global mobile flip bottleneck convolution layer, a seventh stage third global channel attention module, a seventh stage third global spatial attention module, a seventh stage fourth global mobile flip bottleneck convolution layer, a seventh stage fourth global channel attention module, a seventh stage fourth global spatial attention module;

[0061] The eighth stage global face convolution module comprises: an eighth stage first global mobile flip bottleneck convolution layer, an eighth stage first global channel attention module, an eighth stage first global spatial attention module;

[0062] The ninth stage global face convolution module comprises a ninth global convolution layer and a ninth global activation layer.

[0063] The second stage first global mobile flip bottleneck convolution layer, the third stage first global mobile flip bottleneck convolution layer, the third stage second global mobile flip bottleneck convolution, the fourth stage first global mobile flip bottleneck convolution layer, the fourth stage second global mobile flip bottleneck convolution layer, the fifth stage first global mobile flip bottleneck convolution layer, the fifth stage second global mobile flip bottleneck convolution layer, the fifth stage third global mobile flip bottleneck convolution layer, the sixth stage first global mobile flip bottleneck convolution layer, the sixth stage second global mobile flip bottleneck convolution layer, the sixth stage third global mobile flip bottleneck convolution layer, the seventh stage first global mobile flip bottleneck convolution layer, the seventh stage first global channel attention module, the seventh stage second global mobile flip bottleneck convolution layer, the seventh stage third global mobile flip bottleneck convolution layer, the seventh stage fourth global mobile flip bottleneck convolution layer, and the eighth stage first global mobile flip bottleneck convolution layer all comprise:

[0064] 1 1dimensional reduction convolution layer, a depth separable convolution layer, 1 1dimensional reduction convolution layer, a Dropout layer;

[0065] The second stage first global channel attention module, the third stage first global channel attention module, the third stage second global channel attention module, the fourth stage first global channel attention module, the fourth stage second global channel attention module, the fifth stage first global channel attention module, the fifth stage second global channel attention module, the fifth stage third global channel attention module, the sixth stage first global channel attention module, the sixth stage second global channel attention module, the sixth stage third global channel attention module, the seventh stage first global channel attention module, the seventh stage second global channel attention module, the seventh stage third global channel attention module, the seventh stage fourth global channel attention module, and the eighth stage first global channel attention module all comprise:

[0066] a max pooling layer, an average pooling layer, 1 1convolution layer, a SiLU activation layer, 1 1convolution layer, a weighting layer, a Sigmoid activation layer;

[0067] The fourth stage first global spatial attention module, the fourth stage second global spatial attention module, the fifth stage first global spatial attention module, the fifth stage second global spatial attention module, the fifth stage third global spatial attention module, the sixth stage first global spatial attention module, the sixth stage second global spatial attention module, the sixth stage third global spatial attention module, the seventh stage first global spatial attention module, the seventh stage second global spatial attention module, the seventh stage third global spatial attention module, the seventh stage fourth global spatial attention module, and the eighth stage first global spatial attention module all include:

[0068] The max-pooling layer, the average-pooling layer, the 7 7 convolutional layers, Sigmoid activation layers;

[0069] The network layers of the global facial feature convolutional neural network are cascaded in the manner of the internal network layers of the EfficientNet network model;

[0070] The first gaze regression fully connected layer is configured to input the left eye feature of each frame of driver image, the right eye feature of each frame of driver image, the facial feature of each frame of driver image, and the head pose vector, and perform regression processing to obtain the first fusion feature of each frame of driver image;

[0071] The second gaze regression fully connected layer is configured to input the first fusion feature of each frame of driver image, and perform regression processing to obtain the estimated gaze direction of each frame of driver image;

[0072] The facial feature of each frame of driver image is sequentially input into the gaze region 1 1 convolutional layer for convolution feature extraction, the gaze region average-pooling layer for pooling processing, the gaze region linear regression layer for linear regression, and the gaze region LogSoftmax activation layer for feature activation to obtain the probability of each gaze region type of each frame of driver image, and select the gaze region type with the highest probability among all gaze region types of each frame of driver image as the estimated gaze region type of each frame of driver image;

[0073] Preferably, the driver gaze estimation network loss function of step 2 includes a driver gaze direction estimation loss function and a driver gaze region estimation loss function.

[0074] The driver gaze direction estimation loss function is configured to constrain the estimated gaze direction of each frame of driver image, and is defined as follows:

[0075] The mean square error (MSE) is used as the driver gaze direction estimation loss function , which can be defined as:

[0076]

[0077] wherein, N denotes the number of driver images, i denotes the index of each frame of driver image, is the estimated gaze direction of the i frame of driver image, is the gaze direction of the i frame of driver image;

[0078] The driver gaze area estimation loss function is used to constrain the estimated gaze area of each frame of driver image, and the driver gaze area estimation loss function includes a cross-entropy loss function , a center loss function .

[0079] The cross-entropy loss function can be defined as:

[0080]

[0081] wherein, i denotes the index of each frame of driver image, is the gaze area type of the i frame of driver image, is the estimated gaze area type of the i frame of driver image. The cross-entropy loss function is used to measure the difference between the predicted probability value of each gaze area type of each frame of driver image and the true value of the gaze area of each frame of driver image. It can effectively increase the inter-class dispersion, but it has limitations in intra-class dispersion, resulting in relatively large inter-class distance;

[0082] The center loss function can reduce the distance within the class, and can be defined as:

[0083]

[0084] wherein, N denotes the number of driver images, is the feature vector of the i frame of driver image, is the center of the gaze area of the i frame of driver image;

[0085] The driver gaze area estimation loss function can be defined as:

[0086]

[0087] wherein, represents the weight of the center loss function, used to adjust the intra-class compactness. The larger, the lower the intra-class compactness, and conversely, The smaller, the higher the intra-class compactness.

[0088] The application also provides a multi-task driver line-of-sight estimation system for assisting safe driving, comprising:

[0089] The driver image and posture extraction module is used for obtaining multiple frames of driver images from a video sequence, marking the line-of-sight direction and gaze region type of each frame of driver image, extracting multiple facial key feature points of each frame of driver image and a facial region image of each frame of driver image by means of the Dlib CNN method, performing eye region cropping and driver head posture extraction in combination with the multiple facial key feature points of each frame of driver image, and obtaining a left eye region image of each frame of driver image, a right eye region image of each frame of driver image and a head posture of each frame of driver image.

[0090] The driving line-of-sight estimation network training module is used for constructing a driving line-of-sight estimation network, inputting the left eye region image of each frame of driver image, the right eye region image of each frame of driver image, the facial region image of each frame of driver image and the head posture of each frame of driver image into the driver line-of-sight estimation network for line-of-sight estimation, obtaining the estimated line-of-sight direction and estimated gaze region type of each frame of driver image, constructing a driver line-of-sight estimation network loss function in combination with the line-of-sight direction and gaze region type of each frame of driver image, and obtaining an optimized driving line-of-sight estimation network through SGD algorithm optimization training.

[0091] The real-time driver image estimation module is used for obtaining real-time driver images, inputting the real-time driver images into the optimized driving line-of-sight estimation network for line-of-sight estimation, and obtaining the estimated line-of-sight direction and estimated gaze region type of the real-time driver images.

[0092] The application has the following beneficial effects:

[0093] The driver line-of-sight and viewing area estimation model uses the same network structure to simultaneously estimate the line-of-sight direction of the driver during driving and the gaze region of the driver in the cockpit, helps to monitor the line-of-sight direction and gaze region information of the driver in real time, judges the driving line-of-sight blind area, and assists the driver in safe driving.

[0094] Using a common camera as an image acquisition device, compared with other head-mounted line-of-sight tracking devices, the model does not interfere with the driver during driving, has the advantages of more flexible measurement method, simple device operation, low price, and high method universality, etc.

[0095] Compared with other driver vision estimation methods, the model described in this invention uses a lightweight backbone network for feature extraction and a pre-training-fine-tuning strategy to train the network model, thereby reducing network computation and model parameters while improving the accuracy of vision estimation.

[0096] The global facial feature extraction network—gCNN—embeds channel and spatial attention modules in different convolutional stages of the network model, enabling the network to capture overall general features in the early stages and learn local discriminative features in the later stages. It has been verified that the embedding of attention modules improves the estimation of the gaze region by 15%. Attached Figure Description

[0097] Figure 1 : Flowchart of the method according to an embodiment of the present invention;

[0098] Figure 2 Examples of image enhancement, (a) example of eye occlusion, (b) example of free head pose, (c) example of uneven illumination in both eyes, (d) example of blurred eye image, (e) example of closed eye state;

[0099] Figure 3 Normalized confusion matrix of driver gaze region estimation validation results;

[0100] Figure 4 Case studies of driver gaze estimation: (a) gaze estimation in low light conditions, (b) driver gaze estimation while wearing glasses, (c) driver gaze estimation in adjacent areas, and (d) driver gaze estimation with different eye movements under similar head postures. Detailed Implementation

[0101] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0102] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0103] The following is in conjunction with the appendix Figures 1-4 The present invention specifically describes a multi-task driver's line-of-sight estimation method and system for assisting safe driving, as detailed below:

[0104] As Figure 1 shown, it is a kind of safety driving auxiliary multi-task driver line-of-sight estimation method for technical scheme of method embodiment of the application, comprising the following steps:

[0105] Step 1: obtain a plurality of frames of driver images from a video sequence, mark the line-of-sight direction and gaze region type of each frame of driver image, extract 68 facial key feature points of each frame of driver image and facial region image of each frame of driver image by Dlib CNN method, and perform eye region cropping and driver head posture extraction based on the plurality of facial key feature points of each frame of driver image to obtain left eye region image of each frame of driver image, right eye region image of each frame of driver image and head posture of each frame of driver image; the plurality of frames of driver images, the line-of-sight direction and the gaze region type of each frame of driver image can be directly obtained through a public data set. Common data sets are as follows:

[0106] The MPIIFaceGaze data set is a widely used three-dimensional line-of-sight estimation data set, which contains 213659 facial images of 15 subjects collected under different illuminations, head postures and backgrounds. These images are collected using the built-in camera of a notebook computer. The data set also contains 2D line-of-sight landing point and 3D line-of-sight angle information, 6 facial key point coordinates, head posture vector, facial center point coordinate information and other label data. Therefore, the present application first selects the MPIIFaceGaze data set as the training and testing data set of the driver line-of-sight direction estimation network model, and pre-trains the driver gaze region estimation network using the data set and pre-processes the data set;

[0107] The DGW data set contains 586 records captured using an RGB camera at different times of the day (including night). The data set contains image information of 338 different drivers collected from a car. The drivers are aged from 18 to 63. Nine different gaze regions in the car are labeled with corresponding digital labels, including rearview mirror, left and right mirrors, radio, instrument panel and four regions of windshield. The subjects complete the image collection task of different regions by gazing at the corresponding labels. At the same time, the subjects are also required to adjust the seat in a proper posture and control the eye or head movement to gaze at the region in a habitual posture, so that it is not restricted;

[0108] The images in the data set are subjected to occlusion, rotation, illumination adjustment, blurring and other processing to realize the expansion task of the training data set. An example of data set image enhancement is shown in Figure 2 ;

[0109] Step 1 combines the multiple facial key feature points of each frame of the driver image to perform eye region cropping, obtaining the left eye region image of each frame of the driver image, the right eye region image of each frame of the driver image, and the specific process is as follows:

[0110] According to the left eye left corner key point and the left eye right key point of the 68 facial key feature points of each frame of the driver image in step 1, the center point position of the left eye and the distance between the left eye corners are calculated, the side length of the left eye cropping region is 1.7 times the distance between the left eye corners, and the left eye cropping rectangular frame is determined according to the center and the side length, and then the left eye region image of each frame of the driver image is obtained.

[0111] According to the right eye left corner key point and the right eye right key point of the 68 facial key feature points of each frame of the driver image in step 1, the center point position of the right eye and the distance between the right eye corners are calculated, the side length of the right eye cropping region is 1.7 times the distance between the right eye corners, and the right eye cropping rectangular frame is determined according to the center and the side length, and then the right eye region image of each frame of the driver image is obtained.

[0112] Step 1 combines the multiple facial key feature points of each frame of the driver image to perform driver head posture extraction, and the specific extraction process is as follows:

[0113] The 68 facial key feature point pixel coordinates of each frame of the driver image are extracted by the facial key point detection algorithm;

[0114] According to the 14 facial key feature point pixel coordinates of each frame of the driver image and the corresponding key point coordinate values in the standard 3D face model, the rotation matrix and the translation matrix between the 3D face coordinate system and the camera coordinate system are calculated using the classic Perspective-n-Point algorithm;

[0115] The rotation matrix is used as the head posture vector, and the head posture vector includes the pitch angle of the head posture, the yaw angle of the head posture, and the roll angle of the head posture;

[0116] The head posture vector is constructed by the pitch angle of the head posture, the yaw angle of the head posture, and the roll angle of the head posture;

[0117] Step 2: Construct a driving line of sight estimation network, and input the left eye region image of each frame of the driver image, the right eye region image of each frame of the driver image, the face region image of each frame of the driver image, and the head posture of each frame of the driver image into the driver line of sight estimation network for line of sight estimation, to obtain the estimated line of sight direction and the estimated gaze region type of each frame of the driver image, combine the line of sight direction and the gaze region type of each frame of the driver image to construct a driver line of sight estimation network loss function, and optimize and train the driver line of sight estimation network by the SGD algorithm to obtain an optimized driver line of sight estimation network;

[0118] The driving line-of-sight estimation network of step 2 comprises:

[0119] The left eye EE-Net network model, the right eye EE-Net network model, the global face feature convolutional neural network, the first line-of-sight regression fully connected layer, the second line-of-sight regression fully connected layer, the gaze region 1 1 convolutional layer, a gaze region average pooling layer, a gaze region linear regression layer, and a gaze region LogSoftmax activation layer.

[0120] The left eye EE-Net network model is configured to input a left eye region image of each frame of driver image, and extract 64-dimensional left eye features of each frame of driver image.

[0121] The left eye EE-Net network model is composed of a first left eye convolutional layer, a first left eye activation layer, a first left eye expansion convolutional layer, a first left eye expansion activation layer, a first left eye max pooling layer, a second left eye convolutional layer, a second left eye activation layer, a second left eye expansion convolutional layer, a second left eye expansion activation layer, a second left eye max pooling layer, a third left eye convolutional layer, a third left eye activation layer, a third left eye expansion convolutional layer, a third left eye expansion activation layer, a fourth left eye convolutional layer, a fourth left eye activation layer, a fourth left eye expansion convolutional layer, and a fourth left eye expansion activation layer.

[0122] The left eye feature extraction is performed to obtain the 64-dimensional left eye features of each frame of driver image, and the specific process is as follows:

[0123] The left eye region image of each frame of driver image is sequentially subjected to convolution feature extraction by the first left eye convolutional layer, feature activation by the first left eye activation layer, model channel expansion by the first left eye expansion convolutional layer, feature activation by the first left eye expansion activation layer, processing by the first left eye max pooling layer, convolution feature extraction by the second left eye convolutional layer, feature activation by the second left eye activation layer, model channel expansion by the second left eye expansion convolutional layer, feature activation by the second left eye expansion activation layer, processing by the second left eye max pooling layer, convolution feature extraction by the third left eye convolutional layer, feature activation by the third left eye activation layer, model channel expansion by the third left eye expansion convolutional layer, feature activation by the third left eye expansion activation layer, convolution feature extraction by the fourth left eye convolutional layer, feature activation by the fourth left eye activation layer, model channel expansion by the fourth left eye expansion convolutional layer, and feature activation by the fourth left eye expansion activation layer, to obtain the 64-dimensional left eye features of each frame of driver image.

[0124] The right eye EE-Net network model is configured to input a right eye region image of each frame of driver image, and extract 64-dimensional right eye features of each frame of driver image.

[0125] The right eye EE-Net network model is sequentially connected by a first right eye convolutional layer, a first right eye activation layer, a first right eye extended convolutional layer, a first right eye extended activation layer, a first right eye maximum pooling layer, a second right eye convolutional layer, a second right eye activation layer, a second right eye extended convolutional layer, a second right eye extended activation layer, a second right eye maximum pooling layer, a third right eye convolutional layer, a third right eye activation layer, a third right eye extended convolutional layer, a third right eye extended activation layer, a fourth right eye convolutional layer, a fourth right eye activation layer, a fourth right eye extended convolutional layer, and a fourth right eye extended activation layer.

[0126] The right eye feature extraction is performed to obtain the right eye feature of each frame of driver image, and the specific process is as follows:

[0127] The right eye region image of each frame of driver image sequentially passes through the first right eye convolutional layer for convolution feature extraction, the first right eye activation layer for feature activation, the first right eye extended convolutional layer for model channel expansion, the first right eye extended activation layer for feature activation, the first right eye convolution maximum pooling layer for processing, the second right eye convolutional layer for convolution feature extraction, the second right eye activation layer for feature activation, the second right eye extended convolutional layer for model channel expansion, the second right eye extended activation layer for feature activation, the second right eye convolution maximum pooling layer for processing, the third right eye convolutional layer for convolution feature extraction, the third right eye activation layer for feature activation, the third right eye extended convolutional layer for model channel expansion, the third right eye extended activation layer for feature activation, the fourth right eye convolutional layer for convolution feature extraction, the fourth right eye activation layer, the fourth right eye extended convolutional layer for model channel expansion, the fourth right eye extended activation layer for feature activation, and the feature activation, to obtain the 64-dimensional right eye feature of each frame of driver image.

[0128] The global face feature convolutional neural network is used for inputting the face region image of each driver image, and performing face feature extraction to obtain the 128-dimensional face feature of each driver image.

[0129] The global face feature convolutional neural network comprises:

[0130] The first-stage global face convolution module, the second-stage global face convolution module, the third-stage global face convolution module, the fourth-stage global face convolution module, the fifth-stage global face convolution module, the sixth-stage global face convolution module, the seventh-stage global face convolution module, the eighth-stage global face convolution module, and the ninth-stage global face convolution module.

[0131] The first-stage global face convolution module comprises a first global convolutional layer.

[0132] The second-stage global face convolution module comprises a second-stage first global mobile flip bottleneck convolutional layer and a second-stage first global channel attention module.

[0133] The third stage global face convolution module comprises: a third stage first global mobile flip bottleneck convolution layer, a third stage first global channel attention module, a third stage second global mobile flip bottleneck convolution, a third stage second global channel attention module;

[0134] The fourth stage global face convolution module comprises: a fourth stage first global mobile flip bottleneck convolution layer, a fourth stage first global channel attention module, a fourth stage first global spatial attention module, a fourth stage second global mobile flip bottleneck convolution layer, a fourth stage second global channel attention module, a fourth stage second global spatial attention module;

[0135] The fifth stage global face convolution module comprises: a fifth stage first global mobile flip bottleneck convolution layer, a fifth stage first global channel attention module, a fifth stage first global spatial attention module, a fifth stage second global mobile flip bottleneck convolution layer, a fifth stage second global channel attention module, a fifth stage second global spatial attention module, a fifth stage third global mobile flip bottleneck convolution layer, a fifth stage third global channel attention module, a fifth stage third global spatial attention module;

[0136] The sixth stage global face convolution module comprises: a sixth stage first global mobile flip bottleneck convolution layer, a sixth stage first global channel attention module, a sixth stage first global spatial attention module, a sixth stage second global mobile flip bottleneck convolution layer, a sixth stage second global channel attention module, a sixth stage second global spatial attention module, a sixth stage third global mobile flip bottleneck convolution layer, a sixth stage third global channel attention module, a sixth stage third global spatial attention module;

[0137] The seventh stage global face convolution module comprises: a seventh stage first global mobile flip bottleneck convolution layer, a seventh stage first global channel attention module, a seventh stage first global spatial attention module, a seventh stage second global mobile flip bottleneck convolution layer, a seventh stage second global channel attention module, a seventh stage second global spatial attention module, a seventh stage third global mobile flip bottleneck convolution layer, a seventh stage third global channel attention module, a seventh stage third global spatial attention module, a seventh stage fourth global mobile flip bottleneck convolution layer, a seventh stage fourth global channel attention module, a seventh stage fourth global spatial attention module;

[0138] The eighth stage global face convolution module comprises: an eighth stage first global mobile flip bottleneck convolution layer, an eighth stage first global channel attention module, an eighth stage first global spatial attention module;

[0139] The ninth stage global face convolution module comprises a ninth global convolution layer and a ninth global activation layer.

[0140] The second stage first global mobile flip bottleneck convolution layer, the third stage first global mobile flip bottleneck convolution layer, the third stage second global mobile flip bottleneck convolution, the fourth stage first global mobile flip bottleneck convolution layer, the fourth stage second global mobile flip bottleneck convolution layer, the fifth stage first global mobile flip bottleneck convolution layer, the fifth stage second global mobile flip bottleneck convolution layer, the fifth stage third global mobile flip bottleneck convolution layer, the sixth stage first global mobile flip bottleneck convolution layer, the sixth stage second global mobile flip bottleneck convolution layer, the sixth stage third global mobile flip bottleneck convolution layer, the seventh stage first global mobile flip bottleneck convolution layer, the seventh stage first global channel attention module, the seventh stage second global mobile flip bottleneck convolution layer, the seventh stage third global mobile flip bottleneck convolution layer, the seventh stage fourth global mobile flip bottleneck convolution layer, and the eighth stage first global mobile flip bottleneck convolution layer all comprise:

[0141] 1 1dimensional reduction convolution layer, a depth separable convolution layer, 1 1dimensional reduction convolution layer, a Dropout layer;

[0142] The second stage first global channel attention module, the third stage first global channel attention module, the third stage second global channel attention module, the fourth stage first global channel attention module, the fourth stage second global channel attention module, the fifth stage first global channel attention module, the fifth stage second global channel attention module, the fifth stage third global channel attention module, the sixth stage first global channel attention module, the sixth stage second global channel attention module, the sixth stage third global channel attention module, the seventh stage first global channel attention module, the seventh stage second global channel attention module, the seventh stage third global channel attention module, the seventh stage fourth global channel attention module, and the eighth stage first global channel attention module all comprise:

[0143] a max pooling layer, an average pooling layer, 1 1convolution layer, a SiLU activation layer, 1 1convolution layer, a weighting layer, a Sigmoid activation layer;

[0144] The fourth stage first global spatial attention module, the fourth stage second global spatial attention module, the fifth stage first global spatial attention module, the fifth stage second global spatial attention module, the fifth stage third global spatial attention module, the sixth stage first global spatial attention module, the sixth stage second global spatial attention module, the sixth stage third global spatial attention module, the seventh stage first global spatial attention module, the seventh stage second global spatial attention module, the seventh stage third global spatial attention module, the seventh stage fourth global spatial attention module, and the eighth stage first global spatial attention module all include:

[0145] The max-pooling layer, the average-pooling layer, the 7 7 convolutional layers, Sigmoid activation layers;

[0146] The network layers of the global facial feature convolutional neural network are cascaded in the manner of the internal network layers of the EfficientNet network model;

[0147] The first gaze regression fully connected layer is configured to input the 64-dimensional left eye feature of each frame of driver image, the 64-dimensional right eye feature of each frame of driver image, the 128-dimensional facial feature of each frame of driver image, and the 2-dimensional head posture vector, and perform regression processing to obtain the 258-dimensional first fusion feature of each frame of driver image;

[0148] The second gaze regression fully connected layer is configured to input the first fusion feature of each frame of driver image, and perform regression processing to obtain the 2-dimensional estimated gaze direction of each frame of driver image;

[0149] The facial feature of each frame of driver image is sequentially subjected to convolution feature extraction by the 1 1 convolutional layer, pooling processing by the gaze region average-pooling layer, linear regression by the gaze region linear regression layer, and feature activation by the gaze region LogSoftmax activation layer to obtain the probability of each gaze region type of each frame of driver image, and the gaze region type with the highest probability is selected as the estimated gaze region type of each frame of driver image from all gaze region types of each frame of driver image;

[0150] The driver gaze estimation network loss function of step 2 includes a driver gaze direction estimation loss function and a driver gaze region estimation loss function.

[0151] The driver gaze direction estimation loss function is configured to constrain the estimated gaze direction of each frame of driver image, and is defined as follows:

[0152] The mean square error (MSE) is used as the driver gaze direction estimation loss function , which can be defined as:

[0153]

[0154] wherein, N denotes the number of driver images, i denotes the index of each frame of driver image, is the estimated gaze direction of the i frame of driver image, is the gaze direction of the i frame of driver image;

[0155] The driver gaze area estimation loss function is used to constrain the estimated gaze area of each frame of driver image, and the driver gaze area estimation loss function includes a cross-entropy loss function , a center loss function .

[0156] The cross-entropy loss function can be defined as:

[0157]

[0158] wherein, i denotes the index of each frame of driver image, is the gaze area type of the i frame of driver image, is the estimated gaze area type of the i frame of driver image. The cross-entropy loss function is used to measure the difference between the predicted probability value of each gaze area type of each frame of driver image and the true value of the gaze area of each frame of driver image. It can effectively increase the inter-class dispersion, but it has limitations in intra-class dispersion, resulting in relatively large inter-class distance;

[0159] The center loss function can reduce the distance within the class, and can be defined as:

[0160]

[0161] wherein, N denotes the number of driver images, is the feature vector of the i frame of driver image, is the center of the gaze area of the i frame of driver image;

[0162] The driver gaze area estimation loss function can be defined as:

[0163]

[0164] wherein, represents the weight of the center loss function, used to adjust the intra-class compactness. The larger, the lower the intra-class compactness, and conversely, The smaller, the higher the intra-class compactness.

[0165] Step 3: Obtain a real-time driver image, input into the optimized driver line-of-sight estimation network for line-of-sight estimation, and obtain the estimated line-of-sight direction and estimated gaze region type of the real-time driver image.

[0166] The system embodiment of the application is a multi-task driver line-of-sight estimation system for assisting safe driving, comprising:

[0167] The driver image and posture extraction module is used to obtain multiple frames of driver images from a video sequence, mark the line-of-sight direction and gaze region type of each frame of driver image, extract the multiple facial key feature points of each frame of driver image and the facial region image of each frame of driver image by the Dlib CNN method, perform eye region cropping and driver head posture extraction in combination with the multiple facial key feature points of each frame of driver image, and obtain the left eye region image of each frame of driver image, the right eye region image of each frame of driver image, and the head posture of each frame of driver image.

[0168] The driver line-of-sight estimation network training module is used to construct a driver line-of-sight estimation network, input the left eye region image of each frame of driver image, the right eye region image of each frame of driver image, the facial region image of each frame of driver image, and the head posture of each frame of driver image into the driver line-of-sight estimation network for line-of-sight estimation, obtain the estimated line-of-sight direction and estimated gaze region type of each frame of driver image, construct a driver line-of-sight estimation network loss function in combination with the line-of-sight direction and gaze region type of each frame of driver image, and obtain an optimized driver line-of-sight estimation network through SGD algorithm optimization training.

[0169] The real-time driver image estimation module is used to obtain a real-time driver image, input into the optimized driver line-of-sight estimation network for line-of-sight estimation, and obtain the estimated line-of-sight direction and estimated gaze region type of the real-time driver image.

[0170] In order to verify the precision and efficiency of the method of the application, the driver line-of-sight direction estimation model is trained on the MPIIFaceGaze dataset, and the model is fine-tuned on the LISA Gaze dataset in the last 4 stages of the gCNN model, Figure 3 is the normalized confusion matrix of the validation result of the model on the dataset. Figure 3It can be seen that the model performs well in identifying the seven gaze regions, and the test accuracy reaches 98.9%. Compared with the randomly initialized model in this paper, the pre-training-fine-tuning model improves the test accuracy by 2%;

[0171] Figure 4 Experimental results for driver gaze estimation task using the present invention, wherein Figure 4 (a) are the test results in images collected under weak light conditions at night. The first two columns represent the data collected under weak light conditions, and the experimental results show that the model in this paper performs well in distinguishing the adjacent front windshield upper right region 1 and lower right region 2. Figure 4 The third and fourth columns of (a) show uneven lighting conditions due to the interior lights of the cockpit, but the driver's gaze and gaze region estimation task is still relatively robust. Figure 4 (b) illustrates the effect of glasses on driver gaze estimation. The model proposed in this invention can accurately estimate the gaze and gaze region of drivers wearing glasses at night ( Figure 4 (b) second column) and during the day ( Figure 4 Other columns of (b) accurately estimate the gaze and gaze region of drivers wearing glasses. In addition, this invention further verifies the ability of the trained model to identify adjacent gaze regions, and the experimental results are shown in Figure 4 (c). Figure 4 In (c), the driver's head posture is almost the same when gazing at these four regions, but there are slight differences in pupil and iris features. The interference of adjacent regions on the model's accuracy estimation, for example, the rearview mirror region 8 ( Figure 4 the last column of (c)) is a small gaze region, which is very likely to be classified as region 5 ( Figure 4 the first column in (c)), and similar pitch angles are used when gazing at these two regions. In addition, when gazing at region 8 and region 6 ( Figure 4 the second column of (c)), the driver is likely to use a similar gaze yaw angle. At the same time, when the gaze region is far from the driver's face, the interference of adjacent regions will increase accordingly. For example, the left rearview mirror region 7 ( Figure 4 the third column of (c)) is far from the driver's face, and the gaze pitch angle is almost the same, which brings great challenges to gaze region estimation. Overall, Figure 4 (c) experimental results show that the model proposed in this invention has high robustness in distinguishing these adjacent regions. The driver's gaze behavior in the cockpit can be achieved by adjusting the head posture or through eye movement to observe the behavior of a specific region ( Figure 4 Figure 4(d) ). If the model only uses head pose or eye region image for gaze region estimation, it can make the model calculation error larger. The present application uses full face image and line of sight as the input of the model, adopts the gaze region estimation model embedded with attention mechanism, can capture the subtle features of the eyeball, and extract more effective features from the full face image, so as to deal with the situation that the head posture of the driver is not consistent with the eyeball direction; it should be understood that the parts not elaborated in the specification are all prior art.

[0172] It should be understood that the above description of the embodiments is more detailed, and therefore should not be considered as limiting the scope of patent protection of the present application. Those skilled in the art can make substitutions or modifications without departing from the scope of protection of the present application, which falls within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for estimating the line of sight of a multi-tasking driver for assisting safe driving, characterized in that, The method comprises the following steps: Step 1: obtaining multiple frames of driver images from a video sequence, marking the gaze direction and gaze region type of each frame of driver image, extracting multiple facial key feature points of each frame of driver image, a facial region image of each frame of driver image by a Dlib CNN method, performing eye region cropping and driver head posture extraction on the basis of the multiple facial key feature points of each frame of driver image to obtain a left eye region image of each frame of driver image, a right eye region image of each frame of driver image and a head posture of each frame of driver image; Step 2: constructing a driving gaze estimation network, taking the left eye region image of each frame of driver image, the right eye region image of each frame of driver image, the facial region image of each frame of driver image and the head posture of each frame of driver image as input to the driver gaze estimation network for gaze estimation to obtain an estimated gaze direction and an estimated gaze region type of each frame of driver image, constructing a driver gaze estimation network loss function in combination with the gaze direction and the gaze region type of each frame of driver image, and obtaining an optimized driving gaze estimation network through SGD algorithm optimization training; Step 3: obtaining a real-time driver image, inputting the real-time driver image into the optimized driving gaze estimation network for gaze estimation to obtain an estimated gaze direction and an estimated gaze region type of the real-time driver image; The left eye EE-Net network model is used for inputting the left eye region image of each frame of driver image to perform left eye feature extraction to obtain the left eye feature of each frame of driver image. The left eye EE-Net network model is sequentially cascaded by a first left eye convolution layer, a first left eye activation layer, a first left eye expansion convolution layer, a first left eye expansion activation layer, a first left eye maximum pooling layer, a second left eye convolution layer, a second left eye activation layer, a second left eye expansion convolution layer, a second left eye expansion activation layer, a second left eye maximum pooling layer, a third left eye convolution layer, a third left eye activation layer, a third left eye expansion convolution layer, a third left eye expansion activation layer, a fourth left eye convolution layer, a fourth left eye activation layer, a fourth left eye expansion convolution layer and a fourth left eye expansion activation layer. The left eye feature extraction to obtain the left eye feature of each frame of driver image is specifically as follows: The left eye region image of each frame of driver image sequentially passes through the first left eye convolution layer for convolution feature extraction, the first left eye activation layer for feature activation, the first left eye expansion convolution layer for model channel number expansion, the first left eye expansion activation layer for feature activation, the first left eye maximum pooling layer processing, the second left eye convolution layer for convolution feature extraction, the second left eye activation layer for feature activation, the second left eye expansion convolution layer for model channel number expansion, the second left eye expansion activation layer for feature activation, the second left eye maximum pooling layer processing, the third left eye convolution layer for convolution feature extraction, the third left eye activation layer for feature activation, the third left eye expansion convolution layer for model channel number expansion, the third left eye expansion activation layer for feature activation, the fourth left eye convolution layer for convolution feature extraction, the fourth left eye activation layer for feature activation, the fourth left eye expansion convolution layer for model channel number expansion, and the fourth left eye expansion activation layer for feature activation, to obtain the left eye feature of each frame of driver image.

2. The method of claim 1, wherein the method further comprises: Step 1: extracting a plurality of facial key feature points of each frame of driver image, and extracting a driver head pose based on the plurality of facial key feature points, wherein the extracting of the driver head pose comprises: extracting a plurality of facial key feature point pixel coordinates of each frame of driver image by using a facial key point detection algorithm; calculating a rotation matrix and a translation matrix between a 3D facial coordinate system and a camera coordinate system based on the plurality of facial key feature point pixel coordinates of each frame of driver image and corresponding key point coordinate values in a standard 3D facial model by using a classical Perspective-n-Point algorithm; taking the rotation matrix as a head pose vector, wherein the head pose vector comprises a pitch angle of the head pose, a yaw angle of the head pose, and a roll angle of the head pose; and constructing the head pose vector based on the pitch angle of the head pose, the yaw angle of the head pose, and the roll angle of the head pose. Step 1: extracting a plurality of facial key feature points of each frame of driver image, and extracting a driver head pose based on the plurality of facial key feature points, wherein the extracting of the driver head pose comprises: extracting a plurality of facial key feature point pixel coordinates of each frame of driver image by using a facial key point detection algorithm; calculating a rotation matrix and a translation matrix between a 3D facial coordinate system and a camera coordinate system based on the plurality of facial key feature point pixel coordinates of each frame of driver image and corresponding key point coordinate values in a standard 3D facial model by using a classical Perspective-n-Point algorithm; taking the rotation matrix as a head pose vector, wherein the head pose vector comprises a pitch angle of the head pose, a yaw angle of the head pose, and a roll angle of the head pose; and constructing the head pose vector based on the pitch angle of the head pose, the yaw angle of the head pose, and the roll angle of the head pose.

3. The method of claim 2, wherein the method further comprises: Step 1: extracting a plurality of facial key feature points of each frame of driver image, and extracting a driver head pose based on the plurality of facial key feature points, wherein the extracting of the driver head pose comprises: extracting a plurality of facial key feature point pixel coordinates of each frame of driver image by using a facial key point detection algorithm; calculating a rotation matrix and a translation matrix between a 3D facial coordinate system and a camera coordinate system based on the plurality of facial key feature point pixel coordinates of each frame of driver image and corresponding key point coordinate values in a standard 3D facial model by using a classical Perspective-n-Point algorithm; taking the rotation matrix as a head pose vector, wherein the head pose vector comprises a pitch angle of the head pose, a yaw angle of the head pose, and a roll angle of the head pose; and constructing the head pose vector based on the pitch angle of the head pose, the yaw angle of the head pose, and the roll angle of the head pose. ​ ​ ​ ​ ​ 4. The method of claim 3, wherein the method further comprises: The driving line-of-sight estimation network of step 2 comprises: left eye EE-Net network model, right eye EE-Net network model, global face feature convolutional neural network, first gaze regression fully connected layer, second gaze regression fully connected layer, gaze region 1 1 convolutional layer, gaze region average pooling layer, gaze region linear regression layer, gaze region LogSoftmax activation layer.

5. The method of claim 1, wherein the method further comprises: The right eye EE-Net network model is configured to input a right eye region image of each frame of the driver image, and perform right eye feature extraction to obtain a right eye feature of each frame of the driver image. The right eye EE-Net network model is sequentially connected by a first right eye convolution layer, a first right eye activation layer, a first right eye extended convolution layer, a first right eye extended activation layer, a first right eye maximum pooling layer, a second right eye convolution layer, a second right eye activation layer, a second right eye extended convolution layer, a second right eye extended activation layer, a second right eye maximum pooling layer, a third right eye convolution layer, a third right eye activation layer, a third right eye extended convolution layer, a third right eye extended activation layer, a fourth right eye convolution layer, a fourth right eye activation layer, a fourth right eye extended convolution layer, and a fourth right eye extended activation layer. The right eye feature extraction comprises: The right eye region image of each frame of the driver image is sequentially subjected to convolution feature extraction by the first right eye convolution layer, feature activation by the first right eye activation layer, model channel expansion by the first right eye extended convolution layer, feature activation by the first right eye extended activation layer, processing by the first right eye convolution maximum pooling layer, convolution feature extraction by the second right eye convolution layer, feature activation by the second right eye activation layer, model channel expansion by the second right eye extended convolution layer, feature activation by the second right eye extended activation layer, processing by the second right eye convolution maximum pooling layer, convolution feature extraction by the third right eye convolution layer, feature activation by the third right eye activation layer, model channel expansion by the third right eye extended convolution layer, feature activation by the third right eye extended activation layer, convolution feature extraction by the fourth right eye convolution layer, fourth right eye activation layer, model channel expansion by the fourth right eye extended convolution layer, feature activation by the fourth right eye extended activation layer, and feature activation to obtain the right eye feature of each frame of the driver image.

6. The method of claim 5, wherein the method further comprises: The global face feature convolutional neural network is configured to input a face region image of each driver image, and perform face feature extraction to obtain a face feature of each driver image. The global face feature convolutional neural network comprises: The first-stage global face convolution module, the second-stage global face convolution module, the third-stage global face convolution module, the fourth-stage global face convolution module, the fifth-stage global face convolution module, the sixth-stage global face convolution module, the seventh-stage global face convolution module, the eighth-stage global face convolution module, and the ninth-stage global face convolution module. The first-stage global face convolution module comprises a first global convolution layer. The second-stage global face convolution module comprises a second-stage first global mobile flip bottleneck convolution layer and a second-stage first global channel attention module. The third-stage global face convolution module comprises: a third-stage first global mobile flip bottleneck convolution layer, a third-stage first global channel attention module, a third-stage second global mobile flip bottleneck convolution, and a third-stage second global channel attention module. The fourth-stage global face convolution module comprises: a fourth-stage first global mobile flip bottleneck convolution layer, a fourth-stage first global channel attention module, a fourth-stage first global spatial attention module, a fourth-stage second global mobile flip bottleneck convolution layer, a fourth-stage second global channel attention module, and a fourth-stage second global spatial attention module. The fifth-stage global face convolution module comprises: a fifth-stage first global mobile flip bottleneck convolution layer, a fifth-stage first global channel attention module, a fifth-stage first global spatial attention module, a fifth-stage second global mobile flip bottleneck convolution layer, a fifth-stage second global channel attention module, a fifth-stage second global spatial attention module, a fifth-stage third global mobile flip bottleneck convolution layer, a fifth-stage third global channel attention module, and a fifth-stage third global spatial attention module. The sixth-stage global face convolution module comprises: a sixth-stage first global mobile flip bottleneck convolution layer, a sixth-stage first global channel attention module, a sixth-stage first global spatial attention module, a sixth-stage second global mobile flip bottleneck convolution layer, a sixth-stage second global channel attention module, a sixth-stage second global spatial attention module, a sixth-stage third global mobile flip bottleneck convolution layer, a sixth-stage third global channel attention module, and a sixth-stage third global spatial attention module. The seventh-stage global face convolution module comprises: a seventh-stage first global mobile flip bottleneck convolution layer, a seventh-stage first global channel attention module, a seventh-stage first global spatial attention module, a seventh-stage second global mobile flip bottleneck convolution layer, a seventh-stage second global channel attention module, a seventh-stage second global spatial attention module, a seventh-stage third global mobile flip bottleneck convolution layer, a seventh-stage third global channel attention module, a seventh-stage third global spatial attention module, a seventh-stage fourth global mobile flip bottleneck convolution layer, a seventh-stage fourth global channel attention module, and a seventh-stage fourth global spatial attention module. The eighth-stage global face convolution module comprises: an eighth-stage first global mobile flip bottleneck convolution layer, an eighth-stage first global channel attention module, and an eighth-stage first global spatial attention module. The ninth-stage global face convolution module comprises: a ninth global convolution layer and a ninth global activation layer.

7. The method of claim 6, wherein: The second stage first global mobile flip bottleneck convolution layer, the third stage first global mobile flip bottleneck convolution layer, the third stage second global mobile flip bottleneck convolution, the fourth stage first global mobile flip bottleneck convolution layer, the fourth stage second global mobile flip bottleneck convolution layer, the fifth stage first global mobile flip bottleneck convolution layer, the fifth stage second global mobile flip bottleneck convolution layer, the fifth stage third global mobile flip bottleneck convolution layer, the sixth stage first global mobile flip bottleneck convolution layer, the sixth stage second global mobile flip bottleneck convolution layer, the sixth stage third global mobile flip bottleneck convolution layer, the seventh stage first global mobile flip bottleneck convolution layer, the seventh stage first global channel attention module, the seventh stage second global mobile flip bottleneck convolution layer, the seventh stage third global mobile flip bottleneck convolution layer, the seventh stage fourth global mobile flip bottleneck convolution layer, and the eighth stage first global mobile flip bottleneck convolution layer all comprise: 1 1 x 1 convolutional layer, depthwise separable convolutional layer, 1 1 x 1 convolutional layer, Dropout layer; The second stage first global channel attention module, the third stage first global channel attention module, the third stage second global channel attention module, the fourth stage first global channel attention module, the fourth stage second global channel attention module, the fifth stage first global channel attention module, the fifth stage second global channel attention module, the fifth stage third global channel attention module, the sixth stage first global channel attention module, the sixth stage second global channel attention module, the sixth stage third global channel attention module, the seventh stage first global channel attention module, the seventh stage second global channel attention module, the seventh stage third global channel attention module, the seventh stage fourth global channel attention module, and the eighth stage first global channel attention module all comprise: max-pooling layer, average-pooling layer, 1 1 convolutional layer, SiLU activation layer, 1 1 convolutional layer, weighting layer, Sigmoid activation layer; The fourth stage first global spatial attention module, the fourth stage second global spatial attention module, the fifth stage first global spatial attention module, the fifth stage second global spatial attention module, the fifth stage third global spatial attention module, the sixth stage first global spatial attention module, the sixth stage second global spatial attention module, the sixth stage third global spatial attention module, the seventh stage first global spatial attention module, the seventh stage second global spatial attention module, the seventh stage third global spatial attention module, the seventh stage fourth global spatial attention module, and the eighth stage first global spatial attention module all comprise: max-pooling layer, average-pooling layer, 7 7convolutional layer, sigmoid activation layer The network layers of the global facial feature convolutional neural network are cascaded in the manner of the internal network layers of the EfficientNet network model; The first gaze regression fully connected layer is used for inputting the left eye features of each frame of driver image, the right eye features of each frame of driver image, the facial features of each frame of driver image, and the head posture vector, and performing regression processing to obtain the first fusion features of each frame of driver image; The second gaze regression fully connected layer is used for inputting the first fusion features of each frame of driver image, and performing regression processing to obtain the estimated gaze direction of each frame of driver image; The facial features of each frame of the driver image are sequentially passed through the gaze region 1 1convolutional layer for convolutional feature extraction, gaze region average pooling layer for pooling processing, gaze region linear regression layer for linear regression, gaze region LogSoftmax activation layer for feature activation, to obtain the probability of each gaze region type of each frame of the driver image, and select the gaze region type with the highest probability among all gaze region types of each frame of the driver image as the estimated gaze region type of each driver image.

8. The multi-task driver gaze estimation method for assisted safe driving according to claim 7, characterized in that: The driver gaze estimation network loss function described in step 2 includes: driver gaze direction estimation loss function and driver gaze area estimation loss function; The driver gaze direction estimation loss function is used to constrain the estimated gaze direction of each frame of the driver image, and is defined as follows: Using mean square error, MSE, as a loss function for driver gaze direction estimation may be defined as: wherein, N represents the number of driver images, i represents an index of each frame of driver images, is the estimated gaze direction of the i frame driver image, is the gaze direction of the i frame driver image; The driver gaze region estimation loss function is used to constrain the estimated gaze region of each frame of driver image, and the driver gaze region estimation loss function includes a cross-entropy loss function , a center loss function ; The cross-entropy loss function is defined as: wherein, i denotes an index of the driver image per frame, is the gaze region type of the i driver image of the is the estimated gaze region type of the i driver image of the cross-entropy loss function for measuring a difference between each gaze region type predicted probability value of each frame of driver image and a gaze region true value of each frame of driver image; the center loss function reducing intra-class scatter, which can be defined as: wherein, N denotes the number of driver images, is the i feature vector of the is the i center of the gaze region of the driver image; The loss function for estimating the driver's gaze region can be defined as: wherein, denotes a weight of the center loss function, used to adjust the intra-class compactness.

9. A multi-task driver's line-of-sight estimation system for assisting safe driving, based on the multi-task driver's line-of-sight estimation method according to any one of claims 1 to 8, characterized by include: The driver image and pose extraction module is used to acquire multiple frames of driver images from the video sequence, mark the gaze direction and gaze region type of each frame of driver image, extract multiple key facial feature points and facial region images of each frame of driver image using the Dlib CNN method, combine the multiple key facial feature points of each frame of driver image to perform eye region cropping and driver head pose extraction, and obtain the left eye region image, the right eye region image, and the head pose of each frame of driver image. The driver gaze estimation network training module is used to construct the driver gaze estimation network. The left eye region image, right eye region image, facial region image, and head pose of each driver image frame are input into the driver gaze estimation network to estimate gaze, and the estimated gaze direction and estimated gaze region type of each driver image frame are obtained. The driver gaze estimation network loss function is constructed by combining the gaze direction and gaze region type of each driver image frame. The optimized driver gaze estimation network is obtained by training through SGD algorithm. The real-time driver image estimation module is used to acquire real-time driver images, input them into the optimized driver gaze estimation network for gaze estimation, and obtain the estimated gaze direction and estimated gaze region type of the real-time driver image.

Images