Driver's line of sight recognition method and device, vehicle, and storage medium

Abstract

A driver's gaze recognition method and device, a vehicle, and a storage medium are provided, which include the steps of acquiring image data of a target object, processing the image data to acquire a face image and a head image of the target object, acquiring an eye image of the target object based on the face image and acquiring a head pose of the target object based on the head image, processing the eye image to acquire a relative position of the pupil and the sclera, and determining the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera.

Description

[Technical Field]

[0001] The present disclosure relates to the field of vehicle technology, and more particularly to a driver's gaze recognition method, a driver's gaze recognition device, a computer-readable storage medium, and a vehicle. [Background technology]

[0002] The driver's gaze direction is crucial to the safety of driving a vehicle. The number of car accidents caused by distraction and fatigue while driving is countless every year, and bloody lessons are clearly visible. It is particularly important to accurately estimate the driver's gaze direction while driving in real time and assist the driver in driving safely.

[0003] In related technologies, gaze estimation algorithms learn human gaze based on the human eye as the network input, but ignore the influence of head pose, resulting in inaccurate gaze estimation, which affects driving safety and poor user experience. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure solves at least one of the technical problems in the related art. Therefore, a first object of the present disclosure is to provide a driver's gaze recognition method, which simultaneously combines a face image, a head pose, and a relative position of the pupil and the sclera as input to a network to recognize the driver's gaze direction, thereby improving the accuracy of gaze direction recognition, thereby enabling accurate real-time monitoring of the driver's gaze, improving driving safety, and improving user experience.

[0005] A second object of the present disclosure is to provide a driver's line of sight recognition device.

[0006] A third object of the present disclosure is to provide a computer-readable storage medium.

[0007] A fourth object of the present disclosure is to provide a vehicle. [Means for solving the problem]

[0008] To achieve the above object, an embodiment of a first aspect of the present disclosure provides a driver's gaze recognition method, including steps of acquiring image data of a target object, processing the image data to acquire a face image and a head image of the target object, acquiring an eye image of the target object based on the face image and acquiring a head pose of the target object based on the head image, processing the eye image to acquire a relative position of the pupil and the sclera, and determining the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera.

[0009] The driver's gaze recognition method according to the embodiment of the present disclosure first processes image data to obtain a face image and a head image of a target object, then obtains an eye image of the target object based on the face image, obtains the head pose of the target object based on the head image, and further processes the eye image to obtain the relative position of the pupil and the sclera, and finally determines the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera. In this way, the method simultaneously combines the face image, the head pose, and the relative position of the pupil and the sclera as input to the network to recognize the driver's gaze direction, thereby improving the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety, and improving user experience.

[0010] In addition, the driver's gaze recognition method according to the above-described embodiment of the present disclosure may further have the following additional technical features.

[0011] According to one embodiment of the present disclosure, the step of processing the eye image to obtain the relative position of the pupil and the sclera includes the steps of segmenting a pupil region and a sclera region in the eye image using an image segmentation network, acquiring pupil region position information and sclera region position information, and determining the relative position of the pupil and the sclera based on the pupil region position information and the sclera region position information.

[0012] According to one embodiment of the present disclosure, the step of acquiring a facial image of a target object includes a step of processing the image using a face detection model, and the backbone network structure of the face detection model is a lightweight network structure, which adopts a BiFPN (Bidirectional Feature Pyramid Network) structure based on a dilated convolutional neural network to fuse features from different layers.

[0013] According to one embodiment of the present disclosure, the step of acquiring a head image of the target subject includes a step of expanding the facial image of the target subject in the vertical and horizontal directions at a magnification ratio preset for the image data, and acquiring the head image.

[0014] According to another embodiment of the present disclosure, the step of acquiring a head image of the target object includes processing the image data based on one of an RCNN (Region Convolutional Neural Networks) series deep learning algorithm, a Yolo series deep learning algorithm, an SSD (Single Shot MultiBox Detector) series deep learning algorithm, and an anchor-free target detection algorithm to acquire the head image.

[0015] According to one embodiment of the present disclosure, the step of acquiring an eye image of a target object based on a face image includes the steps of acquiring eye keypoints in an area where the eyes of the target object are located based on a face keypoint detection algorithm, acquiring the coordinates of the eye keypoints, acquiring a circumscribing rectangle of the left eye keypoints and a circumscribing rectangle of the right eye keypoints, and expanding the circumscribing rectangles of the left eye / right eye keypoints up, down, left, and right at a preset ratio to determine the eye image of the target object.

[0016] According to one embodiment of the present disclosure, the step of obtaining the circumscribing rectangle of the eye keypoints of the left eye / right eye includes the steps of obtaining the maximum and minimum coordinates of the eye keypoint coordinates of the left eye / right eye along the x-axis and y-axis, respectively, and determining the difference between the maximum and minimum coordinates along the x-axis direction as the length of the circumscribing rectangle, and the difference between the maximum and minimum coordinates along the y-axis direction as the width of the circumscribing rectangle.

[0017] According to one embodiment of the present disclosure, the step of acquiring the head pose of the target object based on the head image includes: acquiring Euler angles of the target object based on a deep learning network; and determining the head pose when a difference between the Euler angles and a predetermined angle is smaller than a preset threshold.

[0018] According to an embodiment of the present disclosure, the step of determining the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera includes: performing feature extraction on the face image, the head pose, and the relative position of the pupil and the sclera based on a multi-branch deep learning network; and acquiring the gaze direction of the target object by connecting the facial features of the face image, the head features of the head pose, and the relative position of the pupil and the sclera.

[0019] In order to achieve the above object, an embodiment of a second aspect of the present disclosure provides a driver's gaze recognition device, comprising: an image acquisition module for acquiring image data of a target object; a face detection module for processing the image data to acquire a facial image of the target object; a head detection module for processing the image data to acquire a head image of the target object; an eye detection module for acquiring an eye image of the target object based on the facial image; a head posture estimation module for acquiring the head posture of the target object based on the head image; an image processing module for processing the eye image to acquire the relative position of the pupil and the sclera; and a gaze estimation module for determining the gaze direction of the target object based on the facial image, the head posture, and the relative position of the pupil and the sclera.

[0020] A driver's gaze recognition device according to an embodiment of the present disclosure acquires image data of a target object using an image acquisition module, processes the image data to acquire a facial image of the target object using a face detection module, processes the image data to acquire a head image of the target object using a head detection module, acquires an eye image of the target object based on the facial image using an eye detection module, acquires a head pose of the target object based on the head image using a head pose estimation module, processes the eye image using an image processing module to acquire the relative position of the pupil and sclera, and determines the gaze direction of the target object based on the facial image, the head pose, and the relative position of the pupil and sclera using a gaze estimation module. In this way, the device can simultaneously combine the facial image, head pose, and relative position of the pupil and sclera as input to a network to recognize the driver's gaze direction and improve the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety, and improving user experience.

[0021] To achieve the above object, an embodiment of a third aspect of the present disclosure provides a computer-readable storage medium, in which a driver's gaze recognition program is stored, and when the driver's gaze recognition program is executed by a processor, the above driver's gaze recognition method is realized.

[0022] The computer-readable storage medium according to the embodiment of the present disclosure can improve the accuracy of gaze direction recognition by executing the above-mentioned driver gaze recognition method, thereby enabling accurate real-time monitoring of the driver's gaze, improving driving safety, and improving the user experience.

[0023] To achieve the above object, an embodiment of a fourth aspect of the present disclosure provides a vehicle, the vehicle comprising a memory, a processor, and a driver's gaze recognition program stored in the memory and operable by the processor, and when the processor executes the driver's gaze recognition program, it realizes the above driver's gaze recognition method. [Effects of the Invention]

[0024] By implementing the driver's gaze recognition method described above, a vehicle according to an embodiment of the present disclosure can improve the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety and improving the user experience.

[0025] Additional aspects and advantages of the present disclosure will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the present disclosure. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a flowchart illustrating a driver's gaze recognition method according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a structural schematic diagram of a face detection model according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a structural schematic diagram of a BiFPN according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a distribution diagram of 68 key points according to one embodiment of the present disclosure. [Figure 5] FIG. 1 is a structural schematic diagram of a multi-branch deep learning network according to an embodiment of the present disclosure. [Figure 6] 1 is a block diagram illustrating a driver's gaze recognition device according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a block diagram illustrating a vehicle according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, the embodiments of the present disclosure shown in the drawings will be described in detail. In all drawings, the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions. The embodiments described through the following reference drawings are illustrative and are intended to explain the present disclosure, and should not be understood as limitations on the present invention.

[0028] Hereinafter, a driver's gaze recognition method, a driver's gaze recognition device, a computer-readable storage medium, and a vehicle according to embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0029] FIG. 1 is a flowchart illustrating a driver's gaze recognition method according to an embodiment of the present disclosure.

[0030] As shown in FIG. 1, a driver's gaze recognition method according to an embodiment of the present disclosure may include the following steps.

[0031] S1, acquire image data of the target object.

[0032] Specifically, an image of the target object (i.e., the driver) can be captured by an in-car camera. The in-car camera can be placed on the A-pillar, rearview mirror, or steering wheel inside the vehicle, and the camera can be an IR camera or an RGB camera. The in-car camera captures images of the target object inside the vehicle in real time, thereby obtaining image data of the target object.

[0033] S2, process the image data to obtain the face image and head image of the target subject.

[0034] Specifically, a face is detected from the image data using a face detection algorithm, the area position where the target face is located is obtained, an image of the target face is obtained, and the head is detected from the image data using a target detection algorithm, the area position where the target head is located is obtained, and an image of the target head is obtained.

[0035] S3: Acquire an eye image of the target subject based on the face image, and acquire a head posture of the target subject based on the head image.

[0036] S4, the eye image is processed to obtain the relative positions of the pupil and the sclera.

[0037] S5, determine the gaze direction of the target subject based on the facial image, head pose, and the relative position of the pupil and sclera.

[0038] Specifically, after capturing an image of the target object with an in-vehicle camera, face detection is performed on the image data of the target object to obtain the position coordinates of the facial area in the image. The facial image of the target object can be obtained by cropping out the facial area. Face detection in image data can be performed using algorithms such as MTCNN, RetinaFace, or the Yolo series, or by using the Face Detector disclosed in OpenCV or a face detector in Dlib. By detecting the head in the image data using a target detector, the position coordinates of the area where the target object's head is located can be obtained, and the head image of the target object can be obtained by cropping out the head area. After obtaining the facial image of the target object, eye area detection is performed on the facial image to obtain the position of the eye area in the facial image, and the eye image can be obtained after cropping. Eye detection in facial images can be performed using a facial keypoint-based detection method, such as PFLD, or by using the OpenCV open source library Dlib to detect facial keypoints. Alternatively, eye detection can be performed using target detection algorithms, such as the RCNN series, Yolo series, SSD series, and CenterNet, with the eye region as a target. Furthermore, the relative position of the pupil and sclera can be obtained by segmenting the eye image. After obtaining a head image of the target object, the head pose of the target object is estimated based on the head image to obtain the target object's head posture. The gaze direction of the target object can be determined by estimating it based on the facial image, head pose, and the relative position of the pupil and sclera. This allows the above gaze recognition method to monitor the driver's gaze direction in real time and determine whether the driver's attention is on the road. When the driver's attention moves away from the road for a certain period of time, a real-time voice reminder is provided, bringing the driver's attention back to driving, protecting them while driving and significantly reducing the risk of accidents.

[0039] It is important to understand that a person's gaze direction is closely related to the position of the eyeballs and the person's head posture. When a person's head posture is fixed, the position of the eyeballs determines the direction of the person's gaze. When the eyeball position is stationary, turning the head will cause the person's gaze direction to change accordingly. The gaze recognition method disclosed herein can determine the gaze direction of a target object by estimating the gaze direction based on a facial image, head posture, and the relative position of the pupil and sclera, thereby improving the accuracy of gaze direction recognition. This allows the driver's gaze to be accurately monitored in real time, thereby improving driving safety and user experience.

[0040] According to one embodiment of the present disclosure, the step of processing the eye image to obtain the relative position of the pupil and the sclera includes the steps of segmenting a pupil region and a sclera region in the eye image using an image segmentation network, acquiring pupil region position information and sclera region position information, and determining the relative position of the pupil and the sclera based on the pupil region position information and the sclera region position information.

[0041] Specifically, after obtaining an eye image, the target eye image is segmented using an image segmentation network to segment the pupil region and sclera region in the eye image, and the relative position of the pupil to the sclera is obtained. The image segmentation network can be one of the following image segmentation algorithms: DDRNet, DeepLab series, PSPNet, UNet series, and Transformer.

[0042] According to one embodiment of the present disclosure, the step of acquiring a face image of a target object includes processing the image using a face detection model, wherein the backbone network structure of the face detection model is a lightweight network structure, and a BiFPN structure based on a dilated convolutional neural network is used to fuse features from different layers.

[0043] Specifically, FIG. 2 is a structural schematic diagram illustrating a face detection model according to one embodiment of the present disclosure. As shown in FIG. 2, mobileNetV3 is used as the backbone network of the face detector, and the backbone network structure of the face detection model is a lightweight network structure. A BiFPN structure based on a dilated convolutional neural network is then used to fuse features from different layers. Image data of the target object is input to the face detection model for face detection, resulting in a face region, which is a face detection frame containing the face image. The face image of the target object is then obtained by cropping the face region. The face detection model can improve the receptive field through dilated convolution, while simultaneously using a depth-separable convolutional structure to reduce the computational complexity of the model, thereby reducing the model size. The BiFPN structure is used to better fuse features at different scales. The BiFPN structure is shown in FIG. 3.

[0044] According to an embodiment of the present disclosure, the step of acquiring a head image of the target object includes, based on a facial image of the target object, expanding the facial image vertically and horizontally at a preset expansion ratio for the image data to acquire the head image. The preset ratio can be set as needed, and may be 20%, thereby saving computational power.

[0045] According to another embodiment of the present disclosure, the step of acquiring the head image of the target object comprises processing the image data to acquire the head image based on one of an RCNN series deep learning algorithm, a Yolo series deep learning algorithm, an SSD series deep learning algorithm, and an anchor-free target detection algorithm, that is, the image data of the target object is subjected to target detection by one of the above algorithms to acquire the head image of the target object.

[0046] According to one embodiment of the present disclosure, the step of acquiring an eye image of a target object based on a face image includes the steps of acquiring eye keypoints in an area where the eyes of the target object are located based on a face keypoint detection algorithm, acquiring the coordinates of the eye keypoints, acquiring a circumscribing rectangle of the left eye keypoints and a circumscribing rectangle of the right eye keypoints, and expanding the circumscribing rectangles of the left eye / right eye keypoints up, down, left, and right at a preset ratio to determine the eye image of the target object.

[0047] According to one embodiment of the present disclosure, the step of obtaining the circumscribing rectangle of the eye keypoints of the left eye / right eye includes the steps of obtaining the maximum and minimum coordinates of the eye keypoint coordinates of the left eye / right eye along the x-axis and y-axis, respectively, and determining the difference between the maximum and minimum coordinates along the x-axis direction as the length of the circumscribing rectangle, and the difference between the maximum and minimum coordinates along the y-axis direction as the width of the circumscribing rectangle.

[0048] Specifically, there are 5-point, 68-point, and 98-point face keypoint detection algorithms. The more keypoints there are, the higher the accuracy of the final human eye region. The following explanation takes 68 keypoints as an example.

[0049] Figure 4 is a distribution diagram of 68 key points. As shown in Figure 4, the left eye corresponds to six key points, such as 37 to 42, and the right eye corresponds to six key points, such as 43 to 48. Based on the position coordinates of the face key points, the coordinates of the eye key points can be obtained. The coordinates of the left eye key point are (x 37 ,y 37 ), (x 38 ,y 38 ), (x 39 ,y 39 ), (x 40 ,y 40 )(x 41 ,y 41 ), (x 42 ,y 42 ) The right eye keypoint coordinates are (x 43 ,y 43 ), (x44 ,y 44 ), (x 45 ,y 45 ), (x 46 ,y 46 )(x 47 ,y 47 ), (x 48 ,y 48 ) Based on the obtained coordinates of the eye keypoints of the left and right eyes, the bounding rectangles of the eye keypoints of the left and right eyes can be obtained. Taking the left eye as an example, the formula for obtaining the bounding rectangles of the eye keypoints of the left eye is as follows:

number

[0050] where X min is the minimum coordinate of the eye keypoint of the left eye along the x-axis, and X max is the maximum coordinate along the x-axis of the eye keypoint coordinates of the left eye, and Y min is the minimum coordinate along the y-axis of the eye keypoint coordinates of the left eye, and Y max is the maximum coordinate along the y-axis of the left eye keypoint coordinate, W is the length of the bounding rectangle, and H is the width of the bounding rectangle. Based on the left eye keypoint coordinate, the bounding rectangle of the left eye keypoint can be obtained by the above formula. The bounding rectangle of the right eye keypoint can be derived in a similar way.

[0051] Because the eye keypoints are too close to the eye, in order to make the circumscribing rectangle include the entire eye, after obtaining the circumscribing rectangle of the left / right eye keypoints, the circumscribing rectangle of the left / right eye keypoints is expanded up / down / left / right at a preset ratio, for example, it can be expanded to the left and right by a size of 0.75*W and at an equal ratio up / down, so as to obtain a complete eye image of the target object.

[0052] According to one embodiment of the present disclosure, the step of acquiring the head pose of the target object based on the head image includes: acquiring Euler angles of the target object based on a deep learning network; and determining the head pose when a difference between the Euler angles and a predetermined angle is smaller than a preset threshold.

[0053] Specifically, after detecting a head image, a deep learning network is used to learn the head pose of the target object from the head image. The three directional angles of the target object, head pitch angle (Pitch), head yaw angle (Yaw), and head roll angle (Roll), i.e., Euler angles, are obtained, and the current head pose can be expressed using the Euler angles. The difference between the Euler angles and a predetermined angle of the pose is calculated. The smaller the obtained difference, the higher the accuracy of the head image. If the difference is smaller than a preset threshold, the head pose of the target object can be determined.

[0054] According to an embodiment of the present disclosure, the step of determining the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera includes: performing feature extraction on the face image, the head pose, and the relative position of the pupil and the sclera based on a multi-branch deep learning network; and acquiring the gaze direction of the target object by connecting the facial features of the face image, the head features of the head pose, and the relative position of the pupil and the sclera.

[0055] Specifically, gaze direction is closely related not only to head pose but also to the position of the pupil. When the head is fixed, the pupil position changes, and so does the gaze direction. Therefore, the facial image, head pose, and the relative position of the pupil and sclera must be input to a multi-branch deep learning network to determine the gaze direction of the target object. The structure of the multi-branch deep learning network is shown in Figure 5.

[0056] Specifically, as shown in Figure 5, the facial image, head pose, and eye image of the target object are input to a multi-branch deep learning network for feature extraction. For the head branch, a head pose estimation network is used to estimate the head pose. For the face branch, a convolutional neural network (CNN) is used to extract features from the facial image. For the eye branch, an image segmentation network is used to segment the image and obtain the relative position of the pupil relative to the sclera. Here, attention mechanisms are incorporated into the feature extraction process, and a loss function is used as a constraint to obtain the facial features of the facial image, the head features of the head pose, and the relative position of the pupil and sclera. The facial features of the facial image, the head features of the head pose, and the relative position of the pupil and sclera are represented by vectors. Finally, the three vectors of the facial features of the facial image, the head features of the head pose, and the relative position of the pupil and sclera are successfully combined at the same level, resulting in a single vector that is input to the transformer structure. By learning the correlation between the facial features of the facial image, the head features of the head posture, and the relative position of the pupil and sclera, the target object's gaze direction is output. This allows the driver's gaze direction to be monitored in real time, and it can be determined whether the driver's attention is on the road. If the driver's attention is away from the road for a certain period of time, a real-time voice reminder will be sent to help the driver return their attention to driving, protecting them while driving and greatly reducing the probability of accidents.

[0057] As described above, the driver's gaze recognition method according to the embodiment of the present disclosure first processes image data to obtain a face image and a head image of the target object, then obtains an eye image of the target object based on the face image, obtains the head pose of the target object based on the head image, and further processes the eye image to obtain the relative position of the pupil and the sclera, and finally determines the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera. In this way, the method simultaneously combines the face image, the head pose, and the relative position of the pupil and the sclera as input to the network to recognize the driver's gaze direction and improve the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety, and improving user experience.

[0058] Corresponding to the above embodiment, the present disclosure further provides a driver's gaze recognition device.

[0059] FIG. 6 is a block diagram illustrating a driver's line of sight recognition device according to an embodiment of the present disclosure.

[0060] As shown in FIG. 6 , the driver's gaze recognition device 100 of an embodiment of the present disclosure may include an image acquisition module 110, a face detection module 120, a head detection module 130, an eye detection module 140, a head pose estimation module 150, an image processing module 160, and a gaze estimation module 170.

[0061] The image acquisition module 110 is used to acquire image data of a target subject. The face detection module 120 is used to process the image data to acquire a facial image of the target subject. The head detection module 130 is used to process the image data to acquire a head image of the target subject. The eye detection module 140 is used to acquire an eye image of the target subject based on the facial image. The head pose estimation module 150 is used to acquire a head pose of the target subject based on the head image. The image processing module 160 is used to process the eye image to acquire the relative position of the pupil and the sclera. The gaze estimation module 170 is used to determine the gaze direction of the target subject based on the facial image, the head pose, and the relative position of the pupil and the sclera.

[0062] According to an embodiment of the present disclosure, the image processing module 160 processes the eye image to obtain the relative positions of the pupil and the sclera, specifically, to segment the pupil region and the sclera region in the eye image using an image segmentation network, to obtain pupil region position information and sclera region position information, and to determine the relative positions of the pupil and the sclera based on the pupil region position information and the sclera region position information.

[0063] According to one embodiment of the present disclosure, the face detection module 120 is used to obtain a face image of a target object and, specifically, to process the image using a face detection model, the backbone network structure of which is a lightweight network structure, which uses a BiFPN structure based on a dilated convolutional neural network to fuse features from different layers.

[0064] According to one embodiment of the present disclosure, the head detection module 130 is used to acquire a head image of the target subject, specifically, based on the facial image of the target subject, expand the facial image up, down, left, and right at a predetermined expansion ratio for the image data to acquire the head image.

[0065] According to another embodiment of the present disclosure, the head detection module 130 is used to acquire a head image of a target object, and specifically, to process the image data based on one of an RCNN series deep learning algorithm, a Yolo series deep learning algorithm, an SSD series deep learning algorithm, and an anchor-free target detection algorithm to acquire the head image.

[0066] According to one embodiment of the present disclosure, the eye detection module 140 acquires the eye image of the target object based on the face image, specifically, it is used to acquire the eye keypoints of the area where the eyes of the target object are located based on the face keypoint detection algorithm, acquire the coordinates of the eye keypoints, acquire the circumscribing rectangle of the left eye keypoints and the circumscribing rectangle of the right eye keypoints, and expand the circumscribing rectangle of the left eye / right eye keypoints up, down, left, and right at a preset ratio to determine the eye image of the target object.

[0067] According to one embodiment of the present disclosure, the eye detection module 140 is used to obtain the circumscribing rectangle of the eye keypoints of the left eye / right eye, specifically, to obtain the maximum and minimum coordinates along the x-axis and y-axis of the eye keypoint coordinates of the left eye / right eye, respectively, and to determine the difference between the maximum and minimum coordinates along the x-axis direction as the length of the circumscribing rectangle, and the difference between the maximum and minimum coordinates along the y-axis direction as the width of the circumscribing rectangle.

[0068] According to one embodiment of the present disclosure, the head pose estimation module 150 is used to obtain the head pose of the target object based on the head image, specifically, to obtain the Euler angles of the target object based on a deep learning network, and to determine the head pose when the difference between the Euler angles and a predetermined angle is smaller than a preset threshold.

[0069] According to an embodiment of the present disclosure, the gaze estimation module 170 determines the gaze direction of the target object based on the face image, the head pose, and the relative position of the pupil and the sclera, specifically, is used to perform feature extraction for each of the face image, the head pose, and the relative position of the pupil and the sclera based on a multi-branch deep learning network, and acquire the gaze direction of the target object by connecting the facial features of the face image, the head features of the head pose, and the relative position of the pupil and the sclera.

[0070] In addition, for details not disclosed in the driver's gaze recognition device according to the embodiment of the present disclosure, please refer to the details disclosed in the driver's gaze recognition method according to the embodiment of the present disclosure, and they will not be explained in detail again here.

[0071] A driver's gaze recognition device according to an embodiment of the present disclosure acquires image data of a target object using an image acquisition module, processes the image data to acquire a facial image of the target object using a face detection module, processes the image data to acquire a head image of the target object using a head detection module, acquires an eye image of the target object based on the facial image using an eye detection module, acquires a head pose of the target object based on the head image using a head pose estimation module, processes the eye image using an image processing module to acquire the relative position of the pupil and sclera, and determines the gaze direction of the target object based on the facial image, the head pose, and the relative position of the pupil and sclera using a gaze estimation module. In this way, the device can simultaneously combine the facial image, head pose, and relative position of the pupil and sclera as input to a network to recognize the driver's gaze direction and improve the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety, and improving user experience.

[0072] Corresponding to the above embodiment, the present disclosure further provides a computer-readable storage medium.

[0073] A computer-readable storage medium according to an embodiment of the present disclosure stores a driver's gaze recognition program, which, when executed by a processor, realizes the driver's gaze recognition method described above.

[0074] The computer-readable storage medium according to the embodiment of the present disclosure can improve the accuracy of gaze direction recognition by executing the above-mentioned driver gaze recognition method, thereby enabling accurate real-time monitoring of the driver's gaze, improving driving safety, and improving the user experience.

[0075] Corresponding to the above embodiment, the present disclosure further provides a vehicle.

[0076] FIG. 7 is a block diagram of a vehicle according to an embodiment of the present disclosure.

[0077] As shown in FIG. 7, a vehicle 200 according to an embodiment of the present disclosure includes a memory 210, a processor 220, and a driver's gaze recognition program that is stored in the memory 210 and can be run by the processor 220, and when the processor 220 executes the driver's gaze recognition program, it realizes the above-mentioned driver's gaze recognition method.

[0078] By implementing the driver's gaze recognition method described above, a vehicle according to an embodiment of the present disclosure can improve the accuracy of gaze direction recognition, thereby accurately monitoring the driver's gaze in real time, improving driving safety and improving the user experience.

[0079] It should be noted that the logic and / or steps shown in flowcharts or otherwise described herein can be considered, for example, as an ordered listing of executable instructions for implementing logical functions, and can be tangibly embodied in any computer-readable medium and used in or in conjunction with an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or a system that reads instructions from and executes instructions from an instruction execution system, device, or device). As used herein, a "computer-readable medium" may refer to any device that contains, stores, communicates, propagates, or transmits a program and can be used in or in conjunction with an instruction execution system, device, or device. More specific examples (non-exhaustive list) of computer-readable media include an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable editable read-only memory (EPROM or flash memory), fiber optic devices, and portable disk read-only memory (CD-ROM). The computer readable medium may also be paper or other suitable medium on which the program is printed, for example by optically scanning the paper or other suitable medium and then editing, interpreting or processing in any other suitable manner as required to obtain the program in electronic form, which is then stored in computer memory.

[0080] It should be understood that each part of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by an appropriate instruction execution system. When implemented in hardware, as in other embodiments, the implementation can be implemented in any one or combination of technologies known in the art, such as a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, a dedicated integrated circuit having appropriate combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0081] In the description herein, the reference words "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In the description herein, the above-mentioned terms, exemplary descriptions, do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples.

[0082] It should be noted that the terms "first" and "second" are for descriptive purposes only and cannot be considered to indicate or imply relative importance or the number of technical features. Thus, a feature qualified as "first" or "second" can expressly or imply the inclusion of one or more of the feature. In the description of this disclosure, unless otherwise expressly and specifically limited, the concept of "plurality" refers to at least two, e.g., two or three.

[0083] In the present disclosure, unless otherwise clearly defined and limited, the terms "attached," "coupled," "connected," "fixed," and the like should be understood in a broad sense, and may mean, for example, fixedly connected, detachably connected, or integrally connected, mechanically connected, directly connected, indirectly connected via an intermediate medium, internal communication between two elements, or an interactive relationship between two elements. Those skilled in the art will be able to understand the specific meaning of the above terms in the present invention depending on the context.

[0084] Although the embodiments of the present disclosure have been shown and described, the above embodiments are illustrative and should not be construed as limiting the present disclosure. Those skilled in the art will appreciate that various changes, modifications, substitutions, and variations can be made to the above embodiments within the scope of the present disclosure.

[0085] [CROSS-REFERENCE TO RELATED APPLICATIONS] This disclosure claims priority to a Chinese patent application filed on September 7, 2022, bearing application number 202211089211.8 and entitled "Driver's gaze recognition method and device, vehicle and storage medium," the entire contents of which are incorporated herein by reference.

Claims

1. A driver's line of sight recognition method, acquiring image data of a target object; processing the image data to obtain face and head images of the target subject; acquiring an eye image of the target subject based on the face image, and acquiring a head pose of the target subject based on the head image; processing the eye image to obtain a relative position of the pupil and the sclera; determining a gaze direction of the target object based on the facial image, the head pose, and the relative position of the pupil and sclera.

2. The step of processing the eye image to obtain a relative position of the pupil and the sclera includes: Segmenting the eye image into pupil and sclera regions using an image segmentation network; acquiring pupil region position information and sclera region position information; The method for recognizing a driver's line of sight according to claim 1 , further comprising: determining a relative position of the pupil and the sclera based on the pupil region position information and the sclera region position information.

3. The step of acquiring a facial image of the target subject includes:

2. The method for driver gaze recognition according to claim 1, further comprising: processing the image using a face detection model; wherein a backbone network structure of the face detection model is a lightweight network structure, and a BiFPN structure based on a dilated convolutional neural network is used to fuse features from different layers.

4. The step of acquiring a head image of the target object includes: The driver's gaze recognition method according to claim 1, further comprising a step of expanding the facial image of the target object in the vertical and horizontal directions at a magnification ratio preset for the image data to acquire the head image.

5. The step of acquiring a head image of the target object includes:

2. The driver's gaze recognition method of claim 1, further comprising: processing the image data based on one of an RCNN series deep learning algorithm, a Yolo series deep learning algorithm, an SSD series deep learning algorithm, and an anchor-free target detection algorithm to obtain the head image.

6. The step of acquiring an eye image of the target subject based on the face image includes: acquiring eye keypoints in the eye area of ​​the target object based on a face keypoint detection algorithm; obtaining the coordinates of the eye keypoints; obtaining a bounding rectangle of the eye keypoints for the left eye and a bounding rectangle of the eye keypoints for the right eye; The driver's gaze recognition method according to claim 1, further comprising a step of determining an eye image of the target object by expanding the circumscribing rectangle of the eye key points of the left eye / right eye in the vertical and horizontal directions at a preset ratio.

7. The step of obtaining the eye keypoint circumscribing rectangle of the left eye / right eye includes: obtaining maximum and minimum coordinates along the x-axis and y-axis of the eye keypoint coordinates of the left eye / right eye, respectively; 7. A driver's gaze recognition method as described in claim 6, including a step of determining the difference between the maximum coordinate and the minimum coordinate along the x-axis direction as the length of the circumscribed rectangle, and determining the difference between the maximum coordinate and the minimum coordinate along the y-axis direction as the width of the circumscribed rectangle.

8. The step of acquiring a head pose of the target object based on the head image includes: obtaining Euler angles of the target object based on a deep learning network; The method for recognizing a driver's line of sight according to claim 1 , further comprising: determining the head posture when a difference between the Euler angles and a predetermined angle is smaller than a preset threshold value.

9. determining a gaze direction of the target subject based on the facial image, the head pose, and the relative position of the pupil and the sclera, performing feature extraction for the face image, the head pose, and the relative position of the pupil and sclera based on a multi-branch deep learning network; and connecting the facial features of the face image, the head features of the head pose, and the relative positions of the pupil and sclera to obtain the gaze direction of the target object.

10. A driver's line of sight recognition device, an image acquisition module for acquiring image data of a target object; a face detection module for processing the image data to obtain a facial image of the target subject; a head detection module for processing the image data to obtain a head image of the target subject; an eye detection module for acquiring an eye image of the target subject based on the face image; a head pose estimation module for obtaining a head pose of the target subject based on the head image; an image processing module for processing the eye image to obtain a relative position of the pupil and the sclera; a gaze estimation module for determining a gaze direction of the target object based on the facial image, the head pose, and the relative position of the pupil and sclera.

11. A computer-readable storage medium in which a driver's gaze recognition program is stored, and when the driver's gaze recognition program is executed by a processor, the driver's gaze recognition method according to any one of claims 1 to 9 is realized.

12. A vehicle comprising a memory, a processor, and a driver's gaze recognition program stored in the memory and operable by the processor, wherein the processor, when executing the driver's gaze recognition program, realizes the driver's gaze recognition method described in any one of claims 1 to 9.

Images