An image-based line of sight impact point estimation method and apparatus

Abstract

The application discloses an image-based line-of-sight landing point estimation method, which comprises the following steps: acquiring image data, performing face detection on the image data to obtain a face detection result, performing face posture detection based on the face detection result to obtain a posture detection result, wherein the posture detection result comprises a first posture meeting a gaze posture condition and a second posture meeting a non-gaze posture condition, acquiring eye region information from the face detection result corresponding to the posture detection result with the first posture, and performing line-of-sight landing point estimation based on the eye region information to obtain a line-of-sight landing point estimation result, wherein the face detection result corresponding to the posture detection result with the second posture is abandoned for line-of-sight landing point estimation. The application has fast response speed, low requirement for hardware resources and improved implementability.

Description

Technical Field

[0001] This invention relates to the field of image detection, and in particular, to a method for estimating the gaze point based on an image. Background Technology

[0002] Image-based gaze point estimation estimates the gaze point of both eyes in an image, i.e., it estimates the point where the binocular gaze focuses. Its typical scenario involves estimating the gaze point of both eyes on a two-dimensional plane. This two-dimensional plane can be a mobile phone screen, tablet screen, or television screen, etc.

[0003] Most existing image-based gaze point estimation methods train on the entire face to determine the gaze point location. This introduces irrelevant information, leading to biased judgments. It also makes the model more complex, harder to train, and more demanding on hardware, requiring large amounts of memory and high-performance CPUs. Some solutions even require specialized equipment to determine the gaze point location. This makes existing image-based gaze point estimation methods difficult to implement and limits their application in practice. Summary of the Invention

[0004] This invention provides an image-based method for estimating the gaze point, thereby improving the feasibility of gaze point estimation.

[0005] This invention provides an image-based method for estimating the gaze point, the method comprising:

[0006] Acquire image data,

[0007] Face detection is performed on the image data to obtain face detection results.

[0008] Based on the facial detection results, facial pose detection is performed to obtain pose detection results. These results include a first pose that meets the gaze pose criteria and a second pose that meets the non-gaze pose criteria. Eye region information is then obtained from the facial detection results corresponding to the pose detection results with the first pose.

[0009] Based on eye region information, the gaze point is estimated, and the gaze point estimation result is obtained.

[0010] in,

[0011] The face detection results corresponding to the pose detection results with the second pose are discarded for gaze point estimation.

[0012] Preferably, the facial pose detection based on the facial detection results includes:

[0013] Based on the facial detection results, facial pose angle information is obtained, which includes: pitch angle information to represent the face tilting up and down, tilt angle information to represent the face tilting left and right, and twist angle information to represent the face twisting forward and backward.

[0014] Based on pitch and twist angle information, identify the first and second attitudes;

[0015] The method further includes:

[0016] Based on the obtained tilt angle information, determine whether the tilt angle is greater than the set tilt angle threshold. If so, correct the face detection results.

[0017] Preferably, the step of identifying the first attitude and the second attitude based on the pitch angle information and the twist angle information includes:

[0018] Based on the acquired pitch angle information, determine whether the pitch angle is within the set pitch angle threshold; and based on the acquired twist angle information, determine whether the twist angle is within the set twist angle threshold.

[0019] If so, it is determined to be the first posture.

[0020] If the pitch angle is not within the set pitch angle threshold, or the twist angle is not within the set twist angle threshold, it is determined to be the second attitude;

[0021] The correction of the facial detection results includes: correcting the facial regions in the facial detection results to be horizontal in the left-right direction.

[0022] Preferably, obtaining facial pose angle information based on facial detection results includes:

[0023] The facial regions in the face detection results are cropped and normalized to obtain preprocessed facial image data.

[0024] The preprocessed facial image data is input into a facial landmark detection model, which is then used to detect landmarks and obtain their location information.

[0025] Based on the conversion relationship between key point location information and facial pose angles, facial pose angles are calculated to obtain pose angle information.

[0026] in,

[0027] The transformation relationship includes:

[0028] The pitch angle conversion relationship determined by the pitch angle coefficient, the first pitch angle parameter, and the second pitch angle parameter.

[0029] The torsion angle conversion relationship determined by the torsion angle coefficient, the first torsion angle parameter, and the second torsion angle parameter; the tilt angle conversion relationship determined by the position information of the key points of the left and right outer corners of the eyes.

[0030] The pitch angle coefficient and twist angle coefficient are determined by linear fitting between key point position information and facial posture angle;

[0031] The step of correcting the facial region in the facial detection results to be horizontal in the left-right direction includes:

[0032] Based on the key point information in the facial region that can characterize the left and right tilt angle of the face, the facial region in the facial detection result is corrected in the left and right direction to be parallel to the u direction in the image coordinate system.

[0033] Among them, key points that can characterize the angle of facial tilt include key points that are symmetrical from left to right.

[0034] Preferably, the step of performing face detection on the image data to obtain face detection results includes:

[0035] Face detection is performed on image data using a face detection model;

[0036] The step of obtaining eye region information from the face detection results corresponding to the pose detection results with the first pose includes:

[0037] Determine the interpupillary distance based on the location information of key points in both pupils.

[0038] Based on the interpupillary distance and a first ratio, a first size for cropping the eye area is determined.

[0039] Based on the interpupillary distance and the second ratio, a second size for cropping the eye area is determined.

[0040] The first proportion is greater than the second proportion.

[0041] Based on the location information of the key points of the pupils, the eye area is cropped from the face detection results using a rectangular area determined by the first and second dimensions.

[0042] Preferably, the facial pose detection based on the facial detection results further includes:

[0043] When the face detection results include multiple facial information, gaze point estimation is performed in single-frame mode.

[0044] In cases where the face detection results do not include facial information or only a single facial information, gaze point estimation is performed in multi-frame mode.

[0045] The process of estimating the gaze point based on eye region information to obtain the gaze point estimation result includes:

[0046] In single-frame mode, a gaze point estimation model is used to identify gaze in the eye region, and the location information of the gaze point is determined based on the gaze recognition results.

[0047] In multi-frame mode, gaze recognition is performed on the number of frames with eye regions that are input gaze point estimation models. When the number of frames of the input gaze point estimation model reaches a first threshold and the number of frames of the eye region image corresponding to the gaze recognition result reaches a set second threshold, the gaze point is determined to be within the target area.

[0048] Preferably, the gaze point estimation model includes: a first convolutional layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer, a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, an eighth feature extraction layer, a ninth feature extraction layer, a tenth feature extraction layer, a second convolutional layer, a pooling layer, a flattening layer, and a fully connected layer, connected in sequence.

[0049] in,

[0050] The outputs of the first feature extraction layer and the second feature extraction layer are first merged, and the result of the first merge is input to the third feature extraction layer. Furthermore, the result of the first merge is second merged with the output of the third feature extraction layer, and the result of the second merge is input to the fourth feature extraction layer.

[0051] The output of the fourth feature extraction layer is merged with the output of the fifth feature extraction layer for the third time. The result of the third merge is input to the sixth feature extraction layer. The result of the third merge is merged with the output of the sixth feature extraction layer for the fourth time. The result of the fourth merge is input to the seventh feature extraction layer. The result of the fourth merge is merged with the output of the seventh feature extraction layer for the fifth time. The result of the fifth merge is input to the eighth feature extraction layer.

[0052] The outputs of the eighth and ninth feature extraction layers are merged in a sixth process. The result of this sixth merge is then fed into the tenth feature extraction layer. Finally, the result of this sixth merge is merged in a seventh process with the output of the tenth feature extraction layer. The result of this seventh merge is then fed into the second convolutional layer.

[0053] The output of the second convolutional layer is processed sequentially by a pooling layer, a flattening layer, and a fully connected layer to obtain the output of the line-of-sight estimation model.

[0054] Preferably, each feature extraction layer comprises n convolutional layers, each convolutional layer having a kernel of less than or equal to 3, where n is a natural number less than or equal to 3;

[0055] The face detection model, face key point detection model, and gaze point estimation model are trained models, and the sample set used for training includes sample image data of people wearing glasses.

[0056] This application also provides an image-based gaze point estimation device, which includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the steps of the computer program to implement any of the gaze point estimation methods.

[0057] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the aforementioned line-of-sight estimation methods.

[0058] This application provides an image-based gaze point estimation method that, through facial pose detection, can filter out poses that do not meet the gaze pose conditions, reducing unnecessary resource consumption and false alarms in gaze point estimation. It uses only eye region information for gaze point estimation, which can fully extract detailed information of the eye region, has a fast response speed, and low requirements for hardware resources. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating an image-based line-of-sight estimation method according to an embodiment of this application.

[0060] Figure 2 This is a flowchart illustrating an image-based line-of-sight estimation method according to an embodiment of this application.

[0061] Figure 3 This is a schematic diagram used to describe the pitch angle, tilt angle, and twist angle of a human face.

[0062] Figure 4 A schematic diagram for defining the eye area.

[0063] Figure 5 This is a schematic diagram of the line-of-sight landing point estimation model structure in an embodiment of this application.

[0064] Figure 6 This is a schematic diagram of a line-of-sight point estimation device according to an embodiment of this application.

[0065] Figure 7 This is another schematic diagram of the line-of-sight point estimation device according to an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0067] This application uses image frames captured by a regular RGB camera to obtain facial region coordinate information through face detection, filters out facial detection results that do not conform to the gaze posture through facial pose detection, obtains eye region information from the facial regions that conform to the gaze posture, and estimates the gaze point based on the eye region information.

[0068] See Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating an image-based gaze point estimation method according to an embodiment of this application. The method includes:

[0069] Step 101: Obtain facial image data.

[0070] Step 102: Perform face detection on the image data to obtain the face detection results.

[0071] Step 103: Based on the face detection results, perform facial pose detection to obtain pose detection results. These results include: a first pose that meets the gaze pose condition, and a second pose that meets the non-gaze pose condition.

[0072] Step 104: Obtain eye region information from the face detection results corresponding to the pose detection results with the first pose.

[0073] Step 105: Based on the eye region information, estimate the gaze point to obtain the gaze point estimation result.

[0074] in,

[0075] The face detection results corresponding to the pose detection results with the second pose are discarded for gaze point estimation.

[0076] This application embodiment requires no additional hardware support. Based on images captured by a regular RGB camera, it can quickly and accurately determine the gaze point, providing a non-contact gaze point estimation method for rapid response to gaze states. When gaze point determination is applied to hardware devices, it can trigger device activation and response through gaze, enriching the interaction methods of some smart companion products and effectively improving the user experience. Simultaneously, it can also serve as a pre-processing algorithm for detection systems such as facial recognition, avoiding facial recognition under non-gaze conditions and reducing the resource consumption of facial recognition systems.

[0077] To facilitate understanding of the embodiments of this application, the following uses face-based image data to illustrate the gaze point estimation method.

[0078] In the following embodiments, the image frame obtains the coordinates of the face region and five key points via a face detection module, including the center of the left pupil, the center of the right pupil, the tip of the nose, the left corner of the mouth, and the right corner of the mouth. The interpupillary distance is calculated based on the coordinates of the left and right pupil centers, and eye region information including both eyes is cropped based on the interpupillary distance. Before determining the gaze point, face pose detection and correction are performed. Only image frames conforming to a gaze pose are cropped for the eye region; image frames not conforming to a gaze pose are determined to be non-gaze poses. Based on image frames with a gaze pose, the gaze point is estimated by combining the eye region information, while ignoring the gaze point estimation for image frames with non-gaze poses.

[0079] See Figure 2 As shown, Figure 2 This is a schematic flowchart illustrating an image-based gaze point estimation method according to an embodiment of this application. The method includes:

[0080] Step 201: Obtain face image data;

[0081] Step 202: Perform face detection on the image data;

[0082] As an example, the steps specifically include:

[0083] Step 2021: Perform size normalization, boundary filling and other preprocessing on the image frames in the acquired image data, and input the processed image data into the face detection model to obtain the output result of the face detection model, which includes the detection bounding box of the detected face;

[0084] As an example, face detection models can be implemented based on the RetinaFace network framework, which integrates a Feature Pyramid Network (FPN) and a context detection module, such as the SSH (Single Stage Headless) network. The introduction of FPN allows the model to detect face images of different sizes, better adapting to complex application scenarios and enhancing the robustness of the algorithm. The SSH network can expand the contextual information of the pre-detection region, making the final output of the face detection model more accurate.

[0085] To prevent redundant calculations, overlapping detection bounding boxes are removed to avoid resource consumption caused by multiple detections of the same target. The output of the face detection model is processed as follows:

[0086] Step 2022 involves calculating the Intersection over Union (IOU) of the detected bounding boxes and performing Non-maximum Suppression (NMS) processing; among which,

[0087] The intersection-union ratio is the area of ​​the intersection of two detection bounding boxes divided by the total area occupied by the merger of the two detection bounding boxes;

[0088] Non-maximum suppression is performed as follows: First, sort all detected bounding boxes by area, select the largest area detected bounding box, and then traverse the remaining detected bounding boxes. If the Intersection over Union (IOU) between the currently traversed detected bounding box and the current largest area detected bounding box is greater than the set threshold, then delete the currently traversed detected bounding box. Then, continue to select the next largest area detected bounding box from the untraversed detected bounding boxes, and repeat the above process.

[0089] Step 203: Determine whether the output of the face detection model contains multiple faces.

[0090] If so, then enter the single-frame discrimination mode, i.e., steps 204 to 207;

[0091] Otherwise, proceed to the multi-frame discrimination mode, i.e., steps 210 to 216;

[0092] Step 204: Determine the location information of the target bounding box.

[0093] As an example, the location information of the target bounding box is obtained from the output of the face detection model after step 2022.

[0094] Furthermore, the face detection model can also obtain key point location information, including: the center of the left pupil, the center of the right pupil, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0095] Step 205: Based on the face detection results, perform face pose detection to obtain the pose detection results.

[0096] As an example, the detection process includes:

[0097] Step 2051: Based on the position information of the target bounding box, crop the target face image from the image frame;

[0098] Step 2052: Perform preprocessing operations on the acquired target face image, including image size normalization, data standardization and other operations, in order to optimize the image data;

[0099] Step 2053: The preprocessed target face image is processed using a facial landmark detection model to detect facial landmarks. Based on a linear transformation model between facial landmarks and face angles, the face pose is calculated to obtain pitch angle information (representing vertical tilt), roll angle information (representing horizontal tilt), and twist angle information (representing forward / backward torsion), denoted as yaw, roll, and pitch, respectively. See also... Figure 3 As shown, Figure 3This is a schematic diagram used to describe the pitch angle, tilt angle, and twist angle of a human face.

[0100] The facial landmark detection model is based on the MobileNetV3 framework. The input data to the facial landmark detection model is optimized according to the characteristics of the image frame. In this way, facial landmark information can be obtained based on a lightweight facial landmark detection model. For example, 106 landmarks can be obtained, including the center of the left pupil, the center of the right pupil, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0101] After obtaining facial landmark information, a linear model was fitted using the facial angle and landmark information. Based on this linear model, the angle information of the three dimensions of facial pose (yaw, roll, and pitch) can be directly obtained.

[0102] Taking 106 facial landmark detections as an example, based on the mathematical transformation model between landmarks and facial angles, the facial pose angles are calculated to obtain three angles: yaw, pitch, and roll. The transformation relationship is expressed mathematically as follows:

[0103] yaw = yaw k *71.58°+0.7037° (1)

[0104] pitch = pitch k *1.497°+18.97° (2)

[0105] roll = arctan((P) 35y -P 93y ) / (P 35x -P 93x ))*180° / π (3)

[0106] in,

[0107] Formula 1 is derived from the torsion angle coefficient yaw k The torsion angle conversion relationship determined by the first torsion angle parameter (e.g., 71.58°) and the second torsion angle parameter (e.g., 0.7037°)

[0108] Formula 2 is derived from the pitch coefficient. k The pitch angle conversion relationship determined by the first pitch angle parameter (e.g., 1.497°) and the second pitch angle parameter (e.g., 18.97°).

[0109] Formula 3 represents the tilt angle conversion relationship determined by the positional information of the key points at the left and right outer corners of the eye.

[0110] The yaw k pitchk P was determined by a linear fitting algorithm. 35x P 93x The x-coordinates of the left and right outer corners of the eyes are respectively, P 35y P 93y These are the y-coordinates of the left and right outer corners of the eyes, respectively.

[0111] Step 2054: Identify whether the target face is in the first pose that meets the gaze pose condition based on the preset pose threshold, so as to effectively filter out some target faces that are obviously not in the gaze pose, thereby reducing some useless calculations and reducing resource consumption.

[0112] As an example, a first attitude and a second attitude that meets the conditions for a non-gazing attitude are identified based on pitch angle information and twist angle information; for example, it is determined whether the pitch angle is within a set pitch angle threshold based on the pitch angle information, and whether the twist angle is within a set twist angle threshold based on the twist angle information.

[0113] If so, it is determined to be the first posture.

[0114] If the pitch angle is not within the set pitch angle threshold, or the twist angle is not within the set twist angle threshold, it is determined to be the second attitude.

[0115] In this way, target faces that are looking up, looking down, turning their heads, or a combination thereof can be eliminated without estimating the point of view.

[0116] Step 2055: For target faces whose pose is the first pose, pose correction is performed to provide a better data basis for visual landing point estimation, thereby improving the accuracy of visual landing point estimation.

[0117] As an example, based on the obtained tilt angle information, it is determined whether the tilt angle is greater than the set tilt angle threshold. If so, the face detection results are corrected.

[0118] During correction, the tilt angle (roll angle) can be corrected based on the detected facial key point position information that can characterize the left-right tilt angle of the face. This ensures that the corrected target face image remains parallel to the u-direction in the image coordinate system; for example, the left and right eyes can remain on the same horizontal line. The facial key points that can characterize the left-right tilt angle are key points with left-right symmetry, including but not limited to: at least one of the left and right eye key points, left and right corner of the mouth key points, and left and right nostril key points.

[0119] After correcting the pose of the target face, the position information of the target face and the position information of the facial key points are updated, laying a good foundation for subsequent gaze point estimation.

[0120] Step 206: Obtain eye region information from the face detection results corresponding to the pose detection results with the first pose.

[0121] As an example, based on the corrected target face image and the recalculated pupil center position information, the eye area is cropped according to the pupil distance and a relative ratio is set.

[0122] For example, based on the interpupillary distance and a first ratio, a first size for cropping the eye region is determined, and based on the interpupillary distance and a second ratio, a second size for cropping the eye region is determined, wherein the first ratio is greater than the second ratio. Thus, the first size is the length along the line connecting the two eyes, and the second size is the height in the direction perpendicular to the line connecting the two eyes. Based on the positional information of the pupils, the eye region image is cropped from the target face image using the rectangular area determined by the first and second sizes. The positional information of the eye region can be expressed mathematically as follows:

[0123] L=x±az

[0124] H=y±bz

[0125] Where L is the first dimension, H is the second dimension, x and y are the coordinates of the center of the line connecting the two pupils, a is the first scale, b is the second scale, and z is the interpupillary distance.

[0126] See Figure 4 As shown, Figure 4 A schematic diagram for defining the eye area.

[0127] Step 207: Based on the eye region information, estimate the gaze point to obtain the gaze point estimation result.

[0128] As an example, an image of the eye region is input into a gaze point estimation model to obtain the user's gaze point estimation result. This result can be a judgment result on whether the gaze falls within the target area. The gaze point estimation model can be a lightweight residual network model based on the MobileNetV2 framework. In the network framework design, it can be based on the MobileNetV2 framework and improved to treat gaze point estimation as a binary classification task.

[0129] Since the input is eye region information and the image size is small, the original MobileNetV2 model's network is too deep. Therefore, this application simplifies the MobileNetV2 framework and readjusts the network architecture. In terms of convolution kernel size design, a small convolution kernel size is still used, which can better extract detailed information of the eye region, while effectively reducing the amount of computation and parameters, making the final model more lightweight. The design of the residual network structure can make the model converge faster.

[0130] See Figure 5 As shown, Figure 5 This is a schematic diagram of the line-of-sight landing point estimation model structure according to an embodiment of this application. The model includes a first convolutional layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer, a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, an eighth feature extraction layer, a ninth feature extraction layer, a tenth feature extraction layer, a second convolutional layer, a pooling layer, a flattening layer, and a fully connected layer, all connected in sequence. Each feature extraction layer includes n convolutional layers, for example, n equals 3. Preferably, n is a natural number less than or equal to 3. The convolutional kernel of each convolutional layer is less than or equal to 3.

[0131] The outputs of the first feature extraction layer and the second feature extraction layer are first merged, and the result of the first merge is input to the third feature extraction layer. Furthermore, the result of the first merge is second merged with the output of the third feature extraction layer, and the result of the second merge is input to the fourth feature extraction layer.

[0132] The output of the fourth feature extraction layer is merged with the output of the fifth feature extraction layer for the third time. The result of the third merge is input to the sixth feature extraction layer. The result of the third merge is merged with the output of the sixth feature extraction layer for the fourth time. The result of the fourth merge is input to the seventh feature extraction layer. The result of the fourth merge is merged with the output of the seventh feature extraction layer for the fifth time. The result of the fifth merge is input to the eighth feature extraction layer.

[0133] The outputs of the eighth and ninth feature extraction layers are merged in a sixth process. The result of this sixth merge is then fed into the tenth feature extraction layer. Finally, the result of this sixth merge is merged in a seventh process with the output of the tenth feature extraction layer. The result of this seventh merge is then fed into the second convolutional layer.

[0134] The output of the second convolutional layer is processed sequentially through a pooling layer, a flattening layer, and a fully connected layer to obtain the model's output. This output represents the floating-point value of each pixel, which includes two different floating-point values. The location information of the gaze point can be obtained based on the distribution of the output.

[0135] Compared to some existing methods that use the entire image as input to the gaze point estimation model for judgment, this embodiment inputs the eye region as a whole image without splitting it into the gaze point estimation model. This allows the gaze point estimation model to focus more on the extraction of eye region features, reducing interference from irrelevant information. At the same time, the reduction in input size results in lower resource consumption and better real-time performance.

[0136] It should be understood that the structure of the line-of-sight point estimation model is not limited to this, and variations and adjustments based on this structure are also possible.

[0137] The following steps are for multi-frame discrimination mode:

[0138] Step 210: Determine whether there are human face images in the multi-frame image data.

[0139] If so, proceed to steps 211-213, which are the same as steps 204-206, and then proceed to step 214.

[0140] Otherwise, proceed directly to step 214;

[0141] Step 214: Determine whether the number of frames with eye region in the pose detection result with first pose reaches the set first quantity threshold. If yes, proceed to step 215; otherwise, determine it as an intermediate frame and end the process.

[0142] Step 215: Based on the eye region information of each frame, estimate the gaze point to obtain the gaze point estimation result. This step is the same as step 207.

[0143] Step 216: Determine whether the number of frames of the eye region image corresponding to the gaze recognition result has reached the set second quantity threshold. If so, determine the position information of the gaze point based on the gaze recognition result of each frame of the eye region. For example, determine the position information of the gaze point by fusing the gaze recognition results of each frame of the eye region, or determine that the gaze point is within the target area; otherwise, end this process.

[0144] In this embodiment, the face detection model, the face key point detection model, and the gaze point estimation model are trained models. The sample set used for training includes sample image data of people wearing glasses. In this way, this embodiment can also perform gaze point estimation when people are wearing glasses, avoiding false alarms when users are wearing glasses and improving robustness.

[0145] This embodiment detects the gaze point by performing face detection, face pose detection and correction, and gaze point estimation. When the user is looking at the device, the smart device can respond quickly, enriching the ways in which the user interacts with the device and improving the user experience.

[0146] By using a lightweight facial landmark detection model and based on a linear transformation model between landmarks and pose, the model obtains three-dimensional angular information of facial pose based on the landmarks. This allows for the rapid filtering out of some non-focused situations, reducing unnecessary resource consumption and effectively decreasing false alarms in such cases.

[0147] The center coordinates of the left and right pupils are located by face detection, and the roll angle is corrected for image frames that meet the requirements based on the pose detection results. Then, the eye area information is obtained by cropping according to the ratio of the distance between the two pupils, which lays a good foundation for subsequent estimation of the gaze point.

[0148] A lightweight gaze-point estimation model based on a residual network structure is used, and only the information of the eye region is input into the model for training. In terms of network structure, a small convolutional kernel size is used to extract the information of the eye region, which can fully extract the detailed information of the eye region and make a correct judgment on whether the user's gaze falls on the target area based on the extracted feature information.

[0149] It offers two selection modes: multi-frame detection and single-frame detection. Users can switch between modes according to different application scenarios to meet the requirements of different indicators in different scenarios.

[0150] This embodiment can be implemented entirely using deep learning algorithms. Compared with traditional methods, the algorithm has stronger robustness and generalization ability, and uses lightweight models, thus making the algorithm more applicable.

[0151] See Figure 6 As shown, Figure 6 This is a schematic diagram of a line-of-sight point estimation device according to an embodiment of this application. The device includes:

[0152] The image acquisition module is used to acquire image data.

[0153] The face detection module is used to perform face detection on the image data and obtain face detection results.

[0154] The facial pose detection module is used to perform facial pose detection based on the facial detection results, and obtain pose detection results. These results include a first pose that meets the gaze pose condition, and a second pose that meets the non-gaze pose condition.

[0155] The gaze point estimation module is used to obtain eye region information from the face detection results corresponding to the pose detection results with the first pose, and to estimate the gaze point based on the eye region information to obtain the gaze point estimation result.

[0156] The device also includes a posture correction module, which determines whether the tilt angle is greater than a set tilt angle threshold based on the tilt angle information obtained by the facial posture detection module. If so, the facial detection results are corrected.

[0157] The device also includes a mode selection module, which is used to estimate the gaze point in a single-frame mode when the face detection result includes multiple facial information, and to estimate the gaze point in a multi-frame mode when the face detection result does not include facial information or includes a single facial information.

[0158] The facial pose detection module is configured to obtain facial pose angle information based on the facial detection results. The facial pose angle information includes: pitch angle information to represent the face tilting in the vertical direction, tilt angle information to represent the face tilting in the horizontal direction, and torsion angle information to represent the face twisting in the front-back direction. Based on the pitch angle information and the torsion angle information, a first pose and a second pose are identified.

[0159] The facial posture detection module is further configured to determine whether the pitch angle is within a set pitch angle threshold based on the acquired pitch angle information, and to determine whether the torsion angle is within a set torsion angle threshold based on the acquired torsion angle information. If it is, it is determined to be a first posture; if the pitch angle is not within the set pitch angle threshold or the torsion angle is not within the set torsion angle threshold, it is determined to be a second posture.

[0160] The facial pose detection module is further configured to crop and normalize the facial region in the facial detection results to obtain preprocessed facial image data. The preprocessed facial image data is then input into the facial key point detection model, which performs key point detection to obtain key point location information. Based on the conversion relationship between key point location information and facial pose angle, facial pose angle is calculated to obtain pose angle information.

[0161] The gaze point estimation module is configured to determine the interpupillary distance based on the location information of key points in both pupils.

[0162] Based on the interpupillary distance and a first ratio, a first size for cropping the eye area is determined.

[0163] Based on the interpupillary distance and the second ratio, a second size for cropping the eye area is determined.

[0164] Based on the location information of the key points of the two pupils, the eye area is cropped from the face detection results using a rectangular area defined by the first and second dimensions.

[0165] The face detection module is configured to perform face detection on image data using a face detection model.

[0166] The gaze point estimation module is configured to perform gaze recognition on the eye region using the gaze point estimation model in single-frame mode, and determine the position information of the gaze point based on the gaze recognition result.

[0167] In multi-frame mode, gaze recognition is performed on the number of frames with eye regions that are input gaze point estimation models. When the number of frames of the input gaze point estimation model reaches a first threshold and the number of frames of the eye region image corresponding to the gaze recognition result reaches a set second threshold, the gaze point is determined to be within the target area.

[0168] The posture correction module is configured to correct the facial region in the face detection result to be parallel to the u direction in the image coordinate system in the left-right direction, based on key point information in the facial region that can characterize the left-right tilt angle of the face.

[0169] See Figure 7 As shown, Figure 7 This is another schematic diagram of a line-of-sight estimation device according to an embodiment of this application. The device includes: an image acquisition device, a memory, and a processor. The memory stores a computer program, and the processor is configured to execute the steps of the computer program to implement any of the line-of-sight estimation methods.

[0170] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0171] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0172] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the aforementioned line-of-sight estimation methods.

[0173] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0174] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating the gaze point based on an image, characterized in that, The method includes: Acquire image data, Face detection is performed on the image data to obtain face detection results. Based on the facial detection results, facial pose detection is performed to obtain pose detection results, which include: a first pose that meets the gaze pose condition, and a second pose that meets the non-gaze pose condition. Eye region information is obtained from the face detection results corresponding to the pose detection results with the first pose. The face detection results corresponding to the pose detection results with the second pose are discarded for gaze point estimation. Based on eye region information, the gaze point is estimated, and the gaze point estimation result is obtained. in, The line-of-sight point estimation is achieved in the following way: An image of the eye region is input into a gaze point estimation model for gaze recognition. Based on the gaze recognition results, the location information of the gaze point is determined. The gaze point estimation model includes: a first convolutional layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer, a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, an eighth feature extraction layer, a ninth feature extraction layer, a tenth feature extraction layer, a second convolutional layer, a pooling layer, a flattening layer, and a fully connected layer, all connected in sequence. in, The outputs of the first feature extraction layer and the second feature extraction layer are first merged, and the result of the first merge is input to the third feature extraction layer. Furthermore, the result of the first merge is second merged with the output of the third feature extraction layer, and the result of the second merge is input to the fourth feature extraction layer. The output of the fourth feature extraction layer is merged with the output of the fifth feature extraction layer for the third time. The result of the third merge is input to the sixth feature extraction layer. The result of the third merge is merged with the output of the sixth feature extraction layer for the fourth time. The result of the fourth merge is input to the seventh feature extraction layer. The result of the fourth merge is merged with the output of the seventh feature extraction layer for the fifth time. The result of the fifth merge is input to the eighth feature extraction layer. The outputs of the eighth and ninth feature extraction layers are merged in a sixth process. The result of this sixth merge is then fed into the tenth feature extraction layer. Finally, the result of this sixth merge is merged in a seventh process with the output of the tenth feature extraction layer. The result of this seventh merge is then fed into the second convolutional layer. The output of the second convolutional layer is processed sequentially through a pooling layer, a flattening layer, and a fully connected layer to obtain the output of the line-of-sight estimation model.

2. The line-of-sight estimation method as described in claim 1, characterized in that, The facial pose detection based on the facial detection results includes: Based on the facial detection results, facial pose angle information is obtained, which includes: pitch angle information to represent the face tilting up and down, tilt angle information to represent the face tilting left and right, and twist angle information to represent the face twisting forward and backward. Based on pitch angle and twist angle information, identify the first attitude and the second attitude; The method further includes: Based on the obtained tilt angle information, determine whether the tilt angle is greater than the set tilt angle threshold. If so, correct the face detection results.

3. The line-of-sight estimation method as described in claim 2, characterized in that, The step of identifying the first attitude and the second attitude based on pitch angle information and twist angle information includes: Based on the acquired pitch angle information, determine whether the pitch angle is within the set pitch angle threshold; and based on the acquired twist angle information, determine whether the twist angle is within the set twist angle threshold. If so, it is determined to be the first posture. If the pitch angle is not within the set pitch angle threshold, or the twist angle is not within the set twist angle threshold, it is determined to be the second attitude; The correction of the facial detection results includes: correcting the facial regions in the facial detection results to be horizontal in the left-right direction.

4. The line-of-sight estimation method as described in claim 3, characterized in that, The process of obtaining facial pose angle information based on facial detection results includes: The facial regions in the face detection results are cropped and normalized to obtain preprocessed facial image data. The preprocessed facial image data is input into a facial landmark detection model, which is then used to detect landmarks and obtain their location information. Based on the conversion relationship between key point location information and facial pose angles, facial pose angles are calculated to obtain pose angle information. in, The transformation relationship includes: The pitch angle conversion relationship determined by the pitch angle coefficient, the first pitch angle parameter, and the second pitch angle parameter. The torsion angle conversion relationship determined by the torsion angle coefficient, the first torsion angle parameter, and the second torsion angle parameter. The tilt angle conversion relationship determined by the position information of the key points of the left and right outer corners of the eyes; The pitch angle coefficient and twist angle coefficient are determined by linear fitting between key point position information and facial posture angle; The step of correcting the facial region in the facial detection results to be horizontal in the left-right direction includes: Based on the key point information in the facial region that can characterize the left and right tilt angle of the face, the facial region in the facial detection result is corrected in the left and right direction to be parallel to the u direction in the image coordinate system. Among them, key points that can characterize the angle of facial tilt include key points that are symmetrical from left to right.

5. The line-of-sight estimation method as described in claim 4, characterized in that, The step of performing face detection on the image data to obtain face detection results includes: Face detection is performed on image data using a face detection model; The step of obtaining eye region information from the face detection results corresponding to the pose detection results with the first pose includes: Determine the interpupillary distance based on the location information of key points in both pupils. Based on the interpupillary distance and a first ratio, a first size for cropping the eye area is determined. Based on the interpupillary distance and the second ratio, a second size for cropping the eye area is determined. The first proportion is greater than the second proportion. Based on the location information of the key points of the pupils, the eye area is cropped from the face detection results using a rectangular area determined by the first and second dimensions.

6. The line-of-sight estimation method as described in claim 5, characterized in that, The facial pose detection based on the facial detection results further includes: When the face detection results include multiple facial information, gaze point estimation is performed in single-frame mode. In cases where the face detection results do not include facial information or only a single facial information, gaze point estimation is performed in multi-frame mode. The process of estimating the gaze point based on eye region information to obtain the gaze point estimation result includes: In single-frame mode, a gaze point estimation model is used to identify gaze in the eye region, and the location information of the gaze point is determined based on the gaze recognition results. In multi-frame mode, gaze recognition is performed on the number of frames with eye regions that are input gaze point estimation models. When the number of frames of the input gaze point estimation model reaches a first threshold and the number of frames of the eye region image corresponding to the gaze recognition result reaches a set second threshold, the gaze point is determined to be within the target area.

7. The line-of-sight estimation method as described in claim 6, characterized in that, Each feature extraction layer comprises n convolutional layers, and the kernel of each convolutional layer is less than or equal to 3, where n is a natural number less than or equal to 3; The face detection model, face key point detection model, and gaze point estimation model are trained models, and the sample set used for training includes sample image data of people wearing glasses.

8. An image-based gaze point estimation device, characterized in that, The device includes a memory and a processor, the memory storing a computer program and the processor being configured to execute the computer program to implement the steps of the line-of-sight estimation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the line-of-sight estimation method as described in any one of claims 1 to 7.

Images