Gaze tracking method, apparatus and system

Abstract

The present disclosure provides a gaze tracking method, device and system, belonging to the technical field of image processing. The gaze tracking method comprises: collecting multiple frames of face images of a viewer when watching a display screen by using at least two cameras; segmenting left and right eye images from the face image of the current frame by using a pre-trained eye detection model; calculating the similarity between the face image of the current frame and each face image in the previous N frames of face images; if the similarity between the face image of the current frame and each face image in the previous N frames of face images is not greater than a preset threshold, detecting the gaze landing position of the viewer on the display screen by using a pre-trained gaze recognition model. The embodiments of the present disclosure can improve the accuracy and efficiency of gaze tracking.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a gaze tracking method, apparatus and system. Background Technology

[0002] Human-computer interaction (HCI) is a technology that studies the interaction between humans and machines. HCI technology enables communication between humans and machines through all possible information channels. With the rapid development of electronic technology, HCI technology is widely used in the interaction between humans and display devices, and eye-tracking-based HCI methods have received widespread attention. Eye-tracking is a technology that uses various detection methods, such as mechanical, electronic, and optical methods, to obtain the user's current "gaze direction."

[0003] Existing eye-tracking technologies have low accuracy and speed in determining the location of the screen where the human eye is looking. Summary of the Invention

[0004] The technical problem to be solved by this disclosure is to provide an eye-tracking method, apparatus and system that can improve the accuracy and efficiency of eye-tracking.

[0005] To address the aforementioned technical problems, the embodiments of this disclosure provide the following technical solutions:

[0006] On the one hand, an eye-tracking method is provided, including:

[0007] Use at least two cameras to capture multiple frames of facial images of the viewer as they view the display screen;

[0008] The left and right eye images are segmented from the face image in the current frame using a pre-trained human eye detection model, wherein the human eye detection model employs a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image;

[0009] Calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, the predicted position of the viewer's gaze on the facial image n is used as the predicted position of the viewer's gaze on the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than the preset threshold, a pre-trained gaze recognition model is used to detect the viewer's gaze position on the display screen. The gaze recognition model uses a convolutional neural network, with the input being the left and right eye images segmented from the facial image and the predicted positions of the viewer's gaze on the previous N frames' facial images, and the output being the predicted position of the viewer's gaze on the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N.

[0010] In some embodiments, the method further includes the step of training a human eye detection model, wherein training the human eye detection model includes:

[0011] Establish a first training dataset, which includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different.

[0012] An initial model for human eye detection is established, which consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image.

[0013] The initial human eye detection model is trained using the multiple sets of first training data, and the model after training K times is saved as the human eye detection model, where K is a positive integer.

[0014] In some embodiments, the downsampling network includes M downsampling layers, and the upsampling network includes M upsampling layers, where M is an integer greater than 1, and the downsampling layers correspond one-to-one with the upsampling layers;

[0015] The input to the first downsampling layer is a facial image. The i-th downsampling layer performs convolution operations on the input features using different convolution kernels, concatenates the calculation results, and passes them to the (i+1)-th downsampling layer. The input to the i-th upsampling layer is the concatenation of the output of the (i+1)-th upsampling layer and the first result, where the first result is the result obtained by convolution of the output of the i-th downsampling layer, and i is greater than or equal to 1 and less than or equal to M-1.

[0016] The Mth downsampling layer performs a convolution operation on the input features and then passes them to the Mth upsampling layer. The output of the Mth upsampling layer is the eye contour in the facial image.

[0017] In some embodiments, the upsampling layer employs interpolation.

[0018] In some embodiments, the method further includes the step of training a gaze recognition model, wherein training the gaze recognition model includes:

[0019] A second training dataset is established, which includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the gaze in the current frame facial image. The facial images in different sets of second training data are different.

[0020] An initial model for gaze recognition is established. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image.

[0021] The initial model for gaze recognition is trained using the multiple sets of second training data, and the model after training L times is saved as the gaze recognition model, where L and A are positive integers.

[0022] In some embodiments, before training the initial gaze recognition model using the multiple sets of second training data, the method further includes preprocessing the second training data, including:

[0023] The distance between the viewer and the display screen is determined based on the facial image;

[0024] The sizes of the facial image and the left and right eye images are adjusted according to the distance.

[0025] In some embodiments, obtaining the second training data includes:

[0026] Display multiple test points on the screen;

[0027] At least two cameras are used to capture facial images of a viewer looking at the test point in front of a display screen, the test point being the location where the viewer's line of sight falls.

[0028] In some embodiments, the gaze recognition model sequentially includes a first layer Dense module, a first layer transition layer, a second layer Dense module, a second layer transition layer, a third layer Dense module, a third layer transition layer, and a fully connected layer. The Dense module includes multiple sequentially connected 1*1 convolutional layers and 3*3 convolutional layers. The outputs of the (2j-1)th and (2j)th convolutional layers are used to calculate residuals and then input into the (2j+5)th convolutional layer, where j is a positive integer.

[0029] This disclosure also provides an eye-tracking device, including:

[0030] An image acquisition module is used to capture multiple frames of facial images of a viewer while viewing a display screen using at least two cameras;

[0031] The human eye detection module is used to segment the left and right eye images from the face image in the current frame using a pre-trained human eye detection model, wherein the human eye detection model adopts a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image;

[0032] The gaze recognition module is used to calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, the predicted gaze position of the facial image n is used as the predicted gaze position of the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than the preset threshold, a pre-trained gaze recognition model is used to detect the viewer's gaze position on the display screen. The gaze recognition model uses a convolutional neural network, with the input being the left and right eye images segmented from the facial image and the predicted gaze positions of the previous N frames' facial images, and the output being the predicted gaze position of the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N.

[0033] In some embodiments, a human eye detection model training module is also included for training a human eye detection model, the human eye detection model training module comprising:

[0034] The first submodule is used to establish the first training dataset, which includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different.

[0035] The second submodule is used to establish an initial model for human eye detection. The initial model for human eye detection consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image.

[0036] The third submodule is used to train the initial human eye detection model using the multiple sets of first training data, and save the model after training K times as the human eye detection model, where K is a positive integer.

[0037] In some embodiments, a gaze recognition model training module is further included for training a gaze recognition model, the gaze recognition model training module comprising:

[0038] The fourth submodule is used to establish a second training dataset. The second training dataset includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the gaze in the current frame facial image. The facial images in different sets of second training data are different.

[0039] The fifth submodule is used to establish an initial model for gaze recognition. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image.

[0040] The sixth submodule is used to train the initial gaze recognition model using the multiple sets of second training data, and save the model after training L times as the gaze recognition model, where L and A are positive integers.

[0041] This disclosure also provides an eye-tracking system, including a display screen, at least two cameras, and an eye-tracking device as described above.

[0042] This disclosure also provides an eye-tracking device, which includes a memory and an actuator, the actuator being used to perform the method described above.

[0043] This disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described above.

[0044] The embodiments disclosed herein have the following beneficial effects:

[0045] In the above scheme, at least two cameras are used to capture multiple frames of facial images of the viewer when viewing the display screen, and a pre-trained human eye detection model and gaze recognition model are used to determine the position of the viewer's gaze on the display screen. The gaze tracking speed is fast. Furthermore, through deep learning, the motion features of the face and environmental influences can be input into the model, which can improve the accuracy of gaze tracking. Attached Figure Description

[0046] Figure 1 This is a schematic flowchart of the gaze tracking method according to an embodiment of the present disclosure;

[0047] Figure 2 This is a schematic diagram of the structure of the human eye detection model according to an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram illustrating the calculation of the distance between a viewer and a display screen according to an embodiment of this disclosure;

[0049] Figure 4 This is a schematic diagram of the structure of the gaze recognition model according to an embodiment of the present disclosure;

[0050] Figure 5 This is a schematic diagram of the structure of the Dense module according to an embodiment of this disclosure;

[0051] Figure 6This is a schematic diagram illustrating the calculation of the similarity between a facial image in the current frame and a facial image n in the previous N frames, according to an embodiment of this disclosure.

[0052] Figure 7 This is a schematic diagram of the structure of the gaze tracking device according to an embodiment of this disclosure. Detailed Implementation

[0053] To make the technical problems, technical solutions and advantages to be solved by the embodiments of this disclosure clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0054] This disclosure provides an eye-tracking method, apparatus, and system that can improve the accuracy and efficiency of eye-tracking.

[0055] Embodiments of this disclosure provide an eye-tracking method, such as... Figure 1 As shown, it includes:

[0056] Step 101: Use at least two cameras to capture multiple frames of facial images of the viewer as they view the display screen;

[0057] Step 102: Use a pre-trained human eye detection model to segment the left and right eye images from the face image in the current frame, wherein the human eye detection model adopts a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image;

[0058] Step 103: Calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, use the predicted position of the viewer's gaze on the facial image n as the predicted position of the viewer's gaze on the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than a preset threshold, use a pre-trained gaze recognition model to detect the viewer's gaze position on the display screen. The gaze recognition model uses a convolutional neural network. The input is the left and right eye images segmented from the facial image and the predicted position of the viewer's gaze on the previous N frames' facial images. The output is the predicted position of the viewer's gaze on the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N.

[0059] Among them, two of the at least two cameras can be two cameras of the same binocular camera.

[0060] In this embodiment, at least two cameras are used to capture multiple frames of facial images of the viewer when viewing the display screen. A pre-trained human eye detection model and a gaze recognition model are used to determine the position of the viewer's gaze on the display screen, resulting in fast gaze tracking. Furthermore, deep learning can incorporate facial motion features and environmental influences into the model, thereby improving the accuracy of gaze tracking.

[0061] In addition, this embodiment adds a judgment step before inputting data into the gaze recognition model to determine whether the facial image of the current frame is highly similar to the facial images of the previous N frames (similarity greater than a preset threshold, which can be fitted based on other data, such as 98%). If the facial image of the current frame is highly similar to the facial images of the previous N frames, it can be assumed that the viewer's gaze has not changed significantly, and the predicted position of the gaze point in facial image n is directly output as the predicted position of the gaze point in the current frame. This reduces the computational load and improves the gaze tracking speed. If the facial image of the current frame is not highly similar to the facial images of the previous N frames, the data is then input into the gaze recognition model for gaze recognition.

[0062] The technical solution of this embodiment can be applied in a variety of scenarios, such as human-computer interaction, assisted driving, psychological research, virtual reality and military.

[0063] In one specific embodiment, the technical solution of this embodiment can be applied in the screen scene. By tracking the gaze, the position of the viewer's gaze can be determined, thereby driving the screen's driving components to achieve adaptive adjustment of the screen's horizontal and vertical placement modes.

[0064] In this embodiment, a camera is used to capture the viewer's facial image. An infrared camera can be used, allowing it to capture facial images even in the absence of visible light, thus overcoming environmental limitations. To improve the accuracy of eye-tracking, multiple cameras can be used to capture the viewer's facial image. Parameters such as the camera's field of view and resolution can be set according to specific needs.

[0065] In this embodiment, when a viewer is looking at the display screen, the camera captures and stores the viewer's facial image. The viewer's facial image can be transmitted to the processor, which stores a pre-trained human eye detection model and a gaze recognition model. The human eye detection model and the gaze recognition model can be used to track the gaze and determine the position of the viewer's gaze on the display screen. This position can be represented by coordinates, and the coordinate system can be based on a point on the display screen as the origin.

[0066] The camera can transmit captured facial images to the processor in real time, or it can capture multiple frames of facial images, such as N frames, and save them. Then, it can transmit the N frames of facial images together to the processor for image processing and output the position of the gaze point.

[0067] In an optional embodiment of this disclosure, the method further includes the step of training a human eye detection model, wherein training the human eye detection model includes:

[0068] Step 1: Establish the first training dataset. The first training dataset includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different.

[0069] Specifically, facial images of viewers of different ages and genders can be collected, and the eye contours in the facial images can be marked. The contours of the left and right eyes are marked separately to obtain multiple sets of first training data.

[0070] Step 2: Establish an initial model for human eye detection. The initial model for human eye detection consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image.

[0071] In this embodiment, the initial model for human eye detection uses an improved U-Net structure. The downsampling network includes M downsampling layers, and the upsampling network includes M upsampling layers, where M is an integer greater than 1. The downsampling layers and the upsampling layers correspond one-to-one. The input of the first downsampling layer is a facial image. The i-th downsampling layer performs convolution operations on the input features using different convolution kernels, concatenates the calculation results, and passes them to the (i+1)-th downsampling layer. The input of the i-th upsampling layer is the concatenation of the output of the (i+1)-th upsampling layer and a first result, where the first result is the result obtained by convolution calculation on the output of the i-th downsampling layer, where i is greater than or equal to 1 and less than or equal to M-1. The M-th downsampling layer performs convolution operations on the input features and passes them to the M-th upsampling layer. The output of the M-th upsampling layer is the eye contour in the facial image.

[0072] In a specific example, M can take the value of 4, and the structure of the initial model for human eye detection is as follows: Figure 2As shown, the left half is the downsampling network and the right half is the upsampling network. The black boxes represent different convolution operations, i.e., performing different convolution calculations on the input features. The downsampling layer concatenates the channels of the calculation results, but all convolution calculations do not change the size of the feature map. The white boxes represent the concatenation of different convolution operations, but the stride is different from that of the black boxes, achieving both feature calculation and downsampling. After this is the ReLU (Rectified Lumen Function) layer. After the three downsampling layers, the resulting feature map is passed to the next block. After being passed to the bottom downsampling layer, another convolution calculation is performed before the upsampling calculation begins. The upsampling process is an interpolation method, directly calculating to the corresponding size (i.e., the same size as the downsampling process). Before each upsampling calculation, the feature map of the downsampling part is concatenated with the input feature map after a 1*1 convolution calculation, i.e., combining shallow features with deep features, which can more clearly reflect the features.

[0073] In this embodiment, the pooling layer is removed from the downsampling network, which simplifies the structure of the downsampling network and improves computational efficiency. The transposed convolution is removed from the upsampling process, and interpolation is used to calculate directly, which can further improve computational efficiency. In addition, feature extraction (convolution) calculation is added during the downsampling and upsampling combination process, which combines shallow features with deep features and can more clearly reflect the features.

[0074] Step 3: Train the initial human eye detection model using the multiple sets of first training data, and save the model after training K times as the human eye detection model, where K is a positive integer.

[0075] The value of K can be set as needed, such as 50, 100, etc.

[0076] In an optional embodiment of this disclosure, the method further includes the step of training a gaze recognition model. The gaze recognition model is an end-to-end algorithm based on deep learning. The input is a facial image, the left and right eye images segmented from the facial image, the predicted gaze positions of the previous N frames of facial images, and their corresponding weight values. The output is the predicted gaze position of the current frame of the facial image. When training the gaze recognition model, a portion of the ground truth is required as input values. After training for a period of time, the trained model and its output results are used to fine-tune the model. The training process is as follows:

[0077] Step 1: Establish a second training dataset. The second training dataset includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the human eye's gaze in the current frame facial image. The facial images in different sets of second training data are different, as are the left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the human eye's gaze in the current frame facial image.

[0078] Specifically, multiple test points can be displayed on the screen; at least two cameras are used to capture facial images of a viewer looking at the test points in front of the screen, and the test points are located at the points where the viewer's line of sight falls, thereby obtaining facial images and the points where the viewer's line of sight falls.

[0079] When acquiring the second training data, images can be captured using a camera. Dots of different sizes, colors, and background colors are randomly displayed on a screen (including the appearance, disappearance, and random movement of fixed dots). The viewer sits in front of the screen; their posture is not restricted, only that their face is within the camera's field of view. The viewer's gaze follows the dots on the screen, and the camera captures their face, creating a dataset. This dataset is then segmented into sequential image streams. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the gaze placement points of the viewer in the previous A-frame facial image, and the gaze placement point in the current frame facial image. The camera can be a left-eye camera and a right-eye camera. The facial image captured by the left-eye camera can be used for eye segmentation to obtain the left and right eye images.

[0080] In an optional embodiment of this disclosure, before training the initial gaze recognition model using the plurality of sets of second training data, the method further includes preprocessing the second training data, including:

[0081] The distance between the viewer and the display screen is determined based on the facial image;

[0082] The sizes of the facial image and the left and right eye images are adjusted according to the distance.

[0083] Preprocessing the second training data can add a distance dimension as input to the model, thereby improving the accuracy of eye-tracking.

[0084] In this system, the relative position between the camera and the display screen is fixed. After capturing facial images, the distance between the viewer and the camera can be calculated. Then, based on the relative position between the camera and the display screen, the distance between the viewer and the display screen is determined. This relative distance between the viewer and the display screen is used as a new channel dimension and pasted onto the facial image. The pasted facial image and the human eye segmentation result image are then stretched to the same size.

[0085] like Figure 3 As shown, the left half displays the corresponding position of the viewer's target point P in the left and right views captured by the left and right eye cameras, while the right half shows the corresponding geometric model. The distance z between the viewer and the display screen can be calculated using the following formula:

[0086] d = u L -u R

[0087]

[0088] Among them, P L P R These are the points corresponding to point P in the left and right views, respectively. L u R Let P be the coordinates of point P in the left and right views, f be the inter-camera focal length, b be the inter-camera baseline, and O be the coordinates of point P in the left and right views, respectively. L The optical center of the left eye camera, O R The optical center of the right eye camera.

[0089] Step 2: Establish an initial model for gaze recognition. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image.

[0090] The initial gaze recognition model built in this embodiment is based on DenseNet, such as... Figure 4 As shown, the gaze recognition model sequentially includes a first layer Dense Block, a first layer Transition Layer, a second layer Dense Block, a second layer Transition Layer, a third layer Dense Block, a third layer Transition Layer, and a Full Connection layer.

[0091] like Figure 5As shown, the Dense module includes multiple sequentially connected 1*1 convolutional layers and 3*3 convolutional layers. The output of every two convolutional layers is input before the next two convolutions, and the residual is calculated before being input into the subsequent convolutions. For example, the output of the (2j-1)th convolutional layer and the 2jth convolutional layer is used to calculate the residual before being input into the 2j+5th convolutional layer, where j is a positive integer.

[0092] Step 3: Train the initial model of gaze recognition using the multiple sets of second training data, and save the model after training L times as the gaze recognition model, where L and A are positive integers.

[0093] In this embodiment, end-to-end model training is performed, and the value of L can be set as needed, such as 50, 100, etc.

[0094] To reduce computational load, before inputting the segmented left and right eye images into the pre-trained gaze recognition model, the method further includes:

[0095] Calculate the similarity between the facial image in the current frame and the facial images in each of the previous N frames;

[0096] If the similarity between the facial image in the current frame and facial image n in the previous N frames is greater than a preset threshold, stop inputting data into the fully connected layer, and use the predicted position of the human eye's gaze point in the facial image n as the predicted position of the human eye's gaze point in the current frame facial image.

[0097] In this embodiment, before inputting data into the fully connected layer, a judgment step is added to determine whether the facial image of the current frame is highly similar to the facial images of the previous N frames (similarity greater than a preset threshold, which can be fitted from other data, such as 98%). If the facial image of the current frame is highly similar to the facial images of the previous N frames, it can be assumed that the viewer's gaze has not changed significantly, and the predicted position of the viewer's gaze in facial image n is directly output as the predicted position of the viewer's gaze in the current frame. This reduces the computational load and improves the speed of gaze tracking. If the facial image of the current frame is not highly similar to the facial images of the previous N frames, the data is then input into the fully connected layer for feature extraction.

[0098] like Figure 6As shown, when calculating the similarity between the current frame's facial image and each of the previous N frames' facial images, the facial image can be segmented into several regions (each region is also a channel), the mean and variance of each channel are calculated, the difference between the current frame's facial image and each of the previous N frames' facial images is calculated, a similarity coefficient is fitted, and it is compared with a threshold. If the difference between the current frame's facial image and facial image n is less than the threshold, the current frame's facial image and facial image n are considered to have the same gaze point; if the difference between the current frame's facial image and facial image n is greater than or equal to the threshold, feature extraction is performed.

[0099] This embodiment constructs a convolutional neural network. The video stream based on facial images is preprocessed and then input into the convolutional neural network for training. The trained model can directly detect the gaze point based on the facial images captured by the camera, thereby improving the accuracy and efficiency of gaze detection.

[0100] This disclosure also provides an eye-tracking device, such as... Figure 7 As shown, it includes:

[0101] Image acquisition module 21 is used to acquire multiple frames of facial images of a viewer while viewing the display screen using at least two cameras;

[0102] The human eye detection module 22 is used to segment the left and right eye images from the face image in the current frame using a pre-trained human eye detection model, wherein the human eye detection model adopts a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image.

[0103] The gaze recognition module 23 is used to calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, the predicted gaze position of the facial image n is used as the predicted gaze position of the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than a preset threshold, the gaze recognition model, which is pre-trained, is used to detect the viewer's gaze position on the display screen. The gaze recognition model uses a convolutional neural network, with the input being the left and right eye images segmented from the facial image and the predicted gaze positions of the previous N frames' facial images, and the output being the predicted gaze position of the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N.

[0104] In this embodiment, at least two cameras are used to capture multiple frames of facial images of the viewer when viewing the display screen. A pre-trained human eye detection model and a gaze recognition model are used to determine the position of the viewer's gaze on the display screen, resulting in fast gaze tracking. Furthermore, deep learning can incorporate facial motion features and environmental influences into the model, thereby improving the accuracy of gaze tracking.

[0105] The technical solution of this embodiment can be applied in a variety of scenarios, such as human-computer interaction, assisted driving, psychological research, virtual reality and military.

[0106] In one specific embodiment, the technical solution of this embodiment can be applied in the screen scene. By tracking the gaze, the position of the viewer's gaze can be determined, thereby driving the screen's driving components to achieve adaptive adjustment of the screen's horizontal and vertical placement modes.

[0107] In this embodiment, a camera is used to capture the viewer's facial image. An infrared camera can be used, allowing it to capture facial images even in the absence of visible light, thus overcoming environmental limitations. To improve the accuracy of eye-tracking, multiple cameras can be used to capture the viewer's facial image. Parameters such as the camera's field of view and resolution can be set according to specific needs.

[0108] In this embodiment, the human eye detection module 22 and the gaze recognition module 23 can be integrated into the processor. When the viewer is looking at the display screen, the camera captures and stores the viewer's facial image. The viewer's facial image can be transmitted to the processor. The processor stores a pre-trained human eye detection model and a gaze recognition model. It can use the human eye detection model and the gaze recognition model to perform gaze tracking and determine the position of the viewer's gaze on the display screen. The position of the gaze can be represented by coordinates, and the coordinate system can be based on a point on the display screen as the origin.

[0109] The camera can transmit captured facial images to the processor in real time, or it can capture multiple frames of facial images, such as N frames, and save them. Then, it can transmit the N frames of facial images together to the processor for image processing and output the position of the gaze point.

[0110] In an optional embodiment of this disclosure, a human eye detection model training module is further included for training a human eye detection model, the human eye detection model training module comprising:

[0111] The first submodule is used to establish the first training dataset, which includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different.

[0112] Specifically, facial images of viewers of different ages and genders can be collected, and the eye contours in the facial images can be marked. The contours of the left and right eyes are marked separately to obtain multiple sets of first training data.

[0113] The second submodule is used to establish an initial model for human eye detection. The initial model for human eye detection consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image.

[0114] In this embodiment, the initial model for human eye detection uses an improved U-Net structure. The downsampling network includes M downsampling layers, and the upsampling network includes M upsampling layers, where M is an integer greater than 1. The downsampling layers and the upsampling layers correspond one-to-one. The input of the first downsampling layer is a facial image. The i-th downsampling layer performs convolution operations on the input features using different convolution kernels, concatenates the calculation results, and passes them to the (i+1)-th downsampling layer. The input of the i-th upsampling layer is the concatenation of the output of the (i+1)-th upsampling layer and a first result, where the first result is the result obtained by convolution calculation on the output of the i-th downsampling layer, where i is greater than or equal to 1 and less than or equal to M-1. The M-th downsampling layer performs convolution operations on the input features and passes them to the M-th upsampling layer. The output of the M-th upsampling layer is the eye contour in the facial image.

[0115] The third submodule is used to train the initial human eye detection model using the multiple sets of first training data, and save the model after training K times as the human eye detection model, where K is a positive integer.

[0116] The value of K can be set as needed, such as 50, 100, etc.

[0117] In an optional embodiment of this disclosure, a gaze recognition model training module is further included for training a gaze recognition model, the gaze recognition model training module comprising:

[0118] The fourth submodule is used to establish a second training dataset. The second training dataset includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the human eye's gaze in the current frame facial image. The facial images in different sets of second training data are different, as are the left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the human eye's gaze in the current frame facial image.

[0119] Specifically, multiple test points can be displayed on the screen; at least two cameras are used to capture facial images of a viewer looking at the test points in front of the screen, and the test points are located at the points where the viewer's line of sight falls, thereby obtaining facial images and the points where the viewer's line of sight falls.

[0120] When acquiring the second training data, images can be captured using a camera. Dots of different sizes, colors, and background colors are randomly displayed on a screen (including the appearance, disappearance, and random movement of fixed dots). The viewer sits in front of the screen; their posture is not restricted, only that their face is within the camera's field of view. The viewer's gaze follows the dots on the screen, and the camera captures their face, creating a dataset. This dataset is then segmented into sequential image streams. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the gaze placement points of the viewer in the previous A-frame facial image, and the gaze placement point in the current frame facial image. The camera can be a left-eye camera and a right-eye camera. The facial image captured by the left-eye camera can be used for eye segmentation to obtain the left and right eye images.

[0121] In an optional embodiment of this disclosure, before training the initial gaze recognition model using the plurality of sets of second training data, the method further includes preprocessing the second training data, including:

[0122] The distance between the viewer and the display screen is determined based on the facial image;

[0123] The sizes of the facial image and the left and right eye images are adjusted according to the distance.

[0124] Preprocessing the second training data can add a distance dimension as input to the model, thereby improving the accuracy of eye-tracking.

[0125] In this system, the relative position between the camera and the display screen is fixed. After capturing facial images, the distance between the viewer and the camera can be calculated. Then, based on the relative position between the camera and the display screen, the distance between the viewer and the display screen is determined. This relative distance between the viewer and the display screen is used as a new channel dimension and pasted onto the facial image. The pasted facial image and the human eye segmentation result image are then stretched to the same size.

[0126] The fifth submodule is used to establish an initial model for gaze recognition. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image.

[0127] The initial gaze recognition model built in this embodiment is based on DenseNet, such as... Figure 4 As shown, the gaze recognition model sequentially includes a first layer Dense Block, a first layer Transition Layer, a second layer Dense Block, a second layer Transition Layer, a third layer Dense Block, a third layer Transition Layer, and a Full Connection layer.

[0128] like Figure 5 As shown, the Dense module includes multiple sequentially connected 1*1 convolutional layers and 3*3 convolutional layers. The output of every two convolutional layers is input before the next two convolutions, and the residual is calculated before being input into the subsequent convolutions. For example, the output of the (2j-1)th convolutional layer and the 2jth convolutional layer is used to calculate the residual before being input into the 2j+5th convolutional layer, where j is a positive integer.

[0129] The sixth submodule is used to train the initial gaze recognition model using the multiple sets of second training data, and save the model after training L times as the gaze recognition model, where L and A are positive integers.

[0130] In this embodiment, end-to-end model training is performed, and the value of L can be set as needed, such as 50, 100, etc.

[0131] This disclosure also provides an eye-tracking system, including a display screen, at least two cameras, and the eye-tracking device described above. The eye-tracking system of this embodiment can determine the viewer's gaze point on the display screen while the viewer is viewing it, thereby enabling human-computer interaction and other operations.

[0132] This disclosure also provides an eye-tracking device, which includes a memory and an actuator. The actuator is used to perform the method described above, and will not be repeated here.

[0133] This disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described above.

[0134] In the various method embodiments of this disclosure, the sequence numbers of each step are not intended to limit the order of the steps. For those skilled in the art, any changes in the order of the steps are within the scope of protection of this disclosure without any creative effort.

[0135] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, since the embodiments are basically similar to the product embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the product embodiments.

[0136] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0137] It is understandable that when a component such as a layer, film, region, or substrate is referred to as being "above" or "below" another component, the component may be "directly" located "above" or "below" the other component, or there may be intermediate components present.

[0138] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0139] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A gaze tracking method, characterized in that, include: Use at least two cameras to capture multiple frames of facial images of the viewer as they view the display screen; The left and right eye images are segmented from the face image in the current frame using a pre-trained human eye detection model, wherein the human eye detection model employs a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image; Calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, the predicted position of the viewer's gaze on the facial image n is used as the predicted position of the viewer's gaze on the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than a preset threshold, the viewer's gaze on the display screen is detected using a pre-trained gaze recognition model. The gaze recognition model uses a convolutional neural network, with the input being the left and right eye images segmented from the facial image and the predicted positions of the viewer's gaze on the previous N frames' facial images, and the output being the predicted position of the viewer's gaze on the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N. The method further includes the step of training a gaze recognition model, wherein training the gaze recognition model includes: A second training dataset is established, which includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the gaze in the current frame facial image. The facial images in different sets of second training data are different. An initial model for gaze recognition is established. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image. The initial model for gaze recognition is trained using the multiple sets of second training data, and the model after training L times is saved as the gaze recognition model, where L and A are positive integers.

2. The gaze tracking method according to claim 1, characterized in that, The method further includes the step of training a human eye detection model, wherein the trained human eye detection model includes: Establish a first training dataset, which includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different. An initial model for human eye detection is established, which consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image. The initial human eye detection model is trained using the multiple sets of first training data, and the model after training K times is saved as the human eye detection model, where K is a positive integer.

3. The gaze tracking method according to claim 2, characterized in that, The downsampling network includes M downsampling layers, and the upsampling network includes M upsampling layers, where M is an integer greater than 1. The downsampling layers and the upsampling layers correspond one-to-one. The input to the first downsampling layer is a facial image. The i-th downsampling layer performs convolution operations on the input features using different convolution kernels, combines the calculation results together, and passes them to the (i+1)-th downsampling layer. The input to the i-th upsampling layer is the concatenation of the output of the (i+1)-th upsampling layer and the first result, where the first result is the result obtained by convolution of the output of the i-th downsampling layer, and i is greater than or equal to 1 and less than or equal to M-1. The Mth downsampling layer performs a convolution operation on the input features and then passes them to the Mth upsampling layer. The output of the Mth upsampling layer is the eye contour in the facial image.

4. The gaze tracking method according to claim 3, characterized in that, The upsampling layer uses an interpolation method.

5. The gaze tracking method according to claim 1, characterized in that, Before training the initial gaze recognition model using the multiple sets of second training data, the method further includes preprocessing the second training data, including: The distance between the viewer and the display screen is determined based on the facial image; The sizes of the facial image and the left and right eye images are adjusted according to the distance.

6. The gaze tracking method according to claim 1, characterized in that, Obtaining the second training data includes: Display multiple test points on the screen; At least two cameras are used to capture facial images of a viewer looking at the test point in front of a display screen, the test point being the location where the viewer's line of sight falls.

7. The gaze tracking method according to claim 1, characterized in that, The gaze recognition model sequentially includes a first-layer Dense module, a first-layer transition layer, a second-layer Dense module, a second-layer transition layer, a third-layer Dense module, a third-layer transition layer, and a fully connected layer. The Dense module includes multiple sequentially connected 1*1 convolutional layers and 3*3 convolutional layers. The outputs of the 2j-1 and 2j convolutional layers are used to calculate the residuals and then input into the 2j+5 convolutional layer, where j is a positive integer.

8. A gaze tracking device, characterized in that, include: An image acquisition module is used to capture multiple frames of facial images of a viewer while viewing a display screen using at least two cameras; The human eye detection module is used to segment the left and right eye images from the face image in the current frame using a pre-trained human eye detection model, wherein the human eye detection model adopts a convolutional neural network, the input of the human eye detection model is the face image, and the output is the left and right eye images segmented from the face image; A gaze recognition module is used to calculate the similarity between the current frame's facial image and each of the previous N frames' facial images. If the similarity between the current frame's facial image and each of the previous N frames' facial images n is greater than a preset threshold, the predicted gaze position of the facial image n is used as the predicted gaze position of the current frame's facial image. If the similarity between the current frame's facial image and each of the previous N frames' facial images is not greater than a preset threshold, a pre-trained gaze recognition model is used to detect the viewer's gaze position on the display screen. The gaze recognition model uses a convolutional neural network, with the input being the left and right eye images segmented from the facial image and the predicted gaze positions of the previous N frames' facial images, and the output being the predicted gaze position of the current frame's facial image. N is a positive integer, and n is an integer not less than 1 and not greater than N. The gaze tracking device further includes a gaze recognition model training module for training a gaze recognition model, the gaze recognition model training module comprising: The fourth submodule is used to establish a second training dataset. The second training dataset includes multiple sets of second training data. Each set of second training data includes a facial image, left and right eye images segmented from the facial image, the position of the human eye's gaze in the previous A-frame facial image, and the position of the gaze in the current frame facial image. The facial images in different sets of second training data are different. The fifth submodule is used to establish an initial model for gaze recognition. The input of the initial model for gaze recognition is the left and right eye images segmented from the facial image and the position of the gaze point of the human eye in the previous A-frame facial image. The output is the position of the gaze point of the human eye in the current frame facial image. The sixth submodule is used to train the initial gaze recognition model using the multiple sets of second training data, and save the model after training L times as the gaze recognition model, where L and A are positive integers.

9. The gaze tracking device according to claim 8, characterized in that, It also includes a human eye detection model training module for training a human eye detection model, the human eye detection model training module comprising: The first submodule is used to establish the first training dataset, which includes multiple sets of first training data. Each set of first training data includes a facial image and the eye contours marked therein. The facial images and eye contours of different sets of first training data are different. The second submodule is used to establish an initial model for human eye detection. The initial model for human eye detection consists of a downsampling network and an upsampling network. The input of the downsampling network is a facial image, and the output of the upsampling network is the eye contour in the facial image. The third submodule is used to train the initial human eye detection model using the multiple sets of first training data, and save the model after training K times as the human eye detection model, where K is a positive integer.

10. A gaze tracking system, characterized in that, It includes a display screen, at least two cameras, and an eye-tracking device as described in any one of claims 8-9.

11. A gaze tracking device, characterized in that, The gaze tracking device includes a memory and an actuator, the actuator being used to perform the method as described in any one of claims 1-7.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

Images