Viewpoint prediction method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a view point prediction method and device, electronic equipment and storage medium, and relates to the technical field of view point prediction. The view point prediction method comprises: obtaining a color image to be predicted and a corresponding depth image thereof; performing target detection on the color image by using a preset target detection model to determine detection boxes corresponding to eyes and a mouth; determining an eye image by using the color image and the detection box corresponding to the eyes; performing horizontal perspective transformation on the eye image to obtain a corrected eye image; determining position and posture information corresponding to the eyes and a head by using the depth image and the detection boxes corresponding to the eyes and the mouth; and performing view point prediction by using the position and posture information and the corrected eye image based on a preset view point prediction model. The present disclosure can realize view point prediction.

Description

Technical Field

[0001] This disclosure relates to the field of viewpoint prediction technology, and in particular to a viewpoint prediction method and apparatus, electronic device and storage medium. Background Technology

[0002] Viewpoint prediction is an emerging topic in computer vision, applicable to many applications such as visual attention analysis and human-computer interaction. However, the performance of gaze tracking can be further improved under conditions of free head movement.

[0003] Viewpoint prediction is a crucial research area in computer vision. It utilizes devices such as color cameras and infrared equipment to capture features like faces and eyes, then calculates the user's gaze position on a screen or in a natural environment. This facilitates user-device interaction or provides data support for operations or testing by collecting the user's physiological information. Viewpoint prediction technology has wide applications in education, scientific research, medicine, entertainment, and other fields. The results of viewpoint prediction can be represented using gaze trajectory maps and heatmaps, which can be used in various eye-tracking applications in daily life. Figure 1 Examples of viewpoint prediction results according to embodiments of this disclosure are shown, such as Figure 1 As shown, the left side is the viewpoint trajectory map, and the right side is the viewpoint heatmap.

[0004] Appearance-based viewpoint prediction is currently a hot research topic in computer vision. This method relies entirely on processing biometric information captured by imaging devices to calculate and predict the gaze direction. It has several advantages, including simplicity, efficiency, high accuracy, few model parameters, and low computational requirements. Furthermore, due to its low hardware requirements, it is suitable for deployment scenarios using lightweight imaging devices. Therefore, it has become one of the mainstream methods for viewpoint prediction tasks.

[0005] There are currently several appearance-based viewpoint prediction algorithms. With the development of deep learning in the field of computer vision, more and more deep learning network models have been applied to viewpoint prediction tasks. Among them, the deep learning-based viewpoint prediction methods mainly include the following: (1) Facial feature point detection algorithm. This method uses a facial feature point model to process facial images, obtain and locate facial feature points, and then uses a neural network model to calculate the gaze direction in combination with the feature points to achieve viewpoint prediction. This method has high accuracy, but it still faces challenges in accurately predicting the gaze direction at large angles and distances. (2) Binocular image processing method. This method uses image processing models such as segmentation and regression in convolutional neural networks to perform regression on binocular images to predict the gaze direction. This method has high accuracy, but it requires binocular images, so it is not suitable for monocular viewpoint prediction. (3) Eye model-based method. The eye model is reconstructed using mathematical modeling methods, and the eye model is fitted based on the eye images captured by the camera.

[0006] Traditional appearance-aware viewpoint prediction methods are limited in their ability to handle tasks involving different positions and angles because they require the use of devices (such as tripods) to fix the user's head in front of the camera for sampling. Therefore, these methods cannot meet the requirements of everyday scenarios and are only suitable for special conditions in the laboratory.

[0007] Currently, appearance-aware viewpoint prediction methods suffer from a certain angular error on home computers. However, when the subject's head is significantly tilted or the distance is considerable, the system may fail to accurately detect the eye position and posture, leading to inaccurate or even non-existent viewpoint prediction. Furthermore, appearance-aware viewpoint prediction methods use conventional cameras for sampling, making them highly susceptible to ambient lighting conditions. Existing algorithms all involve converting the gaze from the camera coordinate system to the world coordinate system and calculating its gaze point on the screen. These algorithms involve multiple manual calculation steps, which can introduce difficult-to-calibrate errors when the camera or screen position is unstable.

[0008] Meanwhile, in existing appearance-based viewpoint prediction methods, the gaze direction can be determined within the camera's coordinate system by analyzing binocular images related to head position. Incorporating 3D head pose allows for the calculation of the gaze's origin and direction within the camera's coordinate system. By transforming the coordinates to the world coordinate system, the gaze point on the display screen, referred to as the target point, can be determined through spatial calculations.

[0009] However, the coordinate transformations and spatial computations involved in this approach are conventional, and the relationship between the gaze vectors before and after the transformation is non-linear. This poses a challenge when the relative positions of the sampling camera and the gaze screen change, as precise adjustments cannot be made using calibration processes with linear fully connected layers. Parameters in conventional computational pipelines are difficult to calibrate, leading to difficulties in handling errors generated by the system in such cases. Summary of the Invention

[0010] This disclosure presents a viewpoint prediction method and apparatus, electronic device and storage medium technical solution.

[0011] According to one aspect of this disclosure, a viewpoint prediction method is provided, comprising:

[0012] Obtain the color image to be predicted and its corresponding depth image;

[0013] Using a preset target detection model, target detection is performed on the color image to determine the detection boxes corresponding to the eyes and mouth;

[0014] The eye image is determined using the color image and the detection box corresponding to the eye;

[0015] Perform a horizontal perspective transformation on the eye image to obtain a corrected eye image;

[0016] Using the depth image and the detection boxes corresponding to the eyes and mouth, the position and pose information of the eyes and head are determined;

[0017] Based on a preset viewpoint prediction model, viewpoint prediction is performed using the position and posture information and the corrected eye image.

[0018] Preferably, the method for determining the eye image using the color image and the detection box corresponding to the eye includes:

[0019] Calculate the lengths of the first and second diagonals of the detection frames for the left and right eyes, respectively.

[0020] Based on a set multiplier, the left eye region and the right eye region are determined using the lengths of the first diagonal and the second diagonal, respectively.

[0021] Based on the left eye region and the right eye region respectively, crop out the corresponding eye images of the left and right eyes from the color image; and / or,

[0022] Before performing a horizontal perspective transformation on the eye image to obtain the corrected eye image, the method further includes:

[0023] Get the set pixel size;

[0024] The pixels of the corresponding eye images for the left and right eyes are adjusted separately to achieve the set pixel size.

[0025] Preferably, the method for performing a horizontal perspective transformation on the eye image to obtain a corrected eye image includes:

[0026] Calculate the first axial distance and the second axial distance between the two sides of the eye image corresponding to the left or right eye and the depth camera;

[0027] Based on the first axial distance and the second axial distance, calculate the perspective transformation ratio corresponding to the eye image of the left eye or the right eye respectively;

[0028] Based on the perspective transformation ratio of the eye image corresponding to the left or right eye, the eye image corresponding to the left or right eye is corrected to obtain the corrected eye image.

[0029] Preferably, the preset viewpoint prediction model includes: a first feature extraction path and a second feature extraction path, a first stitching layer connected to the first feature extraction path and the second feature extraction path respectively, a fully connected layer for feature compression connected to the stitching layer, a second stitching layer for stitching the position and pose information connected to the fully connected layer, and a classification layer connected to the second stitching layer;

[0030] Using the first feature extraction path and the second feature extraction path, feature extraction is performed on the left eye image and the right eye image of the eye image respectively to obtain the first feature map and the second feature map;

[0031] Using the first splicing layer, the first feature map and the second feature map are spliced ​​together to obtain a spliced ​​feature map;

[0032] The fully connected layer is used to compress the stitched feature map to obtain compressed features;

[0033] Using the classification layer and the compressed features, the viewpoint coordinates of the left and right eyes are obtained respectively, completing viewpoint prediction; and / or,

[0034] The first feature extraction path includes: a first convolutional layer and a first pooling layer connected thereto, a first attention mechanism module connected to the first pooling layer, a second convolutional layer connected to the first attention mechanism module and the first pooling layer, and a second pooling layer connected thereto.

[0035] Using the first convolutional layer and the first pooling layer connected thereto, feature extraction and pooling are performed on the left eye image in the eye image to obtain the first feature map to be processed.

[0036] Using the first attention mechanism module, attention is extracted from the first feature map to be processed to obtain the corresponding first weighted feature map;

[0037] Based on the first feature map to be processed and the first weighted feature map, the first feature map is obtained; and / or,

[0038] The method for obtaining the first feature map based on the first feature map to be processed and the first weighted feature map includes: performing a multiplication operation on the first feature map to be processed and the first weighted feature map to obtain the first feature map; and / or,

[0039] The first feature extraction path includes: a third convolutional layer and a third pooling layer connected thereto, a second attention mechanism module connected to the third pooling layer, a fourth convolutional layer connected to the second attention mechanism module and the third pooling layer, and a fourth pooling layer connected thereto.

[0040] Using the third convolutional layer and the third pooling layer connected thereto, feature extraction and pooling are performed on the right eye image in the eye image to obtain a second feature map to be processed.

[0041] The second attention mechanism module is used to extract attention from the second feature map to be processed, thereby obtaining the corresponding second weighted feature map;

[0042] The second feature map is obtained based on the second feature map to be processed and the second weighted feature map; and / or,

[0043] The method for obtaining the second feature map based on the second feature map to be processed and the second weighted feature map includes: performing a multiplication operation on the second feature map to be processed and the second weighted feature map to obtain the second feature map; and / or,

[0044] The fully connected layer includes: a first fully connected layer and a second fully connected layer connected thereto;

[0045] The stitched feature map is compressed using the first fully connected layer and the second fully connected layer connected thereto to obtain compressed features; and / or,

[0046] The classification layer is configured as a regression layer.

[0047] Preferably, before performing viewpoint prediction based on a preset viewpoint prediction model using the position and pose information and the corrected eye image, the method further includes:

[0048] Using the corrected eye images and position and pose information of the training set, the preset viewpoint prediction model is trained according to the preset training rules to obtain the trained preset viewpoint prediction model.

[0049] Furthermore, based on the trained preset viewpoint prediction model, viewpoint prediction is performed using the position and pose information and the corrected eye image; and / or

[0050] The method for training a preset viewpoint prediction model according to preset training rules to obtain a trained preset viewpoint prediction model includes:

[0051] Get the set number of training iterations;

[0052] When the preset viewpoint prediction model has been trained a certain number of times, training of the preset viewpoint prediction model is stopped, and the trained preset viewpoint prediction model is obtained; and / or

[0053] Before performing viewpoint prediction based on the trained preset viewpoint prediction model, using the position and pose information and the corrected eye image, the method further includes:

[0054] Using the corrected eye images and position and pose information of the calibration set, the output layer of the trained preset viewpoint prediction model is calibrated according to the preset calibration rules to obtain the calibrated preset viewpoint prediction model.

[0055] Furthermore, based on the calibrated preset viewpoint prediction model, viewpoint prediction is performed using the position and pose information and the corrected eye image; and / or

[0056] The method for calibrating the output layer of the trained preset viewpoint prediction model according to preset calibration rules to obtain a calibrated preset viewpoint prediction model includes:

[0057] Get the preset loss value

[0058] Calculate the loss value of the viewpoint coordinates and their corresponding viewpoint coordinate labels obtained during the calibration process;

[0059] When the loss value reaches the preset loss value, the calibration of the preset viewpoint prediction model is stopped, and a calibrated preset viewpoint prediction model is obtained.

[0060] Preferably, the method further includes: determining a method for training and / or calibrating the loss function, including:

[0061] Calculate the first norm of the X-coordinate in each predicted viewpoint coordinate during training and / or calibration, corresponding to the true X-coordinate.

[0062] Calculate the second norm of the Y-coordinate in each predicted viewpoint coordinate during training and / or calibration, corresponding to the true Y-coordinate.

[0063] Calculate the third norm of the squared Y-coordinate value in each predicted viewpoint coordinate during training and / or calibration, corresponding to the squared Y-coordinate value of the true Y-coordinate.

[0064] Calculate the fourth norm of the viewpoint coordinates predicted each time during training and / or calibration, corresponding to the true viewpoint coordinates;

[0065] Based on the first norm and its corresponding first weight coefficient, the second norm and its corresponding second weight coefficient, the third norm and its corresponding third weight coefficient, and the fourth norm and its corresponding fourth weight coefficient, a loss function for training and / or calibration is determined; and / or

[0066] The first weighting coefficient, the second weighting coefficient, the third weighting coefficient, and the fourth weighting coefficient are configured as 2, 2, 1, and 4, respectively.

[0067] Preferably, the method further includes: filtering the multiple viewpoint coordinates predicted for adjacent set frame numbers to obtain the final viewpoint coordinates; and / or

[0068] The method for filtering multiple viewpoint coordinates predicted from adjacent set frame numbers includes:

[0069] The X coordinates of the multiple viewpoint coordinates predicted for the adjacent set number of frames are averaged to obtain the final X coordinates of the viewpoint coordinates.

[0070] The Y-coordinates of the multiple predicted viewpoint coordinates corresponding to the adjacent set number of frames are averaged to obtain the final Y-coordinate in the viewpoint coordinates; and / or,

[0071] The method for determining the position and pose information of the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth includes: aligning the depth image with the color image; generating a point cloud array corresponding to the aligned depth image; determining the position of the eyes and mouth based on the index of the centroid unit in the point cloud array and the center of the detection boxes corresponding to the eyes and mouth; calculating the normal vector of the plane where the eyes and mouth are located based on the position of the eyes and mouth, and converting the normal vector into a rotation vector to obtain pose information.

[0072] According to one aspect of this disclosure, a viewpoint prediction device is provided, comprising:

[0073] The acquisition unit is used to acquire the color image to be predicted and its corresponding depth image;

[0074] The first determining unit is used to perform target detection on the color image using a preset target detection model, and determine the detection boxes corresponding to the eyes and mouth;

[0075] The second determining unit is used to determine an eye image using the color image and the detection box corresponding to the eye;

[0076] The correction unit is used to perform a horizontal perspective transformation on the eye image to obtain a corrected eye image;

[0077] The third determining unit is used to determine the position and posture information of the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth;

[0078] The prediction unit is used to predict the viewpoint based on a preset viewpoint prediction model, using the position and posture information and the corrected eye image.

[0079] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0080] processor;

[0081] Memory used to store processor-executable instructions;

[0082] The processor is configured to execute the viewpoint prediction method described above.

[0083] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described viewpoint prediction method.

[0084] In the embodiments of this disclosure, a viewpoint prediction method and apparatus, electronic device and storage medium are proposed, which can improve the accuracy and robustness of appearance-based viewpoint prediction without locking the head, thereby solving the problem that the current appearance-based viewpoint prediction has poor accuracy and robustness and requires head locking.

[0085] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0086] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0088] Figure 1An example of viewpoint prediction results according to embodiments of this disclosure is shown;

[0089] Figure 2 A flowchart illustrating a viewpoint prediction method according to an embodiment of the present disclosure is shown;

[0090] Figure 3 A flowchart illustrating a specific implementation of the viewpoint prediction method according to an embodiment of the present disclosure is shown.

[0091] Figure 4 A flowchart illustrating a specific implementation of a method for calculating the position and posture of the eyes and mouth according to an embodiment of the present disclosure is provided.

[0092] Figure 5 This diagram illustrates the principle of perspective transformation of an eye image according to an embodiment of the present disclosure.

[0093] Figure 6 A schematic diagram illustrating the perspective conversion process of an eye image according to an embodiment of the present disclosure is shown;

[0094] Figure 7 A schematic diagram of the network structure of a preset viewpoint prediction model according to an embodiment of the present disclosure is shown.

[0095] Figure 8 A schematic diagram illustrating the camera coordinate system and viewpoint prediction according to an embodiment of the present disclosure is shown;

[0096] Figure 9 A distribution diagram of the viewpoint prediction training dataset according to an embodiment of the present disclosure is shown;

[0097] Figure 10 Performance analysis of a predictive target detection model (enhanced YOLOv5) for detecting eyes and mouth according to embodiments of the present disclosure is shown;

[0098] Figure 11 Performance analysis of a preset viewpoint prediction model (Gaze-Point-Net) with different variables according to embodiments of the present disclosure is shown;

[0099] Figure 12 The performance analysis of the preset viewpoint prediction model (Gaze-Point-Net) under different lighting conditions according to embodiments of the present disclosure is shown.

[0100] Figure 13 The results of random point tests according to embodiments of the present disclosure are shown;

[0101] Figure 14 The results of data filtering and model calibration analysis according to embodiments of this disclosure are shown.

[0102] Figure 15This paper presents a performance comparison analysis of Gaze-Point-Net with other models according to embodiments of the present disclosure;

[0103] Figure 16 This illustrates an analysis of the impact of confidence level on accuracy according to embodiments of the present disclosure;

[0104] Figure 17 The results of a trajectory tracking test according to an embodiment of this disclosure are shown;

[0105] Figure 18 A block diagram of a viewpoint prediction apparatus according to an embodiment of the present disclosure is shown.

[0106] Figure 19 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment;

[0107] Figure 20 This is a block diagram illustrating an electronic device 1900 according to an exemplary embodiment. Detailed Implementation

[0108] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0109] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0110] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0111] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0112] It is understood that the various viewpoint prediction method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further.

[0113] In addition, this disclosure also provides viewpoint prediction apparatus, electronic equipment, computer-readable storage medium, and program, all of which can be used to implement any viewpoint prediction method provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0114] Figure 2 A flowchart illustrating a viewpoint prediction method according to an embodiment of the present disclosure is shown; Figure 3 A flowchart illustrating a specific implementation of the viewpoint prediction method according to an embodiment of this disclosure is provided. Figure 3 (a) is a flowchart of the deployment of the viewpoint prediction system or the specific implementation of the viewpoint prediction method; Figure 3 (b) Training of a general gaze-pointing network model; Figure 3 (c) Calibration of the viewpoint prediction network model. For example... Figure 2 and Figure 3 As shown, the viewpoint prediction method includes: Step S101: acquiring a color image to be predicted and its corresponding depth image; Step S102: using a preset target detection model to perform target detection on the color image, determining the detection boxes corresponding to the eyes and mouth; Step S103: using the color image and the detection boxes corresponding to the eyes to determine the eye image; Step S104: performing a horizontal perspective transformation on the eye image to obtain a corrected eye image; Step S105: using the depth image and the detection boxes corresponding to the eyes and mouth to determine the position and pose information corresponding to the eyes and head; Step S106: based on the preset viewpoint prediction model, using the position and pose information and the corrected eye image to perform viewpoint prediction. This method can improve the accuracy and robustness of appearance-based viewpoint prediction without locking the head, thus solving the problem of poor accuracy and robustness of current appearance-based viewpoint prediction methods, which require head locking.

[0115] Step S101: Obtain the color image to be predicted and its corresponding depth image.

[0116] In embodiments of this disclosure and other possible embodiments, a camera can be used to acquire a color image to be predicted and its corresponding depth image. For example, the camera is configured as a depth camera, and the color image and its corresponding depth image are acquired using the depth camera.

[0117] Step S102: Using a preset target detection model, perform target detection on the color image to determine the detection boxes corresponding to the eyes and mouth.

[0118] In the embodiments and other possible embodiments disclosed herein, the preset target detection model configuration may be a YOLO target detection model configuration or other target detection models, such as the YOLOv5 target detection model.

[0119] In embodiments and other possible embodiments of this disclosure, such as Figure 3 As shown in (a), a YOLO object detection model is used to locate and crop the eye image, while a depth camera is used to calculate the 3D spatial positions of the eyes and mouth (the detection boxes corresponding to the eyes and mouth). Next, a multimodal (eye image / corrected eye image and its corresponding position and pose information) viewpoint prediction network model is used to predict the gaze point on the screen. To accomplish this task, we need to collect and create two datasets: one for training the YOLO object detection model, and the other for training the viewpoint prediction regression model (the viewpoint prediction model Gaze-Point-Net).

[0120] In the embodiments and other possible embodiments of this disclosure, before performing target detection on the color image using the preset target detection model to determine the detection boxes corresponding to the eyes and mouth, the preset target detection model is trained; and based on the trained preset target detection model (enhanced YOLOv5 model), target detection is performed on the color image to determine the detection boxes corresponding to the eyes and mouth.

[0121] In the embodiments and other possible embodiments disclosed herein, eyes and mouths are located and labeled in RGB images (color images). YOLOv5 is used as the object detection model in this study. YOLOv5 divides the image into several equally sized grids and predicts the location and category of the object for each grid. This model boasts advantages such as fast speed, high efficiency, good accuracy, and the ability to detect multiple object categories. Furthermore, YOLOv5 employs multi-scale training and testing to improve detection accuracy and speed, while also utilizing special techniques such as SPP blocks and PAN.

[0122] In the embodiments and other possible embodiments disclosed herein, YOLOv5 has been improved by adjusting its size and adding prior bounding boxes to better perform the task of detecting eyes and mouths. Specifically, the 20×20 feature layer in YOLOv5 has been removed, and a 160×160 feature layer has been added to better accommodate the detection of small objects. Furthermore, prior bounding boxes with shapes and sizes similar to those for eyes and mouths have been added to improve the performance and accuracy of object detection.

[0123] In embodiments and other possible embodiments of this disclosure, the method for training the preset target detection model includes: rotating a training image to obtain a rotated image; adjusting the rotated image to a set size and filling the set-size image with set pixel values. For example, a rotated image that has already been randomly rotated is further rotated counterclockwise by a set angle; wherein the set angle can be configured as 90 degrees, 180 degrees, or 270 degrees. Then, these images are adjusted to the square size required by the model (the set-size image) and any blank spaces are filled with black pixels to cover the distribution of all possible rotation angles.

[0124] In embodiments and other possible embodiments of this disclosure, the method for training the preset target detection model further includes: adding noise to the filled image obtained by filling the image of the set size with set pixel values ​​to obtain a corresponding noise image. For example, Gaussian noise is introduced, which significantly enhances our model's ability to detect small objects, especially eyes. This is a highlight of our task.

[0125] In embodiments and other possible embodiments of this disclosure, the method for training the preset target detection model further includes: performing histogram equalization on the filled image obtained by filling the image of the set size with set pixel values ​​or the noisy image to obtain an equalized image. For example, although the training dataset covers most lighting conditions, we use histogram equalization to handle any residual effects of lighting differences on model performance. This preprocessing technique helps reduce the impact of different lighting levels and further improves the overall quality of the training images.

[0126] In embodiments and other possible embodiments of this disclosure, the method for training the preset target detection model further includes: standardizing or normalizing the filled image, the noise image, or the equalized image obtained by filling the image of the set size with set pixel values, to obtain a standardized or normalized image. The standardized image has a mean of 0 and a standard deviation of 1, which contributes to a more efficient and stable training process.

[0127] In the embodiments and other possible embodiments of this disclosure, a random data splitting technique is used during training to divide the dataset into a training set, a validation set, and a test set in an 8:1:1 ratio. Stochastic gradient descent (SGD) is used as the optimizer, with a batch size of 16. The initial learning rate is set to 0.01 and dynamically reduced by 0.9 after the twentieth training cycle, for a total of 1000 training cycles.

[0128] In embodiments and other possible embodiments of this disclosure, the step of using a preset target detection model to perform target detection on the color image and determine the detection boxes corresponding to the eyes and mouth includes: obtaining a prediction confidence threshold; and determining the detection boxes corresponding to the eyes and mouth based on the detection boxes to be processed obtained by performing target detection on the color image using the preset target detection model and the prediction confidence threshold.

[0129] In embodiments of this disclosure and other possible embodiments, those skilled in the art can configure the prediction confidence threshold as needed. For example, the prediction confidence threshold can be configured to 0.3.

[0130] In embodiments and other possible embodiments of this disclosure, the method for determining the detection boxes corresponding to the eyes and mouth based on the target detection box obtained by performing target detection on the color image using the preset target detection model and the prediction confidence threshold includes: determining the threshold detection boxes in the target detection boxes whose prediction confidence is greater than or equal to the prediction confidence threshold; and sorting the threshold detection boxes according to the prediction confidence pairs, configuring the two threshold detection boxes with the highest confidence as the detection boxes corresponding to the eyes, and configuring the threshold detection box with the highest confidence as the detection box corresponding to the mouth.

[0131] In embodiments of this disclosure and other possible embodiments, for example, during the deployment phase of the YOLOv5 model, the prediction confidence threshold is set to 0.3, and the results for each image are filtered. The detection results for the eyes and mouth are then sorted according to the prediction confidence, and the two results with the highest confidence are selected as the detection results for the eyes, and the result with the highest confidence is selected as the detection result for the mouth.

[0132] In embodiments and other possible implementations of this disclosure, the dataset used in these embodiments contains over 10,000 face segmentation images. After applying data augmentation techniques, over 70,000 images conforming to the YOLO object detection algorithm and its dataset format requirements were successfully generated. This dataset covers a variety of features, such as age, gender, hair color, facial expression, clothing color, and background color. These images are stored in YOLO format, suitable for training the object detection network.

[0133] In embodiments of this disclosure and other possible embodiments, in order to locate and determine the bounding boxes of the eyes and mouth in a camera-captured image, the object detection model requires a training dataset. This disclosure considers using a facial keypoint segmentation dataset to train the object detection model, using the publicly available CelebA Mask-HQ and FASSEG datasets (training images) to achieve this purpose.

[0134] In the embodiments and other possible embodiments disclosed herein, the training images are configured with corresponding labels. The first step is to convert the two face segmentation datasets (CelebA Mask-HQ and FASSEG datasets) into a single object detection dataset (training images), which includes the original images and bounding box parameters (labels). The obtained face segmentation dataset contains the original images and binary segmentation labels. The CelebA Mask-HQ dataset's annotation format stores the binary segmentation labels for each object category (e.g., eyes, mouth) separately, while the FASSEG dataset stores the annotation data in RGB color images, using different colors to distinguish different segmentation regions. To unify the storage format of the annotation data, the segmentation labels are separated using the RGB values ​​of different segmentation regions, obtaining the segmentation labels for the eyes and mouth, thus processing the labels of the two face datasets into a unified format. Finally, an object detection dataset (the training images and their corresponding labels) is obtained.

[0135] Furthermore, embodiments of this disclosure employ the following data augmentation method. First, the original image is randomly rotated clockwise or counterclockwise by a set angle (e.g., 0 to 45 degrees) to obtain a rotated image; and blank areas of the rotated image are filled with a set value (e.g., zero). Next, the filled rotated image / rotated image is scaled down to a set factor (e.g., 0.2 to 0.8 times). Finally, one of a set number (e.g., seven) of pre-prepared cluttered background images is randomly selected, and the cluttered background image is pasted into any position of the background in the scaled-down / rotated / filled rotated image. Simultaneously, rotation, scaling, and translation operations are used to enhance either the original image or the image after pasting the cluttered background image into any position of the background in the scaled-down / rotated / filled rotated image.

[0136] Step S103: Determine the eye image using the color image and the detection box corresponding to the eye.

[0137] In an embodiment of this disclosure, the method for determining an eye image using the color image and the detection box corresponding to the eye includes: calculating the first diagonal length and the second diagonal length of the left eye detection box and the right eye detection box corresponding to the eye, respectively; determining the left eye region and the right eye region based on a set multiple using the first diagonal length and the second diagonal length, respectively; and cropping the eye images corresponding to the left eye and the right eye from the color image based on the left eye region and the right eye region, respectively.

[0138] In embodiments of this disclosure, before performing a horizontal perspective transformation on the eye image to obtain a corrected eye image, the method further includes: obtaining a set pixel size; and adjusting the pixels of the eye images corresponding to the left and right eyes respectively to achieve the set pixel size.

[0139] In the embodiments and other possible embodiments disclosed herein, those skilled in the art may configure the set multiplier and the set pixel size according to actual needs. For example, the set multiplier and the set pixel size may be configured as 2 and 36×60, respectively.

[0140] In embodiments and other possible embodiments of this disclosure, roll angle errors are introduced due to the rotation of the head along the camera (depth camera) axis. Therefore, we rotate the image (color image) using the coordinates of the center points of the two eyes in the image (color image acquired by the depth camera) to correct the image (color image) and align the two eyes on the same horizontal line. Furthermore, we use a coordinate rotation algorithm (Formula 1) to calculate the center point position of the target bounding box (detection box) and other information such as the bounding box in the rotated image (color image).

[0141] x′=(xc x )cosα-(yc y sinα+c x (1)

[0142] y′=(xc x )sinα+(yc y cosα+c y

[0143] Among them, c x and c y α is the rotation center of the color image captured by the depth camera, x and y are the pixel positions in the original image (the color image captured by the depth camera), x′ and y′ are the positions of the corresponding points after rotation, and α is the rotation angle of the color image captured by the depth camera.

[0144] After acquiring the rotated image (color image) and the detection information for the rotated eyes and mouth (detection boxes corresponding to the eyes and mouth), we define two rectangular regions with an aspect ratio of 5:3, based on the diagonal length of the detection boxes corresponding to the eyes. These rectangular regions are defined with their respective centers based on the centers of the two eyes. The diagonal length of each rectangular region is twice the diagonal length of the predicted eye bounding box. Therefore, we extract the eye images (including the right eye image and the right eye image) by cropping the portion of the original image (color image) contained within these rectangular regions. Subsequently, we resize the two cropped images (eye images) to meet the pixel size (36×60) required by the model for further processing.

[0145] Step S104: Perform a horizontal perspective transformation on the eye image to obtain a corrected eye image.

[0146] In an embodiment of this disclosure, the method for performing a horizontal perspective transformation on the eye image to obtain a corrected eye image includes: calculating a first axial distance and a second axial distance between the two sides of the eye image corresponding to the left or right eye and the depth camera; calculating the perspective transformation ratio corresponding to the eye image corresponding to the left or right eye based on the first axial distance and the second axial distance; and correcting the eye image corresponding to the left or right eye based on the perspective transformation ratio corresponding to the eye image corresponding to the left or right eye to obtain a corrected eye image.

[0147] In the embodiments and other possible embodiments of this disclosure, Figure 5 A schematic diagram illustrating the perspective transformation principle of an eye image according to an embodiment of this disclosure is shown. Figure 5 As shown, when performing a horizontal perspective transformation on the eye images (the corresponding eye images for the left and right eyes), we can consider them as horizontally tilted rectangles in the camera's viewpoint. In the example at the top right, representing the top view from the camera's (depth camera's) field of view, the axial distance between the left and right sides of the top right rectangle (the top view of the depth camera's field of view) is different due to the different distances from the camera to the left and right sides. This distance difference (D) between the left and right sides of the top right rectangle can be represented by the projection of the rectangle's horizontal side onto the depth camera's axis, mathematically expressed as D = L × cos(α), where α is the angle between the projection of the head pose vector onto the horizontal plane and the depth camera's axis, and L is the length of the rectangle's horizontal side. In other words, we obtain the difference in axial distance between the far and near sides and the camera.

[0148] In the embodiments and other possible embodiments disclosed herein, the lower right of the above figure represents a side view of the field of view captured by the camera. The far and near edges can be considered as the edges of two rectangles, namely the front and rear sides. The different distances between the camera and the left and right sides result in different viewing angles, and therefore the lengths of these two sides in the image are also different. In the camera's viewpoint, the near side has a larger viewing angle and therefore appears longer in the image. Therefore, we only need to apply a perspective transformation to the image to make the far side extend to match the length of the near side. According to the principle of perspective, the image size follows the similarity of triangles in the field of view. The original baselines of the far and near sides form two isosceles triangles with equal baselines but unequal leg lengths, such as... Figure 5 As shown. Based on the similarity of perspective triangles, the furthest black baseline in the image has the same length as the nearest original baseline. Therefore, by using perspective transformation, we can enlarge the furthest side to match the length of the black side, thus completing the perspective transformation of the image.

[0149] In the embodiments and other possible embodiments of this disclosure, the first axial distance and the second axial distance between the two sides (left and right sides) of the cropped eye image corresponding to the left or right eye and the depth camera are LN and LF, respectively. However, since the embodiments of this disclosure only obtain the spatial position of the eye center point of the cropped eye image corresponding to the left or right eye, we use the spatial position of the eye center point combined with the distance difference D between the far and near sides (the two sides of the cropped eye image corresponding to the left or right eye) and the position of the depth camera to calculate LN and LF. Finally, using the similarity of triangles, the magnification ratio of the far side in the image perspective transformation can be determined, that is, the perspective transformation ratio between the length of the farthest black side and its adjacent side (Equation 2). The vertical direction process of the perspective transformation is the same. Wherein, Z represents the spatial axial distance between the depth camera and the eye image corresponding to the left or right eye.

[0150]

[0151] Figure 6 A schematic diagram illustrating the perspective transformation process of an eye image according to an embodiment of the present disclosure is shown. Figure 6The diagram illustrates the perspective transformation process, particularly in the horizontal direction, represented by solid and dashed lines. After extracting eye images based on predicted bounding boxes (detection boxes) from the original image (a color image acquired by a depth camera), a perspective transformation is applied to correct distortion caused by distant edges. This yields the corrected image (the corrected eye image). Subsequently, an image of a specific size is cropped again (cropped eye images corresponding to the left and / or right eyes) (represented by black dashed boxes) for further calculations. The corrected image (the corrected eye image) is equivalent to being cropped from the corrected original image (the color image acquired by the depth camera). The shadows in the color image acquired by the depth camera represent the perspective effect on the plane containing the eyes and mouth.

[0152] in, Figure 6 The left image shows a tilted plane in the original image, where the lines connecting the eyes and mouth are not horizontal. Conversely, the right image (the corrected eye image) after perspective transformation depicts a parallel plane, with the planes containing the eyes and mouth parallel to each other. Therefore, perspective transformation effectively eliminates head pose by aligning the facial planes.

[0153] Step S105: Using the depth image and the detection boxes corresponding to the eyes and mouth, determine the position and pose information corresponding to the eyes and head.

[0154] In embodiments and other possible embodiments of this disclosure, the eye coordinates are configured as (x, y, z); where L represents the left eye and R represents the right eye. The attitude information is configured as (α, γ); where α represents the yaw angle and γ represents the pitch angle. The position is configured as the eye position, and the attitude is configured as the head attitude.

[0155] In embodiments and other possible embodiments of this disclosure, both eyes are labeled with the same classification tag. If the head rotation in the color image during the experiment is less than a set angle (e.g., 180 degrees), the first target in the horizontal coordinate is labeled as the left eye, and the second target in the horizontal coordinate is labeled as the right eye. The horizontal coordinate of the first target is greater than the horizontal coordinate of the second target.

[0156] In embodiments of this disclosure, Figure 4 A flowchart illustrating a specific implementation of a method for calculating the position and posture of the eyes and mouth according to an embodiment of this disclosure is shown. Figure 4As shown, the method for determining the position and pose information of the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth includes: aligning the depth image with the color image; generating a point cloud array corresponding to the aligned depth image; determining the position of the eyes and mouth (mouth) based on the index of the centroid unit in the point cloud array and the center of the detection boxes corresponding to the eyes and mouth; calculating the normal vector of the plane where the eyes and mouth (mouth) are located based on the position of the eyes and mouth (mouth), and converting the normal vector into a rotation vector to obtain pose information.

[0157] In embodiments and other possible embodiments of this disclosure, after obtaining the detection results of the eyes and mouth (the detection boxes corresponding to the eyes and mouth), the depth image generated by the depth camera is aligned with the RGB image (color image), such as... Figure 4 As shown, a point cloud array is generated by combining the camera's intrinsic parameters. This point cloud array represents the spatial position of each pixel in the camera's 3D space, corresponding to a point of an object in the scene. The point cloud is structured as a two-dimensional array, the same size as the original RGB image. Each element is a unit, corresponding to the spatial position (x, y, z) of the object point at that pixel.

[0158] In embodiments and other possible embodiments of this disclosure, the spatial positions of the centers of the three objects (detection boxes corresponding to the eyes and mouth) in the detection results and the indices of the centroid cells in the point cloud array are used to obtain the spatial positions of these three object centers. The plane formed by these three center points can be regarded as the facial plane. By using these three points, the normal vector of the plane can be calculated, which can be used as a substitute for the three-dimensional head pose. Then, we use the Rodrigues transform to convert the normal vector into a rotation vector, thereby allowing the components of the vector in three directions to be obtained.

[0159] These components can be considered as the yaw angle, pitch angle, and roll angle of the head in formula (3). Figure 4 (α, γ, and θ in the text).

[0160]

[0161] in, The normal vectors of the plane are [x1,y1,z1], [x2,y2,z2], and [x3,y3,z3], which are the spatial coordinates of the center points of the left eye, right eye, and mouth, respectively.

[0162] Step S106: Based on the preset viewpoint prediction model, perform viewpoint prediction using the position and pose information and the corrected eye image.

[0163] In embodiments of this disclosure, before performing viewpoint prediction based on a preset viewpoint prediction model using the position and pose information and the corrected eye image, the method further includes: training the preset viewpoint prediction model according to preset training rules using the corrected eye images and position and pose information of the training set to obtain a trained preset viewpoint prediction model; and then, performing viewpoint prediction based on the trained preset viewpoint prediction model using the position and pose information and the corrected eye image.

[0164] In the embodiments and other possible embodiments of this disclosure, Figure 3 (b) demonstrates the training method for the viewpoint prediction model Gaze-Point-Net. Training data was collected through volunteer participation, including standardized binocular images (eye images / corrected eye images) and their corresponding 3D information (position and pose information) computed using image depth. These datasets were assembled and used to train a general viewpoint prediction model.

[0165] In embodiments of this disclosure, the method for training a preset viewpoint prediction model according to preset training rules to obtain a trained preset viewpoint prediction model includes: obtaining a set number of training iterations; and stopping training the preset viewpoint prediction model when the set number of training iterations is reached, thereby obtaining a trained preset viewpoint prediction model. Those skilled in the art can configure the set number of training iterations according to actual needs.

[0166] In the embodiments of this disclosure, before performing viewpoint prediction based on the trained preset viewpoint prediction model using the position and pose information and the corrected eye image, the method further includes: calibrating the output layer of the trained preset viewpoint prediction model according to a preset calibration rule using the corrected eye image and position and pose information of the calibration set, to obtain a calibrated preset viewpoint prediction model; and then, performing viewpoint prediction based on the calibrated preset viewpoint prediction model using the position and pose information and the corrected eye image.

[0167] In embodiments and other possible embodiments of this disclosure, the preset viewpoint prediction model is trained on a large dataset, and to further improve its adaptability to each user, this disclosure designs a calibration method. Figure 3 (c) This method can improve the accuracy of viewpoint prediction models and allow the same model to be deployed on screens of different sizes to maintain consistency and accuracy even when the camera position may be unstable.

[0168] In embodiments of this disclosure, the method for calibrating the output layer of the trained preset viewpoint prediction model according to preset calibration rules to obtain a calibrated preset viewpoint prediction model includes: obtaining a loss value of the viewpoint coordinates and their corresponding viewpoint coordinate labels obtained during the calibration process using a preset loss value; and stopping the calibration of the preset viewpoint prediction model when the loss value reaches the preset loss value, thereby obtaining a calibrated preset viewpoint prediction model. Those skilled in the art can configure the preset loss value according to actual needs.

[0169] In the embodiments and other possible embodiments of this disclosure, a data collection and preparation process adapted to the needs of the project is required for training the multimodal viewpoint prediction model (preset viewpoint prediction model). During this process, authorization and ethical approval were obtained, and the legality of the data collection and preparation procedures was ensured through laboratory policies. Furthermore, more than twenty volunteers were invited to participate in data collection and preparation, and their full informed consent was ensured. Throughout the data collection and preparation process, this disclosure strictly adheres to relevant laws, regulations, and ethical guidelines, ensuring the protection of participants' privacy.

[0170] In embodiments of this disclosure and other possible embodiments, the data collection method includes fixing the position of the sampling camera and adjusting lighting conditions to control ambient light. We then begin moving gray target blocks in a serpentine pattern, starting from the top left corner of the display screen. As participants gaze at one of the target points, we capture data using the camera, and the data is processed and stored using algorithms. We collected a total of 749 data samples in each data collection task (i.e., per participant).

[0171] In embodiments and other possible embodiments of this disclosure, conventional gaze tracking tasks are typically performed with the head fixed, thus conventional methods do not require consideration of the head's spatial position and pose information. Since the head's six degrees of freedom are fixed, the neural network models used in conventional methods only accept binocular images or use a monocular computation model, and compute binocular images (left and right eye images) using a mirror-flipping method.

[0172] In the embodiments and other possible embodiments of this disclosure, to address the asymmetry between pupil direction and eyelid features in binocular images, this disclosure designs a preset viewpoint prediction model, Gaze-Point-Net, based on LeNet. Due to the differences between binocular images, feature extractors cannot use the same parameters for feature extraction. Therefore, embodiments of this disclosure employ a dual-channel convolutional neural network feature extractor with non-shared parameters for extracting features from binocular images, and use multiple fully connected layers for feature compression and extraction. Furthermore, by connecting (concatenating) the spatial position and pose vectors of the head in the hidden layers, a multimodal neural network (the preset viewpoint prediction model Gaze-Point-Net) is constructed to complete the multimodal gaze position prediction task.

[0173] Figure 7 A schematic diagram of the network structure of a preset viewpoint prediction model according to an embodiment of this disclosure is shown. Figure 7 As shown, the preset viewpoint prediction model includes: a first feature extraction path and a second feature extraction path, a first stitching layer connected to the first feature extraction path and the second feature extraction path respectively, a fully connected layer for feature compression connected to the stitching layer, a second stitching layer for stitching the position and pose information connected to the fully connected layer, and a classification layer connected to the second stitching layer; using the first feature extraction path and the second feature extraction path, features are extracted from the left-eye image and the right-eye image of the eye image respectively to obtain a first feature map and a second feature map; using the first stitching layer, the first feature map and the second feature map are stitched together to obtain a stitched feature map; using the fully connected layer, the stitched feature map is compressed to obtain compressed features; using the classification layer and the compressed features, the viewpoint coordinates of the left eye and the right eye are obtained respectively, completing the viewpoint prediction. Figure 7 The facial images in the image are in color. Figure 7 The numbers in the diagram represent the dimensions of the feature map. For example, the dimensions of the first feature map and the second feature map are 20×9×15, respectively.

[0174] like Figure 7As shown in the embodiments of this disclosure, the first feature extraction path includes: a first convolutional layer and a first pooling layer connected thereto, a first attention mechanism module connected to the first pooling layer, a second convolutional layer connected to the first attention mechanism module and the first pooling layer, and a second pooling layer connected thereto; using the first convolutional layer and the first pooling layer connected thereto, feature extraction and pooling are performed on the left eye image in the eye image to obtain a first feature map to be processed; using the first attention mechanism module, attention extraction is performed on the first feature map to be processed to obtain a corresponding first weighted feature map; based on the first feature map to be processed and the first weighted feature map, the first feature map is obtained.

[0175] like Figure 7 As shown in the embodiments of this disclosure, the method for obtaining the first feature map based on the first feature map to be processed and the first weighted feature map includes: performing a multiplication operation on the first feature map to be processed and the first weighted feature map to obtain the first feature map.

[0176] like Figure 7 As shown in the embodiments of this disclosure, the first feature extraction path includes: a third convolutional layer and a third pooling layer connected thereto, a second attention mechanism module connected to the third pooling layer, a fourth convolutional layer connected to the second attention mechanism module and the third pooling layer, and a fourth pooling layer connected thereto; using the third convolutional layer and the third pooling layer connected thereto, feature extraction and pooling are performed on the right eye image in the eye image to obtain a second feature map to be processed; using the second attention mechanism module, attention extraction is performed on the second feature map to be processed to obtain a corresponding second weighted feature map; based on the second feature map to be processed and the second weighted feature map, the second feature map is obtained.

[0177] like Figure 7 As shown in the embodiments of this disclosure, the method for obtaining the second feature map based on the second feature map to be processed and the second weighted feature map includes: performing a multiplication operation on the second feature map to be processed and the second weighted feature map to obtain the second feature map.

[0178] like Figure 7 As shown in the embodiments of this disclosure, the fully connected layer includes: a first fully connected layer and a second fully connected layer connected thereto; the first fully connected layer and the second fully connected layer connected thereto are used to compress the spliced ​​feature map to obtain compressed features. The classification layer is configured as a regression layer.

[0179] In embodiments and other possible embodiments of this disclosure, the neural network architecture corresponding to the viewpoint prediction model proposed in this disclosure includes: SELayer (attention mechanism, i.e., a first attention mechanism module and a second attention mechanism module) and GAZE-attention (a first feature extraction path and a second feature extraction path) modules. SELayer is a module used to enhance the feature representation capability of the neural network. It receives the input feature map and performs adaptive average pooling to obtain a global feature vector. Then, after processing by two fully connected layers and an activation function, SELayer generates a weight vector for feature scaling of the input feature map. SELayer includes an adaptive average pooling layer, two fully connected layers connected to the adaptive average pooling layer, a ReLU activation function connected to the last fully connected layer of the two fully connected layers for constructing a nonlinear combination, and a Sigmoid activation function connected to the activation function for parameter range limitation.

[0180] In embodiments and other possible implementations of this disclosure, GAZE-attention is a neural network consisting of convolutional layers and fully connected layers. Its input includes two monochromatic eye images (left and right eye images) and position and pose information (position and orientation information), with all images (left and right eye images) being 36×60 pixels in size. The output is a 2D tensor representing the coordinates of the gaze point.

[0181] The convolutional layers of this neural network employ a dual-channel convolutional neural network structure with non-shared parameters (a first feature extraction path and a second feature extraction path). Each channel (the first and second feature extraction paths) contains two convolutional modules. The first convolutional module (the first and third convolutional layers) consists of 20 convolutional kernels of a predetermined number and size (5×5), with a stride configured as a predetermined stride (e.g., 1) and padding configured as a predetermined padding value (e.g., 2) to ensure that the feature size remains constant. This is followed by a batch normalization layer, a ReLU activation function, and a 2×2 max-pooling layer. After the first pooling operation, a previously defined attention mechanism (i.e., the first and second attention mechanism modules) is introduced to enhance the neural network's attention to the input data. This mechanism adaptively weights each channel, allowing the neural network to focus more on extracting important features and ignore irrelevant features. The second convolutional module (the second and fourth convolutional layers) consists of a second set number and a second set size (e.g., 50 5×5 kernels), with a stride configured as a second set stride (e.g., 1) and padding configured as a second set padding value (e.g., 2×2) to ensure that the feature size remains constant. This is followed by a batch normalization layer, a ReLU activation function, and a max-pooling layer of size 2.

[0182] In embodiments of this disclosure, the method further includes: a method for determining a loss function for training and / or calibration, comprising: calculating a first norm corresponding to the X coordinate and the true X coordinate in each predicted viewpoint coordinate during training and / or calibration; calculating a second norm corresponding to the Y coordinate and the true Y coordinate in each predicted viewpoint coordinate during training and / or calibration; calculating a third norm corresponding to the squared value of the Y coordinate and the squared value of the true Y coordinate in each predicted viewpoint coordinate during training and / or calibration; calculating a fourth norm corresponding to the true viewpoint coordinate in each predicted viewpoint coordinate during training and / or calibration; and determining a loss function for training and / or calibration based on the first norm and its corresponding first weight coefficient, the second norm and its corresponding second weight coefficient, the third norm and its corresponding third weight coefficient, and the fourth norm and its corresponding fourth weight coefficient.

[0183] In the embodiments disclosed herein, the first weighting coefficient, the second weighting coefficient, the third weighting coefficient, and the fourth weighting coefficient are configured as 2, 2, 1, and 4, respectively.

[0184] In the embodiments and other possible embodiments of this disclosure, during the training of the preset viewpoint prediction model, the y-coordinate of the predicted viewpoint (the predicted viewpoint coordinates) of the preset viewpoint prediction model is too small. Therefore, this disclosure redesigns the loss function. The prediction results and labels of the embodiments of this disclosure are 2D coordinates c1 = (x1, y1) and c2 = (x2, y2), respectively. Our loss function includes the Euclidean distance of the x-coordinate, the Euclidean distance of the y-coordinate, the square of the Euclidean distance of the y-coordinate, and the Euclidean distance between the two points on the plane, which are weighted in a ratio of 2:2:1:4 (Formula (4)). Since a convex function is obtained when calculating the square of the Euclidean distance of the y-coordinate, we add error weights to the data close to 1 to effectively correct the model.

[0185] L(c1,c2)=2||x1-x2||+2||y1-y2||+||y1^2-y2^2||+4||(x1,y1)-(x2,y2)||(4)

[0186] In embodiments and other possible implementations of this disclosure, the model is trained using manually collected training data. First, the training data is randomly divided into a training set, a validation set, and a test set in a ratio of 8:2:1. During training, the training epochs are set to 50, the batch size to 16, and a redesigned loss function is used as the objective function. To optimize this objective function, a stochastic gradient descent (SGD) optimizer is employed with an initial learning rate of 0.0001 and a dynamic learning rate strategy. Specifically, the learning rate is gradually reduced by a factor of 0.1 at the 20th and 30th epochs. After each batch of training, the mean absolute error is calculated using the validation set to determine whether to update the optimal model.

[0187] In the embodiments and other possible embodiments of this disclosure, the preset viewpoint prediction model Gaze-Point-Net is trained on data from all volunteers. Therefore, through a calibration process, the accuracy and performance of the model when facing specific user conditions can be further improved. To this end, the following calibration steps are designed in this disclosure.

[0188] First, calibration data is collected using a method similar to that used for training data. Calibration data is collected by sequentially presenting nine gray boxes arranged in two rectangles on the display screen. The collected data includes standardized binocular images, head position and pose vectors, and corresponding label data. Next, the collected data is assembled into a batch of calibration datasets, and the model is fine-tuned using the same loss function and optimizer as in the training process. During the calibration of the preset viewpoint prediction model, to prevent overfitting at the nine calibration points, the parameters of only the last fully connected layer are adjusted for model fine-tuning, which further improves the model's accuracy and performance under specific user conditions.

[0189] In the embodiments of this disclosure, the method further includes: filtering the multiple viewpoint coordinates predicted for adjacent set frame numbers to obtain the final viewpoint coordinates.

[0190] In the embodiments of this disclosure, the method for filtering multiple viewpoint coordinates predicted for adjacent set frame numbers includes: averaging the X coordinates among the multiple viewpoint coordinates predicted for adjacent set frame numbers to obtain the final X coordinates among the viewpoint coordinates; and averaging the Y coordinates among the multiple viewpoint coordinates predicted for adjacent set frame numbers to obtain the final Y coordinates among the viewpoint coordinates.

[0191] Furthermore, in embodiments of this disclosure and other possible embodiments, a mean filter is applied to the predicted gaze position. Specifically, the average of the last five predicted gaze positions is taken as the position of the current gaze, and the filtered result is compared with the unfiltered result.

[0192] In the embodiments and other possible embodiments of this disclosure, the following four experiments were designed: sequential and random point testing, trajectory tracking testing, and browsing pattern analysis. These experiments comprehensively analyzed the system's accuracy, robustness, and real-time performance.

[0193] In the sequential point localization test, the design aimed to track the movement of a gray square in a curved pattern at the top left corner of the screen. Real-time viewpoint prediction was achieved using a viewpoint prediction system. With the participants' consent, binocular images, facial images, head position, head pose information, and standardized screen position information of the gray square were recorded for data analysis.

[0194] Similarly, in the random point localization test, a gray square was used as a fixed target. The target jumped randomly to a specific location and remained stationary for three seconds. To eliminate the influence of rapid eye movement, data recording began one second after the square came to a stop.

[0195] The fixed-target tasks in these experiments are very similar to the primary tasks of current eye-tracking devices, allowing us to accurately evaluate the performance of eye-tracking systems. Data analysis shows that the system significantly outperforms moving-target tasks in fixed-gaze target tasks.

[0196] To evaluate the system's ability to track moving targets, a circular trajectory tracking test was designed. A circular track with a width of 500 pixels was drawn in the center of the screen, with a moving square as the gaze target. The gaze point was predicted and recorded during the test.

[0197] Finally, the performance of the gaze tracking system was demonstrated through a browsing mode analysis task. Five images containing highly salient objects were selected, each displayed in full screen for five seconds. Simultaneously, the viewer's gaze position sequence was captured and visualized as gaze point trajectories and heatmaps to analyze and demonstrate the performance of the preset viewpoint prediction model. By combining the captured gaze points with the positions of target objects in the images, the reliability, real-time performance, and accuracy of the preset viewpoint prediction model were analyzed.

[0198] Figure 8 A schematic diagram illustrating the camera coordinate system and viewpoint prediction according to an embodiment of the present disclosure is shown. Figure 8As shown, after calculating the normalized coordinates of the gaze point, we transform them back to a 1920×1080 screen position to obtain the final projection of the gaze point on the screen, called the gaze point position. Simultaneously, we retain the corresponding true coordinates of each result, i.e., the center position of the experimental block, as a label. After obtaining the predicted and true coordinates, we calculate the actual distance between two points on the screen, taking into account the screen size. By obtaining the spatial distance between the two points, we use the eye spatial position coordinates in the camera coordinate system, combined with the relative positions of the camera and the screen, to calculate the spatial distance between the midpoint of the two eye rays and the positions of the two points. Finally, we use the law of cosines to calculate the angular error between our predicted and true directions, and perform error analysis using various variables.

[0199] Simultaneously, with the participation of over 20 volunteers, a total of 14,820 data samples were collected and processed. These samples can be used to train the gaze prediction model. Furthermore, this disclosure analyzes the distribution of the dataset across different feature dimensions, such as mean and range of brightness, volunteer location, and volunteer head tilt angle.

[0200] Figure 9 A distribution diagram of the viewpoint prediction training dataset according to embodiments of the present disclosure is shown. Figure 9 As shown, (a) the dataset distribution is divided by the RGB average value; (b) the dataset distribution is divided by the RGB range value; (c) the dataset distribution is divided by the distance between the volunteer and the camera; and (d) the dataset distribution is divided by the volunteer's head tilt angle. The RGB average distribution of the eye image is shown below. Figure 9 As shown in (a), the brightness distribution of the image is illustrated. We collected data during both day and night, adjusting the lighting conditions on-site. Therefore, the average brightness distribution of the image spans two ranges, covering most lighting conditions from 70 to 170. Thus, the data collection effectively covered a wide range of lighting conditions. Figure 9 (b) shows the distribution of the RGB range, revealing a relatively balanced distribution. This statistical analysis reveals a wide distribution of light source locations due to the significant difference in RGB range between forward and side lighting. Figure 9 c) illustrates the spatial distribution of volunteer and sampling camera locations, primarily concentrated within a range of 0.6 meters to 1.2 meters. This range essentially covers most locations commonly encountered when using a computer under normal conditions. Figure 9(d) illustrates the distribution of head posture deflection angles among volunteers. Under our guidance and encouragement, volunteers performed gaze by combining eye rotation and head movement. The head rotation angles ranged from 0 to 50 degrees, exhibiting a relatively balanced distribution. Through statistical analysis of the training data features, our collected dataset covers the most typical scenarios, including various lighting conditions, locations, and poses. By utilizing this uniformly distributed dataset, the trained model demonstrated stronger generalization ability and robustness.

[0201] Furthermore, in the embodiments of this disclosure, the strong YOLOv5 model detects eye and mouth targets, demonstrating its ability to accurately identify eye and mouth targets with high accuracy. Therefore, this model performs well in detection tasks at different distances and against various backgrounds, making it suitable for the target localization task of this disclosure.

[0202] Figure 10 Performance analysis of a predictive target detection model (enhanced YOLOv5) for detecting eyes and mouth according to embodiments of this disclosure is shown. Figure 10 As shown, (a) confidence-recall plot; (b) confidence-precision plot; (c) confidence-F1 score plot; (d) recall-precision and mAP plot at 0.5.

[0203] Figure 10 The recall and precision curves in the figure demonstrate the model's high performance across most confidence intervals. The F1 curve shows that the model achieves the best overall performance with an F1 score of 0.91 at a confidence threshold of 0.408. The mAP@50 curve illustrates the model's accuracy at a confidence threshold of 0.5. In this case, the model achieves a precision of 0.918 for eye detection and 0.868 for mouth detection, demonstrating its effectiveness in object detection tasks. Finally, this disclosure successfully trains an enhanced YOLOv5 object detection model for face images. This model can output the center point parameters and bounding box size parameters for each detected object, which can be used for subsequent analysis.

[0204] In embodiments and other possible embodiments of this disclosure, continuous point tests were performed and a total of 6572 sets of experimental data were collected. These test results, along with error analysis results performed at multiple scales, are presented in… Figure 11 and Figure 12 In the error analysis, we considered various variables such as screen gaze position, head spatial position, head spatial pose, and lighting conditions. This comprehensive analysis allowed us to gain a deeper understanding of the factors contributing to errors in the experimental data. By combining continuous point testing with comprehensive error analysis, we were able to fully understand the performance and limitations of the proposed method under different conditions.

[0205] Figure 11 This illustrates the performance analysis of a preset viewpoint prediction model (Gaze-Point-Net) with different variables according to embodiments of the present disclosure. Figure 11 As shown, (a) error analysis of the horizontal coordinates of the target point; (b) error analysis of the vertical coordinates of the target point; (c) error analysis of the horizontal and vertical coordinates of the target point; (d) error analysis of the horizontal spatial position of the volunteer's head; (e) error analysis of the vertical spatial position of the volunteer's head; (f) error analysis of the spatial position of the volunteer's head axis; (g) error analysis of the pitch angle of the volunteer's head; and (h) error analysis of the yaw angle of the volunteer's head.

[0206] like Figure 11 As shown, the error analysis results for various variables are presented. The horizontal axis represents the variables, while the vertical axis represents the average angular error within the corresponding horizontal axis interval. Figure 11 (a) and Figure 11 (b) shows the relationship between the horizontal and vertical positions of the target point on the screen and the angular error. Figure 11 (c) A 3D graph showing the relationship between the target point's position on the screen and the angular error is displayed. The x-axis and y-axis represent the target point's horizontal and vertical positions on the screen, respectively, while the z-axis represents the magnitude of the angular error. The angular error ranges from 6 to 8 degrees. Figure 11 (d) illustrates the relationship between angular error and the horizontal distance between the observer's gaze point and the camera. The independent variable is the horizontal position of the midpoint between the eyes in the camera coordinate system, representing the observer's horizontal displacement. The horizontal distance ranges from -0.1 to 0.15 meters, and the error ranges from 4 to 8 degrees. Figure 11 (e) illustrates the relationship between angular error and the vertical distance between the observer's gaze point and the camera. The independent variable is the vertical position of the midpoint between the eyes in the camera coordinate system, representing the observer's vertical displacement. The vertical distance values ​​range from -0.15 to 0 meters, while the angular error ranges from 5 to 10 degrees. Figure 11 (f) presents the relationship between angular error and the axial distance between the observer's gaze point and the camera. The independent variable is the axial position of the midpoint between the eyes in the camera coordinate system, representing the observer's forward / backward displacement and the distance between the observer and the display screen. The axial distance ranges from 0.5 to 0.85 meters, while the angular error ranges from 4 to 8 degrees. Finally, Figure 11 (g) shows the relationship between angular error and the pitch angle of the observer's head, representing the vertical swing amplitude. The pitch angle ranges from -70 to 20 degrees, while the angular error ranges from 7 to 9 degrees. Figure 11 (h) shows the relationship between the angle error and the yaw angle of the observer's head, representing the horizontal sway. The yaw angle ranges from -10 to 30 degrees, while the angle error ranges from 5 to 8 degrees.

[0207] Figure 12 The performance analysis of a preset viewpoint prediction model (Gaze-Point-Net) under different lighting conditions according to embodiments of the present disclosure is shown. Figure 12 The diagram illustrates the relationship between the mean and range of GB values ​​and angular error. The mean RGB values ​​represent the intensity of the ambient light in the experimental environment, ranging from 40 to 120 degrees. The range of RGB values ​​represents the angle of the laboratory light source, with larger light angles corresponding to a larger RGB range, ranging from 80 to 200 degrees. In both cases, the error is within the range of 5 to 10 degrees.

[0208] This publication collects and analyzes the results of random site testing. Figure 13 The results of random point tests according to embodiments of this disclosure are shown. Figure 13 As shown, a graph plots the point distribution and prediction standard deviation using actual and predicted values. Figure 13 In the left-hand chart, the large light-colored circle represents the fixed target point in the test, centered on the center of the target block, with a diameter equal to the diagonal length of the target block. The small dark-colored circle represents the scattering of the model's predicted gaze points. Figure 13 In the chart on the right, the small dark circles represent the actual fixed gaze points, while the large light-colored circles surrounding them represent the range of predicted points corresponding to the labels. The chart shows the distribution of predicted points around the ground truth and displays the standard deviation of the predictions.

[0209] Figure 14 The results of data filtering and model calibration analysis according to embodiments of this disclosure are shown. During the experiment, we retained the prediction data from the uncalibrated model and the unfiltered results, obtaining a total of four sets of data. These data are... Figure 14 The left and right graphs illustrate the functional relationship between angular error and the true value on the X-axis, with scatter points representing the data distribution. The scatter points are plotted at a scale of 1:100. The four curves in the graph represent the quadratic polynomial fit for each dataset, illustrating how the average angular error changes with the x-coordinate. The right graph shows a similar error analysis, with the Y-axis representing the true value. It can be observed from the graphs that filtering has little effect on reducing the average error, while calibration significantly reduces it.

[0210] To analyze the performance of the models, we conducted a series of controlled experiments and tested the following mathematical models: (1) K-Nearest Neighbors (KNN). In the field of machine learning, KNN is commonly used for pattern recognition tasks. We expanded the image into a one-dimensional vector and concatenated it with head position and pose information, and then tested it using the KNN method with the number of neighbors set to 3. (2) Random Forest (RF). RF is an effective regression method in the field of machine learning. We tested it using the same data processing method, using 300 decision trees with a maximum depth of 20 per node and a maximum number of features of 65 per node. (3) Linear Regression (LR). LR is a simple regression method in the field of machine learning. We tested it using the same data processing method and linear regression.

[0211] Finally, we used various models, including KNN, RL, LR, gaze point network (GPN), and a calibrated GPN, to predict random points. The results of different models were compared, and... Figure 15 The error analysis results are presented in the paper.

[0212] Figure 15 This document illustrates a performance comparison analysis of Gaze-Point-Net with other models according to embodiments of the present disclosure. Figure 15 As shown, (a) scatter plot and fitting curve analysis of the x-coordinates of the target points among multiple models; (b) scatter plot and fitting curve analysis of the y-coordinates of the target points among multiple models; (c) analysis of the mean and standard deviation of the angle errors of multiple models; (d) analysis of the mean and standard deviation of the pixel errors of multiple models.

[0213] Figure 15 (a) shows the error analysis plot with the true value on the x-axis, which is compared with... Figure 14 They share the same pattern, including error scatter plots and fitted curves. On the other hand, Figure 15 (b) shows the error analysis plot with the true value on the y-axis. The scatter points represent the error distribution of the sampled data at a certain proportion, and these scatter points are fitted into a curve using a quadratic function. Blue, cyan, and yellow represent the results of KNN, LR, and RF, respectively. Pink represents the results of the uncalibrated model, while red represents the results of the calibrated model with data filtering. It can be observed that our model has a significantly smaller error distribution and a lower standard deviation. Furthermore, the filtering operation improves the model's performance to a small extent. Additionally, Figure 15 (c) and Figure 15(d) Presents a numerical comparison of the average angle error and average pixel error of the aforementioned models. These two figures, from left to right, represent the KNN, RF, LR, general GPN, and calibrated GPN models, with colors gradually decreasing in color. The results show that our model performs comparably to RF before calibration, but is significantly improved after calibration.

[0214] Table 1. Pixel and angle errors of calibrated Gaze-Point-Net and its corresponding devices

[0215]

[0216] Table 1 presents the mean and standard deviation of pixel and angle errors for different models. The screen has a pixel range of 2202 and a visible viewing angle range of 60 to 70 degrees. The calibrated GPN also includes data filtering operations. Our calibrated GPN achieves a pixel error of 229.60 and an angle error of 5.40, which are significantly smaller than those of KNN, RF, LR, and Gaze-Point-Net.

[0217] Table 2. Evaluation metrics for various regression tasks of Gaze-Point-Net and its corresponding programs

[0218]

[0219] The performance of the various models used in this study was analyzed using multiple regression task evaluation metrics, as shown in Table 2. The calibrated GPN also included data filtering operations. These metrics included root mean square error (RMSE), mean absolute error (MAE), R-squared (R²), mean absolute percentage error (MAPE), and mean square percentage error (MSPE).

[0220] Based on RMSE and MAE analysis, our model shows significantly smaller errors than other models. Furthermore, considering MAPE and MSPE, our model not only exhibits smaller relative errors but also reduces the likelihood of large errors. Additionally, the R² coefficient indicates a stronger correlation between our model's regression results and the reference standard.

[0221] Meanwhile, the relationship between the true values' horizontal and vertical coordinates on the screen and the angle error shows that the proposed model's prediction error is within 5° at most locations. After further processing, including removing obviously incorrect prediction points caused by system latency, the average prediction error reaches 5.13°. This demonstrates that the model can make relatively good predictions for most gaze tracking tasks.

[0222] Similarly, a method similar to that used in classification tasks can be employed for analysis. A confidence radius of a specific size is defined, centered on the target point's location. Predicted points falling within this confidence circle are considered correct predictions, while those falling outside are considered incorrect predictions. By adjusting the size of the confidence radius, we can obtain a curve describing the relationship between the confidence radius and accuracy. Figure 16 An analysis of the impact of confidence level on accuracy is shown according to embodiments of this disclosure. For example... Figure 16 As shown, the relationship between the confidence radius and accuracy is illustrated, and the confidence radius at an accuracy of 0.8 is calculated. Figure 16 The left figure shows the accuracy curve of the Euclidean distance between the predicted point and the target point, with a confidence radius of 320 and an accuracy of 0.8. Figure 16 The right figure shows the accuracy curves of the horizontal and vertical distances between the predicted point and the target point, with confidence radii of 190 and 250, respectively, and an accuracy of 0.8.

[0223] Figure 17 The results of a trajectory tracking test according to an embodiment of this disclosure are shown. Figure 17 The left side shows a heatmap of the prediction error distribution. This heatmap depicts the trajectory of a moving target, with brighter colors representing larger errors and darker colors representing smaller errors. Figure 17 The right side shows the distribution of scatter points within a specific confidence interval. The light-colored rings in the background represent concentric circles centered on the trajectory of the moving target, with a radius of 300 pixels. The scatter plot shows the distribution of predicted points. By establishing points within a specific pixel range as the confidence radius for the effective trajectory, an accuracy of 93.75% was achieved. When the confidence radius was reduced to 200 pixels, the accuracy decreased to 81.91%.

[0224] The viewpoint prediction method can be executed by a viewpoint prediction device, such as a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the viewpoint prediction method can be implemented by a processor calling computer-readable instructions stored in memory.

[0225] Those skilled in the art will understand that in the above-described viewpoint prediction method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0226] Figure 18A block diagram of a viewpoint prediction apparatus according to an embodiment of the present disclosure is shown. Figure 18 As shown, the viewpoint prediction device includes: an acquisition unit 101, used to acquire a color image to be predicted and its corresponding depth image; a first determination unit 102, used to perform target detection on the color image using a preset target detection model, and determine the detection boxes corresponding to the eyes and mouth; a second determination unit 103, used to determine the eye image using the color image and the detection boxes corresponding to the eyes; a correction unit 104, used to perform a horizontal perspective transformation on the eye image to obtain a corrected eye image; a third determination unit 105, used to determine the position and posture information corresponding to the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth; and a prediction unit 106, used to perform viewpoint prediction based on the preset viewpoint prediction model, using the position and posture information and the corrected eye image. This can improve the accuracy and robustness of appearance-based viewpoint prediction without locking the head, thus solving the problem of poor accuracy and robustness of current appearance-based viewpoint prediction, which requires head locking.

[0227] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to execute the viewpoint prediction method described in the above viewpoint prediction method embodiments. The specific implementation can be referred to the description of the above viewpoint prediction method embodiments, which will not be repeated here for the sake of brevity.

[0228] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the aforementioned viewpoint prediction method. The computer-readable storage medium can be a non-volatile computer-readable storage medium. This method can improve the accuracy and robustness of appearance-based viewpoint prediction without requiring head locking, thus addressing the current problems of poor accuracy and robustness in appearance-based viewpoint prediction, which necessitates head locking.

[0229] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured for the aforementioned viewpoint prediction method. The electronic device can be provided as a terminal, a server, or other type of device. This method can improve the accuracy and robustness of appearance-based viewpoint prediction without requiring head locking, thus addressing the current problems of poor accuracy and robustness in appearance-based viewpoint prediction, which necessitates head locking.

[0230] This disclosure presents a viewpoint prediction method, apparatus, electronic device, and computer-readable storage medium, and tests its performance through various experiments. The feasibility of gaze prediction without relying on an infrared camera has been demonstrated, and further, a multimodal approach incorporating head spatial position and pose information can improve the accuracy and generalization performance of gaze prediction. Although methods utilizing this information for gaze prediction exist, we optimize the model for each computational process, thereby improving the system performance to some extent. Furthermore, during the training of the object detection model, we found that increasing the resolution of the YOLOv5 feature map improves the model's ability to recognize small objects. Additionally, merging similar but different object categories (left eye and right eye) into the same category significantly improves the detection performance for that category. Calibrating the gaze prediction model before each prediction task greatly improves its accuracy. Moreover, modifying only a subset of model parameters during calibration has a more significant effect than modifying the entire model. On the other hand, data filtering processes can improve the visual representation of predicted points to some extent, but they do not significantly improve the results. Finally, the proposed model outperforms the fixed-focus gaze prediction task (random point test) in the moving-point saccade task (sequential point test). This is because our random point testing procedure involves averaging the angular error and standard deviation of the prediction results after the random points have been fixed for a period of time, resulting in slightly lower errors than sequential point testing where the target points are continuously moved.

[0231] In this task, embodiments of this disclosure employ the YOLOv5 object detection model as the face marker localization model. This model is fast, accurate, and has excellent generalization capabilities, enabling effective object detection under various extreme conditions. Furthermore, embodiments of this disclosure employ an appearance-based viewpoint prediction method that does not rely on an infrared camera. Instead, embodiments of this disclosure rely on a depth camera to capture color and depth images. By combining the spatial locations of feature points calculated using depth information with the normal vectors of feature planes, and integrating rotation and perspective transformation processes, we significantly improve the system's accuracy and robustness to significant changes in head position and pose. Importantly, embodiments of this disclosure innovatively employ a direct regression method to calculate the gaze point position on the screen, bypassing coordinate transformations from the camera coordinate system to the world coordinate system and the projection of the gaze onto the screen. Embodiments of this disclosure also consider camera position errors, screen size and placement errors, and individual differences in system performance through calibration methods. Finally, the system is capable of capturing information from other modalities, such as sound, in real time, enabling future multimodal research applications.

[0232] Current methods involve conventional coordinate transformations and spatial computations, and the relationship between the gaze vectors before and after the transformation is non-linear. This poses a challenge when the relative positions of the sampling camera and the gaze screen change, as precise adjustments cannot be made using calibration processes with linear fully connected layers. The difficulty in calibrating parameters in traditional computational pipelines leads to difficulties in handling system errors in such cases.

[0233] To address this problem, this disclosure proposes a method for directly regressing the viewpoint position. This method does not rely on transformations from the camera coordinate system to the world coordinate system or further viewpoint position calculations; instead, it directly regresses and reconstructs the standardized gaze coordinates. This is achieved using a neural network that utilizes the output of a model trained using conventional methods as intermediate features. Through this neural network, the final result is directly calculated, significantly improving computational efficiency and accuracy.

[0234] Furthermore, fine-tuning methods can be used to calibrate and compensate for positional errors between the sampling camera and the screen, as well as individual differences. Instead of using a monocular model to compute and mirror the other eye as previous methods did, we directly use a binocular model. This approach uses two feature extractors with different parameters to compute viewpoint predictions, thereby reducing errors caused by differences between eyes.

[0235] In the Gaze-Point-Net model disclosed herein, an attention mechanism is incorporated into two feature extraction channels with different parameters, enabling more efficient feature learning during training. This improvement leads to higher training efficiency, faster convergence, and better generalization. Furthermore, the calibration process of this disclosure is efficient and requires only minimal sample data. During deployment, the model can be calibrated for each individual being tested. Leveraging the model's fast training speed and high accuracy, a coarse general-purpose model can be trained with only a small amount of training data, which can then be further refined into a private model through calibration.

[0236] This disclosure involves multiple experiments to test the performance of our system. Analysis revealed a slightly larger error in predicting the upper and lower boundary positions. Variables such as distance, angle, and lighting conditions have a relatively small impact on system performance. Furthermore, in terms of accuracy, it outperforms the scanning task in the fixed-fog-point prediction task. When comparing the performance of various models in the fixed-fog-point prediction task, the model disclosed herein achieved the best results among all models.

[0237] This disclosure also compares the performance of data filtering and model calibration. Due to the advantages of this disclosure, the calibration process significantly improves system performance, reducing the error level to approximately 5.4 degrees.

[0238] Figure 19 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, or other terminal.

[0239] Reference Figure 19 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0240] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0241] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0242] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0243] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0244] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0245] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0246] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0247] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0248] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0249] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0250] Figure 20 This is a block diagram illustrating an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. (Refer to...) Figure 20 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0251] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0252] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0253] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0254] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0255] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0256] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0257] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0258] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0259] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0260] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0261] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A viewpoint prediction method, characterized in that, include: Obtain the color image to be predicted and its corresponding depth image; Using a preset target detection model, target detection is performed on the color image to determine the detection boxes corresponding to the eyes and mouth; The eye image is determined using the color image and the detection box corresponding to the eye; Perform a horizontal perspective transformation on the eye image to obtain a corrected eye image; Using the depth image and the detection boxes corresponding to the eyes and mouth, the position and pose information of the eyes and head are determined; Based on a preset viewpoint prediction model, viewpoint prediction is performed using the position and posture information and the corrected eye image; The method for performing a horizontal perspective transformation on the eye image to obtain a corrected eye image includes: Calculate the first axial distance and the second axial distance between the two sides of the eye image corresponding to the left eye or right eye and the depth camera, respectively; Based on the first axial distance and the second axial distance, calculate the perspective transformation ratio corresponding to the eye image of the left eye or the right eye respectively; Based on the perspective transformation ratio of the eye image corresponding to the left or right eye, the eye image corresponding to the left or right eye is corrected to obtain the corrected eye image.

2. The viewpoint prediction method according to claim 1, characterized in that, The method for determining an eye image using the color image and the detection box corresponding to the eye includes: Calculate the lengths of the first and second diagonals of the detection frames for the left and right eyes, respectively. Based on a set multiplier, the left eye region and the right eye region are determined using the lengths of the first diagonal and the second diagonal, respectively. Based on the left eye region and the right eye region respectively, crop out the corresponding eye images of the left and right eyes from the color image; and Before performing a horizontal perspective transformation on the eye image to obtain the corrected eye image, the method further includes: Get the set pixel size; The pixels of the corresponding eye images for the left and right eyes are adjusted separately to achieve the set pixel size.

3. The viewpoint prediction method according to claim 1, characterized in that, The preset viewpoint prediction model includes: a first feature extraction path and a second feature extraction path, a first stitching layer connected to the first feature extraction path and the second feature extraction path respectively, a fully connected layer for feature compression connected to the stitching layer, a second stitching layer for stitching the position and pose information connected to the fully connected layer, and a classification layer connected to the second stitching layer. Using the first feature extraction path and the second feature extraction path, feature extraction is performed on the left eye image and the right eye image of the eye image respectively to obtain the first feature map and the second feature map; Using the first splicing layer, the first feature map and the second feature map are spliced ​​together to obtain a spliced ​​feature map; The fully connected layer is used to compress the stitched feature map to obtain compressed features; Using the classification layer and the compressed features, the viewpoint coordinates of the left and right eyes are obtained respectively, thus completing viewpoint prediction; and The first feature extraction path includes: a first convolutional layer and a first pooling layer connected thereto, a first attention mechanism module connected to the first pooling layer, a second convolutional layer connected to the first attention mechanism module and the first pooling layer, and a second pooling layer connected thereto. Using the first convolutional layer and the first pooling layer connected thereto, feature extraction and pooling are performed on the left eye image in the eye image to obtain the first feature map to be processed. Using the first attention mechanism module, attention is extracted from the first feature map to be processed to obtain the corresponding first weighted feature map; Based on the first feature map to be processed and the first weighted feature map, the first feature map is obtained; and The method for obtaining the first feature map based on the first feature map to be processed and the first weighted feature map includes: performing a multiplication operation on the first feature map to be processed and the first weighted feature map to obtain the first feature map; and The first feature extraction path includes: a third convolutional layer and a third pooling layer connected thereto, a second attention mechanism module connected to the third pooling layer, a fourth convolutional layer connected to the second attention mechanism module and the third pooling layer, and a fourth pooling layer connected thereto. Using the third convolutional layer and the third pooling layer connected thereto, feature extraction and pooling are performed on the right eye image in the eye image to obtain a second feature map to be processed. The second attention mechanism module is used to extract attention from the second feature map to be processed, thereby obtaining the corresponding second weighted feature map; The second feature map is obtained based on the second feature map to be processed and the second weighted feature map; and The method for obtaining the second feature map based on the second feature map to be processed and the second weighted feature map includes: performing a multiplication operation on the second feature map to be processed and the second weighted feature map to obtain the second feature map; and The fully connected layer includes: a first fully connected layer and a second fully connected layer connected thereto; Using the first fully connected layer and the second fully connected layer connected thereto, feature compression is performed on the stitched feature map to obtain compressed features; and The classification layer is configured as a regression layer.

4. The method according to claim 1, characterized in that, include: Before performing viewpoint prediction based on the preset viewpoint prediction model using the position and pose information and the corrected eye image, the method further includes: Using the corrected eye images and position and pose information of the training set, the preset viewpoint prediction model is trained according to the preset training rules to obtain the trained preset viewpoint prediction model. Furthermore, based on the trained preset viewpoint prediction model, viewpoint prediction is performed using the position and pose information and the corrected eye image; and The method for training a preset viewpoint prediction model according to preset training rules to obtain a trained preset viewpoint prediction model includes: Get the set number of training iterations; When the preset viewpoint prediction model has been trained a certain number of times, training of the preset viewpoint prediction model is stopped, and the trained preset viewpoint prediction model is obtained; and Before performing viewpoint prediction based on the trained preset viewpoint prediction model, using the position and pose information and the corrected eye image, the method further includes: Using the corrected eye images and position and pose information of the calibration set, the output layer of the trained preset viewpoint prediction model is calibrated according to the preset calibration rules to obtain the calibrated preset viewpoint prediction model. Furthermore, based on the calibrated preset viewpoint prediction model, viewpoint prediction is performed using the position and pose information and the corrected eye image; and The method for calibrating the output layer of the trained preset viewpoint prediction model according to preset calibration rules to obtain a calibrated preset viewpoint prediction model includes: Obtain the preset loss value; Calculate the loss value of the viewpoint coordinates and their corresponding viewpoint coordinate labels obtained during the calibration process; When the loss value reaches the preset loss value, the calibration of the preset viewpoint prediction model is stopped, and a calibrated preset viewpoint prediction model is obtained.

5. The method according to claim 4, characterized in that, Also includes: Determine the methods for training and / or calibrating the loss function, including: Calculate the first norm of the X-coordinate in each predicted viewpoint coordinate during training and / or calibration, corresponding to the true X-coordinate. Calculate the second norm of the Y-coordinate in each predicted viewpoint coordinate during training and / or calibration, corresponding to the true Y-coordinate. Calculate the third norm of the squared Y-coordinate value in each predicted viewpoint coordinate during training and / or calibration, corresponding to the squared Y-coordinate value of the true Y-coordinate. Calculate the fourth norm of the viewpoint coordinates predicted each time during training and / or calibration, corresponding to the true viewpoint coordinates; Based on the first norm and its corresponding first weight coefficient, the second norm and its corresponding second weight coefficient, the third norm and its corresponding third weight coefficient, and the fourth norm and its corresponding fourth weight coefficient, a loss function for training and / or calibration is determined; and The first weighting coefficient, the second weighting coefficient, the third weighting coefficient, and the fourth weighting coefficient are configured as 2, 2, 1, and 4, respectively.

6. The method according to claim 4, characterized in that, Also includes: The multiple viewpoint coordinates predicted for adjacent set frame numbers are filtered to obtain the final viewpoint coordinates. and The method for filtering multiple viewpoint coordinates predicted from adjacent set frame numbers includes: The X coordinates of the multiple viewpoint coordinates predicted for the adjacent set number of frames are averaged to obtain the final X coordinates of the viewpoint coordinates. The Y-coordinates of the multiple predicted viewpoint coordinates corresponding to the adjacent set number of frames are averaged to obtain the final Y-coordinates of the viewpoint coordinates; and The method for determining the position and pose information of the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth includes: aligning the depth image with the color image; generating a point cloud array corresponding to the aligned depth image; determining the position of the eyes and mouth based on the index of the centroid unit in the point cloud array and the center of the detection boxes corresponding to the eyes and mouth; calculating the normal vector of the plane where the eyes and mouth are located based on the position of the eyes and mouth, and converting the normal vector into a rotation vector to obtain pose information.

7. A viewpoint prediction device, characterized in that, include: The acquisition unit is used to acquire the color image to be predicted and its corresponding depth image; The first determining unit is used to perform target detection on the color image using a preset target detection model, and determine the detection boxes corresponding to the eyes and mouth; The second determining unit is used to determine an eye image using the color image and the detection box corresponding to the eye; The correction unit is used to perform a horizontal perspective transformation on the eye image to obtain a corrected eye image; The third determining unit is used to determine the position and posture information of the eyes and head using the depth image and the detection boxes corresponding to the eyes and mouth; The prediction unit is used to predict the viewpoint based on a preset viewpoint prediction model, using the position and posture information and the corrected eye image; The method for performing a horizontal perspective transformation on the eye image to obtain a corrected eye image includes: Calculate the first axial distance and the second axial distance between the two sides of the eye image corresponding to the left eye or right eye and the depth camera, respectively; Based on the first axial distance and the second axial distance, calculate the perspective transformation ratio corresponding to the eye image of the left eye or the right eye respectively; Based on the perspective transformation ratio of the eye image corresponding to the left or right eye, the eye image corresponding to the left or right eye is corrected to obtain the corrected eye image.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the viewpoint prediction method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the viewpoint prediction method according to any one of claims 1 to 7.

Images