Line-of-sight estimation system and line-of-sight estimation method

Abstract

A line-of-sight estimation system and a line-of-sight estimation method, the line-of-sight estimation method comprising deriving a scaled face image and a scaled eye image from an image of a subject captured via a camera device, extracting a first feature representation from the scaled face image using a first feature extraction model, extracting a second feature representation from the scaled eye image using a second feature extraction model, combining the first feature representation and the second feature representation to generate a combined feature representation, and determining a gaze direction of the subject based on the combined feature representation using a regression model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to image processing and image analysis techniques, and in particular, to a gaze estimation system and a gaze estimation method. BACKGROUND

[0002] In recent years, gaze estimation techniques have received increasing attention due to their potential applications in various fields. For example, in automotive systems, gaze estimation systems can be used to monitor the attention of drivers and detect signs of driver distraction, thereby improving driving safety. In digital signage systems, gaze estimation can help determine which areas of a display attract more attention from passersby, thereby providing valuable data for advertising strategies. In retail environments such as convenience stores and supermarkets, surveillance systems equipped with gaze estimation capabilities can identify products that most interest consumers.

[0003] Traditional gaze estimation systems typically rely on analyzing facial images captured by cameras to infer the gaze direction of a subject. However, facial images themselves contain many features unrelated to gaze direction (e.g., individual appearance differences, lighting variations, and background noise). These irrelevant features can interfere with the learning process of the gaze estimation model and reduce the accuracy of the estimation.

[0004] In addition, many existing methods use a single input source to extract gaze-related features without focusing on key areas (e.g., eye regions) separately. This approach can limit the model's ability to focus on obtaining information most relevant to gaze, especially under real-world conditions where lighting, head posture, and occlusions (e.g., sunglasses, glasses) vary frequently.

[0005] Therefore, there is a need for a gaze estimation system and a gaze estimation method that can improve the robustness and accuracy of gaze estimation. SUMMARY

[0006] The present disclosure aims to provide a gaze estimation system and a gaze estimation method to solve at least one of the above problems.

[0007] Embodiments of the present disclosure provide a gaze estimation system. The gaze estimation system includes a camera device and a processing circuit. A gaze estimation system includes a camera device configured to capture an image of a subject. A processing circuit coupled to the camera device and further includes an image processing unit and a gaze inference unit. The image processing unit is configured to derive a scaled face image and a scaled eye image from the image of the subject. The gaze inference unit is configured to extract a first feature representation from the scaled face image using a first feature extraction model, extract a second feature representation from the scaled eye image using a second feature extraction model, combine the first feature representation and the second feature representation to generate a combined feature representation, and determine a gaze direction of the subject based on the combined feature representation using a regression model.

[0008] In an embodiment, the gaze estimation system further includes a model training unit. The model training unit is configured to jointly train the first feature extraction model, the second feature extraction model, and the regression model using a training dataset including pairs of a training face image, a training eye image, and corresponding gaze labels.

[0009] In an embodiment, the model training unit is further configured to optimize the first feature extraction model using a training loss including a gaze estimation loss and an adversarial loss. The gaze estimation loss is minimized to preserve gaze-related features, and the adversarial loss is minimized to suppress gaze-unrelated features. The gaze-unrelated features include brightness features and individual appearance features.

[0010] In an embodiment, the model training unit is further configured to generate a reconstructed face image from the first feature representation using a reconstruction model, and calculate the adversarial loss based on a reconstruction loss between the reconstructed face image and the training face image.

[0011] In an embodiment, the model training unit is further configured to identify a face bounding box and an eye bounding box from a training image, apply a data augmentation to the training image, and derive one of the pairs of the training face image and the training eye image in the training dataset from the face bounding box and the eye bounding box. The data augmentation includes applying a scaling transformation to the face bounding box and the eye bounding box, applying a translation transformation to the face bounding box and the eye bounding box, and / or adjusting a brightness level of the training image.

[0012] In an embodiment, the data augmentation further includes generating a mask based on a plurality of eye landmarks, and applying the mask to the training image to selectively preserve a gaze-related region in the training eye image.

[0013] In an embodiment, the model training unit is further configured to train the second feature extraction model using a side task configured to predict one or more predefined eye features from the training eye images. The side task is designed to encourage the second feature extraction model to preserve information related to the predefined eye features in the second feature representation, the predefined eye features including the plurality of eyelids, the plurality of canthi, the plurality of iris boundaries, the plurality of limbus boundaries, a pupil center, and / or an eyeball center.

[0014] In an embodiment, the model training unit is further configured to train the first feature extraction model using a side task configured to predict one or more predefined face features from the training face images. The side task is designed to encourage the first feature extraction model to preserve information related to the predefined face features in the first feature representation, the predefined face features including face geometry information and / or head pose information.

[0015] In an embodiment, the model training unit is further configured to train the regression model using a training process including applying a classification loss to the coarse gaze direction estimate and applying a regression loss to the refined gaze direction estimate. A weight of the classification loss decreases and another weight of the regression loss increases as a training cycle increases.

[0016] In an embodiment, the gaze inference unit is further configured to determine the gaze direction based on the combined feature representation and the coarse gaze classification by applying a classification model to the combined feature representation to generate a coarse gaze classification, and applying the regression model. The model training unit is further configured to train the classification model using a classification loss and to train the regression model using a regression loss.

[0017] In an embodiment, the image processing unit is further configured to derive the scaled face image and the scaled eye image by performing operations including identifying a face bounding box and an eye bounding box from the image of the subject using an object detection model or a landmark detection model, cropping a cropped face image and a cropped eye image from the image of the subject based on the face bounding box and the eye bounding box, respectively, and downscaling the cropped face image and the cropped eye image to generate the scaled face image and the scaled eye image, respectively.

[0018] In an embodiment, the image processing unit is further configured to downscale the cropped face image and the cropped eye image such that the scaled eye image retains more pixel-level details corresponding to an eye of the subject than a corresponding eye region in the scaled face image.

[0019] In an embodiment, in response to detecting multiple face regions in the image of the subject, the image processing unit is further configured to select the face bounding box based on a proximity to a predefined reference location in the image.

[0020] In an embodiment, the image processing unit is further configured to identify a left eye bounding box and a right eye bounding box from the image of the subject, and select one of the left eye bounding box and the right eye bounding box that is closer to the camera device as the eye bounding box for cropping the cropped eye image.

[0021] In an embodiment, the image processing unit is further configured to maintain the selected eye bounding box across multiple frames, and only switch to other eye bounding boxes to crop the cropped eye image when a predefined condition indicative of sufficient change is satisfied.

[0022] In an embodiment, the camera device is a near-infrared camera configured to capture the image of the subject as a grayscale image, and the first feature extraction model, the second feature extraction model, and the regression model are trained using grayscale training images converted from color images by a model trained to simulate features of near-infrared images.

[0023] In an embodiment, the processing circuitry is further configured to perform at least one of the following actions based on the decided gaze direction: (i) modify a display output of a display device, (ii) blink an indicator light, (iii) generate an audio alert through an audio output device, (iv) activate an autonomous driving unit to take over control of the vehicle, and (v) activate a haptic output device to generate a vibration alert.

[0024] Embodiments of the present disclosure provide a gaze estimation method. The gaze estimation method is performed by one or more processors. The gaze estimation method includes deriving a scaled face image and a scaled eye image from an image of a subject captured via a camera device, extracting a first feature representation from the scaled face image using a first feature extraction model, and extracting a second feature representation from the scaled eye image using a second feature extraction model, combining the first feature representation and the second feature representation to generate a combined feature representation, and determining a gaze direction of the subject based on the combined feature representation using a regression model.

[0025] In an embodiment, the gaze estimation method further includes jointly training the first feature extraction model, the second feature extraction model, and the regression model using a training dataset including pairs of a training face image, a training eye image, and corresponding gaze labels.

[0026] In an embodiment, the gaze estimation method further includes optimizing the first feature extraction model using a training loss including a gaze estimation loss and an adversarial loss. The gaze estimation loss is minimized to preserve gaze-related features, and the adversarial loss is minimized to suppress gaze-unrelated features. The gaze-unrelated features include luminance features and individual appearance features.

[0027] In an embodiment, the gaze estimation method further includes generating a reconstructed face image from the first feature representation using a reconstruction model, and computing the adversarial loss based on a reconstruction loss between the reconstructed face image and the training face image.

[0028] In an embodiment, the gaze estimation method further includes identifying a face bounding box and an eye bounding box from a training image, applying a data augmentation to the training image, and deriving one of a plurality of pairs of training face image and training eye image in the training dataset from the face bounding box and the eye bounding box. The data augmentation includes applying a scaling transformation to the face bounding box and the eye bounding box, applying a translation transformation to the face bounding box and the eye bounding box, and / or adjusting a light level of the training image.

[0029] In an embodiment, the gaze estimation method further includes training the second feature extraction model using a auxiliary task configured to predict one or more predefined eye features from the training eye image. The auxiliary task is designed to encourage the second feature extraction model to preserve information related to the predefined eye features in the second feature representation. The predefined eye features include a plurality of eyelids, a plurality of canthi, a plurality of iris boundaries, a plurality of limbus boundaries, a pupil center, and / or an eyeball center.

[0030] In an embodiment, the gaze estimation method further includes a auxiliary task of predicting one or more predefined face features from the training face image. The auxiliary task is designed to encourage the first feature extraction model to preserve information related to the predefined face features in the first feature representation. The predefined face features include facial geometry information and / or head pose information.

[0031] In an embodiment, the gaze estimation method further includes training the regression model using a training procedure. The training procedure includes applying a classification loss to the coarse gaze direction estimate and applying a regression loss to the refined gaze direction estimate. A weight of the classification loss decreases and another weight of the regression loss increases as a training cycle increases.

[0032] In an embodiment, the gaze estimation method further includes determining the gaze direction based on the combined feature representation and a coarse gaze classification by applying a classification model to the combined feature representation to generate the coarse gaze classification, and applying the regression model. The classification model is trained using a classification loss and the regression model is trained using a regression loss.

[0033] In an embodiment, the operations of deriving the scaled face image and the scaled eye image further include identifying a face bounding box and an eye bounding box from the image of the subject using an object detection model or a landmark detection model, cropping a cropped face image and a cropped eye image from the image of the subject based on the face bounding box and the eye bounding box, respectively, and downscaling the cropped face image and the cropped eye image to generate the scaled face image and the scaled eye image, respectively.

[0034] In an embodiment, the cropped face image and the cropped eye image are downscaled such that the scaled eye image retains more pixel-level details corresponding to an eye of the subject than a corresponding eye region within the scaled face image.

[0035] In an embodiment, the gaze estimation method further includes selecting the face bounding box based on proximity to a predefined reference location within the image in response to detecting multiple face regions in the image of the subject.

[0036] In an embodiment, the gaze estimation method further includes identifying a left eye bounding box and a right eye bounding box from the image of the subject, and selecting the one that is closer to the camera device from the left eye bounding box and the right eye bounding box as the eye bounding box for cropping the cropped eye image.

[0037] In an embodiment, the gaze estimation method further includes maintaining the selected eye bounding box across multiple frames, and only switching to other eye bounding boxes to crop the cropped eye image when a predefined condition indicative of sufficient change is satisfied.

[0038] In an embodiment, the camera device is a near-infrared camera configured to capture the image of the subject as a grayscale image. The first feature extraction model, the second feature extraction model, and the regression model are trained using grayscale training images converted from color images by a model trained to mimic near-infrared image features.

[0039] In an embodiment, the gaze estimation method further includes performing at least one of the following actions based on the determined gaze direction: (i) modifying a display output of a display device, (ii) flashing an indicator light, (iii) producing an audio alert through an audio output device, (iv) activating an autonomous driving unit to take over vehicle control, and (v) activating a haptic output device to produce a vibration alert.

[0040] According to embodiments of the present disclosure, the line-of-sight estimation system and method described herein effectively address various limitations encountered in the prior art. Specifically, by extracting features from both face and eye images and combining them for line-of-sight estimation, the line-of-sight estimation system is able to capture both a broader facial context and detailed eye details, thereby improving the accuracy of line-of-sight estimation. Furthermore, the adversarial training mechanism introduced for the face feature extraction model suppresses line-of-sight irrelevant features, such as variations in brightness and individual appearance, that would otherwise degrade the robustness of the estimation.

[0041] In addition, by using auxiliary tasks to encourage the preservation of facial and eye features relevant to line-of-sight during feature extraction, the line-of-sight estimation system enhances the semantic richness of the intermediate feature representation, thereby facilitating more accurate inference. The use of a dynamic loss weighting strategy during regression model training further mitigates the bias-variance problem commonly encountered in line-of-sight estimation tasks.

[0042] By carefully preprocessing the input images, including intelligently selecting face and eye bounding boxes and employing targeted resizing strategies, the line-of-sight estimation system can ensure that essential visual information is preserved even under hardware and computational constraints. Furthermore, by incorporating mechanisms such as hysteresis control for eye selection, and by enabling responsive operations based on the estimated line-of-sight direction, the line-of-sight estimation system achieves practical reliability and responsiveness suitable for real-world applications such as driving monitoring, digital billboard interaction, and customer behavior analysis. BRIEF DESCRIPTION OF DRAWINGS

[0043] The present disclosure can be more fully understood by reading the subsequent detailed description and examples in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A system architecture diagram of a line-of-sight estimation system according to embodiments of the present disclosure is shown.

[0045] Figure 2 A flowchart of a line-of-sight estimation method according to embodiments of the present disclosure is shown.

[0046] Figure 3 A training process for training a first feature extraction model according to embodiments of the present disclosure is shown.

[0047] Figure 4 A training process for training a second feature extraction model using auxiliary tasks according to embodiments of the present disclosure is shown.

[0048] Figure 5 A training process for training a regression model according to embodiments of the present disclosure is shown.

[0049] Figure 6A training process of training a classification model and a regression model according to embodiments of the present disclosure is shown.

[0050] Figure 7 A process of constructing a training data set according to embodiments of the present disclosure is shown.

[0051] Figure 8 A pre-processing process performed by an image processing unit before a feature extraction operation according to embodiments of the present disclosure is shown.

[0052] Reference signs are as follows:

[0053] 10: line-of-sight estimation system

[0054] 100: image

[0055] 101: face image

[0056] 102: eye image

[0057] 11: camera device

[0058] 111, 311: first feature extraction model

[0059] 112, 412: second feature extraction model

[0060] 12: processing circuitry

[0061] 121, 321: first feature representation

[0062] 122, 322, 422: second feature representation

[0063] 125, 325, 525, 625: combined feature representation

[0064] 13: image processing unit

[0065] 130, 330, 530, 640: regression model

[0066] 14: line-of-sight inference unit

[0067] 140, 340: line-of-sight direction

[0068] 20: line-of-sight estimation method S201, S202, S203, S204, S701, S702, S703, S801, S802, S803: operation

[0069] 301: training face image

[0070] 341: line-of-sight label

[0071] 342: line-of-sight estimation loss

[0072] 350: Reconstruction Model

[0073] 360: Reconstructed facial image

[0074] 361: Reconstruction Losses

[0075] 362: Combating Losses

[0076] 40: Auxiliary Tasks

[0077] 402: Training Eye Image

[0078] 430: Auxiliary Prediction Model

[0079] 431: Predicting Eye Features

[0080] 432: Baseline Facts - Eye Characteristics

[0081] 433: Auxiliary Loss

[0082] 50, 60: Training process

[0083] 531: Classification Output

[0084] 532: Regression Output

[0085] 541: Baseline Fact Classification

[0086] 542, 642: Baseline fact line of sight direction

[0087] 551,633: Classification Loss

[0088] 552,643: Regression Loss

[0089] 560: Loss-Weighted Scheduler

[0090] 561: Total Loss

[0091] 630: Classification Model

[0092] 631: Rough Visual Classification

[0093] 632: Baseline Facts and Visual Classification

[0094] 641: Predicting the direction of sight

[0095] 70: Process

[0096] 80: Preprocessing Detailed Implementation

[0097] The following description is intended to illustrate the general principles of the invention and should not be construed as limiting the invention. The scope of the invention should be determined by the appended claims.

[0098] In the following description of embodiments, same or similar reference numerals may be used to denote same or similar elements or components.

[0099] The ordinal terms such as "first", "second", "third", etc., as used in the claims, are used for distinguishing between other elements or steps and do not supply a serial or numerical preference.

[0100] The following description of embodiments of a device or system also applies to embodiments of a method, and vice versa.

[0101] Figure 1 A system architecture diagram of a line-of-sight estimation system 10 according to embodiments of the disclosure is shown. As shown, the line-of-sight estimation system 10 includes a camera device 11 and a processing circuit 12. The processing circuit 12 further includes an image processing unit 13 and a line-of-sight inference unit 14. Figure 1

[0102] The camera device 11 is configured to capture images of a subject. In different application scenarios, the subject can be a driver inside a vehicle, a passerby in front of a digital billboard display, or a customer in a retail environment. By way of example, the captured images can be color images or grayscale images, depending on the type of camera device 11 used. In some embodiments, the camera device 11 can be a near-infrared (NIR) camera configured to capture grayscale images with enhanced low-light performance.

[0103] The processing circuit 12 is configured to perform various image processing operations on the images captured by the camera device 11, including image processing and line-of-sight estimation. The processing circuit 12 can be implemented using one or more processors, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more programmable logic devices (PLDs), or other combinations of hardware, firmware, and / or software elements. The processing circuit 12 is configured to perform the functions of the image processing unit 13 and the line-of-sight inference unit 14 through dedicated hardware circuitry or a general-purpose processor executing software instructions.

[0104] Details of the operations performed by the image processing unit 13 and the line-of-sight inference unit 14 will be described in detail below with reference to Figure 2

[0105] Figure 2 A flowchart of a line-of-sight estimation method 20 according to embodiments of the disclosure is shown. As shown, the line-of-sight estimation method 20 includes a step S21 of capturing an image of a subject using a camera device 11.​​Figure 2 As shown, the gaze estimation method 20 comprises operations S201 to S204. It should be noted that, Figure 2 The elements involved in the method are as shown in Figure 1 Thus, reference can be made to Figure 1 and Figure 2 to better understand the operation and structure of the present embodiment.

[0106] In operation S201, the image processing unit 13 derives the scaled face image 101 and the scaled eye image 102 from the captured image 100 of the subject by the camera device 11. Specifically, the image processing unit 13 can identify the face region and the eye region within the captured image 100, crop the identified regions, and perform resizing operation to normalize the face image 101 and the eye image 102 to a predetermined size suitable for subsequent feature extraction. The resizing process is beneficial to improve the consistency of the input data and to improve the performance of the downstream machine learning model.

[0107] In operation S202, the gaze inference unit 14 extracts the first feature representation 121 from the scaled face image 101 using the first feature extraction model 111, and extracts the second feature representation 122 from the scaled eye image 102 using the second feature extraction model 112. The first feature extraction model 111 and the second feature extraction model 112 can be implemented by different neural network architectures, respectively, for processing facial and eye features. The extracted feature representations can be embodied as feature maps, vectors or tensors, and contain abstract information of the salient features of the face and eyes required for accurate gaze estimation.

[0108] In operation S203, the gaze inference unit 14 combines the first feature representation 121 and the second feature representation 122 together to generate the combined feature representation 125. In some implementations, the aforementioned combination can be achieved by concatenating the first feature representation 121 and the second feature representation 122 along a specified dimension. Alternatively, other combination techniques (e.g. summation, averaging or feature fusion using learnable layers) can also be used. The combined feature representation 125 is expected to integrate complementary information from the face and eye regions to enhance the robustness of the gaze estimation.

[0109] In operation S204, the gaze inference unit 14 determines the gaze direction 140 of the subject based on the combined feature representation 125 using the regression model 130. Specifically, the combined feature representation 125 is input to the regression model 130, and the regression model 130 outputs the predicted or inferred gaze direction 140 of the subject. The regression model 130 can be implemented using a fully connected neural network, a multilayer perceptron (MLP), a support vector regression (SVR) model, or other suitable regression techniques capable of mapping the combined feature space to a continuous gaze direction as output, although the present disclosure is not limited thereto.

[0110] Depending on the specific application scenario, the gaze direction 140 can be represented in various forms. For example, the gaze direction 140 can be represented as a pair of gaze angles relative to a coordinate system, including a yaw angle representing the horizontal rotation and a pitch angle representing the vertical rotation. In automotive applications, only the yaw and pitch angles are usually needed because the roll angle has little effect on monitoring the driver's attention. Alternatively, the gaze direction 140 can be represented as a three-dimensional unit vector for applications requiring more accurate spatial positioning. In some simplified scenarios, a single angle value (e.g., only the yaw angle) can be used to represent the gaze direction 140 to represent the deviation of the horizontal gaze from a reference axis, thereby reducing the computational complexity.

[0111] In an embodiment, although Figure 1 Although not shown, the gaze estimation system 10 can further include a model training unit. The model training unit can be configured to jointly train the first feature extraction model 111, the second feature extraction model 112, and the regression model 130 using a training dataset including a plurality of pairs of a training face image and a training eye image, and a corresponding gaze label. The model training unit can be implemented in various ways, including as a part of the processing circuit 12, or by a remote server computer configured to perform the model training operation, and then deploy the trained models to the processing circuit 12 for inference operation. The training face images and the training eye images are used to provide visual inputs representing different facial and eye states under different conditions, and the gaze label is used as a supervised ground truth reference value indicating the ground truth gaze direction associated with each image pair, thereby guiding the training of the corresponding models.

[0112] In an embodiment, the model training unit is further configured to optimize the first feature extraction model 111 using a training loss that includes a gaze estimation loss and an adversarial loss. The gaze estimation loss is minimized to encourage the first feature extraction model 111 to preserve gaze-related features that are needed for accurate gaze direction estimation. Conversely, the adversarial loss is minimized to suppress gaze-unrelated features, including but not limited to intensity features caused by lighting changes and individual appearance features (e.g. facial identity features). The combination of the gaze estimation loss and the adversarial loss enables the first feature extraction model 111 to extract features that are robust to changes in the environment and subject-specific differences, thereby improving the generalizability and accuracy of gaze estimation. For more details on the training process of the first feature extraction model, please refer to Figure 3 which will be described below.

[0113] Figure 3 A training process 30 for training the first feature extraction model 311 is shown, according to an embodiment of the disclosure. It should be understood that, Figure 1 The inference phase of the gaze estimation system 10 is described, while Figure 3 The training phase of the gaze estimation system 10 is described, and therefore for the sake of clarity, the same or similar elements will be denoted with different symbols.

[0114] As Figure 3 shown, a training face image 301 of a subject is input to the first feature extraction model 311 to generate a first feature representation 321. The first feature representation 321 is then used to generate a reconstructed face image 360 by a reconstruction model 350. By way of example, the reconstruction model 350 can be implemented as a decoder-type neural network configured to invert an abstract feature representation back to an image that resembles the original input. The purpose of the reconstruction model 350 is to evaluate how much information about the original face image is preserved in the extracted first feature representation 321. During training, the reconstruction model 350 is trained using a reconstruction loss 361 that measures the difference between the reconstructed face image 360 and the training face image 301 (e.g. by pixel-wise mean squared error or other suitable loss metric).

[0115] The reconstruction loss 361 is further used to derive an adversarial loss 362, which penalizes the ability of the first feature extraction model 311 to preserve appearance-related features, thereby promoting the extraction of information related to the gaze while suppressing the extraction of irrelevant details. In one implementation, the adversarial loss 362 can be computed as one minus the reconstruction loss 361, such that a larger reconstruction loss leads to a smaller adversarial loss, and vice versa, although the present disclosure is not limited thereto. Other designs of adversarial loss can be used as long as the reconstruction loss 361 and the adversarial loss 362 exhibit a negative correlation.

[0116] Meanwhile, the first feature representation 321 is also combined with the second feature representation 322 (not detailed for brevity) to form a combined feature representation 325, which is input into a regression model 330 to predict a gaze direction 340. The predicted gaze direction 340 is compared with a ground truth gaze label 341 to compute a gaze estimation loss 342. Both the adversarial loss 362 and the gaze estimation loss 342 are back-propagated to update the parameters of the first feature extraction model 311. Figure 3

[0117] In another embodiment, the adversarial training of the first feature extraction model 311 can be implemented using a gradient reversal mechanism. Specifically, instead of explicitly defining the adversarial loss 362 as a function of the reconstruction loss 361, a gradient reversal layer (GRL) can be inserted between the first feature representation 321 and the reconstruction model 350. During forward pass, the gradient reversal layer allows the first feature representation 321 to be fed into the reconstruction model 350 without modification. However, during backward pass, the gradient reversal layer multiplies the gradient from the reconstruction loss 361 by a negative scalar, effectively reversing the direction of the gradient received by the first feature extraction model 311. As a result, the first feature extraction model 311 is adversarially trained to maximize the reconstruction loss 361, thereby suppressing the encoding of gaze-irrelevant features such as luminance features and individual appearance features. Although this alternative design does not explicitly define an individual adversarial loss, it can increase the computational burden because the reconstruction model 350 has to participate in both forward and full backward pass during each training iteration, and additional operations related to gradient reversal are required.

[0118] ​In one embodiment, the model training unit is further configured to train the second feature extraction model using an auxiliary task, which is configured to predict one or more predefined eye features from training eye images. The auxiliary task is designed to encourage the second feature extraction model to retain information related to the predefined eye features in the second feature representation. The predefined eye features include, but are not limited to, eyelids, canthi, iris boundaries, limbal boundaries, pupil centers, eyeball centers, or combinations thereof. Further details regarding the training process of the second feature extraction model are available as follows: Figure 4 As shown, the following will describe it.

[0119] Figure 4 The training process of using auxiliary task 40 to train second feature extraction model 412 according to an embodiment of this disclosure is illustrated. Figure 4 As shown, a training eye image 402 is provided as input to a second feature extraction model 412, which generates a second feature representation 422. The second feature representation 422 is then input into an auxiliary prediction model 430 within an auxiliary task 40. The auxiliary prediction model 430 is configured to predict one or more predefined eye features, denoted as predicted eye features 431. Simultaneously, the corresponding baseline fact eye features 432 serve as supervision material for training. The predicted eye features 431 and the baseline fact eye features 432 are compared to calculate an auxiliary loss 433 reflecting the prediction error.

[0120] During training, the auxiliary loss 433 is backpropagated through the auxiliary prediction model 430 and the second feature extraction model 412. Specifically, gradients are backpropagated to adjust the parameters of the second feature extraction model 412 in a manner that encourages the second feature representation 422 to retain information related to predefined eye features (e.g., eyelids, canthi, iris boundaries, limbal boundaries, pupil center, and / or eyeball center). The auxiliary prediction model 430 itself can be implemented using any suitable neural network architecture (e.g., multi-layer perceptron (MLP) or convolutional neural network (CNN)), but this disclosure is not limited thereto. In this way, the auxiliary task 40 enhances the primary gaze estimation task by improving the feature richness and task relevance of the second feature extraction model 412.

[0121] In some embodiments, the auxiliary loss 433 can be further combined with the main task loss (e.g., gaze estimation loss) to form a joint optimization objective. During training, a weighting factor can be assigned to the auxiliary loss 433 to balance the contribution of the auxiliary loss 433 relative to the gaze estimation loss. In this case, the total training loss is minimized by adjusting the parameters of the second feature extraction model 412 and the parameters of the auxiliary prediction model 430. The weighting factor can be a constant or dynamically adjusted during the training period, depending on the specific implementation choices. By incorporating the auxiliary loss 433 into the joint optimization framework, the gaze estimation system can improve the relevance of the extracted features to the eye landmarks while maintaining high accuracy of the gaze direction estimation.

[0122] In an embodiment, the first feature extraction model can also be trained using a similar auxiliary task. Specifically, the auxiliary task is configured to predict one or more predefined facial features from the training face images. The auxiliary task is designed to encourage the first feature extraction model to preserve information related to the predefined facial features in the first feature representation. The predefined facial features include, but are not limited to, facial geometry information (e.g., the locations of the nose, mouth, and facial contours) and / or head pose information (e.g., the yaw, pitch, and / or roll angles of the subject’s head). The predicted facial features can be compared against ground truth labels to compute an auxiliary loss, which is then backpropagated to optimize the parameters of the first feature extraction model.

[0123] In some embodiments, the auxiliary task for predicting facial features can incorporate the adversarial mechanism described earlier. That is, the first feature extraction model can be jointly optimized by minimizing the gaze estimation loss, the auxiliary loss related to the facial feature prediction, and the adversarial loss that aims to suppress features irrelevant to the gaze. Through this multi-task training framework, the first feature extraction model can be guided to extract a feature representation that is informative for gaze estimation, predictive of facial structure, and robust to irrelevant variations (e.g., lighting and individual appearance differences).

[0124] In an embodiment, the training process of the regression model involves applying a classification loss to the coarse gaze direction estimation and a regression loss to the fine-grained gaze direction estimation. More details about this training process are described in Figure 5 As shown, the following will be described.

[0125] Figure 5 FIG. 5 shows a training process 50 for training a regression model 530, according to an embodiment of the present disclosure. As shown, the training process 50 includes the following steps: Figure 5As shown, the combined feature representation 525 is input into a regression model 530. The regression model 530 is configured to output both a classification output 531 and a regression output 532. The classification output 531 represents a coarse gaze classification, while the regression output 532 represents a refined gaze direction prediction.

[0126] The classification output 531 is compared against a ground truth classification 541 to compute a classification loss 551, while the regression output 532 is compared against a ground truth gaze direction 542 to compute a regression loss 552. The ground truth classification 541 is obtained by partitioning the gaze space into a plurality of discrete categories, which can correspond to discrete gaze regions or gaze sector regions. In contrast, the ground truth gaze direction 542 is typically represented as a continuous-valued vector or angle representation of the subject’s gaze.

[0127] The classification loss 551 and the regression loss 552 are then combined together using a loss weight scheduler 560 to obtain an overall loss 561. Specifically, the loss weight scheduler 560 dynamically adjusts the relative weights of the classification loss 551 and the regression loss 552 based on the training epoch. Initially, the classification loss 551 is assigned a higher weight to boost the performance of the coarse gaze estimation in the earlier training epochs. As training progresses, the weight of the classification loss 551 is gradually reduced, while the weight of the regression loss 552 is correspondingly increased, shifting the training focus towards refined gaze direction prediction. This strategy enables the regression model 530 to first learn a coarse estimate of the gaze direction, and then refine its prediction to a more accurate level.

[0128] The overall loss 561 computed by the loss weight scheduler 560 is used for backpropagation to update the parameters of the regression model 530. With this epoch-based adjustment mechanism, the training process 50 ensures a smoother optimization trajectory and improves the overall accuracy of the gaze estimation.

[0129] In another embodiment, the gaze estimation system employs a class-informed regression approach. Specifically, in the inference phase, a classification model is applied to the combined feature representation to generate a coarse gaze classification, and a regression model is applied to determine the gaze direction based on the combined feature representation and the coarse gaze classification. In the training phase, the classification model is trained using a classification loss, while the regression model is trained using a regression loss. Figure 6 Further details regarding the training process of the classification model and the regression model are described below.

[0130] Figure 6 A training process 60 for training a classification model 630 and a regression model 640 according to an embodiment of the disclosure is shown. As shown, the training process 60 comprises a training phase 610 and an inference phase 620. Figure 6As shown, the combined feature representation 625 is input into a classification model 630. The classification model 630 outputs a coarse gaze classification 631, representing a discrete classification of the subject gaze direction. The coarse gaze classification 631 is compared against a ground truth gaze classification 632 to compute a classification loss 633, which is used for backpropagation to update the classification model 630.

[0131] Meanwhile, the coarse gaze classification 631 is also used by a regression model 640. The regression model 640 receives the combined feature representation 625 and the coarse gaze classification 631 as input, and generates a predicted gaze direction 641. The predicted gaze direction 641 is compared against a ground truth gaze direction 642 to compute a regression loss 643, which is used for backpropagation to update the regression model 640.

[0132] In this classification-based regression approach, the classification model 630 serves as a pre-processing stage to provide coarse gaze information to the regression model 640, so that the regression model 640 can focus on improving gaze prediction within a narrower and more relevant range. This training strategy effectively reduces the learning difficulty of the regression task, and improves the accuracy of the overall prediction.

[0133] Figure 7 A process 70 of constructing a training dataset according to an embodiment of the disclosure is shown. As shown, the process 70 includes operations S701 to S703. Figure 7

[0134] In operation S701, the model training unit identifies a face bounding box and eye bounding boxes from the training image. The face bounding box refers to a rectangular region that contains the subject’s face region, while the eye bounding boxes refer to rectangular regions that contain the subject’s eye regions. An object detection model trained to detect faces and eyes can be used to generate these bounding boxes, or the bounding regions can be inferred from detected keypoints by a landmark-based approach. Each bounding box is typically represented by coordinates specifying a center point, width, and height, or by coordinates of two opposite corners on the diagonal, but the disclosure is not limited thereto.

[0135] In operation S702, the model training unit applies data augmentation to the training image. Data augmentation can include, but is not limited to:

[0136] (i) Scaling transformation on the face bounding box and eye bounding boxes: scaling operation enlarges or shrinks the size of the bounding boxes by a random factor within a predefined range (e.g., between 0.8 and 1.2), which helps the model to better generalize to variations in face and eye sizes at different distances from the camera;

[0137] ​(ii) applying a translation transformation to the face bounding box and the eye bounding box: the translation operation randomly moves the bounding boxes along the x-axis and / or the y-axis within a limited range (e.g. ±20% of the size of the bounding box), so as to simulate the misalignment or imperfection in detection that can occur in real-world scenarios; and / or

[0138] (iii) adjusting the brightness level of the training images: this operation changes the brightness of the training images to simulate different lighting conditions, so as to make the trained model more robust to lighting variations (e.g. overexposure and underexposure) or low-light scenarios (e.g. night, backlight, shadow, and local illumination non-uniformity).

[0139] These augmentation techniques enhance the diversity of the training samples, and improve the generalization capability of the model and its robustness to real-world variability.

[0140] In operation S703, the model training unit derives a training face image and a training eye image of one of the pairs in the training data set from the face bounding box and the eye bounding box. Specifically, the model training unit crops the training face image and the training eye image from the training image according to the augmented face bounding box and eye bounding box, respectively. Then, the cropped images are adjusted to a standardized resolution as required by the model input. Each pair of training face image and training eye image and the corresponding gaze label forms a piece of data in the training data set, which is subsequently used to train the first feature extraction model, the second feature extraction model and the regression model.

[0141] Although operations S701 to S703 are described in a specific order, it should be understood that the present disclosure is not limited thereto. In some embodiments, the order of the operations can be changed. For example, operation S702 and operation S703 can be performed in a different order (e.g. first derive a pair of training face image and training eye image and then apply data augmentation to the pair of training face image and training eye image).

[0142] In an embodiment, the data augmentation performed in operation S702 can also include generating a mask based on the eye landmark points and applying the mask to the training images to selectively preserve the gaze-related regions in the training eye images. Specifically, a set of eye landmarks corresponding to physical structures such as eyelid contours, eye corners, iris boundaries, and / or pupil centers can be detected or estimated from the training images. Based on the detected eye feature points, a mask is created to delineate the eye regions that are most relevant to gaze estimation. The mask can take the form of a binary or soft-weighted mask that emphasizes the middle portions of the gaze-related regions (e.g., the iris and pupil) while de-emphasizing or excluding the surrounding regions (e.g., the eye socket, skin regions). The mask can then be applied to the training eye images by means such as pixel-wise multiplication or weighted blending to enhance the visibility and prominence of the gaze-related features while reducing the impact of the gaze-unrelated features (e.g., occlusions, reflections, or noise around the eye). This augmentation technique helps to improve the robustness and generalization capability of the second feature extraction model by directing it to focus on the informative regions that are critical to accurately inferring the gaze direction.

[0143] In an embodiment, Figure 1 The illustrated camera device 11 is a near-infrared (NIR) camera configured to capture grayscale images of the subject. With a NIR camera, the system can reliably operate under different lighting conditions and enhance the contrast of certain facial features that are extremely critical to gaze estimation (e.g., the pupil and iris). Since NIR imaging inherently provides grayscale output rather than full-color images, the system benefits from reduced computational demand in the subsequent image processing and feature extraction stages.

[0144] Accordingly, the first feature extraction model, the second feature extraction model, and the regression model are trained using grayscale training images converted from color images by models trained to mimic near-infrared image features. Specifically, a machine learning model (e.g., a convolutional neural network) can be pre-trained to map RGB images captured under visible light to synthetic NIR-like grayscale images. This conversion process helps to align the features of the training dataset with the actual input features captured by the NIR camera device during inference. By training the models on grayscale images exhibiting NIR-like characteristics, the gaze estimation system can achieve higher robustness and consistency across various environmental conditions (e.g., low-light or high-glare scenarios that can be difficult for traditional RGB imaging). Furthermore, using grayscale inputs reduces the number of input channels, which in turn reduces memory consumption and speeds up inference time, particularly advantageous for deployment scenarios on resource-constrained platforms.

[0145] Figure 8Fig. 8 shows a pre-processing procedure 80 performed by the image processing unit 13 prior to the feature extraction operation according to an embodiment of the present disclosure. The pre-processing procedure 80 prepares the image data of the subject in a format suitable for the subsequent line-of-sight estimation analysis. As shown, the pre-processing procedure 80 comprises operations S801 to S803. Figure 1 Figure 8 In operation S801, the image processing unit 13 identifies the face bounding box and the eye bounding box from the image of the subject using an object detection model or a landmark detection model. The object detection model can be a machine learning based model (e.g. convolutional neural network (CNN) and its variants (e.g. R-CNN, Fast R-CNN, Mask R-CNN and YOLO)) trained to locate the facial and eye regions in the image. Alternatively, a landmark detection model can be used to detect specific facial landmark points (e.g. eye corners, nose tip) from which the bounding boxes of the face and eyes can be derived. Detailed explanation of the definition, acquisition method and representation of the face bounding box and the eye bounding box has been given previously and will not be repeated here for brevity.

[0146] In operation S802, the image processing unit 13 crops the face image and the eye image from the image of the subject based on the face bounding box and the eye bounding box respectively. Specifically, the face bounding box defines a region of interest that encompasses the facial features of the subject, while the eye bounding box isolates a region that corresponds specifically to the eyes of the subject. The cropping operation extracts the pixel region within these bounding boxes to generate the face image and the eye image. This operation effectively reduces irrelevant background information and intensifies the focus on the meaningful regions for subsequent feature extraction.

[0147] In operation S803, the image processing unit 13 down-sizes the cropped face image and the cropped eye image to generate the scaled face image and the scaled eye image respectively. The down-sizing operation is used to normalise the input size of the feature extraction model and reduce the computational burden during inference. The down-sizing operation can use interpolation techniques (e.g. bilinear interpolation, bicubic interpolation or other suitable image resizing methods), but the present disclosure is not limited thereto.

[0148]

[0149] ​​In an embodiment, in operation S803, the cropped face image and the cropped eye image are scaled down such that the scaled eye image preserves more pixel-level details corresponding to the subject's eyes than the corresponding eye region in the scaled face image. Specifically, the face image originally containing the entire face region is resized to a standard resolution (e.g., 120x120 pixels), while the eye image originally focusing on the smaller eye region is resized to a different resolution (e.g., 60x36 pixels) that has been optimized to capture detailed eye features. As a result, the scaled eye image preserves a higher pixel density and finer granularity for the eye region than the eye portion in the scaled face image, thereby facilitating more accurate extraction of gaze-related features.

[0150] In fact, compared to inferring from the scaled face image alone, this design ensures that key structures related to gaze (such as the iris, eyelid contour, and pupil center) are maintained with higher fidelity in the scaled eye image. As such, this arrangement enables the feature extraction model to leverage both the broad contextual information from the face and the fine local details from the eye, thereby improving the accuracy of the overall gaze estimation.

[0151] It should be noted that while the specific resizing resolution can vary depending on implementation needs, the principle of assigning a greater pixel density to the isolated eye region is maintained across different implementations.

[0152] In an embodiment, the image processing unit 13 is further configured to detect whether multiple face regions are present in the image 100 of the subject. In response to detecting multiple face regions in the image 100 of the subject, the image processing unit 13 selects the face bounding box based on proximity to a predefined reference location in the image. Specifically, the predefined reference location can correspond to the expected location of the primary subject (e.g., the driver seat region in a vehicle-mounted camera setup). For example, in a vehicle application scenario, the image captured by the camera device 11 can often contain the faces of the driver and one or more passengers. In this case, detecting multiple face regions without additional filtering can lead to the selection of the wrong subject, resulting in an incorrect gaze estimation result.

[0153] By exploiting the relatively fixed spatial relationship between the camera device and the driver seat location, where the driver's face is expected to always appear near the predetermined location in the captured image, the image processing unit 13 can effectively distinguish between the driver and the passengers. This approach helps to prevent confusion caused by accidentally appearing faces (e.g., a rear seat passenger leaning forward), thereby improving the reliability of subsequent gaze estimation.

[0154] In an embodiment, the image processing unit 13 is further configured to identify a left eye bounding box and a right eye bounding box from the image 100 of the subject. Subsequently, the image processing unit 13 selects the eye bounding box that is closer to the camera device 11 from the left eye bounding box and the right eye bounding box and uses it to crop the eye image. The proximity of each eye bounding box to the camera device 11 can be estimated based on the bounding box size, focus sharpness, or other visual cues extracted from the image, but the present disclosure is not limited thereto.

[0155] This selection can improve the quality and reliability of the extracted eye image for subsequent feature extraction. Generally, the eye that is spatially closer to the camera device 11 appears larger, sharper, and provides richer pixel-level details in the captured image, which is crucial for accurate gaze estimation. By selecting the closer eye, the system can maximize the resolution and minimize the distortion caused by oblique viewing angles or perspective effects.

[0156] In a further embodiment, the image processing unit 13 is further configured to include a hysteresis mechanism to avoid frequent switching between the left and right eye bounding boxes across consecutive frames, thereby mitigating the jitter effect in the gaze estimation results. Specifically, the image processing unit 13 is configured to maintain the selected eye bounding box across multiple frames and only switch to the other eye bounding box to crop the eye image when a predefined condition indicative of sufficient change is satisfied.

[0157] The predefined condition can involve some criteria (e.g., a substantial reduction in the relative size or sharpness of the currently selected eye bounding box, or a significant improvement in the proximity or visual quality of the other eye bounding box). For example, the system can define thresholds for the size ratio of the bounding boxes, focus metrics, or detection confidence scores. Only when the thresholds are exceeded, the system allows switching from the currently tracked eye to the other eye.

[0158] By using such a hysteresis mechanism, the gaze estimation system can prevent unnecessary switching triggered by minor or transient changes (e.g., small head movements or detection noise). This helps to provide a smoother and more stable gaze tracking experience, which is particularly important in dynamic environments (e.g., in-vehicle driving monitoring).

[0159] In an embodiment, the processing circuitry 12 is further configured to perform at least one action based on the determined gaze direction 140:

[0160] (i) Modify display output of display devices: In vehicle applications, if the gaze direction indicates that the driver is not paying attention to a critical area (e.g., an intersection or a merging lane), the display unit inside the vehicle can automatically enlarge, highlight, or reposition the navigation prompts or hazard warnings to attract the driver’s attention. As another example, in a digital billboard or advertising display system deployed in a public area, the content displayed on the screen can be dynamically adjusted according to the display area that attracts the gaze of passing pedestrians, thereby optimizing the effectiveness of the advertisement.

[0161] (ii) Blinking indicator light: In environments where safety is a primary concern (e.g., driving a vehicle), if the gaze estimation system detects that the driver’s attention has been diverted from the road for a period of time, an indicator light on the dashboard can start blinking to prompt the driver to refocus their attention on driving.

[0162] (iii) Generate audio alerts through audio output devices: If the driver is detected to be distracted or drowsy based on the gaze pattern, an audio warning tone or voice prompt can be generated through the car’s speaker system, thereby providing an immediate and hard-to-ignore sensory cue even if the visual focus is lost.

[0163] (iv) Activate autonomous driving units to take over vehicle control: In vehicles using advanced driver-assistance systems (ADAS) or autonomous driving, if the gaze direction indicates that the driver has been inattentive for a prolonged period of time, or a dangerous situation is detected while the driver is distracted, the system can activate the autonomous driving control unit to temporarily take over the vehicle control (e.g., maintain driving within the lane, apply the brakes, or perform emergency operations) to prevent accidents.

[0164] (v) Activate haptic output devices to generate vibration alerts: If the driver’s gaze direction indicates that the driver is inattentive, haptic actuators embedded in the steering wheel or driver’s seat can be triggered to generate a vibration alert. The feedback through touch does not require processing through vision or hearing, and can serve as an immediate and intuitive warning, thereby shortening the reaction time in emergency situations.

[0165] However, the present disclosure is not limited to the above examples, and other gaze direction-based response actions can also be implemented.

[0166] According to embodiments of the present disclosure, the gaze estimation system and method described herein effectively address various limitations encountered in the prior art. Specifically, by extracting features from both face and eye images and combining them for gaze estimation, the gaze estimation system is able to capture both a broader facial context and detailed eye details, thereby improving the accuracy of gaze estimation. Furthermore, the adversarial training mechanism introduced for the face feature extraction model suppresses gaze-irrelevant features such as variations in brightness and individual appearance that would otherwise degrade the robustness of the estimation.

[0167] In addition, by using auxiliary tasks to encourage the preservation of face and eye features relevant to gaze during feature extraction, the gaze estimation system enhances the semantic richness of the intermediate feature representation, thereby facilitating more accurate inference. The use of a dynamic loss weighting strategy during regression model training further mitigates the bias-variance problem commonly encountered in gaze estimation tasks.

[0168] By careful preprocessing of the input images, including intelligently selecting face and eye bounding boxes and targeted resizing strategies, the gaze estimation system can ensure that necessary visual information is preserved even under hardware and computational constraints. Furthermore, by incorporating mechanisms such as hysteresis control for eye selection, and by enabling responsive operation based on the estimated gaze direction, the gaze estimation system achieves practical reliability and responsiveness suitable for real-world applications such as driving monitoring, digital billboard interaction, and customer behavior analysis.

[0169] The above paragraphs are described in various aspects. Obviously, the teachings in the present disclosure can be implemented in various ways. Any specific structure or function disclosed in the examples is only a representative context. According to the teachings of the present disclosure, those skilled in the art should understand that any disclosed implementation aspect can be implemented alone, or two or more aspects can be combined and implemented together.

[0170] Although the present application has been described by way of examples and preferred embodiments, it is to be understood that the present application is not limited to the disclosed embodiments. On the contrary, the present application is intended to cover various modifications and equivalent arrangements (as would be apparent to those skilled in the art). It is to be understood that the appended claims are to be interpreted in the broadest way possible in conformity with the principles of the present application.

Claims

1. A gaze estimation system, comprising: A camera device configured to capture an image of a subject; as well as A processing circuit is coupled to the camera device, the processing circuit including: An image processing unit is configured to derive a scaled face image and a scaled eye image from the image of the subject; and A single-line inference unit is configured as follows: A first feature representation is extracted from the scaled face image using a first feature extraction model; A second feature representation is extracted from the scaled eye image using a second feature extraction model; Combine the first feature representation and the second feature representation to generate a combined feature representation; and A regression model is used to determine the subject's line of sight based on the combined feature representation.

2. The gaze estimation system as described in claim 1, further comprising: A model training unit is configured to jointly train the first feature extraction model, the second feature extraction model, and the regression model using a training dataset comprising multiple pairs of training face images, training eye images, and corresponding multiple gaze labels.

3. The gaze estimation system of claim 2, wherein the model training unit is further configured to use a training loss including a gaze estimation loss and an adversarial loss to optimize the first feature extraction model; The gaze estimation loss is minimized to preserve multiple gaze-related features, and the adversarial loss is minimized to suppress multiple gaze-independent features, wherein the multiple gaze-independent features include multiple brightness features and multiple individual appearance features.

4. The gaze estimation system of claim 3, wherein the model training unit is further configured to generate a reconstructed face image from the first feature representation using a reconstruction model, and to calculate the adversarial loss based on a reconstruction loss between the reconstructed face image and the training face image.

5. The gaze estimation system of claim 2, wherein the model training unit is further configured to identify a face bounding box and an eye bounding box from a training image, apply a data augmentation to the training image, and derive one pair of the training face image and the training eye image of the training dataset from the face bounding box and the eye bounding box. This data enhancement includes at least one of the following operations: Apply a scaling transformation to the face bounding box and the eye bounding box; Apply a translation transformation to the face bounding box and the eye bounding box; or Adjust the brightness level of the training image.

6. The gaze estimation system of claim 5, wherein the data augmentation further includes generating a mask based on a plurality of eye markers and applying the mask to the training image to selectively preserve a gaze-related region in the training eye image.

7. The gaze estimation system of claim 2, wherein the model training unit is further configured to use an auxiliary task to train the second feature extraction model, the auxiliary task being configured to predict one or more predefined eye features from the training eye image; The auxiliary task is designed to encourage the second feature extraction model to retain information related to a plurality of predefined eye features in the second feature representation, wherein the plurality of predefined eye features include at least one of the following features: eyelids; corner of the eye; Iris boundary; The limbus boundary; The center of the pupil; or One glance at the center of the ball.

8. The gaze estimation system of claim 2, wherein the model training unit is further configured to train the first feature extraction model using an auxiliary task, the auxiliary task being configured to predict one or more predefined facial features from the training face image; The auxiliary task is designed to encourage the first feature extraction model to retain information related to a plurality of predefined facial features in the first feature representation, wherein the plurality of predefined facial features include at least one of the following features: Facial geometric information; or Head posture information.

9. The gaze estimation system of claim 2, wherein the model training unit is further configured to use a training process to train the regression model, the training process including applying a classification loss to a coarse gaze direction estimate and a regression loss to a detailed gaze direction estimate. The weights of the classification loss and the regression loss decrease and increase respectively with the training period.

10. The gaze estimation system of claim 2, wherein the gaze inference unit is further configured to generate a coarse gaze classification by applying a classification model to the combined feature representation, and to apply the regression model to determine the gaze direction based on the combined feature representation and the coarse gaze classification; The training unit of the model is further configured to train the classification model using a classification loss and the regression model using a regression loss.

11. The gaze estimation system of claim 1, wherein the image processing unit is further configured to derive the scaled face image and the scaled eye image by performing a plurality of operations, the plurality of operations including: Use an object detection model or a marker detection model to identify a face bounding box and an eye bounding box from the image of the subject; Based on the face bounding box and the eye bounding box, a cropped face image and a cropped eye image are respectively cropped from the image of the subject; as well as The cropped face image and the cropped eye image are reduced in size to generate the scaled face image and the scaled eye image, respectively.

12. The gaze estimation system of claim 11, wherein the image processing unit is further configured to reduce the cropped face image and the cropped eye image such that the scaled eye image retains more pixel-level details corresponding to one eye of the subject than a corresponding eye region within the scaled face image.

13. The gaze estimation system of claim 11, wherein in response to detecting a plurality of face regions in the image of the subject, the image processing unit is further configured to select the face bounding box based on proximity to a predefined reference position within the image.

14. The gaze estimation system of claim 11, wherein the image processing unit is further configured to identify a left eye bounding box and a right eye bounding box from the image of the subject, and select the one closer to the camera device from the left eye bounding box and the right eye bounding box as the eye bounding box for cropping the cropped eye image.

15. The gaze estimation system of claim 14, wherein the image processing unit is further configured to maintain the selected eye bounding box across multiple frames, and to switch to other eye bounding boxes to crop the cropped eye image only when a predefined condition indicating a sufficiently varied condition is met.

16. The gaze estimation system of claim 1, wherein the camera device is a near-infrared camera configured to capture the image of the subject as a grayscale image, and the first feature extraction model, the second feature extraction model, and the regression model are trained using multiple grayscale training images converted from multiple color images through a model trained to simulate multiple near-infrared image features.

17. The gaze estimation system of claim 1, wherein the processing circuitry is further configured to perform at least one of a plurality of actions based on the determined gaze direction, the plurality of actions including: Modify a display output of a display device; A flashing indicator light; An audio alarm is generated through an audio output device; Activate an autonomous driving unit to take over vehicle control; or A tactile output device is activated to generate a vibration alarm.

18. A gaze estimation method, executed by one or more processors, the gaze estimation method comprising: Derive a scaled face image and a scaled eye image from an image of a subject captured by a camera device; A first feature representation is extracted from the scaled face image using a first feature extraction model, and a second feature representation is extracted from the scaled eye image using a second feature extraction model; The first feature representation and the second feature representation are combined to generate a combined feature representation; as well as A regression model is used to determine the subject's line of sight based on the combined feature representation.

19. The line-of-sight estimation method as described in claim 18, further comprising: The first feature extraction model, the second feature extraction model, and the regression model are trained together using a training dataset consisting of multiple pairs of training face images, training eye images, and corresponding multiple gaze labels.

20. The line-of-sight estimation method as described in claim 19, further comprising: The first feature extraction model is optimized using a training loss that includes a gaze estimation loss and an adversarial loss. The gaze estimation loss is minimized to preserve multiple gaze-related features, and the adversarial loss is minimized to suppress multiple gaze-independent features, wherein the multiple gaze-independent features include multiple brightness features and multiple individual appearance features.

21. The line-of-sight estimation method as described in claim 20, further comprising: A reconstruction model is used to generate a reconstructed face image from the first feature representation; as well as The adversarial loss is calculated based on a reconstruction loss between the reconstructed face image and the training face image.

22. The line-of-sight estimation method as described in claim 19, further comprising: Identify a face bounding box and an eye bounding box from a training image; Apply a data augmentation to the training image; as well as Derive one pair of training face images and training eye images from the face bounding box and the eye bounding box of the training dataset; This data enhancement includes at least one of the following operations: Apply a scaling transformation to the face bounding box and the eye bounding box; Apply a translation transformation to the face bounding box and the eye bounding box; or Adjust the brightness level of the training image.

23. The gaze estimation method of claim 22, wherein the data augmentation further includes generating a mask based on a plurality of eye markers and applying the mask to the training image to selectively preserve a gaze-related region in the training eye image.

24. The line-of-sight estimation method as described in claim 19, further comprising: The second feature extraction model is trained using an auxiliary task configured to predict one or more predefined eye features from the training eye image. The auxiliary task is designed to encourage the second feature extraction model to retain information related to a plurality of predefined eye features in the second feature representation, wherein the plurality of predefined eye features include at least one of the following features: eyelids; corner of the eye; Iris boundary; The limbus boundary; The center of the pupil; or One glance at the center of the ball.

25. The line-of-sight estimation method as described in claim 19, further comprising: The first feature extraction model is trained using an auxiliary task configured to predict one or more predefined facial features from the training face image. The auxiliary task is designed to encourage the first feature extraction model to retain information related to a plurality of predefined facial features in the first feature representation, wherein the plurality of predefined facial features include at least one of the following features: Facial geometric information; or Head posture information.

26. The line-of-sight estimation method as described in claim 19, further comprising: The regression model is trained using a training process that includes applying a classification loss to a coarse gaze direction estimate and a regression loss to a detailed gaze direction estimate. One weight of the classification loss decreases, while the other weight of the regression loss decreases and increases with the training period, respectively.

27. The line-of-sight estimation method as described in claim 19, further comprising: A coarse gaze classification is generated by applying a classification model to the combined feature representation, and the regression model is applied to determine the gaze direction based on the combined feature representation and the coarse gaze classification. The classification model is trained using a classification loss and the regression model is trained using a regression loss.

28. The gaze estimation method of claim 18, wherein deriving the scaled face image and the scaled eye image further comprises: Use an object detection model or a marker detection model to identify a face bounding box and an eye bounding box from the image of the subject; Based on the face bounding box and the eye bounding box, a cropped face image and a cropped eye image are respectively cropped from the image of the subject; as well as The cropped face image and the cropped eye image are reduced in size to generate the scaled face image and the scaled eye image, respectively.

29. The gaze estimation method of claim 28, wherein the cropped face image and the cropped eye image are reduced such that the scaled eye image retains more pixel-level details corresponding to one eye of the subject than the corresponding eye region in the scaled face image.

30. The line-of-sight estimation method as described in claim 28, further comprising: In response to the detection of multiple face regions in the image of the subject, the face bounding box is selected based on its proximity to a predefined reference position within the image.

31. The line-of-sight estimation method as described in claim 28, further comprising: A left eye bounding box and a right eye bounding box were identified from the image of the subject; as well as Choose the one closer to the camera device from the left eye bounding box and the right eye bounding box as the eye bounding box for cropping the cropped eye image.

32. The line-of-sight estimation method as described in claim 31, further comprising: The selected eye bounding box is maintained across multiple frames, and the process switches to another eye bounding box to crop the cropped eye image only when a predefined condition indicating sufficient variation is met.

33. The gaze estimation method of claim 18, wherein the camera device is a near-infrared camera configured to capture the image of the subject as a grayscale image, and wherein the first feature extraction model, the second feature extraction model, and the regression model are trained using multiple grayscale training images converted from multiple color images through a model trained to simulate multiple near-infrared image features.

34. The line-of-sight estimation method as described in claim 18, further comprising: Based on the determined line-of-sight direction, at least one of a plurality of actions is performed, wherein the plurality of actions includes: Modify a display output of a display device; A flashing indicator light; An audio alarm is generated through an audio output device; Activate an autonomous driving unit to take over vehicle control; or A tactile output device is activated to generate a vibration alarm.

Images