An individualized line-of-sight estimation method and device based on identity-aware feature decoupling

Abstract

This invention provides a personalized gaze estimation method and apparatus based on identity-aware feature decoupling, relating to the technical field of machine vision technology. The method includes: acquiring multimodal input data, including a full-face image, at least one local eye image, and a spatial location vector; extracting mixed features based on the multimodal input data; decoupling the mixed features to obtain shared gaze features and individual bias features; and generating a personalized gaze estimation result based on the shared gaze features and the individual bias features. This invention solves the problem of low gaze estimation accuracy, thereby improving the accuracy of gaze estimation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of machine vision, and more specifically, to an individualized gaze estimation method and apparatus based on identity perception feature decoupling. Background Technology

[0002] Gaze estimation, a core task in computer vision and human-computer interaction, is used to accurately predict the direction of human gaze or the point of focus on the screen by analyzing facial or eye images. This technology plays a crucial role in diverse scenarios such as augmented reality, driver assistance systems, smart education, and medical assisted diagnosis.

[0003] Currently, most mainstream gaze estimation techniques rely on deep learning models, which are typically trained on large datasets containing numerous subjects to learn functions that map image features to gaze direction. However, such general-purpose models reveal a fundamental technical problem in practical applications: they often overlook significant differences between individuals in facial geometry, physiological structure (e.g., the difference in the Kappa angle between the visual gaze line and the anatomical center line of the eye), and gaze behavior habits. This inherent individual variability leads to a common feature coupling problem in existing models, where features directly related to gaze are often mixed with identity-sensitive information such as skin texture, lighting conditions, and eyeglass occlusion. This reduces the robustness of the model when applied to new users (i.e., cross-individual applications), making it difficult to capture and compensate for the physiological biases of specific users, thus limiting further improvements in prediction accuracy. Summary of the Invention

[0004] This invention provides an individualized gaze estimation method and apparatus based on identity-aware feature decoupling, which at least solves the problem of low gaze estimation accuracy in related technologies.

[0005] According to an embodiment of the present invention, a personalized gaze estimation method based on identity-aware feature decoupling is provided, comprising: Acquire multimodal input data, which includes a full-face image, at least one local image of an eye, and a spatial location vector; Based on the multimodal input data, extract hybrid features; The hybrid features are decoupled to obtain shared line-of-sight features and individual bias features; Based on the shared gaze features and the individual bias features, an individualized gaze estimation result is generated.

[0006] In an exemplary embodiment, the decoupling of the hybrid features to obtain shared gaze features and individual bias features includes: The shared gaze feature and the individual bias feature are generated by mapping from the hybrid features through a shared feature branch network and an individual bias branch network, respectively. The individual bias features are input into the identity discriminator, and the individual bias branch network and the identity discriminator are supervised by the identity classification loss function so that the individual bias features carry identity information. The shared gaze features are input into the identity discriminator via a gradient inversion layer, and the shared feature branch network is supervised by an adversarial purification loss function to extract identity information from the shared gaze features.

[0007] In one exemplary embodiment, the method further includes: Meta-training of the pre-defined model is performed on a gaze estimation task involving multiple individuals to obtain initial model parameters; Obtain a support set for new subjects that includes at least one calibration sample; Based on the support set, the initial model parameters are adaptively gradient-adapted to generate personalized model parameters for new subjects. The individualized gaze estimation result is generated based on the personalized model parameters.

[0008] In one exemplary embodiment, extracting hybrid features based on the multimodal input data includes: Global facial semantic vectors are extracted from the full-face image using facial feature branches; Spatial position vectors are extracted from the spatial position vectors using position feature branches; The global facial semantic vector and the spatial location vector are concatenated to obtain the modulation factor; Eye feature maps are extracted from at least one of the said local eye images using eye feature branches; The eye feature map is nonlinearly modulated according to the modulation factor to obtain eye modulation features; The eye modulation features, the global facial semantic vector, and the spatial location vector are fused to obtain the hybrid features.

[0009] In one exemplary embodiment, acquiring multimodal input data includes: Based on the spatial coordinate system of facial key points, calculate the geometric center of the set of facial key points; Centered on the geometric center, and with a preset multiple of the maximum horizontal or vertical span of the set of facial key points as the side length, a square cropping area for the face is constructed, and the full-face image is cropped from the original image. Based on the proportional relationship between key points of the eye, the rectangular bounding boxes of the left and right eyes are dynamically calculated, and at least one local image of the eye is cropped from the original image.

[0010] According to another embodiment of the present invention, a personalized gaze estimation device based on identity-aware feature decoupling is provided, comprising: The data acquisition module is used to acquire multimodal input data, which includes a full-face image, at least one local image of an eye, and a spatial location vector. The feature extraction module is used to extract mixed features based on the multimodal input data; A feature decoupling module is used to decouple the hybrid features to obtain shared line-of-sight features and individual bias features; The result generation module is used to generate individualized gaze estimation results based on the shared gaze features and the individual bias features.

[0011] In an exemplary embodiment, the decoupling of the hybrid features to obtain shared gaze features and individual bias features includes: The shared gaze feature and the individual bias feature are generated by mapping from the hybrid features through a shared feature branch network and an individual bias branch network, respectively. The individual bias features are input into the identity discriminator, and the individual bias branch network and the identity discriminator are supervised by the identity classification loss function so that the individual bias features carry identity information. The shared gaze features are input into the identity discriminator via a gradient inversion layer, and the shared feature branch network is supervised by an adversarial purification loss function to extract identity information from the shared gaze features.

[0012] In one exemplary embodiment, the apparatus further includes: Meta-training of the pre-defined model is performed on a gaze estimation task involving multiple individuals to obtain initial model parameters; Obtain a support set for new subjects that includes at least one calibration sample; Based on the support set, the initial model parameters are adaptively gradient-adapted to generate personalized model parameters for new subjects. The individualized gaze estimation result is generated based on the personalized model parameters.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0014] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This invention achieves explicit decoupling between gaze-related features and identity-sensitive information, purifying mixed features into identity-independent general gaze representations and individual-related behavioral features. This solves the feature coupling problem of existing models, improves the robustness of the model in cross-individual applications, and thus addresses the problem of low gaze estimation accuracy, thereby improving gaze estimation accuracy. Attached Figure Description

[0016] Figure 1 This is a flowchart of an individualized gaze estimation method based on identity-aware feature decoupling according to an embodiment of the present invention; Figure 2 This is a structural block diagram illustrating the structure of an individualized gaze estimation network model according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the identity-aware feature decoupling module according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0019] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0020] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0021] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0022] This embodiment provides a personalized gaze estimation method based on identity-aware feature decoupling. This method employs a deep learning framework including a feature extraction network, an identity-aware feature decoupling module, and a model-independent meta-learning mechanism. It can explicitly decompose the input mixed image and geometric features into identity-independent shared gaze features and individual features related to individual physiological biases. Furthermore, it rapidly adapts the model to new subjects using a small number of calibration samples. This method solves the technical problems of existing gaze estimation techniques, such as severe feature coupling due to neglecting individual differences, weak cross-individual generalization ability, and inability to compensate for individual physiological biases. It achieves the beneficial effects of eliminating interference from individual physiological differences and realizing high-precision personalized gaze estimation.

[0023] like Figures 1-3 As shown, the specific steps include the following:

[0024] S100: Acquire multimodal input data containing a full-face image, at least one local image of an eye, and a spatial location vector.

[0025] By traversing and reading the subject dataset, the system synchronously loads multimodal annotation information, including but not limited to two-dimensional gaze screen landing point coordinates, position coordinates of 68 or more key points on the face, and head posture parameters (such as yaw, pitch, and roll).

[0026] In the process of precise localization of the image region, in order to ensure complete coverage of the subject's global facial features, identity information, and ambient lighting, the system first accurately calculates the geometric center of the set of all key points on the face. Then, using 1.5 times the maximum span of all key points in the horizontal or vertical direction as a preset multiple, and using this as the side length, a square cropping region that can completely accommodate the face is constructed. This strategy not only ensures the integrity of the facial region, but also provides a stable and standardized input for subsequent feature extraction.

[0027] For the extraction of local eye details, the system dynamically calculates the bounding boxes surrounding the left and right eyes based on the inherent proportional relationship between key eye points (such as the inner and outer corners of the eyes and the contour points of the upper and lower eyelids). In order to prevent the cropped area from exceeding the effective boundary of the original image, the system also implements boundary constraint checks.

[0028] To ensure the consistency of the dimensions of each component in the subsequent loss function calculation, this method also normalizes the true value of the coordinates of the two-dimensional line of sight landing point. This normalization operation is based on the physical size of the display screen.

[0029] For example, suppose the true coordinates of a subject's gaze point are ( ), and its corresponding display screen size is ( Normalized coordinates () The calculation method is as follows: For example, on a screen with a width of 50cm and a height of 30cm, a gaze point with physical coordinates (25cm, 15cm) has normalized coordinates of (0.5, 0.5). Subsequently, all gaze coordinates are mapped to the dimensionless interval [0, 1], so that the gaze regression loss (such as mean squared error loss) used for subsequent supervised model training also becomes a dimensionless scalar value, and so on.

[0030] Finally, through a series of spatial transformation operations, the original image was finely cropped into a single image with a resolution of [resolution value missing]. A full-face image of pixels, and two images with a resolution of [resolution value missing]. A pixel-level local image of the eye. While performing image cropping, the system simultaneously extracts and encapsulates a series of geometric features reflecting the spatial relationship between the eyes and the face, forming a 12-dimensional geometric feature vector (i.e., a spatial position vector). This vector, together with the cropped image and the normalized gaze coordinates, constitutes the multimodal input data source required by the subsequent identity-aware feature decoupling model.

[0031] For example, suppose an image frame is read from a dataset, and its corresponding set of facial keypoint coordinates is: Then perform the following operations: S110: Facial area positioning and cropping.

[0032] First, calculate the geometric center of the keypoint set ( ),in , Assuming the calculation yields... .

[0033] Next, calculate the horizontal span. and vertical span ; Assumption Pixels pixels, take the maximum span Pixel.

[0034] Then, determine the side length of the cropping frame. Pixel.

[0035] Finally, with Centered on the cropping box, define the coordinates as follows: ,Right now ,get Then crop the area from the original image and scale it to... Pixels, to obtain a full-face image .

[0036] S120: Eye area positioning and cropping.

[0037] Taking the left eye as an example, its bounding rectangle is calculated using the coordinates of six keypoints of the left eye (e.g., keypoint indices 37 to 42); assuming the calculated rectangle is... The system enlarges the rectangle by a certain scale (e.g., 20% in width and height) to include periorbital information and performs boundary checks; finally, it is cropped and scaled to... Pixels, to obtain the left eye image Perform the same operation on the right eye to obtain .

[0038] S130: Construction of geometric feature vectors.

[0039] This 12-dimensional vector It can include the following information: Normalized coordinates and dimensions of the full-face cropping frame ( ), normalized coordinates and dimensions of the left and right eye cropping frames ( These coordinates are all normalized relative to the original image size, for example, The final vector obtained Possibly These data together constitute the input to the model.

[0040] S200: Based on multimodal input data, it extracts mixed features through a feature extraction network.

[0041] This is performed within a deep neural network consisting of multiple parallel branches to extract complementary information from data of different modalities and achieve deep fusion.

[0042] In the facial feature branch, for the input shape is The full-face image tensor is processed, where B is the batch size. This branch begins with a convolutional block containing a convolutional kernel of size B. A convolutional layer with a stride of 2 and 48 filters. After this layer, the shape of the output feature map becomes... The second convolutional block follows, containing a kernel size of [size missing]. A convolutional layer with a stride of 1 and 96 filters outputs a feature map of shape [shape missing]. Next, a series of operations including convolution, batch normalization, ReLU activation, and max pooling (pooling size is...) are performed. The residual blocks are processed by progressively compressing the spatial dimension of the feature map while increasing the channel dimension. After multiple stages of downsampling, the final generated shape is... The feature map is flattened and passed through a fully connected layer, ultimately mapped to a shape of... Vectors that contain global facial semantics .

[0043] At the same time, the auxiliary shape is Location feature vector The data is fed into a multilayer perceptron (MLP) consisting of four fully connected layers for dimensionality upscaling and reconstruction. The dimensions of each hidden layer in this MLP can be set to 32, 64, 128, and 64, respectively, and each layer is followed by a ReLU activation function. The output of this MLP is a shape that is also... Spatial position vector .

[0044] Next, the 64-dimensional global facial semantic vector With 64-dimensional spatial position vector By splicing along the feature dimension, a shape is formed. The modulation factor, as a conditional information, directly affects the extraction process of eye features.

[0045] In the eye feature extraction stage, the left and right eye images (shaped as follows) Each feature extraction branch passes through its own sub-branch, but they share the same network weights; the image is first processed by an initial convolutional module containing a kernel size of [missing information]. A convolutional layer with a stride of 2 and 48 filters generates a shape of... The feature map is then processed via an Adaptive Group Normalization (AGN) layer, which utilizes the aforementioned 128-dimensional modulation factor to non-linearly regulate the eye feature map. Specifically, the modulation factor generates the scaling and translation parameters required for the normalization layer through a small network, thereby dynamically injecting global facial geometry and positional information into the extraction of local eye features. After a second stage of convolution and attention mechanisms (such as SELayer) enhancement, the feature map dimension is transformed to... Finally, after the third stage of downsampling and multi-scale convolution, the network extracts two sets of key feature maps, both with spatial dimensions of [missing information]. The two feature maps, with 64 and 64 channels respectively, are then concatenated to form a 128-dimensional monocular fusion feature. The fusion features of the left and right eyes are ultimately integrated to form the overall eye feature, which is then combined with... and Together, they constitute the hybrid features input to the next module. .

[0046] For example, taking the AGN module of the eye feature branch as an example, its workflow is as follows: Input eye feature map Its tensor shape is ,in Enter the batch size; then enter the modulation factor. Its shape is .

[0047] The AGN module contains two small MLPs, respectively called... and .

[0048] The input is simultaneously fed into both MLPs.

[0049] Mapping a 128-dimensional input to a 48-dimensional vector ,Right now ,this The vector will be used as a scaling parameter for Group Normalization (GN).

[0050] Mapping a 128-dimensional input to a 48-dimensional vector ,Right now ,this The vector will be used as the offset parameter of GN.

[0051] Then, the input feature map Perform GN operation. Assume the 48 channels are divided into... Groups (e.g.) (6 channels per group). Within each group, the mean and variance of the features are calculated and normalized.

[0052] Finally, the dynamically generated and Applying the channel to the normalized feature map: ,in This indicates element-wise multiplication.

[0053] Through this process, the geometric information of the entire face (encoded in...) (The middle part) was used to dynamically adjust the statistical properties of eye features, realizing nonlinear deep fusion of cross-modal features.

[0054] S300: Adversarial purification training, which involves inputting the mixed features into the identity-aware feature decoupling module, which decouples the mixed features into shared gaze features and individual bias features.

[0055] Extracting hybrid features from the backbone network The core task of this module is then to explicitly divide the feature into two semantically independent subspaces using two parallel multilayer perceptron (MLP) branches; specifically: S310: Shared gaze feature branch.

[0056] The goal of this branch is to extract features that are completely unrelated to identity and only carry pure gaze direction semantics. : (1) in, It is determined by parameters Defined multilayer perceptron network.

[0057] For example, a shared feature branch network It can be composed of three fully connected layers, with the input dimension being a mixture of features. The dimensions of the hidden layers are 256 and 128 respectively, and the output dimension (i.e.) The dimension can be set to 64. Each layer, except the output layer, is followed by a Modified Linear Unit (ReLU) activation function.

[0058] S320: Individual bias feature branch.

[0059] The goal of this branch is to capture identity-related information such as individual-specific facial structure, physiological biases (such as the Kappa angle), and behavioral habits. : (2) in, It is determined by parameters Defined multilayer perceptron network.

[0060] For example, individual biased branch network Can be adopted with A similar architecture, for example, consists of three fully connected layers, with hidden layer dimensions of 256 and 128 respectively, and an output dimension (i.e. The dimension can be set to 64.

[0061] To ensure It truly stripped away identity information, and The identity information was accurately captured, and the present invention uses an adversarial purification mechanism based on a gradient inversion layer (GRL) to process it: The core of this mechanism is the identity detector. Its goal is to determine which subject a given feature belongs to based on that feature (i.e., to predict the identity label). ).

[0062] For example, an identity discriminator It can be composed of two fully connected layers, with the input dimension being the feature dimension (e.g., 64), the hidden layer dimension being 128, and the output layer dimension being the total number of subjects in the dataset. The classification probability is then output through the Softmax function.

[0063] During the training process, After being sent to the identity verification device Previously, it would pass through a gradient inversion layer (GRL). The mathematical definition of this layer is as follows: Its forward propagation is an identity transformation: (3) Its backpropagation is the gradient multiplied by a negative constant: (4) in, It is the identity matrix. It is a hyperparameter used to control the intensity of gradient reversal; preferably, this hyperparameter... It can be set to 1.0.

[0064] Shared feature branches via GRL Identity detector This forms an adversarial game, with a discriminator. from Identify identity information in the middle, and Driven by gradient reversal, the parameters are actively adjusted to generate a gradient that allows for gradient reversal. Even if the characteristics that cannot be identified are obtained Their "identity has become blurred".

[0065] The entire module is optimized by combining multiple loss functions: Identity Classification Loss Used to supervise individual bias branches It accurately captures identity information and is also used to train an identity discriminator. It typically employs cross-entropy loss: (5) Anti-purification loss : Acts on shared feature branches, driven by GRL destroy The accuracy of identity recognition is also represented by cross-entropy loss: (6) The module's ultimate optimization goal is a minimization-maximization game problem. By balancing loss terms with different weights, it aims to achieve complete purification of gaze features in the latent space; the total loss function... It can be represented as: (7) in It is the regression loss used to supervise the final line-of-sight estimation results. and These are hyperparameters used to adjust the decoupling strength; preferably, these hyperparameters... and They can be set to 0.1 and 0.5 respectively.

[0066] like Figure 3 As shown, the operating logic of this module is as follows: after the initial fusion of mixed features, it is divided into parallel MLP branches. and ; The data is directly fed into the identity detector for supervised training. Then, the data is fed back into the same identity discriminator with shared parameters through a gradient reversal layer for adversarial training.

[0067] S400: Generates individualized gaze estimation results based on shared gaze features and individual bias features.

[0068] After feature decoupling, and These features are concatenated to form a complete feature vector, which contains both general gaze information and individual bias information. This vector is then fed into a regression head, typically an MLP, to map high-dimensional features to two-dimensional normalized gaze point coordinates. For example, this regression head can consist of two fully connected layers with an input dimension of... and The concatenated dimensions (e.g.) The hidden layer can have a dimension of 64, and the output layer has a dimension of 2, corresponding to two-dimensional normalized coordinates. .

[0069] The supervision of this step is based on the loss of visual return. Complete. Since the line-of-sight coordinates have already been normalized in S100, here... It is a dimensionless value, and the mean squared error (MSE) loss function is typically used: (10) in( ) represents the two-dimensional normalized line-of-sight coordinates predicted by the model, ( () represents the true two-dimensional normalized line-of-sight coordinates, and N is the number of samples in the batch. The optimizer used here is the Adam optimizer, and the learning rate is set to 0.0001.

[0070] In the final output, the model predicts the normalized coordinates ( The coordinates need to be converted to physically meaningful coordinates through an inverse normalization operation. This operation involves multiplying the normalized coordinates by the corresponding physical screen size. This is accomplished by ( ), that is, the final output physical coordinates. That is, to obtain the point where the line of sight falls.

[0071] Preferably, a model-agnostic meta-learning (MAML) mechanism is also introduced to achieve rapid calibration and personalization for new subjects, specifically: S500: Meta-learning adaptive calibration, also known as MAML meta-learning fast calibration.

[0072] Here, we ensure that the model can keenly perceive and adapt to the differences in gaze characteristics among different individuals. This is achieved by meta-training on gaze tasks with multiple different individuals to find a set of initial model parameters that are highly sensitive to individual differences and have excellent transfer capabilities. The calibration process is divided into two stages: meta-training and rapid adaptation. S510: Meta-training phase.

[0073] This stage comprises two cycles, one internal and one external.

[0074] S511: Inner-loop: Individual-specific task adaptation.

[0075] For a selected subject task A small number of calibration samples (e.g., 3-5 frames of images and their corresponding gaze labels) are drawn from the subject's support set. The model calculates the gaze regression loss on this task. The parameters are updated through one or more gradient descent steps to obtain a set of temporary parameters specific to the subject. The formula is as follows: (8) in, The learning rate for the inner loop is used to quickly capture the subject's specific physiological biases. Preferably, the inner loop learning rate... It can be set to 1e-4.

[0076] For example, for a task Assuming its support set contains 3 samples, and their regression losses are 0.02, 0.03, and 0.025 (all dimensionless), then the average loss for this task is... The system calculates this loss with respect to the current model parameters. gradient Assuming This is a weight parameter in the model, currently valued at 1.5, with a calculated gradient of -0.8. After the inner loop update, what is the temporary value of this parameter? .

[0077] S512: Outer-loop: Global optimization of meta-parameters.

[0078] In all tasks After all inner loops have completed adaptation, the adapted models are evaluated on their respective query sets. The performance of the task is then evaluated. Then, based on the total loss of all tasks on the query set, its performance relative to the original initialization parameters is calculated. The gradient is used to update the value. : (9) in, This is the learning rate for meta-learning (outer loop). Preferably, the outer loop learning rate is... It can be set to 1e-3. Through this second-order gradient optimization, the model finally learns the initial parameters. It can guarantee that when facing a completely new subject, only a very small parameter shift (i.e., one or a few internal loop updates) is needed to achieve a high-precision estimate.

[0079] S520: Rapid adaptation and inference phase.

[0080] When a new subject appears, a very small number of calibration samples (e.g., 3-5 images) are first collected to form a support set. Then, high-quality initialization parameters obtained from meta-training are used. Starting from this point, execute one inner loop operation as described in S511 to obtain personalized model parameters tailored for the new subject. All subsequent gaze estimation tasks for this subject will use this set of personalized parameters. This allows for reasoning, thereby enabling efficient and precise individualized calibration.

[0081] This invention was experimentally validated on the publicly available dataset MPIIFaceGaze. This dataset contains over 40,000 eye and facial images collected from 15 participants in their daily lives. Leave-one-out training and testing were conducted. During training, the MSE loss function was used to supervise the regression of gaze coordinates, and Euclidean distance was introduced as an evaluation metric. Experimental results show that the personalized gaze estimation network model based on identity-aware feature decoupling designed in this invention achieves an average gaze error of 3.12 cm on the test set, outperforming several similar advanced methods, demonstrating the effectiveness and superiority of the technical solution of this invention.

[0082] In summary, this application successfully addresses the individual variability problem in traditional gaze estimation models by combining an innovative identity-aware feature decoupling framework with a model-independent meta-learning mechanism. First, through adversarial training, this method achieves clear separation of gaze information and identity information at the feature level, laying a solid foundation for subsequent personalized modeling. Then, leveraging the rapid adaptability of meta-learning, the model can accurately calibrate new users with only a few frames, effectively compensating for errors caused by individual physiological biases. The entire process is logically rigorous and technically advanced, significantly improving the accuracy and practicality of personalized gaze estimation.

[0083] The following examples illustrate this point.

[0084] A personalized gaze estimation network model based on identity-aware feature decoupling was designed to perform personalized calibration of gaze estimation. The model structure is as follows: Figure 2 As shown, the specific implementation steps are as follows: 1. Input and Data Preprocessing In the input preprocessing stage, the system first traverses and reads the subject dataset, simultaneously loading multimodal annotation information, including the coordinates of the two-dimensional gaze point, the positions of facial key points, and head pose parameters. During image region localization, the invention accurately calculates the geometric center of the facial key point set based on the spatial coordinate system of facial key points, and constructs a square cropping region of the face using 1.5 times the maximum horizontal or vertical span of the key points as the side length, thereby ensuring complete coverage of the subject's global facial features, identity information, and ambient lighting. For local detail extraction, the system dynamically calculates the left and right eye rectangles based on the proportional relationship between eye key points and performs boundary constraint checks to ensure that the cropping range does not exceed the original image boundary. Finally, through spatial transformation, the original image is finely cropped into a 224×224 pixel full-face image and two 112×112 pixel local eye images. Simultaneously, 12-dimensional geometric feature vectors (Rects) reflecting the spatial relationship between the eyes and the face are extracted and encapsulated, forming a high-quality input data source for the subsequent identity-aware feature decoupling model.

[0085] 2. Preliminary extraction and fusion of branch features In the facial feature branch, a full-face image with a resolution of 224×224×3 is input and processed by two layers of large-size convolutional kernels (5×5), resulting in feature map sizes of 110×110×48 and 106×106×96 respectively. Subsequently, through multi-level max pooling and hierarchical convolution, the feature space is compressed to 5×5×64, and finally mapped to a 64-dimensional global facial semantic vector through a fully connected layer. At the same time, the auxiliary position feature vector (1×12) is up-dimensioned and reconstructed through four fully connected layers, outputting a 64-dimensional spatial position vector. The 128-dimensional modulation factor formed by concatenating the two vectors directly affects the eye branch.

[0086] In the eye feature extraction stage, the binocular images (112×112×3) are first processed into a 50×50×48 feature map by the initial convolution module, and adaptive group normalization (AGN) is introduced for non-linear adjustment. After the second stage of convolution and attention mechanism (SELayer) enhancement, the feature map dimension is transformed to 22×22×64. Then, after the third stage of downsampling and multi-scale convolution, two sets of key feature maps are extracted, both with a spatial size of 10×10 and 64 channels respectively. The two are spliced ​​to form a 128-dimensional monocular fusion feature.

[0087] 3. Identity-aware feature decoupling module First, there's the feature space partitioning and mapping logic. After the backbone network extracts the mixed features, this module uses two parallel multilayer perceptron (MLP) branches to explicitly partition the features into two independent subspaces: Shared gaze feature branch: Extracting features that are identity-independent and only carry gaze direction semantics. .

[0088] Individual bias feature branch: Captures information such as individual-specific facial structure and physiological biases (e.g., Kappa angle). .

[0089] Then, adversarial purification will be performed based on a gradient inversion layer (GRL). To ensure... This invention truly strips away identity information and introduces an identity discriminator. The goal of this discriminator is to determine the identity label of the subject based on the input features. During the training process, it is necessary to... Apply a Gradient Reversal Layer (GRL) operation. The mathematical definitions of its forward and backward propagation are as follows: Forward propagation: Backpropagation: Through GRL, the shared feature branch and its identity discriminator engage in an adversarial game. The discriminator attempts to accurately identify identities, while the feature extraction layer actively filters out unwanted features driven by gradient inversion. The sensitive identity information within makes it "identity ambiguous".

[0090] For the loss function used in the model, since in addition to constraining the direction or landing point of gaze regression, it is also necessary to supervise identity recognition, this module guides the decoupling process by combining multiple constraint terms: Identity classification loss is used to supervise the accuracy of individual branches in capturing identity information, while also supervising the performance of the discriminator. Cross-entropy loss is typically used, with the specific formula as follows: The adversarial purification loss, applied to the shared branch, corrupts the accuracy of identity recognition by driving the feature layer through GRL. The specific formula is as follows: The ultimate optimization goal of this module is a minimization-maximization (Min-Max) game problem. By balancing loss terms with different weights, it aims to achieve complete purification of gaze features in the latent space. in, and It is a hyperparameter used to adjust the decoupling strength.

[0091] like Figure 3 As shown, the identity-aware feature decoupling module receives the initial mixed features from branch fusion and explicitly divides them into shared gaze features through parallel multilayer perceptron branches. With individual behavioral characteristics Furthermore, an adversarial learning framework is constructed by introducing a gradient inversion layer (GRL) and a parameter-sharing identity feature classifier. In the module's operational logic, the shared branch uses the GRL to invert the gradient of the identity classifier during backpropagation, thereby forcibly stripping away identity-related noise information in the latent space and ensuring... It contains only pure gaze direction semantics; the corresponding individual branches are enhanced by supervised classification tasks to extract subject-specific physiological structures and gaze biases. By jointly optimizing gaze regression loss, supervised identity classification loss, and adversarial identity loss, the system achieves complete decoupling of gaze features and identity features. This not only improves the model's cross-individual generalization ability in complex scenes but also provides accurate and applicable feature vectors to eliminate interference from physiological differences such as the Kappa angle.

[0092] 4. MAML Meta-Learning Fast Calibration The Model-Agnostic Meta-Learning (MAML) framework is introduced to ensure that the model can keenly perceive the differences in gaze features among different individuals. Unlike traditional fine-tuning methods, the core of MAML lies in "learning how to learn," that is, by meta-training on gaze tasks of multiple different individuals, to find a set of initialization parameters that are extremely sensitive to individual differences and have excellent transferability.

[0093] The calibration process is divided into two stages: meta-training and fast adaptation. Inner-loop: Individual-specific task adaptation, tailored to the subject's task. A small number of samples (e.g., 3-5 frames) are drawn from the support set. The model calculates the gaze regression loss on this task. The parameters are updated through one or more gradient descent iterations to obtain parameters specific to that subject. The formula is as follows: in, The internal learning rate is used to capture the subject's specific physiological biases.

[0094] Outer loop: Global optimization of meta-parameters across all tasks. After adaptation, evaluate the adapted model on the query set. The performance of this gradient is used to update the original initialization parameters. The specific formula is as follows: (9) in, The meta-learning rate. Through this second-order gradient optimization, the model ultimately learns... It can ensure that high-precision estimation can be achieved with only a very small parameter shift when dealing with new subjects.

[0095] Model training Training was performed on the publicly available MPIIFaceGaze dataset, a widely used gaze estimation dataset containing over 40,000 eye and face images collected from 15 participants in real-world everyday environments. Leave-one-out training was used for the fifteen different subjects within the dataset.

[0096] In the experiment, the MSE loss function was used as the supervised loss function for the gaze estimation task, and its formula is as follows: in , This represents the two-dimensional line-of-sight coordinates predicted by the model. Represents the true 2D line-of-sight coordinates, and N is the number of samples in the batch. The optimizer used is the Adam optimizer with a learning rate of 0.0001.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0098] This embodiment also provides a personalized gaze estimation device based on identity-aware feature decoupling. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 2 This is a structural block diagram of an individualized gaze estimation device based on identity-aware feature decoupling according to an embodiment of the present invention, such as... Figure 2 As shown, the device includes: The data acquisition module is used to acquire multimodal input data, which includes a full-face image, at least one local image of an eye, and a spatial location vector. The feature extraction module is used to extract mixed features based on the multimodal input data; A feature decoupling module is used to decouple the hybrid features to obtain shared line-of-sight features and individual bias features; The result generation module is used to generate individualized gaze estimation results based on the shared gaze features and the individual bias features.

[0100] In an optional embodiment, the decoupling of the mixed features to obtain shared gaze features and individual bias features includes: The shared gaze feature and the individual bias feature are generated by mapping from the hybrid features through a shared feature branch network and an individual bias branch network, respectively. The individual bias features are input into the identity discriminator, and the individual bias branch network and the identity discriminator are supervised by the identity classification loss function so that the individual bias features carry identity information. The shared gaze features are input into the identity discriminator via a gradient inversion layer, and the shared feature branch network is supervised by an adversarial purification loss function to extract identity information from the shared gaze features.

[0101] In an optional embodiment, the apparatus further includes: Meta-training of the pre-defined model is performed on a gaze estimation task involving multiple individuals to obtain initial model parameters; Obtain a support set for new subjects that includes at least one calibration sample; Based on the support set, the initial model parameters are adaptively gradient-adapted to generate personalized model parameters for new subjects. The individualized gaze estimation result is generated based on the personalized model parameters.

[0102] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0103] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0104] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0105] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0106] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A personalized gaze estimation method based on identity-aware feature decoupling, characterized in that, include: Acquire multimodal input data, which includes a full-face image, at least one local image of an eye, and a spatial location vector; Based on the multimodal input data, extract hybrid features; The hybrid features are decoupled to obtain shared line-of-sight features and individual bias features; Based on the shared gaze features and the individual bias features, an individualized gaze estimation result is generated.

2. The method according to claim 1, characterized in that, The decoupling of the mixed features to obtain shared gaze features and individual bias features includes: The shared gaze feature and the individual bias feature are generated by mapping from the hybrid features through a shared feature branch network and an individual bias branch network, respectively. The individual bias features are input into the identity discriminator, and the individual bias branch network and the identity discriminator are supervised by the identity classification loss function so that the individual bias features carry identity information. The shared gaze features are input into the identity discriminator via a gradient inversion layer, and the shared feature branch network is supervised by an adversarial purification loss function to extract identity information from the shared gaze features.

3. The method according to claim 2, characterized in that, The method further includes: Meta-training of the pre-defined model is performed on a gaze estimation task involving multiple individuals to obtain initial model parameters; Obtain a support set for new subjects that includes at least one calibration sample; Based on the support set, the initial model parameters are adaptively gradient-adapted to generate personalized model parameters for new subjects. The individualized gaze estimation result is generated based on the personalized model parameters.

4. The method according to claim 1, characterized in that, The extraction of hybrid features based on the multimodal input data includes: Global facial semantic vectors are extracted from the full-face image using facial feature branches; Spatial position vectors are extracted from the spatial position vectors using position feature branches; The global facial semantic vector and the spatial location vector are concatenated to obtain the modulation factor; Eye feature maps are extracted from at least one of the said local eye images using eye feature branches; The eye feature map is nonlinearly modulated according to the modulation factor to obtain eye modulation features; The eye modulation features, the global facial semantic vector, and the spatial location vector are fused to obtain the hybrid features.

5. The method according to claim 1, characterized in that, The acquisition of multimodal input data includes: Based on the spatial coordinate system of facial key points, calculate the geometric center of the set of facial key points; Centered on the geometric center, and with a preset multiple of the maximum horizontal or vertical span of the set of facial key points as the side length, a square cropping area for the face is constructed, and the full-face image is cropped from the original image. Based on the proportional relationship between key points of the eye, the rectangular bounding boxes of the left and right eyes are dynamically calculated, and at least one local image of the eye is cropped from the original image.

6. A personalized gaze estimation device based on identity perception feature decoupling, characterized in that, include: The data acquisition module is used to acquire multimodal input data, which includes a full-face image, at least one local image of an eye, and a spatial location vector. The feature extraction module is used to extract mixed features based on the multimodal input data; A feature decoupling module is used to decouple the hybrid features to obtain shared line-of-sight features and individual bias features; The result generation module is used to generate individualized gaze estimation results based on the shared gaze features and the individual bias features.

7. The apparatus according to claim 6, characterized in that, The decoupling of the mixed features to obtain shared gaze features and individual bias features includes: The shared gaze feature and the individual bias feature are generated by mapping from the hybrid features through a shared feature branch network and an individual bias branch network, respectively. The individual bias features are input into the identity discriminator, and the individual bias branch network and the identity discriminator are supervised by the identity classification loss function so that the individual bias features carry identity information. The shared gaze features are input into the identity discriminator via a gradient inversion layer, and the shared feature branch network is supervised by an adversarial purification loss function to extract identity information from the shared gaze features.

8. The apparatus according to claim 6, characterized in that, The device further includes: Meta-training of the pre-defined model is performed on a gaze estimation task involving multiple individuals to obtain initial model parameters; Obtain a support set for new subjects that includes at least one calibration sample; Based on the support set, the initial model parameters are adaptively gradient-adapted to generate personalized model parameters for new subjects. The individualized gaze estimation result is generated based on the personalized model parameters.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images