Gaze tracking method and apparatus based on autoencoding transformation

Abstract

The gaze tracking method and device based on automatic coding transformation rotate a face image around x, y and z axes of a three-dimensional coordinate system, and rotate the face image around the projection to the image plane, calculate the homography matrix by using the point pairs before and after the face image projection, and store the homography matrix data as a homography array; an automatic coding transformation network is constructed, the weight of the automatic coding network is obtained by learning the transformation matrix; the automatic coding transformation network is trained, the encoder and the transformation encoder of the trained automatic coding transformation network are taken out, a multi-layer perceptron is composed, the gaze direction regression network is obtained, and the gaze direction is regressed on a small sample eye movement label training set; the gaze direction regression network is deployed in an application environment, and the gaze direction is estimated through the gaze direction regression network. The face feature aggregation can be better performed, and the performance of the gaze estimation task is improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a gaze tracking method and apparatus based on automatic encoding transformation. Background Technology

[0002] Currently, appearance-based gaze estimation has important applications in human-computer interaction, behavior understanding, and autonomous driving. In representation-based isovariant learning, it is generally assumed that applying transformations to data will cause covariance in the data feature space. Therefore, the transformations applied to the data can be reconstructed from the feature representations of the data before and after the transformation, thereby learning the feature representations of the data.

[0003] In many learning problems, the success of deep neural networks often depends on large amounts of labeled data, which are costly to collect. Therefore, the small data challenge has become increasingly important. To address this, many unsupervised and semi-supervised methods have been proposed to train complex models on small datasets. The amount of labeled data is typically small, while the amount of unlabeled data is large. The distribution of unlabeled data often provides clues for learning feature representations with strong generalization ability. The difference between unsupervised and semi-supervised methods lies in whether or not additional labeled data samples are used to train the model. Unlabeled data can help models reduce inter-domain differences between different tasks, which has also driven the development of numerous unsupervised and semi-supervised domain adaptation methods.

[0004] In many applications, a more flexible form of feature representation is desired, enabling unsupervised training by exploring the distribution of unlabeled data. However, CNNs, with their local receptive fields, struggle to extract features over a wide range. Transformers, on the other hand, have long receptive fields, but the computation of attention parameters is computationally expensive. Therefore, improving the performance of gaze estimation tasks has practical application significance. Summary of the Invention

[0005] Therefore, the present invention provides a gaze tracking method and apparatus based on automatic encoding transformation to improve the performance of gaze estimation tasks.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a gaze tracking method based on automatic encoding transformation, comprising:

[0007] Step 1: Rotate the face image around the x, y, and z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane. Calculate the homography matrix using the point pairs before and after the face image projection, and collect the homography matrix data as a homography array.

[0008] Step 2: Construct an autoencoder transformation network and obtain the weights of the autoencoder network by learning the transformation matrix;

[0009] Step 3: Train the autoencoder transformation network, extract the encoder and transformation encoder of the trained autoencoder transformation network, combine them with the multilayer perceptron to form a gaze direction regression network, and regress the gaze direction on a small sample eye-tracking labeled training set.

[0010] Step 4: Deploy the line-of-sight regression network in the application environment and use the line-of-sight regression network to estimate the line of sight.

[0011] As a preferred approach for gaze tracking based on autoencoder transformation, in step one:

[0012] The face image is rotated around the x, y, and z axes of a 3D coordinate system by rotation angles α, β, and γ, respectively. The face image is then projected onto an image plane. The four vertices A, B, C, and D of the face image correspond to points A′, B′, C′, and D′ on the image plane. A 3×3 homography matrix H is calculated using the four point pairs before and after the projection of the face image. The data in the homography matrix H is then collected and stored as a homography array.

[0013] As a preferred approach for gaze tracking based on autoencoding transformation, in step two:

[0014] Automatic encoding transformation networks consist of encoders and decoders with shared weights, and the encoder uses a backbone network such as ResNet or DenseNet.

[0015] The embedded features of the output are input into the transformer, and the long-range dependencies of the embedded features are learned through the attention mechanism.

[0016] The outputs of the encoder and decoder are cascaded and input into the multilayer perceptron, which outputs the homography matrix H′.

[0017] As a preferred approach for gaze tracking based on autoencoder transformation, the loss function of the autoencoder transformation network is:

[0018]

[0019] In the formula, H is the homography matrix of the perspective projection transformation, h i,j H' is the coefficient of the i-th row and j-th column of the homography matrix H; H′ is the output homography matrix of the autocoding network; h′ i,j Let ||·|| be the coefficient of the i-th row and j-th column of matrix H′; F Let be the F-norm of the matrix.

[0020] As a preferred scheme for the gaze tracking method based on autoencoder transformation, in step three, the weights of the fixed encoder and the transform encoder are adjusted, the weights of the multilayer perceptron are transferred to the gaze direction estimation.

[0021] The present invention also provides a gaze tracking device based on automatic encoding transformation, comprising:

[0022] The face image processing module is used to rotate the face image around the x, y, and z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane. It calculates the homography matrix using the point pairs before and after the face image projection, and collects the homography matrix data as a homography array.

[0023] The encoding transformation network construction module is used to build an autoencoding transformation network and obtain the weights of the autoencoding network by learning the transformation matrix.

[0024] The encoding transformation network processing module is used to train the autoencoder transformation network, extract the encoder and transformation encoder of the trained autoencoder transformation network, form a gaze direction regression network with the multilayer perceptron, and regress the gaze direction on a small sample eye-tracking labeled training set.

[0025] The line-of-sight regression network deployment module is used to deploy the line-of-sight regression network in the application environment and perform line-of-sight estimation through the line-of-sight regression network.

[0026] As a preferred embodiment of the gaze tracking device based on automatic encoding transformation, the face image processing module includes:

[0027] The face image is rotated around the x, y, and z axes of a 3D coordinate system by rotation angles α, β, and γ, respectively. The face image is then projected onto an image plane. The four vertices A, B, C, and D of the face image correspond to points A′, B′, C′, and D′ on the image plane. A 3×3 homography matrix H is calculated using the four point pairs before and after the projection of the face image. The data in the homography matrix H is then collected and stored as a homography array.

[0028] As a preferred embodiment of the gaze tracking device based on automatic encoding transformation, the automatic encoding transformation network construction module includes an encoder and a decoder with shared weights, and the encoder adopts a backbone network ResNet or DenseNet.

[0029] The encoding transformation network construction module inputs the output embedded features into the transformer, and learns the long-range dependencies of the embedded features through an attention mechanism;

[0030] The encoding transformation network construction module cascades the outputs of the encoder and decoder into the multilayer perceptron and outputs the homography matrix H′.

[0031] As a preferred embodiment of the gaze tracking device based on autoencoder transformation, the loss function of the autoencoder transformation network in the encoding transformation network construction module is:

[0032]

[0033] In the formula, H is the homography matrix of the perspective projection transformation, h i,j H' is the coefficient of the i-th row and j-th column of the homography matrix H; H′ is the output homography matrix of the autocoding network; h′ i,j Let ||·|| be the coefficient of the i-th row and j-th column of matrix H′; F Let be the F-norm of the matrix.

[0034] As a preferred embodiment of the gaze tracking device based on automatic encoding transformation, the encoding transformation network processing module fixes the weights of the encoder and transforms the encoder, adjusts the weights of the multilayer perceptron, and transfers the self-supervised learning weights to gaze direction estimation.

[0035] The beneficial effects of this invention are as follows: A face image is rotated around the x, y, and z axes of a three-dimensional coordinate system, and the rotated face image is projected onto an image plane. A homography matrix is ​​calculated using the point pairs before and after the projection, and the homography matrix data is collected and stored as a homography array. An autoencoder transform network is constructed, and the weights of the autoencoder network are obtained by learning the transform matrix. The autoencoder transform network is trained, and the encoder and transform encoder of the trained autoencoder transform network are extracted and combined with a multilayer perceptron to form a gaze direction regression network. The gaze direction is regressed on a small sample eye-tracking labeled training set. The gaze direction regression network is deployed in the application environment, and gaze estimation is performed through the gaze direction regression network. This invention can better aggregate face features and improve the performance of the gaze estimation task. Attached Figure Description

[0036] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0037] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0038] Figure 1 This is a schematic flowchart of the gaze tracking method based on automatic encoding transformation provided in Embodiment 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of face image rotation and projection in the gaze tracking method based on automatic encoding transformation provided in Embodiment 1 of the present invention;

[0040] Figure 3 This is a schematic diagram of the autoencoder transformation network principle in the gaze tracking method based on autoencoder transformation provided in Embodiment 1 of the present invention;

[0041] Figure 4 This is a schematic diagram of gaze direction estimation in the gaze tracking method based on autoencoder transformation provided in Embodiment 1 of the present invention;

[0042] Figure 5 This is a schematic diagram of an eye-tracking device based on automatic encoding transformation provided in Embodiment 2 of the present invention. Detailed Implementation

[0043] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] In many applications, a more flexible form of feature representation is desired, which can be achieved through unsupervised training by exploring the distribution of unlabeled data. Autoencoded data learns representations by reconstructing data, while autoencoded transforms train unsupervised models by decoding the transformation from the original feature representation and the feature representation of the transformed image. The assumption of autoencoded transforms is that if a transformation can be reconstructed, then the learned representation should contain all the information about the visual structure of the image before and after the transformation; thus, the representation is transform-equivalent. Furthermore, the lack of strict restrictions on the form of the representation enhances the flexibility in choosing the representation format.

[0046] Based on the above theoretical foundation, see Figure 1 Embodiment 1 of the present invention provides a gaze tracking method based on automatic encoding transformation, comprising the following steps:

[0047] S1. Rotate the face image around the x, y, and z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane. Calculate the homography matrix using the point pairs before and after the face image projection, and collect the homography matrix data as a homography array.

[0048] S2. Construct an autoencoder transformation network and obtain the weights of the autoencoder network by learning the transformation matrix;

[0049] S3. Train the autoencoder transformation network, extract the encoder and transformation encoder of the trained autoencoder transformation network, form a gaze direction regression network with the multilayer perceptron, and regress the gaze direction on a small sample eye-tracking labeled training set.

[0050] S4. Deploy the line-of-sight regression network in the application environment and use the line-of-sight regression network to estimate the line of sight.

[0051] See Figure 2 In step S1 of this embodiment: the face image is rotated around the x, y, and z axes of the three-dimensional coordinate system by rotation angles α, β, and γ, respectively; the face image is projected onto the image plane; the corresponding points of the four vertices A, B, C, and D of the face image on the image plane are A′, B′, C′, and D′; a 3×3 homography matrix H is calculated using the four point pairs before and after the face image projection; and the data in the homography matrix H is collected and stored as a homography array.

[0052] In step S2 of this embodiment: the autoencoder transform network includes a shared-weight encoder and a decoder. The autoencoder transform network is fed into the face image X and the transformed image H(X) to the shared-weight encoder. The encoder uses a ResNet or DenseNet backbone network. The output embedded features are input to the transformer, and the long-range dependencies of the embedded features are learned through an attention mechanism. The outputs of the encoder and decoder are cascaded and input to a multilayer perceptron, and the homography matrix H′ is output.

[0053] See Figure 3 and Figure 4 In this embodiment, during the learning process, the weights of the fixed encoder and the transformation encoder are adjusted, and the weights of the multilayer perceptron are transferred to the gaze direction estimation. Because CNNs have local receptive fields, they struggle to extract features over a wide range. While Transformers have long receptive fields, the computation of attention requires significant resources.

[0054] In this embodiment, the autoencoder transform network adopts a CNN-Transformer hybrid architecture, which can efficiently stack CNN and Transformer in a hybrid paradigm, thereby combining the advantages of both to better aggregate features and improve the performance of gaze estimation tasks.

[0055] In this embodiment, the loss function of the autoencoder transform network is:

[0056]

[0057] In the formula, H is the homography matrix of the perspective projection transformation, h i,jH' is the coefficient of the i-th row and j-th column of the homography matrix H; H′ is the output homography matrix of the autocoding network; h′ i,j Let ||·|| be the coefficient of the i-th row and j-th column of matrix H′; F Let be the F-norm of the matrix.

[0058] The CNN-Transformer architecture incorporates information interaction in its encoder structure, simulating short-term and long-term dependencies in visual data through convolutional and transform blocks. The transform block fuses local and global information, further enhancing its modeling capabilities.

[0059] The CNN-Transformer architecture integrates convolutional and transform blocks using a hybrid strategy, stacking them to create a hybrid CNN-Transformer architecture. Following a hierarchical pyramid structure, the CNN-Transformer architecture consists of an initial layer, a combination block, and a transform block. Each processing stage is equipped with a series of combination blocks (several convolutional blocks (CBs) and transform blocks (TBs)) and a transform block (TRB). The spatial resolution is progressively reduced to 1 / 32 of its original size, while the channel dimension expands stage by stage. N convolutional and transform blocks are stacked sequentially in each processing stage.

[0060] The convolutional blocks follow a Multi-Head Convolutional Attention (MHCA) architecture, achieving outstanding performance of the transform blocks while maintaining the deployment advantages of the bottleneck blocks. The transform blocks are placed at the end of the combination blocks and employ an efficient multi-head self-attention mechanism (E-MHSA) to learn long-range dependencies between labels, enabling the model to learn global representations in shallow layers.

[0061] The process involves learning a global representation of a face image using a CNN-Transformer architecture. The input image has a resolution of H×W (224×224). After passing through an initial layer consisting of N0 3×3 conv2d layers, the output image resolution is H / 4×W / 4. The first processing stage includes 11 Combination Blocks (CMBs) and one Transformer Block (TRB). Each CMB consists of N1 convolutional blocks (CBs), and each TRB consists of a 1×1 conv2d layer and an average pooling layer with a stride of 2, resulting in an output image resolution of H / 8×W / 8. The second processing stage consists of 12 Combination Blocks (CMBs) and one Transformer Block (TRB). Each CMB consists of N2 convolutional blocks (CBs) and one Transformer Block (TB). Each TRB consists of a 1×1 conv2d layer and an average pooling layer with a stride of 2, resulting in an output image resolution of H / 8×W / 8. The first stage of processing involves an nv2d layer and an average pooling layer with a stride of 2, resulting in an output image resolution of H / 16×W / 16. The second stage consists of a third processing stage, which includes 13 convolutional blocks (CMBs) and a fourth processing stage, also consisting of 14 convolutional blocks (CMBs) and a fifth processing layer (TRB). The third stage, after batch regularization, outputs an image with a resolution of H / 32×W / 32.

[0062] In summary, this invention rotates a face image around the x, y, and z axes of a three-dimensional coordinate system, projects the rotated face image onto an image plane, calculates the homography matrix using point pairs before and after projection, and stores the homography matrix data as a homography array. An autoencoder transform network is constructed, and its weights are obtained by learning the transform matrix. The autoencoder transform network is trained, and the encoder and transform encoder of the trained network are extracted and combined with a multilayer perceptron to form a gaze direction regression network. The gaze direction is regressed on a small-sample eye-tracking labeled training set. The gaze direction regression network is deployed in the application environment for gaze estimation. The autoencoder transform network includes a weighted encoder and decoder, with the encoder using a ResNet or DenseNet backbone network. The output embedded features are input to the transformer, and the long-range dependencies of the embedded features are learned through an attention mechanism. The outputs of the encoder and decoder are cascaded and input to the multilayer perceptron, outputting the homography matrix H′. This invention can better aggregate face features and improve the performance of the gaze estimation task.

[0063] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0064] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] Example 2

[0066] See Figure 5 Embodiment 2 of the present invention provides a gaze tracking device based on automatic encoding transformation, comprising:

[0067] The face image processing module 1 is used to rotate the face image around the x, y, and z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane. It calculates the homography matrix using the point pairs before and after the face image projection, and collects the homography matrix data as a homography array.

[0068] Encoding Transformation Network Construction Module 2 is used to construct an autoencoding transformation network and obtain the weights of the autoencoding network by learning the transformation matrix;

[0069] The encoding transformation network processing module 3 is used to train the autoencoder transformation network, extract the encoder and transformation encoder of the trained autoencoder transformation network, form a gaze direction regression network with the multilayer perceptron, and regress the gaze direction on a small sample eye-tracking labeled training set.

[0070] The line-of-sight regression network deployment module 4 is used to deploy the line-of-sight regression network in the application environment and perform line-of-sight estimation through the line-of-sight regression network.

[0071] In this embodiment, the face image processing module 1 includes:

[0072] The face image is rotated around the x, y, and z axes of a 3D coordinate system by rotation angles α, β, and γ, respectively. The face image is then projected onto an image plane. The four vertices A, B, C, and D of the face image correspond to points A′, B′, C′, and D′ on the image plane. A 3×3 homography matrix H is calculated using the four point pairs before and after the projection of the face image. The data in the homography matrix H is then collected and stored as a homography array.

[0073] In this embodiment, the automatic encoding transformation network construction module 2 includes an encoder and a decoder with shared weights, and the encoder adopts a backbone network ResNet or DenseNet.

[0074] The encoding transformation network construction module 2 inputs the output embedded features into the transformer, and learns the long-range dependencies of the embedded features through an attention mechanism;

[0075] The encoding transformation network construction module 2 cascades the outputs of the encoder and decoder and inputs them into the multilayer perceptron, and outputs the homography matrix H′.

[0076] In this embodiment, the loss function of the autoencoder transformation network in the encoding transformation network construction module 2 is:

[0077]

[0078] In the formula, H is the homography matrix of the perspective projection transformation, h i,j H' is the coefficient of the i-th row and j-th column of the homography matrix H; H′ is the output homography matrix of the autocoding network; h′ i,j Let ||·|| be the coefficient of the i-th row and j-th column of matrix H′; F Let be the F-norm of the matrix.

[0079] In this embodiment, in the encoding transformation network processing module 3, the weights of the fixed encoder and the transformation encoder are adjusted, and the weights of the multilayer perceptron are transferred to the gaze direction estimation.

[0080] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0081] Example 3

[0082] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a gaze tracking method based on autoencoder transformation. The program code includes instructions for executing the gaze tracking method based on autoencoder transformation of Embodiment 1 or any possible implementation thereof.

[0083] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives, SSDs).

[0084] Example 4

[0085] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0086] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the gaze tracking method based on autoencoder transformation according to Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0087] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0088] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0089] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0090] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method of gaze tracking based on auto-encoding transformation, characterized in that, The method comprises the following steps: Step one, rotate the face image around the x, y, z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane, calculate the homography matrix using the point pairs before and after the face image projection, collect the homography matrix data and store it as a homography array; Step two, construct an auto-encoding transformation network, learn the transformation matrix, and obtain the weight value of the auto-encoding network; Step three, train the auto-encoding transformation network, extract the encoder and the transformation encoder of the trained auto-encoding transformation network, and combine them with a multilayer perceptron to form a gaze direction regression network, and regress the gaze direction on a small sample eye movement labeling training set; Step four, deploy the gaze direction regression network in an application environment and estimate the gaze direction through the gaze direction regression network; In step two: The auto-encoding transformation network includes a shared weight encoder and a decoder, and the encoder uses a backbone network Resnet or Densenet; The output embedded features are input to the transformer, and the long-range dependencies of the embedded features are learned through the attention mechanism; concatenating the outputs of the encoder and the decoder into a multilayer perceptron and outputting a homography matrix ; The loss function of the auto-encoding transformation network is: ， where H is a homography matrix of perspective projection transformation, is the i-th row, j-th column coefficient of the homography matrix H; is the output homography matrix of the auto-encoding network; is is the i-th row, j-th column coefficient of the matrix; is the F-norm of the matrix.

2. The automatic-coding-transformation-based gaze tracking method according to claim 1, wherein, In step one: Rotation angles of rotating the face image around the x, y, z axes of the three-dimensional coordinate system are respectively , and the corresponding points of the four vertices A, B, C, D of the face image on the image plane are ; A homography matrix H is calculated using four points before and after projection of a human face image Data in the homography matrix H is stored as a homography array .

3. The automatic-coding- transformation-based line-of-sight tracking method of claim 1, wherein, In step three, fix the weight values of the encoder and the transformation encoder, adjust the weight values of the multilayer perceptron, and migrate the weight values of the self-supervised learning to the gaze direction estimation.

4. A gaze tracking device based on auto-encoding transformation, characterized in that The method comprises the following steps: A face image processing module is configured to rotate the face image around the x, y, z axes of the three-dimensional coordinate system, and project the rotated face image onto the image plane, calculate the homography matrix using the point pairs before and after the face image projection, collect the homography matrix data and store it as a homography array; An encoding transformation network construction module is configured to construct an auto-encoding transformation network, learn the transformation matrix, and obtain the weight value of the auto-encoding network; An encoding transformation network processing module is configured to train the auto-encoding transformation network, extract the encoder and the transformation encoder of the trained auto-encoding transformation network, and combine them with a multilayer perceptron to form a gaze direction regression network, and regress the gaze direction on a small sample eye movement labeling training set; A gaze direction regression network deployment module is configured to deploy the gaze direction regression network in an application environment and estimate the gaze direction through the gaze direction regression network; In the encoding transformation network construction module, the auto-encoding transformation network includes a shared weight encoder and a decoder, and the encoder uses a backbone network Resnet or Densenet; In the encoding transformation network construction module, the output embedded features are input to the transformer, and the long-range dependencies of the embedded features are learned through the attention mechanism; The encoding transformation network building module cascades the outputs of the encoder and the decoder into a multi-layer perceptron and outputs a homography matrix ; In the encoding transformation network construction module, the loss function of the auto-encoding transformation network is: ， where H is a homography matrix of perspective projection transformation, is the i-th row, j-th column coefficient of the homography matrix H; is the output homography matrix of the auto-encoding network; is the i-th row, j-th column coefficient of the matrix H; is the i-th row, j-th column coefficient of the matrix H; is the F-norm of the matrix H.

5. The automatic-coding-transformation-based gaze tracking device according to claim 4, characterized in that, In the face image processing module: Rotation angles of rotating the face image around the x, y, z axes of the three-dimensional coordinate system are respectively , and the corresponding points of the four vertices A, B, C, D of the face image on the image plane are ; A homography matrix H is calculated using four points before and after projection of a human face image Data in the homography matrix H is stored as a homography array .

6. The automatic-coding-transformation-based gaze tracking device of claim 4, wherein, In the encoding transformation network processing module, the weight values of the encoder and the transformation encoder are fixed, the weight values of the multilayer perceptron are adjusted, and the weight values of the self-supervised learning are migrated to the gaze direction estimation.

Images