An isometric self-supervised line of sight estimation method and apparatus

Abstract

An isometric self-supervised line-of-sight estimation method and device, the method establishes a camera coordinate system and a world coordinate system, calibrates the camera, and estimates the extrinsic parameters of the world coordinate system; adjust the camera posture, rotate and scale the camera coordinate system; a virtual camera coordinate system is established, and after the head of the human body moves in three directions for two-dimensional projection, a homography matrix corresponding to the image transformation is obtained; the transformed picture is sent into the encoder for feature extraction, and after the features are extracted, they are projected to the vector space through the multilayer perceptron network, and the Barlow Twins loss is calculated through the hidden space feature mutual correlation matrix; the obtained encoder is connected with the multilayer perceptron network, and in the line-of-sight gaze data environment, the 3D line-of-sight direction is fitted by using transfer learning. The application can efficiently utilize unlabeled data for 3D line-of-sight estimation; it is invariant to appearance transformation and isometric to geometric transformation.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to an isovariant self-supervised line-of-sight estimation method and apparatus. Background Technology

[0002] Eye gaze is a fundamental nonverbal signal in human communication and a powerful tool for inferring attention and intent. Appearance-based gaze estimation, including direct gaze direction regression from facial or eye images, is the preferred method for remote camera scenarios, supporting numerous applications including human-computer interaction, cognitive and behavioral understanding, and autonomous driving.

[0003] Currently, 3D pose estimation remains a challenging problem, primarily due to the difficulty of data annotation. Ordinary 2D annotation is relatively easy to complete—simply take a photograph and record the gaze point on the computer screen. However, collecting 3D annotation information requires specialized equipment, often in laboratory environments. The limitations of these equipment in terms of space and placement lead to a lack of diverse background information and a single gaze direction. This significantly restricts the scale of 3D annotation data, making it difficult to improve model performance through common data manipulation methods such as increasing data volume and diversity.

[0004] Convolutional Neural Network (CNN)-based methods have achieved good performance driven by large-scale datasets. However, their performance remains unsatisfactory when generalized to distributions outside of these datasets. Training deep neural networks typically requires a large amount of labeled data to learn sufficient representations to perform visual understanding tasks. When only a limited amount of labeled data is available for training the network, this greatly limits the applicability of deep neural networks. Summary of the Invention

[0005] To address this, the present invention provides an isovariable self-supervised line-of-sight estimation method and apparatus, which can be trained on small batches of data and use unlabeled data for 3D line-of-sight estimation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an isovariant self-supervised line-of-sight estimation method, comprising:

[0007] Establish a camera coordinate system and a world coordinate system, calibrate the camera, and estimate the extrinsic parameters of the world coordinate system;

[0008] Adjust the camera pose, perform rotation and scaling transformations on the camera coordinate system; establish a virtual camera coordinate system, and after the human head moves in three directions in the virtual camera coordinate system and is projected in two dimensions, obtain the homography matrix corresponding to the image transformation;

[0009] N images are transformed using two preset data transformations to obtain 2N transformed images from preset viewpoints. The transformed images are then used as samples to be fed into the encoder for feature extraction. After feature extraction, the images are projected into the vector space through a multilayer perceptron network. The BarlowTwins loss is calculated using the cross-correlation matrix of the latent space features.

[0010] The obtained encoder is connected to a multilayer perceptron network, and transfer learning is used to fit the 3D gaze direction in a gaze-attention data environment.

[0011] As a preferred embodiment of the self-supervised gaze estimation method, the origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face plane model. The x-axis of the world coordinate system points from the right eye to the left eye of the face plane model. The y-axis of the world coordinate system is perpendicular to the x-axis of the world coordinate system and points downward on the plane of the assumed center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth of the face plane model. The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

[0012] As a preferred scheme for the self-supervised line-of-sight estimation method, in the virtual camera coordinate system, the human head moves along the pitch angle, yaw angle and roll angle;

[0013] The homography matrix H corresponding to the image transformation is:

[0014]

[0015]

[0016] In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

[0017] As a preferred embodiment of the self-supervised line-of-sight estimation method, the z-axis of the camera coordinate system is pointed to the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x-axis and z-axis of the camera coordinate system and points downwards.

[0018] As a preferred scheme for the self-supervised line-of-sight estimation method, the diagonal elements of the cross-correlation matrix and the identity matrix are close to 1, and the off-diagonal elements are close to 0.

[0019] The present invention also provides an isovariable self-supervised line-of-sight estimation device, comprising:

[0020] The coordinate system construction module is used to establish the camera coordinate system and the world coordinate system, calibrate the camera, and estimate the extrinsic parameters of the world coordinate system.

[0021] The camera pose adjustment module is used to adjust the camera pose by rotating and scaling the camera coordinate system.

[0022] The virtual camera coordinate system processing module is used to establish a virtual camera coordinate system. In the virtual camera coordinate system, after the human head moves in three directions and is projected in two dimensions, the homography matrix corresponding to the image transformation is obtained.

[0023] The data transformation module is used to transform N images through two preset data transformations to obtain 2N transformed images from preset perspectives.

[0024] The feature extraction module is used to feed the transformed image as a sample into the encoder for feature extraction.

[0025] The projection processing module is used to extract features and project them onto the vector space through a multilayer perceptron network, and calculate the BarlowTwins loss through the cross-correlation matrix of latent space features.

[0026] The gaze direction fitting module is used to connect the obtained encoder to the multilayer perceptron network and use transfer learning to fit the 3D gaze direction in the gaze-attention data environment.

[0027] As a preferred embodiment of the self-supervised line-of-sight estimation device, the coordinate system construction module includes:

[0028] The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face plane model. The x-axis of the world coordinate system points from the right eye to the left eye of the face plane model. The y-axis of the world coordinate system is perpendicular to the x-axis of the world coordinate system and points downward on the plane of the assumed center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth of the face plane model. The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

[0029] As a preferred embodiment of the self-supervised line-of-sight estimation device, in the virtual camera coordinate system processing module, the human head moves along the pitch angle, yaw angle and roll angle in the virtual camera coordinate system.

[0030] The homography matrix H corresponding to the image transformation is:

[0031]

[0032]

[0033] In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

[0034] As a preferred embodiment of the self-supervised line-of-sight estimation device, the coordinate system construction module includes:

[0035] The z-axis of the camera coordinate system is pointed to the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x-axis and z-axis of the camera coordinate system and points downwards.

[0036] As a preferred embodiment of the self-supervised line-of-sight estimation device, in the projection processing module, the diagonal elements of the cross-correlation matrix and the identity matrix are close to 1, and the off-diagonal elements are close to 0.

[0037] The beneficial effects of this invention are as follows: A camera coordinate system and a world coordinate system are established; the camera is calibrated; and the extrinsic parameters of the world coordinate system are estimated. The camera pose is adjusted, and the camera coordinate system undergoes rotation and scaling transformations. A virtual camera coordinate system is established; in this system, the human head moves along three directions and is projected in two dimensions to obtain the homography matrix corresponding to the image transformation. N images are transformed using two preset data transformations to obtain 2N transformed images from preset viewpoints. These transformed images are then used as samples in an encoder for feature extraction. After feature extraction, the images are projected onto a vector space via a multilayer perceptron network, and the Barlow Twins loss is calculated using the latent space feature cross-correlation matrix. The obtained encoder is connected to the multilayer perceptron network, and transfer learning is used to fit the 3D gaze direction under gaze-based data conditions. This invention can efficiently utilize unlabeled data for 3D gaze estimation; it is invariant to appearance transformations and isotropic to geometric transformations; it does not heavily rely on samples and does not require training with negative samples. Therefore, it is not limited by batch size and can be trained on small batches of data. Starting from the embedded features themselves, it does not require asymmetric structural design. Compared with other methods that use large batches of sample data, this method can use high-dimensional embedded features. Attached Figure Description

[0038] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0039] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0040] Figure 1 This is a schematic diagram of the flow chart of the self-supervised line-of-sight estimation method provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram illustrating the principle of homography matrix acquisition in the equal-variable self-supervised line-of-sight estimation method provided in Embodiment 1 of the present invention;

[0042] Figure 3 This is a schematic diagram of the equivalent self-supervised representation learning based on combined transformation in the equivalent self-supervised gaze estimation method provided in Embodiment 1 of the present invention;

[0043] Figure 4 This is a schematic diagram of gaze estimation based on transfer learning in the equal-variable self-supervised gaze estimation method provided in Embodiment 1 of the present invention;

[0044] Figure 5 This is a schematic diagram of the apparatus for the self-supervised line-of-sight estimation method provided in Embodiment 2 of the present invention. Detailed Implementation

[0045] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Self-supervised models based on contrastive learning outperform the state-of-the-art supervised baselines on various downstream tasks. Gaze estimation, as a downstream task sensitive to spatial information, is particularly vulnerable because any data augmentation that alters spatial information will change the gaze direction. Contrastive learning relies on labels—whether two objects have the same or different labels. If they are the same, the models bring the two features closer together in the feature space; otherwise, they separate them. Self-supervised contrastive learning tasks mostly involve data augmentation of unlabeled data, encouraging the model to learn feature representations that do not change with data augmentation—a property known as invariance.

[0047] The aforementioned invariance is not very applicable to gaze estimation tasks. When a face image undergoes transformations that do not involve geometry, the model's representation is expected to be invariant; however, once any geometric transformation occurs, the keypoint positions change, and the model's extracted representation should also change accordingly, requiring the model's representation to be equivariant. For a function / feature and a transformation, if:

[0048] H j (H i (X))=H i (H j (X))

[0049] Then it is said to have equivariance to transformation.

[0050] In view of this, to address the unique nature of the pose estimation task and the challenges of data labeling, this invention proposes an equivariant self-supervised gaze estimation method. Specifically, on an unlabeled eye-tracking dataset, an equivariant non-contrast learning objective function is designed to learn equivariant features in geometric transformations without utilizing any labeled information. Then, gaze direction estimation is learned through transfer learning using a small amount of labeled data. The specific details of this invention are as follows.

[0051] Example 1

[0052] See Figure 1 Embodiment 1 of the present invention provides an isovariant self-supervised line-of-sight estimation method, comprising the following steps:

[0053] S1. Establish the camera coordinate system and the world coordinate system, calibrate the camera, and estimate the external parameters of the world coordinate system;

[0054] S2. Adjust the camera posture and perform rotation and scaling transformations on the camera coordinate system;

[0055] S3. Establish a virtual camera coordinate system. In the virtual camera coordinate system, after the human head moves in three directions and is projected in two dimensions, the corresponding homography matrix of the image transformation is obtained.

[0056] S4. Transform N images using two preset data methods to obtain 2N transformed images from preset viewpoints. Use the transformed images as samples to input the encoder for feature extraction.

[0057] S5. After extracting features, the features are projected onto the vector space through a multilayer perceptron network, and the Barlow Twins loss is calculated using the cross-correlation matrix of the latent space features.

[0058] S6. Connect the obtained encoder to a multilayer perceptron network, and use transfer learning to fit the 3D gaze direction in a gaze-attention data environment.

[0059] The key to improving the performance of self-supervised contrastive learning lies in two points: the large size of positive and negative sample pairs and sample consistency. This embodiment uses a feature cross-correlation function to measure sample consistency, requiring only positive samples and being insensitive to the batch size. Furthermore, a data preprocessing method is used before training to obtain a normalized perspective transformation view of the face image.

[0060] See Figure 2 During data preprocessing, a camera coordinate system and a world coordinate system are established. The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face screen model. The x-axis is from the right eye to the left eye of the face screen model. The y-axis of the world coordinate system is perpendicular to the x-axis and pointing downwards on the four-point plane (center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth). The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

[0061] This includes calibrating the camera and estimating the world coordinate system. External parameters. Establish a normalized virtual camera coordinate system, adjust the camera pose, and set the z-axis of the camera coordinate system. c The axis points to the origin o of the world coordinate system. w The x-axis of the camera coordinate system c The x-axis is parallel to the world coordinate system. w y in the camera coordinate system c The x-axis is perpendicular to the camera coordinate system. c z c The resulting plane is facing downwards, and a rotational transformation is performed. Move the camera along z c Axis translation, i.e., scaling transformation S.

[0062] In this embodiment, in the virtual camera coordinate system, the human head moves along the pitch angle, yaw angle, and roll angle; after 2D projection, the corresponding homography matrix H of the image transformation is:

[0063]

[0064]

[0065] In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

[0066] In this embodiment, the diagonal elements of the cross-correlation matrix and the identity matrix are close to 1, and the off-diagonal elements are close to 0.

[0067] See Figure 3 Because self-supervised learning employs an end-to-end solution, it transforms N images through two different data transformations H... i and H j We obtain 2N images from different perspectives. These samples are fed into an encoder to extract features, and then projected into a vector space through a multilayer perceptron network. The Barlow Twins loss is calculated using the cross-correlation matrix of the latent space features. The loss function is used to reduce redundancy, making the cross-correlation matrix as close to the identity matrix as possible, i.e., the diagonal elements are close to 1 and the off-diagonal elements are close to 0. The following relationship exists:

[0068]

[0069] Here, λ is a positive constant used to weigh the importance of the first and second terms of the loss function. The first term is the invariant term, and the second term is the redundancy reduction term. The invariant term is used to reduce C... ii The diagonal elements are equal to 1, so that the equivariant embedded features of different views of the same sample remain unchanged. The redundancy reduction term equals the off-diagonal elements to 0 to reduce redundancy, and is related to different embedded vector solutions.

[0070] Wherein, the cross-correlation matrix C ii The calculation is as follows:

[0071]

[0072] In the formula, The input face image is processed by H i The feature z is obtained by transformation and encoder transformation. i After H j Latent space features output by transformation and feature mapping; The input face image is processed by H j The feature z is obtained by transformation and encoder transformation. i After H iLatent space features output by transformation and feature mapping.

[0073] See Figure 4 3D gaze estimation requires the model to be equivariant to geometric transformations, as these transformations alter the positions of keypoints. A combined transformation of these two features is necessary to maximize the consistency of the resulting features. Since 3D geometric transformations are essentially rigid body motions, the 3D to 2D transformation can be modeled using perspective transformation, and image-to-image transformations are implemented using a 3×3 homography matrix. Finally, the encoder is connected to a new multilayer perceptron network to establish a new network architecture. In this new architecture, the encoder weights are kept constant while the weights of the multilayer perceptron network are changed to obtain the 3D gaze direction.

[0074] In summary, this invention establishes a camera coordinate system and a world coordinate system, calibrates the camera, and estimates the extrinsic parameters of the world coordinate system. The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face model. The x-axis of the world coordinate system points from the right eye to the left eye of the face model. The y-axis of the world coordinate system lies on a plane containing the centers of the left and right eyes, the left corner of the mouth, and the right corner of the mouth of the assumed face model, perpendicular to the x-axis of the world coordinate system and pointing downwards. The z-axis of the world coordinate system is perpendicular to the screen plane formed by the x and y axes of the world coordinate system according to the right-hand screw principle. The z-axis of the camera coordinate system is aligned with the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x and z axes of the camera coordinate system and pointing downwards. The camera pose is adjusted, and the camera coordinate system is rotated and scaled. A virtual camera coordinate system is established. In this system, the human head is projected in three directions to obtain a two-dimensional image transformation homography matrix. N images are transformed using two preset data transformations to obtain 2N transformed images from preset viewpoints. These transformed images are then fed into an encoder for feature extraction. After feature extraction, the images are projected onto a vector space via a multilayer perceptron network. The Barlow Twins loss is calculated using the cross-correlation matrix of the latent space features. The cross-correlation matrix is ​​close to 1 for diagonal elements and close to 0 for off-diagonal elements. The encoder is then connected to the multilayer perceptron network, and transfer learning is used to fit the 3D gaze direction under gaze-based data. This invention can efficiently utilize unlabeled data for 3D gaze estimation; it is invariant to appearance transformations and isotropic to geometric transformations; it does not heavily rely on samples and does not require training with negative samples. Therefore, it is not limited by batch size and can be trained on small batches of data. Starting from the embedded features themselves, it does not require asymmetric structural design. Compared to other methods that use large batches of sample data, this method can use high-dimensional embedded features.

[0075] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0076] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] Example 2

[0078] See Figure 5 Embodiment 2 of the present invention provides an isovariant self-supervised line-of-sight estimation device, comprising:

[0079] Coordinate system construction module 1 is used to establish the camera coordinate system and the world coordinate system, calibrate the camera, and estimate the extrinsic parameters of the world coordinate system;

[0080] Camera pose adjustment module 2 is used to adjust the camera pose by rotating and scaling the camera coordinate system.

[0081] Virtual camera coordinate system processing module 3 is used to establish a virtual camera coordinate system. In the virtual camera coordinate system, after the human head moves in three directions and is projected in two dimensions, the homography matrix corresponding to the image transformation is obtained.

[0082] Data transformation module 4 is used to transform N images through two preset data transformations to obtain 2N transformed images from preset perspectives;

[0083] Feature extraction module 5 is used to feed the transformed image as a sample into the encoder for feature extraction;

[0084] Projection processing module 6 is used to project the extracted features into the vector space through a multilayer perceptron network and calculate the BarlowTwins loss through the cross-correlation matrix of the latent space features.

[0085] The gaze direction fitting module 7 is used to connect the obtained encoder to the multilayer perceptron network and use transfer learning to fit the 3D gaze direction in the gaze-attention data environment.

[0086] In this embodiment, in the coordinate system construction module 1:

[0087] The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face plane model. The x-axis of the world coordinate system points from the right eye to the left eye of the face plane model. The y-axis of the world coordinate system is perpendicular to the x-axis of the world coordinate system and points downward on the plane of the assumed center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth of the face plane model. The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

[0088] In this embodiment, in the coordinate system construction module 1:

[0089] The z-axis of the camera coordinate system is pointed to the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x-axis and z-axis of the camera coordinate system and points downwards.

[0090] In this embodiment, in the virtual camera coordinate system processing module 3, the human head moves along the pitch angle, yaw angle and roll angle in the virtual camera coordinate system;

[0091] The homography matrix H corresponding to the image transformation is:

[0092]

[0093]

[0094] In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

[0095] In this embodiment, in the projection processing module 6, the diagonal elements of the cross-correlation matrix and the identity matrix are close to 1, and the off-diagonal elements are close to 0.

[0096] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0097] Example 3

[0098] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for an equal-variable self-supervised line-of-sight estimation method. The program code includes instructions for executing the equal-variable self-supervised line-of-sight estimation method of Embodiment 1 or any possible implementation thereof.

[0099] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives, SSDs).

[0100] Example 4

[0101] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0102] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the isovariant self-supervised line-of-sight estimation method of Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0103] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0104] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0105] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0106] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A self-supervised line-of-sight estimation method with equal variation, characterized in that, include: Establish a camera coordinate system and a world coordinate system, calibrate the camera, and estimate the extrinsic parameters of the world coordinate system; Adjust the camera pose, perform rotation and scaling transformations on the camera coordinate system; establish a virtual camera coordinate system, and after the human head moves in three directions in the virtual camera coordinate system and is projected in two dimensions, obtain the homography matrix corresponding to the image transformation; N images are transformed using two preset data transformations to obtain 2N transformed images from preset viewpoints. The transformed images are then used as samples to be fed into the encoder for feature extraction. After feature extraction, the images are projected into the vector space through a multilayer perceptron network. The BarlowTwins loss is calculated using the cross-correlation matrix of the latent space features. The obtained encoder is connected to a multilayer perceptron network, and transfer learning is used to fit the 3D gaze direction in a gaze-attention data environment.

2. The self-supervised line-of-sight estimation method according to claim 1, characterized in that, The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face plane model. The x-axis of the world coordinate system points from the right eye to the left eye of the face plane model. The y-axis of the world coordinate system is perpendicular to the x-axis of the world coordinate system and points downward on the plane of the assumed center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth of the face plane model. The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

3. The self-supervised line-of-sight estimation method according to claim 2, characterized in that, In the virtual camera coordinate system, the human head moves along the pitch angle, yaw angle, and roll angle; The homography matrix H corresponding to the image transformation is: In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

4. The self-supervised line-of-sight estimation method according to claim 3, characterized in that, The z-axis of the camera coordinate system is pointed to the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x-axis and z-axis of the camera coordinate system and points downwards.

5. The self-supervised line-of-sight estimation method according to claim 4, characterized in that, The cross-correlation matrix and the identity matrix have diagonal elements close to 1 and off-diagonal elements close to 0.

6. A self-supervised line-of-sight estimation device with equal variation, characterized in that, include: The coordinate system construction module is used to establish the camera coordinate system and the world coordinate system, calibrate the camera, and estimate the extrinsic parameters of the world coordinate system. The camera pose adjustment module is used to adjust the camera pose by rotating and scaling the camera coordinate system. The virtual camera coordinate system processing module is used to establish a virtual camera coordinate system. In the virtual camera coordinate system, after the human head moves in three directions and is projected in two dimensions, the homography matrix corresponding to the image transformation is obtained. The data transformation module is used to transform N images through two preset data transformations to obtain 2N transformed images from preset perspectives. The feature extraction module is used to feed the transformed image as a sample into the encoder for feature extraction. The projection processing module is used to extract features and project them onto the vector space through a multilayer perceptron network, and calculate the BarlowTwins loss through the cross-correlation matrix of latent space features. The gaze direction fitting module is used to connect the obtained encoder to the multilayer perceptron network and use transfer learning to fit the 3D gaze direction in the gaze-attention data environment.

7. The isovariable self-supervised line-of-sight estimation device according to claim 6, characterized in that, In the coordinate system construction module: The origin of the world coordinate system is at the midpoint of the line connecting the centers of the two eyes of the face plane model. The x-axis of the world coordinate system points from the right eye to the left eye of the face plane model. The y-axis of the world coordinate system is perpendicular to the x-axis of the world coordinate system and points downward on the plane of the assumed center of the left eye, center of the right eye, left corner of the mouth, and right corner of the mouth of the face plane model. The z-axis of the world coordinate system is perpendicular to the screen surface formed by the x and y axes of the world coordinate system according to the right-hand screw principle.

8. The self-supervised line-of-sight estimation device according to claim 7, characterized in that, In the virtual camera coordinate system processing module, the human head moves along the pitch angle, yaw angle and roll angle in the virtual camera coordinate system; The homography matrix H corresponding to the image transformation is: In the formula, s is the scale factor, p is the homogeneous coordinate of the two-dimensional image, and P w The homogeneous coordinates of the three dimensions of the image; t represents the extrinsic parameters of the real camera; K represents the intrinsic parameters of the real camera. n Internal parameters for the virtual camera; Let S be the rotation of the virtual camera relative to the camera coordinate system, S be the scaling transformation of the virtual camera relative to the camera coordinate system, and R be the matrix of the virtual camera's rotation transformation along three directions.

9. The isovariant self-supervised line-of-sight estimation device according to claim 8, characterized in that, In the coordinate system construction module: The z-axis of the camera coordinate system is pointed to the origin of the world coordinate system, the x-axis of the camera coordinate system is parallel to the x-axis of the world coordinate system, and the y-axis of the camera coordinate system is perpendicular to the plane formed by the x-axis and z-axis of the camera coordinate system and points downwards.

10. The isovariant self-supervised line-of-sight estimation device according to claim 9, characterized in that, In the projection processing module, the diagonal elements of the cross-correlation matrix and the identity matrix are close to 1, and the off-diagonal elements are close to 0.

Images