A difference-guided gaze estimation method and device based on 6D rotation matrix representation

Abstract

The present invention discloses a difference-guided gaze estimation method and device based on 6D rotation matrix representation. The method includes data preprocessing: extracting facial regions and key points from original facial images, performing perspective transformation and scaling through a virtual camera, and generating a fixed-size corrected image; constructing input image pairs: in the training phase, using samples in the training set as the main input image, and randomly sampling another sample as a reference frame; in the testing phase, using samples in the testing set as the main input image, and randomly sampling from the training set or a predefined reference image set as a reference frame; processing image pairs through a twin encoder with shared weights to generate a high-dimensional feature representation. X 1 and X 2. Calculate the correlation between features through the cross attention module to generate difference features X C Through the multi-task learning strategy and the compact representation of the 6D rotation matrix, the learning efficiency and estimation accuracy of the network are effectively improved, and the accurate quantification of the line of sight difference is achieved.

Description

Technical Field

[0001] The present application relates to the field of video image processing technology, and more specifically to a method and device for difference-guided sight estimation based on 6D rotation matrix representation. Background Art

[0002] Gaze estimation is a non-invasive technology based on mechanics, electronics, optics, and computer vision. It analyzes facial images, eye features, or head posture to determine a user's current gaze direction or point of gaze. This technology provides crucial insights for user behavior analysis, attention assessment, and interactive experience optimization, and is widely used in fields such as human-computer interaction and virtual reality. Traditional gaze estimation methods typically rely on dedicated hardware, such as infrared cameras or head-mounted displays (HMDs), to achieve high-precision gaze tracking by capturing eye movements or pupil reflexes. While these devices demonstrate high accuracy and reliability in laboratory environments or specific application scenarios, they are hardware-dependent, expensive, and have limited applicability. Furthermore, infrared light sources can potentially cause eye discomfort and visual fatigue with prolonged use. Interference from infrared signals in bright sunlight or outdoor environments can also reduce system robustness, limiting their practicality for low-cost, portable, and continuous monitoring in everyday life.

[0003] With the rapid development of deep learning and image processing technologies, appearance-based gaze estimation methods have gradually attracted attention. Relying on a single RGB camera, these methods analyze facial features, eye region characteristics, or head pose information, combined with neural network models to predict gaze direction. They offer the advantages of low equipment cost and ease of integration into everyday devices such as smartphones and laptops. Early appearance-based gaze estimation methods primarily used monocular images or a combination of eye and facial regions as input. These methods typically required initial eye localization preprocessing to obtain eye images. However, this time-consuming preprocessing and significant degradation in robustness under large head pose variations and complex lighting conditions make them difficult to meet practical application requirements. In recent years, methods that directly regress gaze direction from full-face images have gradually emerged and have become mainstream due to their efficiency and adaptability. However, full-face images contain a large amount of redundant information unrelated to gaze, which limits the accuracy of gaze estimation methods based on full-face images. Summary of the Invention

[0004] In this embodiment, a difference-guided gaze estimation method, device, electronic device and storage medium based on 6D rotation matrix representation are provided to solve the problems of poor accuracy and poor robustness of current full-face gaze estimation methods in related technologies.

[0005] In a first aspect, an embodiment of the present invention provides a method for estimating sight lines based on a difference in 6D rotation matrix representation. The method comprises:

[0006] Data preprocessing: Extract the facial area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image I ;

[0007] Input image pair construction: During the training phase, samples in the training set are used as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frame I 2;

[0008] Processing image pairs through a siamese encoder with shared weights ( , ), generate high-dimensional feature representation X 1 and X 2;

[0009] Calculate the correlation between features through the cross attention module to generate difference features X C , and enhance the original feature representation;

[0010] Use the fully connected regression module to transform the features X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d ;

[0011] The 6D rotation matrix is ​​converted to a standard 9D rotation matrix through the orthogonal transformation module R 1. R 2 and R d ;

[0012] The network parameters are optimized based on a multi-task loss function constructed from a rotation matrix geodesic distance loss function.

[0013] In an optional embodiment, the data preprocessing step includes:

[0014] Taking the center of the eye or the center of the face as the reference point, rotate the virtual camera to point to the reference point;

[0015] Rotate the virtual camera so that its perspective faces forward;

[0016] Image scaling ensures that the distance between the subject and the camera is consistent, generating a calibrated image of a fixed size.

[0017] In an optional embodiment, the calculation method of the cross attention module includes:

[0018] Project one input feature of the cross-attention module into the query vector Q, and the other input feature into the query vector K and the value vector V;

[0019] The feature correlation is calculated through the attention mechanism to generate enhanced feature representation.

[0020] In an optional embodiment, the difference feature X C The generation methods include:

[0021] In high-dimensional feature representation X 1 and X 2 Perform cross attention calculations between them to generate difference features X C ;

[0022] The difference characteristics X C Respectively with high-dimensional feature representation X 1 and X 2 Perform feature enhancement;

[0023] The original semantic information is preserved through residual connections.

[0024] In an optional embodiment, the fully connected regression module includes three independent fully connected layers, which respectively represent the high-dimensional features X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d .

[0025] In an optional embodiment, the method in which the orthogonal transformation module converts the 6D rotation matrix into a 9D rotation matrix includes:

[0026] Split the 6D rotation matrix into two 3D vectors and ;

[0027] Generate orthogonal vectors by normalization and cross product ;

[0028] Combined 3D vectors 、 and orthogonal vectors Get the 9D rotation matrix.

[0029] In an optional embodiment, the rotation matrix geodesic distance loss function is calculated as follows:

[0030] ;

[0031] in, represents the arc cosine function, R represents the predicted rotation matrix, Represents the true rotation matrix.

[0032] In an optional embodiment, the multi-task loss function is:

[0033] ;

[0034] in 、 and The geodesic distance loss corresponding to the line of sight and the difference between the two frames,

[0035] in, They represent the true rotation matrix of the sight line corresponding to frame 1, the true rotation matrix of the sight line corresponding to frame 2, and the difference in the true rotation matrix between the two frames. , , They represent the predicted rotation matrix of the sight line corresponding to frame 1, the predicted rotation matrix of the sight line corresponding to frame 2, and the difference in the predicted rotation matrix between the two frames. 、 、 is the weight.

[0036] Compared with the prior art, the difference-guided gaze estimation method based on 6D rotation matrix representation of the present invention has the following beneficial effects:

[0037] In the present invention, the twin encoder processes the input image pair and extracts high-dimensional feature representations. Subsequently, through the feature interaction module and the difference-guided cross-attention module, the correlation information between the two frames of images is effectively captured to strengthen the attention to the line of sight related features. Specifically, the module generates difference features and enhances the original feature representation through multi-level cross-attention calculations. In the regression stage, three parallel fully connected layers are used to map the features into 6D rotation matrix representations, corresponding to the two frames of line of sight and their differences. The 6D representation is converted into a standard 9D rotation matrix through orthogonal transformation. In order to optimize the network performance, a multi-task loss function based on the geodesic distance loss of the rotation matrix is ​​also designed. This method effectively improves the learning efficiency and estimation accuracy of the network through a multi-task learning strategy and a compact representation of the 6D rotation matrix, and achieves accurate quantification of line of sight differences.

[0038] In a second aspect, an embodiment of the present invention provides a device for estimating a difference-guided sight line based on 6D rotation matrix representation, comprising:

[0039] The data preprocessing module is used to extract the facial area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image. I ;

[0040] Input image pair construction module, used in the training phase, to use samples in the training set as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frame I 2;

[0041] The twin encoder processing module processes image pairs through a twin encoder with shared weights ( , ), generate high-dimensional feature representation X 1 and X 2;

[0042] Cross-attention calculation module, used to calculate the correlation between features through the cross-attention module to generate difference features X C , and enhance the original feature representation;

[0043] Fully connected regression processing module, used to use the fully connected regression module to transform features X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 andA d ;

[0044] Orthogonal transformation processing module, used to convert the 6D rotation matrix into a standard 9D rotation matrix through the orthogonal transformation module R 1. R 2 and R d ;

[0045] An optimization module is used to optimize network parameters based on a multi-task loss function constructed by a rotation matrix geodesic distance loss function.

[0046] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0047] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the difference-guided line of sight estimation method based on 6D rotation matrix representation as described in the first aspect.

[0048] Compared with the prior art, the beneficial effects of the difference-guided sight estimation device, electronic device and storage medium based on 6D rotation matrix representation of the present invention are the same as the difference-guided sight estimation method based on 6D rotation matrix representation described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 Flowchart of a difference-guided gaze estimation method based on 6D rotation matrix representation according to an embodiment of the present invention;

[0051] Figure 2 This is a model framework diagram for difference-guided gaze estimation based on 6D rotation matrix representation in an embodiment of the present invention;

[0052] Figure 3 This is a structural block diagram of a device for difference-guided sight line estimation based on 6D rotation matrix representation in an embodiment of the present invention;

[0053] Figure 4 2 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of this application, this application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0055] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] In an embodiment of the present invention, a difference-guided sight estimation method based on 6D rotation matrix representation is provided. Figure 1 Flowchart of the difference-guided sight estimation method based on 6D rotation matrix representation of the present invention. Figure 1 and Figure 2 As shown, the process includes the following steps:

[0057] S100, Data Preprocessing: Extract the facial area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image I ;

[0058] In this embodiment, the data preprocessing step includes:

[0059] Taking the center of the eye or the center of the face as the reference point, rotate the virtual camera to point to the reference point;

[0060] Rotate the virtual camera so that its perspective faces forward;

[0061] Image scaling ensures that the distance between the subject and the camera is consistent, generating a calibrated image of a fixed size.

[0062] Specifically, in this embodiment, the YOLO5Face face detection tool is first used to extract the facial area and key points from the original facial image, and perform data correction for line of sight estimation. Specifically, the captured eye image is regarded as a plane in 3D space, and the original camera is replaced by a virtual camera for perspective transformation and scaling to achieve rotation and translation of the virtual camera. The processing flow is as follows: First, with the center of the eye or the center of the face as the reference point, the virtual camera is rotated to point to the reference point, thereby eliminating the deviation caused by the camera position; secondly, the virtual camera is further rotated so that the rotated camera angle of view is facing forward; finally, the image is scaled to ensure that the distance between the subject and the camera remains consistent. This ensures that the face will be located in the center of the image and maintain a relatively uniform size. After the above processing, the image is uniformly adjusted to a fixed size. (H=224, W=224), the final shape is where H and W represent the height and width of the image respectively.

[0063] S200, input image pair construction: In the training phase, the samples in the training set are used as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, samples in the test set are used as the main input image I1, and random samples from the training set or predefined reference image set are used as reference frames I2;

[0064] In this embodiment, the network is based on two sizes The image is taken as input, respectively and In the training and testing phases, the input image pairs ( , ) is constructed as follows: During the training phase, As the main input image, traverse the training set in sequence Samples in From the same training set In the test / inference phase, From the test set , that is, the test image to be processed; From the training set or a predefined set of reference images Randomly sampled from as reference frames.

[0065] S300, processing image pairs through a twin encoder with shared weights ( , ), generate high-dimensional feature representation X 1 and X 2;

[0066] The twin encoder receives an input image pair ( , ), processed by a two-branch network with shared weights. Considering the excellent performance of convolutional neural networks in line of sight estimation tasks, this model uses ConvNextv2-tiny as the encoder architecture. Image pair ( , ) passes through the twin encoder and is mapped to a high-dimensional feature space with semantic similarity, denoted as and .

[0067] S400, calculate the correlation between features through the cross attention module to generate difference features X C , and enhance the original feature representation;

[0068] S500, use the fully connected regression module to transform the features X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d ;

[0069] Differentiating features X C The generation methods include:

[0070] In high-dimensional feature representation X 1 and X 2 Perform cross attention calculations between them to generate difference features X C ;

[0071] The difference characteristics X C Respectively with high-dimensional feature representation X 1 and X 2 Perform feature enhancement;

[0072] The original semantic information is preserved through residual connections.

[0073] It should be noted that the calculation method of the cross attention module includes:

[0074] Project one input feature of the cross-attention module into the query vector Q, and the other input feature into the query vector K and the value vector V;

[0075] The feature correlation is calculated through the attention mechanism to generate enhanced feature representation.

[0076] For example, the module has two features and As input, it is projected through a fully connected layer, where is projected into a query vector (Query, Q), They are projected into key vectors (Key, K) and value vectors (Value, V) respectively. Subsequently, the correlation between features is calculated through the attention mechanism, and further normalization operations and residual connections are used. The specific calculation method can be expressed as:

[0077] ;

[0078] ;

[0079]

[0080] Among them, Q, K, V represent the query vector, key vector and value vector in the multi-head attention mechanism respectively, softmax is the activation function, norm represents normalization, and MLP stands for multi-layer perceptron. and is an intermediate feature, Output of the cross attention module result.

[0081] Then, the characteristics and As input, the intermediate features are obtained by feature interaction through the fully connected layer This module contains multi-level cross attention calculations. The specific process is as follows: First, perform cross attention calculations between X1 and X2 to generate difference features , this feature effectively captures the spatiotemporal interaction information between two frames. As guidance signals, respectively and Perform feature enhancement. On this basis, cross-attention calculations are further performed to deeply explore the correlation between the features of each frame and the difference features. To maintain the integrity of the features, residual connections are introduced in each calculation step to ensure that the original semantic information is retained while enhancing the feature representation. This process can be formally expressed as:

[0082]

[0083]

[0084]

[0085]

[0086] in, represents the initial difference feature, Represents frame 1 features, frame 2 features, difference features, represents the crisscross attention module.

[0087] The fully connected regression module outputs features Specifically, three independent fully connected layers are used to process the input features respectively. Each fully connected layer maps the N×C dimensional features to the N×6 dimensional output space, and obtains three 6D rotation matrix representations. , corresponding to the line of sight , sight and sight differences Where N is the batch size and C is the input feature dimension.

[0088] Specifically, the fully connected regression module consists of three independent fully connected layers, which represent high-dimensional features. X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d .

[0089] The method of converting the 6D rotation matrix into a 9D rotation matrix by the orthogonal transformation module includes:

[0090] Split the 6D rotation matrix into two 3D vectors and ;

[0091] Generate orthogonal vectors by normalization and cross product ;

[0092] Combined 3D vectors 、 and orthogonal vectors Get the 9D rotation matrix.

[0093] S600, convert the 6D rotation matrix into a standard 9D rotation matrix through the orthogonal transformation module R 1. R 2 and R d ;

[0094] The 6D rotation matrix output by the fully connected regression module is represented by the orthogonal transformation module without parameter learning. Convert to a standard 9D rotation matrix Specifically, each 6D rotation matrix can be split into two three-dimensional vectors , and then obtain the 9D rotation matrix through normalization and cross product. The calculation method is as follows:

[0095]

[0096]

[0097]

[0098]

[0099] Among them, R represents the predicted rotation matrix, Represents the three component column vectors of the rotation matrix, the initial Predicted by the network, represents the cross product operation, Represents vector normalization.

[0100] Convert the real 3D gaze vector label to a rotation matrix label. We follow the general operation of converting rotation vectors to rotation matrices. First, define the standard coordinate system, with the direction from the left eye to the right eye in the face image as the positive x-axis, the vertical downward direction as the positive y-axis, and the direction of the camera pointing to the face as the positive z-axis. The standard gaze direction (i.e., looking straight ahead) is defined as the vector All gaze vectors are first normalized and then rotated into the normalized coordinate system using the zyx Euler angle sequence to calculate the rotation matrix.

[0101] S700 , optimizing network parameters based on a multi-task loss function, where the multi-task loss function is constructed by a rotation matrix geodesic distance loss function.

[0102] Prediction value obtained based on orthogonal transformation module and the true value , a multi-task loss function is used for optimization.,First, the rotation matrix geodesic distance loss is introduced to,effectively quantify the distance between rotations.

[0103] The calculation method of the rotation matrix geodesic distance loss function is:

[0104] ;

[0105] in, represents the arc cosine function, R represents the true rotation matrix, Represents the predicted rotation matrix.

[0106] Based on the rotation matrix geodesic distance loss, the multi-task loss function consists of three parts, which supervise the line of sight between two frames and the difference between the line of sight frames. The specific calculation method is as follows:

[0107]

[0108] in, They represent the true rotation matrix of the sight line corresponding to frame 1, the true rotation matrix of the sight line corresponding to frame 2, and the difference in the true rotation matrix between the two frames. , , They represent the predicted rotation matrix of the sight line corresponding to frame 1, the predicted rotation matrix of the sight line corresponding to frame 2, and the difference in the predicted rotation matrix between the two frames.

[0109] The loss function is calculated by weighted summing of three contrast losses. The multi-task loss function is:

[0110] ;

[0111] in 、 and The geodesic distance loss corresponding to the line of sight and the difference between the two frames, 、 、 is the weight.

[0112] It can be seen from the technical solution provided by the present invention that, specifically: first, the twin encoder processes the input image pair and extracts high-dimensional feature representations. Subsequently, through the feature interaction module and the difference-guided cross-attention module, the correlation information between the two frames of images is effectively captured to strengthen the attention to the line of sight related features. Specifically, the module generates difference features and enhances the original feature representation through multi-level cross-attention calculations. In the regression stage, three parallel fully connected layers are used to map the features into 6D rotation matrix representations, corresponding to the two frames of line of sight and their differences. The 6D representation is converted into a standard 9D rotation matrix through orthogonal transformation. In order to optimize network performance, a multi-task loss function based on the geodesic distance loss of the rotation matrix is ​​also designed. This method effectively improves the learning efficiency and estimation accuracy of the network through a multi-task learning strategy and a compact representation of the 6D rotation matrix, and achieves accurate quantification of line of sight differences.

[0113] Below we use the public dataset Gaze360 dataset for verification. The Gaze360 dataset is a dataset for studying physically unconstrained line of sight estimation in the wild. The dataset contains 172,000 images of 238 subjects, which are collected by 360-degree cameras in various indoor and outdoor environments. The dataset is known for its wide range of line of sight directions. We divide the dataset into training sets, validation sets, and test sets in advance. After we build the network, we train a total of 80 epochs, and take the model weights obtained from the best results of the validation set for testing on the test set. Finally, the Angular Error of line of sight estimation on the test set of the Gaze360 dataset is counted, that is, the angle between the predicted line of sight vector and the actual line of sight vector. The statistical results are shown in Table 1:

[0114] Table 1

[0115]

[0116] An embodiment of the present invention further provides a difference-guided sight estimation device based on 6D rotation matrix representation, which is used to implement the above-mentioned method embodiment, and the details that have been described will not be repeated. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, implementation by hardware or a combination of software and hardware is also possible and conceived.

[0117] like Figure 3 As shown, Figure 3 : is a structural block diagram of a difference-guided sight line estimation device based on 6D rotation matrix representation in the present invention, which includes:

[0118] The data preprocessing module 101 is used to extract the face area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image. I ;

[0119] Input image pair construction module 102 is used to use samples in the training set as the main input images during the training phase. I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frame I 2;

[0120] The twin encoder processing module 103 processes the image pair through the twin encoder with shared weights ( , ), generate high-dimensional feature representation X1 and X 2;

[0121] Cross attention calculation module 104 is used to calculate the correlation between features through the cross attention module to generate difference features X C , and enhance the original feature representation;

[0122] Fully connected regression processing module 105 is used to use the fully connected regression module to transform the feature X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d ;

[0123] The orthogonal transformation processing module 106 is used to convert the 6D rotation matrix into a standard 9D rotation matrix through the orthogonal transformation module. R 1. R 2 and R d ;

[0124] The optimization module 107 is used to optimize network parameters based on a multi-task loss function, wherein the multi-task loss function is constructed by a rotation matrix geodesic distance loss function.

[0125] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0126] Data preprocessing: Extract the face area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a Corrected image I ;

[0127] Input image pair construction: During the training phase, samples in the training set are used as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frameI 2;

[0128] Processing image pairs through a siamese encoder with shared weights ( , ), generate high-dimensional feature representation X 1 and X 2;

[0129] Calculate the correlation between features through the cross attention module to generate difference features X C , and enhance the original feature representation;

[0130] Use the fully connected regression module to transform the features X 1. X 2 and differential characteristics X C Mapping to 6D rotation matrix representation A 1. A 2 and A d ;

[0131] The 6D rotation matrix is ​​converted to a standard 9D rotation matrix through the orthogonal transformation module R 1. R 2 and R d ;

[0132] The network parameters are optimized based on a multi-task loss function constructed from a rotation matrix geodesic distance loss function.

[0133] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0134] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0135] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A difference-guided gaze estimation method based on 6D rotation matrix representation, characterized in that: The difference-guided sight estimation method based on 6D rotation matrix representation includes: Data preprocessing: Extract the facial area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image I ; Input image pair construction: During the training phase, samples in the training set are used as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frame I 2; Processing image pairs through a siamese encoder with shared weights ( , ), generate the original high-dimensional feature representation X 1 and X 2; Generate difference features through multi-level cross attention calculation X C , and enhance the original feature representation. The specific process is as follows: Among them, the original features Based on original high-dimensional features X 1. X 2 is taken as input and obtained by feature interaction through the fully connected layer, represents the cross attention module; Use the fully connected regression module to transform the features X 1. X 2 and Mapping to 6D rotation matrix representation A 1. A 2 and A d ; The 6D rotation matrix is ​​converted into the standard 9D rotation matrices R1, R2 and Rd through the orthogonal transformation module; The network parameters are optimized based on a multi-task loss function constructed from a rotation matrix geodesic distance loss function.

2. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 1 is characterized in that: The data preprocessing step includes: Taking the center of the eye or the center of the face as the reference point, rotate the virtual camera to point to the reference point; Rotate the virtual camera so that its perspective faces forward; Image scaling ensures that the distance between the subject and the camera is consistent, generating a calibrated image of a fixed size.

3. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 1 is characterized in that: The calculation method of the cross attention module includes: Project one input feature of the cross-attention module into the query vector Q, and the other input feature into the query vector K and the value vector V; The feature correlation is calculated through the attention mechanism to generate enhanced feature representation.

4. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 1 is characterized in that: The fully connected regression module consists of three independent fully connected layers, which represent high-dimensional features. X 1. X 2 and Mapping to 6D rotation matrix representation A 1. A 2 and A d .

5. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 1 is characterized in that: The method for the orthogonal transformation module to convert the 6D rotation matrix into a 9D rotation matrix includes: Split the 6D rotation matrix into two 3D vectors and ; Generate orthogonal vectors by normalization and cross product ; Combined 3D vectors 、 and orthogonal vectors Get the 9D rotation matrix.

6. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 1 is characterized in that: The calculation method of the rotation matrix geodesic distance loss function is: ； in, represents the arc cosine function, R represents the true rotation matrix, Represents the predicted rotation matrix.

7. The difference-guided sight line estimation method based on 6D rotation matrix representation according to claim 6, characterized in that: The multi-task loss function is: ； in, 、 and The geodesic distance loss corresponding to the line of sight and the difference between the two frames, , in, They represent the true rotation matrix of the sight line corresponding to frame 1, the true rotation matrix of the sight line corresponding to frame 2, and the difference in the true rotation matrix between the two frames. , , They represent the predicted rotation matrix of the sight line corresponding to frame 1, the predicted rotation matrix of the sight line corresponding to frame 2, and the difference in the predicted rotation matrix between the two frames. 、 、 is the weight.

8. A difference-guided sight line estimation device based on 6D rotation matrix representation, characterized in that: include: The data preprocessing module is used to extract the facial area and key points from the original face image, perform perspective transformation and scaling through a virtual camera, and generate a fixed-size corrected image. I ; Input image pair construction module, used in the training phase, to use samples in the training set as the main input images I 1. Randomly sample another sample as a reference frame I 2. In the testing phase, the samples in the test set are used as the main input images I 1. Randomly sample from the training set or predefined reference image set as the reference frame I 2; The twin encoder processing module is used to process image pairs through the twin encoder with shared weights ( , ), generate the original high-dimensional feature representation X 1 and X 2; Cross-attention calculation module, used to generate difference features through multi-level cross-attention calculation X C , and enhance the original feature representation. The specific process is as follows: Among them, the original features Based on original high-dimensional features X 1. X 2 is taken as input and obtained by feature interaction through the fully connected layer, represents the cross attention module; Fully connected regression processing module, used to use the fully connected regression module to transform features X 1. X 2 and Mapping to 6D rotation matrix representation A 1. A 2 and A d ; Orthogonal transformation processing module, used to convert the 6D rotation matrix into a standard 9D rotation matrix through the orthogonal transformation module R 1. R 2 and R d ; An optimization module is used to optimize network parameters based on a multi-task loss function constructed by a rotation matrix geodesic distance loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the difference-guided sight line estimation method based on 6D rotation matrix representation is implemented according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the difference-guided sight line estimation method based on 6D rotation matrix representation are implemented as described in any one of claims 1 to 7.

Images