Sight line estimation method and related equipment thereof

Abstract

The invention discloses a sight line estimation method and related equipment thereof. The method comprises the following steps: acquiring to-be-processed image data; performing sight line direction prediction on the to-be-processed image data through a preset sight line estimation branch in a preset sight line estimation model to obtain a target sight line direction; the preset line-of-sight estimation model comprises a preset line-of-sight estimation branch and a modal alignment branch, and the modal alignment branch aligns the text feature with the preset line-of-sight label with the image feature during training, and aligns the image feature with the text feature with the preset line-of-sight label; when the preset line-of-sight estimation branch predicts the image data to be processed, a target image feature having a first feature distance with the negative sample text feature and having a second feature distance with the positive sample text feature is extracted from the feature layer, and the negative sample text feature is a feature having a reaction on prediction; the positive sample feature is a feature having a positive effect on prediction, and the first feature distance is greater than the second feature distance. The prediction accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication computers, and particularly relates to a line-of-sight estimation method and device, equipment, a storage medium and a computer program product. BACKGROUND

[0002] In recent years, line-of-sight estimation technology is increasingly common in the fields of human-computer interaction, augmented / virtual / mixed reality, eye movement behavior analysis, and assisted driving (intelligent cockpit). In current technology, a line-of-sight estimation scheme generally uses labeled line-of-sight data, such as using collected face images with line-of-sight labels, to supervise the learning and training of a general model. However, the corresponding collected face images still contain many irrelevant features to line-of-sight estimation in addition to the effective human eye region information, which results in low robustness and low prediction accuracy of the line-of-sight estimation model obtained by current conventional training.

[0003] Therefore, how to improve the prediction accuracy of the line-of-sight estimation model is a problem to be solved at present.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not mean that the above content is prior art. SUMMARY

[0005] The main purpose of the present application is to provide a line-of-sight estimation method, device, equipment, storage medium and computer program product, which aims to solve the technical problems of low robustness and low prediction accuracy of the line-of-sight estimation model in related technologies.

[0006] To achieve the above purpose, the present application provides a line-of-sight estimation method, which comprises:

[0007] obtaining image data to be processed;

[0008] predicting the line-of-sight direction of the image data to be processed by a preset line-of-sight estimation branch in a preset line-of-sight estimation model to obtain a target line-of-sight direction; wherein the preset line-of-sight estimation model comprises a preset line-of-sight estimation branch and a modal alignment branch, the modal alignment branch is used to align the text features with the preset line-of-sight labels with the corresponding image features in the training process, and align the image features with the text features with the preset line-of-sight labels, the modal alignment branch enables the preset line-of-sight estimation branch to extract target image features with a corresponding first feature distance from negative sample text features and a corresponding second feature distance from positive sample text features in the process of predicting the image data to be processed at the feature level, the negative sample text features are the learned text features that have an opposite effect on correct prediction, the positive sample features are the learned text features that have a positive effect on correct prediction, and the first feature distance is greater than the second feature distance.

[0009] In an embodiment, before the step of predicting the gaze direction of the image data to be processed by a preset gaze estimation branch in a preset gaze estimation model, the method further comprises:

[0010] obtaining text sample data and image sample data with preset gaze labels;

[0011] determining text features based on the text sample data with preset gaze labels, and determining image features based on the image sample data;

[0012] determining positive sample feature pairs and negative sample feature pairs based on the text features and the image features, wherein the positive sample pairs are sample pairs in which the image features and the text features are successfully matched, and the negative sample pairs are sample pairs in which the image features and the text features are not successfully matched;

[0013] performing bidirectional modal alignment based on the positive sample feature pairs and the negative sample feature pairs, training to obtain a modal alignment branch, and training a preset gaze estimation branch based on the image features until a preset gaze estimation model is obtained.

[0014] In an embodiment, the preset bidirectional modal alignment loss function comprises an image alignment text loss function and a text alignment image loss function, the positive sample feature pairs comprise the negative sample pairs comprise wherein t is a text feature correlation, g is an image feature correlation, is the i-th text feature, is the i-th image feature, is the j-th text feature, is the j-th image feature, i and j have a value range of B, and B is the number of sample feature pairs;

[0015] The step of determining a first loss result corresponding to the modal alignment branch based on the positive sample feature pairs, the negative sample feature pairs, and the preset bidirectional modal alignment loss function comprises:

[0016] determining an image alignment text loss result based on the positive sample feature pairs the negative sample feature pairs and the image alignment text loss function;

[0017] determining a text alignment image loss result based on the positive sample feature pairs the negative sample feature pairs and the text alignment image loss function;

[0018] taking the image alignment text loss result and the text alignment image loss result as the first loss result.

[0019] In an embodiment, a weight parameter is arranged in the preset bidirectional modal alignment loss function, and the weight parameter is used to adjust the size of the first feature distance, so that the distance relationship between the target image feature and different negative sample text features is different.

[0020] In an embodiment, the step of determining the text feature based on the text sample data with the preset line-of-sight label comprises:

[0021] obtaining a context feature sequence corresponding to the text sample data;

[0022] obtaining a label matrix of the line-of-sight label in a label space, and converting the label feature into an embedding feature in an embedding space;

[0023] concatenating the context feature sequence and the embedding feature to obtain a semantic label feature;

[0024] determining the semantic label feature as the text feature.

[0025] In an embodiment, the label space is uniformly distributed with a first basis direction vector, and the embedding space is uniformly distributed with a second basis direction vector. The step of obtaining the label matrix of the line-of-sight label in the label space and converting the label feature into the embedding feature in the embedding space comprises:

[0026] obtaining a first association relationship between the line-of-sight label and its preset nearest neighbor first basis direction vector in the label space;

[0027] based on the first association relationship, obtaining a second association relationship between the line-of-sight label and its preset nearest neighbor second basis direction vector, wherein the first association relationship and the second association relationship satisfy the requirement of a preset geometric constraint loss function;

[0028] based on the second association relationship, converting the label feature into the embedding feature in the embedding space.

[0029] In an embodiment, the step of obtaining the context feature sequence corresponding to the text sample data comprises:

[0030] learning different context prompt words from the text sample data;

[0031] based on the different context prompt words, determining a context feature sequence.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a line-of-sight estimation device, which comprises:

[0033] an acquisition module, configured to acquire image data to be processed;

[0034] The prediction module is configured to predict a target gaze direction by a preset gaze estimation branch in a preset gaze estimation model, to obtain the target gaze direction; wherein the preset gaze estimation model comprises the preset gaze estimation branch and a modal alignment branch, the modal alignment branch is configured to align a text feature with a preset gaze label with a corresponding image feature, and the modal alignment branch enables the preset gaze estimation branch to extract a target image feature having a corresponding first feature distance with a negative sample text feature and a corresponding second feature distance with a positive sample text feature at a feature level in a process of predicting the image data to be processed, the negative sample text feature is a learned text feature having an opposite effect on correct prediction, the positive sample feature is a learned text feature having a positive effect on correct prediction, and the first feature distance is greater than the second feature distance.

[0035] In an embodiment, the gaze estimation device is configured to implement:

[0036] obtain text sample data and image sample data with a preset gaze label;

[0037] determine a text feature based on the text sample data with the preset gaze label, and determine an image feature based on the image sample data;

[0038] determine a positive sample feature pair and a negative sample feature pair based on the text feature and the image feature, wherein the positive sample pair is a sample pair of image features and text features that are successfully matched, and the negative sample pair is a sample pair of image features and text features that are not successfully matched;

[0039] perform bidirectional modal alignment based on the positive sample feature pair and the negative sample feature pair, train to obtain a modal alignment branch, and train a preset gaze estimation branch based on the image feature, until a preset gaze estimation model is obtained.

[0040] In an embodiment, the overall loss function of the preset gaze estimation model comprises a preset bidirectional modal alignment loss function and a preset supervision loss function;

[0041] The gaze estimation device is configured to implement:

[0042] determine a first loss result corresponding to the modal alignment branch based on the positive sample feature pair, the negative sample feature pair, and the preset bidirectional modal alignment loss function; and determine a second loss result based on the image feature and a preset supervision loss function;

[0043] determine whether the preset overall loss function is minimum based on the first loss result and the second loss result, and if it is determined that the preset overall loss function is not minimum, return to the step of redetermining the first loss result and the second loss result until the preset number of training times is reached or the preset overall loss function is minimum, so as to train the preset line-of-sight estimation model.

[0044] In an embodiment, the preset bidirectional modality alignment loss function comprises an image alignment text loss function and a text alignment image loss function, the positive sample feature pair comprises and The negative sample pair comprises and wherein t is a text feature association, g is an image feature association, is the i-th text feature, is the i-th image feature, is the j-th text feature, is the j-th image feature, i and j are in the range of B, and B is the number of sample feature pairs;

[0045] The line-of-sight estimation device is used to implement:

[0046] based on the positive sample feature pair the negative sample feature pair and the image alignment text loss function, determine an image alignment text loss result;

[0047] based on the positive sample feature pair the negative sample feature pair and the text alignment image loss function, determine a text alignment image loss result;

[0048] The image alignment text loss result and the text alignment image loss result are taken as the first loss result.

[0049] In an embodiment, a weight parameter is set in the preset bidirectional modality alignment loss function, and the weight parameter is used to adjust the size of the first feature distance, so that the distance relationship of the target image feature and different negative sample text features is different.

[0050] In an embodiment, the line-of-sight estimation device is used to implement:

[0051] obtain a context feature sequence corresponding to the text sample data;

[0052] obtain a label matrix of the line-of-sight label in a label space, and convert the label feature into an embedding feature in an embedding space;

[0053] concatenate the context feature sequence and the embedding feature to obtain a semantic label feature;

[0054] determine the semantic label feature as the text feature.

[0055] In an embodiment, the line-of-sight estimation apparatus is configured to implement:

[0056] obtain a first association relationship between the line-of-sight label and a first basis direction vector of its preset nearest neighbor in a label space;

[0057] obtain a second association relationship between the line-of-sight label and a second basis direction vector of its preset nearest neighbor based on the first association relationship, wherein the first association relationship and the second association relationship satisfy the requirement of a preset geometric constraint loss function;

[0058] convert the label feature into an embedding feature in an embedding space based on the second association relationship.

[0059] In an embodiment, the line-of-sight estimation apparatus is configured to implement:

[0060] learn different context cue words from the text sample data;

[0061] determine a context feature sequence based on the different context cue words.

[0062] In addition, to achieve the above-mentioned purposes, the present application further provides a line-of-sight estimation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the line-of-sight estimation method as described above.

[0063] In addition, to achieve the above-mentioned purposes, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the line-of-sight estimation method as described above.

[0064] In addition, to achieve the above-mentioned purposes, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the line-of-sight estimation method as described above.

[0065] The above-mentioned technical solutions of the present application have the following technical effects:

[0066] The application provides a line-of-sight estimation method, device, equipment, storage medium and computer program product. Compared with the related art, which only trains a line-of-sight estimation model based on a conventional supervised manner, resulting in low robustness and low accuracy of the corresponding model in predicting a line-of-sight direction, the application obtains target line-of-sight direction by predicting a line-of-sight direction of the to-be-processed image data through a preset line-of-sight estimation branch in a preset line-of-sight estimation model after obtaining the to-be-processed image data. The preset line-of-sight estimation model comprises a preset line-of-sight estimation branch and a modality alignment branch. The modality alignment branch is used to align a text feature with a preset line-of-sight label and a corresponding image feature. The modality alignment branch enables the preset line-of-sight estimation branch to extract, in a feature layer, a target image feature having a corresponding first feature distance from a negative sample text feature and a corresponding second feature distance from a positive sample text feature in the process of predicting the to-be-processed image data. The negative sample text feature is a text feature learned to have an opposite effect on correct prediction. The positive sample feature is a text feature learned to have a positive effect on correct prediction. The first feature distance is greater than the second feature distance.

[0067] It can be understood that, in the application, the preset line-of-sight estimation model comprises two branches, namely, a preset line-of-sight estimation branch and a modality alignment branch. The modality alignment branch enables the text feature with the preset line-of-sight label to be aligned with the corresponding image feature, and enables the corresponding image feature to be aligned with the text feature with the preset line-of-sight label. That is, the modality alignment branch can constrain (mutually constrain) the corresponding image feature based on the corresponding text feature (for training the preset line-of-sight estimation branch). The specific constraint process is that the modality alignment branch enables the corresponding preset line-of-sight estimation branch to extract, in a feature layer, a target image feature having a corresponding first feature distance (a distance that is farther) from a negative sample text feature and a corresponding second feature distance (a distance that is closer) from a positive sample text feature in the process of prediction. The first feature distance is greater than the second feature distance. That is, the modality alignment branch in the application enables the preset line-of-sight estimation branch to shorten the feature distance from the positive sample text feature and lengthen the feature distance from the negative sample text feature in the process of prediction. Since the language text feature provides more abundant and accurate information, the preset line-of-sight estimation branch can learn the image feature better in the training process (that is, the constraint related to the text layer is increased, and the text layer feature provides more abundant and accurate information, and the image feature is close to the text layer feature), thereby obtaining a model with better robustness and more accurate prediction of the line-of-sight direction. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0070] Figure 1 The first flowchart provided for the first embodiment of the line-of-sight estimation method of the present application;

[0071] Figure 2 The second flowchart provided for the second embodiment of the line-of-sight estimation method of the present application;

[0072] Figure 3 The third flowchart related to the second embodiment of the line-of-sight estimation method of the present application;

[0073] Figure 4 The first scene diagram provided for the first embodiment of the line-of-sight estimation method of the present application;

[0074] Figure 5 The second scene diagram provided for the second embodiment of the line-of-sight estimation method of the present application;

[0075] Figure 6 The third scene diagram provided for the second embodiment of the line-of-sight estimation method of the present application;

[0076] Figure 7 The device structure diagram of the hardware running environment related to the line-of-sight estimation method in the embodiments of the present application;

[0077] Figure 8 The module structure diagram of the line-of-sight estimation device in the embodiments of the present application.

[0078] The purposes, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0079] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0080] In order to better understand the technical solutions of the present application, the embodiments will be described in detail with reference to the drawings and specific embodiments in the description.

[0081] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a line-of-sight estimation device, etc. The present embodiment and the following embodiments will be described taking the line-of-sight estimation device as an example.

[0082] Based on this, embodiments of this application provide a line-of-sight estimation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the line-of-sight estimation method of this application.

[0083] In this embodiment, the line-of-sight estimation method includes steps S10 to S20:

[0084] Step S10: Obtain the image data to be processed;

[0085] In this embodiment, it should be noted that the image to be processed can be an image input to the gaze estimation device through any means. For example, the image to be processed can be an image directly transmitted from a camera to the gaze estimation device, or the image to be processed can be an image uploaded to the gaze estimation device by a client such as a user's mobile terminal, or the image to be processed can be an image uploaded to the gaze estimation device by the advertising promotion terminal corresponding to the shopping mall advertising screen, or the image to be processed can be an image collected and uploaded to the gaze estimation device by the vehicle unit, etc., and there is no specific limitation.

[0086] It should be emphasized that when the application scenario is to determine whether a child has strabismus using the gaze estimation method of this application, the corresponding image to be processed can be an image uploaded to the gaze estimation device by the user's mobile terminal. When the application scenario is to determine whether there is a violation of driving operation using the gaze estimation method of this application, the corresponding image to be processed can be an image collected by the vehicle unit and uploaded to the gaze estimation device. When the application scenario is to conduct targeted advertising promotion using the gaze estimation method of this application, the corresponding image to be processed can be an image uploaded to the gaze estimation device by the advertising promotion terminal corresponding to the shopping mall advertising screen, etc., and there is no specific limitation.

[0087] Furthermore, the image to be processed can be any type of image; for example, the image to be processed can be an captured image, or it can be an image obtained by taking a screenshot.

[0088] Furthermore, the image to be processed can be in various formats, such as gpg or BMP, etc., without any specific limitation.

[0089] After acquiring the image to be processed, the gaze estimation device can process the image. For example, after receiving an image uploaded to the gaze estimation device from an advertising promotion terminal, the gaze estimation device can predict or recognize the image (including the direction of the human eye's gaze) to determine the user's focus and then promote the content that the user is interested in, such as specific sports events or specific products. Or, for example, after the gaze estimation device receives an image uploaded to the gaze estimation device from an in-vehicle unit, it can predict or recognize the image (including the direction of the human eye's gaze) to determine whether the user's gaze is in line with driving habits and then provide targeted reminders.

[0090] Step S20: The gaze direction of the image data to be processed is predicted by the preset gaze estimation branch in the preset gaze estimation model to obtain the target gaze direction. The preset gaze estimation model includes a preset gaze estimation branch and a modal alignment branch. The modal alignment branch is used to align text features with preset gaze labels with corresponding image features. The modal alignment branch enables the preset gaze estimation branch to extract target image features at the feature level during the prediction of the image data to be processed. The target image features have a first feature distance with the negative sample text features and a second feature distance with the positive sample text features. The negative sample text features are the learned text features that have a negative effect on correct prediction, and the positive sample features are the learned text features that have a positive effect on correct prediction. The first feature distance is greater than the second feature distance.

[0091] In this embodiment, the network structure of the preset gaze estimation model is... Figure 5 As shown, the preset gaze estimation model includes a preset gaze estimation branch and a modal alignment branch. Specifically, the training of the preset gaze estimation branch and the modal alignment branch can be performed simultaneously, such as... Figure 4 As shown, in Figure 4In this context, content Prompt is the text prompt vector, Gaze label is the gaze label, embedding space is the embedding space, Gaze embedding is the gaze embedding feature, Semantic label is the semantic label, text encoder is the text encoder, latent space is the latent space, vision encoder is the image encoder, image feature is the feature image, Geometric semantic interpolation is spherical linear interpolation, text feature extraction is text feature extraction, feature alignment is feature alignment, image feature extraction is image feature extraction, Trainable means trainable, concatenate means continuous, negative text feature is negative sample text feature, and positive text feature is positive sample text feature. Overall, from... Figure 4 As can be seen, it is necessary to obtain text features based on gaze labels and text cue vectors (at the same time, extract image features based on image data), and then the extracted text features and image features need to be aligned (modal alignment branch).

[0092] It should be noted that in this embodiment, the modality alignment branch has a special design (a dual-modality bidirectional alignment setting and a special design for the alignment loss function). This special design can use text features to constrain the learning of image features (after constraint, in the next round of training, it will in turn affect the preset gaze estimation branch). Thus, the preset gaze estimation branch trained accordingly extracts target image features at the feature level that are far away from the corresponding negative sample text features and close to the corresponding positive sample text features during the prediction process.

[0093] Specifically, before the step of identifying the license plate number in the image to be processed using a preset gaze estimation model, the following steps S01-S04 are included:

[0094] Before the step of predicting the gaze direction of the image data to be processed using a preset gaze estimation branch in a preset gaze estimation model, the method further includes:

[0095] Step S01: Obtain text sample data and image sample data with preset gaze labels;

[0096] In this embodiment, the different branches of the preset gaze estimation model are specifically explained. Specifically, the sample data of the preset gaze estimation model includes two parts: one part is text sample data with preset gaze labels, and the other part is its corresponding image sample data.

[0097] In this embodiment, the preset gaze label can be 60 degrees upward, 30 degrees to the left, etc. The text sample data can specifically be reflected as: "the direction of the person's eyes" or "the direction of the head". Therefore, the overall text sample data with preset gaze labels can be: "the direction of the person's eyes is 60 degrees upward" or "the direction of the head is 30 degrees to the left".

[0098] In this embodiment, the image sample data may include images of the head, such as... Figure 4 As shown.

[0099] Step S02: Based on the text sample data with preset gaze labels, determine text features; based on the image sample data, determine image features.

[0100] In this embodiment, the text sample data with preset gaze labels is subjected to feature extraction using a combination of spherical interpolation and bilinear interpolation techniques to obtain text features. Similarly, the image sample data is subjected to feature extraction to obtain image features.

[0101] Step S03: Based on the text features and the image features, determine positive sample feature pairs and negative sample feature pairs, wherein the positive sample pair is a sample pair in which the image features and text features are successfully matched, and the negative sample pair is a sample pair in which the image features and text features are not successfully matched;

[0102] In this embodiment, a positive sample pair is a sample pair in which image features and text features are successfully matched, and a negative sample pair is a sample pair in which image features and text features are not successfully matched. For example, if the image features are... Text features are Image features Text features These are positive sample pairs, image features. Text features These are positive sample pairs, image features. Text features If the positive samples are equivalent, then the image features Text features as well as These are negative sample pairs, image features. Text features as well as These are negative sample pairs, image features. Text features as well as It is a negative sample pair.

[0103] Step S04: Based on the positive sample feature pairs and the negative sample feature pairs, perform bidirectional modal alignment to train the modal alignment branch, and train the preset gaze estimation branch based on image features until the preset gaze estimation model is obtained.

[0104] After obtaining the positive sample feature pairs and the negative sample feature pairs, bidirectional modal alignment is performed based on the positive sample feature pairs and the negative sample feature pairs to train a modal alignment branch. Specifically, the positive sample feature pairs and the negative sample feature pairs are substituted into the loss function corresponding to the modal alignment branch, the corresponding loss function value is calculated, and the process is iterated until the cutoff condition is met, such as when the training reaches the corresponding number of times, then the training is complete.

[0105] In this embodiment, a preset gaze estimation branch is also trained based on image features until a preset gaze estimation model is obtained.

[0106] It should be noted that, for the same batch, the training data for the preset modality alignment branch is positive sample feature pairs and negative sample feature pairs, while for the preset gaze estimation branch, the corresponding training data is image features. It should also be noted that, for the next batch, the corresponding preset modality alignment branch of the previous batch will affect the learning of image features in the next batch (through the constraints of the loss function).

[0107] The overall loss function of the preset gaze estimation model includes a preset bidirectional modality alignment loss function and a preset supervision loss function.

[0108] The steps of performing bidirectional modal alignment based on the positive sample feature pairs and the negative sample feature pairs, training a modal alignment branch, and training a preset gaze estimation branch based on image features until a preset gaze estimation model is obtained, include steps S041-S042:

[0109] Step S041: Based on the positive sample feature pair, the negative sample feature pair, and the preset bidirectional modal alignment loss function, determine the first loss result corresponding to the modal alignment branch; and based on the image features and the preset supervised loss function, determine the second loss result.

[0110] In this embodiment, the overall loss function of the preset gaze estimation model includes a preset bidirectional modal alignment loss function and a preset supervision loss function. The preset bidirectional modal alignment loss function belongs to the modal alignment branch, and the preset supervision loss function belongs to the preset gaze estimation branch.

[0111] Specifically, the preset bidirectional modal alignment loss function includes an image-to-text loss function and a text-to-image loss function, and the positive sample feature pairs include... and The negative sample pairs include and Where t represents text feature association and g represents image feature association. For the i-th text feature, For the i-th image feature, For the j-th text feature, Let i be the j-th image feature, and let B be the range of values ​​for i and j, where B is the logarithm of the sample features.

[0112] The step of determining the first loss result corresponding to the modality alignment branch based on the positive sample feature pair, the negative sample feature pair, and the preset bidirectional modality alignment loss function includes:

[0113] Based on the positive sample features The negative sample feature pair And the image-to-text loss function, to determine the image-to-text loss result;

[0114] Based on the positive sample features The negative sample feature pair And the text-aligned image loss function, to determine the text-aligned image loss result;

[0115] The image-to-text loss result and the text-to-image loss result are used as the first loss result.

[0116] Specifically, the preset bidirectional modal alignment loss function can include two parts: l MCR (Preset bidirectional modal alignment loss function) = l i2t (Image-to-text alignment loss function) + l t2i (Text-to-Image Loss Function), where,

[0117]

[0118] in and w(i,j)=1-cos(g i ,g j B is the logarithm of the sample features. For positive sample feature pairs, For negative sample feature pairs, t represents text feature association, and g represents image feature association. i With g j The gaze labels associated with the text features (directional vectors in different directions, which can be three-dimensional).

[0119] In this embodiment, the preset supervised loss function is specifically as follows:

[0120]

[0121] Among them, the predicted line of sight direction is g represents the direction of the label's view.

[0122] Step S041 is based on the first loss result and the second loss result, and determines whether the preset overall loss function is minimized. If it is determined that the preset overall loss function is not minimized, the step of re-determining the first loss result and the second loss result is returned until the number of training times reaches the preset number or the preset overall loss function is minimized, so as to train the preset gaze estimation model.

[0123] In this embodiment, based on the first loss result and the second loss result, it is determined whether the preset overall loss function is minimized. If it is determined that the preset overall loss function is not minimized, the step of re-determining the first loss result and the second loss result is returned until the number of training times reaches a preset number or the preset overall loss function is minimized, so as to train the preset gaze estimation model.

[0124] In this embodiment, since the preset gaze estimation model is a pre-trained model, the gaze direction can be predicted for the image data to be processed through the preset gaze estimation branch in the preset gaze estimation model, thereby obtaining the target gaze direction.

[0125] It is understood that in this application, the preset gaze estimation model includes two branches: a preset gaze estimation branch and a modal alignment branch. The modal alignment branch aligns text features with preset gaze labels with corresponding image features, and aligns corresponding image features with text features with preset gaze labels. In other words, the modal alignment branch can constrain (mutually constrain) corresponding image features based on corresponding text features (used for training the preset gaze estimation branch). The specific constraint process is as follows: the modal alignment branch ensures that, during the prediction process, the preset gaze estimation branch extracts features with a first feature distance (greater distance) from negative sample text features and a second feature distance from positive sample text features. The target image features with a closer feature distance (the first feature distance is greater than the second feature distance). That is, the modal alignment branch described in this application enables the preset gaze estimation branch to narrow the feature distance with positive sample text features and widen the feature distance with negative sample text features during the prediction process. Since language text features provide richer and more accurate information, the preset gaze estimation branch learns image features better during the training process than before (i.e., it adds text-level constraints, and the text-level features provide richer and more accurate information, while image features are closer to text-level features). As a result, the model obtained has better robustness and can predict gaze direction more accurately.

[0126] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of determining text features based on the text sample data with preset gaze labels includes:

[0127] Step A1: Obtain the context feature sequence corresponding to the text sample data;

[0128] In this embodiment, the text sample data needs to be converted into feature vectors, and the resulting context feature sequence is obtained after the conversion.

[0129] Specifically, the step of obtaining the context feature sequence corresponding to the text sample data includes:

[0130] Different contextual cue words are learned from the text sample data;

[0131] Based on the different contextual cue words, a contextual feature sequence is determined.

[0132] In this embodiment, the contextual prompts are learnable:

[0133] Specifically, as an example, the text sample data includes words such as "human eye," "head," "face," "of," and "direction." Different contextual cue words are learned from the text sample data. For example, based on the text sample data "of," "direction," "human eye," "face," and "head," different contextual cue words are learned such as: "the direction of the human eye," "the direction of the head is," and "the direction of the face is." Furthermore, a contextual feature sequence is determined, which can specifically be Ti = {v1, v2, ..., vL-1}.

[0134] Step A2: Obtain the label matrix of the gaze label in the label space, and convert the label features into embedding features in the embedding space;

[0135] In this embodiment, the label matrix of the gaze label in the label space is obtained, and then the label features are transformed into embedding features in the embedding space. Specifically, the gaze label is transformed from a three-dimensional space representation to a high-dimensional space representation (consistent with the context feature sequence). The embedding feature can specifically be vg.

[0136] The label space contains a first fundamental direction vector, such as... Figure 6 As shown in (a), the embedding space is uniformly distributed with a second fundamental direction vector, such as Figure 6 As shown in (b), refer to Figure 3 The step of obtaining the label matrix of the gaze label in the label space and converting the label features into embedding features in the embedding space includes:

[0137] Step B1: Obtain the first association relationship between the line-of-sight label and its preset nearest neighbor's first basic direction vector in the label space;

[0138] This embodiment illustrates how to combine spherical interpolation and bilinear interpolation techniques to transform the label features into embedding features in the embedding space. Specifically, as follows: Figure 6 As shown in Figure (a), the label space is a space in which the line-of-sight label can be expressed by the first basic direction vector of its preset nearest neighbor. For example, the line-of-sight label 60 degrees can be expressed by the first basic direction vector of its preset nearest neighbor (such as 30 degrees and 90 degrees). In Figure (a), g1 is 30 degrees, g2 is 90 degrees, and the line-of-sight label is gi. gi can be expressed by the first basic direction vector (such as 30 degrees and 90 degrees).

[0139] Specifically, the relationship between the gaze label and its preset nearest neighbor's first basic direction vector is the first association relationship.

[0140] Specifically, in Figure (a):

[0141] Step B2: Based on the first association relationship, obtain the second association relationship between the gaze label and its preset nearest neighbor's second basic direction vector, wherein the first association relationship and the second association relationship satisfy the requirements of the preset geometric constraint loss function;

[0142] Step B3: Based on the second association relationship, the label features are transformed into embedded features in the embedding space.

[0143] In this embodiment, based on the first association relationship, a second association relationship is obtained between the gaze label and its preset nearest neighbor's second basic direction vector. The first association relationship and the second association relationship satisfy the requirements of a preset geometric constraint loss function. That is, in this embodiment, the preset geometric constraint loss function minimizes the loss in the geometric dimension of the high-dimensional representation (embedding space) of the gaze label. Specifically, the relationship between the gaze label and its surrounding basic direction vectors in the embedding space (i.e., in the high-dimensional space) is consistent with the relationship between the gaze label and its surrounding basic direction vectors in the label space.

[0144] In this embodiment, the preset geometric constraint loss function is:

[0145]

[0146] In this embodiment, L1 is the LI loss function, g1 and g2 are the first basic direction vectors of the preset nearest neighbors in the label space, and A1 and A2 are the second basic direction vectors of the preset nearest neighbors in the embedding space. Based on the first association relationship, the second association relationship between the gaze label and the second basic direction vector of its preset nearest neighbor is obtained, thereby obtaining the embedding features of the label features in the embedding space. The specific calculation process can be described as follows: Figure 6 As shown in (b), (c), and (d):

[0147] in,

[0148]

[0149] Among them, W a1 W a2 W b1 W b2 W ia W ib All of these are corresponding weights.

[0150] Step A3: Concatenate the context feature sequence and the embedded features to obtain semantic label features;

[0151] In this embodiment, the gaze label is used as a word unit vg in the semantic label feature. Specifically, the semantic label feature can be {v1,v2,...,vL-1,vg}.

[0152] Step A4: The semantic tag features are determined as the text features.

[0153] In this embodiment, the semantic tag features are determined as the text features, that is, the text features are {v1,v2,...,vL-1,vg}.

[0154] In this embodiment, since the transformation of label features into embedded features in the embedding space is subject to a preset geometric constraint loss function, excessive loss in the geometric space during the transformation process is avoided, which could lead to inaccurate model training. Therefore, in this embodiment, the prediction accuracy of the model can be improved.

[0155] Based on the first and second embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, the preset bidirectional modal alignment loss function is provided with a weight parameter, which is used to adjust the magnitude of the first feature distance, so that the distance relationship between the target image features and different negative sample text features is different.

[0156] In this embodiment, the preset bidirectional modal alignment loss function is specifically as follows:

[0157]

[0158] in and w(i,j)=1-cos(g i ,g j B is the logarithm of the sample features. For positive sample feature pairs, For negative sample feature pairs, g represents text feature association, t represents image feature association, and g i With g j Different vectors associated with text features.

[0159] In this embodiment, w(i,j) in the loss function is the weight parameter, which is used to adjust the magnitude of the first feature distance so that the distance relationship between the target image feature and different negative sample text features is different. Specifically, the weight parameter can make the distance relationship between the target image feature and the negative sample text feature with a small distance smaller, and the distance relationship with the negative sample text feature with a large distance larger. For example, the weight parameter can make the distance relationship between the target image feature (corresponding to a text feature of 30 degrees) and the negative sample text feature with a small distance (29 degrees) smaller, and the distance relationship with the negative sample text feature with a large distance (80 degrees) larger.

[0160] In this embodiment, the distance between negative samples is controlled by weights, which avoids excessive shrinkage of negative samples and thus avoids model overfitting.

[0161] This application also provides a line-of-sight estimation device, please refer to... Figure 8 The line-of-sight estimation device includes:

[0162] Acquisition module 10 is used to acquire image data to be processed;

[0163] The prediction module 20 is used to predict the gaze direction of the image data to be processed through a preset gaze estimation branch in a preset gaze estimation model, thereby obtaining the target gaze direction. The preset gaze estimation model includes a preset gaze estimation branch and a modal alignment branch. During training, the modal alignment branch aligns text features with preset gaze labels with corresponding image features, and aligns image features with text features with preset gaze labels. The modal alignment branch enables the preset gaze estimation branch to extract target image features at the feature level during the prediction process of the image data to be processed. These target image features have a first feature distance to negative sample text features and a second feature distance to positive sample text features. The negative sample text features are learned text features that have a negative effect on correct prediction, and the positive sample features are learned text features that have a positive effect on correct prediction. The first feature distance is greater than the second feature distance.

[0164] In one embodiment, the line-of-sight estimation device is used to achieve:

[0165] Acquire text sample data and image sample data with preset gaze labels;

[0166] Based on the text sample data with preset gaze labels, text features are determined; based on the image sample data, image features are determined.

[0167] Based on the text features and the image features, positive sample feature pairs and negative sample feature pairs are determined, wherein the positive sample pair is a sample pair in which the image features and text features are successfully matched, and the negative sample pair is a sample pair in which the image features and text features are not successfully matched;

[0168] Based on the positive sample feature pairs and the negative sample feature pairs, bidirectional modal alignment is performed to train a modal alignment branch, and a preset gaze estimation branch is trained based on image features until a preset gaze estimation model is obtained.

[0169] In one embodiment, the overall loss function of the preset gaze estimation model includes a preset bidirectional modal alignment loss function and a preset supervision loss function;

[0170] The line-of-sight estimation device is used to achieve:

[0171] Based on the positive sample feature pairs, the negative sample feature pairs, and the preset bidirectional modal alignment loss function, the first loss result corresponding to the modal alignment branch is determined; and based on the image features and the preset supervised loss function, the second loss result is determined.

[0172] Based on the first loss result and the second loss result, it is determined whether the preset overall loss function is minimized. If it is determined that the preset overall loss function is not minimized, the step of re-determining the first loss result and the second loss result is returned until the number of training times reaches the preset number or the preset overall loss function is minimized, so as to train the preset gaze estimation model.

[0173] In one embodiment, the preset modality loss function includes an image-to-text loss function and a text-to-image loss function, and the positive sample feature pairs include... and The negative sample pairs include and Where g represents text feature association, t represents image feature association, and the range of values ​​for i and j is B, where B is the logarithm of the sample features;

[0174] The line-of-sight estimation device is used to achieve:

[0175] Based on the positive sample features The negative sample feature pair And the image-to-text loss function, to determine the image-to-text loss result;

[0176] Based on the positive sample features The negative sample feature pair And the text-aligned image loss function, to determine the text-aligned image loss result;

[0177] The image-to-text loss result and the text-to-image loss result are used as the first loss result.

[0178] In one embodiment, the preset bidirectional modal alignment loss function is provided with a weight parameter, which is used to adjust the magnitude of the first feature distance so that the distance relationship between the target image feature and different negative sample text features is different.

[0179] In one embodiment, the line-of-sight estimation device is used to achieve:

[0180] Obtain the context feature sequence corresponding to the text sample data;

[0181] Obtain the label matrix of the gaze label in the label space, and transform the label features into embedding features in the embedding space;

[0182] By concatenating the context feature sequence and the embedded features, semantic label features are obtained;

[0183] The semantic label features are determined as the text features.

[0184] In one embodiment, the line-of-sight estimation device is used to achieve:

[0185] When obtaining the label space, the first association relationship between the line-of-sight label and the first basic direction vector of its preset nearest neighbor;

[0186] Based on the first association relationship, a second association relationship is obtained between the line-of-sight label and its preset nearest neighbor's second basic direction vector, wherein the first association relationship and the second association relationship satisfy the requirements of a preset geometric constraint loss function;

[0187] Based on the second association relationship, the label features are transformed into embedded features in the embedding space.

[0188] In one embodiment, the line-of-sight estimation device is used to achieve:

[0189] Different contextual cue words are learned from the text sample data;

[0190] Based on the different contextual cue words, a contextual feature sequence is determined.

[0191] The line-of-sight estimation device provided in this application, employing the line-of-sight estimation method in the above embodiments, can solve the technical problem of line-of-sight estimation. Compared with the prior art, the beneficial effects of the line-of-sight estimation device provided in this application are the same as those of the line-of-sight estimation method provided in the above embodiments, and other technical features in the line-of-sight estimation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0192] This application provides a line-of-sight estimation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the line-of-sight estimation method in Embodiment 1 above.

[0193] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a gaze estimation device suitable for implementing embodiments of this application. The gaze estimation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The line-of-sight estimation device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0194] like Figure 7 As shown, the gaze estimation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the gaze estimation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the line-of-sight estimation device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show line-of-sight estimation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0195] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0196] The line-of-sight estimation device provided in this application, employing the line-of-sight estimation method in the above embodiments, can solve the technical problem of line-of-sight estimation. Compared with the prior art, the beneficial effects of the line-of-sight estimation device provided in this application are the same as those of the line-of-sight estimation method provided in the above embodiments, and other technical features of the line-of-sight estimation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0197] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0199] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the line-of-sight estimation method in the above embodiments.

[0200] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, system, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0201] The aforementioned computer-readable storage medium may be included in the line-of-sight estimation device; or it may exist independently and not be assembled into the line-of-sight estimation device.

[0202] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the line-of-sight estimation device, cause the line-of-sight estimation device to perform the aforementioned line-of-sight estimation method.

[0203] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0204] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0205] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0206] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described line-of-sight estimation method, thereby solving the technical problem of line-of-sight estimation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the line-of-sight estimation method provided in the above embodiments, and will not be repeated here.

[0207] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the line-of-sight estimation method as described above.

[0208] The computer program product provided in this application can solve the technical problem of line-of-sight estimation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the line-of-sight estimation method provided in the above embodiments, and will not be repeated here.

[0209] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A line of sight estimation method, characterized by, The method comprises: acquiring image data to be processed; predicting the gaze direction of the image data to be processed by a preset gaze estimation branch in a preset gaze estimation model, to obtain a target gaze direction; wherein the preset gaze estimation model comprises a preset gaze estimation branch and a modal alignment branch, the modal alignment branch is used to align text features with preset gaze labels with corresponding image features in a training process, and align image features with text features with preset gaze labels, the modal alignment branch enables the preset gaze estimation branch to extract target image features having a corresponding first feature distance with negative sample text features and a corresponding second feature distance with positive sample text features at a feature level in the process of predicting the image data to be processed, the negative sample text features are text features learned to have an opposite effect on correct prediction, the positive sample features are text features learned to have a positive effect on correct prediction, and the first feature distance is greater than the second feature distance.

2. The method of claim 1, wherein, Before the step of predicting the gaze direction of the image data to be processed by a preset gaze estimation branch in a preset gaze estimation model, the method further comprises: acquiring text sample data and image sample data with preset gaze labels; determining text features based on the text sample data with preset gaze labels, and determining image features based on the image sample data; determining positive sample feature pairs and negative sample feature pairs based on the text features and the image features, wherein the positive sample pairs are sample pairs in which image features and text features are successfully matched, and the negative sample pairs are sample pairs in which image features and text features are not successfully matched; based on the positive sample feature pairs and the negative sample feature pairs, performing bidirectional modal alignment to train the modal alignment branch, and training the preset gaze estimation branch based on the image features, until a preset gaze estimation model is obtained.

3. The method of claim 2, wherein, The overall loss function of the preset gaze estimation model comprises a preset bidirectional modal alignment loss function and a preset supervision loss function; The step of training the modal alignment branch based on the positive sample feature pairs and the negative sample feature pairs, and training the preset gaze estimation branch based on the image features, until a preset gaze estimation model is obtained, comprises: determining a first loss result corresponding to the modal alignment branch based on the positive sample feature pairs, the negative sample feature pairs, and the preset bidirectional modal alignment loss function; and determining a second loss result based on the image features and the preset supervision loss function; determining whether the preset overall loss function is minimum based on the first loss result and the second loss result, if it is determined that the preset overall loss function is not minimum, returning to the step of redetermining the first loss result and the second loss result, until the training number reaches a preset number or the preset overall loss function is minimum, to train the preset gaze estimation model.

4. The method of claim 3, wherein, The preset bidirectional modal alignment loss function comprises an image alignment text loss function and a text alignment image loss function, the positive sample feature pair comprises and The negative sample pair comprises and Wherein, t is a text feature correlation, g is an image feature correlation, is the i th text feature, is the i th image feature, is the j th text feature, is the j th image feature, the value range of i and j is B, and B is the number of sample feature pairs. The step of determining a first loss result corresponding to the modal alignment branch based on the positive sample feature pairs, the negative sample feature pairs, and the preset bidirectional modal alignment loss function comprises: based on the positive sample feature pair the negative sample feature pair and an image alignment text loss function, determine an image alignment text loss result; based on the positive sample feature pair the negative sample feature pair and a text alignment image loss function, determine a text alignment image loss result; The image alignment text loss result and the text alignment image loss result are taken as the first loss result.

5. The method of claim 3, wherein, The preset bidirectional modal alignment loss function is provided with a weight parameter, which is used to adjust the size of the first feature distance, so that the distance relationship of the target image feature and different negative sample text features is different.

6. The method of claim 2, wherein, The step of determining the text feature based on the text sample data with the preset gaze label comprises: obtaining a context feature sequence corresponding to the text sample data; obtaining a label matrix of the gaze label in a label space, and converting the label feature into an embedding feature in an embedding space; concatenating the context feature sequence and the embedding feature to obtain a semantic label feature; determining the semantic label feature as the text feature.

7. The method of claim 6, wherein, The label space is uniformly distributed with a first basic direction vector, and the embedding space is uniformly distributed with a second basic direction vector, and the step of obtaining the label matrix of the gaze label in the label space and converting the label feature into the embedding feature in the embedding space comprises: obtaining a first association relationship between the gaze label and its preset nearest neighbor first basic direction vector in the label space; based on the first association relationship, obtaining a second association relationship between the gaze label and its preset nearest neighbor second basic direction vector, wherein the first association relationship and the second association relationship meet the requirements of a preset geometric constraint loss function; based on the second association relationship, converting the label feature into the embedding feature in the embedding space.

8. The method of claim 6, wherein, The step of obtaining the context feature sequence corresponding to the text sample data comprises: learning different context prompt words from the text sample data; based on the different context prompt words, determining a context feature sequence.

9. A line of sight estimation device, comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the gaze estimation method according to any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the gaze estimation method according to any one of claims 1 to 8.

Images