Sight line estimation method, device and equipment

Abstract

The invention provides a sight line estimation method, device and equipment. The method comprises the following steps: inputting a sample image into an initial sight line estimation model to obtain a sight line image feature, a sight line related feature and a sight line predicted value; inputting the sample image to an image encoder to obtain facial image features; inputting the sight-line-independent description text into a text encoder to obtain sight-line-independent features; wherein the sight-line-independent description text is used for describing sight-line-independent interference information in the sample image; determining a target loss value based on the sight line image features, the sight line related features, the sight line predicted value, the facial image features, the sight line unrelated features and the sight line label value; network parameters of the initial sight line estimation model are adjusted based on the target loss value, a trained target sight line estimation model is obtained, and the target sight line estimation model is used for performing sight line estimation based on the to-be-detected image. According to the technical scheme, the influence of sight irrelevant factors on sight estimation performance is reduced, and the accuracy of a sight estimation result is improved.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and particularly to a gaze estimation method, device, and equipment. Background Art

[0002] Gaze Estimation is a technology related to the eyeball, eye movement, gaze, etc., which takes facial images (including the facial area and the eye area) as the processing object. In gaze estimation, the gaze direction of the user is estimated based on the facial image, and the fixation point coordinates intersecting with the gaze direction are obtained by combining the regional coordinates.

[0003] Gaze Estimation can be applied to scenarios such as games, VR, assisted driving (intelligent cockpit), screen applications (such as mobile phone unlocking, short video special effects, etc.). Among them, in the game scenario, game interaction is carried out through gaze estimation. In the VR scenario, the gaze direction of the user is accurately estimated through the built-in camera of the helmet, so as to perform local fine rendering on the scene. In the assisted driving scenario, by analyzing the gaze direction of the user, it is detected whether the driver is fatigued and whether the attention is concentrated, improving safety. In the screen application scenario, interaction is carried out through gaze estimation. In the screen application scenario, the user's concentration can also be obtained through gaze analysis, scoring for classes, or products, or videos, and estimating the user's interest, etc.

[0004] However, in addition to including effective human eye region information, facial images also include a large number of gaze-irrelevant features, such as identity features (such as face shape, eyebrows, mouth, nose, etc.), expression changes (such as smiling, crying, laughing, etc.), texture attributes (such as the color and style of hair and beard), and other features (such as gender and age, etc.). These gaze-irrelevant features damage the generalization performance of gaze estimation, that is, the accuracy of the gaze estimation result is greatly reduced, so that an accurate gaze estimation result cannot be obtained. Summary of the Invention

[0005] The present application provides a gaze estimation method, and the method includes:

[0006] Obtain an initial gaze estimation model to be trained, input a sample image into the initial gaze estimation model, and obtain a gaze image feature, a gaze-related feature, and a gaze prediction value;

[0007] Input the sample image into an image encoder to obtain a facial image feature;

[0008] Input a gaze-irrelevant description text into a text encoder to obtain a gaze-irrelevant feature; wherein, the gaze-irrelevant description text is used to describe the interference information irrelevant to the gaze in the sample image;

[0009] Determine a target loss value based on the line-of-sight image features, the line-of-sight related features, the line-of-sight prediction value, the facial image features, the line-of-sight unrelated features, and the line-of-sight label value;

[0010] Adjust the network parameters of the initial line-of-sight estimation model based on the target loss value to obtain a trained target line-of-sight estimation model, where the target line-of-sight estimation model is used to perform line-of-sight estimation based on the image to be detected.

[0011] This application provides a line-of-sight estimation device, and the device includes:

[0012] An acquisition module, configured to acquire an initial line-of-sight estimation model to be trained, input a sample image to the initial line-of-sight estimation model to obtain line-of-sight image features, line-of-sight related features, and a line-of-sight prediction value; input the sample image to an acquired image encoder to obtain facial image features; input line-of-sight unrelated description text to an acquired text encoder to obtain line-of-sight unrelated features; where the line-of-sight unrelated description text is used to describe interference information unrelated to the line of sight in the sample image;

[0013] A determination module, configured to determine a target loss value based on the line-of-sight image features, the line-of-sight related features, the line-of-sight prediction value, the facial image features, the line-of-sight unrelated features, and the line-of-sight label value;

[0014] A processing module, configured to adjust the network parameters of the initial line-of-sight estimation model based on the target loss value to obtain a trained target line-of-sight estimation model; where the target line-of-sight estimation model is used to perform line-of-sight estimation based on the image to be detected.

[0015] This application provides an electronic device, including: a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the line-of-sight estimation method in the above examples of this application.

[0016] As can be seen from the above technical solutions, in the embodiments of the present application, the view-independent description text can be input to the text encoder to obtain view-independent features. The view-independent description text is used to describe the interference information unrelated to the view in the sample image. Then, based on the view-independent features, the network parameters of the initial view estimation model are adjusted to obtain the target view estimation model, so as to train an accurate and reliable target view estimation model based on the view-independent features. When performing view estimation based on the target view estimation model, the interference of view-independent factors can be reduced, the influence of view-independent factors on the view estimation performance can be alleviated, the accuracy of the view estimation result can be improved, an accurate view estimation result can be obtained, and the performance of the target view estimation model can be enhanced. In view of the fact that the facial features of different users are different, the facial personalized features of different users can be mapped to the text space to improve the accuracy of describing the face in the context, and further improve the accuracy of describing the current individual with view-independent features, and finally enhance the robustness of the target view estimation model in the view estimation generalization framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments of the present application or the prior art. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings of the embodiments of the present application.

[0018] Figure 1 is a flowchart of the view estimation method in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the training process of the target view estimation model in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the training of the target view estimation model in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the training process of the target view estimation model in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of training to obtain the target text generation network in an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of the training of the target text generation network in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of the training process of the target text generation network in an embodiment of the present application;

[0025] Figure 8 is a schematic flowchart of the line-of-sight estimation method in an embodiment of the present application;

[0026] Figure 9A is a schematic structural diagram of the line-of-sight estimation device in an embodiment of the present application;

[0027] Figure 9B is a hardware structure diagram of an electronic device in an embodiment of the present application. Specific embodiments

[0028] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and do not limit the present application. The singular forms "a", "the", and "said" used in the present application and the claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "when" or "while" or "in response to determining".

[0030] In the embodiments of the present application, a line-of-sight estimation method is proposed. This method can be applied to an electronic device. Refer to Figure 1 as shown, which is a schematic flowchart of the line-of-sight estimation method. The method may include:

[0031] Step 101: Obtain an initial line-of-sight estimation model to be trained, input a sample image into the initial line-of-sight estimation model, and obtain line-of-sight image features, line-of-sight related features, and a line-of-sight prediction value.

[0032] Step 102: Input the sample image into an image encoder to obtain facial image features.

[0033] Step 103: Input line-of-sight irrelevant description text into a text encoder to obtain line-of-sight irrelevant features; wherein, the line-of-sight irrelevant description text is used to describe the interference information unrelated to the line of sight in the sample image.

[0034] Step 104: Determine a target loss value based on the line-of-sight image features, the line-of-sight related features, the line-of-sight prediction value, the facial image features, the line-of-sight irrelevant features, and a line-of-sight label value.

[0035] Step 105: Adjust the network parameters of the initial line-of-sight estimation model based on the target loss value (i.e., train the initial line-of-sight estimation model) to obtain the trained target line-of-sight estimation model. The target line-of-sight estimation model can be used to estimate the line of sight based on the image to be detected.

[0036] Exemplarily, the initial line-of-sight estimation model may include a line-of-sight feature extractor, a multi-layer perceptron, and a fully connected layer. Input the sample image into the initial line-of-sight estimation model to obtain line-of-sight image features, line-of-sight related features, and line-of-sight prediction values, which may include, but are not limited to: input the sample image into the line-of-sight feature extractor to obtain the line-of-sight image features; input the line-of-sight image features into the multi-layer perceptron to obtain the line-of-sight related features; input the line-of-sight related features into the fully connected layer to obtain the line-of-sight prediction values.

[0037] Exemplarily, determining the target loss value based on the line-of-sight image features, the line-of-sight related features, the line-of-sight prediction values, the facial image features, the line-of-sight unrelated features, and the line-of-sight label values may include, but are not limited to: determining a first loss value based on the line-of-sight image features and the facial image features; determining a second loss value based on the line-of-sight related features and the line-of-sight unrelated features; determining a third loss value based on the line-of-sight prediction values and the line-of-sight label values; determining the target loss value based on the first loss value, the second loss value, and the third loss value.

[0038] Exemplarily, determining the first loss value based on the line-of-sight image features and the facial image features includes, but is not limited to: determining the first loss value using the following formula: where represents the first loss value, f represents the line-of-sight image features, and f v represents the facial image features.

[0039] Exemplarily, determining the second loss value based on the line-of-sight related features and the line-of-sight unrelated features may include, but is not limited to: determining the second loss value using the following formula: where represents the second loss value, f represents the line-of-sight image features, f re represents the line-of-sight related features, represents the line-of-sight unrelated features corresponding to the kth line-of-sight unrelated category, and the value range of k is 1-K; among them, the line-of-sight unrelated description text includes text prompts corresponding to K line-of-sight unrelated categories, and the line-of-sight unrelated features include line-of-sight unrelated features corresponding to K line-of-sight unrelated categories.

[0040] Exemplarily, determining the third loss value based on the line-of-sight prediction values and the line-of-sight label values may include, but is not limited to: determining the third loss value using the following formula: where Represents the third loss value, represents the gaze prediction value, and g represents the gaze label value.

[0041] Exemplarily, before inputting the gaze-independent description text into the text encoder to obtain the gaze-independent feature, it may further include: obtaining a trained target text generation network, inputting the sample image into the target text generation network to obtain the sample context corresponding to the sample image; for K gaze-independent categories supported by the text encoder, generating text prompts corresponding to each gaze-independent category based on the sample context and each gaze-independent category; generating the gaze-independent description text based on the K text prompts corresponding to the K gaze-independent categories.

[0042] Exemplarily, obtaining a trained target text generation network may include, but is not limited to: obtaining an initial text generation network to be trained; wherein, the initial text generation network includes a 3DMM (3D Morphable Model) network, a meta-network, and multiple personalized parameters. Inputting the sample facial image into the 3DMM network to obtain the personalized feature; inputting the personalized feature into the meta-network to obtain the facial token; generating multiple personalized contexts based on the facial token and the multiple personalized parameters; selecting the gaze-independent category corresponding to the sample facial image from the K gaze-independent categories supported by the text encoder, and generating multiple text prompts based on the multiple personalized contexts and the gaze-independent category; inputting the multiple text prompts into the text encoder to obtain the gaze-independent feature. Inputting the sample facial image into the image encoder to obtain the facial image feature. Determining a reference loss value based on the gaze-independent feature and the facial image feature, and adjusting the meta-network and the multiple personalized parameters based on the reference loss value to obtain the trained target text generation network.

[0043] Exemplarily, determining a reference loss value based on the gaze-independent feature and the facial image feature may include, but is not limited to: the reference loss value can be determined by the following formula: where L represents the reference loss value, f v represents the facial image feature, and represents the gaze-independent feature, τ represents the obtained hyperparameter, which can be configured according to experience, sim can represent calculating similarity, and exp can represent the exponential function. Wherein, when generating multiple text prompts based on the multiple personalized contexts and the gaze-independent category, for each personalized context, generating a pair of text prompts with opposite semantics based on the personalized context and the gaze-independent category, and after inputting the pair of text prompts into the text encoder, the text encoder can output gaze-independent features with opposite semantics and gaze-independent features

[0044] As can be seen from the above technical solutions, in the embodiments of the present application, the view-independent description text can be input to the text encoder to obtain view-independent features. The view-independent description text is used to describe the interference information unrelated to the view in the sample image. Then, based on the view-independent features, the network parameters of the initial view estimation model are adjusted to obtain the target view estimation model, so as to train an accurate and reliable target view estimation model based on the view-independent features. When performing view estimation based on the target view estimation model, the interference of view-independent factors can be reduced, the influence of view-independent factors on the view estimation performance can be alleviated, the accuracy of the view estimation result can be improved, an accurate view estimation result can be obtained, and the performance of the target view estimation model can be enhanced. In view of the fact that the facial features of different users are different, the facial personalized features of different users can be mapped to the text space, the accuracy of describing the face in the context can be improved, and then the accuracy of describing the current individual by view-independent features can be enhanced. Finally, the robustness of the target view estimation model can be enhanced in the view estimation generalization framework.

[0045] The view estimation method of the embodiments of the present application will be described below in combination with specific application scenarios.

[0046] View estimation is a technology related to the eyeball, eye movement, view, etc. It can take a facial image as the processing object and estimate the three-dimensional visual fixation direction. For example, in view estimation, the view direction of the user is estimated based on the facial image, and the fixation point coordinates intersecting with the view direction are obtained by combining the region coordinates.

[0047] However, in addition to including effective human eye region information, the facial image also includes a large number of view-independent features, such as identity features (such as face shape, eyebrows, mouth, nose, etc.), expression changes (such as smiling, crying, laughing, etc.), texture attributes (such as the color and style of hair and beard), and other features (such as gender and age, etc.). These view-independent features damage the generalization performance of view estimation, that is, the accuracy of the view estimation result drops significantly, so that an accurate view estimation result cannot be obtained. In addition to the above facial features, view estimation is also affected by external factors, such as wearing glasses, hats, helmets, and makeup, and view estimation is also affected by image quality, such as sensor noise, motion blur, and environment.

[0048] In view of the above findings, an embodiment of the present application proposes a view estimation method, which may involve the training process of the target view estimation model and the view estimation process based on the target view estimation model. When performing view estimation based on the target view estimation model, the interference of view-independent factors can be reduced, the influence of view-independent factors on the view estimation performance can be alleviated, the accuracy of the view estimation result can be improved, an accurate view estimation result can be obtained, and the performance of the target view estimation model can be enhanced. The "training process of the target view estimation model" and the "view estimation process based on the target view estimation model" will be described below.

[0049] First, the training process of the target line-of-sight estimation model. Refer to Figure 2 As shown, it is a schematic diagram of the training process of the target line-of-sight estimation model. In order to train the target line-of-sight estimation model, this process may include:

[0050] Step 201, obtain an initial line-of-sight estimation model to be trained, a trained image encoder, and a text encoder. The initial line-of-sight estimation model may include a line-of-sight feature extractor, a multi-layer perceptron, and a fully connected layer.

[0051] Exemplarily, refer to Figure 3 As shown, it is a training schematic diagram of the target line-of-sight estimation model. An initial line-of-sight estimation model can be obtained. The initial line-of-sight estimation model may include a line-of-sight feature extractor E (which can also be referred to as a line-of-sight encoder E), a multi-layer perceptron M, and a fully connected layer F. The line-of-sight feature extractor E is learnable, the multi-layer perceptron M is learnable, and the fully connected layer F is learnable. That is, during the training process of the target line-of-sight estimation model, the network parameters of the line-of-sight feature extractor E can be adjusted, the network parameters of the multi-layer perceptron M can be adjusted, and the network parameters of the fully connected layer F can be adjusted.

[0052] The line-of-sight feature extractor E is a network layer for extracting image features. It can be a deep learning network or a neural network, etc. The network structure of the line-of-sight feature extractor E is not limited. The multi-layer perceptron M is a binary classification linear classification model. The basic principle is to calculate the linear combination of input features and then use an activation function to perform binary classification on the result. It can be a deep learning network or a neural network, etc. The network structure of the multi-layer perceptron M is not limited. The fully connected layer F is such that each node is connected to all nodes in the previous layer, and is used to synthesize the previously extracted features, having the characteristic of full connection. It can be a deep learning network or a neural network, etc. The network structure of the fully connected layer F is not limited.

[0053] Exemplarily, refer to Figure 3 As shown, an image encoder C v and a text encoder C t can be obtained. The image encoder C v is not learnable, and the text encoder C t is not learnable. That is, during the training process of the target line-of-sight estimation model, the network parameters of the image encoder C v are not adjusted, and the network parameters of the text encoder C t are not adjusted. That is to say, the image encoder C v and the text encoder C t are already pre-trained image encoder C v and text encoder C t, no longer train the image encoder C v and the text encoder C t In summary, it is necessary to freeze the image encoder C v and the text encoder C t Freezing means that during training, the network parameters of the image encoder C v and the text encoder C t are fixed so that they do not participate in gradient updates.

[0054] The image encoder C v and the text encoder C t can be the image encoder and text encoder of a vision-language model. For example, a vision-language model can process images and natural language texts simultaneously, learn the correspondence between vision and language from a large number of image-text pairs, and be used for various visual recognition tasks such as image classification, object detection, semantic segmentation, etc. A vision-language model consists of three key elements: an image encoder, a text encoder, and a strategy for fusing the information of the two encoders. The pre-training methods of vision-language models can be classified according to different objectives, such as contrastive learning, generative tasks, and alignment tasks. The transfer methods and knowledge distillation methods of vision-language models are used to improve the performance and generalization ability of the model on downstream tasks.

[0055] In summary, the image encoder in the vision-language model can be used as the image encoder C of this embodiment v , and the text encoder in the vision-language model can be used as the text encoder C of this embodiment t .

[0056] Referring to Figure 3 as shown, the image encoder C v can be a CLIP image encoder, and the text encoder C t can be a CLIP text encoder. For example, CLIP (Contrastive Language-Image Pre-Training, pre-training based on contrastive text-image pairs) is a joint training model that can process images and texts simultaneously, thus connecting visual and language information. CLIP can effectively learn visual information through natural language supervision without using a large amount of labeled data. The training process of CLIP can use contrastive learning techniques, enabling the model to learn the correlation between images and texts. CLIP can be applied to any visual classification benchmark test, only requiring the name of the visual category to be recognized to be provided.

[0057] Step 202: Input the sample image into the gaze feature extractor to obtain gaze image features.

[0058] Step 203: Input the sample image into the image encoder to obtain the facial image features.

[0059] Exemplarily, a training data set can be obtained. The training data set can include multiple training data (hereinafter, one training data will be taken as an example for illustration). For each training data, the training data can include a sample image and the corresponding line-of-sight label value of the sample image. Among them, the line-of-sight label value is used to represent the true value of the line-of-sight direction of the sample image and can be calibration data.

[0060] Exemplarily, the sample image can be input into the line-of-sight feature extractor E to obtain the line-of-sight image feature f. The line-of-sight image feature f is a feature used to reflect the line-of-sight direction. For example, after the sample image is input into the line-of-sight feature extractor E, the line-of-sight feature extractor E extracts the feature in the sample image that is used to reflect the line-of-sight direction, calls the feature used to reflect the line-of-sight direction the line-of-sight image feature f, and outputs the line-of-sight image feature f. The process of the line-of-sight feature extractor E extracting the line-of-sight image feature f is not limited in this embodiment.

[0061] Exemplarily, the sample image can be input into the image encoder C v to obtain the facial image feature f v , and the facial image feature f v is a feature used to reflect facial information, such as human eye region information, identity features (such as face shape, eyebrows, mouth, nose) information, expression changes (such as smiling, crying, laughing) information, texture attributes (such as hair, beard color, style) information, etc. For example, after the sample image is input into the image encoder C v , the image encoder C v extracts the feature in the sample image that is used to reflect facial information, calls the feature used to reflect facial information the facial image feature f v , and outputs the facial image feature f v . The process of the image encoder C v extracting the facial image feature f v is not limited in this embodiment.

[0062] Step 204: Determine the first loss value based on the line-of-sight image feature and the facial image feature.

[0063] Exemplarily, based on the line-of-sight image feature and the facial image feature, the first loss value can be determined using the following formula (1). Of course, formula (1) is just an example, and the determination method of this first loss value is not limited, as long as the first loss value is related to the line-of-sight image feature and the facial image feature.

[0064]

[0065] In formula (1), represents the first loss value, f represents the line-of-sight image feature, f v represents the facial image feature, ||f|| represents the norm of the line-of-sight image feature, ||f v || represents the norm of the facial image feature.

[0066] Since the adjustment direction of the initial line-of-sight estimation model is to make the first loss value smaller and smaller, the line-of-sight image feature f can be aligned to the feature space of the image encoder C v That is, by formula (1), f and f are brought closer v .

[0067] Step 205: Input the line-of-sight image feature into a multi-layer perceptron to obtain line-of-sight related features.

[0068] Exemplarily, after the line-of-sight feature extractor E outputs the line-of-sight image feature f, the line-of-sight image feature f can be input into the multi-layer perceptron M to obtain the line-of-sight related feature f re , the line-of-sight related feature f re is a feature related to the line-of-sight direction, that is, the line-of-sight unrelated features are separated from the line-of-sight image feature f to obtain the line-of-sight related feature f re .

[0069] For example, after the line-of-sight image feature f is input into the multi-layer perceptron M, the multi-layer perceptron M can obtain the features related to the line-of-sight direction in the line-of-sight image feature f, separate the line-of-sight unrelated features from the line-of-sight image feature f, and call the features related to the line-of-sight direction the line-of-sight related feature f re , and output the line-of-sight related feature f re . Regarding the process of the multi-layer perceptron M extracting the line-of-sight related feature f re in this embodiment, there is no limitation.

[0070] For example, the line-of-sight related feature f re = M(f), M(f) represents processing based on the line-of-sight image feature f, M represents the processing method of the multi-layer perceptron M, and is related to the network structure of the multi-layer perceptron M.

[0071] Step 206: Input the line-of-sight unrelated description text into a text encoder to obtain line-of-sight unrelated features. Among them, the line-of-sight unrelated description text is used to describe the interference information unrelated to the line-of-sight in the sample image.

[0072] Exemplarily, the line-of-sight unrelated description text can be input into the text encoder C tObtain view-independent features, which are used to reflect features independent of the viewing direction, such as identity features (e.g., face shape, eyebrows, mouth, nose) information, expression change (e.g., smiling, crying, laughing) information, texture attributes (e.g., hair, beard color, style) information, etc. For example, when inputting the view-independent description text to the text encoder C t After that, the text encoder C t Obtains features used to reflect features independent of the viewing direction based on the view-independent description text, calls the features used to reflect features independent of the viewing direction view-independent features, and outputs the view-independent features. Regarding the process of the text encoder C t For extracting the view-independent features, there is no limitation in this embodiment, as long as it can output the view-independent features.

[0073] Exemplarily, the text encoder C t Can support K view-independent categories, where K can be a positive integer. For example, the text encoder C t Can support view-independent categories such as "happy", "glasses", "dark", "hat", etc. There is no limitation on these view-independent categories, and they can be any category independent of the viewing direction.

[0074] For each view-independent category supported by the text encoder C t The view-independent description text includes a text prompt corresponding to this view-independent category, that is, the view-independent description text includes text prompts corresponding to K view-independent categories. For example, for the text prompt corresponding to the view-independent category "happy", the text prompt is used to describe the interference information related to the view-independent category "happy" in the sample image. For example, the text prompt can be like "An image of a face with{c k}", {c k} represents "happy". For the text prompt corresponding to the view-independent category "glasses", the text prompt is used to describe the interference information related to the view-independent category "glasses" in the sample image. For example, the text prompt can be like "An image of a face with{c k}", {c k} represents "glasses", and so on.

[0075] Obviously, for each view-independent category supported by the text encoder C t The corresponding text prompt for this view-independent category can be obtained. This text prompt can include the template context and this view-independent category, that is, An image of aface with represents the template context, {c k} represents this view-independent category.

[0076] Since the text encoder C tSupports K line-of-sight independent categories. Therefore, K text prompts corresponding to the K line-of-sight independent categories can be obtained, and the K text prompts can be combined to obtain the line-of-sight independent description text, that is, the line-of-sight independent description text includes the text prompts corresponding to the K line-of-sight independent categories. When inputting the line-of-sight independent description text into the text encoder C t After that, the text encoder C t Outputs the line-of-sight independent feature, and this line-of-sight independent feature can include the line-of-sight independent features corresponding to the K line-of-sight independent categories, and this line-of-sight independent feature can be denoted as

[0077] For example, {c k} is a line-of-sight independent category in the set of line-of-sight independent categories, that is, the set of line-of-sight independent categories When k is 1, {c k} represents the 1st line-of-sight independent category, Represents the line-of-sight independent feature corresponding to the 1st line-of-sight independent category, and this line-of-sight independent feature is obtained after the text encoder C t Processes the text prompt of the 1st line-of-sight independent category. When k is 2, {c k} represents the 2nd line-of-sight independent category, Represents the line-of-sight independent feature corresponding to the 2nd line-of-sight independent category, and this line-of-sight independent feature is obtained after the text encoder C t Processes the text prompt of the 2nd line-of-sight independent category, and so on. In this way, the text encoder C t Can output a total of K line-of-sight independent features.

[0078] Step 207: Determine the second loss value based on the line-of-sight related feature and the line-of-sight independent feature.

[0079] Exemplarily, based on the line-of-sight related feature and the line-of-sight independent feature, the second loss value can be determined by the following formula (2). Of course, formula (2) is just an example, and the determination method of this second loss value is not limited, as long as the second loss value is related to the line-of-sight related feature and the line-of-sight independent feature.

[0080]

[0081] In formula (2), Represents the second loss value, f represents the line-of-sight image feature, f re Represents the line-of-sight related feature, Represents the line-of-sight independent feature corresponding to the kth line-of-sight independent category, and the value range of k is 1 - K. ||f|| represents the norm of the line-of-sight image feature, Represents the norm of the line-of-sight independent feature corresponding to the kth line-of-sight independent category, ||f re|| represents the magnitude of the line-of-sight related feature.

[0082] In formula (2), when k = 1, represents the line-of-sight independent feature corresponding to the first line-of-sight independent category. When k = 2, represents the line-of-sight independent feature corresponding to the second line-of-sight independent category, and so on. When k = K, represents the line-of-sight independent feature corresponding to the Kth line-of-sight independent category. Obviously, the line-of-sight independent description text includes the text prompts corresponding to K line-of-sight independent categories, and the line-of-sight independent features include the line-of-sight independent features corresponding to K line-of-sight independent categories. That is, K in formula (2) can be obtained.

[0083] In formula (2), represents the similarity w between the line-of-sight image feature f and the line-of-sight independent feature corresponding to the kth line-of-sight independent category i.e., k Obviously, can be obtained. On this basis, can be substituted into formula (2), that is, keep unchanged. Or, can be normalized to obtain the normalized There is no limit to this normalization process, and is substituted into formula (2), that is, in formula (2) is replaced by the normalized

[0084] In formula (2), represents the line-of-sight related feature f re and the line-of-sight independent feature corresponding to the kth line-of-sight independent category with similarity w k’ i.e., Obviously, can be obtained. On this basis, can be substituted into formula (2), that is, keep unchanged. Or, can be normalized to obtain the normalized and the normalized is substituted into formula (2), that is,

[0085] Since the adjustment direction of the initial line-of-sight estimation model is to make the second loss value smaller and smaller, thus reducing the loss through formula (2). When reducing the loss, it can pull apart the line-of-sight independent feature and the line-of-sight related feature fre 。

[0086] Step 208: Input the gaze-related features into a fully connected layer to obtain a gaze prediction value.

[0087] Exemplarily, after the multi-layer perceptron M outputs the gaze-related feature f re the gaze-related feature f can be re input into the fully connected layer F to obtain a gaze prediction value The gaze prediction value represents the predicted value of the gaze direction. For example, after inputting the gaze-related feature f re into the fully connected layer F, the fully connected layer F (i.e., the linear fully connected layer) can perform mapping prediction based on the gaze-related feature f re to obtain a gaze prediction value and output the gaze prediction value The process of determining the gaze prediction value regarding the fully connected layer F is not limited in this embodiment.

[0088] Step 209: Determine a third loss value based on the gaze prediction value and the gaze label value.

[0089] Exemplarily, based on the gaze prediction value and the gaze label value, the third loss value can be determined using the following formula (3). Of course, formula (3) is just an example, and the determination method of this third loss value is not limited, as long as the third loss value is related to the gaze prediction value and the gaze label value.

[0090]

[0091] In formula (3), represents the third loss value, represents the gaze prediction value, and g represents the gaze label value, which is used to represent the true value of the gaze direction of the sample image.

[0092] Since the adjustment direction of the initial gaze estimation model is to make the third loss value smaller and smaller, the loss is reduced through formula (3), and when reducing the loss, the gaze prediction value and the gaze label value g can be approximated.

[0093] Step 210: Determine a target loss value based on the first loss value, the second loss value, and the third loss value. For example, the sum of the first loss value, the second loss value, and the third loss value can be used as the target loss value.

[0094] Step 211: Adjust the network parameters of the initial gaze estimation model (i.e., train the initial gaze estimation model) based on the target loss value to obtain a trained target gaze estimation model.

[0095] Exemplarily, the network parameters of the initial line of sight estimation model (such as a line of sight feature extractor, a multi-layer perceptron, and a fully connected layer, etc.) can be adjusted based on the target loss value to obtain an adjusted line of sight estimation model. The goal of adjusting the network parameters is to make the target loss value smaller and smaller, such as by using a gradient descent method to adjust the network parameters of the initial line of sight estimation model, and there is no restriction on this adjustment process.

[0096] After obtaining the adjusted line of sight estimation model, it can be determined whether the adjusted line of sight estimation model has converged. For example, if the target loss value is less than the preset threshold, the adjusted line of sight estimation model has converged; if the target loss value is not less than the preset threshold, the adjusted line of sight estimation model has not converged. For another example, if the number of iterations of the line of sight estimation model reaches the number threshold, the adjusted line of sight estimation model has converged; if the number of iterations of the line of sight estimation model does not reach the number threshold, the adjusted line of sight estimation model has not converged. For another example, if the iteration duration of the line of sight estimation model reaches the duration threshold, the adjusted line of sight estimation model has converged; if the iteration duration of the line of sight estimation model does not reach the duration threshold, the adjusted line of sight estimation model has not converged. Of course, the above are just a few examples of the convergence conditions of the adjusted line of sight estimation model, and there is no limitation on this.

[0097] If the adjusted sight line estimation model has converged, the adjusted sight line estimation model is used as the trained target sight line estimation model, and the training process of the target sight line estimation model is now completed.

[0098] If the adjusted sight line estimation model has not converged, the adjusted sight line estimation model is used as the initial sight line estimation model, and the process returns to step 202 to repeat the above steps until the adjusted sight line estimation model has converged.

[0099] In a possible implementation, for the training process of the target sight line estimation model, see Figure 4 As shown. The facial image (i.e., sample image) can be input into the sight line feature extractor to obtain sight line image features (also called sight line mixed features). The sight line image features can be input into the multi-layer perceptron to obtain sight line related features. The sight line related features can be input into the fully connected layer to obtain the sight line prediction value.

[0100] The facial image and the sight-independent description text (i.e., personalized prompt information) can be input into the CLIP model to obtain facial image features and sight-independent features. For example, the facial image can be input into the image encoder of the CLIP model to obtain facial image features, and the sight-independent description text can be input into the text encoder of the CLIP model to obtain sight-independent features (sight-independent features are personalized sight-independent text features).

[0101] The line-of-sight image features and facial image features can be pulled closer through a loss function, and the line-of-sight related features and line-of-sight unrelated features can be pulled farther apart through the loss function, and then a target line-of-sight estimation model can be trained.

[0102] In a possible implementation manner, for step 206, in order to obtain the text prompt corresponding to the line-of-sight unrelated category, the template context and the line-of-sight unrelated category are combined to obtain the text prompt, and the template context is a pre-configured template, such as "An image of a face with", etc. Different from this, considering that there are differences in the facial features of different users, the facial change amplitudes of different users under the same expression are different, and there are also deviations between different ages and genders. Therefore, personalized context can also be provided instead of using the template context, that is, using the personalized context to replace the template context "An image of a face with", that is to say, the template context can be transformed into a learnable and personalized prompt (that is, the personalized context), so as to improve the accuracy of the line-of-sight unrelated features and the reliability of the target line-of-sight estimation model.

[0103] In order to obtain the personalized context, a target text generation network can be trained. After obtaining the target text generation network, the personalized context can be generated based on the target text generation network. Refer to Figure 5 As shown, it is a schematic diagram for training the target text generation network. This process can include:

[0104] Step 501, obtain the initial text generation network to be trained, the trained image encoder and text encoder. The initial text generation network can include a 3DMM network, a meta network and multiple personalized parameters.

[0105] Exemplarily, refer to Figure 6 As shown, it is a training schematic diagram of the target text generation network. The initial text generation network can be obtained. The initial text generation network can include a 3DMM network, a meta network and multiple personalized parameters. The 3DMM network is not learnable, that is, the network parameters of the 3DMM network are not adjusted during the training process. The meta network is learnable, that is, the network parameters of the meta network are adjusted during the training process. The multiple personalized parameters are learnable, that is, the personalized parameters are adjusted during the training process.

[0106] The 3DMM network can be a deep learning network or a neural network, etc., and there is no restriction on the network structure of this network. The 3DMM network is a statistical model used to describe the three-dimensional shape and texture of the face, and can represent the face as a parameterized model, including shape parameters and texture parameters. The shape parameters describe the geometric structure of the face, such as position, size and shape, while the texture parameters describe the appearance characteristics of the face, such as skin color and texture. By adjusting the values of these parameters, face models with different shapes and textures can be generated.

[0107] The meta-network can be a deep learning network or a neural network, etc., and there is no restriction on the network structure of this meta-network. For example, the meta-network can be a neural network, or other types of meta-networks.

[0108] The multiple personalized parameters can include personalized parameters u1, u2, …, u M , that is, M personalized parameters can be pre-configured, and the M personalized parameters need to be adjusted and optimized during the training process.

[0109] Exemplarily, referring to Figure 6 as shown, the image encoder C v and the text encoder C t can be obtained. The image encoder C v and the text encoder C t can refer to step 201, which will not be repeated here.

[0110] Step 502: Input the sample face image into the 3DMM network to obtain personalized features.

[0111] Step 503: Input the personalized features into the meta-network to obtain face tokens.

[0112] Step 504: Generate multiple personalized contexts based on the face tokens and multiple personalized parameters.

[0113] Exemplarily, a sample face image (it can be multiple sample face images, and hereinafter a single sample face image is taken as an example) can be obtained, and the sample face image is input into the 3DMM network to obtain personalized features (such as personalized identity features). Then, the personalized features are input into the meta-network, and the meta-network maps the personalized features to text features, that is, the face token π. Then, the face token is superimposed on the learnable personalized parameters (u1, u2, …, u M ) to obtain multiple personalized contexts.

[0114] For example, the face token can be superimposed on the personalized parameter u1 to obtain personalized context 1, the face token can be superimposed on the personalized parameter u2 to obtain personalized context 2, and so on. The face token can be superimposed on the personalized parameter uM to obtain a personalized context M.

[0115] Step 505: Select the line-of-sight independent category corresponding to the sample facial image from the K line-of-sight independent categories supported by the text encoder, and generate multiple text prompts based on the multiple personalized contexts and this line-of-sight independent category.

[0116] For example, the text encoder C t can support K line-of-sight independent categories, that is, in the line-of-sight independent category dataset, the facial images themselves have various attribute labels These attribute labels are also line-of-sight independent factors. For the introduction of the K line-of-sight independent categories, please refer to Step 206 and will not be elaborated here.

[0117] Select the line-of-sight independent category corresponding to the sample facial image from the K line-of-sight independent categories. For example, when there is information related to "happy" in the sample facial image, select the line-of-sight independent category "happy"; when there is information related to "glasses" in the sample facial image, select the line-of-sight independent category "glasses", and so on.

[0118] Generate a text prompt based on personalized context 1 and the line-of-sight independent category, generate a text prompt based on personalized context 2 and the line-of-sight independent category, and so on, generate a text prompt based on personalized context M and the line-of-sight independent category, that is, generate multiple text prompts based on the multiple personalized contexts and the line-of-sight independent category.

[0119] Exemplarily, when generating multiple text prompts based on the multiple personalized contexts and the line-of-sight independent category, for each personalized context, generate a pair of text prompts with opposite semantics based on this personalized context and the line-of-sight independent category. For example, when generating text prompts based on personalized context 1 and the line-of-sight independent category, generate a pair of text prompts with opposite semantics, such as the line-of-sight independent categories at the end of the text prompts are a pair of words with opposite semantics, such as "not happy" and "happy". That is, personalized context 1 + "not happy" and personalized context 1 + "happy" form a pair of text prompts with opposite semantics. Similarly, personalized context 2 + "not happy" and personalized context 2 + "happy" form a pair of text prompts with opposite semantics, and so on. Each personalized context corresponds to a pair of text prompts with opposite semantics.

[0120] Step 506: Input the multiple text prompts into the text encoder to obtain line-of-sight independent features.

[0121] For example, after inputting text prompt 1 (composed of personalized context 1 and a gaze-independent category, and text prompt 1 is a text prompt pair with opposite semantics) into the text encoder, the text encoder can output the gaze-independent features corresponding to text prompt 1 and gaze-independent features and the gaze-independent features and gaze-independent features are gaze-independent features with opposite semantics. After inputting text prompt 2 (composed of personalized context 2 and a gaze-independent category) into the text encoder, the text encoder can output the gaze-independent features corresponding to text prompt 2 and gaze-independent features And so on.

[0122] In summary, for each text prompt pair, after inputting the text prompt pair into the text encoder, the text encoder can output gaze-independent features with opposite semantics and gaze-independent features

[0123] Step 507: Input the sample facial image into the image encoder to obtain facial image features.

[0124] Step 508: Determine the reference loss value based on the gaze-independent features and the facial image features.

[0125] Exemplarily, based on the gaze-independent features and the facial image features, the reference loss value can be determined using the following formula (4). Of course, formula (4) is just an example, and there is no limitation on the way to determine this reference loss value, as long as the reference loss value is related to the gaze-independent features and the facial image features.

[0126]

[0127] In formula (4), L represents the reference loss value, f v represents the facial image features, and represent the gaze-independent features, τ represents the obtained hyperparameter, which can be configured according to experience, sim represents calculating similarity, such as calculating similarity, calculating similarity, etc., and exp represents the exponential function.

[0128] Obviously, for each text prompt (text prompt pair), the reference loss value corresponding to the text prompt can be determined based on the gaze-independent features and gaze-independent features corresponding to the text prompt. Then, sum up the reference loss values corresponding to all text prompts to obtain the final reference loss value.

[0129] Based on the loss function shown in formula (4), by optimizing the learnable parameters, the matching degree between the text features (i.e., gaze-independent features and ) and the image features (i.e., the facial image features f v ) can be increased.

[0130] Step 509: Adjust the meta-network and multiple personalized parameters based on the reference loss value to obtain the trained target text generation network. Thus, the training process of the target text generation network is completed.

[0131] For example, the network parameters of the meta-network can be adjusted based on the reference loss value, and the personalized parameters u1, u2, …, u M are adjusted to obtain the adjusted text generation network. After obtaining the adjusted text generation network, it can be determined whether the adjusted text generation network has converged. If so, the adjusted text generation network is used as the trained target text generation network. If not, the adjusted text generation network is used as the initial text generation network, and the above steps are repeated until the adjusted text generation network has converged.

[0132] Based on the trained target text generation network, in step 206, in order to obtain the text prompt corresponding to the gaze-independent category, the sample image can be input into the target text generation network to obtain the sample context corresponding to the sample image. For example, the sample image can be input into the 3DMM network to obtain the personalized features, the personalized features are input into the meta-network to obtain the facial tokens, and the personalized context is generated based on the facial tokens and the personalized parameters, and the personalized context is recorded as the sample context corresponding to the sample image.

[0133] On this basis, for the K gaze-independent categories supported by the text encoder, the text prompt corresponding to each gaze-independent category is generated based on the sample context and each gaze-independent category; the gaze-independent description text is generated based on the K text prompts corresponding to the K gaze-independent categories. For example, this process is similar to step 206, except that the template context in step 206 is replaced with the sample context, which will not be elaborated here.

[0134] In a possible implementation manner, for the training process of the target text generation network, reference can also be made to Figure 7 as shown. The sample facial image can be input into the 3DMM network to obtain the personalized features. The personalized features are input into the meta-network and encoded as facial tokens, the facial tokens are superimposed with the personalized parameters to generate the personalized context, and the personalized context is superimposed with the gaze-independent category to generate the text prompt (i.e., the text vector).

[0135] The text prompt is input into the text encoder to obtain view-independent features. The sample facial image is input into the image encoder to obtain facial image features. Based on the view-independent features and facial image features, the meta-network and personalized parameters are fine-tuned to maximize the matching degree between the view-independent features and facial image features.

[0136] Second, the gaze estimation process based on the target gaze estimation model. Refer to Figure 8 As shown, it is a schematic diagram of the gaze estimation process based on the target gaze estimation model, and this process may include:

[0137] Step 801: Input the image to be detected into the gaze feature extractor to obtain gaze image features.

[0138] Exemplarily, gaze estimation can be applied to scenarios such as games, VR, assisted driving (intelligent cockpit), screen applications (such as mobile phone unlocking, short video special effects), etc. In the game scenario, game interaction is performed through gaze estimation, and the image to be detected is an image collected for the game scenario. In the VR scenario, the gaze direction of the user is accurately estimated through the built-in camera of the helmet, so as to perform local fine rendering on the scene, and the image to be detected is an image collected by the built-in camera of the helmet. In the assisted driving scenario, by analyzing the gaze direction of the user, it is detected whether the driver is fatigued or distracted, improving safety, and the image to be detected is an image collected by in-vehicle equipment. In the screen application scenario, interaction is performed through gaze estimation, such as obtaining the user's concentration through gaze analysis, scoring a class, or a product, or a video, and estimating the user's interest, etc., and the image to be detected is an image collected by a smart phone. Of course, the above are just a few examples.

[0139] Exemplarily, after obtaining the image to be detected, the image to be detected can be input into the gaze feature extractor (i.e., the gaze feature extractor of the target gaze estimation model) to obtain gaze image features.

[0140] Step 802: Input the gaze image features into the multi-layer perceptron to obtain gaze-related features.

[0141] Step 803: Input the gaze-related features into the fully connected layer to obtain a gaze prediction value, and the gaze prediction value is the result of gaze estimation based on the image to be detected, and is used to represent the gaze direction prediction value.

[0142] As can be seen from the above technical solutions, in the embodiments of the present application, an accurate and reliable target gaze estimation model is trained based on gaze-independent features. When performing gaze estimation based on the target gaze estimation model, it is possible to reduce the interference of gaze-independent factors, mitigate the impact of gaze-independent factors on gaze estimation performance, improve the accuracy of gaze estimation results, obtain accurate gaze estimation results, and enhance the performance of the target gaze estimation model. In view of the fact that the facial features of different users are different, the facial personalized features of different users can be mapped to the text space, improving the accuracy of describing the face in the context, and further improving the accuracy of describing the current individual with gaze-independent features. Finally, the robustness of the target gaze estimation model is enhanced in the gaze estimation generalization framework. It can enhance the domain generalization ability of the target gaze estimation model and solve the robustness problem of the gaze estimation model in cross-scenario applications. By leveraging the advantages of both visual and language modalities, it is convenient to quickly construct various gaze-independent features, providing a flexible and easily expandable plug-and-play solution to improve the model performance at low cost and high efficiency. The purpose of domain generalization is to train a model from multiple datasets with different distributions (referred to as source domains) so that it can perform well on a new unseen dataset (referred to as the target domain).

[0143] Based on the same inventive concept as the above method, an embodiment of the present application proposes a gaze estimation device. Refer to Figure 9A As shown, it is a schematic structural diagram of the device. The device may include: an acquisition module 911, configured to acquire an initial gaze estimation model to be trained, input a sample image to the initial gaze estimation model to obtain a gaze image feature, a gaze-related feature, and a gaze prediction value; input the sample image to an acquired image encoder to obtain a facial image feature; input a gaze-independent description text to an acquired text encoder to obtain a gaze-independent feature; wherein the gaze-independent description text is used to describe the interference information unrelated to gaze in the sample image; a determination module 912, configured to determine a target loss value based on the gaze image feature, the gaze-related feature, the gaze prediction value, the facial image feature, the gaze-independent feature, and a gaze label value; a processing module 913, configured to adjust the network parameters of the initial gaze estimation model based on the target loss value to obtain a trained target gaze estimation model; wherein the target gaze estimation model is used to perform gaze estimation based on an image to be detected.

[0144] Exemplarily, the initial line-of-sight estimation model includes a line-of-sight feature extractor, a multi-layer perceptron, and a fully connected layer. When the acquisition module 911 inputs a sample image into the initial line-of-sight estimation model to obtain a line-of-sight image feature, a line-of-sight related feature, and a line-of-sight prediction value, it is specifically used for: inputting the sample image into the line-of-sight feature extractor to obtain a line-of-sight image feature; inputting the line-of-sight image feature into the multi-layer perceptron to obtain a line-of-sight related feature; and inputting the line-of-sight related feature into the fully connected layer to obtain a line-of-sight prediction value.

[0145] Exemplarily, when the determination module 912 determines a target loss value based on the line-of-sight image feature, the line-of-sight related feature, the line-of-sight prediction value, the facial image feature, the line-of-sight unrelated feature, and the line-of-sight label value, it is specifically used for: determining a first loss value based on the line-of-sight image feature and the facial image feature; determining a second loss value based on the line-of-sight related feature and the line-of-sight unrelated feature; determining a third loss value based on the line-of-sight prediction value and the line-of-sight label value; and determining the target loss value based on the first loss value, the second loss value, and the third loss value.

[0146] Exemplarily, when the determination module 912 determines a first loss value based on the line-of-sight image feature and the facial image feature, it is specifically used for: determining the first loss value using the following formula: where represents the first loss value, f represents the line-of-sight image feature, and f v represents the facial image feature; where, when the determination module 912 determines a second loss value based on the line-of-sight related feature and the line-of-sight unrelated feature, it is specifically used for: determining the second loss value using the following formula: where represents the second loss value, f represents the line-of-sight image feature, and f re represents the line-of-sight related feature, represents the line-of-sight unrelated feature corresponding to the k-th line-of-sight unrelated category, and the value range of k is 1-K; where the line-of-sight unrelated description text includes text prompts corresponding to K line-of-sight unrelated categories, and the line-of-sight unrelated features include the line-of-sight unrelated features corresponding to the K line-of-sight unrelated categories; where, when the determination module 912 determines a third loss value based on the line-of-sight prediction value and the line-of-sight label value, it is specifically used for: determining the third loss value using the following formula: where represents the third loss value, represents the line-of-sight prediction value, and g represents the line-of-sight label value.

[0147] Exemplarily, the obtaining module 911 is further configured to, before inputting the line-of-sight-independent description text to the text encoder to obtain the line-of-sight-independent feature, obtain a trained target text generation network, input the sample image to the target text generation network to obtain the sample context corresponding to the sample image; for K line-of-sight-independent categories supported by the text encoder, generate text prompts corresponding to each line-of-sight-independent category based on the sample context and each line-of-sight-independent category; and generate the line-of-sight-independent description text corresponding to the sample image based on the K text prompts corresponding to the K line-of-sight-independent categories.

[0148] Exemplarily, when the obtaining module 911 obtains the trained target text generation network, it is specifically configured to: obtain an initial text generation network to be trained; the initial text generation network includes a 3DMM network, a meta-network, and multiple personalized parameters; input the sample facial image to the 3DMM network to obtain personalized features; input the personalized features to the meta-network to obtain facial tokens; generate multiple personalized contexts based on the facial tokens and the multiple personalized parameters; select the line-of-sight-independent category corresponding to the sample facial image from the K line-of-sight-independent categories supported by the text encoder, and generate multiple text prompts based on the multiple personalized contexts and the line-of-sight-independent category; input the multiple text prompts to the text encoder to obtain line-of-sight-independent features; input the sample facial image to the image encoder to obtain facial image features; determine a reference loss value based on the line-of-sight-independent features and the facial image features, and adjust the meta-network and the multiple personalized parameters based on the reference loss value to obtain the trained target text generation network.

[0149] Exemplarily, when the obtaining module 911 determines the reference loss value based on the line-of-sight-independent features and the facial image features, it is specifically configured to: determine the reference loss value using the following formula: where p k represents the reference loss value, f v represents the facial image feature, and represent the line-of-sight-independent features, τ represents a hyperparameter, sim represents calculating similarity, and exp represents an exponential function; wherein, when generating multiple text prompts based on the multiple personalized contexts and the line-of-sight-independent category, for each personalized context, generate a pair of text prompts with opposite semantics based on the personalized context and the line-of-sight-independent category, and after inputting the pair of text prompts to the text encoder, the text encoder outputs line-of-sight-independent features and line-of-sight-independent features

[0150] Based on the same inventive concept as the above method, an electronic device is proposed in an embodiment of the present application. Refer to Figure 9BAs shown, the electronic device includes: a processor 921 and a machine-readable storage medium 922, where the machine-readable storage medium 922 stores machine-executable instructions that can be executed by the processor 921; the processor 921 is used to execute the machine-executable instructions to implement the gaze estimation method disclosed in the above examples of the present application.

[0151] Based on the same application concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which a number of computer instructions are stored, and when the computer instructions are executed by a processor, the gaze estimation method disclosed in the above examples of the present application can be implemented.

[0152] Among them, the above machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0153] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0154] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0155] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows or blocks Figure 1 or multiple flows and / or blocks.

[0157] Moreover, these computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in one or more flows Figure 1 or multiple flows and / or blocks Figure 1 or multiple blocks.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows Figure 1 or multiple flows and / or blocks Figure 1 or multiple blocks.

[0159] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for gaze estimation, characterized in that, The method includes: Obtain an initial gaze estimation model to be trained, input a sample image into the initial gaze estimation model, and obtain gaze image features, gaze-related features, and a gaze prediction value; Input the sample image into an image encoder to obtain facial image features; Input gaze-unrelated description text into a text encoder to obtain gaze-unrelated features; wherein, the gaze-unrelated description text is used to describe interference information in the sample image that is unrelated to the gaze; Determine a target loss value based on the gaze image features, the gaze-related features, the gaze prediction value, the facial image features, the gaze-unrelated features, and a gaze label value; Adjust the network parameters of the initial gaze estimation model based on the target loss value to obtain a trained target gaze estimation model, and the target gaze estimation model is used to perform gaze estimation based on an image to be detected.

2. The method according to claim 1, wherein The initial gaze estimation model includes a gaze feature extractor, a multi-layer perceptron, and a fully connected layer. The step of inputting a sample image into the initial gaze estimation model to obtain gaze image features, gaze-related features, and a gaze prediction value includes: Input the sample image into the gaze feature extractor to obtain the gaze image features; Input the gaze image features into the multi-layer perceptron to obtain the gaze-related features; Input the gaze-related features into the fully connected layer to obtain the gaze prediction value.

3. The method according to claim 1 or 2, wherein The step of determining a target loss value based on the gaze image features, the gaze-related features, the gaze prediction value, the facial image features, the gaze-unrelated features, and a gaze label value includes: Determine a first loss value based on the gaze image features and the facial image features; Determine a second loss value based on the gaze-related features and the gaze-unrelated features; Determine a third loss value based on the gaze prediction value and the gaze label value; Determine the target loss value based on the first loss value, the second loss value, and the third loss value.

4. The method according to claim 3, wherein Determining a first loss value based on the line-of-sight image features and the facial image features includes: determining the first loss value using the following formula: wherein represents the first loss value, f represents the line-of-sight image features, and f v represents the facial image features; Determining a second loss value based on the line-of-sight related features and the line-of-sight independent features includes: determining the second loss value using the following formula: Wherein, represents the second loss value, f represents the line-of-sight image feature, f re represents the line-of-sight related feature, represents the line-of-sight independent feature corresponding to the k-th line-of-sight independent category, and the value range of k is 1-K; wherein, the line-of-sight independent description text includes text prompts corresponding to K line-of-sight independent categories, and the line-of-sight independent features include line-of-sight independent features corresponding to K line-of-sight independent categories; Determining a third loss value based on the line-of-sight prediction value and the line-of-sight label value, including: determining the third loss value using the following formula: Wherein, represents the third loss value, represents the line-of-sight prediction value, and g represents the line-of-sight label value.

5. The method according to claim 1, wherein Before inputting the gaze-unrelated description text into the text encoder to obtain gaze-unrelated features, the method further includes: Obtain a trained target text generation network, input the sample image into the target text generation network, and obtain a sample context corresponding to the sample image; For K gaze-unrelated categories supported by the text encoder, generate a text prompt corresponding to each gaze-unrelated category based on the sample context and each gaze-unrelated category; Generate gaze-unrelated description text based on the K text prompts corresponding to the K gaze-unrelated categories.

6. The method according to claim 5, wherein The step of obtaining a trained target text generation network includes: Obtain an initial text generation network to be trained; wherein, the initial text generation network includes a 3D facial statistical model (3DMM) network, a meta-network, and multiple personalized parameters; Input the sample facial image into the 3DMM network to obtain personalized features; input the personalized features into the meta-network to obtain facial tokens; generate multiple personalized contexts based on the facial tokens and multiple personalized parameters; select the view-independent category corresponding to the sample facial image from the K view-independent categories supported by the text encoder, and generate multiple text prompts based on the multiple personalized contexts and the view-independent category; input the multiple text prompts into the text encoder to obtain view-independent features; Input the sample facial image into the image encoder to obtain facial image features; Determine a reference loss value based on the view-independent features and the facial image features, and adjust the meta-network and multiple personalized parameters based on the reference loss value to obtain a trained target text generation network.

7. The method according to claim 6, wherein, Determining the reference loss value based on the view-independent feature and the facial image feature includes: determining the reference loss value by using the following formula: where L represents the reference loss value, f v represents the facial image feature, and represents the gaze-independent feature, τ represents the obtained hyperparameter, sim represents calculating similarity, and exp represents the exponential function; Among them, when generating a plurality of text prompts based on the plurality of personalized contexts and the line-of-sight independent category, for each personalized context, a pair of text prompts with opposite semantics is generated based on the personalized context and the line-of-sight independent category. After inputting the pair of text prompts into the text encoder, the text encoder outputs line-of-sight independent features with opposite semantics and line-of-sight independent features 8. A line-of-sight estimation device, characterized in that, The device includes: An acquisition module, configured to acquire an initial gaze estimation model to be trained, input a sample image into the initial gaze estimation model to obtain gaze image features, gaze-related features, and a gaze prediction value; input the sample image into an acquired image encoder to obtain facial image features; input a view-independent description text into an acquired text encoder to obtain view-independent features; wherein, the view-independent description text is used to describe interference information unrelated to gaze in the sample image; A determination module, configured to determine a target loss value based on the gaze image features, the gaze-related features, the gaze prediction value, the facial image features, the view-independent features, and a gaze label value; A processing module, configured to adjust network parameters of the initial gaze estimation model based on the target loss value to obtain a trained target gaze estimation model; wherein, the target gaze estimation model is used to perform gaze estimation based on an image to be detected.

9. The device according to claim 8, Wherein, Wherein, The initial gaze estimation model includes a gaze feature extractor, a multi-layer perceptron, and a fully connected layer. When the acquisition module inputs the sample image into the initial gaze estimation model to obtain gaze image features, gaze-related features, and a gaze prediction value, it is specifically configured to: input the sample image into the gaze feature extractor to obtain gaze image features; input the gaze image features into the multi-layer perceptron to obtain gaze-related features; input the gaze-related features into the fully connected layer to obtain a gaze prediction value; Wherein, when the determination module determines the target loss value based on the gaze image features, the gaze-related features, the gaze prediction value, the facial image features, the view-independent features, and the gaze label value, it is specifically configured to: determine a first loss value based on the gaze image features and the facial image features; determine a second loss value based on the gaze-related features and the view-independent features; determine a third loss value based on the gaze prediction value and the gaze label value; determine the target loss value based on the first loss value, the second loss value, and the third loss value; Among them, when the determination module determines the first loss value based on the line-of-sight image feature and the facial image feature, it is specifically used to: determine the first loss value by using the following formula: Among them, represents the first loss value, f represents the line-of-sight image feature, and f v represents the facial image feature; among them, when the determination module determines the second loss value based on the line-of-sight related feature and the line-of-sight unrelated feature, it is specifically used to: determine the second loss value by using the following formula: Among them, represents the second loss value, f represents the line-of-sight image feature, and f re represents the line-of-sight related feature, represents the line-of-sight unrelated feature corresponding to the k-th line-of-sight unrelated category, and the value range of k is 1-K; among them, the line-of-sight unrelated description text includes text prompts corresponding to K line-of-sight unrelated categories, and the line-of-sight unrelated features include line-of-sight unrelated features corresponding to the K line-of-sight unrelated categories; among them, when the determination module determines the third loss value based on the line-of-sight prediction value and the line-of-sight label value, it is specifically used to: determine the third loss value by using the following formula: Among them, represents the third loss value, represents the line-of-sight prediction value, and g represents the line-of-sight label value; Wherein, the obtaining module is further configured to, before inputting the line-of-sight independent description text to the text encoder to obtain the line-of-sight independent feature, obtain a trained target text generation network, input the sample image to the target text generation network to obtain the sample context corresponding to the sample image; for K line-of-sight independent categories supported by the text encoder, generate text prompts corresponding to each line-of-sight independent category based on the sample context and each line-of-sight independent category; generate the line-of-sight independent description text corresponding to the sample image based on the K text prompts corresponding to the K line-of-sight independent categories; Wherein, when the obtaining module obtains the trained target text generation network, it is specifically configured to: obtain an initial text generation network to be trained; wherein, the initial text generation network includes a 3DMM network, a meta-network and a plurality of personalized parameters; input the sample facial image to the 3DMM network to obtain personalized features; input the personalized features to the meta-network to obtain facial tokens; generate a plurality of personalized contexts based on the facial tokens and the plurality of personalized parameters; select the line-of-sight independent category corresponding to the sample facial image from the K line-of-sight independent categories supported by the text encoder, and generate a plurality of text prompts based on the plurality of personalized contexts and the line-of-sight independent category; input the plurality of text prompts to the text encoder to obtain line-of-sight independent features; input the sample facial image to the image encoder to obtain facial image features; determine a reference loss value based on the line-of-sight independent features and the facial image features, and adjust the meta-network and the plurality of personalized parameters based on the reference loss value to obtain the trained target text generation network; Wherein, when determining the reference loss value based on the line-of-sight independent feature and the facial image feature, the obtaining module is specifically configured to: determine the reference loss value by using the following formula: Wherein, L represents the reference loss value, and f v represents the facial image feature, and represents the line-of-sight independent feature, τ represents a hyperparameter, sim represents calculating similarity, and exp represents an exponential function; wherein, when generating multiple text prompts based on multiple personalized contexts and the line-of-sight independent category, for each personalized context, a text prompt pair with opposite semantics is generated based on the personalized context and the line-of-sight independent category. After inputting the text prompt pair into the text encoder, the text encoder outputs line-of-sight independent features with opposite semantics and line-of-sight independent features 10. An electronic device, characterized in that, Comprising: A processor and a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1-7.

Images