Sight line estimation method, device, equipment and medium

Abstract

The embodiment of the invention relates to a sight line estimation method and device, equipment and a medium. The method comprises the following steps: acquiring a target image of a to-be-estimated sight line direction and a plurality of calibration images; wherein the calibration image carries sight line direction information, and the target image and the calibration image are eye images; inputting the target image and the plurality of calibration images into a preset sight line estimation model to obtain a sight line estimation direction of the target image output by the sight line estimation model; wherein the sight line estimation model is used for acquiring first related information among the plurality of calibration images, and acquiring second related information between the target image and the plurality of calibration images, and generating the sight line estimation direction of the target image based on the first related information, the second related information and the sight line direction information corresponding to the calibration image. The sight line estimation method provided by the embodiment of the invention is relatively convenient and rapid, and has higher reliability.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a line of sight estimation method, device, equipment and medium. Background Art

[0002] In many scenarios such as gaming, medical, and intelligent control, it is necessary to identify the line of sight of the human eye so as to adopt corresponding strategies based on the line of sight estimation results of the human eye. The inventor has found through research that the line of sight estimation method used in the related art is relatively inconvenient, and the reliability of the line of sight estimation results obtained is relatively poor. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a line of sight estimation method, device, equipment and medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: obtaining a target image whose line of sight direction is to be estimated and multiple calibration images; wherein the calibration image carries line of sight direction information, and the target image and the calibration image are both eye images; inputting the target image and the multiple calibration images into a preset line of sight estimation model to obtain the line of sight estimation direction of the target image output by the line of sight estimation model; wherein the line of sight estimation model is used to obtain first related information between the multiple calibration images, and to obtain second related information between the target image and the multiple calibration images, and to generate the line of sight estimation direction of the target image based on the first related information, the second related information and the line of sight direction information corresponding to the calibration image.

[0005] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, used to acquire a target image whose line of sight direction is to be estimated and a plurality of calibration images; wherein the calibration image carries line of sight direction information, and the target image and the calibration image are both eye images; a line of sight estimation module, used to input the target image and the plurality of calibration images into a preset line of sight estimation model, so as to acquire the line of sight estimation direction of the target image output by the line of sight estimation model; wherein the line of sight estimation model is used to acquire first related information between the plurality of calibration images, and to acquire second related information between the target image and the plurality of calibration images, and to generate the line of sight estimation direction of the target image based on the first related information, the second related information and the line of sight direction information corresponding to the calibration image.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the line of sight estimation method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method as described in the first aspect.

[0008] The above-mentioned technical solution provided by the embodiment of the present disclosure can use a preset line of sight estimation model to directly output the line of sight estimation direction of the target image, which is relatively convenient and quick. In addition, the line of sight estimation model can obtain the first relevant information between multiple calibration images, and obtain the second relevant information between the target image and the multiple calibration images, and further generate the line of sight estimation direction of the target image based on the first relevant information, the second relevant information and the line of sight direction information corresponding to the calibration image. The above-mentioned model analyzes the relationship between multiple calibration images and the relationship between the target image and the multiple calibration images, and directly estimates the line of sight estimation direction of the target image in combination with the line of sight direction information of the calibration image, which is more reliable.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Figure 1 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram of the structure of a sight line estimation model provided by an embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram of the structure of a sight line estimation model provided by an embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram of the structure of a self-attention unit provided in an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram of the structure of a sight line estimation device provided by an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0020] The inventor has found through research that the sight estimation method used in the related art is not good. Specifically, in the related art, the sight difference estimation model is first used to estimate the sight difference between the eye image to be tested and the calibration image, and then the sight direction of the eye image to be tested is determined based on the sight difference and the sight direction marked on the calibration image. However, this method cannot directly obtain the eye sight direction. It is necessary to estimate the sight difference first and then determine the eye sight direction. It is not only inconvenient, but also only based on the sight difference between the two images (that is, the eye image of the sight direction to be estimated and the calibration image with a known sight direction) to make an estimate, so as to determine the sight direction of the eye image to be tested. Even if multiple calibration images are used, the sight difference between each calibration image and the eye image of the sight direction to be estimated is obtained respectively, and then the sight direction marked on each calibration image is combined to obtain the sight estimation result of the eye image to be estimated corresponding to each calibration image, and then the multiple sight estimation results are weighted to obtain the final sight estimation result. However, the multiple calibration images are independent of each other, and there may be defective calibration images, so the reliability is relatively poor, and there is a problem of inaccurate sight estimation results. In addition, it should be noted that the above-mentioned defects existing in the relevant technology are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned defects and the solutions proposed in the embodiments of the present disclosure for the above-mentioned defects below should be recognized as the contributions made by the inventor to this application.

[0021] To improve at least one of the above problems, the embodiments of the present disclosure provide a line of sight estimation method, apparatus, device and medium, which are described in detail below.

[0022] Figure 1 The flowchart of a line of sight estimation method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a line of sight estimation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S104:

[0023] Step S102, obtaining a target image whose sight direction is to be estimated and a plurality of calibration images; wherein the calibration image carries sight direction information, and both the target image and the calibration image are eye images.

[0024] In actual applications, the target user may wear electronic devices such as VR glasses, smart glasses, eye trackers, or other devices that need to estimate the direction of the user's eye line of sight. In order to accurately realize eye tracking based on the physiological structure of the user's eyes, the user may be provided with a gaze point in advance, and the user may be asked to stare at the gaze point, and the corresponding eye image may be collected as a calibration image. Since the gaze point is pre-set, the line of sight direction corresponding to the calibration image may be directly determined, and a label carrying line of sight direction information may be attached to the calibration image, so as to assist the target image with unknown line of sight direction in estimating the line of sight direction in the future. It should be noted that before collecting the eye image, a collection prompt message may be initiated to the user, and the eye image may be collected after obtaining the user's authorization, and the user may be further informed of the purpose of the image collection in the prompt message. In addition, the embodiment of the present disclosure does not limit the specific number of the multiple calibration images obtained, and may be two or more calibration images, which may be flexibly set according to the actual situation.

[0025] Step S104, inputting the target image and the multiple calibration images into a preset sight line estimation model to obtain the sight line estimation direction of the target image output by the sight line estimation model; wherein the sight line estimation model is used to obtain first related information between the multiple calibration images, and obtain second related information between the target image and the multiple calibration images, and generate the sight line estimation direction of the target image based on the first related information, the second related information and the sight line direction information corresponding to the calibration images. The disclosed embodiment does not limit the structure of the sight line estimation model. Exemplarily, the sight line estimation model can be a differential model.

[0026] Among them, the first relevant information can indicate the correlation between multiple calibration images, and the second relevant information can indicate the correlation between the target image and each calibration image. In practical applications, the first relevant information and the second relevant information can both be represented in the form of feature vectors. The embodiment of the present disclosure does not limit the expression form of the first relevant information and the second relevant information. In practical applications, any method that can obtain the correlation between images can be applied, such as, the line of sight estimation model can use the attention mechanism to obtain the first relevant information and the second relevant information, which is not limited here. It can be understood that the attention mechanism can establish the correlation between multiple features extracted from the image (calibration image, target image) and learn the dependency between multiple features. By using the attention mechanism to analyze and process the image, it is helpful to obtain the relevant information between images more accurately.

[0027] In summary, the above model analyzes the correlation between multiple calibration images and the correlation between the target image and multiple calibration images, and directly estimates the line of sight estimation direction of the target image based on the line of sight direction information of the calibration image. It is more reliable and can directly predict the line of sight estimation direction of the target image, which is more convenient and quick.

[0028] In order to obtain the first relevant information and the second relevant information more accurately and reliably, illustratively, when obtaining the first relevant information between the multiple calibration images, the line of sight estimation model can obtain the first relevant information between the multiple calibration images based on the self-attention mechanism. The self-attention mechanism can reduce the dependence on external information and is better at capturing the internal correlation of data or features, thereby relatively reliably obtaining the association between the two calibration images.

[0029] When obtaining the second related information between the target image and the multiple calibration images, the line of sight estimation model can obtain the second related information between the target image and the multiple calibration images based on the cross attention mechanism. The cross attention mechanism can not only determine the association between the target image and each calibration image, but also help reduce the consumption of computing resources and improve the processing efficiency of image features.

[0030] To facilitate understanding of the line of sight estimation model provided by the embodiment of the present disclosure, the line of sight estimation model is explained in detail below.

[0031] First, the embodiment of the present disclosure provides an example of a sight line estimation model. In some implementation examples, see Figure 2 A structural diagram of a line of sight estimation model shown in FIG. 1 , including a feature extraction network, a self-attention network, a cross-attention network, and a prediction network.

[0032] The feature extraction network is used to extract the first image features corresponding to each of the multiple calibration images, and to extract the second image features corresponding to the target image. The feature extraction network can perform feature extraction on the multiple calibration images respectively to obtain the image features corresponding to each calibration image (i.e., the first image features), and the feature extraction network can perform feature extraction on the target image to obtain the image features corresponding to the target image (i.e., the second image features). It should be noted that in practical applications, the line of sight estimation model can only set up one feature extraction network, through which both the first image features and the second image features are extracted; the line of sight estimation model can also set up two feature extraction networks with shared weight parameters, one feature extraction network is used to extract the first image features, and the other feature extraction network is used to extract the second image features. The specific settings can be flexibly set and are not limited here.

[0033] The self-attention network is used to obtain the first relevant information between the multiple calibration images through the self-attention mechanism based on the first image feature. That is, the input of the self-attention network is the first image feature, and the output of the self-attention network is the first relevant information.

[0034] The cross attention network is used to obtain the second related information between the target image and the multiple calibration images through the cross attention mechanism based on the second image feature and the first related information. That is, the input of the cross attention network is the second image feature and the first related information, and the output of the cross attention network is the second related information.

[0035] The prediction network is used to generate the estimated sight line direction of the target image according to the first relevant information, the second relevant information, and the sight line direction information respectively carried by the multiple calibration images. That is, the input of the prediction network is the first relevant information, the second relevant information, and the sight line direction information respectively carried by the multiple calibration images, and the output of the prediction network is the estimated sight line direction of the target image.

[0036] In practical applications, a self-attention network includes one or more self-attention units, and multiple self-attention units are connected in sequence; a cross-attention network includes one or more cross-attention units, and multiple cross-attention units are connected in sequence. Figure 2 Based on Figure 3The structural diagram of a line of sight estimation model shown in the figure, the self-attention network includes N self-attention units, the cross-attention network includes M cross-attention units, N and M are both positive integers, N and M can be the same or different, and can be flexibly set according to needs. By setting multiple self-attention units and multiple cross-attention units, the association between images can be analyzed and processed layer by layer, so as to fully guarantee the accuracy and stability of the first relevant information and the second relevant information, so as to further guarantee the reliability of the line of sight estimation direction of the target image generated based on the line of sight direction information respectively carried by the first relevant information, the second relevant information and the multiple calibration images.

[0037] For ease of understanding, the self-attention unit and the cross-attention unit are explained in detail below.

[0038] The self-attention unit is used to obtain the first relevant feature vectors corresponding to the multiple calibration images through the self-attention mechanism based on the received input information. Among them, the input information corresponding to the first self-attention unit includes the first image feature, and the input information corresponding to the non-first self-attention unit includes the first relevant feature vector output by the previous self-attention unit; the first relevant information includes the first relevant feature vector output by the last self-attention unit. The structure of each self-attention unit is the same, and the main difference is that the input is different. The input of the first self-attention unit is the first image feature output by the feature extraction network, and the input of the remaining self-attention units is the output of the previous self-attention unit. The output of the last self-attention unit is the output of the self-attention network. It can be understood that multiple calibration images correspond to the same target image and are all used to assist in estimating the sight line direction of the target image. When there are defective calibration images among the multiple calibration images, by establishing the association between the multiple calibration images, the information of the defective calibration images can be corrected by using other calibration images. The first relevant feature vector corresponding to the calibration image obtained can more accurately reflect the image information of the calibration image, and also help to further ensure the reliability of the sight line estimation direction of the target image finally obtained.

[0039] Specifically, the embodiment of the present disclosure provides a specific implementation method in which the self-attention unit obtains the first relevant feature vectors corresponding to the multiple calibration images respectively through the self-attention mechanism based on the received input information, which can be performed with reference to the following steps a to c:

[0040] Step a, based on the input information received by the self-attention unit, obtain the first query vector and the first index vector corresponding to the multiple calibration images respectively. Exemplarily, the self-attention unit may include a first FC (fully connected) network layer, and the first FC network layer is used to convert the input information of the self-attention unit (first image feature or first related feature vector) into a first query vector and a first index vector. The number of first query vectors is equal to the number of calibration images, and the number of first index vectors is equal to the number of calibration images. Among them, the query vector is the Key vector, and the index vector is the Query vector, both of which are vectors required in the attention mechanism.

[0041] Step b, based on the first query vectors and the first index vectors corresponding to the multiple calibration images, obtain multiple first splicing vectors; wherein the first splicing vector is obtained by splicing the first query vector and the first index vector, and the first query vectors and / or first index vectors corresponding to different first splicing vectors are different. Exemplarily, the self-attention unit may include a splicing layer, and the first query vector and the first index vector are spliced ​​in pairs to obtain multiple first splicing vectors. Assuming that there are 3 calibration images in total, calibration image 1 corresponds to the first query vector Q1 and the first index vector K1, calibration image 2 corresponds to the first query vector Q2 and the first index vector K2, and calibration image 3 corresponds to the first query vector Q3 and the first index vector K3. The first splicing vector includes KiQj, i takes values ​​of 1 to n, j takes values ​​of 1 to n, and n is the number of calibration images. If the number of calibration images is 3, 9 first splicing vectors can be obtained by combining the two. Among them, Ki = FC_k (f_i), that is, the first index vector is obtained by processing the feature f_i corresponding to the i-th calibration image through the FC network layer; Qj = FC_q (f_j), that is, the first query vector is obtained by processing the feature corresponding to the j-th calibration image through the FC network layer. The FC network layer for obtaining the first index vector and the FC network layer for obtaining the first query vector can be the same or different, and can be flexibly set, and are collectively referred to as the first FC network layer.

[0042] Step c, based on the multiple first stitching vectors, obtain the first related feature vectors corresponding to the multiple calibration images. Based on the analysis of the multiple first stitching vectors, the correlation between any two calibration images can be determined one by one, and the first related feature vector corresponding to each calibration image can reflect the related information between the calibration image and other calibration images. Exemplarily, step c can be performed with reference to the following steps c1 to c3:

[0043] Step c1, according to the plurality of first splicing vectors, obtain the first correlation matrix corresponding to the plurality of calibration images. The self-attention unit may include a second FC network layer, and the second FC network layer is used to generate the first correlation matrix corresponding to the plurality of calibration images based on the plurality of first splicing vectors. The relevant information between any two calibration images may be used as a matrix element in the first correlation matrix, and the relevant information between all available calibration images may form the first correlation matrix. Exemplarily, the first correlation matrix may be a three-row and three-column matrix consisting of the relevant information (which may be expressed in the form of eigenvectors) corresponding to K1Q1, K1Q2, K1Q3, K2Q1, K2Q2, K2Q3, K3Q1, K3Q2, and K3Q3, respectively. The K corresponding to each row element is the same, and the Q corresponding to each column element is the same, such as, the relevant information corresponding to K1Q1, K1Q2, and K1Q3 is the first row, the relevant information corresponding to K2Q1, K2Q2, and K2Q3 is the second row, and the relevant information corresponding to K3Q1, K3Q2, and K3Q3 is the third row. The first correlation matrix diff_ij=FC_dij(cat(Ki,Qj)). Specifically, cat(Ki,Qj) indicates that Ki and Qj are concatenated to obtain the first concatenated vector, and FC_dij(cat(Ki,Qj)) indicates that the first correlation matrix diff_ij is obtained by performing correlation analysis on the first concatenated vector through the FC network layer.

[0044] Step c2, obtaining the first correlation probability corresponding to the first correlation matrix through the first preset algorithm. The first correlation probability can be used to indicate the correlation probability between each calibration image and other calibration images. In specific implementation, the self-attention unit uses the first preset algorithm to normalize the elements in the first correlation matrix to obtain the correlation probability between each calibration image and other calibration images. Exemplarily, the first preset algorithm includes a softmax function, then the first correlation probability P_ij=softmax(Diff_ij,-1).

[0045] Step c3, generating first correlation feature vectors corresponding to the plurality of calibration images respectively according to the first index vectors, the first correlation matrix and the first correlation probability corresponding to the plurality of calibration images respectively.

[0046] Exemplarily, the first index vector and the first correlation matrix may be fused to enhance the features of each calibration image to avoid the calibration image from losing its own features during the above processing. The above fusion process may be an addition process, such as adding Ki to the matrix element corresponding to Ki in the first correlation matrix. Afterwards, the fusion feature obtained based on the first index vector and the first correlation matrix may be multiplied with the first correlation probability, and the product result may be averaged to obtain the first correlation feature vectors corresponding to the multiple calibration images. Exemplarily, the first correlation feature vector F_i corresponding to the i-th calibration image is mean((diff_ij+Ki)*P_ij,-1). Wherein, diff_ij+Ki represents the fusion feature obtained by fusing the first index vector and the first correlation matrix, and mean represents the average processing of the product result of the fusion feature (diff_ij+Ki) and the first correlation probability P_ij, and finally obtains the first correlation feature vector F_i.

[0047] For ease of understanding, based on the above, you can refer to Figure 4 The structural schematic diagram of a self-attention unit shown in the figure shows that the self-attention unit includes a first FC network layer, a concatenation layer, a second FC network layer, a softmax layer, a fusion layer, and an average processing layer. The network layer in the self-attention unit can refer to the aforementioned steps a to c to perform corresponding operations, so as to output the first related feature vectors corresponding to the multiple calibration images. The specific implementation method can refer to the above-mentioned related content, which will not be repeated here. The above is only an implementation example of the self-attention unit. In practical applications, other network structures can also be used for implementation, which is not limited here.

[0048] The cross attention unit is used to obtain the second related feature vector between the target image and the multiple calibration images through the cross attention mechanism based on the received input information. Among them, the input information corresponding to the first cross attention unit includes the second image feature and the first related information, and the input information corresponding to the non-first cross attention unit includes the first related information and the second related feature vector output by the previous cross attention unit; the second related information includes the second related feature vector output by the last cross attention unit. The structure of each cross attention unit is the same, and the main difference lies in the different inputs. The input of the first cross attention unit is the output of the self-attention network and the second image feature output by the feature extraction network. The input of the remaining cross attention units is the output of the self-attention network and the output of the previous cross attention unit. The output of the last cross attention unit is the output of the cross attention network.

[0049] Specifically, the embodiment of the present disclosure provides a specific implementation method of the cross attention unit obtaining the second related feature vector between the target image and the multiple calibration images through the cross attention mechanism based on the received input information, which can be performed with reference to the following steps A to D:

[0050] Step A, based on the first relevant information in the input information received by the cross-attention unit, obtain the second index vectors corresponding to the multiple calibration images respectively. In some specific implementation examples, the first relevant information includes the first relevant feature vectors corresponding to the multiple calibration images output by the self-attention network. In practical applications, the first relevant feature vectors corresponding to the multiple calibration images can be converted into second index vectors with the help of the third FC network layer. For example, assuming that there are 3 calibration images in total, the first relevant feature vectors corresponding to the 3 calibration images are converted into second index vectors corresponding to the 3 calibration images, which are K1', K2', and K3', respectively. That is, the number of second index vectors is equal to the number of calibration images. Among them, Ki'=FC_k(f_i'), that is, the first relevant feature vector f_i' corresponding to the i-th calibration image is processed by the FC network layer to obtain the second index vector Ki'.

[0051] Step B, based on the second image feature or the second related feature vector in the input information received by the cross attention unit, obtain the second query vector corresponding to the target image. In practical applications, the second image feature or the second related feature vector can be converted into a second query vector Q with the help of the fourth FC network layer, and the number of second query vectors is 1. Among them, Q = FC_q (f), that is, the second query vector is obtained by processing the feature f corresponding to the target image through the FC network layer.

[0052] Step C, respectively splicing the second index vectors corresponding to the multiple calibration images with the second query vector corresponding to the target image to obtain multiple second splicing vectors. For example, the second index vectors K1', K2' and K3' corresponding to the three calibration images are respectively spliced ​​with Q to obtain three second splicing vectors K1'Q, K2'Q and K3'Q, where the second splicing vector can be expressed as Ki'Q, where i is 1 to n, and n is the number of calibration images.

[0053] Step D, based on the multiple second stitching vectors, obtain the second correlation feature vector between the target image and the multiple calibration images. Based on the analysis of the multiple second stitching vectors, the correlation between the target image and each calibration image can be determined one by one, and the second correlation feature vector between the target image and the multiple calibration images can reflect the relevant information between the target image and other calibration images. Exemplarily, step D can be performed with reference to the following steps D1 to D3:

[0054] Step D1, according to the plurality of second stitching vectors, obtain the second correlation matrix corresponding to the target image and the plurality of calibration images. In practical applications, the second correlation matrix corresponding to the target image and the plurality of calibration images can be generated based on the plurality of second stitching vectors with the aid of the fifth FC network layer, and the relevant information between the target image and each calibration image obtained based on the second stitching vector can be used as a matrix element in the second correlation matrix. Exemplarily, the second correlation matrix can be a three-row and one-column matrix consisting of the relevant information (which can be expressed in the form of eigenvectors) corresponding to K1'Q, K2'Q and K3'Q respectively. Among them, the second correlation matrix diff_i=FC_di(cat(Ki',Q)). Specifically, cat(Ki',Q) means that Ki' and Q are spliced ​​to obtain the second stitching vector, and FC_di(cat(Ki',Q)) means that the second splicing vector is analyzed through the FC network layer to obtain the second correlation matrix diff_i.

[0055] Step D2, obtaining the second correlation probability corresponding to the second correlation matrix through the second preset algorithm. The second correlation probability can be used to indicate the correlation probability between the target image and each calibration image. In specific implementation, the cross-attention unit uses the softmax function to normalize the elements in the second correlation matrix to obtain the correlation probability between the target image and each calibration image. Exemplarily, the second preset algorithm includes a softmax function, and the second correlation probability P_i=softmax(Diff_i).

[0056] Step D3, generating a second correlation feature vector between the target image and the plurality of calibration images according to the second index vectors, the second correlation matrix and the second correlation probability respectively corresponding to the plurality of calibration images.

[0057] Exemplarily, the second index vector and the second correlation matrix may be fused (such as by addition), and then the fused feature obtained based on the second index vector and the second correlation matrix may be multiplied by the second correlation probability, and the product result may be averaged to obtain the second correlation feature vector between the target image and the multiple calibration images. Exemplarily, the second correlation feature vector F between the target image and the multiple calibration images is mean((diff_i+Ki')*P_i). Wherein, diff_i+Ki' represents the fused feature obtained by fusing the second index vector and the second correlation matrix, and mean represents the average of the product result of the fused feature (diff_ij+Ki) and the second correlation probability P_i, and finally the second correlation feature vector F is obtained.

[0058] In practical applications, the structure of the cross-attention unit is similar to that of the self-attention unit, and the structure of the self-attention unit can be referred to, which will not be repeated here.

[0059] Further, the embodiment of the present disclosure provides a specific implementation method for generating a sight line estimation direction of a target image by a prediction network according to the first relevant information, the second relevant information, and the sight line direction information respectively carried by a plurality of calibration images, which can be performed with reference to the following steps 1 to 4:

[0060] Step 1, according to the first relevant information, obtain the second index vectors corresponding to the multiple calibration images respectively. The specific acquisition method is similar to the acquisition method in the aforementioned step A. The prediction network and the cross-attention network can obtain the second index vector through the same network layer (such as the third FC network layer). The prediction network and the cross-attention network can also obtain the second index vector through different network layers. For example, the prediction network and the cross-attention network each have an FC network layer to obtain the second index vector based on the first relevant information. The specific setting can be flexibly set and is not limited here. As mentioned above, the number of second index vectors is equal to the number of calibration images. Specifically, the second index vector Ki'=FC_k(f_i').

[0061] Step 2, according to the second relevant information, obtain the third query vector corresponding to the target image. In practical applications, the second relevant information (specifically, the second relevant feature vector output by the cross-attention network) can be converted into a third query vector Q' with the help of the sixth FC network layer, and the number of the third query vector is 1. Among them, Q'=FC_q(F), that is, the third query vector is obtained by processing the second relevant feature vector F through the FC network layer.

[0062] Step 3, the second index vectors corresponding to the multiple calibration images are spliced ​​with the third query vector corresponding to the target image to obtain multiple third splicing vectors. For example, the second index vectors K1', K2' and K3' corresponding to the three calibration images are spliced ​​with Q' to obtain three third splicing vectors K1'Q', K2'Q' and K3'Q', where the third splicing vector can be expressed as Ki'Q', where i is 1 to n, and n is the number of calibration images.

[0063] Step 4, based on the multiple third stitching vectors and the line of sight direction information respectively carried by the multiple calibration images, generate the estimated line of sight direction of the target image. Based on the analysis of the multiple third stitching vectors, the correlation between the target image and each calibration image is determined, and combined with the line of sight direction of each calibration image, the line of sight direction of the target image can be reasonably and reliably estimated. Exemplarily, step 4 can be performed with reference to the following steps 4.1 to 4.3:

[0064] Step 4.1, according to the plurality of third splicing vectors, obtain the third correlation matrix corresponding to the target image and the plurality of calibration images. In practical applications, the third correlation matrix corresponding to the target image and the plurality of calibration images can be generated based on the plurality of third splicing vectors with the help of the seventh FC network layer, and the relevant information between the target image and each calibration image obtained based on the third splicing vector can be used as a matrix element in the third correlation matrix. Exemplarily, the third correlation matrix can be a three-row and one-column matrix consisting of the relevant information (which can be expressed in the form of eigenvectors) corresponding to K1'Q', K2'Q' and K3'Q' respectively. Among them, the third correlation matrix diff_i'=FC_di(cat(Ki',Q')). Specifically, cat(Ki',Q') means that Ki' and Q' are spliced ​​to obtain the third splicing vector, and FC_di(cat(Ki',Q)) means that the third splicing vector is analyzed through the FC network layer to obtain the third correlation matrix diff_i'.

[0065] Step 4.2, obtain the third correlation probability corresponding to the third correlation matrix through the third preset algorithm. The third correlation probability can be used to indicate the association probability between the target image and each calibration image. Compared with the second correlation probability, the third correlation probability is obtained by processing on the basis of the first correlation feature vector corresponding to multiple calibration images and the second correlation feature vector between the target image and the multiple calibration images, which fully considers the association between the calibration images and the association between the target image and the calibration image, and further improves the accuracy of the association probability between the target image and each calibration image. In addition, in the related technology, it is necessary to separately determine the line of sight difference between the target image and each calibration image, and then combine the line of sight direction marked by each calibration image itself to obtain the line of sight estimation result of the target image corresponding to each calibration image, and then use a weighted method to determine the final line of sight estimation result of the target image, and different calibration images are independent of each other. Compared with the above method, the disclosed embodiment can use the association probability between the target image and each calibration image to subsequently comprehensively estimate the line of sight direction of the target image, that is, it can allow the model to learn the impact of different correlations between the target image and each calibration image on the line of sight estimation result of the target image, and can better comprehensively utilize the information of multiple calibration images to assist in evaluating the line of sight estimation result of the target image, thereby effectively improving the accuracy of the line of sight estimation direction. In specific implementation, the cross-attention unit uses a third preset algorithm to normalize the elements in the third correlation matrix, and can accurately and reliably obtain the association probability between the target image and each calibration image. Exemplarily, the third preset algorithm can include a softmax function, and the third correlation probability P_i'=softmax(diff_i').

[0066] Step 4.3, generating the estimated line of sight direction of the target image according to the line of sight direction information, the third correlation matrix and the third correlation probability respectively carried by the multiple calibration images.

[0067] In some specific implementation examples, the gaze direction information and the third correlation matrix respectively carried by multiple calibration images can be fused to obtain a fusion feature; then, the fusion feature is averaged according to the product of the fusion feature and the third correlation probability to obtain the estimated gaze direction of the target image. Specifically, the above fusion process can be an addition process, such as adding the gaze direction information gt_i corresponding to the i-th calibration image and the matrix element diff_i' corresponding to Ki' in the third correlation matrix. Exemplarily, the estimated gaze direction of the target image gaze = mean((diff_i'+gt_i)*P_i').

[0068] In practical applications, the structure of the prediction network is similar to that of the self-attention unit and will not be described here.

[0069] Furthermore, the disclosed embodiment also provides a training method for the above sight line estimation model.

[0070] Exemplarily, an eye image can be randomly sampled from an eye image sample set (each eye image included in the sample set is marked with a theoretical line of sight direction) as a target image sample for which the line of sight direction is to be estimated, and at least two eye images can be sampled as multiple calibration image samples obtained, and then the target image sample and the multiple calibration image samples are input into the initial network model to be trained to obtain the line of sight estimation direction of the target image sample output by the initial network model in the aforementioned manner (that is, the manner in which the line of sight estimation model described above generates the line of sight estimation direction of the target image), and then a loss value is determined based on the difference between the theoretical line of sight direction marked by the target image sample and the line of sight estimation direction output by the initial network model, and the parameters of the initial network model are iteratively updated based on the loss value until a preset iteration termination condition is reached (such as the loss value converges to a preset threshold), and the initial network model that has finished training is used as a line of sight estimation model, which can output a relatively accurate line of sight estimation result.

[0071] In summary, the above-mentioned line of sight estimation model provided by the embodiment of the present disclosure directly estimates the line of sight estimation direction of the target image by analyzing the relationship between multiple calibration images and the relationship between the target image and multiple calibration images, combined with the line of sight direction information of the calibration image, which is more reliable and convenient.

[0072] Corresponding to the aforementioned line of sight estimation method, Figure 5 FIG. 1 is a schematic diagram of a structure of a sight line estimation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 5 As shown, the sight line estimation device includes:

[0073] An image acquisition module 502 is used to acquire a target image of a to-be-estimated sight line direction and a plurality of calibration images; wherein the calibration image carries sight line direction information, and both the target image and the calibration image are eye images;

[0074] A sight line estimation module 504, used to input the target image and the plurality of calibration images into a preset sight line estimation model to obtain a sight line estimation direction of the target image output by the sight line estimation model;

[0075] Among them, the line of sight estimation model is used to obtain first related information between multiple calibration images, and to obtain second related information between the target image and the multiple calibration images, and to generate the line of sight estimation direction of the target image based on the first related information, the second related information and the line of sight direction information corresponding to the calibration image.

[0076] The above model analyzes the correlation between multiple calibration images and the correlation between the target image and multiple calibration images, and directly estimates the line of sight estimation direction of the target image based on the line of sight direction information of the calibration image. It is more reliable and can directly predict the line of sight estimation direction of the target image, which is more convenient and quick.

[0077] In some embodiments, the line of sight estimation model is specifically used to: obtain first relevant information between the multiple calibration images based on a self-attention mechanism; the line of sight estimation model is specifically used to: obtain second relevant information between the target image and the multiple calibration images based on a cross-attention mechanism.

[0078] In some embodiments, the line of sight estimation model includes a feature extraction network, a self-attention network, a cross-attention network and a prediction network; wherein the feature extraction network is used to extract the first image features corresponding to each of the multiple calibration images, and to extract the second image features corresponding to the target image; the self-attention network is used to obtain the first related information between the multiple calibration images through a self-attention mechanism based on the first image features; the cross-attention network is used to obtain the second related information between the target image and the multiple calibration images through a cross-attention mechanism based on the second image features and the first related information; the prediction network is used to generate the line of sight estimation direction of the target image according to the first related information, the second related information and the line of sight direction information respectively carried by the multiple calibration images.

[0079] In some embodiments, the self-attention network includes one or more self-attention units, and multiple self-attention units are connected in sequence; the self-attention unit is used to obtain the first relevant feature vectors corresponding to the multiple calibration images respectively through the self-attention mechanism based on the received input information; wherein, the input information corresponding to the first self-attention unit includes the first image feature, and the input information corresponding to the non-first self-attention unit includes the first relevant feature vector output by the previous self-attention unit; the first relevant information includes the first relevant feature vector output by the last self-attention unit.

[0080] In some embodiments, the self-attention unit is specifically used to: obtain a first query vector and a first index vector respectively corresponding to the multiple calibration images based on the input information received by the self-attention unit; obtain a plurality of first stitching vectors based on the first query vectors and the first index vector respectively corresponding to the multiple calibration images; wherein the first stitching vector is obtained by stitching the first query vector and the first index vector, and the first query vectors and / or first index vectors corresponding to different first stitching vectors are different; and obtain the first related feature vectors respectively corresponding to the multiple calibration images according to the multiple first stitching vectors.

[0081] In some embodiments, the self-attention unit is specifically used to: obtain a first correlation matrix corresponding to the multiple calibration images based on the multiple first stitching vectors; obtain a first correlation probability corresponding to the first correlation matrix through a first preset algorithm; and generate first correlation feature vectors corresponding to the multiple calibration images respectively based on the first index vectors, the first correlation matrix and the first correlation probability respectively corresponding to the multiple calibration images.

[0082] In some embodiments, the cross-attention network includes one or more cross-attention units, and multiple cross-attention units are connected in sequence; the cross-attention unit is used to obtain a second related feature vector between the target image and the multiple calibration images through a cross-attention mechanism based on the received input information; wherein, the input information corresponding to the first cross-attention unit includes the second image feature and the first related information, and the input information corresponding to the non-first cross-attention unit includes the first related information and the second related feature vector output by the previous cross-attention unit; the second related information includes the second related feature vector output by the last cross-attention unit.

[0083] In some embodiments, the cross-attention unit is specifically used to: obtain second index vectors corresponding to the multiple calibration images respectively based on first related information in the input information received by the cross-attention unit; obtain a second query vector corresponding to the target image based on a second image feature or a second related feature vector in the input information received by the cross-attention unit; splice the second index vectors corresponding to the multiple calibration images with the second query vector corresponding to the target image respectively to obtain multiple second splicing vectors; and obtain second related feature vectors between the target image and the multiple calibration images based on the multiple second splicing vectors.

[0084] In some embodiments, the cross-attention unit is specifically used to: obtain a second correlation matrix corresponding to the target image and the multiple calibration images based on the multiple second stitching vectors; obtain a second correlation probability corresponding to the second correlation matrix through a second preset algorithm; and generate a second correlation feature vector between the target image and the multiple calibration images based on the second index vectors, the second correlation matrix and the second correlation probability respectively corresponding to the multiple calibration images.

[0085] In some embodiments, the prediction network is specifically used to: obtain second index vectors corresponding to the multiple calibration images respectively according to the first relevant information; obtain a third query vector corresponding to the target image according to the second relevant information; splice the second index vectors corresponding to the multiple calibration images with the third query vector corresponding to the target image to obtain multiple third splicing vectors; and generate a line of sight estimated direction of the target image based on the multiple third splicing vectors and the line of sight direction information respectively carried by the multiple calibration images.

[0086] In some embodiments, the prediction network is specifically used to: obtain a third correlation matrix corresponding to the target image and the multiple calibration images based on the multiple third stitching vectors; obtain a third correlation probability corresponding to the third correlation matrix through a third preset algorithm; and generate a line of sight estimated direction of the target image based on the line of sight direction information respectively carried by the multiple calibration images, the third correlation matrix and the third correlation probability.

[0087] In some embodiments, the prediction network is specifically used to: fuse the gaze direction information respectively carried by the multiple calibration images and the third correlation matrix to obtain a fusion feature; and perform averaging processing based on the product result of the fusion feature and the third correlation probability to obtain the estimated gaze direction of the target image.

[0088] The line of sight estimation device provided in the embodiments of the present disclosure can execute the line of sight estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0090] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing instructions executable by the processor; and a processor for reading the executable instructions from the memory and executing the instructions to implement a line of sight estimation method.

[0091] Figure 6 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and a memory 602 .

[0092] The processor 601 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0093] The memory 602 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may run the program instructions to implement the line of sight estimation method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0094] In one example, the electronic device 600 may further include: an input device 603 and an output device 604 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0095] In addition, the input device 603 may also include, for example, a keyboard, a mouse, and the like.

[0096] The output device 604 can output various information to the outside, including the determined distance information, direction information, etc. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0097] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device 600 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 600 may also include any other appropriate components.

[0098] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the line of sight estimation method provided by the embodiment of the present disclosure.

[0099] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0100] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the line of sight estimation method provided by the embodiment of the present disclosure.

[0101] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0102] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the line of sight estimation method in the embodiments of the present disclosure when executed by a processor.

[0103] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0104] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. For example, when collecting an eye image of the user, a prompt message for collecting the eye image is sent to the user, and the user can be further informed of the purpose of the image collection in the prompt message. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present disclosure according to the prompt message, and perform the operation of the technical solution of the present disclosure after obtaining the user's authorization.

[0105] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0106] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0107] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight estimation method, characterized in that: include: Acquire a target image whose sight direction is to be estimated and a plurality of calibration images; wherein the calibration image carries sight direction information, and both the target image and the calibration image are eye images; Inputting the target image and the plurality of calibration images into a preset sight line estimation model to obtain a sight line estimation direction of the target image output by the sight line estimation model; Among them, the line of sight estimation model is used to obtain first related information between the multiple calibration images, and to obtain second related information between the target image and the multiple calibration images, and to generate a line of sight estimation direction of the target image based on the first related information, the second related information and the line of sight direction information corresponding to the calibration image.

2. The method according to claim 1, characterized in that The obtaining of the first related information between the plurality of calibration images comprises: obtaining the first related information between the plurality of calibration images based on a self-attention mechanism; The obtaining of second related information between the target image and the plurality of calibration images includes: obtaining second related information between the target image and the plurality of calibration images based on a cross-attention mechanism.

3. The method according to claim 1 or 2, characterized in that: The line of sight estimation model includes a feature extraction network, a self-attention network, a cross-attention network and a prediction network; wherein, The feature extraction network is used to extract first image features corresponding to each of the plurality of calibration images, and to extract second image features corresponding to the target image; The self-attention network is used to obtain first related information between the multiple calibration images through a self-attention mechanism based on the first image feature; The cross-attention network is used to obtain second related information between the target image and the plurality of calibration images through a cross-attention mechanism based on the second image feature and the first related information; The prediction network is used to generate the estimated line of sight direction of the target image according to the first relevant information, the second relevant information and the line of sight direction information respectively carried by the multiple calibration images.

4. The method according to claim 3, characterized in that The self-attention network includes one or more self-attention units, and a plurality of the self-attention units are connected in sequence; The self-attention unit is used to obtain first relevant feature vectors corresponding to the multiple calibration images respectively through a self-attention mechanism based on the received input information; Among them, the input information corresponding to the first self-attention unit includes the first image feature, and the input information corresponding to the non-first self-attention unit includes the first related feature vector output by the previous self-attention unit; the first related information includes the first related feature vector output by the last self-attention unit.

5. The method according to claim 4, characterized in that The method of obtaining first related feature vectors corresponding to the plurality of calibration images respectively through a self-attention mechanism based on the received input information includes: Based on the input information received by the self-attention unit, obtain a first query vector and a first index vector respectively corresponding to the plurality of calibration images; Based on the first query vectors and the first index vectors respectively corresponding to the multiple calibration images, a plurality of first stitching vectors are obtained; wherein the first stitching vector is obtained by stitching the first query vector and the first index vector, and different first stitching vectors correspond to different first query vectors and / or first index vectors; According to the multiple first stitching vectors, first related feature vectors respectively corresponding to the multiple calibration images are obtained.

6. The method according to claim 5, characterized in that The step of acquiring first related feature vectors respectively corresponding to the plurality of calibration images according to the plurality of first stitching vectors includes: Acquire a first correlation matrix corresponding to the plurality of calibration images according to the plurality of first stitching vectors; Obtaining a first correlation probability corresponding to the first correlation matrix by a first preset algorithm; First correlation feature vectors respectively corresponding to the plurality of calibration images are generated according to the first index vectors respectively corresponding to the plurality of calibration images, the first correlation matrix, and the first correlation probability.

7. The method according to claim 3, characterized in that The cross attention network includes one or more cross attention units, and a plurality of the cross attention units are connected in sequence; The cross attention unit is used to obtain a second correlation feature vector between the target image and the plurality of calibration images through a cross attention mechanism based on the received input information; Among them, the input information corresponding to the first cross-attention unit includes the second image feature and the first related information, and the input information corresponding to the non-first cross-attention unit includes the first related information and the second related feature vector output by the previous cross-attention unit; the second related information includes the second related feature vector output by the last cross-attention unit.

8. The method according to claim 7, characterized in that The step of obtaining a second related feature vector between the target image and the plurality of calibration images through a cross attention mechanism based on the received input information includes: Based on the first relevant information in the input information received by the cross attention unit, obtaining second index vectors corresponding to the plurality of calibration images respectively; Obtaining a second query vector corresponding to the target image based on a second image feature or a second related feature vector in the input information received by the cross-attention unit; Splicing the second index vectors corresponding to the plurality of calibration images with the second query vector corresponding to the target image to obtain a plurality of second splicing vectors; Based on the multiple second stitching vectors, a second correlation feature vector between the target image and the multiple calibration images is obtained.

9. The method according to claim 8, characterized in that The step of acquiring a second correlation feature vector between the target image and the plurality of calibration images based on the plurality of second stitching vectors includes: Acquire a second correlation matrix corresponding to the target image and the plurality of calibration images according to the plurality of second stitching vectors; Obtaining a second correlation probability corresponding to the second correlation matrix by a second preset algorithm; A second correlation feature vector between the target image and the plurality of calibration images is generated according to the second index vectors, the second correlation matrix, and the second correlation probability respectively corresponding to the plurality of calibration images.

10. The method according to claim 3, characterized in that The generating the estimated line of sight direction of the target image according to the first relevant information, the second relevant information and the line of sight direction information respectively carried by the plurality of calibration images comprises: According to the first relevant information, obtaining second index vectors respectively corresponding to the plurality of calibration images; acquiring a third query vector corresponding to the target image according to the second relevant information; splicing the second index vectors corresponding to the plurality of calibration images with the third query vector corresponding to the target image respectively to obtain a plurality of third splicing vectors; Based on the plurality of third stitching vectors and the sight direction information respectively carried by the plurality of calibration images, a sight line estimation direction of the target image is generated.

11. The method according to claim 10, characterized in that The generating the estimated sight line direction of the target image based on the sight line direction information respectively carried by the plurality of third stitching vectors and the plurality of calibration images comprises: Acquire a third correlation matrix corresponding to the target image and the plurality of calibration images according to the plurality of third stitching vectors; Obtaining a third correlation probability corresponding to the third correlation matrix by a third preset algorithm; The estimated line of sight direction of the target image is generated according to the line of sight direction information respectively carried by the multiple calibration images, the third correlation matrix and the third correlation probability.

12. The method according to claim 11, characterized in that The generating the estimated sight line direction of the target image according to the sight line direction information respectively carried by the plurality of calibration images, the third correlation matrix and the third correlation probability comprises: Fusing the line of sight direction information respectively carried by the plurality of calibration images and the third correlation matrix to obtain a fusion feature; An average process is performed based on the product result of the fusion feature and the third related probability to obtain the estimated sight line direction of the target image.

13. A sight line estimation device, characterized in that: include: An image acquisition module, used to acquire a target image of a to-be-estimated sight line direction and a plurality of calibration images; wherein the calibration image carries sight line direction information, and both the target image and the calibration image are eye images; A sight line estimation module, used for inputting the target image and the plurality of calibration images into a preset sight line estimation model to obtain a sight line estimation direction of the target image output by the sight line estimation model; Among them, the line of sight estimation model is used to obtain first related information between the multiple calibration images, and to obtain second related information between the target image and the multiple calibration images, and to generate a line of sight estimation direction of the target image based on the first related information, the second related information and the line of sight direction information corresponding to the calibration image.

14. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the line of sight estimation method described in any one of claims 1-12 above.

15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method described in any one of claims 1 to 12.

Images