Line of sight estimation method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a line-of-sight estimation method, device, equipment and medium, wherein the method comprises: obtaining a first eye image to be estimated for a line-of-sight direction, and a plurality of second eye images carrying line-of-sight direction information; combining the first eye image and each of the plurality of second eye images to obtain a plurality of image groups; obtaining correlation information between the plurality of image groups, and based on the correlation information between the plurality of image groups, obtaining line-of-sight difference information and weight information corresponding to each of the plurality of image groups; and based on the line-of-sight difference information and the weight information corresponding to each of the plurality of image groups, and the line-of-sight direction information carried by each of the plurality of second eye images, determining a target line-of-sight direction of the first eye image. Embodiments of the present disclosure can additionally determine the correlation information between the plurality of image groups, and determine the line-of-sight direction of the first eye image based on the correlation information, thereby effectively guaranteeing the accuracy and reliability of the final line-of-sight direction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a line-of-sight estimation method, apparatus, device, and medium. Background Technology

[0002] In many fields such as gaming, healthcare, and intelligent control, it is necessary to identify the direction of human eye gaze in order to implement appropriate strategies based on the estimated gaze direction. The inventors discovered that most existing gaze estimation techniques struggle to obtain accurate and reliable gaze directions, thus failing to adequately meet these requirements. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a line-of-sight estimation method, apparatus, device, and medium.

[0004] This disclosure provides a gaze estimation method, the method comprising: acquiring a first eye image whose gaze direction is to be estimated, and multiple second eye images carrying gaze direction information; combining the first eye image and each of the multiple second eye images to obtain multiple image groups; acquiring correlation information between the multiple image groups, and acquiring gaze difference information and weight information corresponding to each of the multiple image groups based on the correlation information between the multiple image groups; wherein, the gaze difference information corresponding to the image group is used to characterize the gaze direction angle difference between the first eye image and the second eye image in the image group; and determining the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the multiple image groups, and the gaze direction information carried by each of the multiple second eye images.

[0005] Optionally, the step of obtaining the correlation information between the plurality of image groups, and obtaining the gaze difference information and weight information corresponding to each of the plurality of image groups based on the correlation information between the plurality of image groups, includes: for each of the plurality of image groups, performing feature extraction on the first eye image in the image group to obtain a first feature, and performing feature extraction on the second eye image in the image group to obtain a second feature; fusing the first feature and the second feature to obtain the fused feature corresponding to the image group; obtaining the correlation information between the plurality of image groups based on the fused feature corresponding to each of the plurality of image groups; and obtaining the gaze difference information and weight information corresponding to each of the plurality of image groups based on the correlation information between the plurality of image groups.

[0006] Optionally, obtaining the association information between the multiple image groups based on the fusion features corresponding to each of the multiple image groups includes: processing the fusion features corresponding to each of the multiple image groups based on a cross-attention mechanism to obtain the association information between the multiple image groups.

[0007] Optionally, the step of processing the fusion features corresponding to each of the plurality of image groups based on the cross-attention mechanism to obtain the association information between the plurality of image groups includes: performing a first cross-attention processing on the fusion features corresponding to each of the plurality of image groups to obtain a first association information between the plurality of image groups; and performing a second cross-attention processing on the fusion features corresponding to each of the plurality of image groups to obtain a second association information between the plurality of image groups.

[0008] Optionally, obtaining the gaze difference information and weight information corresponding to each of the plurality of image groups based on the association information between the plurality of image groups includes: determining the gaze difference information corresponding to each of the plurality of image groups using a difference algorithm based on the first association information between the plurality of image groups; obtaining the attention weight corresponding to each of the plurality of image groups based on the second association information between the plurality of image groups; and obtaining the weight information corresponding to each of the plurality of image groups based on the attention weight corresponding to each of the plurality of image groups.

[0009] Optionally, the step of obtaining the correlation information between the plurality of image groups and, based on the correlation information between the plurality of image groups, obtaining the gaze difference information and weight information corresponding to each of the plurality of image groups is performed by a preset target network model; wherein, the target network model is obtained in the following way: obtaining a plurality of sample groups; wherein, each sample group includes a first image sample and a second image sample, the first image sample in different sample groups is the same, and the second image sample is different; both the first image sample and the second image sample are eye images with known actual gaze directions; obtaining the gaze difference information and weight information corresponding to each of the plurality of sample groups through a preset initial network model; obtaining the estimated gaze direction of the first image sample based on the gaze difference information and weight information corresponding to each of the plurality of sample groups, and the actual gaze direction corresponding to each of the plurality of second image samples; training the initial network model based on the difference between the estimated gaze direction of the first image sample and the actual gaze direction of the first image sample, and obtaining the target network model based on the trained initial network model.

[0010] Optionally, determining the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the plurality of image groups, and the gaze direction information carried by each of the plurality of second eye images, includes: obtaining a plurality of estimated gaze directions corresponding to the first eye image based on the gaze difference information between the plurality of second eye images and the first eye image, and the gaze direction information carried by each of the plurality of second eye images; and performing weighted fusion processing on the plurality of estimated gaze directions based on the weight information corresponding to each of the plurality of image groups to obtain the target gaze direction of the first eye image.

[0011] This disclosure also provides a gaze estimation device, comprising: an image acquisition module for acquiring a first eye image with a gaze direction to be estimated, and multiple second eye images carrying gaze direction information; an image combination module for combining the first eye image with each of the multiple second eye images to obtain multiple image groups; an information acquisition module for acquiring correlation information between the multiple image groups, and acquiring gaze difference information and weight information corresponding to each of the multiple image groups based on the correlation information between the multiple image groups; wherein the gaze difference information corresponding to the image group is used to characterize the gaze direction angle difference between the first eye image and the second eye image in the image group; and a gaze direction determination module for determining the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the multiple image groups, and the gaze direction information carried by each of the multiple second eye images.

[0012] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the line-of-sight estimation method as provided in this disclosure.

[0013] This disclosure also provides a computer-readable storage medium storing a computer program for performing the line-of-sight estimation method as provided in this disclosure.

[0014] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the line-of-sight estimation method provided in this disclosure.

[0015] The technical solution provided in this disclosure combines a first eye image with a known gaze direction and several second eye images with known gaze directions to obtain multiple image groups. Then, it acquires the correlation information between these image groups, and further acquires the gaze difference information and weight information corresponding to each image group. Based on the gaze difference information and weight information of each image group, and the gaze direction information carried by each of the second eye images, the target gaze direction of the first eye image is determined. Compared to the independent approach used in related technologies, this method additionally determines the correlation information between multiple image groups, and comprehensively determines the gaze direction of the first eye image based on this correlation information, thereby effectively ensuring the accuracy and reliability of the final obtained gaze direction.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic flowchart illustrating a line-of-sight estimation method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a schematic diagram of the structure of a target network model provided in an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram of the structure of a target network model provided in an embodiment of the present disclosure;

[0022] Figure 4 A schematic flowchart illustrating a line-of-sight estimation method provided in an embodiment of this disclosure;

[0023] Figure 5 This is a schematic diagram of the structure of a line-of-sight estimation device provided in an embodiment of the present disclosure;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0026] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0027] Figure 1 This is a flowchart illustrating a gaze estimation method provided in an embodiment of the present disclosure. The method can be executed by a gaze estimation device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0028] Step S102: Obtain a first eye image whose gaze direction is to be estimated, and multiple second eye images carrying gaze direction information. In other words, the gaze direction of the eye in the first eye image is unknown, while the gaze direction of the eye in the second eye images is known. Furthermore, the aforementioned "multiple images" specifically refers to at least two images.

[0029] The eyes included in the first and second eye images can belong to the same target object (such as a user) or to different target objects; this is not limited. For example, the target user can wear electronic devices such as VR (Virtual Reality) glasses, smart glasses, eye trackers, or other electronic devices that require estimation of the user's eye gaze direction. These electronic devices can integrate a camera to capture images of the target user's eyes. Alternatively, the user can also directly capture eye images using a camera at a preset position without wearing any device; the method of capturing eye images is not limited. In practical applications, a second eye image containing gaze direction information can be acquired by displaying a gaze point or similar method, and a gaze direction label can be attached to the second eye image to obtain a second eye image carrying gaze direction information.

[0030] It should be noted that a prompt message can be sent to the user before collecting eye images, and the user's eye images can only be collected after authorization is obtained. For example, the prompt message can be sent to the user in the form of a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control for the user to choose "agree" or "disagree" to allow the electronic device to collect their eye images. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure.

[0031] Step S104: Combine the first eye image with each of the multiple second eye images to obtain multiple image groups. For example, combine the first eye image with each of the N second eye images to obtain N image groups. Image group 1 includes the first eye image and second eye image 1, image group 2 includes the first eye image and second eye image 2, and so on.

[0032] Step S106: Obtain the association information between multiple image groups, and based on the association information between multiple image groups, obtain the gaze difference information and weight information corresponding to each of the multiple image groups; wherein, the gaze difference information corresponding to the image group is used to characterize the gaze direction angle difference between the first eye image and the second eye image in the image group.

[0033] This disclosure fully considers that if the gaze difference of each of multiple image groups is obtained individually, the multiple second eye images are completely independent of each other and have no information interaction, which means that the correlation between the data of multiple image groups will be ignored. Moreover, based on obtaining the gaze difference of each of multiple image groups independently, it is usually necessary to manually design relevant fusion parameters, which is not only complicated, but also prone to poor parameter design, and still cannot accurately and objectively determine the weights of different image groups, resulting in low accuracy of the subsequent gaze direction estimation results. Based on the above considerations, this disclosure first obtains the correlation information between multiple image groups. This correlation information can be used to indicate the degree of correlation or dependency between different image groups. In practical applications, based on multiple image groups and the correlation information between multiple image groups, it is helpful to more reasonably and reliably analyze and obtain the gaze difference information and weight information corresponding to each of the multiple image groups.

[0034] Step S108: Based on the gaze difference information and weight information corresponding to each of the multiple image groups, and the gaze direction information carried by each of the multiple second eye images, determine the target gaze direction of the first eye image.

[0035] In some implementations, multiple estimated gaze directions corresponding to the first eye image can be obtained based on the gaze difference information between multiple second eye images and the first eye image, as well as the gaze direction information carried by each of the multiple second eye images. Then, based on the weight information corresponding to each of the multiple image groups, a weighted fusion process is performed on the multiple estimated gaze directions to obtain the target gaze direction of the first eye image. Specifically, for each image group, the estimated gaze direction of the first eye image corresponding to that image group can be obtained based on the gaze difference information between the second eye images and the first eye image in that image group, as well as the gaze direction of the second eye images in that image group. Then, a weighted sum is performed based on the weights corresponding to each of the multiple image groups and the estimated gaze direction of the first eye image to obtain the target gaze direction of the first eye image.

[0036] In summary, compared to the independent approach used in related technologies, the above approach can additionally determine the correlation information between multiple image groups, and comprehensively determine the gaze direction of the first eye image based on this correlation information, thereby effectively ensuring the accuracy and reliability of the final gaze direction.

[0037] In some implementations, step S106, which is to obtain the correlation information between multiple image groups and, based on the correlation information between multiple image groups, obtain the line-of-sight difference information and weight information corresponding to each of the multiple image groups, can be performed with reference to steps A to C below:

[0038] Step A: For each image group in the multiple image groups, feature extraction is performed on the first eye image in the image group to obtain a first feature, and feature extraction is performed on the second eye image in the image group to obtain a second feature. The first and second features are then fused to obtain the fused feature corresponding to the image group. In practical applications, the first eye images in multiple image groups are the same, so only one feature extraction operation needs to be performed on the first eye image. The first and second features corresponding to each of the multiple image groups are fused, and the resulting fused feature carries both the eye information of the first eye image and the eye information of the second eye image. Compared to related technologies that directly input the first and second eye images into the gaze difference prediction network, this embodiment first extracts features from the images separately and then fuses them. The fusion method does not have to be a conventional stitching method, but can be achieved through methods such as convolution, dot product, or multiplication. For example, the first and second features can be input into a convolutional layer, which then fuses them to obtain the fused features. Different strategies can be selected based on the spatial location, semantic level, and fusion purpose of the features; no limitations are imposed here. This approach helps to achieve functions such as feature complementarity and measuring the correlation between different features, facilitating subsequent analysis and processing.

[0039] Step B: Based on the fusion features corresponding to each of the multiple image groups, obtain the correlation information between the multiple image groups.

[0040] For example, different fusion features can be compared and analyzed to measure whether they are similar or share common features, thereby obtaining the correlation between different fusion features. To efficiently and reliably determine the correlation information, in some specific implementation examples, the fusion features corresponding to multiple image groups can be processed based on a cross-attention mechanism to obtain the correlation information between the multiple image groups. The cross-attention mechanism allows elements in one sequence (or feature set) to focus on elements in another different sequence (or feature set). The cross-attention mechanism has three main inputs: query (Q), key (K), and value (V). Q comes from one sequence (specifically, the fusion features corresponding to one of the aforementioned image groups), while K and V come from another different sequence (i.e., fusion features corresponding to different image groups). By calculating the similarity score between Q and K, which can also be called the attention score, and normalizing it using a function such as softmax as the attention weight, the weighted sum V is used to obtain the output. This output can be used to present the correlation information between the fusion features corresponding to different image groups. The above-mentioned cross-attention mechanism can capture the complex interaction relationship between different fusion features, and thus present the correlation information between multiple image groups relatively accurately and reliably. The correlation information between multiple image groups can indicate the dependency relationship or correlation degree, similarity degree, common features, etc. between different image groups.

[0041] Step C: Based on the correlation information between multiple image groups, obtain the line-of-sight difference information and weight information corresponding to each of the multiple image groups.

[0042] In some specific implementation examples, cross-attention processing can be uniformly applied to the fusion features corresponding to multiple image groups to obtain the association information of multiple image groups. Then, based on this association information, the gaze difference information and weight information corresponding to each of the multiple image groups can be obtained separately. In other specific implementation examples, a first cross-attention processing can be applied to the fusion features corresponding to multiple image groups to obtain the first association information between the multiple image groups; a second cross-attention processing can be applied to the fusion features corresponding to multiple image groups to obtain the second association information between the multiple image groups. That is, different cross-attention processing can be performed based on the fusion features. The processing flow of the first and second cross-attention processing is roughly the same, but the network parameters and output results may differ. For example, the first association information can be used to represent the dependency relationship between different fusion features, which can be represented by association features between different fusion features, while the second association information can be represented only by the attention weights corresponding to each fusion feature. Based on the above, the gaze difference information corresponding to each of the multiple image groups can be further determined using a difference algorithm based on the first association information between the multiple image groups; wherein, the above difference algorithm can be implemented through a difference model. Furthermore, attention weights corresponding to each of the multiple image groups are obtained based on the second association information between the multiple image groups, and weight information corresponding to each of the multiple image groups is obtained based on the attention weights corresponding to each of the multiple image groups. For example, the attention weights corresponding to the fusion features of each image group can be used as the weights of that image group.

[0043] In practical applications, the steps of obtaining correlation information between multiple image groups and, based on this correlation information, obtaining line-of-sight difference information and weight information corresponding to each image group are performed through a preset target network model. For ease of understanding, this disclosure provides two structures for the target network model, which are described below in conjunction with... Figure 2 and Figure 3 Explain each:

[0044] See Figure 2The diagram illustrates the structure of a target network model, showing that it includes a feature extraction network, a feature fusion network, a cross-attention network, a gaze difference determination network, and a weight determination network. A first eye image and N second eye images are combined to form N image groups. For example, image group 1 contains a first eye image and a second eye image 1, and so on. The N image groups are input into the feature extraction network for extraction, yielding features corresponding to each image group. These features are then fed in parallel into the feature fusion network to obtain fused features 1 through N. The N fused features are then input into the cross-attention network to obtain the correlation information between them. The gaze difference determination network uses this correlation information to determine the gaze difference for each of the N image groups. Combined with the gaze direction of the second eye images in the N image groups, this yields the estimated gaze direction of the first eye image for each of the N image groups. The weight determination network uses this correlation information to determine the weights for each of the N image groups. The N estimated gaze directions are then weighted and fused to obtain the target gaze direction of the first eye image. Furthermore... Figure 2 This is for illustrative purposes only. In practical applications, N fusion features can also be combined with correlation information and input into the line-of-sight difference determination network; this is not a limitation. See also... Figure 3 The diagram shown is a structural schematic of a target network model. Figure 3 and Figure 2 The main difference is that, Figure 2 A separate cross-attention network was set up. The gaze difference determination network and the weight determination network both process the output of the cross-attention network. Figure 3 In this design, separate cross-attention units are incorporated into both the line-of-sight difference determination network and the weight determination network. This allows for flexible implementation in practical applications. Figure 2 or Figure 3 The method is not restricted here. Figure 2 and Figure 3 All the line-of-sight difference determination networks can be implemented based on differential networks, and Figure 3 In addition to the difference network, the line-of-sight determination network also includes cross-attention units. Figure 2 and Figure 3 The weight determination network in the process includes layers such as softmax, which convert the correlation information between the fused features of N image groups (such as attention score representation) into a probability distribution with a sum of 1, thereby obtaining the weights of each image group.

[0045] It should be noted that even in related technologies where a line-of-sight difference prediction model exists, only two eye images from an image group are input into the model, and the model outputs the corresponding line-of-sight difference estimation result. The data from different image groups are completely independent, and the model processes each image group separately. However, the embodiments of this disclosure require inputting relevant information from multiple image groups into the model simultaneously. The model needs to comprehensively determine the line-of-sight difference corresponding to each image group based on the information from multiple image groups.

[0046] In some specific implementation examples, the target network model is trained through the following steps a to d:

[0047] Step a, obtain multiple sample groups; wherein, each sample group contains a first image sample and a second image sample, the first image sample in different sample groups is the same, and the second image sample is different; both the first image sample and the second image sample are eye images with the actual gaze direction known.

[0048] Step b involves obtaining the line-of-sight difference information and weight information for each of the multiple sample groups using a pre-defined initial network model. The method for obtaining the line-of-sight difference information and weight information for each of the multiple sample groups using the initial network model can refer to the relevant content in the preceding steps. For example, it also requires first obtaining the correlation information between the multiple sample groups, and then obtaining the line-of-sight difference information and weight information for each sample group based on the correlation information. This will not be elaborated further here.

[0049] Step c involves obtaining the estimated gaze direction of the first image sample based on the gaze difference information and weight information corresponding to each of the multiple sample groups, as well as the actual gaze directions corresponding to each of the multiple second image samples. For details, please refer to the specific implementation of step S108 above, which will not be repeated here.

[0050] Step d: Based on the difference between the estimated gaze direction of the first image sample and the actual gaze direction of the first image sample, the initial network model is trained, and the target network model is obtained based on the trained initial network model. Specifically, the trained initial network model can be directly used as the target network model. The loss function described above can be implemented using the following formula:

[0051]

[0052] Among them, w i This represents the weight corresponding to the i-th sample group, where N is the number of sample groups, which is also the number of samples in the second image. (CalibGaze) i For the actual viewing direction of the i-th second image sample, Diff i The line-of-sight difference between the first image sample and the i-th second image sample. Let GtGaze be the estimated gaze direction of the first image sample and GtGaze be the actual gaze direction of the first image sample. The difference between the estimated gaze direction and the actual gaze direction of the first image sample can be obtained through the above method. This loss can be quantified and represented by the loss function value. The parameters of the initial network model are adjusted with the goal of reducing this loss until the loss function value converges to the preset threshold and the training ends. The trained initial network model can be used as the target network model.

[0053] It should be noted that in related technologies, the line-of-sight difference estimation model is usually trained by directly targeting the difference between the estimated line-of-sight difference output by the model and the theoretical line-of-sight difference (also known as the actual line-of-sight difference). Moreover, multiple image groups are not associated or multiple line-of-sight difference estimation results are not weighted during the training process. Therefore, the embodiments of this disclosure and related technologies have made significant improvements in model training. The model trained in the above manner helps to directly obtain more accurate and reliable line-of-sight direction estimation results.

[0054] See Figure 4 The flowchart of a line-of-sight estimation method shown here mainly includes the following steps S402 to S414:

[0055] Step S402: Obtain a first eye image of the gaze direction to be estimated, and multiple second eye images carrying gaze direction information.

[0056] Step S404: Combine the first eye image with each of the multiple second eye images to obtain multiple image groups.

[0057] Step S406: For each image group in the multiple image groups, feature extraction is performed on the first eye image in the image group to obtain a first feature, and feature extraction is performed on the second eye image in the image group to obtain a second feature. The first feature and the second feature are fused together to obtain the fused feature corresponding to the image group.

[0058] Step S408: Based on the fusion features corresponding to each of the multiple image groups, obtain the correlation information between the multiple image groups.

[0059] Step S410: Based on the correlation information between multiple image groups, obtain the line-of-sight difference information and weight information corresponding to each of the multiple image groups.

[0060] Step S412: Based on the line-of-sight difference information between multiple second eye images and the first eye image, and the line-of-sight direction information carried by each of the multiple second eye images, multiple estimated line-of-sight directions corresponding to the first eye image are obtained.

[0061] Step S414: Based on the weight information corresponding to each of the multiple image groups, perform weighted fusion processing on the multiple estimated gaze directions to obtain the target gaze direction of the first eye image.

[0062] In summary, the above-described method provided by the embodiments of this disclosure can associate data from multiple image groups and analyze the gaze difference and weight of each image group on this basis. It also eliminates the need for manual design of fusion parameters, enabling the model to adaptively learn a reasonable fusion method for multiple image groups, which helps to obtain more accurate and reliable gaze direction estimation results.

[0063] Corresponding to the aforementioned line-of-sight estimation method, this disclosure further provides a line-of-sight estimation device. Figure 5 This is a schematic diagram of a line-of-sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 5 As shown, the line-of-sight estimation device includes:

[0064] The image acquisition module 502 is used to acquire a first eye image of the gaze direction to be estimated, and multiple second eye images carrying gaze direction information;

[0065] The image combination module 504 is used to combine the first eye image with each of the multiple second eye images to obtain multiple image groups;

[0066] The information acquisition module 506 is used to acquire the correlation information between multiple image groups, and based on the correlation information between multiple image groups, acquire the line-of-sight difference information and weight information corresponding to each of the multiple image groups; wherein, the line-of-sight difference information corresponding to the image group is used to characterize the line-of-sight direction angle difference between the first eye image and the second eye image in the image group;

[0067] The gaze direction determination module 508 is used to determine the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the multiple image groups, as well as the gaze direction information carried by each of the multiple second eye images.

[0068] Compared to the independent approach used in related technologies, the above-mentioned device can additionally determine the correlation information between multiple image groups, and comprehensively determine the gaze direction of the first eye image based on the correlation information, thereby effectively ensuring the accuracy and reliability of the final gaze direction.

[0069] In some embodiments, the information acquisition module 506 is specifically used for: for each of the plurality of image groups, performing feature extraction on the first eye image in the image group to obtain a first feature, and performing feature extraction on the second eye image in the image group to obtain a second feature; fusing the first feature and the second feature to obtain a fused feature corresponding to the image group; acquiring correlation information between the plurality of image groups based on the fused features corresponding to each of the plurality of image groups; and acquiring line-of-sight difference information and weight information corresponding to each of the plurality of image groups based on the correlation information between the plurality of image groups.

[0070] In some implementations, the information acquisition module 506 is specifically used to: process the fusion features corresponding to each of the plurality of image groups based on a cross-attention mechanism to obtain the association information between the plurality of image groups.

[0071] In some embodiments, the information acquisition module 506 is specifically used to: perform a first cross-attention processing on the fusion features corresponding to each of the plurality of image groups to obtain a first association information between the plurality of image groups; and perform a second cross-attention processing on the fusion features corresponding to each of the plurality of image groups to obtain a second association information between the plurality of image groups.

[0072] In some implementations, the information acquisition module 506 is specifically used to: determine the line-of-sight difference information corresponding to each of the plurality of image groups based on the first association information between the plurality of image groups using a difference algorithm; obtain the attention weights corresponding to each of the plurality of image groups based on the second association information between the plurality of image groups; and obtain the weight information corresponding to each of the plurality of image groups based on the attention weights corresponding to each of the plurality of image groups.

[0073] In some implementations, the step of obtaining the correlation information between the plurality of image groups and, based on the correlation information between the plurality of image groups, obtaining the gaze difference information and weight information corresponding to each of the plurality of image groups is performed by a preset target network model; the method further includes a model training module, which is used to obtain the target network model in the following manner: obtaining a plurality of sample groups; wherein, the sample group includes a first image sample and a second image sample, the first image sample in different sample groups is the same, and the second image sample is different; the first image sample and the second image sample are both eye images with known actual gaze directions; obtaining the gaze difference information and weight information corresponding to each of the plurality of sample groups through a preset initial network model; obtaining the estimated gaze direction of the first image sample based on the gaze difference information and weight information corresponding to each of the plurality of sample groups, and the actual gaze direction corresponding to each of the plurality of second image samples; training the initial network model based on the difference between the estimated gaze direction of the first image sample and the actual gaze direction of the first image sample, and obtaining the target network model based on the trained initial network model.

[0074] In some embodiments, the gaze direction determination module 508 is specifically used to: obtain multiple estimated gaze directions corresponding to the first eye image based on the gaze difference information between the multiple second eye images and the first eye image, and the gaze direction information carried by each of the multiple second eye images; and perform weighted fusion processing on the multiple estimated gaze directions based on the weight information corresponding to each of the multiple image groups to obtain the target gaze direction of the first eye image.

[0075] The line-of-sight estimation device provided in this disclosure can execute the line-of-sight estimation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0077] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0078] The following is for reference. Figure 6This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0079] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0080] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0081] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0082] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0083] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the line-of-sight estimation method provided in embodiments of this disclosure.

[0084] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0085] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the line-of-sight estimation method in this disclosure.

[0086] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0087] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0088] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0089] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0090] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line-of-sight estimation method, characterized in that, include: Acquire a first eye image of the gaze direction to be estimated, and multiple second eye images carrying gaze direction information; The first eye image is combined with each of the multiple second eye images to obtain multiple image groups; The association information between the plurality of image groups is obtained, and based on the association information between the plurality of image groups, the gaze difference information and weight information corresponding to each of the plurality of image groups are obtained; wherein, the gaze difference information corresponding to the image group is used to characterize the gaze direction angle difference between the first eye image and the second eye image in the image group; Based on the gaze difference information and weight information corresponding to each of the multiple image groups, and the gaze direction information carried by each of the multiple second eye images, the target gaze direction of the first eye image is determined.

2. The method according to claim 1, characterized in that, The step of obtaining the correlation information between the plurality of image groups, and based on the correlation information between the plurality of image groups, obtaining the line-of-sight difference information and weight information corresponding to each of the plurality of image groups, includes: For each of the plurality of image groups, feature extraction is performed on the first eye image in the image group to obtain a first feature, and feature extraction is performed on the second eye image in the image group to obtain a second feature. The first feature and the second feature are then fused to obtain the fused feature corresponding to the image group. Based on the fusion features corresponding to each of the multiple image groups, the association information between the multiple image groups is obtained; Based on the correlation information between the multiple image groups, the line difference information and weight information corresponding to each of the multiple image groups are obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the association information between the multiple image groups based on the fusion features corresponding to each of the multiple image groups includes: The fusion features corresponding to each of the multiple image groups are processed based on the cross-attention mechanism to obtain the association information between the multiple image groups.

4. The method according to claim 3, characterized in that, The process of processing the fusion features corresponding to each of the multiple image groups based on the cross-attention mechanism to obtain the association information between the multiple image groups includes: A first cross-attention process is performed on the fusion features corresponding to each of the multiple image groups to obtain the first association information between the multiple image groups; A second cross-attention process is performed on the fusion features corresponding to each of the multiple image groups to obtain the second association information between the multiple image groups.

5. The method according to claim 4, characterized in that, The step of obtaining the line-of-sight difference information and weight information corresponding to each of the multiple image groups based on the correlation information between the multiple image groups includes: Based on the first correlation information between the plurality of image groups, the line-of-sight difference information corresponding to each of the plurality of image groups is determined by the difference algorithm; Based on the second association information between the multiple image groups, the attention weights corresponding to each of the multiple image groups are obtained, and based on the attention weights corresponding to each of the multiple image groups, the weight information corresponding to each of the multiple image groups is obtained.

6. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the correlation information between the plurality of image groups, and obtaining the line-of-sight difference information and weight information corresponding to each of the plurality of image groups based on the correlation information between the plurality of image groups, is performed by a preset target network model; wherein, the target network model is obtained in the following manner: Multiple sample groups are obtained; wherein each sample group contains a first image sample and a second image sample, the first image sample is the same in different sample groups, and the second image sample is different; both the first image sample and the second image sample are eye images with the actual gaze direction known. The line-of-sight difference information and weight information corresponding to each of the multiple sample groups are obtained through a preset initial network model; Based on the gaze difference information and weight information corresponding to each of the multiple sample groups, and the actual gaze direction corresponding to each of the multiple second image samples, the estimated gaze direction of the first image sample is obtained. Based on the difference between the estimated gaze direction of the first image sample and the actual gaze direction of the first image sample, the initial network model is trained, and the target network model is obtained based on the trained initial network model.

7. The method according to claim 1, characterized in that, The step of determining the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the plurality of image groups, and the gaze direction information carried by each of the plurality of second eye images, includes: Based on the line-of-sight difference information between the multiple second eye images and the first eye image, and the line-of-sight direction information carried by each of the multiple second eye images, multiple estimated line-of-sight directions corresponding to the first eye image are obtained. Based on the weight information corresponding to each of the multiple image groups, the multiple estimated gaze directions are weighted and fused to obtain the target gaze direction of the first eye image.

8. A line-of-sight estimation device, characterized in that, include: The image acquisition module is used to acquire a first eye image of the gaze direction to be estimated, and multiple second eye images carrying gaze direction information; An image combination module is used to combine the first eye image with each of the multiple second eye images to obtain multiple image groups; The information acquisition module is used to acquire the correlation information between the plurality of image groups, and based on the correlation information between the plurality of image groups, acquire the line-of-sight difference information and weight information corresponding to each of the plurality of image groups; wherein, the line-of-sight difference information corresponding to the image group is used to characterize the line-of-sight direction angle difference between the first eye image and the second eye image in the image group; The gaze direction determination module is used to determine the target gaze direction of the first eye image based on the gaze difference information and weight information corresponding to each of the plurality of image groups, as well as the gaze direction information carried by each of the plurality of second eye images.

9. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the line-of-sight estimation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the line-of-sight estimation method according to any one of claims 1-7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the line-of-sight estimation method according to any one of claims 1-7.

Images