Methods for constructing sight estimation models, sight estimation methods and apparatus

Abstract

This disclosure relates to a method for constructing a gaze estimation model, a gaze estimation method, and an apparatus. The method includes: acquiring an eye image of a first target object and an eye image of a second target object; wherein the eye images of the first and second target objects respectively correspond to labels containing gaze directions; acquiring an estimated gaze difference between the eye images of the first and second target objects based on a model to be constructed; constructing a loss function based on the labels corresponding to the eye images of the first and second target objects, the labels corresponding to the eye images of the second target objects, and the gaze difference; and training the model to be constructed based on training samples and the loss function to obtain a gaze estimation model. This can improve the accuracy of the gaze direction estimated by the gaze estimation model in the image to be estimated.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for constructing a line-of-sight estimation model, a line-of-sight estimation method, and an apparatus. Background Technology

[0002] Gaze is a crucial clue revealing how humans understand their external environment. Gaze estimation technology refers to the technique of calculating the direction of a user's gaze using captured images of their eyes. Gaze estimation technology has found numerous applications in human-computer interaction, virtual reality, augmented reality, and medical analysis.

[0003] When users wear related devices, such as AR (Augmented Reality), VR (Virtual Reality), or MR (Mixed Reality), images of the user's eyes can be captured. However, when directly estimating the gaze direction from these eye images, the estimated gaze direction may be inaccurate. Summary of the Invention

[0004] This disclosure provides a method for constructing a line-of-sight estimation model, a line-of-sight estimation method, and an apparatus.

[0005] According to one aspect of this disclosure, a method for constructing a gaze estimation model is provided, the method comprising:

[0006] Obtain eye images of a first target object and a second target object; wherein the eye images of the first target object and the second target object respectively correspond to labels containing the direction of gaze;

[0007] Based on the model to be constructed, the estimated gaze difference between the eye image of the first target object and the eye image of the second target object is obtained;

[0008] A loss function is constructed based on the labels contained in the eye images of the first target object, the labels contained in the eye images of the second target object, and the gaze difference.

[0009] The model to be constructed is trained based on the training samples and the loss function to obtain a gaze estimation model; wherein, the training samples include eye images of the first target object and eye images of the second target object, during the training process, the loss value is determined based on the loss function, if the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated, and if the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

[0010] According to a second aspect of this disclosure, a line-of-sight estimation method is provided, the method comprising:

[0011] Acquire a calibration image; the calibration image corresponds to a known line-of-sight direction;

[0012] Obtain the image to be estimated;

[0013] Based on the calibration image, the image to be estimated, and the gaze estimation model, the gaze direction estimated for the image to be estimated is obtained; wherein, the gaze estimation model is obtained by the method described above.

[0014] According to a third aspect of this disclosure, an apparatus for constructing a gaze estimation model is provided, the apparatus comprising:

[0015] An image acquisition module is used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object are respectively labeled with a gaze direction.

[0016] The gaze difference acquisition module is used to acquire the estimated gaze difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed.

[0017] The loss function construction module is used to construct a loss function based on the labels contained in the eye image of the first target object, the labels contained in the eye image of the second target object, and the gaze difference.

[0018] The model training module is used to train the model to be constructed based on training samples and the loss function to obtain a gaze estimation model. The training samples include eye images of the first target object and eye images of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

[0019] According to a fourth aspect of this disclosure, a line-of-sight estimation device is provided, the device comprising:

[0020] A calibration image acquisition module is used to acquire a calibration image; the calibration image corresponds to a known line-of-sight direction;

[0021] The image acquisition module is used to acquire the image to be estimated.

[0022] A gaze estimation module is used to obtain the estimated gaze direction of the image to be estimated based on the calibration image, the image to be estimated, and the gaze estimation model; wherein the gaze estimation model is obtained by the method described above.

[0023] According to a fifth aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0024] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0025] This embodiment of the disclosure acquires eye images of a first target object and a second target object, and obtains the estimated gaze difference between the eye images of the first target object and the second target object based on the model to be constructed. A loss function is constructed using the labels corresponding to the eye images of the first target object, the labels corresponding to the eye images of the second target object, and the gaze difference. By training the model to be constructed based on training samples and the loss function, the resulting gaze estimation model can offset the influence of individual differences between different users, enabling the eye image of the first target object to be simulated using the eye data of the second target object, thereby improving the accuracy of the gaze direction estimated by the gaze estimation model. Attached Figure Description

[0026] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A schematic diagram of a line-of-sight estimation model provided as an exemplary embodiment of this disclosure;

[0028] Figure 2 A flowchart illustrating a method for constructing a gaze estimation model provided as an exemplary embodiment of this disclosure;

[0029] Figure 3 A schematic block diagram of the functional modules of an apparatus for constructing a line-of-sight estimation model provided in an exemplary embodiment of the present disclosure;

[0030] Figure 4 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure;

[0031] Figure 5 A block diagram of a computer system provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0037] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0038] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0039] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0040] When users wear AR, VR, or MR target devices, with the user's authorization and consent, images of the user's eyes can be captured in real time through image acquisition devices set in the target device. The direction of gaze can be estimated from the captured eye images. For example, the scene displayed on the target device can be rendered in a local fine detail based on the estimated direction of gaze, that is, fine rendering can be performed only on the scene within the user's gaze range, thereby reducing the hardware cost of the target device.

[0041] When estimating the gaze direction from images of a user's eyes, related technologies may input a calibration image and the image to be estimated into a differential gaze estimation model to obtain the estimated gaze difference between the two images. Based on this gaze difference and the known gaze direction corresponding to the calibration image, the gaze direction of the image to be estimated is then estimated. However, since the calibration image may be taken when the user first wears the target device, while the image to be estimated may be taken on another occasion when the user wears the target device, differences in wearing position between these different wearing experiences can affect the estimated gaze direction, leading to inaccuracies in the estimated gaze direction.

[0042] like Figure 1 As shown, Figure 1This is a gaze estimation model that extracts image features from two images and estimates the gaze difference between them. Two images can be acquired: a calibration image and the image to be estimated. These two images are input into the gaze estimation model, which outputs the estimated gaze difference between the gaze directions of the two images. The calibration image is acquired by guiding the user to gaze at a calibration point with a known gaze direction. Therefore, the calibration image corresponds to a known gaze direction. Combining this difference with the calibration image, the true gaze direction of the image to be estimated can be estimated. It should be noted that the images in this embodiment can be image samples. Image samples can include a user's monocular image, such as an image of the left or right eye; image samples can also include a user's binocular images, such as an image of the left and right eyes; image samples can also be a stitched image of a user's left and right eyes, and the embodiment is not limited to these.

[0043] Therefore, to improve the accuracy of gaze estimation models in estimating gaze direction, it is necessary to obtain eye images of the same user wearing the target device at different times. This allows the gaze estimation model to learn the impact of different wearing states on gaze direction estimation through training. However, in practice, obtaining a large number of eye images of the same user under different wearing states as image samples would significantly increase the cost of collecting training samples. Furthermore, research has found that due to individual differences among users, including variations in eye structure such as the angle between the optical axis and visual axis (where the eye image represents the direction of the optical axis, while the direction of the visual axis is the actual gaze direction), and because these individual differences cannot be eliminated, the gaze estimation accuracy of the resulting gaze estimation model cannot be guaranteed if eye images obtained from different users wearing the target device are used to simulate eye images of the same user under different wearing states for model training.

[0044] For example, when training the gaze estimation model using the two image samples mentioned above, the gaze labels corresponding to the image samples include the gaze direction represented by the image and the influence of individual factors caused by the aforementioned individual differences on the true value of the gaze direction. If the gaze direction of the image to be estimated is estimated using a non-differential gaze estimation model, the estimated gaze direction can be directly output based on a single image sample. In this case, the model's learned result can be divided into two situations. The first situation is that the image samples come from the same user. The non-differential gaze estimation model learns the gaze direction represented by the image and the influence of individual factors caused by the user's individual differences on the true value of the gaze direction. During the testing phase, if the gaze direction estimation is for that user, the gaze estimation of the image to be estimated is relatively accurate because both the training samples and the image to be estimated during the testing phase come from that user. However, if the image to be estimated during the testing phase does not come from the same user as the image samples used in the training phase, the individual differences between different individuals will affect the estimated gaze direction, leading to a significant deviation in the estimated gaze direction.

[0045] If a differential gaze estimation model is used to estimate the gaze direction of an image, the model learns the gaze difference between two image samples. The true value of this gaze difference can include the gaze difference represented by the images and the difference in individual factors. If images from the same user are used as input to the model, the difference in individual factors is zero, ensuring that the output is not affected by individual differences. However, if the model input comes from two different users, the difference in individual factors will not be zero, and since the user's individual factors are unknown, there will be negative impacts on gaze estimation due to individual differences between different users.

[0046] Therefore, when training a differential gaze estimation model using two image samples, if the first image sample is an eye image captured by the first user wearing the target device, and the second image sample is an eye image captured by the second user wearing the target device, the gaze estimation accuracy of the trained gaze estimation model will decrease if the influence of this individual difference on the gaze direction cannot be eliminated when training the model using eye images from different users.

[0047] For example, due to the aforementioned individual differences among users, the influence of individualized eye structure on gaze direction can be characterized by a personality factor. That is, the label of an eye image acquired when a user wears the target device can include two parts: the gaze direction represented by the image and the influence of the personality factor on the true value of the gaze direction.

[0048] label = image gaze + personalized gaze (1)

[0049] Here, label represents the gaze label corresponding to the image, image gaze represents the gaze direction represented by the image, and personalized gaze represents the influence of personalized factors on the ground truth of the gaze direction.

[0050] In this embodiment, when a user wears the target device, a video of the user's eyes can be acquired through an image acquisition device installed in the target device. When any two image samples are acquired from the video, the difference in labels between the two image samples can be represented in the following way:

[0051] label1_A - label1_B = (image_gaze_1A + personalized_gaze_1) - (image_gaze_1B + personalized_gaze_1)

[0052] =Image gaze_1A - Image gaze_1B(2)

[0053] Where label1_A represents the gaze label corresponding to user 1 in the first gaze direction; label1_B represents the gaze label corresponding to user 1 in the second gaze direction; image gaze_1A represents the gaze direction represented by the image corresponding to user 1; image gaze_1B represents the gaze direction represented by the image corresponding to user 1; and personalized gaze_1 represents the influence of the personalized factor on the ground truth of the gaze direction.

[0054] As can be seen from equation (2) above, the personalized gaze_1 can cancel each other out. Here, the same video refers to the image sequence collected by the same user while wearing the target device.

[0055] However, when two image samples are collected from videos corresponding to different users, the difference in labels between the two image samples can be represented as follows:

[0056] label1_A - label2_B = (image_gaze_1A + personalized_gaze_1) - (image_gaze_2B + personalized_gaze_2) (3)

[0058] Wherein, label1_A represents the gaze label corresponding to user 1 in the first gaze direction, label2_B represents the gaze label corresponding to user 2 in the second gaze direction, image gaze_1A represents the gaze direction represented by the image corresponding to user 1, image gaze_2B represents the gaze direction represented by the image corresponding to user 2, personalized gaze_1 represents the influence of user 1's personality factor on the ground truth of the gaze direction, and personalized gaze_2 represents the influence of user 2's personality factor on the ground truth of the gaze direction. Here, user 1 is equivalent to the first user in the following description, and user 2 is equivalent to the second user in the following description.

[0059] As can be seen from equation (3) above, due to individual differences between User 1 and User 2, the personalized gaze_1 of User 1 and the personalized gaze_2 of User 2 cannot be eliminated. Therefore, in order to use the eye images collected by different users when wearing VR and other devices as the eye images of the same person in different wearing states, it is necessary to eliminate this difference, that is, to bridge the differences between individual factors caused by individual differences. Only in this way can the training data of different users' eye images be used to simulate the different wearing states of the same person, so as to solve the problems caused by the lack or insufficiency of training data.

[0060] In the embodiments provided in this disclosure, two image samples can be collected for different users wearing the target device, denoted as person1_A, person1_B, person2_C, and person2_D, respectively. Here, person1_A and person1_B represent eye images of one user corresponding to two different gaze directions; person2_C and person2_D represent eye images of another user corresponding to two different gaze directions. person1 represents the first user, person2 represents the second user, and A, B, C, and D represent the corresponding gaze directions. It should be noted that the image samples obtained in the embodiments are randomly acquired, and the gaze directions represented by A, B, C, and D can be the same, different, or partially the same; the embodiments are not limited to this.

[0061] In this embodiment, two image samples can be acquired. The first image sample can be an eye image of a first user, and the second image sample can be an eye image of a second user. These two image samples are then input into... Figure 1 The model in the image estimates the gaze direction difference between the two image samples. The eye images of the first and second target objects are labeled with gaze direction information.

[0062] Specifically, inputting person1_A and person2_C into the gaze estimation model outputs the estimated gaze direction difference between the two images, diff_12_AC, i.e., diff_12_AC = F(person1_A, person2_C). The gaze difference label between the two images can be determined by the difference between their labels.

[0063] label1_A - label2_C = (image gaze1_A + personalized gaze_1) - (image gaze2_C + personalized gaze_2) (4)

[0065] Inputting person1_B and person2_D into the gaze estimation model outputs the estimated gaze direction difference between the two images, diff_12_BD, i.e., diff_12_BD = F(person1_B, person2_D). The gaze difference label between the two images can be determined by the difference between their labels.

[0066] label1_B - label2_D = (image gaze1_B + personalized gaze_1) - (image gaze2_D + personalized gaze_2) (5)

[0068] From equations (4) and (5) above, we can obtain:

[0069] label=(label1_A-label2_C)-(label1_B-label2_D)

[0070] =(image_gaze1_A + personalized_gaze_1) -(image_gaze2_C + personalized_gaze_2) –[(image_gaze1_B + personalized_gaze_1) -(image_gaze2_D + personalized_gaze_2)]

[0071] = image gaze1_A - image gaze2_C - image gaze1_B + image gaze2_D (6)

[0073] As can be seen from the above equation (6), the obtained label is equivalent to the difference in the gaze direction between the two images. The above process can cancel out the personalized gaze_1 and personalized gaze_2 corresponding to the first user and the second user respectively, which can avoid the negative impact of different individual differences on gaze estimation.

[0074] Therefore, a loss function can be constructed: loss = diff_12_AC - diff12_BD – label, where label = (label1_A - label2_C) - (label1_B - label2_D), i.e., loss = Loss(diff_12_AC - diff12_BD, label).

[0075] Since label1_A, label1_B, label2_C, and label2_D are the gaze labels corresponding to the four images, the constructed loss function and the eye images collected when the first user wears the target device and the second user wear the target device can be used as training samples. By constructing the above loss function, the model can learn the distribution characteristics of the gaze difference between two sample images under different wearing states during the model training process. By offsetting the individual factors brought about by individual differences of different users, it is possible to achieve the purpose of simulating the eye data of the first user under different wearing states using the eye data of the second user. This allows the model to improve its gaze estimation ability in the scenario where the same user collects calibration images and images to be estimated under different wearing states after training the model with image samples from different users.

[0076] Based on the above embodiments, this disclosure provides a method for constructing a gaze estimation model, such as... Figure 2 As shown, the method may include the following steps:

[0077] In step S210, eye images of the first target object and the second target object are acquired. The eye images of the first and second target objects each correspond to labels containing the direction of gaze.

[0078] In step S220, the estimated gaze difference between the eye image of the first target object and the eye image of the second target object is obtained based on the model to be constructed.

[0079] In this embodiment, the first target object can be represented by a first user, and the second target object can be represented by a second user. The first user's eye image can be obtained when the first user wears the target device, and the second user's eye image can be obtained when the second user wears the target device. As mentioned earlier, due to individual differences, the gaze estimation accuracy of the resulting gaze estimation model will be affected when the second user's eye image is used to simulate the first user's eye image for model training.

[0080] Based on this, in one embodiment provided in this disclosure, the negative impact of the differences in individual factors among different users on the accuracy of gaze estimation can be reduced by estimating the first personality factor corresponding to the first user and the second personality factor corresponding to the second user, and then substituting the estimation results into the loss calculation process. To obtain the first personality factor corresponding to the first user, multiple eye image samples of the first user can be collected. For example, these multiple eye image samples can be collected when the first user is wearing the target device. Alternatively, multiple eye image samples can be obtained by directly capturing images of the first user's eyes using an image acquisition device when the first user is not wearing the target device, thus obtaining higher image quality. During the image acquisition of the first user's eyes, the user can be guided to gaze at various pre-defined fixation points. When the user gazes at a specific fixation point, an eye image sample is acquired at that time. Each acquired eye image sample corresponds to a gaze direction label. This label can be a tag indicating the gaze direction. Using relevant image recognition algorithms, the pupil center can be determined from each high-quality eye image sample to estimate the gaze direction represented by each sample. By subtracting the estimated gaze direction from the corresponding label, multiple differences are obtained. The average of these differences can be used as the first user's personality factor. Similarly, a second personality factor for the second user can be obtained in the same way. These first and second personality factors can then be applied to the training process of the model to be built. This allows the model to be trained using eye images from different users, improving the accuracy of the gaze direction estimation model in predicting the gaze direction of the image. It should be noted that, according to equation (3) above, since the personality factors of the first user and the second user are different, in order to eliminate the influence of the personality factors, in another embodiment provided in this disclosure, the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be eliminated. For example, an eye image from the first user can be obtained to obtain at least two image samples, such as a first image and a second image. At the same time, an eye image from the second user can be obtained to obtain at least two image samples, such as a third image and a fourth image. Of course, the eye image of the first user obtained in the embodiment can be a sequence of images in a video of the user's eyes obtained by an image acquisition device set in the target device when the user is wearing the target device.

[0081] Therefore, in this embodiment, the first image and the third image can be input to... Figure 1The gaze estimation model to be constructed obtains the estimated first gaze difference between the first image and the third image; and the second image and the fourth image are input into the gaze estimation model to obtain the estimated second gaze difference between the second image and the fourth image.

[0082] Specifically, by inputting the first image and the third image into... Figure 1 The gaze estimation model extracts image features from the first image and the third image, respectively. Based on these features, an estimated first gaze difference between the first and third images can be obtained. Similarly, by inputting the second and fourth images... Figure 1 The gaze estimation model extracts image features from the second image and the fourth image, respectively. Based on these features, an estimated second gaze difference between the second and fourth images can be obtained. The difference between the first and second gaze differences is then calculated; this difference represents the gaze difference between the eye images of the first and second target objects.

[0083] Since the first and second images correspond to the first sex factor, and the third and fourth images correspond to the second sex factor, the difference between the first and second gaze differences obtained above can cancel out the first and second sex factors. This can avoid the negative impact on gaze estimation caused by individual differences between different users when wearing the target device.

[0084] In step S230, a loss function is constructed based on the label corresponding to the eye image of the first target object, the label corresponding to the label of the eye image of the second target object, and the gaze difference.

[0085] In the embodiment, the loss function can be constructed after canceling out the first and second personality factors.

[0086] Since the first, second, third, and fourth images each correspond to labels containing the viewing direction, a specific example can be given where the first image contains a first label, the second image contains a second label, the third image contains a third label, and the fourth image contains a fourth label. The first label can be label1_A in the above embodiment, the second label can be label1_B in the above embodiment, the third label can be label2_C in the above embodiment, and the fourth label can be label2_D in the above embodiment. label1_A and label1_B share the same first characteristic factor, and label2_C and label2_D share the same second characteristic factor.

[0087] We can obtain the estimated first gaze difference (diff_12_AC) between the first and third images, the estimated second gaze difference (diff12_BD) between the second and fourth images, the first label difference between the first and third labels, the second label difference between the second and fourth labels, and the difference (label) between the first and second label differences. This allows us to obtain the constructed loss function loss, loss = diff_12_AC - diff12_BD – label.

[0088] In step S240, the model to be constructed is trained based on training samples and a loss function to obtain a gaze estimation model. The training samples include eye images of the first target object and the second target object. During training, a loss value is determined based on the loss function. If the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

[0089] In this embodiment, the model to be constructed is trained by constructing a loss function and using the image sequences of the first user's eyes and the second user's eyes as training samples. For example, when the loss value corresponding to the loss function is minimized, it is determined that the preset convergence condition is met. At this point, training can be stopped, and the trained model to be constructed is obtained. This trained model to be constructed is then used as the gaze estimation model.

[0090] Based on the above embodiments, in another embodiment provided in this disclosure, the method may further include the following steps:

[0091] Step S251: Obtain the training sample set.

[0092] In this embodiment, the training sample set includes at least two subsets, with each subset containing eye images corresponding to a different target object. For example, eye images captured by the same user wearing the target device can be considered as one subset.

[0093] Step S252: Obtain the first image sample and the second image sample from the target subset in the training sample set.

[0094] In this embodiment, the first image sample and the second image sample can be any two images in the target subset, which can be any subset of the training sample set.

[0095] Step S253: Obtain the target label difference between the label of the first image sample corresponding to the viewing direction and the label of the second image sample corresponding to the viewing direction, and construct a first target loss function based on the target label difference.

[0096] In the embodiment, the first image sample and the second image sample contain labels corresponding to the viewing direction, and a loss function can be constructed by the difference between the two labels.

[0097] Step S254: Based on the first image sample, the second image sample, and the first target loss function, train the model to be built and update the model parameters of the model to be built.

[0098] In this embodiment, the model to be built can be trained using first and second image samples. If the loss value of the first objective loss function does not meet a preset convergence condition, the model parameters are updated. This allows for rapid training of the model using eye images of the same user, improving the model's gaze estimation capability under the same wearing conditions.

[0099] Step S255: Obtain the third image sample from the first subset and the fourth image sample from the second subset in the training sample set.

[0100] In this embodiment, the first subset and the second subset can be any two subsets in the training sample set, the third image sample can be any image sample in the first subset, and the fourth image sample can be any image sample in the second subset.

[0101] Step S256: Obtain the estimated target gaze difference between the third image sample and the fourth image sample based on the model to be constructed.

[0102] In this embodiment, the third and fourth image samples can be input into the model to be built to obtain the estimated target gaze difference between the third and fourth image samples.

[0103] Step S257: Based on the labels of the gaze direction corresponding to the third image sample, the labels of the gaze direction corresponding to the fourth image sample, and the target gaze difference, construct a second target loss function, and train the model to be constructed based on the third image sample, the fourth image sample, and the second target loss function, and update the model parameters of the model to be constructed.

[0104] In this embodiment, the model to be constructed is trained using third and fourth image samples. During training, a loss value is determined based on a second objective loss function. If the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. Retraining the model using eye images from different users can improve the accuracy of gaze estimation under different wearing conditions.

[0105] It should be noted that the construction process of the second objective loss function is the same as the construction process of the loss function in step S230 above. For details, please refer to the construction process of the loss function above, which will not be repeated here.

[0106] Based on the above embodiments, in another embodiment provided in this disclosure, a line-of-sight estimation method is also provided, which may include the following steps:

[0107] S310, acquire the calibration image. The calibration image corresponds to a known line-of-sight direction.

[0108] In this embodiment, the calibration image is an eye image obtained by the user while wearing the target device, and the gaze direction of the calibration image is known. The acquisition of both the calibration image and the image to be estimated is performed with the user's authorization and consent. For example, an application on the target acquisition device can receive a user's request for gaze estimation and, with the user's authorization and consent to acquire the eye image, acquire the user's eye image as the image to be estimated in order to estimate the gaze direction of the image to be estimated.

[0109] S320, acquire the image to be estimated.

[0110] In this embodiment, the image to be estimated may be an eye image acquired when the user wears the target device again.

[0111] S330, based on the calibration image, the image to be estimated, and the gaze estimation model, obtain the estimated gaze direction of the image to be estimated.

[0112] The calibration image and the image to be estimated can be input into the gaze estimation model trained according to the above embodiments to obtain the estimated gaze direction of the image to be estimated.

[0113] In the embodiment, the gaze estimation model obtained after training can learn the distribution characteristics of the gaze difference between two sample images under different wearing states. By offsetting the individual factors brought about by individual differences of different users, it can achieve the purpose of simulating the eye data of the same user under different wearing states using the eye data of different users. This allows the model to be trained by image samples from different users, thereby improving the accuracy of gaze estimation in the scenario where the same user collects calibration images and images to be estimated under different wearing states.

[0114] By dividing each functional module according to its corresponding function, this disclosure provides an apparatus for constructing a line-of-sight estimation model. The apparatus for constructing a line-of-sight estimation model can be a server or a chip applied to a server. Figure 3 A schematic block diagram of the functional modules of an apparatus for constructing a line-of-sight estimation model provided as an exemplary embodiment of this disclosure. Figure 3 As shown, the apparatus for constructing the line-of-sight estimation model includes:

[0115] The image acquisition module 10 is used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object are respectively labeled with a gaze direction.

[0116] The gaze difference acquisition module 20 is used to acquire the estimated gaze difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0117] The loss function construction module 30 is used to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the gaze difference.

[0118] The model training module 40 is used to train the model to be constructed based on training samples and the loss function to obtain a gaze estimation model; wherein, the training samples include eye images of the first target object and eye images of the second target object, during the training process, a loss value is determined based on the loss function, if the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated, and if the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

[0119] Optionally, the eye image of the first target object includes a first image and a second image, and the eye image of the second target object includes a third image and a fourth image; the gaze difference acquisition module is specifically used for:

[0120] Based on the first image and the third image, an estimated first gaze difference is obtained, and based on the second image and the fourth image, an estimated second gaze difference is obtained;

[0121] The estimated line-of-sight difference is determined based on the difference between the first line-of-sight difference and the second line-of-sight difference.

[0122] Optionally, the line-of-sight difference acquisition module is further used for:

[0123] Based on the first image, the third image, and the model to be constructed, the first gaze difference is obtained;

[0124] The second line-of-sight difference is obtained based on the second image, the fourth image, and the model to be constructed.

[0125] Optionally, the first image corresponds to a label containing a first viewing direction, the second image corresponds to a label containing a second viewing direction, the third image corresponds to a label containing a third viewing direction, and the fourth image corresponds to a label containing a fourth viewing direction; the loss function construction module is specifically used for:

[0126] Obtain a first label difference between the label in the first viewing direction and the label in the third viewing direction, and obtain a second label difference between the label in the second viewing direction and the label in the fourth viewing direction;

[0127] Obtain the label difference between the first label difference and the second label difference;

[0128] A loss function is constructed based on the label difference and the line-of-sight difference.

[0129] Based on the above embodiments, in another embodiment provided in this disclosure, the model training module is further configured to:

[0130] Obtain a training sample set; the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects;

[0131] Obtain the first image sample and the second image sample from the target subset in the training sample set;

[0132] Obtain the target label difference between the label of the first image sample corresponding to the viewing direction and the label of the second image sample corresponding to the viewing direction, and construct a first target loss function based on the target label difference;

[0133] Based on the first image sample, the second image sample, and the first target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0134] Obtain the third image sample from the first subset and the fourth image sample from the second subset of the training sample set;

[0135] Based on the model to be constructed, the estimated target gaze difference between the third image sample and the fourth image sample is obtained;

[0136] Based on the labels of the gaze direction corresponding to the third image sample, the labels of the gaze direction corresponding to the fourth image sample, and the target gaze difference, a second target loss function is constructed. Based on the third image sample, the fourth image sample, and the second target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0137] Based on the above embodiments, in another embodiment provided in this disclosure, a line-of-sight estimation device is provided, the device comprising:

[0138] A calibration image acquisition module is used to acquire a calibration image; the calibration image includes the line-of-sight direction.

[0139] The image acquisition module is used to acquire the image to be estimated.

[0140] The gaze estimation module is used to obtain the estimated gaze direction of the image to be estimated based on the calibration image, the image to be estimated, and the gaze estimation model. The gaze estimation model can be constructed using the method provided in the above embodiments.

[0141] For details regarding the apparatus, please refer to the description of the above embodiments, which will not be repeated here.

[0142] The apparatus provided in this disclosure acquires eye images of a first target object and a second target object, and obtains the gaze difference between the eye images of the first target object and the second target object based on the model to be constructed. A loss function is constructed using the labels corresponding to the eye images of the first target object, the labels corresponding to the eye images of the second target object, and the gaze difference. By training the model to be constructed based on training samples and the loss function, the resulting gaze estimation model can offset the influence of individual differences between different users, enabling the eye image of the first target object to be simulated using the eye data of the second target object, thereby improving the accuracy of the gaze direction estimated by the gaze estimation model.

[0143] This disclosure also provides an electronic device, including: at least one processor; a memory for storing processor-executable instructions; wherein the at least one processor is configured to execute the instructions to implement the methods disclosed in this disclosure.

[0144] Figure 4 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 4 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0145] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the method described above.

[0146] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 5 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 5 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0147] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 5As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0149] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0150] The computing unit 1901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1900 via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0151] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0152] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0154] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0155] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0158] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0159] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0160] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for constructing a gaze estimation model, characterized in that, The method includes: Obtain eye images of a first target object and a second target object; wherein the eye images of the first target object and the second target object respectively correspond to labels containing the direction of gaze; Based on the model to be constructed, the estimated gaze difference between the eye image of the first target object and the eye image of the second target object is obtained; A loss function is constructed based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the gaze difference; The model to be constructed is trained based on the training samples and the loss function to obtain a gaze estimation model; wherein, the training samples include eye images of the first target object and eye images of the second target object, during the training process, the loss value is determined based on the loss function, if the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated, and if the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

2. The method according to claim 1, characterized in that, The first target object's eye image includes a first image and a second image, and the second target object's eye image includes a third image and a fourth image; obtaining the estimated gaze difference between the first target object's eye image and the second target object's eye image includes: Based on the first image and the third image, an estimated first gaze difference is obtained, and based on the second image and the fourth image, an estimated second gaze difference is obtained; The estimated line-of-sight difference is determined based on the difference between the first line-of-sight difference and the second line-of-sight difference.

3. The method according to claim 2, characterized in that, The step of obtaining an estimated first gaze difference based on the first image and the second image, and obtaining an estimated second gaze difference based on the second image and the fourth image, includes: Based on the first image, the third image, and the model to be constructed, the first gaze difference is obtained; The second line-of-sight difference is obtained based on the second image, the fourth image, and the model to be constructed.

4. The method according to claim 2, characterized in that, The first image corresponds to a label containing a first viewing direction, the second image corresponds to a label containing a second viewing direction, the third image corresponds to a label containing a third viewing direction, and the fourth image corresponds to a label containing a fourth viewing direction. The construction of the loss function includes: Obtain a first label difference between the label in the first viewing direction and the label in the third viewing direction, and obtain a second label difference between the label in the second viewing direction and the label in the fourth viewing direction; Obtain the label difference between the first label difference and the second label difference; A loss function is constructed based on the label difference and the line-of-sight difference.

5. The method according to claim 1, characterized in that, The method includes: Obtain a training sample set; the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects; Obtain the first image sample and the second image sample from the target subset in the training sample set; Obtain the target label difference between the label of the first image sample corresponding to the viewing direction and the label of the second image sample corresponding to the viewing direction, and construct a first target loss function based on the target label difference; Based on the first image sample, the second image sample, and the first target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated. Obtain the third image sample from the first subset and the fourth image sample from the second subset of the training sample set; Based on the model to be constructed, the estimated target gaze difference between the third image sample and the fourth image sample is obtained; Based on the labels of the gaze direction corresponding to the third image sample, the labels of the gaze direction corresponding to the fourth image sample, and the target gaze difference, a second target loss function is constructed. Based on the third image sample, the fourth image sample, and the second target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

6. A line-of-sight estimation method, characterized in that, The method includes: Acquire a calibration image, which corresponds to a known line-of-sight direction; Obtain the image to be estimated; Based on the calibration image, the image to be estimated, and the gaze estimation model, the gaze direction estimated for the image to be estimated is obtained; wherein the gaze estimation model is obtained by the method described in any one of claims 1 to 5.

7. A device for constructing a gaze estimation model, characterized in that, The device includes: An image acquisition module is used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object are respectively labeled with a gaze direction. The gaze difference acquisition module is used to acquire the estimated gaze difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed. The loss function construction module is used to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the gaze difference. The model training module is used to train the model to be constructed based on training samples and the loss function to obtain a gaze estimation model. The training samples include eye images of the first target object and eye images of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

8. A line-of-sight estimation device, characterized in that, The device includes: A calibration image acquisition module is used to acquire a calibration image; the calibration image corresponds to a known line-of-sight direction; The image acquisition module is used to acquire the image to be estimated. A gaze estimation module is used to obtain the estimated gaze direction of the image to be estimated based on the calibration image, the image to be estimated, and the gaze estimation model; wherein the gaze estimation model is obtained by the method described in any one of claims 1 to 5.

9. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

Images