Gaze estimation model construction method and apparatus, and gaze estimation method and apparatus

Abstract

The present disclosure relates to a gaze estimation model construction method and apparatus, and a gaze estimation method and apparatus. The gaze estimation model construction method comprises: obtaining an eye image of a first target object and an eye image of a second target object, wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels containing gaze directions; obtaining an estimated gaze difference between the eye image of the first target object and the eye image of the second target object on the basis of a model to be constructed; constructing a loss function on the basis of the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the gaze difference; and on the basis of a training sample and the loss function, training the model to be constructed to obtain a gaze estimation model.

Description

Method for constructing sight line estimation model, sight line estimation method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to the Chinese patent application with application number 202310876838.6 and application date July 17, 2023. The disclosed content of the Chinese patent application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a method for constructing a sight line estimation model, a sight line estimation method, and a sight line estimation device. Background Art

[0004] Gaze provides a crucial clue to how people perceive the external environment. Gaze estimation technology, based on eye gaze, calculates the user's gaze direction using captured eye images. This technology has found numerous applications in human-computer interaction, virtual reality, augmented reality, medical analysis, and other fields.

[0005] Summary of the Invention

[0006] The present disclosure provides a method for constructing a line of sight estimation model, a line of sight estimation method, and a device.

[0007] According to one aspect of the present disclosure, a method for constructing a line of sight estimation model is provided, the method comprising:

[0008] Acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels including gaze directions;

[0009] Obtaining an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0010] constructing a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference;

[0011] The model to be constructed is trained based on the training samples and the loss function to obtain a line of sight estimation model; wherein, the training samples include the eye image of the first target object and the eye image of the second target object. During the training process, the loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0012] According to a second aspect of the present disclosure, a line of sight estimation method is provided, the method comprising:

[0013] Acquire a calibration image; the calibration image corresponds to a known line of sight direction;

[0014] Obtain the image to be estimated;

[0015] Based on the calibration image, the image to be estimated and the line of sight estimation model, an estimated line of sight direction of the image to be estimated is obtained; wherein the line of sight estimation model is obtained by the method described above.

[0016] According to a third aspect of the present disclosure, a device for constructing a line of sight estimation model is provided, the device comprising:

[0017] An image acquisition module, configured to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a gaze direction;

[0018] a sight line difference acquisition module, configured to acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0019] a loss function construction module, configured to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference;

[0020] A model training module is used to train the model to be constructed based on training samples and the loss function to obtain a line of sight estimation model; wherein, the training samples include the eye image of the first target object and the eye image of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0021] According to a fourth aspect of the present disclosure, a sight line estimation device is provided, the device comprising:

[0022] A calibration image acquisition module, configured to acquire a calibration image corresponding to a known sight line direction;

[0023] An image acquisition module to be estimated is used to acquire the image to be estimated;

[0024] A sight line estimation module is used to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated and a sight line estimation model; wherein the sight line estimation model is obtained by the method described above.

[0025] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the program.

[0026] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0028] FIG1 is a schematic diagram of a line-of-sight estimation model provided by an exemplary embodiment of the present disclosure;

[0029] FIG2 is a flowchart of a method for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure;

[0030] FIG3 is a schematic block diagram of functional modules of a device for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure;

[0031] FIG4 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure;

[0032] FIG5 is a structural block diagram of a computer system provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0039] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to receiving a user's active request, the method for sending a prompt to the user can be, for example, a pop-up window, in which the prompt information can be presented in text form. Furthermore, the pop-up window can also contain a selection control for the user to select "Agree" or "Disagree" to provide personal information to the electronic device. It will be understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0041] When a user wears relevant equipment, such as AR (Augmented Reality), VR (Virtual Reality) or MR (Mixed Reality), with the user's authorization and consent, images of the user's eyes can be collected in real time by an image acquisition device set in the target device, and the gaze direction can be estimated for the collected eye image. For example, local fine rendering of the scene displayed in the target device can be achieved based on the estimated gaze direction, that is, fine rendering can be performed only on the scene within the user's gaze range, so as to reduce the hardware cost of the target device. However, when the gaze direction is directly estimated from the eye image, the estimated gaze direction may be inaccurate.

[0042] When estimating the gaze direction of an image of a user's eye, related technologies can input a calibration image and an image to be estimated into a differential gaze estimation model to obtain an estimated gaze difference between the calibration image and the image to be estimated, and estimate the gaze direction of the image to be estimated based on the gaze difference and the known gaze direction corresponding to the calibration image. Since the calibration image may be an image captured when the user first wears the target device, and the image to be estimated may be an image captured when the user wears the target device again, due to differences in wearing positions between different wears, these differences will affect the estimated gaze direction of the image to be estimated, resulting in inaccurate estimates of the gaze direction of the image to be estimated.

[0043] As shown in Figure 1, a gaze estimation model can extract image features from images and, based on the extracted features, obtain an estimated gaze difference between two images. Two images can be obtained: a calibration image and an image to be estimated. These two images are input into the gaze estimation model, which outputs an estimated gaze difference between the gaze directions of the two images. The calibration image is captured by guiding a user to gaze at a calibration point with a known gaze direction. Therefore, the calibration image corresponds to a known gaze direction. Combined with this difference, the true gaze direction of the image to be estimated can be estimated. It should be noted that in the embodiments, the images can be image samples. These image samples can include a single-eye image of a user, such as a left eye image or a right eye image; they can also include binocular images of a user, such as a left eye image and a right eye image; or they can be a spliced ​​image of a user's left eye image and right eye image. The embodiments are not limited to this.

[0044] Therefore, in order to improve the accuracy of the gaze estimation model in estimating the gaze direction, it is necessary to obtain eye images of the same user wearing the target device at different times, so that the gaze estimation model can learn the impact of different wearing states on the gaze direction estimation through training. However, in practice, if a large number of eye images of the same user in different wearing states are required as image samples, the acquisition cost of the training samples will be significantly increased. In addition, the study found that due to individual differences between different users, such individual differences include differences in the eyeball structure of different individuals, such as the angle between the optical axis and the visual axis (where the eye image represents the direction of the optical axis of the eye, and the direction of the visual axis of the eye is the actual gaze direction), and such individual differences cannot be eliminated. Therefore, if the eye images obtained when different users wear the target device are used to simulate the eye images of the same user in different wearing states for model training, the gaze estimation accuracy of the obtained gaze estimation model cannot be guaranteed.

[0045] For example, when training the gaze estimation model using the two image samples mentioned above, the gaze labels corresponding to the image samples include the gaze direction represented by the image and the influence of the individual factors caused by the aforementioned individual differences on the true value of the gaze direction. If the gaze direction of the image to be estimated is estimated using a non-differential gaze estimation model, the corresponding estimated gaze direction can be directly output based on a single image sample. The model's learned results can be divided into two cases. The first case is that the image samples come from the same user. The non-differential gaze estimation model learns the gaze direction represented by the image and the influence of the individual factors caused by the individual differences of the user on the true value of the gaze direction. During the testing phase, if the gaze direction is estimated for that user, the gaze estimation of the image to be estimated in the test results is relatively accurate because both the training samples and the image to be estimated during the test phase come from the same user. However, if the image to be estimated during the test process does not come from the same user as the image samples used during the training phase, due to individual differences between different individuals, the individual factors caused by the individual differences between users will affect the estimated gaze direction, resulting in significant deviations in the estimated gaze direction.

[0046] If a differential gaze estimation model is used to estimate the gaze direction of an image to be estimated, the model learns the gaze difference between the two image samples. The true value of this gaze difference can include both the gaze difference represented by the images and the difference in personality factors. If the model inputs images from the same user, the difference in personality factors is zero, making the output unaffected by individual differences. However, if the model inputs come from two different users, the difference in personality factors will not be zero, and the user's personality factors are unknown. This means that individual differences between users may negatively impact gaze estimation.

[0047] Therefore, when training the model through two image samples to obtain a differential gaze estimation model, if the first image sample is an eye image collected when the first user wears the target device, and the second image sample is an eye image collected when the second user wears the target device, since the first user and the second user are different users, and there will be individual differences between different users as mentioned above, when training the model through eye images of different users, if the influence of the individual differences on the gaze direction cannot be eliminated, the gaze estimation accuracy of the trained gaze estimation model will be reduced.

[0048] For example, due to the aforementioned individual differences among different users, the influence of the personalized eye structure of different users on the gaze direction can be represented by a personality factor. That is, the label of an eye image collected when a user wears a target device can include two parts, namely, the gaze direction represented by the image and the influence of the personality factor on the true value of the gaze direction: label = image gaze + personalized gaze (1)

[0049] Among them, label represents the gaze label corresponding to the image, image gaze represents the gaze direction represented by the image, and personalized gaze represents the influence of personality factors on the true value of the gaze direction.

[0050] In the embodiment, when a user wears the target device, an image acquisition device provided in the target device can be used to acquire a video of the user's eyes. When two image samples are arbitrarily acquired from the video, the label difference between the two image samples can be expressed as follows: label1_A-label1_B=(image gaze_1A+personalized gaze_1)-(image gaze_1B+personalized gaze_1)=image gaze_1A-image gaze_1B (2)

[0051] Among them, label1_A represents the gaze label corresponding to user 1 in the first gaze direction; label1_B represents the gaze label corresponding to user 1 in the second gaze direction; image gaze_1A represents the gaze direction corresponding to the image representation of user 1, image gaze_1B represents the gaze direction corresponding to the image representation of user 1, and personalized gaze_1 represents the influence of the personality factor on the true value of the gaze direction.

[0052] From the above formula (2), we can see that personalized gaze_1 can cancel each other out. Here, the same video refers to the image sequence collected by the same user while wearing the target device.

[0053] However, when two image samples are collected from videos corresponding to different users, the label difference between the two image samples can be expressed as follows: label1_A-label2_B=(image gaze_1A+personalized gaze_1)-(image gaze_2B+personalized gaze_2) (3)

[0054] Here, label1_A represents the gaze label corresponding to user 1's first gaze direction, label2_B represents the gaze label corresponding to user 2's second gaze direction, image gaze_1A represents the gaze direction represented by the image for user 1, image gaze_2B represents the gaze direction represented by the image for user 2, personalized gaze_1 represents the influence of user 1's personality factors on the true value of the gaze direction, and personalized gaze_2 represents the influence of user 2's personality factors on the true value of the gaze direction. User 1 corresponds to the first user in the following description, and user 2 corresponds to the second user in the following description.

[0055] From the above formula (3), we can see that due to the individual differences between user 1 and user 2, user 1's personalized gaze_1 and user 2's personalized gaze_2 cannot be eliminated. Therefore, in order to use the eye images collected when different users wear VR devices as eye images of the same person wearing different devices, it is necessary to eliminate this difference, that is, to bridge the differences between the personality factors caused by individual differences. In this way, we can use the eye images of different users to simulate the training data of the same person in different wearing states, so as to solve the above problems caused by the lack or insufficiency of training data.

[0056] In the embodiments provided herein, two image samples can be collected for different users wearing the target device, represented by person1_A, person1_B, person2_C, and person2_D, respectively. Person1_A and person1_B represent eye images of one user corresponding to two gaze directions, respectively; person2_C and person2_D represent eye images of another user corresponding to two gaze directions, respectively. Person1 represents the first user, person2 represents the second user, and A, B, C, and D represent corresponding gaze directions, respectively. It should be noted that the image samples acquired in the embodiments are randomly acquired, and the gaze directions represented by A, B, C, and D can be the same, different, or partially the same, respectively, and the embodiments are not limited thereto.

[0057] In this embodiment, two image samples can be obtained. The first image sample can be an eye image of a first user, and the second image sample can be an eye image of a second user. By inputting these two image samples into the gaze estimation model in Figure 1, the gaze estimation model estimates the gaze difference between the gaze directions of the two image samples. The eye image of the first target object and the eye image of the second target object are each labeled with a gaze direction.

[0058] Specifically, inputting person1_A and person2_C into the gaze estimation model can output the estimated gaze direction difference between the two images, diff_12_AC, i.e., diff_12_AC = F(person1_A, person2_C). The gaze difference label between the two can be determined by the difference between the two labels: label1_A - label2_C = (image gaze1_A + personalized gaze_1) - (image gaze2_C + personalized gaze_2) (4)

[0059] Inputting person1_B and person2_D into the gaze estimation model can output the estimated gaze direction difference between the two images, diff_12_BD, i.e., diff_12_BD = F(person1_B, person2_D). The gaze difference label between the two can be determined by the difference between the two labels: label1_B - label2_D = (image gaze1_B + personalized gaze_1) - (image gaze2_D + personalized gaze_2) (5)

[0060] Through the above formulas (4) and (5), we can get: label = (label1_A-label2_C)-(label1_B-label2_D) = (image gaze1_A+personalized gaze_1)-(image gaze2_C+personalized gaze_2)-[(image gaze1_B+personalized gaze_1)-(image gaze2_D+personalized gaze_2)] = image gaze1_A-image gaze2_C-image gaze1_B+image gaze2_D (6)

[0061] From the above formula (6), we can see that the obtained label is equivalent to the difference between the gaze direction differences represented by the two images. The above process can offset the personalized gaze_1 and personalized gaze_2 corresponding to the first user and the second user respectively, and can avoid the negative impact of individual differences on gaze estimation.

[0062] Therefore, the loss function loss = diff_12_AC - diff12_BD–label can be constructed, where label = (label1_A - label2_C) - (label1_B - label2_D), that is, loss = Loss (diff_12_AC - diff12_BD, label).

[0063] Since label1_A, label1_B, label2_C and label2_D are the sight labels corresponding to the four images respectively, the constructed loss function and the eye images collected when the first user wears the target device and the eye images collected when the second user wears the target device can be used as training samples. By constructing the above loss function, the distribution characteristics of the sight line difference between the two sample images under different wearing states can be learned during the model training process. By offsetting the personality factors brought about by individual differences of different users, the purpose of simulating the eye data of the first user under different wearing states through the eye data of the second user can be achieved. After the model is trained with image samples from different users, the model's sight line estimation ability in the scenario where the same user collects calibration images and images to be estimated under different wearing states can be improved.

[0064] Based on the above embodiments, the present disclosure provides a method for constructing a line of sight estimation model. As shown in FIG2 , the method may include the following steps:

[0065] In step S210, an eye image of the first target object and an eye image of the second target object are acquired, wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a gaze direction.

[0066] In step S220 , an estimated sight line difference between the eye image of the first target object and the eye image of the second target object is obtained based on the model to be constructed.

[0067] In this embodiment, the first target object may be represented by a first user, and the second target object may be represented by a second user. An eye image of the first user may be obtained when the first user wears the target device, and an eye image of the second user may be obtained when the second user wears the target device. As previously mentioned, due to individual differences between individuals, when the eye image of the second user is used to simulate the eye image of the first user for model training, the gaze estimation accuracy of the resulting gaze estimation model may be affected.

[0068] Based on this, in one embodiment provided by the present disclosure, the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be estimated, and the estimated results can be substituted into the loss calculation process to reduce the negative impact of the personality factor differences between the above-mentioned different users on the accuracy of line of sight estimation. In order to obtain the first personality factor corresponding to the first user, multiple eye image samples of the first user can be collected. For example, the multiple eye image samples can be collected when the first user wears the target device. Of course, the image acquisition device can also be used to capture the eyes of the first user when the first user is not wearing the target device to obtain multiple eye image samples of the first user. In this way, the image quality of the image samples obtained is higher. During the image acquisition process of the first user's eyes, the first user can be guided to gaze at each pre-set gaze point, and eye image samples of the first user at the corresponding moment are collected when the first user gazes at the corresponding gaze point. Therefore, each collected eye image sample is respectively associated with gaze annotation information, and the gaze annotation information can be a label corresponding to the gaze direction. In this way, the pupil center can be determined from each high-quality eye image sample through a relevant image recognition algorithm to estimate the gaze direction represented by each sample image sample. By subtracting the estimated gaze direction from the corresponding label, multiple difference values ​​can be obtained, and the average of the multiple difference values ​​can be used as the personality factor of the first user. Similarly, the second personality factor corresponding to the second user can be obtained in the above manner. In this way, the obtained first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be applied to the training process of the model to be constructed, so that the model can be trained using eye images from different users to improve the accuracy of the gaze direction of the image to be estimated estimated by the constructed gaze estimation model. It should be noted that, according to the above formula (3), since the personality factors of the first user and the second user are different, in order to eliminate the influence of the personality factors, in another embodiment provided by the present disclosure, the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be eliminated. For example, the eye image from the first user can be obtained to obtain at least two image samples, for example, including the first image and the second image, and the eye image from the second user can be obtained to obtain at least two image samples, for example, including the third image and the fourth image. Of course, the eye image of the first user obtained in the embodiment can be a sequence of images in a video of the user's eyes that can be obtained by an image acquisition device provided in the target device when the user wears the target device.

[0069] Therefore, in an embodiment, the first image and the third image can be input into the line of sight estimation model to be constructed in Figure 1 to obtain the estimated first line of sight difference between the first image and the third image; and the second image and the fourth image can be input into the line of sight estimation model to obtain the estimated second line of sight difference between the second image and the fourth image.

[0070] Specifically, by inputting the first image and the third image into the line of sight estimation model in FIG1 , the image features of the first image and the image features of the third image are extracted respectively, and based on the image features of the first image and the image features of the third image, a first line of sight difference value estimated between the first image and the third image can be obtained. Similarly, by inputting the second image and the fourth image into the line of sight estimation model in FIG1 , the image features of the second image and the image features of the fourth image are extracted respectively, and based on the image features of the second image and the image features of the fourth image, a second line of sight difference value estimated between the second image and the fourth image can be obtained. The difference between the first line of sight difference value and the second line of sight difference value is obtained, and the difference is the line of sight difference value between the eye image of the first target object and the eye image of the second target object.

[0071] Since the first image and the second image correspond to the first characteristic factor respectively, and the third image and the fourth image correspond to the second characteristic factor respectively, the difference between the first line of sight difference value obtained above and the second line of sight difference value obtained above can offset the first characteristic factor and the second characteristic factor, thereby avoiding the negative impact on line of sight estimation caused by individual differences between different users when wearing the target device.

[0072] In step S230 , a loss function is constructed based on the label corresponding to the eye image of the first target object, the label corresponding to the label of the eye image of the second target object, and the sight line difference.

[0073] In the embodiment, after the first personality factor and the second personality factor are offset, the loss function can be constructed.

[0074] Since the first, second, third, and fourth images each correspond to a label containing a gaze direction, the first image corresponds to the first label, the second image corresponds to the second label, the third image corresponds to the third label, and the fourth image corresponds to the fourth label as an example. The first label can be label1_A in the above implementation, the second label can be label1_B in the above implementation, the third label can be label2_C in the above implementation, and the fourth label can be label2_D in the above implementation. Label1_A and label1_B have the same first characteristic factor, and label2_C and label2_D have the same second characteristic factor.

[0075] The estimated first sight line difference diff_12_AC between the first image and the third image, the estimated second sight line difference diff12_BD between the second image and the fourth image can be obtained, and the first label difference between the first label and the third label, the second label difference between the second label and the fourth label, and the difference label between the first label difference and the second label difference can be obtained. In this way, the constructed loss function loss can be obtained, loss = diff_12_AC - diff12_BD – label.

[0076] In step S240, the model to be constructed is trained based on the training samples and the loss function to obtain a gaze estimation model. The training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the gaze estimation model.

[0077] In an embodiment, the model to be constructed is trained by constructing a loss function and using the image sequence of the first user's eyes and the image sequence of the second user's eyes as training samples. For example, when the loss value corresponding to the loss function is minimized, it is determined that the preset convergence condition is met. At this time, the training can be stopped to obtain the trained model to be constructed, and the trained model to be constructed is used as a line of sight estimation model.

[0078] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0079] Step S251: Obtain a training sample set.

[0080] In an embodiment, the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects. For example, eye images collected by the same user wearing the target device can be used as a subset.

[0081] Step S252: Acquire a first image sample and a second image sample in a target subset of a training sample set.

[0082] In an embodiment, the first image sample and the second image sample may be any two images in a target subset, and the target subset may be any subset in a training sample set.

[0083] Step S253: Obtain a target label difference between the label of the first image sample corresponding to the line of sight direction and the label of the second image sample corresponding to the line of sight direction, and construct a first target loss function based on the target label difference.

[0084] In an embodiment, the first image sample and the second image sample include labels corresponding to the viewing directions, respectively, and a loss function can be constructed by the difference between the two labels.

[0085] Step S254: Based on the first image sample, the second image sample and the first target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0086] In this embodiment, the model to be constructed can be trained using the first and second image samples. If the loss value of the first objective loss function does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. This allows the model to be quickly trained using the same user's eye images, improving the model's line of sight estimation capabilities under the same wearing conditions.

[0087] Step S255 : Acquire the third image sample in the first subset and the fourth image sample in the second subset of the training sample set.

[0088] In the embodiment, the first subset and the second subset may be any two subsets in the training sample set respectively, the third image sample may be any image sample in the first subset, and the fourth image sample may be any image sample in the second subset.

[0089] Step S256 : obtaining an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed.

[0090] In an embodiment, the third image sample and the fourth image sample may be input into the model to be constructed to obtain an estimated target sight line difference between the third image sample and the fourth image sample.

[0091] Step S257, constructing a second target loss function based on the label of the third image sample corresponding to the line of sight direction, the label of the fourth image sample corresponding to the line of sight direction and the target line of sight difference, and training the model to be constructed based on the third image sample, the fourth image sample and the second target loss function, and updating the model parameters of the model to be constructed.

[0092] In this embodiment, the model to be constructed is trained using the third and fourth image samples. During the training process, a loss value is determined based on the second objective loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. Further training of the model to be constructed using eye images of different users can improve the accuracy of the model's line of sight estimation under different wearing conditions.

[0093] It should be noted that the construction process of the second objective loss function is consistent with the construction process of the loss function in the above step S230. For details, please refer to the construction process of the above loss function, which will not be repeated here.

[0094] Based on the above embodiment, in another embodiment provided by the present disclosure, a line of sight estimation method is further provided, which may include the following steps:

[0095] S310: Acquire a calibration image, wherein the calibration image corresponds to a known sight direction.

[0096] In this embodiment, the calibration image is an eye image obtained by the user wearing the target device, and the gaze direction of the calibration image is known. The acquisition of the calibration image and the image to be estimated is performed with the user's authorization and consent. For example, an application on the target acquisition device can receive a user's request for gaze estimation, obtain the user's authorization and consent to acquire an eye image, and then acquire the user's eye image as the image to be estimated, in order to estimate the gaze direction of the image to be estimated.

[0097] S320: Acquire an image to be estimated.

[0098] In an embodiment, the image to be estimated may be an eye image acquired when the user wears the target device another time.

[0099] S330 : Obtaining an estimated line of sight direction of the image to be estimated based on the calibration image, the image to be estimated, and a line of sight estimation model.

[0100] The calibration image and the image to be estimated can be input into the sight line estimation model obtained after training according to the above embodiment, and the estimated sight line direction of the image to be estimated can be obtained.

[0101] In an embodiment, the line of sight estimation model obtained after the above training can learn the distribution characteristics of the line of sight difference between two sample images under different wearing states. By offsetting the personality factors brought about by individual differences of different users, it can achieve the purpose of simulating the eye data of the same user under different wearing states through the eye data of different users. After the model is trained with image samples from different users, the accuracy of the line of sight estimation of the model in the scenario where the calibration image and the image to be estimated are collected by the same user under different wearing states can be improved.

[0102] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a device for constructing a sight line estimation model. The device for constructing the sight line estimation model can be a server or a chip applied to a server. Figure 3 is a schematic block diagram of the functional modules of the device for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure. As shown in Figure 3, the device for constructing the sight line estimation model includes:

[0103] An image acquisition module 10 is configured to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a gaze direction;

[0104] a sight line difference acquisition module 20, configured to acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0105] a loss function construction module 30, configured to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference;

[0106] The model training module 40 is used to train the model to be constructed based on the training samples and the loss function to obtain a line of sight estimation model; wherein, the training samples include the eye image of the first target object and the eye image of the second target object. During the training process, the loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0107] Optionally, the eye image of the first target object includes a first image and a second image, and the eye image of the second target object includes a third image and a fourth image; and the sight difference value acquisition module is specifically configured to:

[0108] Obtaining an estimated first sight line difference value based on the first image and the third image, and obtaining an estimated second sight line difference value based on the second image and the fourth image;

[0109] The estimated sight line difference value is determined based on a difference between the first sight line difference value and the second sight line difference value.

[0110] Optionally, the sight difference value acquisition module is further configured to:

[0111] Obtaining the first sight difference value based on the first image, the third image, and the model to be constructed;

[0112] The second sight line difference value is obtained based on the second image, the fourth image and the model to be constructed.

[0113] Optionally, the first image corresponds to a label containing a first sight line direction, the second image corresponds to a label containing a second sight line direction, the third image corresponds to a label containing a third sight line direction, and the fourth image corresponds to a label containing a fourth sight line direction; the loss function construction module is specifically used to:

[0114] Obtaining a first label difference between the label in the first sight direction and the label in the third sight direction, and obtaining a second label difference between the label in the second sight direction and the label in the fourth sight direction;

[0115] Obtaining a label difference between the first label difference and the second label difference;

[0116] A loss function is constructed based on the label difference and the sight difference.

[0117] Based on the above embodiment, in another embodiment provided by the present disclosure, the model training module is further configured to:

[0118] Acquire a training sample set; the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects;

[0119] Acquire a first image sample and a second image sample in a target subset of the training sample set;

[0120] Obtaining a target label difference between a label of the first image sample corresponding to the line of sight direction and a label of the second image sample corresponding to the line of sight direction, and constructing a first target loss function based on the target label difference;

[0121] Based on the first image sample, the second image sample and the first objective loss function, training the model to be constructed, and updating model parameters of the model to be constructed;

[0122] Acquire a third image sample in the first subset and a fourth image sample in the second subset of the training sample set;

[0123] Obtaining an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed;

[0124] Based on the label of the third image sample corresponding to the line of sight direction, the label of the fourth image sample corresponding to the line of sight direction and the target line of sight difference, a second target loss function is constructed, and based on the third image sample, the fourth image sample and the second target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0125] Based on the above embodiment, in another embodiment provided by the present disclosure, the present disclosure provides a sight line estimation device, which includes:

[0126] A calibration image acquisition module, configured to acquire a calibration image; the calibration image corresponds to and includes a sight line direction;

[0127] An image acquisition module to be estimated is used to acquire the image to be estimated;

[0128] The sight line estimation module is configured to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated, and a sight line estimation model, wherein the sight line estimation model can be constructed using the method provided in the above embodiment.

[0129] For the device part, please refer to the description of the above embodiment and will not be repeated here.

[0130] The device provided by the embodiments of the present disclosure obtains an eye image of a first target object and an eye image of a second target object, and obtains the line of sight difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed, and constructs a loss function through the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the line of sight difference. In this way, the line of sight estimation model obtained by training the model to be constructed based on the training samples and the loss function can offset the influence of individual differences between different users, so that the eye image of the first target object can be simulated through the eye data of the second target object, and the accuracy of the line of sight direction of the image to be estimated estimated by the line of sight estimation model can be improved.

[0131] An embodiment of the present disclosure further provides an electronic device, comprising: at least one processor; a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0132] Figure 4 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. As shown in Figure 4, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can execute the corresponding steps of the above method disclosed in the embodiment of the present disclosure.

[0133] The processor 1801 can also be referred to as a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in the embodiments of the present disclosure can be completed by hardware integrated logic circuits in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present disclosure can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in the memory 1802, such as a storage medium mature in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The processor 1801 reads the information in the memory 1802 and, in conjunction with its hardware, completes the steps of the method.

[0134] In addition, when various operations / processes according to the present disclosure are implemented via software and / or firmware, the programs constituting the software can be installed from a storage medium or a network to a computer system having a dedicated hardware structure, such as computer system 1900 shown in FIG5 . When the various programs are installed, the computer system can perform various functions, including those described above. FIG5 is a block diagram of the structure of a computer system provided by an exemplary embodiment of the present disclosure.

[0135] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] As shown in FIG5 , computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. Various programs and data required for the operation of computer system 1900 may also be stored in RAM 1903. Computing unit 1901, ROM 1902, and RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to bus 1904.

[0137] Several components within computer system 1900 are connected to I / O interface 1905, including an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input numeric or character information and generate key input signals related to user settings and / or function control of an electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 1908 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices over a network, such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0138] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above-mentioned methods disclosed in the embodiments of the present disclosure by any other appropriate means (e.g., by means of firmware).

[0139] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0140] The computer-readable storage medium in the embodiments of the present disclosure can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The above-mentioned computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specifically, the above-mentioned computer-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0141] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0142] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method disclosed in the embodiments of the present disclosure is implemented.

[0143] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules, components, or units described in the embodiments of the present disclosure may be implemented in software or hardware. The names of the modules, components, or units do not necessarily limit the modules, components, or units themselves.

[0146] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] The above descriptions are merely some embodiments of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present disclosure.

[0148] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for constructing a line of sight estimation model, the method comprising: Acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels including sight lines directions; Acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed; constructing a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference; The model to be constructed is trained based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. When the loss value does not satisfy a preset convergence condition, the model parameters of the model to be constructed are updated. When the loss value satisfies a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

2. The method according to claim 1, wherein the eye image of the first target object includes a first image and a second image, and the eye image of the second target object includes a third image and a fourth image; and obtaining an estimated sight difference between the eye image of the first target object and the eye image of the second target object comprises: Obtaining an estimated first sight difference value based on the first image and the third image, and obtaining an estimated second sight difference value based on the second image and the fourth image; The estimated sight line difference value is determined based on a difference between the first sight line difference value and the second sight line difference value.

3. The method according to claim 2, wherein obtaining an estimated first sight difference value based on the first image and the second image, and obtaining an estimated second sight difference value based on the second image and the fourth image, comprises: Obtaining the first sight difference value based on the first image, the third image and the model to be constructed; The second sight difference value is obtained based on the second image, the fourth image and the model to be constructed.

4. The method according to claim 2 or 3, wherein the first image corresponds to a label containing a first sight line direction, the second image corresponds to a label containing a second sight line direction, the third image corresponds to a label containing a third sight line direction, and the fourth image corresponds to a label containing a fourth sight line direction; The constructing of the loss function comprises: Obtaining a first label difference between the label of the first sight line direction and the label of the third sight line direction, and obtaining a second label difference between the label of the second sight line direction and the label of the fourth sight line direction; Obtaining a label difference between the first label difference and the second label difference; A loss function is constructed based on the label difference and the sight difference.

5. The method according to any one of claims 1 to 4, comprising: Obtain a training sample set; The training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects; Acquire a first image sample and a second image sample in a target subset of the training sample set; Obtaining a target label difference between a label of the first image sample corresponding to the line of sight direction and a label of the second image sample corresponding to the line of sight direction, and constructing a first target loss function based on the target label difference; Based on the first image sample, the second image sample and the first objective loss function, training the model to be constructed, and updating the model parameters of the model to be constructed; Acquire a third image sample in the first subset and a fourth image sample in the second subset of the training sample set; Acquire an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed; Based on the label of the line of sight direction corresponding to the third image sample, the label of the line of sight direction corresponding to the fourth image sample and the target line of sight difference, a second objective loss function is constructed, and based on the third image sample, the fourth image sample and the second objective loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

6. A line of sight estimation method, the method comprising: Acquire a calibration image, wherein the calibration image corresponds to a known sight direction; Obtaining an image to be estimated; Based on the calibration image, the image to be estimated and the line of sight estimation model, the line of sight direction estimated by the image to be estimated is obtained; wherein the line of sight estimation model is obtained by the method described in any one of claims 1 to 5.

7. A device for constructing a sight line estimation model, the device comprising: An image acquisition module, used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a sight line direction; A sight line difference acquisition module, used for acquiring an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed; A loss function construction module, used to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference; a model training module, configured to train the model to be constructed based on training samples and the loss function to obtain a sight line estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object; During the training process, a loss value is determined based on the loss function. When the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. When the loss value meets the preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

8. A sight line estimation device, comprising: A calibration image acquisition module, used for acquiring a calibration image; The calibration image corresponds to a known line of sight direction; An image acquisition module to be estimated is used to acquire the image to be estimated; A sight line estimation module, used to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated and a sight line estimation model; wherein the sight line estimation model is obtained by the method described in any one of claims 1 to 5.

9. An electronic device, comprising: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

11. A computer program product, wherein the computer program product stores program instructions, and when the program instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

12. A computer program, when executed by a processor of an electronic device, enables the electronic device to perform the method according to any one of claims 1 to 6.

Images