Construction method of sight line estimation model and sight line estimation method and device

Abstract

The invention relates to a sight line estimation model construction method, a sight line estimation method and a sight line estimation device. The method comprises the following steps: acquiring an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels containing sight directions; obtaining an estimated sight line difference value between the eye image of the first target object and the eye image of the second target object based on a to-be-constructed model; constructing a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object and the sight line difference value; and training the to-be-constructed model based on a training sample and the loss function to obtain a sight line estimation model. Therefore, the accuracy of the line-of-sight direction of the to-be-estimated image estimated by the line-of-sight estimation model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for constructing a sight line estimation model, a sight line estimation method and a sight line estimation device. Background Art

[0002] Line of sight is an important clue to reveal how people understand the external environment. Line of sight estimation technology based on line of sight refers to the technology that calculates the user's line of sight through the captured user's eye image. Line of sight estimation technology has been widely used in human-computer interaction, virtual reality, augmented reality, medical analysis and other fields.

[0003] When a user wears relevant devices, such as AR (Augmented Reality), VR (Virtual Reality) or MR (Mixed Reality), the user's eye image can be collected. However, when the line of sight is directly estimated based on the eye image, the estimated line of sight direction may be inaccurate. Summary of the invention

[0004] The present disclosure provides a method for constructing a line of sight estimation model, a line of sight estimation method and a device.

[0005] According to one aspect of the present disclosure, a method for constructing a line of sight estimation model is provided, the method comprising:

[0006] Acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels including sight lines directions;

[0007] Acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0008] Constructing a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference;

[0009] The model to be constructed is trained based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. When the loss value does not satisfy a preset convergence condition, the model parameters of the model to be constructed are updated. When the loss value satisfies a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0010] According to a second aspect of the present disclosure, a line of sight estimation method is provided, the method comprising:

[0011] Acquire a calibration image; the calibration image corresponds to a known sight line direction;

[0012] Obtaining an image to be estimated;

[0013] Based on the calibration image, the image to be estimated and the line of sight estimation model, the line of sight direction estimated by the image to be estimated is obtained; wherein the line of sight estimation model is obtained by the method described above.

[0014] According to a third aspect of the present disclosure, a device for constructing a sight line estimation model is provided, the device comprising:

[0015] An image acquisition module, used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a sight line direction;

[0016] A sight line difference acquisition module, used for acquiring an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0017] A loss function construction module, used to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference;

[0018] A model training module is used to train the model to be constructed based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0019] According to a fourth aspect of the present disclosure, a sight line estimation device is provided, the device comprising:

[0020] A calibration image acquisition module, used to acquire a calibration image; the calibration image corresponds to a known sight line direction;

[0021] An image acquisition module to be estimated is used to acquire the image to be estimated;

[0022] A sight line estimation module is used to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated and the sight line estimation model; wherein the sight line estimation model is obtained by the method described above.

[0023] According to a fifth aspect of the present disclosure, an electronic device is provided. The electronic device comprises: a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0024] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0025] The disclosed embodiment obtains an eye image of a first target object and an eye image of a second target object, and obtains an estimated line of sight difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed, and constructs a loss function through the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the line of sight difference. In this way, the line of sight estimation model obtained by training the model to be constructed based on the training samples and the loss function can offset the influence of individual differences between different users, so that the eye image of the first target object can be simulated by the eye data of the second target object, thereby improving the accuracy of the line of sight direction of the image to be estimated estimated by the line of sight estimation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A schematic diagram of a sight line estimation model provided for an exemplary embodiment of the present disclosure;

[0028] Figure 2 A flowchart of a method for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure;

[0029] Figure 3 A schematic block diagram of functional modules of a device for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure;

[0030] Figure 4 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0031] Figure 5 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0038] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0039] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] When a user wears a target device such as AR, VR or MR, with the user's authorization and consent, images of the user's eyes can be collected in real time through an image acquisition device provided in the target device, and the line of sight direction can be estimated for the collected eye image. For example, local fine rendering of the scene displayed in the target device can be achieved based on the estimated line of sight direction, that is, fine rendering can be performed only on the scene within the user's gaze range, so as to reduce the hardware cost of the target device.

[0041] When estimating the sight direction of an image of the user's eye, in the related art, the calibration image and the image to be estimated can be input into a differential sight estimation model to obtain the estimated sight difference between the calibration image and the image to be estimated, and estimate the sight direction of the image to be estimated based on the sight difference and the known sight direction corresponding to the calibration image. Since the calibration image may be an image captured when the user wears the target device for the first time, and the image to be estimated may be an image captured when the user wears the target device another time, due to the difference in wearing position between different wearings, such difference will affect the estimated sight direction of the image to be estimated, resulting in inaccurate estimation of the sight direction of the image to be estimated.

[0042] like Figure 1 As shown, Figure 1It is a sight estimation model, which can extract image features of the image and obtain the estimated sight difference between the two images according to the extracted image features. Two images can be obtained, the first image can be a calibration image, and the second image can be an image to be estimated. By inputting these two images into the sight estimation model, the sight estimation model will output the estimated sight difference between the sight directions of the two images. Among them, the calibration image is an image collected by guiding the user to look at the calibration point of the known sight direction, so the calibration image corresponds to the known sight direction, so combined with the difference, the real sight direction of the image to be estimated can be estimated. It should be noted that the image in the embodiment can be an image sample, and the image sample can include a single eye image of a user, such as an eye image of the left eye or an eye image of the right eye; the image sample can also include a binocular image of a user, such as an eye image of the left eye and an eye image of the right eye; the image sample can also be an image spliced ​​by the eye image of the left eye and the eye image of the right eye of a user, and the embodiment is not limited to this.

[0043] Therefore, in order to improve the accuracy of the line of sight estimation model in estimating the line of sight direction, it is necessary to obtain eye images of the same user wearing the target device at different times, so that the line of sight estimation model can learn the influence of different wearing states on the line of sight direction estimation through training. However, in practice, if a large number of eye images of the same user in different wearing states are required as image samples, the acquisition cost of training samples will be significantly increased. In addition, the study found that due to individual differences between different users, such individual differences include differences in the eyeball structure of different individuals, such as different angles between the optical axis and the visual axis (where the eye image represents the direction of the optical axis of the eyeball, and the direction of the visual axis of the eyeball is the actual line of sight direction), and such individual differences cannot be eliminated, so that if the eye images obtained when different users wear the target device are used to simulate the eye images of the same user in different wearing states for model training, the line of sight estimation accuracy of the obtained line of sight estimation model cannot be guaranteed.

[0044] For example, when the sight line estimation model is trained by the above two image samples, the sight line labels corresponding to the image samples include the sight line direction represented by the image and the influence of the personality factors brought by the above individual differences on the true value of the sight line direction. If the sight line direction of the image to be estimated is estimated by the non-differential sight line estimation model, the corresponding estimated sight line direction can be directly output according to the single image sample, and the results learned by the model can be divided into two cases. The first case is that the image samples come from the same user, and the non-differential sight line estimation model learns the sight line direction represented by the image and the influence of the personality factors brought by the individual differences of the corresponding user on the true value of the sight line direction. In the test stage, if the sight line direction of the user is estimated, because the training samples and the images to be estimated in the test are all from the user, the sight line estimation of the image to be estimated in the test results is relatively accurate. However, if the image to be estimated in the test process and the image samples used in the training stage are not from the same user, due to the individual differences between different individuals, the personality factors brought by the individual differences between different users will have an impact on the estimated sight line direction, resulting in a large deviation in the estimated sight line direction.

[0045] If the line of sight direction of the image to be estimated is estimated by a differential line of sight estimation model, the model learns the line of sight difference between the two image samples. The true value of the line of sight difference can include the line of sight difference represented by the image and the difference in personality factors. If the images from the same user are used as the input of the model, the difference in personality factors is zero, so that the output result will not be affected by individual differences. If the input of the model comes from two different users, the difference in personality factors will not be zero, and the personality factors of the users are unknown items, that is, there will be a negative impact on line of sight estimation due to individual differences between different users.

[0046] Therefore, when training the model through two image samples to obtain a differential line of sight estimation model, if the first image sample is an eye image collected when the first user wears the target device, and the second image sample is an eye image collected when the second user wears the target device, since the first user and the second user are different users, and there will be individual differences between different users as mentioned above, when training the model through eye images of different users, if the influence of the individual differences on the line of sight direction cannot be eliminated, the line of sight estimation accuracy of the trained line of sight estimation model will be reduced.

[0047] Exemplarily, due to the individual differences among different users, the influence of the personalized eyeball structure of different users on the line of sight direction can be represented by the personality factor. That is, the label of the eye image collected when a user wears the target device can include two parts, namely, the line of sight direction represented by the image and the influence of the personality factor on the true value of the line of sight direction:

[0048] label = image gaze + personalized gaze (1)

[0049] Among them, label represents the gaze label corresponding to the image, image gaze represents the gaze direction represented by the image, and personalized gaze represents the influence of personality factors on the true value of the gaze direction.

[0050] In the embodiment, when the user wears the target device, a video of the user's eyes can be obtained through an image acquisition device set in the target device. When two image samples are arbitrarily acquired from the video, the difference in the labels between the two image samples can be expressed in the following manner:

[0051] label1_A-label1_B=(image gaze_1A+personalized gaze_1)-(image gaze_1B+personalized gaze_1)

[0052] = Image gaze_1A - Image gaze_1B (2)

[0053] Among them, label1_A represents the gaze label corresponding to the first gaze direction of user 1; label1_B represents the gaze label corresponding to the second gaze direction of user 1; image gaze_1A represents the gaze direction represented by the image corresponding to user 1, image gaze_1B represents the gaze direction represented by the image corresponding to user 1, and personalized gaze_1 represents the influence of personality factors on the true value of the gaze direction.

[0054] From the above formula (2), it can be seen that personalized gaze_1 can cancel each other out. Here, the same video refers to the image sequence collected by the same user while wearing the target device.

[0055] However, when two image samples are collected from videos corresponding to different users, the label difference between the two image samples can be expressed as follows:

[0056] label1_A-label2_B=(image gaze_1A+personalized gaze_1)-(image gaze_2B+personalized gaze_2) (3)

[0058] Among them, label1_A represents the gaze label corresponding to the first gaze direction of user 1, label2_B represents the gaze label corresponding to the second gaze direction of user 2, image gaze_1A represents the gaze direction represented by the image corresponding to user 1, image gaze_2B represents the gaze direction represented by the image corresponding to user 2, personalized gaze_1 represents the influence of the personality factor of user 1 on the true value of the gaze direction, and personalized gaze_2 represents the influence of the personality factor of user 2 on the true value of the gaze direction. Among them, user 1 is equivalent to the first user in the following, and user 2 is equivalent to the second user in the following.

[0059] It can be seen from the above formula (3) that due to the individual differences between user 1 and user 2, the personalized gaze_1 of user 1 and the personalized gaze_2 of user 2 cannot be eliminated. Therefore, in order to use the eye images collected when different users wear VR devices as the eye images of the same person in different wearing conditions, it is necessary to eliminate this difference, that is, to bridge the differences between the personality factors caused by individual differences, so that the eye images of different users can be used to simulate the training data of the same person in different wearing conditions, so as to solve the above problems caused by the lack or insufficiency of training data.

[0060] In the embodiments provided by the present disclosure, two image samples can be collected for different users wearing the target device, and they are represented by person1_A, person1_B, person2_C, and person2_D, respectively. Among them, person1_A and person1_B represent eye images of one user corresponding to two sight lines, respectively; person2_C and person2_D represent eye images of another user corresponding to two sight lines, respectively. person1 represents the first user, person2 represents the second user, and A, B, C, and D represent the corresponding sight lines, respectively. It should be noted that the image samples obtained in the embodiments are randomly obtained, and the sight lines represented by A, B, C, and D can be the same, different, or partially the same, and the embodiments are not limited thereto.

[0061] In the embodiment, two image samples can be obtained, the first image sample can be an eye image of the first user, and the second image sample can be an eye image of the second user. By inputting these two image samples into Figure 1 The line of sight estimation model in the embodiment of the present invention estimates the line of sight difference between the line of sight directions of the two image samples. The eye image of the first target object and the eye image of the second target object are respectively labeled with line of sight directions.

[0062] Specifically, inputting person1_A and person2_C into the line of sight estimation model can output the estimated line of sight direction difference diff_12_AC between the two images, that is, diff_12_AC = F(person1_A, person2_C). The line of sight difference label between the two can be determined by the difference between the two labels:

[0063] label1_A-label2_C=(image gaze1_A+personalized gaze_1)-(image gaze2_C+personalized gaze_2) (4)

[0065] By inputting person1_B and person2_D into the line of sight estimation model, the estimated line of sight direction difference diff_12_BD between the two images can be output, that is, diff_12_BD = F(person1_B, person2_D). The line of sight difference label between the two can be determined by the difference between the two labels:

[0066] label1_B-label2_D=(image gaze1_B+personalized gaze_1)-(image gaze2_D+personalized gaze_2) (5)

[0068] Through the above formulas (4) and (5), we can get:

[0069] label=(label1_A-label2_C)-(label1_B-label2_D)

[0070] =(image gaze1_A+personalized gaze_1)-(image gaze2_C+personalized gaze_2)-[(image gaze1_B+personalized gaze_1)-(image gaze2_D+personalized gaze_2)]

[0071] = image gaze1_A - image gaze2_C - image gaze1_B + image gaze2_D (6)

[0073] It can be seen from the above formula (6) that the obtained label is equivalent to the difference of the gaze direction difference represented by the two images. The above process can offset the personalized gaze_1 and personalized gaze_2 corresponding to the first user and the second user respectively, and avoid the negative impact of different individual differences on gaze estimation.

[0074] Therefore, the loss function loss = diff_12_AC-diff12_BD–label can be constructed, where label = (label1_A-label2_C)-(label1_B-label2_D), that is, loss = Loss(diff_12_AC-diff12_BD, label).

[0075] Since label1_A, label1_B, label2_C and label2_D are the sight lines labels corresponding to the four images respectively, the constructed loss function and the eye images collected when the first user wears the target device and the eye images collected when the second user wears the target device can be used as training samples. By constructing the above loss function, the distribution characteristics of the sight line difference between the two sample images under different wearing states can be learned during the model training process. By offsetting the personality factors brought by individual differences of different users, the purpose of simulating the eye data of the first user under different wearing states through the eye data of the second user can be achieved. After the model is trained with image samples from different users, the sight line estimation ability of the model in the scenario where the same user collects calibration images and images to be estimated under different wearing states can be improved.

[0076] Based on the above embodiments, the present disclosure provides a method for constructing a sight line estimation model. Figure 2 As shown, the method may include the following steps:

[0077] In step S210, an eye image of the first target object and an eye image of the second target object are acquired, wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a sight line direction.

[0078] In step S220, an estimated sight line difference between the eye image of the first target object and the eye image of the second target object is obtained based on the model to be constructed.

[0079] In an embodiment, the first target object may be represented by a first user, and the second target object may be represented by a second user. The eye image of the first user may be obtained when the first user wears the target device, and the eye image of the second user may be obtained when the second user wears the target device. As mentioned above, due to individual differences between individuals, when the eye image of the first user is simulated by the eye image of the second user for model training, the line of sight estimation accuracy of the obtained line of sight estimation model will be affected.

[0080] Based on this, in an embodiment provided by the present disclosure, the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be estimated, and the estimated result can be substituted into the loss calculation process to reduce the negative impact of the personality factor differences between the above-mentioned different users on the accuracy of line of sight estimation. In order to obtain the first personality factor corresponding to the first user, multiple eye image samples of the first user can be collected. For example, the multiple eye image samples can be collected when the first user wears the target device. Of course, when the first user is not wearing the target device, the image acquisition device can be used to capture the eyes of the first user to obtain multiple eye image samples of the first user, so that the image quality of the image samples obtained is higher. In the process of collecting images of the eyes of the first user, the first user can be guided to look at each pre-set gaze point respectively, and the eye image samples of the first user at the corresponding moment are collected when the first user looks at the corresponding gaze point. Therefore, each collected eye image sample corresponds to the sight line annotation information, and the sight line annotation information can be a label corresponding to the sight line direction. In this way, the pupil center can be determined from each high-quality eye image sample through a related image recognition algorithm to estimate the sight line direction represented by each sample image sample. By subtracting the estimated sight line direction from the corresponding label, multiple differences can be obtained, and the average value of the multiple differences can be used as the personality factor of the first user. Similarly, the second personality factor corresponding to the second user can be obtained in the above manner, so that the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be applied to the training process of the model to be constructed, so that the model can be trained through eye images from different users to improve the accuracy of the sight line direction of the image to be estimated estimated by the constructed sight line estimation model. It should be noted that, according to the above formula (3), since the personality factor of the first user is different from the personality factor of the second user, in order to eliminate the influence of the personality factors, in another embodiment provided by the present disclosure, the first personality factor corresponding to the first user and the second personality factor corresponding to the second user can be eliminated. For example, the eye image from the first user can be obtained to obtain at least two image samples, for example, including the first image and the second image, and the eye image from the second user can be obtained to obtain at least two image samples, for example, including the third image and the fourth image. Of course, the eye image of the first user obtained in the embodiment can be a sequence of images in a video of the user's eyes that can be obtained by an image acquisition device set in the target device when the user wears the target device.

[0081] Therefore, in the embodiment, the first image and the third image can be input into Figure 1The sight line estimation model to be constructed in the embodiment of the present invention is used to obtain a first sight line difference value estimated between the first image and the third image; and the second image and the fourth image are input into the sight line estimation model to obtain a second sight line difference value estimated between the second image and the fourth image.

[0082] Specifically, by inputting the first image and the third image into Figure 1 The line of sight estimation model in , respectively extracts the image features of the first image and the image features of the third image, and according to the image features of the first image and the image features of the third image, the estimated first line of sight difference between the first image and the third image can be obtained. Similarly, by inputting the second image and the fourth image into Figure 1 The sight line estimation model in the embodiment extracts the image features of the second image and the image features of the fourth image respectively, and obtains the second sight line difference value estimated between the second image and the fourth image according to the image features of the second image and the image features of the fourth image. The difference between the first sight line difference value and the second sight line difference value is obtained, and the difference is the sight line difference value between the eye image of the first target object and the eye image of the second target object.

[0083] Since the first image and the second image correspond to the first characteristic factor respectively, and the third image and the fourth image correspond to the second characteristic factor respectively, the difference between the first line of sight difference value obtained above and the second line of sight difference value obtained above can offset the first characteristic factor and the second characteristic factor, thereby avoiding the negative impact on line of sight estimation caused by individual differences between different users when wearing the target device.

[0084] In step S230, a loss function is constructed based on the label corresponding to the eye image of the first target object, the label corresponding to the label of the eye image of the second target object, and the sight line difference.

[0085] In the embodiment, after the first characteristic factor and the second characteristic factor are offset, the loss function can be constructed.

[0086] Since the first image, the second image, the third image and the fourth image respectively correspond to labels including the sight line direction, it can be specifically described by taking the first image corresponding to the first label, the second image corresponding to the second label, the third image corresponding to the third label and the fourth image corresponding to the fourth label as an example. The first label can be label1_A in the above implementation, the second label can be label1_B in the above implementation, the third label can be label2_C in the above implementation, and the fourth label can be label2_D in the above implementation. label1_A and label1_B include the same first property factor, and label2_C and label2_D include the same second property factor.

[0087] The estimated first sight difference diff_12_AC between the first image and the third image, the estimated second sight difference diff12_BD between the second image and the fourth image can be obtained, and the first label difference between the first label and the third label, the second label difference between the second label and the fourth label, and the difference label between the first label difference and the second label difference can be obtained. In this way, the constructed loss function loss can be obtained, loss = diff_12_AC-diff12_BD–label.

[0088] In step S240, the model to be constructed is trained based on the training samples and the loss function to obtain a line of sight estimation model. The training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets the preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0089] In an embodiment, the model to be constructed is trained by constructing a loss function and taking the image sequence of the first user's eyes and the image sequence of the second user's eyes as training samples. For example, when the loss value corresponding to the loss function is minimized, it is determined that the preset convergence condition is met. At this time, the training can be stopped to obtain the trained model to be constructed, and the trained model to be constructed is used as a line of sight estimation model.

[0090] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0091] Step S251, obtaining a training sample set.

[0092] In an embodiment, the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects. For example, eye images collected by the same user wearing the target device can be used as a subset.

[0093] Step S252: Acquire a first image sample and a second image sample in a target subset in a training sample set.

[0094] In an embodiment, the first image sample and the second image sample may be any two images in a target subset, respectively, and the target subset may be any subset in a training sample set.

[0095] Step S253, obtaining a target label difference between a label of the first image sample corresponding to the line of sight direction and a label of the second image sample corresponding to the line of sight direction, and constructing a first target loss function based on the target label difference.

[0096] In an embodiment, the first image sample and the second image sample include labels corresponding to the sight lines directions respectively, and the loss function can be constructed by the difference between the labels of the two.

[0097] Step S254: Based on the first image sample, the second image sample and the first target loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0098] In an embodiment, the model to be constructed can be trained by the first image sample and the second image sample, and when the loss value of the first objective loss function does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. In this way, the model to be constructed can be quickly trained by the eye image of the same user, thereby improving the line of sight estimation capability of the model under the same wearing state.

[0099] Step S255 , obtaining the third image sample in the first subset and the fourth image sample in the second subset in the training sample set.

[0100] In the embodiment, the first subset and the second subset may be any two subsets in the training sample set respectively, the third image sample may be any image sample in the first subset, and the fourth image sample may be any image sample in the second subset.

[0101] Step S256, obtaining an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed.

[0102] In an embodiment, the third image sample and the fourth image sample may be input into the model to be constructed to obtain an estimated target sight line difference between the third image sample and the fourth image sample.

[0103] Step S257, constructing a second target loss function based on the label of the third image sample corresponding to the line of sight direction, the label of the fourth image sample corresponding to the line of sight direction and the target line of sight difference, and training the model to be constructed based on the third image sample, the fourth image sample and the second target loss function, and updating the model parameters of the model to be constructed.

[0104] In the embodiment, the model to be constructed is trained by the third image sample and the fourth image sample, and during the training process, a loss value is determined based on the second objective loss function, and when the loss value does not meet the preset convergence condition, the model parameters of the model to be constructed are updated. The model to be constructed is trained again by eye images of different users, so that the accuracy of the line of sight estimation of the model in different wearing states can be improved.

[0105] It should be noted that the construction process of the second objective loss function is consistent with the construction process of the loss function in the above step S230. For details, please refer to the construction process of the above loss function, which will not be repeated here.

[0106] Based on the above embodiment, in another embodiment provided by the present disclosure, a line of sight estimation method is further provided, and the method may include the following steps:

[0107] S310, obtaining a calibration image, wherein the calibration image corresponds to a known sight line direction.

[0108] In an embodiment, the calibration image is an eye image obtained by the user wearing the target device, and the sight direction of the calibration image is known. When the calibration image and the image to be estimated are collected, it is done with the authorization and consent of the user. For example, the application on the target acquisition device can receive the user's request operation for sight estimation, and obtain the user's authorization and consent for collecting the eye image, and collect the user's eye image as the image to be estimated, so as to estimate the sight direction of the image to be estimated.

[0109] S320: Acquire an image to be estimated.

[0110] In an embodiment, the image to be estimated may be an eye image acquired when the user wears the target device another time.

[0111] S330, obtaining an estimated line of sight direction of the image to be estimated based on the calibration image, the image to be estimated and a line of sight estimation model.

[0112] The calibration image and the image to be estimated may be input into the sight line estimation model obtained after training according to the above embodiment, and the estimated sight line direction of the image to be estimated may be obtained.

[0113] In an embodiment, the line of sight estimation model obtained after the above training can learn the distribution characteristics of the line of sight difference between two sample images under different wearing states, and by offsetting the personality factors brought about by individual differences of different users, it can achieve the purpose of simulating the eye data of the same user under different wearing states through the eye data of different users. After the model is trained with image samples from different users, the accuracy of the line of sight estimation of the model in the scenario where the calibration image and the image to be estimated are collected by the same user under different wearing states can be improved.

[0114] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a device for constructing a line of sight estimation model, and the device for constructing the line of sight estimation model may be a server or a chip applied to a server. Figure 3 Schematic block diagram of functional modules of a device for constructing a sight line estimation model provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the construction device of the sight line estimation model includes:

[0115] An image acquisition module 10 is used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a sight line direction;

[0116] A sight line difference acquisition module 20, configured to acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed;

[0117] A loss function construction module 30, configured to construct a loss function based on a label corresponding to the eye image of the first target object, a label corresponding to the eye image of the second target object, and the sight line difference;

[0118] The model training module 40 is used to train the model to be constructed based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. When the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. When the loss value meets a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

[0119] Optionally, the eye image of the first target object includes a first image and a second image, and the eye image of the second target object includes a third image and a fourth image; the sight difference value acquisition module is specifically used to:

[0120] Obtaining an estimated first sight difference value based on the first image and the third image, and obtaining an estimated second sight difference value based on the second image and the fourth image;

[0121] The estimated sight line difference value is determined based on a difference between the first sight line difference value and the second sight line difference value.

[0122] Optionally, the sight difference value acquisition module is further used to:

[0123] Obtaining the first sight difference value based on the first image, the third image and the model to be constructed;

[0124] The second sight difference value is obtained based on the second image, the fourth image and the model to be constructed.

[0125] Optionally, the first image corresponds to a label containing a first sight line direction, the second image corresponds to a label containing a second sight line direction, the third image corresponds to a label containing a third sight line direction, and the fourth image corresponds to a label containing a fourth sight line direction; the loss function construction module is specifically used to:

[0126] Obtaining a first label difference between the label of the first sight line direction and the label of the third sight line direction, and obtaining a second label difference between the label of the second sight line direction and the label of the fourth sight line direction;

[0127] Obtaining a label difference between the first label difference and the second label difference;

[0128] A loss function is constructed based on the label difference and the sight difference.

[0129] Based on the above embodiment, in another embodiment provided by the present disclosure, the model training module is further used to:

[0130] Acquire a training sample set; the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects;

[0131] Acquire a first image sample and a second image sample in a target subset of the training sample set;

[0132] Obtaining a target label difference between a label of the first image sample corresponding to the line of sight direction and a label of the second image sample corresponding to the line of sight direction, and constructing a first target loss function based on the target label difference;

[0133] Based on the first image sample, the second image sample and the first objective loss function, training the model to be constructed, and updating the model parameters of the model to be constructed;

[0134] Acquire a third image sample in the first subset and a fourth image sample in the second subset of the training sample set;

[0135] Acquire an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed;

[0136] Based on the label of the line of sight direction corresponding to the third image sample, the label of the line of sight direction corresponding to the fourth image sample and the target line of sight difference, a second objective loss function is constructed, and based on the third image sample, the fourth image sample and the second objective loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

[0137] Based on the above embodiment, in another embodiment provided by the present disclosure, the present disclosure provides a sight line estimation device, the device comprising:

[0138] A calibration image acquisition module, used to acquire a calibration image; the calibration image corresponds to and includes a sight line direction;

[0139] An image acquisition module to be estimated is used to acquire the image to be estimated;

[0140] The sight line estimation module is used to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated and the sight line estimation model. The sight line estimation model can be constructed by the method provided in the above embodiment.

[0141] For the device part, please refer to the description of the above embodiment, which will not be repeated here.

[0142] The device provided by the embodiment of the present disclosure obtains an eye image of a first target object and an eye image of a second target object, and obtains the line of sight difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed, and constructs a loss function through the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the line of sight difference. In this way, the line of sight estimation model obtained by training the model to be constructed based on the training samples and the loss function can offset the influence of individual differences between different users, so that the eye image of the first target object can be simulated by the eye data of the second target object, and the accuracy of the line of sight direction of the image to be estimated estimated by the line of sight estimation model can be improved.

[0143] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0144] Figure 4 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0145] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0146] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 5 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 5 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0147] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 5As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0149] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0150] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0151] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0152] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0154] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0155] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0158] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0159] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for constructing a sight line estimation model, characterized in that: The method comprises: Acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to labels including sight lines directions; Acquire an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed; constructing a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference; The model to be constructed is trained based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. When the loss value does not satisfy a preset convergence condition, the model parameters of the model to be constructed are updated. When the loss value satisfies a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

2. The method according to claim 1, characterized in that The eye image of the first target object includes a first image and a second image, and the eye image of the second target object includes a third image and a fourth image; and obtaining an estimated sight difference between the eye image of the first target object and the eye image of the second target object includes: Obtaining an estimated first sight difference value based on the first image and the third image, and obtaining an estimated second sight difference value based on the second image and the fourth image; The estimated sight line difference value is determined based on a difference between the first sight line difference value and the second sight line difference value.

3. The method according to claim 2, characterized in that The obtaining an estimated first sight difference value based on the first image and the second image, and the obtaining an estimated second sight difference value based on the second image and the fourth image, comprises: Obtaining the first sight difference value based on the first image, the third image and the model to be constructed; The second sight difference value is obtained based on the second image, the fourth image and the model to be constructed.

4. The method according to claim 2, characterized in that: The first image corresponds to a label containing a first sight line direction, the second image corresponds to a label containing a second sight line direction, the third image corresponds to a label containing a third sight line direction, and the fourth image corresponds to a label containing a fourth sight line direction; The constructing of the loss function comprises: Obtaining a first label difference between the label of the first sight line direction and the label of the third sight line direction, and obtaining a second label difference between the label of the second sight line direction and the label of the fourth sight line direction; Obtaining a label difference between the first label difference and the second label difference; A loss function is constructed based on the label difference and the sight difference.

5. The method according to claim 1, characterized in that The method comprises: Acquire a training sample set; the training sample set includes at least two subsets, and different subsets include eye images corresponding to different target objects; Acquire a first image sample and a second image sample in a target subset of the training sample set; Obtaining a target label difference between a label of the first image sample corresponding to the line of sight direction and a label of the second image sample corresponding to the line of sight direction, and constructing a first target loss function based on the target label difference; Based on the first image sample, the second image sample and the first objective loss function, training the model to be constructed, and updating the model parameters of the model to be constructed; Acquire a third image sample in the first subset and a fourth image sample in the second subset of the training sample set; Acquire an estimated target sight line difference between the third image sample and the fourth image sample based on the model to be constructed; Based on the label of the line of sight direction corresponding to the third image sample, the label of the line of sight direction corresponding to the fourth image sample and the target line of sight difference, a second objective loss function is constructed, and based on the third image sample, the fourth image sample and the second objective loss function, the model to be constructed is trained, and the model parameters of the model to be constructed are updated.

6. A line of sight estimation method, characterized in that: The method comprises: Acquire a calibration image, wherein the calibration image corresponds to a known sight direction; Obtaining an image to be estimated; Based on the calibration image, the image to be estimated and the line of sight estimation model, the line of sight direction estimated by the image to be estimated is obtained; wherein the line of sight estimation model is obtained by any one of the methods described in claims 1 to 5.

7. A device for constructing a sight line estimation model, characterized in that: The device comprises: An image acquisition module, used to acquire an eye image of a first target object and an eye image of a second target object; wherein the eye image of the first target object and the eye image of the second target object respectively correspond to a label including a sight line direction; A sight line difference acquisition module, used for acquiring an estimated sight line difference between the eye image of the first target object and the eye image of the second target object based on the model to be constructed; A loss function construction module, used to construct a loss function based on the label corresponding to the eye image of the first target object, the label corresponding to the eye image of the second target object, and the sight line difference; A model training module is used to train the model to be constructed based on training samples and the loss function to obtain a line of sight estimation model; wherein the training samples include an eye image of the first target object and an eye image of the second target object. During the training process, a loss value is determined based on the loss function. If the loss value does not meet a preset convergence condition, the model parameters of the model to be constructed are updated. If the loss value meets a preset convergence condition, the trained model to be constructed is determined as the line of sight estimation model.

8. A sight line estimation device, characterized in that: The device comprises: A calibration image acquisition module, used to acquire a calibration image; the calibration image corresponds to a known sight line direction; An image acquisition module to be estimated is used to acquire the image to be estimated; A sight line estimation module, used to obtain the estimated sight line direction of the image to be estimated based on the calibration image, the image to be estimated and a sight line estimation model; wherein the sight line estimation model is obtained by any method described in claims 1 to 5.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

Images