Sight line estimation method and device, electronic equipment and storage medium

Abstract

The invention relates to a sight line estimation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a calibration image and a to-be-estimated image; the calibration image corresponds to a known sight line direction; a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the to-be-estimated image are acquired, the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device; obtaining an estimated sight line difference value between the calibration image and the to-be-estimated image, and obtaining a sight line difference estimation deviation under the common influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor; and obtaining the sight line direction of the to-be-estimated image based on the sight line direction of the calibration image, the sight line difference value and the sight line difference estimation deviation. Therefore, the sight line difference between the obtained calibration image and the to-be-estimated image can be corrected, and the calculation accuracy of the sight line direction of the to-be-estimated image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a line of sight estimation method, device, electronic device and storage medium. Background Art

[0002] Line of sight is an important clue to reveal how people understand the external environment. Line of sight estimation technology based on line of sight refers to the technology that calculates the user's line of sight through the user's facial image. Line of sight estimation technology has been widely used in human-computer interaction, virtual reality, augmented reality, medical analysis and other fields.

[0003] When a user wears relevant devices, such as AR (Augmented Reality), VR (Virtual Reality) or MR (Mixed Reality), the user's eye image can be collected. When the line of sight is estimated directly from the eye image, the estimated line of sight direction may be inaccurate. Summary of the invention

[0004] The present disclosure provides a line of sight estimation method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, a line of sight estimation method is provided, the method comprising:

[0006] Acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0007] Acquire a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device;

[0008] Acquire an estimated sight line difference value between the calibration image and the image to be estimated, and acquire a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor;

[0009] The sight direction of the image to be estimated is obtained based on the sight direction of the calibration image, the sight difference value, and the sight difference estimation deviation.

[0010] According to another aspect of the present disclosure, a sight line estimation device is provided, the device comprising:

[0011] An image acquisition module is used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0012] a wearing factor acquisition module, configured to acquire a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device;

[0013] a difference acquisition module, configured to acquire an estimated sight line difference between the calibration image and the image to be estimated, and acquire a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor;

[0014] The sight line direction acquisition module is used to obtain the sight line direction of the image to be estimated based on the sight line direction of the calibration image, the sight line difference value and the sight line difference estimation deviation.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] The line of sight estimation method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain the calibration image and the image to be estimated, and obtain the first wearing factor corresponding to the calibration image and the second wearing factor corresponding to the image to be estimated, obtain the estimated line of sight difference between the calibration image and the image to be estimated, and obtain the line of sight difference estimation deviation under the influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor, so as to obtain the line of sight direction of the image to be estimated. In this way, on the basis of the calibration image, combined with the line of sight difference estimation deviation caused by the wearing state that may exist between the calibration image and the image to be estimated, the estimated line of sight difference between the obtained calibration image and the image to be estimated can be corrected, and the calculation accuracy of the line of sight direction of the image to be estimated can be improved, thereby avoiding the problem of inaccurate calculation of the line of sight direction of the image to be estimated due to ignoring the above-mentioned line of sight difference estimation deviation by directly using the estimated line of sight difference between the calibration image and the image to be estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a line of sight estimation model provided for an exemplary embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a wearing factor network model provided for an exemplary embodiment of the present disclosure;

[0021] Figure 3 A flowchart of a line of sight estimation method provided for an exemplary embodiment of the present disclosure;

[0022] Figure 4 A schematic block diagram of functional modules of a sight line estimation device is provided for an exemplary embodiment of the present disclosure;

[0023] Figure 5 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0024] Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0032] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] When a user wears a target device such as AR, VR or MR, with the user's authorization and consent, images of the user's eyes can be collected in real time through an image acquisition device provided in the target device, and the line of sight direction can be estimated for the collected eye image. For example, local fine rendering of the scene displayed in the target device can be achieved based on the estimated line of sight direction, that is, fine rendering can be performed only on the scene within the user's gaze range, so as to reduce the hardware cost of the target device.

[0034] When estimating the line of sight direction of an image of a user's eye, the related technology usually inputs the calibration image and the image to be estimated into a differential model to obtain the estimated line of sight difference between the calibration image and the image to be estimated, and estimates the line of sight direction of the image to be estimated based on the line of sight difference and the line of sight direction corresponding to the calibration image. Since the calibration image may be an image captured when the user wears the target device at a certain time, and the image to be estimated may be an image captured when the user wears the target device at another time, due to the difference in wearing position between different wearings, such difference will affect the line of sight estimation accuracy of the image to be estimated, resulting in inaccurate line of sight direction of the image to be estimated.

[0035] like Figure 1 As shown, Figure 1 For the line of sight estimation model, two images can be obtained, the first image can be a calibration image, and the second image can be an image to be estimated. By inputting these two images into the line of sight estimation model, the line of sight estimation model will output the estimated line of sight difference between the line of sight directions of the two images. Among them, the calibration image is an image collected by guiding the user to look at the calibration point of the known line of sight direction. Since the line of sight direction corresponding to the calibration image is known, the real line of sight direction of the image to be estimated can be obtained by combining the difference. It should be noted that the image in the embodiment may include a single-eye image of a user, such as an eye image of the left eye or an eye image of the right eye; the image may also include a binocular image of a user, such as an eye image of the left eye and an eye image of the right eye; the image may also be an image obtained by splicing an eye image of the left eye and an eye image of the right eye of a user, and the embodiment is not limited to this.

[0036] In the embodiments, relevant wearing refers to the user wearing a target device such as AR, VR or MR, and different wearing refers to the user taking off the target device and then wearing the target device again after wearing it, or it may also refer to the two wearing states before and after the user adjusts the wearing position of the target device during the wearing process. The wearing state may include the relative position between the eye and the target device, etc.

[0037] Since the calibration image and the image to be estimated may be images collected under different wearing states, the related art often ignores the influence of the wearing state on the time estimation result when estimating the image through the line of sight estimation model. For example, when the related art trains the line of sight estimation model, the training data may obtain images corresponding to different wearing states, and the wearing state corresponding to the training data may be different from the wearing state corresponding to the image to be estimated during the line of sight estimation. When the line of sight estimation model obtained by training the model with the above training data estimates the line of sight difference between the calibration image and the image to be estimated, if the image to be estimated and the calibration image correspond to different wearing states respectively, there will be a difference between the wearing states corresponding to the image to be estimated and the calibration image respectively. Both of the above differences will affect the accuracy of the line of sight estimation and have a negative impact on the line of sight estimation. Therefore, the embodiment of the present disclosure introduces corresponding wearing factors respectively under different wearing states, and online fits the line of sight difference estimation deviation caused by the corresponding wearing state based on the wearing factors to improve the accuracy of the line of sight direction estimation result.

[0038] like Figure 2 As shown in the figure, by pre-building the wearing factor network model, the line of sight difference estimation deviation under different wearing conditions is obtained, and the line of sight difference estimation deviation is combined with Figure 1 The estimated sight difference value output by the middle network can more accurately obtain the sight direction of the image to be estimated.

[0039] In the embodiment, A is used to identify the calibration image collected by the user during one wearing process. Since the calibration image is an image collected by guiding the user to look at the calibration point in a known sight direction, the calibration image is equivalent to having a known sight direction. The calibration image may include multiple calibration images such as A1 and A2, and A1 and A2 may respectively represent calibration images in different sight directions. Corresponding to the calibration image A, B may be used to identify the image collected by the user during another wearing process, which may include multiple images such as B1 and B2.

[0040] In an embodiment, a wearing factor is introduced during the training and testing of the model, and F(x, y) is used to represent the estimated line of sight difference between the estimated image x and the calibration image y; β is used to represent the wearing factor, and f(β1, β2) is used to represent the degree of influence of different wearing on the line of sight direction, for example, the degree of influence of the Nth wearing on the line of sight direction compared to the Mth wearing, where M and N are unequal positive integers.

[0041] During the test, the embodiment of the present disclosure combines the wearing factor into the line of sight estimation model to obtain the estimated line of sight difference value, and the following formula (1) can be obtained:

[0042] F(A1, A2) + f(βA, βA) = label A1 – label A2 (1)

[0043] Where F(A1, A2) represents the estimated line of sight difference between calibration image A1 and calibration image A2, which can be obtained by Figure 1 βA represents the first wearing factor corresponding to the calibration image, and f(βA,βA) represents the estimated deviation of the sight difference due to the first wearing factor in the first wearing state when obtaining the calibration image, which can be obtained by Figure 2 The corresponding network is obtained; label A1 represents the sight direction of calibration image A1, and label A2 represents the sight direction of calibration image A2.

[0044] From the above formula (1), we can get f(βA,βA)=label A1–label A2-F(A1,A2). Since labelA1, label A2 and F(A1,A2) are known quantities, we can get f(βA,βA). During the test, since the parameters of the model remain unchanged, βA can be obtained by back-propagating the wearing factor network model f().

[0045] In the embodiment provided in the present disclosure, by incorporating the wearing factor into the line of sight estimation model to obtain the estimated line of sight difference value, the following formula (2) and formula (3) can also be obtained.

[0046] F(A1, B1) + f(βA, βB) = label A1 –gaze_ B1 (2)

[0047] F(A2, B2) + f(βA, βB) = label A2 – gaze_B2 (3)

[0048] Among them, F(A1, B1) represents the estimated sight difference between the calibration image A1 and the image to be estimated B1, and F(A2, B2) represents the estimated sight difference between the calibration image A2 and the image to be estimated B2. Figure 1 The gaze estimation model in is obtained; f(βA, βB) represents the estimated deviation of the gaze difference between the first wearing factor corresponding to the calibration image in the first wearing state and the second wearing factor corresponding to the image to be estimated in the second wearing state; label A1 represents the gaze direction of the calibration image A1, label A2 represents the gaze direction of the calibration image A2, gaze_B1 represents the estimated gaze direction of the image to be estimated B1, and gaze_B2 represents the estimated gaze direction of the image to be estimated B2. Among them, image B1 and image B2 represent images obtained from another wearing of the corresponding calibration image. For example, the target user obtains calibration images A1 and A2 when wearing the target device for the Mth time, and the target user obtains calibration images B1 and B2 when wearing the target device for the Nth time. Among them, M and N are unequal positive integers.

[0049] Based on the above formula (2) and formula (3), the left side of formula (2) is subtracted from the left side of formula (3), and the right side of formula (2) is subtracted from the right side of formula (3), and the following formula (4) can be obtained:

[0050] F(A1,B1)+f(βA,βB)-F(A2,B2)-f(βA,βB)=(label A1–gaze_B1)–(label A2–gaze_B2); (4)

[0051] That is: F(A1,B1)-F(A2,B2)=(label A1–gaze_B1)–(label A2–gaze_B2)

[0052] By expressing F(A1,B1)-F(A2,B2) through diff^AB, we can obtain the following formula (5):

[0053] diff^AB = (label A1 – label A2) + (gaze_B2 – gaze_B1) (5)

[0054] Based on the deformation of the above formula (1), the following formula (6) can be obtained:

[0055] F(B1, B2) + f(βB, βB) = gaze_B1 – gaze_B2 (6)

[0056] Among them, F(B1, B2) represents the estimated sight difference between the image to be estimated B1 and the image to be estimated B2, which can be specifically expressed by Figure 1 βB represents the second wearing factor corresponding to the image to be estimated, and f(βB,βB) represents the estimated deviation of the sight difference due to the second wearing factor in the second wearing state when acquiring the image to be estimated, which can be obtained by Figure 2 The corresponding network is obtained; gaze_B1 represents the estimated sight direction of the image B1 to be estimated, and gaze_B2 represents the estimated sight direction of the image B2 to be estimated.

[0057] The following formula (7) can be obtained by using the above formula (5) and formula (6):

[0058] F(B1, B2) + f(βB, βB) = (labelA1 - labelA2) - diff^AB (7)

[0059] Since F(B1, B2), labelA1, labelA2 and diff^AB are known quantities that can be obtained, f(βB, βB) can be obtained through the above formula (7). Similarly, βB can be obtained by back-propagating the wearing factor network model f().

[0060] In the embodiment, since βA and βB are obtained by the above method, f(βA, βB) can be obtained, and since F(A1, B1) and label A1 are known quantities that can be obtained, combined with the above formula (2), gaze_B1 can be obtained, that is:

[0061] gaze_B1=label A1–F(A1,B1)-f(βA,βB)

[0062] Through the embodiments provided in the present disclosure, the sight direction of the image to be estimated under different wearing conditions can be obtained.

[0063] In an embodiment, the above-mentioned line of sight estimation model and wearing factor network model can be independent of each other. When training the model, the line of sight estimation model and the wearing factor network model can be trained jointly, or they can be trained separately. For example, the line of sight estimation model can be trained first, and after the line of sight estimation model is trained, the wearing factor network model can be trained using the trained line of sight estimation model.

[0064] In an embodiment, during the training of the wearing factor network model, a vector value may be randomly assigned as the value of the wearing factor.

[0065] In the model testing stage, by obtaining a calibration image, solving the first wearing factor corresponding to the calibration image by formula (1), and obtaining the image to be estimated, the line of sight estimation of the image to be estimated includes two processes. In the first process, the line of sight estimation model and the calibration image are used to obtain the estimated line of sight difference of the image to be estimated; in the second process, the second wearing factor corresponding to the image to be estimated can be solved according to the above formulas (5) and (6), and then the first wearing factor and the second wearing factor are input into the wearing factor network model to obtain the line of sight difference estimation deviation. According to the estimated line of sight difference value obtained in the first process and the line of sight difference estimation deviation obtained in the second process, the line of sight direction estimation result of the image to be estimated can be obtained. In the embodiment, the model training stage can be based on the same principle as above, which will not be repeated here. It should be noted that in the model training stage, the training samples used can be any two eye images collected by the same person when wearing the target device. For example, a video containing a sequence of eye images is collected by an image acquisition device set in the target device, and two images are selected arbitrarily from the video.

[0066] Based on the above embodiments, the present disclosure first provides a line of sight estimation method, such as Figure 3 As shown, the method may include the following steps:

[0067] In step S310, a calibration image and an image to be estimated are acquired.

[0068] Among them, the calibration image corresponds to the known viewing direction.

[0069] In an embodiment, for example, when a user wears VR glasses, the VR glasses can be used to guide the user to look in a certain direction, such as setting 9 calibration points, guiding the user to look at the 9 calibration points in turn, and collecting images containing the user's current eyes while the user is looking at the calibration points. In this way, 9 calibration images can be obtained, and the calibration images will contain corresponding known sight directions, such as left front, right front, or straight ahead, etc.

[0070] The image to be estimated in the embodiment may be an image captured by the user during another time of wearing VR glasses, the purpose of which is to obtain the sight direction of the image to be estimated. For example, by obtaining the user's current sight direction, a high-resolution video image can be displayed in the corresponding area of ​​the VR glasses, while a non-high-resolution image can be displayed in the user's non-annotated area, so as to improve data processing efficiency.

[0071] In step S320, a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated are obtained, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device.

[0072] Since the eye wearing position will be different each time when the user puts on the target device such as VR glasses again, and this difference will affect the accuracy of the line of sight estimation, the first wearing factor and the second wearing factor are introduced to respectively characterize the first wearing state and the second wearing state of the target user when wearing the target device, so as to estimate the line of sight direction of the image to be estimated more accurately in the future. Among them, the acquisition method of the first wearing factor and the second wearing factor can refer to the description in the above embodiment, which will not be repeated here.

[0073] In step S330, an estimated sight line difference value between the calibration image and the image to be estimated is obtained, and a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state is obtained based on the first wearing factor and the second wearing factor.

[0074] In an embodiment, the line of sight estimation model can be used to extract image features of the calibration image and the image to be estimated, and based on the extracted image features, the estimated line of sight difference between the calibration image and the image to be estimated is obtained. The line of sight estimation model is trained using image samples obtained when the target object wears the target device. Specifically, the estimated line of sight difference between the calibration image and the image to be estimated can be obtained by Figure 1 The calibration image and the image to be estimated are input into the line of sight estimation model, and the estimated line of sight difference between the line of sight directions of the two is obtained through feature extraction and feature comparison.

[0075] In an embodiment, the calibration image and the image to be estimated can be input into a pre-built line of sight estimation model to obtain an estimated line of sight difference between the calibration image and the image to be estimated. In addition, the first wearing factor and the second wearing factor can be input into a pre-built wearing factor network model to obtain an estimated deviation of the line of sight difference under the combined influence of the first wearing state and the second wearing state.

[0076] Since the calibration image is an image obtained when the user wears the target device in the first wearing state, and the image to be estimated is generally an image obtained when the user wears the target device in the second wearing state, the two different wearing states will cause wearing differences between the wearing states, so that the estimated line of sight difference between the calibration image directly obtained by the line of sight estimation model and the image to be estimated will be affected by the two wearing states. Therefore, the estimated line of sight difference deviation between the two different wearing states can be obtained by obtaining the first wearing factor and the second wearing factor to offset the influence of the estimated line of sight difference under different wearing states.

[0077] It should be noted that the line of sight difference estimation deviation in the embodiment may correspond to one wearing state or two wearing states, and is related to the two images to be used for line of sight difference estimation. For example, when the two images are collected under the same wearing state, the obtained line of sight difference estimation deviation corresponds to one wearing state; when the two images are collected under different wearing states, the obtained line of sight difference estimation deviation corresponds to the aforementioned two different wearing states.

[0078] In step S340, the sight line direction of the image to be estimated is obtained based on the sight line direction of the calibration image, the sight line difference value and the sight line difference estimation deviation.

[0079] In an embodiment, since the calibration image corresponds to a known line of sight direction, the estimated line of sight difference between the calibration image and the image to be estimated can be combined with the calibration image to improve the calculation accuracy of the line of sight direction of the image to be estimated. By introducing the line of sight difference estimation deviation under the influence of the first wearing state and the second wearing state, the problem of inaccurate line of sight direction estimation of the image to be estimated caused by ignoring the line of sight difference estimation deviation while directly using the estimated line of sight difference between the calibration image and the image to be estimated can be avoided.

[0080] Based on the above embodiment, in another embodiment provided by the present disclosure, the calibration image includes a first calibration image and a second calibration image of the eye of the target object acquired when the target object wears the target device in the first wearing state; the above step S320 may further include the following steps:

[0081] S321, obtaining a first line of sight difference estimated between the first calibration image and the second calibration image.

[0082] S322, determining a first wearing factor based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image. In the embodiment, the sight difference estimation deviation in the first wearing state can be obtained based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image, and the first wearing factor can be obtained by back propagation based on the sight difference estimation deviation in the first wearing state and the pre-constructed wearing factor network model.

[0083] In the embodiment provided by the present disclosure, two calibration images A1 and A2 can be obtained, and the two have different sight lines. By inputting the two calibration images into the sight line estimation model, the estimated first sight line difference between the two calibration images can be obtained, and since the two correspond to known sight line directions respectively, the sight line difference estimation deviation generated by the two calibration images in the first wearing state can be obtained by the above formula (1), and combined with the pre-constructed wearing factor network model, the first wearing factor can be obtained by back propagation.

[0084] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S320 may further include the following steps:

[0085] S323, obtaining a reference image, wherein the reference image and the image to be estimated are respectively eye images obtained when the target object wears the target device in the first wearing state.

[0086] S324, obtaining a target sight line difference value; the target sight line difference value includes a difference between a first value and a second value, the first value being an estimated sight line difference value between the first calibration image and the image to be estimated, and the second value being an estimated sight line difference value between the second calibration image and the reference image.

[0087] S325, obtaining a second sight line difference value estimated between the reference image and the image to be estimated;

[0088] S326, determining a second wearing factor based on the first sight difference value, the second sight difference value, and the target sight difference value. In the embodiment, the sight difference estimation deviation estimated between the reference image and the image to be estimated can be obtained based on the first sight difference value, the second sight difference value, and the target sight difference value, and the second wearing factor can be obtained by back propagation based on the sight difference estimation deviation estimated between the reference image and the image to be estimated and the pre-constructed wearing factor network model.

[0089] In the embodiment provided by the present disclosure, f(βB, βB) can be obtained by combining formulas (2) to (7) in the above embodiment, and then the second wearing factor βB can be obtained. Since βA and βB are obtained in the above manner, f(βA, βB) can be obtained, and since F(A1, B1) and label A1 are known quantities that can be obtained, combined with the above formula (2), gaze_B1 can be obtained, that is, the gaze direction of the image to be estimated can be obtained.

[0090] In the embodiments provided by the present disclosure, the above-mentioned sight estimation model and the wearing factor network model can be independent of each other, and can be jointly trained online when training the models. Therefore, based on the above-mentioned embodiments, when the sight estimation model to be constructed and the wearing factor network model to be constructed are trained by joint training, the method provided by the embodiments of the present disclosure can also include the following steps:

[0091] Step S410, obtaining training samples.

[0092] In an embodiment, the training sample includes a first image sample and a second image sample, the first image sample includes an eye image obtained when the first subject wears the target device in a third wearing state, the second image sample includes an eye image obtained when the first subject wears the target device in a fourth wearing state, and the first image sample and the second image sample correspond to line of sight annotation information respectively. The first subject may be the same person as the target subject in the above embodiment, or may be a different person. The third wearing state and the fourth wearing state are different wearing states, and the third wearing state and the fourth wearing state may be wearing states different from the first wearing state and the second wearing state in the above embodiment.

[0093] Step S420: Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated third sight line difference between the first image sample and the second image sample.

[0094] In an embodiment, the first image sample and the second image sample are input into a sight line estimation model to be constructed, and an estimated third sight line difference value between the first image sample and the second image sample can be output.

[0095] Step S430: Obtain the third wearing factor and the fourth wearing factor which are randomly initialized respectively.

[0096] The third wearing factor represents a third wearing state when the first subject wears the target device, and the fourth wearing factor represents a fourth wearing state when the first subject wears the target device. In the embodiment, a random vector can be assigned to the third wearing factor and the fourth wearing factor respectively to randomly initialize the third wearing factor and the fourth wearing factor.

[0097] Step S440: taking the third wearing factor and the fourth wearing factor as inputs of the wearing factor network model to be constructed, and obtaining a second sight line difference estimation deviation under the combined influence of the third wearing state and the fourth wearing state.

[0098] In the embodiment, the third wearing factor and the fourth wearing factor are input into the wearing factor network model to be constructed, so that the second sight line difference estimation deviation under the joint influence of the third wearing state and the fourth wearing state can be input.

[0099] Step S450: constructing a first loss function based on the third sight line difference value, the second sight line difference estimation deviation, and sight line annotation information corresponding to the first image sample and the second image sample respectively.

[0100] In an embodiment, the sight line annotation information corresponding to the second image sample may be subtracted from the sight line annotation information corresponding to the first image sample to obtain a first sight line annotation information difference; the third sight line difference value and the second sight line difference estimation deviation may be added and then subtracted from the first sight line annotation information difference to obtain a first loss function.

[0101] Step S460: When the first training stop condition is not met, the sight line estimation model to be constructed and the wearing factor network model to be constructed are updated based on the first loss function. When the first training stop condition is met, the updated sight line estimation model to be constructed is used as the sight line estimation model, and the updated wearing factor network model to be constructed is used as the wearing factor network model.

[0102] In an embodiment, the first training stop condition may be reaching at least one of a preset training time, a preset number of iterations, and a preset loss convergence condition, wherein the preset training time and the preset number of iterations may be set separately according to actual needs, and the preset loss convergence condition may be, for example, determining that the preset loss convergence condition is satisfied when the loss value obtained according to the first loss function is the smallest, etc. The embodiment is not limited thereto.

[0103] In the embodiment provided by the present disclosure, the sight estimation model and the wearing factor network model may be trained separately, so the method may further include the following steps:

[0104] Step S510: acquiring training samples, where the training samples include a first image sample and a second image sample.

[0105] Among them, the first image sample includes an eye image obtained when the first object wears the target device in a third wearing state, and the second image sample includes an eye image obtained when the first object wears the target device in a fourth wearing state. The first image sample and the second image sample correspond to line of sight annotation information respectively.

[0106] In the embodiment, the first object may be the same person as the target object in the above embodiment, or may be a different person. The third wearing state and the fourth wearing state are different wearing states, and the third wearing state and the fourth wearing state may be wearing states different from the first wearing state and the second wearing state in the above embodiment.

[0107] Step S520: Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated fourth sight line difference value between the first image sample and the second image sample.

[0108] Step S530, constructing a second loss function based on the fourth sight difference value, the sight annotation information corresponding to the first image sample and the second image sample, and updating the sight estimation model to be constructed based on the second loss function when the second stop training condition is not met. When the second stop training condition is met, the updated sight estimation model to be constructed is used as the sight estimation model.

[0109] In an embodiment, the sight line annotation information corresponding to the second image sample may be subtracted from the sight line annotation information corresponding to the first image sample to obtain a second sight line annotation information difference. The second loss function may be obtained by subtracting the second sight line annotation information difference from the obtained fourth sight line difference value. The second training stop condition may be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition. Among them, the preset training time and the preset number of iterations may be set separately according to actual needs, and the preset loss convergence condition may be, for example, when the loss value obtained according to the second loss function is minimized, determining that the preset loss convergence condition is satisfied, etc. The embodiments are not limited thereto.

[0110] Step S540, taking the first image sample and the second image sample as inputs of the line of sight estimation model, obtaining an estimated fifth line of sight difference value between the first image sample and the second image sample, and obtaining supervision information of the line of sight difference estimation deviation based on the fifth line of sight difference value and the line of sight annotation information corresponding to the first image sample and the second image sample respectively.

[0111] In an embodiment, the fifth sight line difference value may be subtracted from the second sight line annotation information difference to obtain supervision information of the sight line difference estimation deviation.

[0112] Step S550: Obtain the third wearing factor and the fourth wearing factor which are randomly initialized respectively.

[0113] The third wearing factor represents a third wearing state when the first subject wears the target device, and the fourth wearing factor represents a fourth wearing state when the first subject wears the target device.

[0114] Step S560: taking the third wearing factor and the fourth wearing factor as inputs of the wearing factor network model to be constructed, and obtaining a third sight line difference estimation deviation under the joint influence of the third wearing state and the fourth wearing state.

[0115] Step S570, constructing a third loss function based on the third line of sight estimation deviation and the supervision information; when the third stop training condition is not met, updating the wearing factor network model to be constructed based on the third loss function; and when the third stop training condition is met, using the updated wearing factor network model to be constructed as the wearing factor network model.

[0116] In an embodiment, the difference between the third sight difference estimation deviation and the supervision information can be used as a third loss function. The third training stop condition can be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition. Among them, the preset training time and the preset number of iterations can be set separately according to actual needs, and the preset loss convergence condition can be, for example, when the loss value obtained according to the third loss function is the minimum, it is determined that the preset loss convergence condition is met, etc. The embodiment is not limited to this.

[0117] In the embodiment, the sight estimation model to be constructed and the constructed wearing factor network model can be trained respectively by training samples. After reaching the stop training conditions respectively, the constructed sight estimation model and the constructed wearing factor network model can be obtained respectively. The training samples in the embodiment can be obtained in advance. The training methods of the sight estimation model and the wearing factor network model in the embodiment can be selected as needed, and the embodiment is not limited thereto.

[0118] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a sight line estimation device, which may be a server or a chip applied to a server. Figure 4 Schematic block diagram of functional modules of a sight line estimation device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the sight line estimation device comprises:

[0119] An image acquisition module 10 is used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0120] A wearing factor acquisition module 20, configured to acquire a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device;

[0121] a difference acquisition module 30, configured to acquire an estimated sight line difference between the calibration image and the image to be estimated, and acquire a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor;

[0122] The sight line direction acquisition module 40 is used to obtain the sight line direction of the image to be estimated based on the sight line direction of the calibration image, the sight line difference value and the sight line difference estimation deviation.

[0123] In another embodiment provided by the present disclosure, the calibration image includes a first calibration image and a second calibration image of the eye of the target object acquired when the target object wears the target device in the first wearing state, and the first calibration image and the second calibration image correspond to different preset sight lines, respectively; the wearing factor acquisition module is specifically used to:

[0124] Obtaining an estimated first sight difference value between the first calibration image and the second calibration image;

[0125] The first wearing factor is determined based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image.

[0126] In another embodiment provided by the present disclosure, the wearing factor acquisition module is further used to:

[0127] Obtaining a sight difference estimation deviation in the first wearing state based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image;

[0128] Based on the line of sight difference estimation deviation in the first wearing state and a pre-constructed wearing factor network model, the first wearing factor is obtained by back propagation.

[0129] In another embodiment provided by the present disclosure, the wearing factor acquisition module is specifically used to:

[0130] Acquire a reference image; wherein the reference image and the image to be estimated are respectively eye images acquired when the target object wears the target device in the first wearing state;

[0131] Acquire a target sight line difference value; the target sight line difference value comprises a difference between a first value and a second value, the first value being an estimated sight line difference value between the first calibration image and the image to be estimated, and the second value being an estimated sight line difference value between the second calibration image and the reference image;

[0132] Obtaining an estimated second sight line difference value between the reference image and the image to be estimated;

[0133] The second wearing factor is determined based on the first sight line difference value, the second sight line difference value, and the target sight line difference value.

[0134] In another embodiment provided by the present disclosure, the wearing factor acquisition module is further used to:

[0135] obtaining a sight line difference estimation deviation in the second wearing state based on the first sight line difference value, the second sight line difference value and the target sight line difference value;

[0136] Based on the line of sight difference estimation deviation in the second wearing state and a pre-constructed wearing factor network model, the second wearing factor is obtained by back propagation.

[0137] In another embodiment provided by the present disclosure, the device further includes a processing module, and the processing module is specifically configured to:

[0138] Inputting the calibration image and the image to be estimated into a pre-built line of sight estimation model to obtain an estimated line of sight difference between the calibration image and the image to be estimated;

[0139] The first wearing factor and the second wearing factor are input into a pre-constructed wearing factor network model to obtain a sight difference estimation deviation under the combined influence of the first wearing state and the second wearing state.

[0140] In another embodiment provided by the present disclosure, the device further includes a first training module, and the first training module is specifically configured to:

[0141] Acquire a training sample, where the training sample includes a first image sample and a second image sample, where the first image sample includes an eye image obtained when the first subject wears the target device in a third wearing state, and the second image sample includes an eye image obtained when the first subject wears the target device in a fourth wearing state, and the first image sample and the second image sample correspond to line of sight annotation information respectively;

[0142] Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated third sight line difference value between the first image sample and the second image sample;

[0143] Acquire a randomly initialized third wearing factor and a fourth wearing factor; wherein the third wearing factor represents the third wearing state when the first subject wears the target device, and the fourth wearing factor represents the fourth wearing state when the first subject wears the target device;

[0144] Using the third wearing factor and the fourth wearing factor as inputs of a wearing factor network model to be constructed, and obtaining a second sight line difference estimation deviation under the combined influence of the third wearing state and the fourth wearing state;

[0145] Constructing a first loss function based on the third sight line difference value, the second sight line difference estimation deviation, and sight line annotation information corresponding to the first image sample and the second image sample respectively;

[0146] When the first training stop condition is not met, the line of sight estimation model to be constructed and the wearing factor network model to be constructed are updated based on the first loss function; when the first training stop condition is met, the updated line of sight estimation model to be constructed is used as the line of sight estimation model, and the updated wearing factor network model to be constructed is used as the wearing factor network model.

[0147] In another embodiment provided by the present disclosure, the device further includes a second training module, and the second training module is specifically configured to:

[0148] Acquire a training sample, where the training sample includes a first image sample and a second image sample, where the first image sample includes an eye image obtained when the first subject wears the target device in a third wearing state, and the second image sample includes an eye image obtained when the first subject wears the target device in a fourth wearing state, and the first image sample and the second image sample correspond to line of sight annotation information respectively;

[0149] Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated fourth sight line difference value between the first image sample and the second image sample;

[0150] Based on the fourth sight line difference value, the sight line annotation information corresponding to the first image sample and the second image sample, respectively, constructing a second loss function, and when the second stop training condition is not met, updating the sight line estimation model to be constructed based on the second loss function; and when the second stop training condition is met, using the updated sight line estimation model to be constructed as the sight line estimation model;

[0151] Using the first image sample and the second image sample as inputs of the line of sight estimation model, obtaining a fifth line of sight difference value estimated between the first image sample and the second image sample; and obtaining supervision information of the line of sight difference estimation deviation based on the fifth line of sight difference value and line of sight annotation information corresponding to the first image sample and the second image sample respectively;

[0152] Acquire a randomly initialized third wearing factor and a fourth wearing factor; wherein the third wearing factor represents the third wearing state when the first subject wears the target device, and the fourth wearing factor represents the fourth wearing state when the first subject wears the target device;

[0153] Using the third wearing factor and the fourth wearing factor as inputs of a wearing factor network model to be constructed, and obtaining a third sight line difference estimation deviation under the combined influence of the third wearing state and the fourth wearing state;

[0154] Based on the third sight difference estimation deviation and the supervision information, a third loss function is constructed; when the third stop training condition is not met, the wearing factor network model to be constructed is updated based on the third loss function; and when the third stop training condition is met, the updated wearing factor network model to be constructed is used as the wearing factor network model.

[0155] The relevant device part corresponds to the above method embodiment. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0156] The line of sight estimation device provided by the embodiment of the present disclosure obtains the calibration image and the image to be estimated, and obtains the first wearing factor corresponding to the calibration image and the second wearing factor corresponding to the image to be estimated, obtains the estimated line of sight difference between the calibration image and the image to be estimated, and obtains the line of sight difference estimation deviation under the influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor, so as to obtain the line of sight direction of the image to be estimated. In this way, on the basis of the calibration image, combined with the line of sight difference estimation deviation caused by the wearing state that may exist between the calibration image and the image to be estimated, the estimated line of sight difference between the obtained calibration image and the image to be estimated can be corrected, and the calculation accuracy of the line of sight direction of the image to be estimated can be improved, thereby avoiding the problem of inaccurate calculation of the line of sight direction of the image to be estimated caused by ignoring the above-mentioned line of sight difference estimation deviation by directly using the estimated line of sight difference between the calibration image and the image to be estimated.

[0157] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0158] Figure 5 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 5As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0159] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0160] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 6 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0161] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0162] like Figure 6 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0163] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0164] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0165] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0166] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0168] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0169] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0170] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0171] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0172] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0173] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0174] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A line of sight estimation method, characterized in that: The method comprises: Acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction; Acquire a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device; Acquire an estimated sight line difference value between the calibration image and the image to be estimated, and acquire a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor; The sight direction of the image to be estimated is obtained based on the sight direction of the calibration image, the sight difference value, and the sight difference estimation deviation.

2. The method according to claim 1, characterized in that The calibration image includes a first calibration image and a second calibration image of the eye of the target object obtained when the target object wears the target device in a first wearing state, the first calibration image and the second calibration image respectively corresponding to different preset sight lines directions; The obtaining a first wearing factor corresponding to the calibration image includes: Obtaining an estimated first sight difference value between the first calibration image and the second calibration image; The first wearing factor is determined based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image.

3. The method according to claim 2, characterized in that The determining the first wearing factor comprises: Obtaining a sight difference estimation deviation in the first wearing state based on the first sight difference value, the sight direction of the first calibration image, and the sight direction of the second calibration image; Based on the line of sight difference estimation deviation in the first wearing state and a pre-constructed wearing factor network model, the first wearing factor is obtained by back propagation.

4. The method according to claim 2, characterized in that: Obtaining a second wearing factor corresponding to the image to be estimated, including: Acquire a reference image; wherein the reference image and the image to be estimated are respectively eye images acquired when the target object wears the target device in the first wearing state; Acquire a target sight line difference value; the target sight line difference value comprises a difference between a first value and a second value, the first value being an estimated sight line difference value between the first calibration image and the image to be estimated, and the second value being an estimated sight line difference value between the second calibration image and the reference image; Obtaining an estimated second sight line difference value between the reference image and the image to be estimated; The second wearing factor is determined based on the first sight line difference value, the second sight line difference value, and the target sight line difference value.

5. The method according to claim 4, characterized in that The determining the second wearing factor comprises: obtaining a sight line difference estimation deviation in the second wearing state based on the first sight line difference value, the second sight line difference value and the target sight line difference value; Based on the line of sight difference estimation deviation in the second wearing state and a pre-constructed wearing factor network model, the second wearing factor is obtained by back propagation.

6. The method according to claim 5, characterized in that The method further comprises: Inputting the calibration image and the image to be estimated into a pre-built line of sight estimation model to obtain an estimated line of sight difference between the calibration image and the image to be estimated; The first wearing factor and the second wearing factor are input into a pre-constructed wearing factor network model to obtain a sight difference estimation deviation under the combined influence of the first wearing state and the second wearing state.

7. The method according to claim 6, characterized in that The method further comprises: Acquire a training sample, where the training sample includes a first image sample and a second image sample, where the first image sample includes an eye image obtained when the first subject wears the target device in a third wearing state, and the second image sample includes an eye image obtained when the first subject wears the target device in a fourth wearing state, and the first image sample and the second image sample correspond to line of sight annotation information respectively; Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated third sight line difference value between the first image sample and the second image sample; Acquire a third wearing factor and a fourth wearing factor that are randomly initialized respectively; wherein the third wearing factor represents the third wearing state when the first subject wears the target device, and the fourth wearing factor represents the fourth wearing state when the first subject wears the target device; Using the third wearing factor and the fourth wearing factor as inputs of a wearing factor network model to be constructed, and obtaining a second sight line difference estimation deviation under the combined influence of the third wearing state and the fourth wearing state; Constructing a first loss function based on the third sight line difference value, the second sight line difference estimation deviation, and sight line annotation information corresponding to the first image sample and the second image sample respectively; When the first training stop condition is not met, the line of sight estimation model to be constructed and the wearing factor network model to be constructed are updated based on the first loss function; when the first training stop condition is met, the updated line of sight estimation model to be constructed is used as the line of sight estimation model, and the updated wearing factor network model to be constructed is used as the wearing factor network model.

8. The method according to claim 6, characterized in that The method further comprises: Acquire a training sample, where the training sample includes a first image sample and a second image sample, where the first image sample includes an eye image obtained when the first subject wears the target device in a third wearing state, and the second image sample includes an eye image obtained when the first subject wears the target device in a fourth wearing state, and the first image sample and the second image sample correspond to line of sight annotation information respectively; Using the first image sample and the second image sample as inputs of a sight line estimation model to be constructed, and obtaining an estimated fourth sight line difference value between the first image sample and the second image sample; Based on the fourth sight line difference value, the sight line annotation information corresponding to the first image sample and the second image sample, respectively, constructing a second loss function, and when the second stop training condition is not met, updating the sight line estimation model to be constructed based on the second loss function; and when the second stop training condition is met, using the updated sight line estimation model to be constructed as the sight line estimation model; Using the first image sample and the second image sample as inputs of the line of sight estimation model, obtaining a fifth line of sight difference value estimated between the first image sample and the second image sample; and obtaining supervision information of the line of sight difference estimation deviation based on the fifth line of sight difference value and line of sight annotation information corresponding to the first image sample and the second image sample respectively; Acquire a third wearing factor and a fourth wearing factor that are randomly initialized respectively; wherein the third wearing factor represents the third wearing state when the first subject wears the target device, and the fourth wearing factor represents the fourth wearing state when the first subject wears the target device; Using the third wearing factor and the fourth wearing factor as inputs of a wearing factor network model to be constructed, and obtaining a third sight line difference estimation deviation under the combined influence of the third wearing state and the fourth wearing state; Based on the third sight difference estimation deviation and the supervision information, a third loss function is constructed; when the third stop training condition is not met, the wearing factor network model to be constructed is updated based on the third loss function; and when the third stop training condition is met, the updated wearing factor network model to be constructed is used as the wearing factor network model.

9. A sight line estimation device, characterized in that: The device comprises: An image acquisition module, used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction; a wearing factor acquisition module, configured to acquire a first wearing factor corresponding to the calibration image and a second wearing factor corresponding to the image to be estimated, wherein the first wearing factor represents a first wearing state when the target object wears the target device, and the second wearing factor represents a second wearing state when the target object wears the target device; a difference acquisition module, configured to acquire an estimated sight line difference between the calibration image and the image to be estimated, and acquire a sight line difference estimation deviation under the combined influence of the first wearing state and the second wearing state based on the first wearing factor and the second wearing factor; The sight line direction acquisition module is used to obtain the sight line direction of the image to be estimated based on the sight line direction of the calibration image, the sight line difference value and the sight line difference estimation deviation.

10. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.

Images