Sight line estimation method and device, electronic equipment and storage medium

Abstract

The invention relates to a sight line estimation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a calibration image and a to-be-estimated image, and enabling the calibration image to correspond to a known sight line direction; obtaining an estimated sight line difference value between the calibration image and the to-be-estimated image; obtaining accumulated sight line estimation deviations of the calibration image and the to-be-estimated image, wherein the accumulated sight line estimation deviations comprise a first sight line estimation deviation and a second sight line estimation deviation; and obtaining the sight line direction of the to-be-estimated image based on the known sight line direction, the sight line difference value and the accumulated sight line estimation deviation corresponding to the calibration image. Therefore, by introducing the accumulated sight line estimation deviation and combining the estimated sight line direction information and the sight line difference value corresponding to the calibration image, the sight line direction of the to-be-estimated image can be accurately obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a line of sight estimation method, device, electronic device and storage medium. Background Art

[0002] Line of sight is an important clue to reveal how people understand the external environment. Line of sight estimation technology based on line of sight refers to the technology that calculates the user's line of sight through the user's facial image. Line of sight estimation technology has been widely used in human-computer interaction, virtual reality, augmented reality, medical analysis and other fields.

[0003] When a user wears relevant devices, such as AR (Augmented Reality) or VR (Virtual Reality) devices, the user's eye image can be collected. However, when the line of sight is estimated directly from the eye image, the estimated line of sight direction may be inaccurate. Summary of the invention

[0004] The present disclosure provides a line of sight estimation method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, a line of sight estimation method is provided, the method comprising:

[0006] Acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0007] Obtaining an estimated sight line difference between the calibration image and the image to be estimated;

[0008] Obtaining a cumulative line of sight estimation deviation between the calibration image and the image to be estimated, the cumulative line of sight estimation deviation comprising a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation characterizes the influence of a first wearing state on a line of sight direction, the first wearing state corresponding to a wearing state when a target object wears a target device to acquire the calibration image; the second line of sight estimation deviation characterizes the influence deviation of a second wearing state on line of sight estimation compared with the first wearing state, the second wearing state corresponding to a wearing state when the target object wears the target device to acquire the image to be estimated;

[0009] The sight line direction of the image to be estimated is obtained based on the known sight line direction corresponding to the calibration image, the sight line difference value and the accumulated sight line estimation deviation.

[0010] According to another aspect of the present disclosure, a sight line estimation device is provided, the device comprising:

[0011] An image acquisition module, used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0012] A sight line difference value acquisition module, used to acquire an estimated sight line difference value between the calibration image and the image to be estimated;

[0013] A cumulative line of sight estimation deviation acquisition module, used to acquire the cumulative line of sight estimation deviation between the calibration image and the image to be estimated, the cumulative line of sight estimation deviation comprising a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation represents the influence of a first wearing state on the line of sight direction, the first wearing state corresponds to the wearing state of a target object when wearing a target device to acquire the calibration image, the second line of sight estimation deviation represents the influence deviation of a second wearing state on line of sight estimation compared with the first wearing state, the second wearing state corresponds to the wearing state of the target object when wearing the target device to acquire the image to be estimated;

[0014] The sight line direction estimation module is used to obtain the sight line direction of the image to be estimated based on the known sight line direction corresponding to the calibration image, the sight line difference value and the accumulated sight line estimation deviation.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] The line of sight estimation method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain a calibration image and an image to be estimated, obtain an estimated line of sight difference between the calibration image and the image to be estimated, and obtain a cumulative line of sight estimation deviation between the calibration image and the image to be estimated, and obtain the line of sight direction of the image to be estimated based on the known line of sight direction, line of sight difference and cumulative line of sight estimation deviation corresponding to the calibration image. In this way, by introducing the cumulative line of sight estimation deviation and combining the known line of sight direction and line of sight difference corresponding to the calibration image, the line of sight direction of the image to be estimated can be accurately obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a sight line estimation model provided for an exemplary embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a line of sight deviation estimation model provided by an exemplary embodiment of the present disclosure;

[0021] Figure 3 A flowchart of a line of sight estimation method provided for an exemplary embodiment of the present disclosure;

[0022] Figure 4 A schematic block diagram of functional modules of a sight line estimation device provided by an exemplary embodiment of the present disclosure;

[0023] Figure 5 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0024] Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0032] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] When a user wears a target device such as AR or VR, with the user's authorization and consent, images of the user's eyes can be collected in real time through the image acquisition device of the target device, and the line of sight direction can be estimated for the collected eye images. For example, local fine rendering of the scene displayed in the target device can be achieved based on the estimated line of sight direction, that is, fine rendering can be performed only on the scene within the user's gaze range, so as to reduce the hardware cost of the target device.

[0034] Figure 1 is a differential line of sight estimation model, such as Figure 1As shown, the calibration image and the image to be estimated can be input into a differential line of sight estimation model to obtain the estimated line of sight difference between the calibration image and the image to be estimated, and estimate the line of sight direction of the image to be estimated based on the line of sight difference and the known line of sight direction corresponding to the calibration image. Among them, the calibration image is an image collected by guiding the user to look at the calibration point with a known line of sight direction, so the calibration image corresponds to a known line of sight direction, so combined with the difference, the real line of sight direction of the image to be estimated can be estimated. It should be noted that the image involved in the embodiment may include a single-eye image of a user, such as an eye image of the left eye or an eye image of the right eye; the image may also include a binocular image of a user, such as an eye image of the left eye and an eye image of the right eye; it may also be an image obtained by splicing an eye image of the left eye and an eye image of the right eye of a user, and the embodiment is not limited to this.

[0035] However, when obtaining the calibration image, since the calibration image is an eye image collected when the user wears the target device once, when the user wears the target device again, the user's wearing position is likely to change. Since the change in wearing position will affect the estimation of the line of sight direction, the accuracy of estimating the line of sight difference of eye images collected during different wearing processes will decrease. If two eye images are collected at different times during the same wearing process, the estimated line of sight difference between the two eye images will be relatively accurate.

[0036] This is because when estimating the sight line direction of the image of the user's eye, the related technology inputs the calibration image and the image to be estimated into a differential sight line estimation model to obtain the estimated sight line difference between the calibration image and the image to be estimated, and estimates the sight line direction of the image to be estimated based on the sight line difference and the known sight line direction corresponding to the calibration image. Since the calibration image may be an image captured when the user wears the target device in the first wearing state, and the image to be estimated may be an image captured when the user wears the target device in the second wearing state, since there are usually differences in wearing positions between different wearing states, such differences will affect the estimated sight line direction of the image to be estimated, resulting in inaccurate sight line direction of the estimated image to be estimated. In the embodiment, relevant wearing refers to the user wearing a target device such as AR, VR or MR, and different wearing refers to the user taking off the target device and wearing the target device again after wearing it, or it may also refer to the two wearing states before and after the user adjusts the wearing position of the target device during the wearing process. Among them, the wearing state may include the relative position between the eye and the target device.

[0037] Since wearing in different wearing states will have a negative impact on the estimated line of sight direction, in the embodiment provided in the present disclosure, in order to improve the accuracy of the realization direction of the image to be estimated, the cumulative line of sight estimation deviation caused by wearing in different wearing states will be introduced.

[0038] In the embodiments provided by the present disclosure, in the embodiments, A can be used to identify the calibration image collected by the user during a wearing process. Since the calibration image is an image collected by guiding the user to look at the calibration point in a known line of sight direction, the calibration image is equivalent to having a known line of sight direction. The calibration image may include multiple calibration images such as A1 and A2, and A1 and A2 can respectively represent calibration images of different line of sight directions. Corresponding to the calibration image A, B can be used to identify the image collected by the user during another wearing process, and image B can be used as the image to be estimated. Among them, the calibration image A can be an eye image collected when the target user wears the target device in the first wearing state, and the image to be estimated B can be an eye image collected when the target user wears the target device in the second wearing state.

[0039] In an embodiment, the calibration image A and the image to be estimated B can be input into Figure 1 The sight line estimation model outputs the estimated sight line difference diff_AB between the calibration image A and the image to be estimated B. Similarly, the calibration image A1 and the calibration image A2 can be input into the sight line estimation model, and the estimated sight line difference diff_A1A2 between the calibration image A1 and the calibration image A2 can be output.

[0040] In an embodiment, a plurality of calibration images can be obtained, and the estimated sight line differences between the plurality of calibration images and the image to be estimated can be estimated by a sight line estimation model to obtain a plurality of sight line difference values. The plurality of sight line difference values ​​can be weighted averaged to obtain a weighted average value, and the weighted average value can be used as the estimated sight line difference between the calibration image and the image to be estimated, so that the sight line difference value obtained can be more accurate. The weights corresponding to the plurality of calibration images are positively correlated with the absolute values ​​of the estimated sight line differences between the calibration image and the image to be estimated.

[0041] Since the calibration image A1 and the calibration image correspond to known sight lines respectively, it is assumed that the calibration image A1 corresponds to the known first sight line direction labelA1, and the calibration image A2 corresponds to the known second sight line direction labelA2. In this way, the first sight line estimation deviation offset_A corresponding to the calibration image A in the first wearing state can be calculated based on labelA1, labelA2 and diff_A1A2. That is:

[0042] offset_A= labelA1 - labelA2 - diff_A1A2 (1)

[0043] Among them, offset_A represents the influence of the first wearing state on the line of sight direction.

[0044] In an embodiment, the calibration image A and the image B to be estimated can be input into a line of sight deviation estimation model to output a second line of sight estimation deviation offset_diffAB estimated between the calibration image A and the image B to be estimated, wherein offset_diffAB represents the deviation of the influence of the second wearing state on the line of sight estimation compared with the first wearing state.

[0045] It should be noted that in the embodiment, when the calibration image A and the image to be estimated B are input into the line of sight deviation estimation model, the uncropped original image can be used as the input of the line of sight deviation estimation model. Since the size of the original image is relatively large, the area proportion of the target user's eye area in the calibration image A and the image to be estimated B may be relatively small. For example, the area proportion of the target user's eye area in the calibration image A and the image to be estimated B is less than the first threshold. And because the uncropped calibration image and the image to be estimated also include a non-eye area with a certain area proportion, when the uncropped calibration image and the image to be estimated are used as the input of the line of sight deviation estimation model, the information input into the line of sight deviation estimation model will also include information related to the wearing state, so that the influence of the wearing state on the line of sight estimation can be estimated more accurately, so that the accuracy of the line of sight direction estimated by the image to be estimated can be improved.

[0046] Before inputting the calibration image A and the image to be estimated B into the line of sight estimation model, the calibration image A and the image to be estimated B can be cropped in advance, for example, at least part of the area outside the target user's eye area can be cropped out of the calibration image A and the image to be estimated B, respectively, so that the area ratio of the target user's eye area in the cropped calibration image A and the image to be estimated B will be relatively large, for example, the area ratio of the target user's eye area in the cropped calibration image A and the cropped image to be estimated B is greater than the second threshold. Among them, the first threshold is less than the second threshold, and in the embodiment, the first preset and the second threshold can be set respectively according to actual needs. In this way, by using the cropped calibration image and the image to be estimated as the input of the line of sight estimation model, the area ratio of the non-eye area in the image can be reduced, and the negative impact of the non-eye area on the line of sight estimation difference can be minimized to improve the accuracy of the line of sight estimation difference.

[0047] Therefore, in the embodiment, the cumulative sight estimation deviation offset_AB of the calibration image and the image to be estimated can be obtained according to offset_A and offset_diffAB.

[0048] offset_AB = offset_A+ offset_diffAB (2)

[0049] Since the calibration image A and the image to be estimated B are eye images obtained when the target user wears the target device in different wearing states, respectively, in the process of estimating the gaze direction of the image to be estimated B based on the known gaze direction corresponding to the calibration image A, the cumulative gaze estimation deviation offset_AB between the calibration image A and the image to be estimated B will affect the gaze direction of the image to be estimated. Therefore, the gaze direction gaze_B of the image to be estimated B can be estimated by the following method:

[0050] gaze_B = labelA-(diff_AB +offset_AB) (3)

[0051] Where labelA indicates that the calibration image A corresponds to the known estimated sight direction.

[0052] According to the above formula (2) and formula (3), the gaze direction gaze_B estimated by the image to be estimated B can be obtained, so that the gaze direction of the image to be estimated when the target object wears the target device in different wearing states can be accurately obtained.

[0053] In an embodiment, the cumulative line of sight estimation deviations of multiple calibration images and images to be estimated can be obtained respectively, so that multiple cumulative line of sight estimation deviations can be obtained. For example, the average value of the multiple cumulative line of sight estimation deviations can be used as the cumulative line of sight estimation deviation of the calibration image and the image to be estimated. The multiple calibration images can be calibration images such as calibration image A1 and calibration image A2 included in the above-mentioned embodiments.

[0054] In order to obtain the model to be estimated and the line of sight deviation estimation model in the above embodiment, the first preset model and the second preset model may be trained respectively through training samples to obtain the model to be estimated and the line of sight deviation estimation model.

[0055] Therefore, in the embodiment provided by the present disclosure, a first training sample set obtained when the first object wears the target device in the third wearing state can be obtained, and a second training sample set obtained when the first object wears the target device in the fourth wearing state can be obtained. The first training sample set may include multiple sample images such as a first sample image and a second sample image, and the second training sample set may include multiple sample images such as a third sample image and a fourth sample image. The first object may be a user different from the above-mentioned target object, the third wearing state is different from the fourth wearing state, and each image sample corresponds to line of sight annotation information.

[0056] The first sample image and the third image are used as inputs of the first preset model, and the estimated line of sight difference between the first sample image and the third image is output. The second sample image and the fourth sample image are used as inputs of the second preset model, and the estimated line of sight deviation of the fourth wearing state compared to the third wearing state is output. Wherein, in the embodiment, the first sample image and the third sample image are respectively cropped sample images, and the area proportion of the eye area of ​​the first object in the first sample image and the third eye image is greater than the second threshold. The second sample image and the fourth sample image are respectively sample images that have not been collected, and the area proportion of the eye area of ​​the first object in the second sample image and the fourth sample image is less than the first threshold, wherein the first preset is less than the second threshold.

[0057] In this way, the first loss function Loss1 can be constructed based on the sight difference, the sight estimation deviation and the sight annotation information, where:

[0058] Loss1 = label_1 - label_3 _diff_13 - offset_24 (4)

[0059] Among them, label_1 represents the line of sight annotation information corresponding to the first sample image, label_3 represents the line of sight annotation information corresponding to the third sample image, diff_13 represents the estimated line of sight difference between the first sample image and the third sample image, and offset_24 represents the line of sight estimation deviation estimated in the fourth wearing state compared with the third wearing state.

[0060] When the condition for stopping training is not met, the model parameters of the first preset model and the second preset model can be updated according to the first loss function. When the condition for stopping training is met, the updated first preset model can be used as a line of sight estimation model, and the updated second preset model can be used as a line of sight deviation estimation model. Among them, the condition for stopping training can be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition, wherein the preset training time and the preset number of iterations can be set separately according to actual needs, and the preset loss convergence condition can be, for example, when the loss value obtained according to the first loss function is the smallest, it is determined that the preset loss convergence condition is met, etc. The embodiment is not limited to this.

[0061] Based on the above embodiments, in the embodiments provided by the present disclosure, the above-mentioned first sample image and the second sample image are different, and the third sample image and the fourth sample image are different. The first sample image and the second sample image can be used as inputs of the first preset model to output the estimated line of sight difference between the first sample image and the second sample image. In this way, the line of sight estimation deviation corresponding to the third wearing state can be obtained based on the line of sight difference value and the line of sight annotation information corresponding to the first sample image and the second sample image, respectively. In this way, the second loss function Loss2 can be constructed according to the line of sight estimation deviation corresponding to the third wearing state, the estimated line of sight difference value between the first sample image and the second sample image, the line of sight estimation deviation estimated for the fourth wearing state compared to the third wearing state, and the line of sight annotation information:

[0062] Loss2 = label_1 - diff_12 - offset_3 - offset_24 - label_3 (5)

[0063] Wherein, label_1 represents the sight line annotation information corresponding to the first sample image, label_3 represents the sight line annotation information corresponding to the third sample image, diff_12 represents the estimated sight line difference between the first sample image and the second sample image, offset_3 represents the sight line estimation deviation corresponding to the third wearing state, and offset_4 represents the sight line estimation deviation estimated for the fourth wearing state compared with the third wearing state. Wherein, offset_3 can be obtained by substituting the estimated sight line difference between the first sample image and the second sample image, and the sight line annotation information corresponding to the first sample image and the second sample image respectively into the above formula (1); offset_24 represents the sight line estimation deviation estimated for the fourth wearing state compared with the third wearing state, which can be obtained by substituting the estimated sight line difference between the first image and the third image and offset_3 into the above formula (2), or by substituting the estimated sight line difference between the second image and the fourth image and offset_3 into the above formula (2).

[0064] When the condition for stopping training is not met, the model parameters of the first preset model and the second preset model can be updated according to the second loss function. When the condition for stopping training is met, the updated first preset model can be used as a line of sight estimation model, and the updated second preset model can be used as a line of sight deviation estimation model. Among them, the condition for stopping training can be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition, wherein the preset training time and the preset number of iterations can be set separately according to actual needs, and the preset loss convergence condition can be, for example, when the loss value obtained according to the second loss function is the minimum, it is determined that the preset loss convergence condition is met, etc. The embodiment is not limited to this.

[0065] Based on the above embodiment, in another embodiment provided by the present disclosure, a line of sight estimation method is also provided, such as Figure 3 As shown, the method may include the following steps:

[0066] In step S310, a calibration image and an image to be estimated are obtained, wherein the calibration image corresponds to a known sight line direction.

[0067] In an embodiment, for example, when the target user is wearing the target device, an eye image sequence of the target user can be acquired in real time by an image acquisition device provided in the target device, so as to estimate the line of sight direction of the target user in real time based on the acquired eye image. Therefore, the image to be estimated in the embodiment can be any image in the eye image sequence. The calibration image can be an image acquired when the user wears the target device in a first wearing state, and the image to be estimated can be an image acquired when the user wears the target device in a second wearing state. Since the calibration image is an image acquired by guiding the user to look at a calibration point of a known line of sight direction, the calibration image corresponds to a known line of sight direction.

[0068] In step S320, the estimated sight line difference between the calibration image and the image to be estimated is obtained.

[0069] In an embodiment, the calibration image and the image to be estimated can be input into Figure 1 The corresponding line of sight estimation model outputs the estimated line of sight difference between the calibration image and the image to be estimated.

[0070] For example, when the target user wears the target device for the first time, a calibration image is obtained, including calibration image A, and when the target user wears the target device for the second time, an image B to be estimated is obtained. The calibration image A and the image B to be estimated can be input into a differential model, and the estimated line of sight difference between the calibration image A and the image B to be estimated is output.

[0071] In step S330, the accumulated sight line estimation deviation of the calibration image and the image to be estimated is obtained.

[0072] Among them, the cumulative line of sight estimation deviation includes a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation characterizes the influence of a first wearing state on the line of sight direction, the first wearing state corresponds to the wearing state when the target object wears the target device to obtain a calibration image, and the second line of sight estimation deviation characterizes the influence deviation of the second wearing state on the line of sight estimation compared with the first wearing state, the second wearing state corresponds to the wearing state when the target object wears the target device to obtain the image to be estimated.

[0073] In the embodiment, since the calibration image may be an image captured when the user wears the target device for the first time, and the image to be estimated may be an image captured when the user wears the target device for another time, there may be differences in wearing positions between different wearings, so that such differences may affect the line of sight direction of the estimated image to be estimated. Assuming that the target user wears the target device in a standard manner, and in the standard wearing state corresponding to the standard wearing, no deviation in line of sight estimation will be caused, that is, in the standard wearing state, the deviation in line of sight estimation is zero, then when the target user wears the target device for the first time to obtain the calibration image A, the corresponding wearing state is the first wearing state, and the line of sight estimation deviation of the first wearing state compared to the standard wearing state can be obtained, and the line of sight estimation deviation can be used as the first line of sight estimation deviation.

[0074] Similarly, for example, when the target user wears the target device for the second time to obtain the image B to be estimated, the corresponding wearing state is the second wearing state, then the sight line estimation deviation in the second wearing state compared with the first wearing state can be obtained, and the sight line estimation deviation can be used as the second sight line estimation deviation. The cumulative sight line estimation deviation can be determined based on the first sight line estimation deviation and the second sight line estimation deviation.

[0075] It should be noted that, when the target user wears the target device at the same time, if the wearing position changes, the corresponding wearing state will also change, and then there will be a deviation in the line of sight estimation.

[0076] In step S340, the gaze direction of the image to be estimated is obtained based on the known gaze direction, gaze difference and accumulated gaze estimation deviation corresponding to the calibration image.

[0077] In the embodiment, when the target user wears the target device, the wearing states corresponding to different wearing positions may affect the estimation of the line of sight direction, and when the first wearing state corresponding to the calibration image is different from the second wearing state corresponding to the image to be estimated, the line of sight estimation deviation will be further increased. Therefore, the embodiment of the present disclosure introduces the cumulative line of sight estimation deviation, and combines the known line of sight direction and line of sight difference corresponding to the calibration image, so as to accurately obtain the line of sight direction of the image to be estimated.

[0078] Based on the above embodiment, in another embodiment provided by the present disclosure, the calibration image may include a first calibration image and a second calibration image; the above step S330 may further include the following steps:

[0079] Step S331, obtaining an estimated sight line difference between the first calibration image and the second calibration image.

[0080] The first calibration image corresponds to a known first sight line direction, and the second calibration image corresponds to a known second sight line direction.

[0081] In an embodiment, for example, when the target user wears the target device for the first time, multiple calibration images can be obtained. For example, the calibration images can include calibration image A1 and calibration image A2. Calibration image A1 and calibration image A2 can be input into a line of sight estimation model to output an estimated line of sight difference between calibration image A1 and calibration image A2.

[0082] Step S332: determining a first sight line estimation deviation of the calibration image based on the sight line difference value, the first sight line direction, and the second sight line direction.

[0083] In the embodiment, the second line of sight estimation deviation represents the deviation of the influence of the second wearing state on the line of sight estimation compared with the first wearing state. Specifically, the first line of sight estimation deviation can be obtained by formula (1) in the above embodiment, which will not be repeated here.

[0084] Based on the above embodiment, in another embodiment provided by the present disclosure, the calibration image and the image to be estimated respectively include an eye region of the target object, and the area proportion of the eye region of the target object in the calibration image and the image to be estimated is respectively less than a first preset threshold, and the above step S320 may further specifically include the following steps:

[0085] Step S321 , target processing is performed on the calibration image and the image to be estimated, respectively, to obtain a processed calibration image and a processed image to be estimated, respectively.

[0086] The area proportions of the eye regions of the target object in the processed calibration image and the processed image to be estimated are both greater than the second preset threshold, and the first preset threshold is less than the second preset threshold.

[0087] Step S322: using the processed calibration image and the processed image to be estimated as inputs of a line of sight estimation model to determine an estimated line of sight difference between the calibration image and the image to be estimated.

[0088] In an embodiment, the calibration image A and the image to be estimated B may be cropped in advance, and the cropped calibration image A and the image to be estimated B may be used as inputs of the sight line estimation model, so as to crop out the areas outside the eye area of ​​the target user in the calibration image A and the image to be estimated B, respectively, so that the area proportion of the eye area of ​​the target user in the cropped calibration image A and the image to be estimated B will be relatively large. In this way, by using the cropped calibration image and the image to be estimated as inputs of the sight line estimation model, the area proportion of the non-eye area in the image can be reduced, and the negative impact of the non-eye area on the sight line estimation difference can be minimized, so as to improve the accuracy of the sight line estimation difference.

[0089] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S330 may further include the following steps:

[0090] Step S333: using the calibration image and the image to be estimated as inputs of a line of sight deviation estimation model to determine a second line of sight estimation deviation estimated between the calibration image and the image to be estimated.

[0091] In an embodiment, when the calibration image A and the image to be estimated B are input into the line of sight deviation estimation model, the uncropped original image can be used as the input of the line of sight estimation model. Since the size of the original image is large, the area proportion of the target user's eye area in the calibration image A and the image to be estimated B may be relatively small. Therefore, the uncropped calibration image and the image to be estimated also include a non-eye area with a certain area proportion, respectively. In this way, when the uncropped calibration image and the image to be estimated are used as the input of the line of sight deviation estimation model, the information input into the line of sight deviation estimation model will also include information related to the wearing status, so that the impact of the wearing status on the line of sight estimation can be more accurately estimated, thereby improving the accuracy of the line of sight direction estimated by the image to be estimated.

[0092] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0093] Step S411, obtaining N calibration images.

[0094] Wherein, N is a positive integer greater than 1.

[0095] In an embodiment, the N calibration images may include the calibration image A1 and the calibration image A2 in the above embodiment.

[0096] Step S412: respectively determine N estimated sight line differences between the N calibration images and the image to be estimated.

[0097] The N calibration images and the image to be estimated may be respectively input into the line of sight estimation model to obtain N estimated line of sight difference values ​​between the N calibration images and the image to be estimated.

[0098] Step S413: Obtain a weighted average value of the N sight line differences, and use the weighted average value as the estimated sight line difference value between the calibration image and the image to be estimated.

[0099] The weights corresponding to the N calibration images are positively correlated with the absolute values ​​of the estimated sight line differences between the calibration images and the image to be estimated.

[0100] In an embodiment, corresponding weights may be assigned to the N calibration images respectively, and then a weighted average may be taken for the N obtained sight line difference values, and the weighted average may be used as the estimated sight line difference value between the calibration image and the image to be estimated, so that the obtained sight line difference value may be more accurate.

[0101] Step S414, respectively obtain N cumulative sight line estimation deviations of the N calibration images and the image to be estimated, and take the average value of the N cumulative sight line estimation deviations as the cumulative sight line estimation deviation of the calibration image and the image to be estimated.

[0102] The N calibration images are all eye images obtained when the target object wears the target device in the first wearing state.

[0103] In an embodiment, the cumulative line of sight estimation deviation between each calibration image and the image to be estimated in N calibration images can be obtained in the manner of the above embodiment, so that N cumulative line of sight estimation deviations can be obtained, and the N cumulative line of sight estimation deviations are averaged, and the obtained average value is used as the cumulative line of sight estimation deviation between the calibration image and the image to be estimated.

[0104] It should be noted that the above step S414 can be performed on the basis of step S413, and step S414 can be performed on the basis of step S412, and the embodiment is not limited thereto.

[0105] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0106] Step S511, obtaining a first training sample set and a second training sample set.

[0107] Among them, the first training sample set includes multiple sample images of the eyes obtained when the first subject wears the target device in a third wearing state, and the second training sample set includes multiple sample images of the eyes obtained when the first subject wears the target device in a fourth wearing state, each sample image corresponds to line of sight annotation information, and the third wearing state is different from the fourth wearing state.

[0108] In an embodiment, the first object may be a user different from the target object, and the third wearing state is different from the fourth wearing state.

[0109] Step S512: Acquire a first sample image and a second sample image in the first training sample set, and acquire a third sample image and a fourth sample image in the second training sample set.

[0110] In the embodiment, the first sample image and the second sample image may be any image in the first training sample set, respectively. The third sample image and the fourth sample image may be any image in the second training sample set, respectively.

[0111] Step S513: Based on the first sample image and the third sample image, determine an estimated first sight difference between the first sample image and the third sample image by using a first preset model.

[0112] In an embodiment, the first sample image and the second sample image may be used as inputs of a first preset model, and a first sight difference value estimated between the first sample image and the third sample image may be output.

[0113] Step S514: Based on the second sample image and the fourth sample image, determine a third sight line estimation deviation estimated in the fourth wearing state compared to the third wearing state through a second preset model.

[0114] In an embodiment, the second sample image and the fourth sample image may be used as inputs of the second preset model, so that a third line of sight estimation deviation estimated in the fourth wearing state compared to the third wearing state may be output.

[0115] It should be noted that, in the embodiment, the first sample image and the third sample image are respectively cropped sample images, and the area ratio of the eye region of the first object in the first sample image and the third eye image is greater than the second threshold. The second sample image and the fourth sample image are respectively sample images that have not been collected, and the area ratio of the eye region of the first object in the second sample image and the fourth sample image is respectively less than the first threshold, wherein the first preset is less than the second threshold.

[0116] Step S515, constructing a first loss function based on the first sight difference value, the third sight estimation deviation and the sight annotation information, and updating the model parameters of the first preset model and the second preset model based on the first loss function when the first stop training condition is not met. When the first stop training condition is met, the updated first preset model is used as the sight estimation model, and the updated second preset model is used as the sight deviation estimation model.

[0117] In an embodiment, the first loss function can be obtained by formula (4) in the above embodiment. During the training of the first preset model and the second preset model, when the first stop training condition is not reached, the model parameters of the first preset model and the second preset model can be updated according to the loss function. When the first stop training condition is reached, the updated first preset model can be used as a line of sight estimation model, and the updated second preset model can be used as a line of sight deviation estimation model. Among them, the first stop training condition can be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition, and the preset training time and the preset number of iterations can be set separately according to actual needs. The preset loss convergence condition can be, for example, when the loss value obtained according to the first loss function is the minimum, it is determined that the preset loss convergence condition is met, etc. The embodiment is not limited to this.

[0118] Based on the above embodiment, in another embodiment provided by the present disclosure, the above first sample image is different from the second sample image, the third sample image is different from the fourth sample image, and the method may further include the following steps:

[0119] Step S516: Based on the first sample image and the second sample image, determine an estimated second sight line difference between the first sample image and the second sample image by using a first preset model.

[0120] Step S517: determining a fourth line of sight estimation deviation corresponding to the third wearing state based on the second line of sight difference value and the line of sight annotation information corresponding to the first sample image and the second sample image respectively.

[0121] In an embodiment, the first sample image and the second sample image are used as inputs of the first preset model, and an estimated second sight line difference value between the first sample image and the second sample image can be output. In this way, a corresponding fourth sight line estimation deviation can be obtained based on the sight line difference value and the sight line annotation information corresponding to the first sample image and the second sample image respectively. The fourth sight line estimation deviation corresponding to the third wearing state can be obtained by substituting the estimated sight line difference value between the first sample image and the second sample image, and the sight line annotation information corresponding to the first sample image and the second sample image respectively into the above formula (1).

[0122] Step S518, constructing a second loss function based on the first sight difference value, the third sight estimation deviation, the fourth sight estimation deviation and the sight annotation information, and updating the model parameters of the first preset model and the second preset model based on the second loss function when the second stop training condition is not met. When the second stop training condition is met, the updated first preset model is used as the sight estimation model, and the updated second preset model is used as the sight deviation estimation model.

[0123] In an embodiment, the second loss function can be obtained according to formula (5) in the above embodiment. During the training of the first preset model and the second preset model, when the second stop training condition is not reached, the model parameters of the first preset model and the second preset model can be updated according to the loss function. When the second stop training condition is reached, the updated first preset model can be used as a line of sight estimation model, and the updated second preset model can be used as a line of sight deviation estimation model. Among them, the second stop training condition can be at least one of reaching a preset training time, a preset number of iterations, and a preset loss convergence condition, and the preset training time and the preset number of iterations can be set separately according to actual needs. The preset loss convergence condition can be, for example, when the loss value obtained according to the second loss function is the minimum, it is determined that the preset loss convergence condition is met, etc. The embodiment is not limited to this.

[0124] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a sight line estimation device, which may be a server or a chip applied to a server. Figure 4 Schematic block diagram of functional modules of a sight line estimation device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the sight line estimation device comprises:

[0125] An image acquisition module 10 is used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction;

[0126] A sight line difference value acquisition module 20, used to acquire an estimated sight line difference value between the calibration image and the image to be estimated;

[0127] The cumulative line of sight estimation deviation acquisition module 30 is used to acquire the cumulative line of sight estimation deviation between the calibration image and the image to be estimated, wherein the cumulative line of sight estimation deviation includes a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation represents the influence of a first wearing state on the line of sight direction, the first wearing state corresponds to the wearing state when the target object wears the target device to acquire the calibration image, and the second line of sight estimation deviation represents the influence deviation of the second wearing state on the line of sight estimation compared with the first wearing state, the second wearing state corresponds to the wearing state when the target object wears the target device to acquire the image to be estimated;

[0128] The sight line direction estimation module 40 is used to obtain the sight line direction of the image to be estimated based on the known sight line direction corresponding to the calibration image, the sight line difference value and the accumulated sight line estimation deviation.

[0129] In another embodiment provided by the present disclosure, the calibration image includes a first calibration image and a second calibration image, and the cumulative sight line estimation deviation acquisition module is further specifically used to:

[0130] Obtaining an estimated sight line difference between the first calibration image and the second calibration image, wherein the first calibration image corresponds to a known first sight line direction and the second calibration image corresponds to a known second sight line direction;

[0131] A first line of sight estimation deviation of the calibration image is determined based on the line of sight difference value, the first line of sight direction, and the second line of sight direction.

[0132] In another embodiment provided by the present disclosure, the calibration image and the image to be estimated respectively include an eye region of the target object, and an area ratio of the eye region of the target object in the calibration image and the image to be estimated is respectively less than a first preset threshold, and the cumulative line of sight estimation deviation acquisition module is further specifically used to:

[0133] performing target processing on the calibration image and the image to be estimated respectively, to obtain the processed calibration image and the processed image to be estimated respectively; the area ratios of the eye regions of the target object in the processed calibration image and the processed image to be estimated are both greater than a second preset threshold, and the first preset threshold is less than the second preset threshold;

[0134] The calibration image after the target processing and the image to be estimated after the target processing are used as inputs of a line of sight estimation model to determine an estimated line of sight difference between the calibration image and the image to be estimated.

[0135] In another embodiment provided by the present disclosure, the sight difference value acquisition module is further used to:

[0136] The calibration image and the image to be estimated are used as inputs of a line of sight deviation estimation model to determine the second line of sight estimation deviation estimated between the calibration image and the image to be estimated.

[0137] In another embodiment provided by the present disclosure, the device further includes a processing module, which is specifically configured to:

[0138] Acquire N calibration images; where N is a positive integer greater than 1;

[0139] Respectively determining N estimated sight line difference values ​​between the N calibration images and the image to be estimated;

[0140] Obtaining a weighted average of the N sight line difference values, and using the weighted average as the sight line difference value estimated between the calibration image and the image to be estimated; wherein the weights corresponding to the N calibration images are positively correlated with the absolute values ​​of the sight line difference values ​​estimated between the calibration image and the image to be estimated;

[0141] And / or, respectively obtain N cumulative line of sight estimation deviations of the N calibration images and the image to be estimated, and take the average value of the N cumulative line of sight estimation deviations as the cumulative line of sight estimation deviation of the calibration image and the image to be estimated; wherein the N calibration images are all eye images obtained when the target object wears the target device in the first wearing state.

[0142] In another embodiment provided by the present disclosure, the device further includes a first training module, and the first training module is specifically used to:

[0143] Acquire a first training sample set and a second training sample set; wherein the first training sample set includes a plurality of sample images of eyes obtained when the first subject wears the target device in a third wearing state, and the second training sample set includes a plurality of sample images of eyes obtained when the first subject wears the target device in a fourth wearing state, each of the sample images corresponds to line of sight annotation information, and the third wearing state is different from the fourth wearing state;

[0144] Acquire a first sample image and a second sample image in the first training sample set, and acquire a third sample image and a fourth sample image in the second training sample set;

[0145] Based on the first sample image and the third sample image, determining an estimated first sight difference value between the first sample image and the third sample image by using a first preset model;

[0146] Based on the second sample image and the fourth sample image, determining a third sight line estimation deviation of the fourth wearing state compared to the third wearing state through a second preset model;

[0147] A first loss function is constructed based on the first line of sight difference value, the third line of sight estimation deviation and the line of sight annotation information. When the first stop training condition is not met, the model parameters of the first preset model and the second preset model are updated based on the first loss function. When the first stop training condition is met, the updated first preset model is used as the line of sight estimation model, and the updated second preset model is used as the line of sight deviation estimation model.

[0148] In another embodiment provided by the present disclosure, the first sample image is different from the second sample image, and the third sample image is different from the fourth sample image; the device further includes a second training module, and the second training module is specifically used to:

[0149] Based on the first sample image and the second sample image, determining an estimated second sight difference value between the first sample image and the second sample image by using the first preset model;

[0150] Determine a fourth line of sight estimation deviation corresponding to the third wearing state based on the second line of sight difference value and the line of sight annotation information respectively corresponding to the first sample image and the second sample image;

[0151] A second loss function is constructed based on the first line of sight difference value, the third line of sight estimation deviation, the fourth line of sight estimation deviation, and the line of sight annotation information corresponding to the first sample image and the second sample image respectively; when the second stop training condition is not met, the model parameters of the first preset model and the second preset model are updated based on the second loss function; when the second stop training condition is met, the updated first preset model is used as the line of sight estimation model, and the updated second preset model is used as the line of sight deviation estimation model.

[0152] Since the device part corresponds to the above method, please refer to the corresponding description of the above method embodiment for details, which will not be repeated here.

[0153] The sight line estimation device provided by the embodiment of the present disclosure obtains a calibration image and an image to be estimated, obtains an estimated sight line difference between the calibration image and the image to be estimated, and obtains a cumulative sight line estimation deviation between the calibration image and the image to be estimated, and obtains the sight line direction of the image to be estimated based on the known sight line direction, sight line difference and cumulative sight line estimation deviation corresponding to the estimated calibration image. In this way, by introducing the cumulative sight line estimation deviation and combining the known sight line direction and sight line difference corresponding to the calibration image, the sight line direction of the image to be estimated can be accurately obtained.

[0154] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0155] Figure 5 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0156] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0157] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 6 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0158] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0159] like Figure 6As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0160] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0161] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0162] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0163] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0165] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0166] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0169] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0170] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0171] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A line of sight estimation method, characterized in that: The method comprises: Acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction; Obtaining an estimated sight line difference between the calibration image and the image to be estimated; Obtaining a cumulative line of sight estimation deviation between the calibration image and the image to be estimated, the cumulative line of sight estimation deviation comprising a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation characterizes the influence of a first wearing state on a line of sight direction, the first wearing state corresponding to a wearing state when a target object wears a target device to acquire the calibration image; the second line of sight estimation deviation characterizes the influence deviation of a second wearing state on line of sight estimation compared with the first wearing state, the second wearing state corresponding to a wearing state when the target object wears the target device to acquire the image to be estimated; The sight line direction of the image to be estimated is obtained based on the known sight line direction corresponding to the calibration image, the sight line difference value and the accumulated sight line estimation deviation.

2. The method according to claim 1, characterized in that The calibration image includes a first calibration image and a second calibration image; The obtaining the first sight estimation deviation of the calibration image includes: Obtaining an estimated sight line difference between the first calibration image and the second calibration image, wherein the first calibration image corresponds to a known first sight line direction and the second calibration image corresponds to a known second sight line direction; A first sight line estimation deviation of the calibration image is determined based on the sight line difference value, the first sight line direction, and the second sight line direction.

3. The method according to claim 1, characterized in that The calibration image and the image to be estimated respectively include an eye region of the target object, and an area ratio of the eye region of the target object in the calibration image and the image to be estimated is respectively smaller than a first preset threshold; The obtaining an estimated sight line difference between the calibration image and the image to be estimated comprises: performing target processing on the calibration image and the image to be estimated respectively, to obtain the processed calibration image and the processed image to be estimated respectively; the area ratios of the eye regions of the target object in the processed calibration image and the processed image to be estimated are both greater than a second preset threshold, and the first preset threshold is less than the second preset threshold; The calibration image after the target processing and the image to be estimated after the target processing are used as inputs of a line of sight estimation model to determine an estimated line of sight difference between the calibration image and the image to be estimated.

4. The method according to claim 3, characterized in that The obtaining of a second line of sight estimation deviation estimated between the calibration image and the image to be estimated comprises: The calibration image and the image to be estimated are used as inputs of a line of sight deviation estimation model to determine the second line of sight estimation deviation estimated between the calibration image and the image to be estimated.

5. The method according to claim 1, characterized in that The method further comprises: Acquire N calibration images; where N is a positive integer greater than 1; Respectively determining N estimated sight line difference values ​​between the N calibration images and the image to be estimated; Obtaining a weighted average of the N sight line difference values, and using the weighted average as the sight line difference value estimated between the calibration image and the image to be estimated; wherein the weights corresponding to the N calibration images are positively correlated with the absolute values ​​of the sight line difference values ​​estimated between the calibration image and the image to be estimated; And / or, respectively obtain N cumulative line of sight estimation deviations of the N calibration images and the image to be estimated, and take the average value of the N cumulative line of sight estimation deviations as the cumulative line of sight estimation deviation of the calibration image and the image to be estimated; wherein the N calibration images are all eye images obtained when the target object wears the target device in the first wearing state.

6. The method according to claim 4, characterized in that The method further comprises: Acquire a first training sample set and a second training sample set; wherein the first training sample set includes a plurality of sample images of eyes obtained when the first subject wears the target device in a third wearing state, and the second training sample set includes a plurality of sample images of eyes obtained when the first subject wears the target device in a fourth wearing state, each of the sample images corresponds to line of sight annotation information, and the third wearing state is different from the fourth wearing state; Acquire a first sample image and a second sample image in the first training sample set, and acquire a third sample image and a fourth sample image in the second training sample set; Based on the first sample image and the third sample image, determining an estimated first sight difference value between the first sample image and the third sample image by using a first preset model; Based on the second sample image and the fourth sample image, determining a third sight line estimation deviation of the fourth wearing state compared to the third wearing state through a second preset model; A first loss function is constructed based on the first line of sight difference value, the third line of sight estimation deviation and the line of sight annotation information. When the first stop training condition is not met, the model parameters of the first preset model and the second preset model are updated based on the first loss function. When the first stop training condition is met, the updated first preset model is used as the line of sight estimation model, and the updated second preset model is used as the line of sight deviation estimation model.

7. The method according to claim 6, characterized in that The first sample image is different from the second sample image, and the third sample image is different from the fourth sample image; the method further includes: Based on the first sample image and the second sample image, determining an estimated second sight difference value between the first sample image and the second sample image by using the first preset model; Determine a fourth line of sight estimation deviation corresponding to the third wearing state based on the second line of sight difference value and the line of sight annotation information respectively corresponding to the first sample image and the second sample image; A second loss function is constructed based on the first line of sight difference value, the third line of sight estimation deviation, the fourth line of sight estimation deviation and the line of sight annotation information. When the second stop training condition is not met, the model parameters of the first preset model and the second preset model are updated based on the second loss function. When the second stop training condition is met, the updated first preset model is used as the line of sight estimation model, and the updated second preset model is used as the line of sight deviation estimation model.

8. A sight line estimation device, characterized in that: The device comprises: An image acquisition module, used to acquire a calibration image and an image to be estimated; the calibration image corresponds to a known sight line direction; A sight line difference value acquisition module, used to acquire an estimated sight line difference value between the calibration image and the image to be estimated; A cumulative line of sight estimation deviation acquisition module, used to acquire the cumulative line of sight estimation deviation between the calibration image and the image to be estimated, the cumulative line of sight estimation deviation comprising a first line of sight estimation deviation and a second line of sight estimation deviation; the first line of sight estimation deviation represents the influence of a first wearing state on the line of sight direction, the first wearing state corresponds to the wearing state of a target object when wearing a target device to acquire the calibration image, the second line of sight estimation deviation represents the influence deviation of a second wearing state on line of sight estimation compared with the first wearing state, the second wearing state corresponds to the wearing state of the target object when wearing the target device to acquire the image to be estimated; The sight line direction estimation module is used to obtain the sight line direction of the image to be estimated based on the known sight line direction corresponding to the calibration image, the sight line difference value and the accumulated sight line estimation deviation.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images