Method for selecting calibration image, method for estimating line of sight, apparatus, and device

Abstract

Embodiments of the present disclosure relate to a method for selecting a calibration image, a method for estimating line of sight, an apparatus, a device, and a medium. The method for selecting a calibration image comprises: obtaining a plurality of candidate calibration images, a theoretical line-of-sight direction when a target eye fixates on a specified calibration point being labeled in each candidate calibration image; obtaining an estimated line-of-sight difference value and a labeled line-of-sight difference value between every two among the plurality of candidate calibration images, the estimated line-of-sight difference value being obtained by performing estimation by using a pre-trained network model, and the labeled line-of-sight difference value being obtained on the basis of the theoretical line-of-sight directions of the candidate calibration images; and on the basis of the estimated line-of-sight difference value and the labeled line-of-sight difference value between every two among the plurality of candidate calibration images, determining an effective calibration image from among the plurality of candidate calibration images.

Description

Calibration image screening method, sight line estimation method, device and equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese Patent Application No. 202310897038.2 filed on July 20, 2023. The entire teachings of the above application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a calibration image screening method, a sight line estimation method, an apparatus, a device, and a medium. Background Art

[0004] Identifying the direction of a person's gaze is crucial in many scenarios, including gaming, healthcare, and intelligent control. This allows for the implementation of appropriate strategies based on the estimated gaze direction. However, the accuracy of gaze estimation results obtained using related technologies is poor, which can adversely affect their application scenarios. Therefore, improving the accuracy of gaze estimation results is urgently needed.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a calibration image screening method, a line of sight estimation method, an apparatus, a device and a medium.

[0007] An embodiment of the present disclosure provides a method for screening calibration images, comprising: obtaining a plurality of candidate calibration images; wherein the candidate calibration images are eye images taken when a target eye is required to look at a designated calibration point, different calibration images correspond to different designated calibration points, and the calibration images are annotated with a theoretical line of sight direction when the target eye looks at the designated calibration point; obtaining line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the line of sight difference estimation values ​​are estimated using a pre-trained network model, and the line of sight difference annotation values ​​are obtained based on the theoretical line of sight directions of the candidate calibration images; determining a valid calibration image from the plurality of candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated with the valid calibration image meet a preset matching condition.

[0008] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: acquiring multiple candidate calibration images, and using the calibration image screening method provided in the first aspect to screen valid calibration images from the multiple candidate calibration images; in response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; in response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; obtaining a line of sight difference estimate between the valid calibration image and the target image through a pre-trained network model; and estimating the line of sight direction corresponding to the target image based on the theoretical line of sight direction of the valid calibration image and the line of sight difference estimate.

[0009] In a third aspect, an embodiment of the present disclosure provides a calibration image screening device, comprising: a calibration image acquisition module, for acquiring multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are annotated with the theoretical line of sight direction when the target eye looks at the specified calibration point; a line of sight difference value acquisition module, for acquiring line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the multiple candidate calibration images; wherein the line of sight difference estimation values ​​are estimated using a pre-trained network model, and the line of sight difference annotation values ​​are obtained based on the theoretical line of sight direction of the candidate calibration images; a calibration image screening module, for determining a valid calibration image from the multiple candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated with the valid calibration image meet a preset matching condition.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, configured to acquire multiple candidate calibration images, and to screen a valid calibration image from the multiple candidate calibration images using the calibration image screening method provided in the first aspect; an end acquisition module, configured to end acquiring candidate calibration images in response to a screening result in which the number of valid calibration images is not zero; a line of sight difference estimation module, configured to acquire a target image to be estimated in response to a line of sight estimation request, and to obtain the valid calibration image, and to obtain a line of sight difference estimation value between the valid calibration image and the target image through a pre-trained network model; and a line of sight direction estimation module, configured to estimate the line of sight direction corresponding to the target image based on the theoretical line of sight direction of the valid calibration image and the line of sight difference estimation value.

[0011] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor for reading the executable instructions from the memory and executing the instructions to implement the calibration image screening method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for executing the calibration image screening method of the first aspect or implementing the line of sight estimation method of the second aspect.

[0013] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the calibration image screening method of the first aspect, or implements the line of sight estimation method of the second aspect.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] FIG1 is a schematic diagram of a calibration panel provided by an embodiment of the present disclosure;

[0018] FIG2 is a schematic diagram of a flow chart of a calibration image screening method provided by an embodiment of the present disclosure;

[0019] FIG3 is a schematic diagram of an error matrix provided by an embodiment of the present disclosure;

[0020] FIG4 is a flow chart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0021] FIG5 is a schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0022] FIG6 is a schematic structural diagram of a calibration image screening device provided by an embodiment of the present disclosure;

[0023] FIG7 is a schematic structural diagram of a sight line estimation device provided by an embodiment of the present disclosure;

[0024] FIG8 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0027] In related art, in order to accurately track eye movements based on the user's eye activity, it is necessary to first have the user look at a specified calibration point, capture an image of the user's eye as a calibration image, and annotate the theoretical gaze direction (i.e., the direction the user should look when looking at the specified calibration point as required). This allows subsequent gaze estimation and other operations to be performed based on the captured calibration image. In specific implementations, the user can wear devices such as VR glasses, smart glasses, or eye trackers, and look at the calibration points provided by such devices as required. The camera of such a device can then capture the user's eye image. Typically, multiple calibration points are provided to the user. For example, a calibration panel diagram can be shown in FIG1 , where nine calibration points are simultaneously displayed on the calibration panel. It should be noted that the nine calibration points are not displayed to the user at the same time. Only one calibration point is displayed to the user at a time. After the user looks at the calibration point and the corresponding calibration image is captured, the next calibration point is displayed, and so on, so that multiple calibration images can be captured. The number on each circle in FIG1 represents the offset angle of the calibration point on the circle in the specified direction. FIG1 is only an example. In actual applications, calibration points may be displayed in other forms, which are not limited here.

[0028] However, the inventors have discovered through research that some users may not look at the designated calibration point as required. In this case, the theoretical sight direction marked on the calibration image corresponding to the calibration point is incorrect, and the actual sight direction of the captured calibration image does not match the marked theoretical sight direction. In other words, the calibration image is invalid, resulting in inaccurate results from subsequent processing using the calibration image. Although some related technologies take the above issues into consideration and filter the collected calibration images, the filtering method is unreliable. Taking calibration points 1 to 9 as an example, the related technology will select one calibration point (assuming calibration point 1) from the 9 calibration points as the test point, and estimate the sight direction of the calibration image corresponding to calibration point 1 based on the calibration images corresponding to the remaining 8 calibration points through a network model. Specifically, the network model can be used to estimate the sight difference between the calibration image of calibration point 1 and the calibration images of other calibration points. On this basis, combined with the theoretical sight directions of other calibration points, the sight direction estimation result of calibration point 1 can be obtained. Further, the difference between the sight direction estimation result of calibration point 1 and the theoretical sight direction of calibration point 1 can be obtained. The above method is also used to process the remaining calibration points. Each calibration point is used as a test point in turn for line of sight direction estimation. In this way, the difference between the line of sight direction estimation results corresponding to each of the nine calibration points and the theoretical line of sight direction can be obtained. Calibration images whose difference exceeds a preset threshold are then considered invalid calibration points. In other words, the calibration images corresponding to invalid calibration points are invalid. However, the inventors have found that the main problem with the above method is that when estimating the line of sight direction corresponding to a test point based on the calibration image corresponding to the invalid calibration point, the difference corresponding to the test point will be large, that is, the invalid calibration point will interfere with the line of sight direction estimation results of other calibration points. If there are multiple invalid calibration points at the same time, it will be impossible to effectively calculate the line of sight direction estimation results corresponding to other points, resulting in inaccurate differences between the line of sight direction estimation results and the theoretical line of sight direction. It will also make it impossible to reasonably determine the threshold value. It is also difficult to reliably screen out valid calibration images for subsequent applications through the comparison relationship between the threshold value and the difference.

[0029] In summary, the screening method adopted in the related art is not reliable. To improve this problem, the embodiments of the present disclosure provide a calibration image screening method, a line of sight estimation method, an apparatus, a device and a medium, which are described in detail below.

[0030] FIG2 is a flow chart of a calibration image screening method provided by an embodiment of the present disclosure. The method can be performed by a valid calibration image screening device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG2 , the method mainly includes the following steps S202 to S206:

[0031] In step S202, multiple candidate calibration images are obtained. The candidate calibration images are eye images captured while the target eye is directed to gaze at a designated calibration point. Different calibration images correspond to different designated calibration points, and the candidate calibration images are annotated with the theoretical gaze direction of the target eye when gazing at the designated calibration point. Any method for obtaining the candidate calibration images is acceptable and is not a limitation.

[0032] Step S204 , obtaining line-of-sight difference estimation values ​​and line-of-sight difference annotation values ​​between each pair of the plurality of candidate calibration images; wherein the line-of-sight difference estimation values ​​are estimated using a pre-trained network model, and the line-of-sight difference annotation values ​​are obtained based on the theoretical line-of-sight directions of the candidate calibration images.

[0033] The network model can be a neural network model. In practical applications, the network model can also be referred to as a calibration model. The disclosed embodiments do not limit the structure of the network model; any network model capable of estimating the visual line difference between two eye images can be used. For example, the network model can be a differential model.

[0034] Multiple candidate calibration images are paired together to form multiple image groups. These groups are then fed into the network model to generate line-of-sight difference estimates for each image group. This yields line-of-sight difference estimates between each pair of candidate calibration images. Since these line-of-sight difference estimates are derived using a pretrained network model, they are generally accurate and reflect the true line-of-sight difference. Furthermore, based on the difference between the theoretical line-of-sight directions of the two candidate calibration images in each image group, line-of-sight difference annotations are generated for each image group. This yields line-of-sight difference annotations between each pair of candidate calibration images. However, since the theoretical line-of-sight directions of the candidate calibration images may be incorrect, the line-of-sight difference annotations may also be inaccurate.

[0035] In step S206, a valid calibration image is determined from the multiple candidate calibration images based on the estimated sight line differences and the annotated sight line differences between each pair of the multiple candidate calibration images. The estimated sight line direction of the valid calibration image satisfies a preset matching condition with the annotated theoretical sight line direction of the valid calibration image. For example, the difference between the estimated sight line direction of the valid calibration image and the annotated theoretical sight line direction of the valid calibration image is no greater than a preset threshold. The preset threshold can be flexibly set based on needs and can be determined based on empirical values. That is, for a candidate calibration image, only if the difference between its estimated sight line direction and the theoretical sight line direction is small (i.e., no greater than the preset threshold) is it considered a valid calibration image. The estimated sight line direction of the candidate calibration image is relatively accurate and can be estimated using a pre-trained network model or obtained using a sight line direction estimation tool. Any method that can obtain a relatively accurate actual sight line direction is acceptable, and there is no limitation here. The estimated sight line direction of the candidate calibration image can be used to represent the actual sight line direction. If the difference between the theoretical sight line direction and the estimated sight line direction is small, it indicates that the annotated theoretical sight line direction is also relatively accurate and reliable.

[0036] In the embodiment of the present disclosure, the network model is trained and can output relatively accurate results. Therefore, the line of sight difference estimate can be used to characterize the relatively real line of sight difference between the two candidate calibration images. However, considering that the user may not look at the specified calibration point as required, the theoretical line of sight direction annotated on the candidate calibration image may not match the actual line of sight direction of the eye in the candidate calibration image. In other words, the theoretical line of sight direction corresponding to the candidate calibration image may not be accurate. Therefore, the line of sight difference annotation value between the two candidate calibration images obtained based on the theoretical line of sight direction of the candidate calibration image may not be accurate. Based on this, the above method can be analyzed by the estimated value (relatively real and accurate, also called actual value) and the annotation value of the line of sight difference between different candidate calibration images, which helps to reasonably and reliably find a valid calibration image, and the theoretical line of sight direction annotated on the final valid calibration image matches the actual line of sight direction.

[0037] In some embodiments, step S206, i.e., determining a valid calibration image from the plurality of candidate calibration images based on the estimated sight difference values ​​and the sight difference labeled values ​​between each pair of the plurality of candidate calibration images, may be performed with reference to steps A and B below:

[0038] In step A, based on the line of sight difference estimates and line of sight difference labeled values ​​between the multiple candidate calibration images, the error between each pair of the multiple candidate calibration images is obtained; wherein the error is determined based on the difference between the line of sight difference estimates and the line of sight difference labeled values. For example, a specified loss function can be used to determine the error based on the difference between the line of sight difference estimates and the line of sight difference labeled values ​​(i.e., the loss value obtained based on the loss function). The presently disclosed embodiments do not limit the loss function. Taking candidate calibration image A1 and candidate calibration image A2 as an example, the error between candidate calibration image A1 and candidate calibration image A2, err_12 = Loss(F(A1, A2), label1 - label2). Where F(A1, A2) represents the line of sight difference estimate obtained after inputting candidate calibration image A1 and candidate calibration image A2 into the network model, label1 and label2 represent the theoretical line of sight directions corresponding to candidate calibration image A1 and candidate calibration image A2, respectively, label1-label2 represents the line of sight difference annotation value between candidate calibration image A1 and candidate calibration image A2, and Loss represents the loss value calculated using a preset loss function for the difference between the line of sight difference estimate and the line of sight difference annotation value.

[0039] Step B: Determine a valid calibration image from the multiple candidate calibration images based on the errors between each pair of the multiple candidate calibration images. In some embodiments, a valid calibration image can be determined directly from the multiple candidate calibration images based on the error magnitudes. In other embodiments, invalid calibration images can be determined first, followed by valid calibration images. Specifically, considering that if an invalid calibration image exists, the sight line difference annotation values ​​between it and other candidate calibration images will be substantially inaccurate, and the corresponding errors will be relatively significant, invalid calibration images can be determined first, and the images other than the invalid calibration images from the multiple candidate calibration images can be designated as valid calibration images.

[0040] To further ensure that a valid calibration image can be accurately found, in some specific implementation examples, the number of candidate calibration images is no less than three. On this basis, step B can be performed with reference to the following steps B1 to B3:

[0041] In step B1, the errors between the target calibration image and the remaining candidate calibration images in the plurality of candidate calibration images are averaged to obtain a first mean value. Each of the plurality of candidate calibration images is sequentially used as a target calibration image. That is, steps B1 to B3 are executed for each candidate calibration image to determine whether each candidate calibration image is a valid calibration image. The "sequentially" mentioned above can be any order and can be flexibly set. In other words, each candidate calibration image can be used as a target calibration image, and steps B1 to B3 are executed to determine whether it is a valid calibration image. The order in which each candidate calibration image is used as a target calibration image is not limited.

[0042] Step B2: averaging the errors between each pair of the candidate calibration images except the target calibration image in the plurality of candidate calibration images to obtain a second mean value.

[0043] Step B3: Determine whether the target calibration image is a valid calibration image based on the first mean and the second mean. In practical applications, whether the target calibration image is a valid calibration image can be determined based on the difference between the first mean and the second mean. For example, step B3 can be performed with reference to steps B3.1 and B3.2 below:

[0044] Step B3.1: Determine whether the difference between the first mean and the second mean is not greater than a preset threshold. The preset threshold can be flexibly set based on experience and is not limited here.

[0045] Step B3.2: If yes, determine whether the target calibration image is a valid calibration image.

[0046] It is understandable that the errors between an invalid calibration image and other candidate calibration images are generally large, that is, the first mean is generally large, which results in a correspondingly large difference between the first mean and the second mean, thus accurately and reliably determining it as an invalid calibration image. In contrast, the errors between a valid calibration image and other candidate calibration images are generally relatively small, that is, the first mean is generally small, and the difference between the first mean and the second mean is also correspondingly small, and will not exceed a preset threshold, thus allowing it to be relatively accurately and reliably determined as a valid calibration image. Furthermore, for ease of understanding, the following further illustrates steps B1 to B3 provided in the present embodiment in conjunction with an error matrix schematic diagram shown in FIG3 :

[0047] FIG3 illustrates a 9*9 matrix grid, where each horizontal row and each vertical column represents a corresponding calibration point. The grid AiAj corresponding to the i-th row and j-th column represents the error between the candidate calibration image Ai corresponding to calibration point Ai and the candidate calibration image Aj corresponding to calibration point Aj. For example, the grid A1A2 corresponding to the first row and second column represents the error between the candidate calibration image A1 corresponding to calibration point A1 and the candidate calibration image A2 corresponding to calibration point A2. If i=j, the error is zero. For example, the error of A1A1 is 0. In practical applications, it can be directly eliminated or included in the calculation (since the error is 0, it does not affect the final result). Candidate calibration images A1-A9 are analyzed one by one as the above-mentioned target calibration images. In FIG3, candidate calibration image A2 is used as the target calibration image for analysis as an example. At this time, the errors corresponding to all grids in the second row and second column can be averaged to obtain Err_this (corresponding to the first mean value mentioned above). Then, the second row and second column are deleted, and the errors corresponding to the remaining squares are averaged to obtain Err_other (corresponding to the aforementioned second mean). If the difference between Err_this and Err_other is greater than a preset threshold, candidate calibration image A2 is considered an invalid calibration image. In other words, if the difference between Err_this and Err_other is not greater than the preset threshold, candidate calibration image A2 is considered a valid calibration image. It should be noted that FIG3 is merely a simplified example for ease of understanding. The errors between AiAj and AjAi are the same, for example, the errors between A1A2 and A2A1 are the same. In practical applications, only one calculation is required. That is, for A2, the errors between A2 and A1 and A3-A9 are averaged to obtain a first mean, and the errors between A1 and A3-A9 (i.e., the candidate calibration images remaining after A2 is removed) are averaged to obtain a second mean.

[0048] In conjunction with FIG3 , taking the case where the target calibration image is candidate calibration image A2 for analysis as an example, the following four cases are given to fully illustrate the reliability of the above-mentioned method for determining the valid calibration image provided by the embodiment of the present disclosure:

[0049] Case 1: Among multiple candidate calibration images A1 to A9, only one candidate calibration image is invalid (assuming it is A2). When the above operation is performed on candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and other candidate calibration images are generally large, that is, the first mean is usually large, while the other candidate calibration images (A1, A3 to A9) are normal, and the corresponding errors are generally small, that is, the second mean is usually small. Therefore, the difference between the first mean and the second mean will be large, so it can be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0050] Case 2: Among multiple candidate calibration images A1-A9, only one is invalid (assuming it's A3). When performing the above operation using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1 and A4-A9 are relatively small, with only the error between candidate calibration image A2 and A3 being relatively large. Therefore, the first mean is affected by the errors of A2 and A3. However, the errors between A3 and A1 and A4-A9 are also relatively large, and the second mean is affected by the errors between A3 and A1 and A4-A9. Subtracting the first and second means can effectively offset the influence of A3 to a certain extent. Therefore, the difference between the first and second means is not significantly increased, and the current candidate calibration image A2 can be accurately determined to be a valid calibration image.

[0051] Case 3: At least two of the multiple candidate calibration images A1-A9 are invalid (assuming they are A2 and A3). When performing the above operation using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and the other candidate calibration images (A1, A4-A9) are generally large, which will cause the first mean to be biased. Since candidate calibration image A3 is also invalid, the theoretical line of sight direction annotated is inaccurate, so the error between A2 and A3 will also affect the first mean. However, when calculating the second mean of the errors between candidate calibration images A1, A3-A9, the errors between A3 and A1 and A4-A9 are all large, which will also affect the second mean to some extent. When the first mean is subtracted from the second mean, the error between A2 and A3 can be considered to be offset to a certain extent by the error between A3 and A1, A4 to A9. The remaining error is mainly caused by the invalidity of A2 itself. Therefore, the difference between the first mean and the second mean is still large. Therefore, it can still be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0052] Case 4: At least two of the multiple candidate calibration images A1-A9 are invalid (assuming they are A3 and A4). When the above operation is performed on candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1 and A5-A9 are small, with only large errors between candidate calibration image A2 and A3 and A4. The first mean is affected by the errors of A2A3 and A2A4. However, the errors between A3 and A1 and A4-A9 are also large, and the errors between A4 and A1, A3, and A5-A9 are also large. The second mean is affected by these errors. Subtracting the first mean from the second mean can effectively offset the effects of the errors of A3 and A4 to a certain extent. Therefore, the difference between the first mean and the second mean does not increase, and the current candidate calibration image A2 can still be accurately determined to be a valid calibration image.

[0053] From the above analysis, it can be seen that the method for determining the valid calibration image provided by the embodiment of the present disclosure can reasonably and reliably find the valid calibration image from multiple candidate calibration images. Compared with the method adopted in the related art, it can effectively avoid the invalid calibration image interfering with the line of sight direction estimation results of the remaining calibration images. When there are multiple invalid calibration images at the same time, it can also avoid the problem of inaccurate difference between the line of sight direction estimation result and the theoretical line of sight direction caused by the inability to effectively calculate the line of sight direction estimation results corresponding to other calibration images in the related art.

[0054] In some embodiments, the method further includes: initiating a prompt message to recapture the calibration image if none of the multiple candidate calibration images are valid calibration images. In actual applications, if a valid calibration image is present among the multiple candidate calibration images, the valid calibration image can be directly used to perform subsequent operations, such as using the valid calibration image to estimate the user's line of sight. If none of the multiple candidate calibration images are valid, a prompt message to recapture the calibration image is sent to the user in a specified format (such as a voice prompt, graphic prompt, etc.). After obtaining the user's authorization, a valid calibration image can be recaptured, thereby ensuring the reliability of subsequent line of sight estimation.

[0055] Based on the above, the present disclosure also provides a line of sight estimation method. Figure 4 is a flow chart of a line of sight estimation method provided by the present disclosure. The method can be performed by a line of sight estimation device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in Figure 4, the method mainly includes the following steps S402 to S408:

[0056] In step S402, a plurality of candidate calibration images are collected, and a valid calibration image is selected from the plurality of candidate calibration images using the aforementioned calibration image selection method. Detailed information can be found in the aforementioned related content and will not be described in detail here.

[0057] Step S404 : In response to the screening result that the number of valid calibration images is not zero, the acquisition of candidate calibration images is terminated.

[0058] In addition, in response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images may continue to be acquired until a valid calibration image is screened out from the candidate calibration images that continue to be acquired.

[0059] In practical applications, multiple candidate calibration images can be captured each time. The aforementioned screening method can be used to select valid calibration images from these captured candidate calibration images. If the number of valid calibration images screened out is not zero, the acquisition of candidate calibration images can be terminated. If the number of valid calibration images is zero, multiple candidate calibration images need to be captured again until a valid calibration image is screened out, and then the acquisition operation can be terminated. The number of candidate calibration images captured in different times can be the same or different, and there is no limitation here. Through the above method, it is fully guaranteed that a valid calibration image will eventually be obtained.

[0060] Step S406 , in response to the line of sight estimation request, capturing the target image to be estimated, and obtaining a valid calibration image, and obtaining an estimated value of the line of sight difference between the valid calibration image and the target image through a pre-trained network model.

[0061] The line of sight estimation request can be issued directly by the user, or triggered by the user when performing a target operation, or set by the background, and there is no restriction here. In addition, before collecting the target image to be estimated, a prompt message to collect the target image can be sent to the user to obtain user authorization. The network model can be a neural network model. In practical applications, the network model can also be called a calibration model. The embodiment of the present disclosure does not limit the structure of the network model, and any network model that can estimate the line of sight difference between two eye images can be used. For example, the network model can be a differential model.

[0062] Step S408 : Estimate the sight line direction corresponding to the target image according to the theoretical sight line direction of the effective calibration image and the sight line difference estimation value.

[0063] Specifically, the theoretical sight line direction of the effective calibration image can be added to the sight line difference estimate to obtain the sight line direction corresponding to the target image. Since the theoretical sight line direction of the effective calibration image is accurate and the trained network model can output accurate sight line difference estimates, the sight line direction corresponding to the target image is also accurate and reliable.

[0064] When there are multiple valid calibration images, multiple gaze direction estimates corresponding to the target image can be obtained based on the theoretical gaze directions annotated on the multiple valid calibration images and the estimated gaze differences between the multiple valid calibration images and the target image. These multiple gaze direction estimates are then weighted, and the gaze direction corresponding to the target image is estimated based on the weighted result. The gaze direction obtained by combining the two methods in this manner effectively ensures the accuracy of the gaze direction corresponding to the target image.

[0065] Since the calibration image screening method provided by the embodiment of the present disclosure can obtain accurate and reliable effective calibration images, the theoretical line of sight direction annotated on the effective calibration image matches the actual line of sight direction. On this basis, the accuracy of line of sight direction estimation based on the theoretical line of sight direction of the effective calibration image and the estimated line of sight difference between the effective calibration image and the target image is better guaranteed.

[0066] For example, a line of sight estimation process diagram can be shown in FIG5 , which illustrates that a valid calibration image and a target image marked with a theoretical line of sight direction are simultaneously input into a pre-trained network model. The network model can output a line of sight difference estimate. Based on the line of sight difference estimate and the theoretical line of sight direction marked on the valid calibration image, the line of sight direction corresponding to the target image can be obtained by summing the line of sight difference estimate and the theoretical line of sight direction marked on the valid calibration image. Furthermore, the network model can be a differential model. FIG5 also illustrates that the network model includes a feature extraction network and a line of sight difference prediction network. The feature extraction network is used to extract the feature vectors of the valid calibration image and the target image respectively. The line of sight difference prediction network can analyze and process the feature vectors of the valid calibration image and the target image respectively to generate a line of sight difference estimate. In specific implementation, the feature vectors of the valid calibration image and the target image can be spliced ​​to obtain spliced ​​features, and then the line of sight difference prediction network can analyze and process the spliced ​​features to obtain a line of sight difference estimate. The above is only an implementation example of the network model and should not be regarded as limiting.

[0067] In practical applications, post-processing such as smoothing can be performed based on the output results of the network model to further optimize the estimation results of the line of sight direction, which is not limited here.

[0068] In summary, through the above-mentioned method provided by the embodiment of the present disclosure, a more accurate and reliable line of sight direction estimation result can be obtained.

[0069] Corresponding to the aforementioned calibration image screening method, FIG6 is a schematic structural diagram of a calibration image screening device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG6 , the calibration image screening device includes:

[0070] The calibration image acquisition module 602 is configured to acquire multiple candidate calibration images. The candidate calibration images are eye images captured when the target eye is required to gaze at a designated calibration point. Different calibration images correspond to different designated calibration points, and the calibration images are annotated with the theoretical gaze direction of the target eye when gazing at the designated calibration point.

[0071] The sight difference acquisition module 604 is used to obtain sight difference estimates and sight difference annotations between each of the multiple candidate calibration images. The sight difference estimates are estimated using a pre-trained network model, and the sight difference annotations are obtained based on the theoretical sight directions of the candidate calibration images.

[0072] The calibration image screening module 606 is configured to determine a valid calibration image from the plurality of candidate calibration images based on the estimated sight line differences and the annotated sight line differences between the plurality of candidate calibration images; wherein the estimated sight line direction of the valid calibration image and the theoretical sight line direction annotated on the valid calibration image meet a preset matching condition.

[0073] The above-mentioned device can analyze the estimated values ​​and labeled values ​​of the line of sight differences between different candidate calibration images, which helps to reasonably and reliably find a valid calibration image. The theoretical line of sight direction labeled in the final valid calibration image matches the estimated line of sight direction that can be used to characterize the actual situation.

[0074] In some embodiments, the calibration image screening module 606 is specifically used to: obtain the error between the multiple candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images; wherein the error is determined based on the difference between the line of sight difference estimation values ​​and the line of sight difference annotation values; based on the error between the multiple candidate calibration images, determine a valid calibration image from the multiple candidate calibration images.

[0075] In some embodiments, the number of the candidate calibration images is no less than three; the calibration image screening module 606 is specifically used to: average the errors between the target calibration image and the remaining candidate calibration images in the multiple candidate calibration images to obtain a first mean; wherein each candidate calibration image in the multiple candidate calibration images is sequentially used as the target calibration image; average the errors between each two candidate calibration images in the multiple candidate calibration images except the target calibration image to obtain a second mean; and determine whether the target calibration image is a valid calibration image based on the first mean and the second mean.

[0076] In some implementations, the calibration image screening module 606 is specifically configured to: determine whether a difference between the first mean and the second mean is not greater than a preset threshold; if so, determine that the target calibration image is a valid calibration image.

[0077] In some embodiments, the apparatus further includes an information initiating module configured to initiate a prompt message for re-capturing a calibration image when none of the multiple candidate calibration images are valid calibration images.

[0078] The calibration image screening device provided in the embodiment of the present disclosure can execute the calibration image screening method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0080] Corresponding to the aforementioned line of sight estimation method, FIG7 is a schematic structural diagram of a line of sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG7 , the line of sight estimation device includes:

[0081] An image acquisition module 702 is configured to acquire a plurality of candidate calibration images and select a valid calibration image from the plurality of candidate calibration images using the calibration image screening method.

[0082] an end acquisition module 704, configured to end acquisition of candidate calibration images in response to a screening result indicating that the number of valid calibration images is not zero;

[0083] The sight difference estimation module 706 is configured to, in response to a sight line estimation request, capture a target image to be estimated, obtain the valid calibration image, and obtain a sight line difference estimation value between the valid calibration image and the target image using a pre-trained network model;

[0084] The sight line direction estimation module 708 is used to estimate the sight line direction corresponding to the target image based on the theoretical sight line direction of the effective calibration image and the sight line difference estimation value.

[0085] Since the calibration image screening method provided by the embodiment of the present disclosure can obtain accurate and reliable effective calibration images, the theoretical line of sight direction annotated on the effective calibration image matches the actual line of sight direction. On this basis, the accuracy of line of sight direction estimation based on the theoretical line of sight direction of the effective calibration image and the estimated line of sight difference between the effective calibration image and the target image is better guaranteed.

[0086] In some embodiments, the apparatus further comprises: in response to a screening result that the number of valid calibration images is zero, continuing to acquire multiple candidate calibration images until a valid calibration image is screened out from the continuously acquired candidate calibration images.

[0087] The gaze direction estimation device provided in the embodiments of the present disclosure can execute the gaze direction estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0089] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned calibration image screening method or the above-mentioned line of sight estimation method.

[0090] FIG8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG8 , the electronic device 800 includes one or more processors 801 and a memory 802 .

[0091] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0092] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the calibration image screening method, line of sight estimation method, and / or other desired functions of the embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0093] In one example, the electronic device 800 may further include an input device 803 and an output device 804 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0094] In addition, the input device 803 may also include, for example, a keyboard, a mouse, and the like.

[0095] The output device 804 can output various information to the outside, including determined distance information, direction information, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0096] Of course, for simplicity, FIG8 shows only some of the components related to the present disclosure in the electronic device 800, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 800 may further include any other appropriate components according to specific application scenarios.

[0097] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the calibration image screening method or line of sight estimation method provided by the embodiment of the present disclosure.

[0098] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0099] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the calibration image screening method or line of sight estimation method provided by the embodiment of the present disclosure.

[0100] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0101] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the calibration image screening method or the line of sight estimation method in the embodiments of the present disclosure.

[0102] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0103] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. For example, when capturing an eye image of the user, a prompt message is sent to the user, and the user can be further informed of the purpose of the image capture in the prompt message. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message, and only perform the operations of the disclosed technical solution after obtaining user authorization.

[0104] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0105] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0107] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for screening a calibration image, comprising: Acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the candidate calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point; Obtaining line-of-sight difference estimation values ​​and line-of-sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the line-of-sight difference estimation values ​​are estimated using a pre-trained network model, and the line-of-sight difference annotation values ​​are obtained based on theoretical line-of-sight directions of the candidate calibration images; According to the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

2. The method according to claim 1, wherein: The determining a valid calibration image from the plurality of candidate calibration images according to the sight difference estimation values ​​and sight difference annotation values ​​between the plurality of candidate calibration images comprises: Obtaining errors between the plurality of candidate calibration images according to the estimated sight difference values ​​and the marked sight difference values ​​between the plurality of candidate calibration images; wherein the errors are determined based on the difference between the estimated sight difference values ​​and the marked sight difference values; Based on the errors between the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images.

3. The method according to claim 2, wherein: The number of the candidate calibration images is no less than three; The determining a valid calibration image from the plurality of candidate calibration images based on errors between each of the plurality of candidate calibration images comprises: Averaging the errors between the target calibration image and the remaining candidate calibration images in the plurality of candidate calibration images to obtain a first mean value; wherein each candidate calibration image in the plurality of candidate calibration images is sequentially used as the target calibration image; Averaging the errors between each of the candidate calibration images except the target calibration image among the plurality of candidate calibration images to obtain a second mean value; Whether the target calibration image is a valid calibration image is determined according to the first mean value and the second mean value.

4. The method according to claim 3, wherein: The determining, based on the first mean value and the second mean value, whether the target calibration image is a valid calibration image includes: Determining whether a difference between the first mean and the second mean is not greater than a preset threshold; If so, it is determined that the target calibration image is a valid calibration image.

5. The method according to claim 1, wherein: The method further comprises: In the case that none of the multiple candidate calibration images is a valid calibration image, a prompt message for re-capturing the calibration image is initiated.

6. A line of sight estimation method, comprising: Acquire a plurality of candidate calibration images, and use the calibration image screening method according to any one of claims 1 to 5 to screen a valid calibration image from the plurality of candidate calibration images; In response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; In response to a line of sight estimation request, a target image to be estimated is acquired, and the effective calibration image is obtained, and a line of sight difference estimation value between the effective calibration image and the target image is obtained through a pre-trained network model; The line of sight direction corresponding to the target image is estimated according to the theoretical line of sight direction of the effective calibration image and the line of sight difference estimation value.

7. The method according to claim 6, wherein: The method further comprises: In response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images continue to be acquired until a valid calibration image is screened out from the candidate calibration images that continue to be acquired.

8. A screening device for calibration images, comprising: A calibration image acquisition module, used to acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point; A sight difference value acquisition module, used to acquire sight difference estimation values ​​and sight difference annotation values ​​between the plurality of candidate calibration images; wherein the sight difference estimation values ​​are estimated by using a pre-trained network model, and the sight difference annotation values ​​are obtained based on theoretical sight directions of the candidate calibration images; A calibration image screening module is used to determine a valid calibration image from the multiple candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

9. A sight line estimation device, comprising: An image acquisition module, used for acquiring a plurality of candidate calibration images, and using the calibration image screening method according to any one of claims 1 to 5 to screen a valid calibration image from the plurality of candidate calibration images; an end acquisition module, configured to end acquisition of candidate calibration images in response to a screening result that the number of valid calibration images is not zero; A sight difference estimation module, configured to, in response to a sight estimation request, acquire a target image to be estimated, obtain the effective calibration image, and obtain a sight difference estimation value between the effective calibration image and the target image through a pre-trained network model; The sight line direction estimation module is used to estimate the sight line direction corresponding to the target image according to the theoretical sight line direction of the effective calibration image and the sight line difference estimation value.

10. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

11. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

Images