Calibration image screening method, sight line estimation method, device and equipment

Abstract

The embodiment of the invention relates to a calibration image screening method and device, a sight line estimation method and device, equipment and a medium. The calibration image screening method comprises the steps that multiple candidate calibration images are acquired; the candidate calibration image is marked with a theoretical line-of-sight direction when the target eye watches the specified calibration point; obtaining a sight line difference estimation value and a sight line difference annotation value between every two candidate calibration images; the line-of-sight estimation value is obtained by estimating a pre-trained network model, and the line-of-sight annotation value is obtained based on the theoretical line-of-sight direction of the candidate calibration image; and determining an effective calibration image from the plurality of candidate calibration images according to the sight line difference estimation value and the sight line difference annotation value between every two of the plurality of candidate calibration images. According to the embodiment of the invention, the screening reliability of the effective calibration image can be effectively guaranteed, and the accuracy of sight line direction estimation is further guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a calibration image screening method, a sight line estimation method, a device, a equipment and a medium. Background Art

[0002] In many scenarios such as gaming, medical, and intelligent control, it is necessary to identify the direction of human gaze so that corresponding strategies can be adopted based on the estimation results of the direction of human gaze. However, the accuracy of the gaze estimation results obtained by related technologies is poor, which can easily bring adverse effects to the application scenarios of the gaze estimation results. Therefore, it is urgent to improve the accuracy of the gaze estimation results. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a calibration image screening method, a line of sight estimation method, an apparatus, a device and a medium.

[0004] The present disclosure provides a method for screening calibration images, comprising: obtaining a plurality of candidate calibration images; wherein the candidate calibration images are eye images taken when a target eye is required to look at a designated calibration point, different calibration images correspond to different designated calibration points, and the calibration images are annotated with a theoretical line of sight direction when the target eye looks at the designated calibration point; obtaining line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the line of sight difference estimation values ​​are estimated using a pre-trained network model, and the line of sight difference annotation values ​​are obtained based on the theoretical line of sight direction of the candidate calibration images; determining a valid calibration image from the plurality of candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

[0005] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: acquiring multiple candidate calibration images, and using the calibration image screening method provided in the first aspect to screen valid calibration images from the multiple candidate calibration images; in response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; in response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; obtaining a line of sight difference estimate between the valid calibration image and the target image through a pre-trained network model; and estimating the line of sight direction corresponding to the target image based on the theoretical line of sight direction of the valid calibration image and the line of sight difference estimate.

[0006] In a third aspect, an embodiment of the present disclosure provides a calibration image screening device, comprising: a calibration image acquisition module, used to acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are annotated with a theoretical line of sight direction when the target eye looks at the specified calibration point; a line of sight difference value acquisition module, used to acquire line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the multiple candidate calibration images; wherein the line of sight difference estimation values ​​are estimated using a pre-trained network model, and the line of sight difference annotation values ​​are obtained based on the theoretical line of sight direction of the candidate calibration images; a calibration image screening module, used to determine a valid calibration image from the multiple candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between each of the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, used to acquire multiple candidate calibration images, and use the calibration image screening method provided in the first aspect to screen a valid calibration image from the multiple candidate calibration images; an end acquisition module, used to end the acquisition of candidate calibration images in response to a screening result that the number of valid calibration images is not zero; a line of sight difference estimation module, used to acquire a target image to be estimated in response to a line of sight estimation request, and obtain the valid calibration image, and obtain a line of sight difference estimation value between the valid calibration image and the target image through a pre-trained network model; a line of sight direction estimation module, used to estimate the line of sight direction corresponding to the target image based on the theoretical line of sight direction of the valid calibration image and the line of sight difference estimation value.

[0008] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the calibration image screening method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0009] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for executing the calibration image screening method of the first aspect, or implementing the line of sight estimation method of the second aspect.

[0010] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the calibration image screening method of the first aspect, or implements the line of sight estimation method of the second aspect.

[0011] The above technical solution provided by the embodiment of the present disclosure can determine a valid calibration image from multiple candidate calibration images based on the estimated values ​​of the sight difference between the multiple candidate calibration images (obtained based on the network model estimation) and the sight difference annotation values ​​(obtained based on the theoretical sight direction annotated by the candidate calibration image). The above method can be analyzed by the estimated values ​​and annotation values ​​of the sight difference between different candidate calibration images, which helps to reasonably and reliably find the valid calibration image. The theoretical sight direction annotated by the valid calibration image finally obtained meets the preset matching condition with the estimated sight direction. On this basis, the accuracy of the sight direction estimation based on the annotated sight direction of the valid calibration image and the estimated sight difference between the valid calibration image and the target image is better guaranteed.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 A schematic diagram of a calibration panel provided in an embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of a flow chart of a calibration image screening method provided in an embodiment of the present disclosure;

[0017] Figure 3 A schematic diagram of an error matrix provided in an embodiment of the present disclosure;

[0018] Figure 4 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram of the structure of a calibration image screening device provided in an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of the structure of a sight line estimation device provided by an embodiment of the present disclosure;

[0022] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0025] In related technologies, in order to accurately implement eye tracking based on the activity state of the user's eyes, it is necessary to let the user look at the specified calibration points in advance, so as to collect the user's eye image as the calibration image, and mark the theoretical line of sight direction (that is, the line of sight direction that the user should have when looking at the specified calibration points as required), so as to perform line of sight estimation and other operations based on the collected calibration image. In specific implementation, the user can wear devices such as VR glasses, smart glasses, eye trackers, etc., and look at the calibration points provided by such devices as required, and the user's eye image can be collected through the camera of such devices. Usually, multiple calibration points will be provided to the user. For example, you can refer to Figure 1 The schematic diagram of a calibration panel shown in FIG. 9 shows 9 calibration points on the calibration panel at the same time. It should be noted that the 9 calibration points are not displayed to the user at the same time. Only one calibration point is displayed to the user each time. After the user looks at the calibration point and collects the corresponding calibration image, the next calibration point is displayed, and so on. Multiple calibration images can be collected. Among them, Figure 1 The number on each circle in represents the offset angle of the calibration point on the circle in the specified direction. Figure 1 This is only an example. In actual applications, the calibration points may be displayed in other forms, which are not limited here.

[0026] However, the inventor has found through research that some users may not look at the specified calibration point as required, then the theoretical sight line direction marked on the calibration image corresponding to the calibration point is wrong, and the actual sight line direction of the calibration image obtained by shooting does not match the marked theoretical sight line direction, that is, the calibration image is invalid, resulting in inaccurate results of subsequent processing using the calibration image. Although some related technologies take the above problems into consideration and filter the collected calibration images, the screening method is unreliable. Taking calibration points 1 to 9 as an example, the related technology will select one calibration point (assuming calibration point 1) among the 9 calibration points as the test point, and estimate the sight line direction of the calibration image corresponding to calibration point 1 through the network model based on the calibration images corresponding to the remaining 8 calibration points. Specifically, the network model can be used to estimate the sight line difference estimation value between the calibration image of calibration point 1 and the calibration images of other calibration points. On this basis, combined with the theoretical sight line directions of other calibration points, the sight line direction estimation result of calibration point 1 can be obtained, and further, the difference between the sight line direction estimation result of calibration point 1 and the theoretical sight line direction of calibration point 1 can be obtained. The above method is also used for the remaining calibration points. Each calibration point is used as a test point in turn to estimate the line of sight direction, so that the difference between the line of sight direction estimation results corresponding to each of the 9 calibration points and the theoretical line of sight direction can be obtained. Then the calibration image whose difference exceeds the preset threshold is regarded as an invalid calibration point. In other words, the calibration image corresponding to the invalid calibration point is invalid. However, the inventor found that the main problem with the above method is that when estimating the line of sight direction corresponding to the test point based on the calibration image corresponding to the invalid calibration point, the difference corresponding to the test point will be large, that is, the invalid calibration point will interfere with the line of sight direction estimation results of other calibration points. If there are multiple invalid calibration points at the same time, it is even more impossible to effectively calculate the line of sight direction estimation results corresponding to other points, which will cause the difference between the line of sight direction estimation result and the theoretical line of sight direction to be inaccurate, and it will also lead to the inability to reasonably determine the threshold. It is also difficult to reliably screen out effective calibration images for subsequent applications through the comparison relationship between the threshold and the difference.

[0027] In summary, the screening method adopted by the related art is not reliable. To improve this problem, the embodiments of the present disclosure provide a screening method for calibration images, a line of sight estimation method, device, equipment and medium, which are described in detail below.

[0028] Figure 2 The present invention provides a flowchart of a calibration image screening method provided by an embodiment of the present invention. The method can be performed by a screening device for effective calibration images, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 2 As shown, the method mainly includes the following steps S202 to S206:

[0029] Step S202, obtaining multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the candidate calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point. Any method of obtaining the candidate calibration images is acceptable and is not limited here.

[0030] Step S204, obtaining line of sight difference estimation values ​​and line of sight difference annotation values ​​between each pair of the plurality of candidate calibration images; wherein the line of sight difference estimation values ​​are estimated by using a pre-trained network model, and the line of sight difference annotation values ​​are obtained based on theoretical line of sight directions of the candidate calibration images.

[0031] The network model may be a neural network model. In practical applications, the network model may also be referred to as a calibration model. The disclosed embodiments do not limit the structure of the network model. Any network model that can estimate the line of sight difference between two eye images may be used. For example, the network model may be a differential model.

[0032] Multiple candidate calibration images are combined in pairs to obtain multiple image groups, and the multiple image groups are respectively input into the network model to obtain the line of sight difference estimation value corresponding to each image group output by the network model, that is, the line of sight difference estimation value between the multiple candidate calibration images is obtained. Since the line of sight difference estimation value is estimated by using a pre-trained network model, it is generally more accurate and can better reflect the actual line of sight difference. In addition, based on the difference between the theoretical line of sight directions of the two candidate calibration images in each image group, the line of sight difference annotation value corresponding to each image group can be obtained, that is, the line of sight difference annotation value between the multiple candidate calibration images is obtained. Since the theoretical line of sight directions of the candidate calibration images may be wrong, the line of sight difference annotation value may also be inaccurate.

[0033] Step S206, determining a valid calibration image from the multiple candidate calibration images according to the estimated sight line difference values ​​and sight line difference annotation values ​​between the multiple candidate calibration images; wherein the estimated sight line direction of the valid calibration image and the theoretical sight line direction annotated by the valid calibration image meet the preset matching condition. Exemplarily, the direction difference between the estimated sight line direction of the valid calibration image and the theoretical sight line direction annotated by the valid calibration image is not greater than a preset threshold value, and the preset threshold value can be flexibly set according to the demand, and can be specifically determined based on an empirical value. That is, for a candidate calibration image, only if the difference between its estimated sight line direction and the theoretical sight line direction is small (that is, not greater than the preset threshold value), can it be regarded as a valid calibration image, wherein the estimated sight line direction of the candidate calibration image is relatively accurate, and can be estimated by a pre-trained network model, or obtained by a sight line direction estimation tool, and any method that can obtain a relatively accurate actual sight line direction is acceptable, and is not limited here, and the estimated sight line direction of the candidate calibration image can be used to characterize the actual sight line direction, and if the difference between the theoretical sight line direction and the estimated sight line direction is small, it means that the annotated theoretical sight line direction is also relatively accurate and reliable.

[0034] In the disclosed embodiment, the network model is trained and can output relatively accurate results. Therefore, the line of sight difference estimation value can be used to characterize the relatively real line of sight difference between two candidate calibration images. However, considering that the user may not look at the specified calibration point as required, the theoretical line of sight direction annotated by the candidate calibration image may not match the actual line of sight direction of the eye in the candidate calibration image. In other words, the theoretical line of sight direction corresponding to the candidate calibration image may not be accurate. Therefore, the line of sight difference annotation value between the two candidate calibration images obtained based on the theoretical line of sight direction of the candidate calibration image may not be accurate. Based on this, the above method can be analyzed through the estimated value (relatively real and accurate, also referred to as the actual value) and the annotation value of the line of sight difference between different candidate calibration images, which helps to reasonably and reliably find a valid calibration image, and the theoretical line of sight direction annotated by the final valid calibration image matches the actual line of sight direction.

[0035] In some implementations, the above step S206, i.e., determining a valid calibration image from a plurality of candidate calibration images according to the estimated sight difference values ​​and sight difference labeled values ​​between each of the plurality of candidate calibration images, may be performed with reference to the following steps A and B:

[0036] Step A, according to the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images, obtain the error between the multiple candidate calibration images; wherein the error is determined based on the difference between the line of sight difference estimation value and the line of sight difference annotation value, and illustratively, the error (that is, the loss value obtained based on the loss function) can be determined based on the difference between the line of sight difference estimation value and the line of sight difference annotation value using a specified loss function. The disclosed embodiment does not limit the loss function. Taking candidate calibration image A1 and candidate calibration image A2 as an example, the error err_12 between candidate calibration image A1 and candidate calibration image A2 is err_12=Loss(F(A1,A2),label1-label2). Among them, F(A1, A2) represents the line of sight difference estimation value obtained after inputting candidate calibration image A1 and candidate calibration image A2 into the network model, label1 and label2 represent the theoretical line of sight directions corresponding to candidate calibration image A1 and candidate calibration image A2 respectively, label1-label2 represents the line of sight difference annotation value between candidate calibration image A1 and candidate calibration image A2, and Loss represents the difference between the line of sight difference estimation value and the line of sight difference annotation value, and the loss value is calculated using the preset loss function.

[0037] Step B, based on the errors between the multiple candidate calibration images, determine the valid calibration image from the multiple candidate calibration images. In some embodiments, the valid calibration image can be directly determined from the multiple candidate calibration images based on the error size. In other embodiments, the invalid calibration image can be determined first, and then the valid calibration image can be determined. Specifically, considering that if there is an invalid calibration image, the sight difference annotation value between it and other candidate calibration images is basically inaccurate, and the corresponding error will also be more obvious, so the invalid calibration image can be determined first, and the images other than the invalid calibration image in the multiple candidate calibration images are used as valid calibration images.

[0038] In order to further ensure that a valid calibration image can be accurately found, in some specific implementation examples, the number of candidate calibration images is not less than three; on this basis, step B can be performed with reference to the following steps B1 to B3:

[0039] Step B1, averaging the errors between the target calibration image and the remaining candidate calibration images in the multiple candidate calibration images to obtain a first mean value; wherein, each of the multiple candidate calibration images is sequentially used as the target calibration image, that is, for each candidate calibration image, steps B1 to B3 are executed to determine whether each candidate calibration image is a valid calibration image one by one. The above "sequentially" can be any order, which can be flexibly set. In other words, each candidate calibration image can be used as a target calibration image, and steps B1 to B3 are used to determine whether it is a valid calibration image. The order in which each candidate calibration image is used as a target calibration image is not limited here.

[0040] Step B2: average the errors between each pair of the candidate calibration images except the target calibration image in the plurality of candidate calibration images to obtain a second mean value.

[0041] Step B3, judging whether the target calibration image is a valid calibration image according to the first mean and the second mean. In practical applications, whether the target calibration image is a valid calibration image can be judged according to the difference between the first mean and the second mean. Exemplarily, step B3 can be performed with reference to the following steps B3.1 and B3.2:

[0042] Step B3.1, determine whether the difference between the first mean and the second mean is not greater than a preset threshold. The preset threshold can be flexibly set based on experience and is not limited here.

[0043] Step B3.2: If yes, determine whether the target calibration image is a valid calibration image.

[0044] It can be understood that the errors between the invalid calibration image and other candidate calibration images are generally large, that is, the first mean is generally large, which will lead to a correspondingly large difference between the first mean and the second mean, so that it can be accurately and reliably determined to be an invalid calibration image. The errors between the valid calibration image and other candidate calibration images are mostly relatively small, that is, the first mean is generally small, and the difference between the first mean and the second mean is also correspondingly small, and will not be greater than the preset threshold, so it can be relatively accurately and reliably determined to be a valid calibration image. Further, for ease of understanding, the following is combined with Figure 3 The error matrix diagram shown in FIG. 1 further illustrates the above steps B1 to B3 provided in the embodiment of the present disclosure:

[0045] Figure 3A 9*9 matrix grid is shown, where each horizontal row and each vertical column represent the corresponding calibration point. The grid AiAj corresponding to the i-th row and j-th column represents the error between the candidate calibration image Ai corresponding to the calibration point Ai and the candidate calibration image Aj corresponding to the calibration point Aj. For example, the grid A1A2 corresponding to the first row and second column represents the error between the candidate calibration image A1 corresponding to the calibration point A1 and the candidate calibration image A2 corresponding to the calibration point A2. If i=j, the error is zero. For example, the error of A1A1 is 0. In practical applications, it can be directly eliminated or included in the calculation (since the error is 0, it does not affect the final result). The candidate calibration images A1~A9 are analyzed one by one as the above-mentioned target calibration images. Figure 3 In the example of analyzing the candidate calibration image A2 as the target calibration image, the errors corresponding to all the squares in the second row and the second column can be averaged to obtain Err_this (corresponding to the aforementioned first mean). Then the second row and the second column are deleted, and the errors corresponding to the remaining squares are averaged to obtain Err_other (corresponding to the aforementioned second mean). If the difference between Err_this and Err_other is greater than the preset threshold, it means that the candidate calibration image A2 is an invalid calibration image. In other words, if the difference between Err_this and Err_other is not greater than the preset threshold, it means that the candidate calibration image A2 is a valid calibration image. It should be noted that Figure 3 A simple example is only given for ease of understanding. The errors between AiAj and AjAi are the same. For example, the errors between A1A2 and A2A1 are the same. In practical applications, it only needs to be calculated once. That is, for A2, the errors between A2 and A1 and A3 to A9 are averaged to obtain a first mean, and the errors between A1 and A3 to A9 (that is, the candidate calibration images remaining after A2 is eliminated) are averaged to obtain a second mean.

[0046] Combination Figure 3 Taking the target calibration image as the candidate calibration image A2 for analysis as an example, the following four cases are given to fully illustrate the reliability of the above-mentioned method for determining the effective calibration image provided by the embodiment of the present disclosure:

[0047] Case 1: Among multiple candidate calibration images A1~A9, only one candidate calibration image is invalid (assuming it is A2). When the above operation is performed using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and other candidate calibration images are basically large, that is, the first mean is usually large, while other candidate calibration images (A1, A3~A9) are normal, and the corresponding errors are basically small, that is, the second mean is usually small. Therefore, the difference between the first mean and the second mean will be large, so it can be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0048] Case 2: Among multiple candidate calibration images A1 to A9, only one candidate calibration image is invalid (assuming it is A3). When the above operation is performed on candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1 and A4 to A9 are small, and only the error between candidate calibration image A2 and A3 is large. The first mean value will be affected by the errors of A2 and A3. However, the errors between A3 and A1 and A4 to A9 are also large, and the second mean value is affected by the errors between A3 and A1 and A4 to A9. The difference between the first mean value and the second mean value can offset the influence of A3 to a certain extent. Therefore, the difference between the first mean value and the second mean value will not be large, and the current candidate calibration image A2 can be accurately determined to be a valid calibration image.

[0049] Case 3: At least two of the multiple candidate calibration images A1 to A9 are invalid (assuming they are A2 and A3). When the above operation is performed using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and other candidate calibration images (A1, A4 to A9) are generally large, which will make the first mean value larger. Since candidate calibration image A3 is also invalid, the annotated theoretical sight direction is not accurate, so the error between A2A3 will also affect the first mean value. However, when calculating the second mean of the errors between candidate calibration images A1, A3 to A9, the errors between A3 and A1, A4 to A9 are all large, which will also affect the second mean value to a certain extent. When the first mean is subtracted from the second mean, the error between A2A3 can be regarded as offsetting the error between A3 and A1, A4~A9 to a certain extent, and the remaining error is mainly caused by the invalidity of A2 itself. Therefore, the difference between the first mean and the second mean is still large. Therefore, it can still be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0050] Case 4: At least two of the multiple candidate calibration images A1 to A9 are invalid (assuming they are A3 and A4). When the above operation is performed on candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1, A5 to A9 are small, and only the errors between candidate calibration image A2 and A3 and A4 are large. The first mean value will be affected by the errors of A2A3 and A2A4. However, the errors between A3 and A1, A4 to A9 are also large, and the errors between A4 and A1, A3, A5 to A9 are also large. The second mean value is affected by the above errors. The difference between the first mean value and the second mean value can offset the error influence of A3 and A4 to a certain extent. Therefore, the difference between the first mean value and the second mean value will not increase, and it can still be accurately determined that the current candidate calibration image A2 is a valid calibration image.

[0051] From the above analysis, it can be seen that the method for determining the valid calibration image provided by the embodiment of the present disclosure can reasonably and reliably find the valid calibration image from multiple candidate calibration images. Compared with the method adopted in the related art, it can effectively avoid the invalid calibration image interfering with the line of sight direction estimation results of the remaining calibration images. When there are multiple invalid calibration images at the same time, it can also avoid the problem of inaccurate difference between the line of sight direction estimation result and the theoretical line of sight direction caused by the inability to effectively calculate the line of sight direction estimation results corresponding to other calibration images in the related art.

[0052] In some embodiments, the method further includes: initiating a prompt message for re-collecting the calibration image when none of the multiple candidate calibration images is a valid calibration image. In practical applications, if there is a valid calibration image among the multiple candidate calibration images, the valid calibration image can be directly used to perform subsequent operations, such as using the valid calibration image to estimate the user's line of sight. If the multiple candidate calibration images are invalid, it is necessary to initiate a prompt message for re-collecting the calibration image to the user in a specified form (such as a voice prompt, graphic prompt, etc.). After obtaining the user's authorization, the valid calibration image can be re-collected, thereby ensuring the reliability of the subsequent line of sight estimation.

[0053] Based on the above, the embodiment of the present disclosure further provides a line of sight estimation method. Figure 4 The flowchart of a line of sight estimation method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a line of sight estimation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 4 As shown, the method mainly includes the following steps S402 to S408:

[0054] Step S402, multiple candidate calibration images are collected, and the aforementioned calibration image screening method is used to screen effective calibration images from the multiple candidate calibration images. For details, please refer to the aforementioned related content, which will not be repeated here.

[0055] Step S404 , in response to the screening result that the number of valid calibration images is not zero, the acquisition of candidate calibration images is terminated.

[0056] In addition, in response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images may continue to be acquired until a valid calibration image is screened out from the candidate calibration images that continue to be acquired.

[0057] In practical applications, multiple candidate calibration images can be collected each time, and the above-mentioned screening method can be used to screen valid calibration images for the multiple collected candidate calibration images. If the number of screened valid calibration images is not zero, the collection of candidate calibration images can be stopped. If the number of valid calibration images is zero, it is necessary to continue to collect multiple candidate calibration images until a valid calibration image is screened and the collection operation is stopped. The number of candidate calibration images collected at different times can be the same or different, and there is no limitation here. Through the above method, it can be fully guaranteed that a valid calibration image can be obtained in the end.

[0058] Step S406, in response to the line of sight estimation request, collecting the target image to be estimated, and obtaining the effective calibration image, and obtaining the line of sight difference estimation value between the effective calibration image and the target image through the pre-trained network model.

[0059] The line of sight estimation request can be directly issued by the user, or triggered by the user when performing a target operation, or set by the background, and is not limited here. In addition, before collecting the target image to be estimated, a prompt message for collecting the target image can be sent to the user to obtain user authorization. The network model can be a neural network model. In practical applications, the network model can also be called a calibration model. The embodiment of the present disclosure does not limit the structure of the network model, and any network model that can estimate the line of sight difference between two eye images can be used. Exemplarily, the network model can be a differential model.

[0060] Step S408, estimating the sight direction corresponding to the target image according to the theoretical sight direction of the effective calibration image and the sight difference estimation value.

[0061] Specifically, the theoretical sight line direction of the effective calibration image and the sight line difference estimation value can be added together to obtain the sight line direction corresponding to the target image. Since the theoretical sight line direction of the effective calibration image is accurate and the trained network model can output accurate sight line difference estimation values, the sight line direction corresponding to the target image obtained in the end is also accurate and reliable.

[0062] In the case where there are multiple valid calibration images, multiple estimated sight line directions corresponding to the target image can be obtained based on the theoretical sight line directions annotated by the multiple valid calibration images and the estimated sight line differences between the multiple valid calibration images and the target image; then the multiple sight line direction estimated values ​​are weighted, and the sight line direction corresponding to the target image is estimated based on the weighted processing result. The sight line direction obtained by synthesizing in the above manner can effectively ensure the accuracy of the sight line direction corresponding to the target image obtained in the end.

[0063] Since the calibration image screening method provided in the embodiment of the present disclosure can obtain accurate and reliable effective calibration images, the theoretical line of sight direction annotated on the effective calibration image matches the actual line of sight direction. On this basis, the accuracy of line of sight direction estimation based on the theoretical line of sight direction of the effective calibration image and the estimated line of sight difference between the effective calibration image and the target image is better guaranteed.

[0064] For example, reference may be made to Figure 5 The schematic diagram of a line of sight estimation process shown in the figure illustrates that a valid calibration image with annotated theoretical line of sight direction and a target image are simultaneously input into a pre-trained network model, and the network model can output a line of sight difference estimation value, and the line of sight direction corresponding to the target image can be obtained by adding the line of sight difference estimation value and the theoretical line of sight direction annotated by the valid calibration image. Furthermore, the network model can be a difference model, Figure 5 It is also illustrated that the network model includes a feature extraction network and a sight difference prediction network. The feature extraction network is used to extract the feature vectors of the effective calibration image and the target image respectively. The sight difference prediction network can analyze and process the feature vectors of the effective calibration image and the target image respectively, thereby generating a sight difference estimation value. In specific implementation, the feature vectors of the effective calibration image and the target image can be spliced ​​to obtain spliced ​​features, and then the sight difference prediction network is used to analyze and process the spliced ​​features to obtain a sight difference estimation value. The above is only an implementation example of the network model and should not be regarded as a limitation.

[0065] In practical applications, post-processing such as smoothing can be performed based on the output results of the network model to further optimize the estimation results of the line of sight direction, which is not limited here.

[0066] In summary, through the above method provided by the embodiment of the present disclosure, a more accurate and reliable line of sight direction estimation result can be obtained.

[0067] Corresponding to the screening method of the aforementioned calibration image, Figure 6 FIG. 1 is a schematic diagram of a calibration image screening device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6 As shown, the screening device for the calibration image includes:

[0068] The calibration image acquisition module 602 is used to acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point;

[0069] The sight difference value acquisition module 604 is used to obtain the sight difference estimation value and sight difference annotation value between the plurality of candidate calibration images; wherein the sight difference estimation value is estimated by using the pre-trained network model, and the sight difference annotation value is obtained based on the theoretical sight direction of the candidate calibration image;

[0070] The calibration image screening module 606 is used to determine a valid calibration image from multiple candidate calibration images based on the estimated sight line difference values ​​and sight line difference annotation values ​​between the multiple candidate calibration images; wherein the estimated sight line direction of the valid calibration image and the theoretical sight line direction annotated by the valid calibration image meet a preset matching condition.

[0071] The above device can analyze the estimated values ​​and labeled values ​​of the line of sight differences between different candidate calibration images, which helps to reasonably and reliably find a valid calibration image, and the theoretical line of sight direction annotated in the final valid calibration image matches the estimated line of sight direction that can be used to characterize the actual situation.

[0072] In some embodiments, the calibration image screening module 606 is specifically used to: obtain the error between the multiple candidate calibration images based on the line of sight difference estimation values ​​and the line of sight difference annotation values ​​between the multiple candidate calibration images; wherein the error is determined based on the difference between the line of sight difference estimation values ​​and the line of sight difference annotation values; and determine a valid calibration image from the multiple candidate calibration images based on the error between the multiple candidate calibration images.

[0073] In some embodiments, the number of the candidate calibration images is not less than three; the calibration image screening module 606 is specifically used to: average the errors between the target calibration image and the remaining candidate calibration images among the multiple candidate calibration images to obtain a first mean; wherein each candidate calibration image among the multiple candidate calibration images is sequentially used as a target calibration image; average the errors between each pair of the candidate calibration images other than the target calibration image among the multiple candidate calibration images to obtain a second mean; and determine whether the target calibration image is a valid calibration image based on the first mean and the second mean.

[0074] In some implementations, the calibration image screening module 606 is specifically configured to: determine whether a difference between the first mean and the second mean is not greater than a preset threshold; if so, determine that the target calibration image is a valid calibration image.

[0075] In some embodiments, the device further comprises an information initiating module, configured to initiate a prompt message for re-capturing the calibration image when none of the plurality of candidate calibration images are valid calibration images.

[0076] The calibration image screening device provided in the embodiments of the present disclosure can execute the calibration image screening method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0078] Corresponding to the aforementioned line of sight estimation method, Figure 7 FIG. 1 is a schematic diagram of a structure of a sight line estimation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 7 As shown, the sight line estimation device includes:

[0079] An image acquisition module 702 is used to acquire a plurality of candidate calibration images, and select a valid calibration image from the plurality of candidate calibration images using the calibration image screening method;

[0080] An end acquisition module 704 is used to end acquisition of candidate calibration images in response to the screening result that the number of valid calibration images is not zero;

[0081] The sight difference estimation module 706 is used to collect the target image to be estimated and obtain the effective calibration image in response to the sight estimation request, and obtain the sight difference estimation value between the effective calibration image and the target image through the pre-trained network model;

[0082] The sight line direction estimation module 708 is used to estimate the sight line direction corresponding to the target image according to the theoretical sight line direction of the effective calibration image and the sight line difference estimation value.

[0083] Since the calibration image screening method provided in the embodiment of the present disclosure can obtain accurate and reliable effective calibration images, the theoretical line of sight direction annotated on the effective calibration image matches the actual line of sight direction. On this basis, the accuracy of line of sight direction estimation based on the theoretical line of sight direction of the effective calibration image and the estimated line of sight difference between the effective calibration image and the target image is better guaranteed.

[0084] In some embodiments, the apparatus further comprises: in response to a screening result that the number of valid calibration images is zero, continuing to acquire multiple candidate calibration images until a valid calibration image is screened out from the candidate calibration images that continue to be acquired.

[0085] The gaze direction estimation device provided in the embodiments of the present disclosure can execute the gaze direction estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0087] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing instructions executable by the processor; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned calibration image screening method or the above-mentioned line of sight estimation method.

[0088] Figure 8 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the electronic device 800 includes one or more processors 801 and a memory 802 .

[0089] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0090] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may run the program instructions to implement the calibration image screening method, the line of sight estimation method, and / or other desired functions of the embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0091] In one example, the electronic device 800 may further include: an input device 803 and an output device 804, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0092] In addition, the input device 803 may also include, for example, a keyboard, a mouse, etc.

[0093] The output device 804 can output various information to the outside, including the determined distance information, direction information, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0094] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device 800 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 800 may also include any other appropriate components.

[0095] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the calibration image screening method or line of sight estimation method provided by the embodiments of the present disclosure.

[0096] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0097] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the calibration image screening method or line of sight estimation method provided by the embodiments of the present disclosure.

[0098] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0099] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the calibration image screening method or the line of sight estimation method in the embodiments of the present disclosure.

[0100] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0101] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. For example, when collecting an eye image of the user, a prompt message for collecting the eye image is sent to the user, and the user can be further informed of the purpose of the image collection in the prompt message. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium or other software or hardware that performs the operation of the technical solution of the present disclosure according to the prompt message, and perform the operation of the technical solution of the present disclosure after obtaining the user's authorization.

[0102] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0103] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0104] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0105] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for screening calibration images, characterized in that: include: Acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the candidate calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point; Obtaining line-of-sight difference estimation values ​​and line-of-sight difference annotation values ​​between each of the plurality of candidate calibration images; wherein the line-of-sight difference estimation values ​​are estimated using a pre-trained network model, and the line-of-sight difference annotation values ​​are obtained based on theoretical line-of-sight directions of the candidate calibration images; According to the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

2. The method according to claim 1, characterized in that The determining a valid calibration image from the plurality of candidate calibration images according to the sight difference estimation values ​​and sight difference annotation values ​​between the plurality of candidate calibration images comprises: Obtaining errors between the plurality of candidate calibration images according to the estimated sight difference values ​​and the marked sight difference values ​​between the plurality of candidate calibration images; wherein the errors are determined based on the difference between the estimated sight difference values ​​and the marked sight difference values; Based on the errors between the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images.

3. The method according to claim 2, characterized in that The number of the candidate calibration images is no less than three; and determining a valid calibration image from the plurality of candidate calibration images based on errors between each of the plurality of candidate calibration images comprises: Averaging the errors between the target calibration image and the remaining candidate calibration images in the plurality of candidate calibration images to obtain a first mean value; wherein each candidate calibration image in the plurality of candidate calibration images is sequentially used as the target calibration image; Averaging the errors between each of the candidate calibration images except the target calibration image among the multiple candidate calibration images to obtain a second mean value; Whether the target calibration image is a valid calibration image is determined according to the first mean value and the second mean value.

4. The method according to claim 3, characterized in that The determining, based on the first mean value and the second mean value, whether the target calibration image is a valid calibration image includes: Determining whether a difference between the first mean and the second mean is not greater than a preset threshold; If so, it is determined that the target calibration image is a valid calibration image.

5. The method according to claim 1, characterized in that The method further comprises: In the case that none of the multiple candidate calibration images is a valid calibration image, a prompt message for re-capturing the calibration image is initiated.

6. A line of sight estimation method, characterized in that: include: Acquire a plurality of candidate calibration images, and use the calibration image screening method according to any one of claims 1 to 5 to screen a valid calibration image from the plurality of candidate calibration images; In response to a screening result that the number of valid calibration images is not zero, ending the acquisition of candidate calibration images; In response to a line of sight estimation request, a target image to be estimated is acquired, and the effective calibration image is obtained, and a line of sight difference estimation value between the effective calibration image and the target image is obtained through a pre-trained network model; The line of sight direction corresponding to the target image is estimated according to the theoretical line of sight direction of the effective calibration image and the line of sight difference estimation value.

7. The method according to claim 6, characterized in that The method further comprises: In response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images continue to be acquired until a valid calibration image is screened out from the candidate calibration images that continue to be acquired.

8. A calibration image screening device, characterized in that: include: A calibration image acquisition module, used to acquire multiple candidate calibration images; wherein the candidate calibration images are eye images taken when the target eye is required to look at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are annotated with theoretical sight lines when the target eye looks at the specified calibration point; A sight difference value acquisition module, used to acquire sight difference estimation values ​​and sight difference annotation values ​​between the plurality of candidate calibration images; wherein the sight difference estimation values ​​are estimated by using a pre-trained network model, and the sight difference annotation values ​​are obtained based on theoretical sight directions of the candidate calibration images; A calibration image screening module is used to determine a valid calibration image from the multiple candidate calibration images based on the line of sight difference estimation values ​​and line of sight difference annotation values ​​between the multiple candidate calibration images; wherein the estimated line of sight direction of the valid calibration image and the theoretical line of sight direction annotated by the valid calibration image meet a preset matching condition.

9. A sight line estimation device, characterized in that: include: An image acquisition module, used for acquiring a plurality of candidate calibration images, and using the calibration image screening method according to any one of claims 1 to 5 to screen a valid calibration image from the plurality of candidate calibration images; an end acquisition module, configured to end acquisition of candidate calibration images in response to a screening result that the number of valid calibration images is not zero; A sight difference estimation module, configured to, in response to a sight estimation request, acquire a target image to be estimated, obtain the effective calibration image, and obtain a sight difference estimation value between the effective calibration image and the target image through a pre-trained network model; The sight line direction estimation module is used to estimate the sight line direction corresponding to the target image according to the theoretical sight line direction of the effective calibration image and the sight line difference estimation value.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

12. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the calibration image screening method described in any one of claims 1 to 5 or the line of sight estimation method described in any one of claims 6 to 7.

Images