Screening method, line-of-sight estimation method, device and equipment for calibration image

Abstract

The embodiment of the present disclosure relates to a kind of calibration image screening method, line-of-sight estimation method, device, equipment and medium, and the screening method of calibration image includes: obtaining multiple candidate calibration images;Candidate calibration image is marked with the theoretical line-of-sight direction of target eye when gazing specified calibration point;The line-of-sight difference estimation value and line-of-sight difference label value between the two of multiple candidate calibration images are obtained;Line-of-sight difference estimation value is estimated using a pre-trained network model, and line-of-sight difference label value is obtained based on the theoretical line-of-sight direction of candidate calibration image;According to the line-of-sight difference estimation value and line-of-sight difference label value between the two of multiple candidate calibration images, determine effective calibration image from multiple candidate calibration images.The embodiment of the present disclosure can effectively guarantee the screening reliability of effective calibration image, further guarantee the accuracy of line-of-sight direction estimation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for screening calibration images, a method for estimating line of sight, an apparatus, a device, and a medium. Background Technology

[0002] In many fields such as gaming, healthcare, and intelligent control, it is necessary to identify the direction of human eye gaze in order to implement appropriate strategies based on the estimated gaze direction. However, the accuracy of gaze estimation results obtained through related technologies is poor, which can negatively impact the application scenarios. Therefore, there is an urgent need to improve the accuracy of gaze estimation results. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method for screening calibration images, a method for estimating line of sight, an apparatus, a device, and a medium.

[0004] This disclosure provides a method for screening calibration images, including: acquiring multiple candidate calibration images; wherein the candidate calibration images are eye images captured when a target eye is required to gaze at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are labeled with the theoretical gaze direction when the target eye gazes at the specified calibration point; acquiring estimated gaze difference values ​​and labeled gaze difference values ​​between each pair of the multiple candidate calibration images; wherein the estimated gaze difference values ​​are obtained using a pre-trained network model, and the labeled gaze difference values ​​are obtained based on the theoretical gaze direction of the candidate calibration images; determining a valid calibration image from the multiple candidate calibration images based on the estimated gaze difference values ​​and labeled gaze difference values ​​between each pair of the multiple candidate calibration images; wherein the estimated gaze direction of the valid calibration image and the labeled theoretical gaze direction of the valid calibration image satisfy a preset matching condition.

[0005] Secondly, embodiments of this disclosure provide a gaze estimation method, comprising: acquiring multiple candidate calibration images; using the calibration image filtering method provided in the first aspect to filter valid calibration images from the multiple candidate calibration images; terminating the acquisition of candidate calibration images in response to a filtering result where the number of valid calibration images is not zero; terminating the acquisition of candidate calibration images in response to a filtering result where the number of valid calibration images is not zero; obtaining a gaze difference estimate between the valid calibration image and the target image through a pre-trained network model; and estimating the gaze direction corresponding to the target image based on the theoretical gaze direction of the valid calibration image and the gaze difference estimate.

[0006] Thirdly, embodiments of this disclosure provide a calibration image screening device, comprising: a calibration image acquisition module for acquiring multiple candidate calibration images; wherein the candidate calibration images are eye images captured when a target eye is required to gaze at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are labeled with the theoretical gaze direction when the target eye gazes at the specified calibration point; a gaze difference acquisition module for acquiring estimated gaze difference values ​​and labeled gaze difference values ​​between each pair of the multiple candidate calibration images; wherein the estimated gaze difference values ​​are obtained by estimating using a pre-trained network model, and the labeled gaze difference values ​​are obtained based on the theoretical gaze direction of the candidate calibration images; and a calibration image screening module for determining a valid calibration image from the multiple candidate calibration images based on the estimated gaze difference values ​​and labeled gaze difference values ​​between each pair of the multiple candidate calibration images; wherein the estimated gaze direction of the valid calibration image and the labeled theoretical gaze direction of the valid calibration image satisfy a preset matching condition.

[0007] Fourthly, embodiments of this disclosure provide a gaze estimation device, comprising: an image acquisition module for acquiring multiple candidate calibration images and selecting valid calibration images from the multiple candidate calibration images using the calibration image filtering method provided in the first aspect; an acquisition termination module for terminating the acquisition of candidate calibration images in response to a filtering result where the number of valid calibration images is not zero; a gaze difference estimation module for acquiring a target image to be estimated in response to a gaze estimation request, obtaining the valid calibration images, and obtaining a gaze difference estimate between the valid calibration images and the target image through a pre-trained network model; and a gaze direction estimation module for estimating the gaze direction corresponding to the target image based on the theoretical gaze direction of the valid calibration images and the gaze difference estimate.

[0008] Fifthly, embodiments of this disclosure provide an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement a calibration image screening method of the first aspect, or a gaze estimation method of the second aspect.

[0009] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium storing a computer program for executing a calibration image screening method of the first aspect, or implementing a gaze estimation method of the second aspect.

[0010] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements a method for screening calibration images in the first aspect, or a method for estimating gaze in the second aspect.

[0011] The technical solution provided in this disclosure can determine the effective calibration image from multiple candidate calibration images based on the estimated line-of-sight difference (obtained based on a network model) and the labeled line-of-sight difference (obtained based on the theoretical line-of-sight direction labeled on the candidate calibration images). This method analyzes the estimated and labeled line-of-sight differences between different candidate calibration images, which helps to find the effective calibration image reasonably and reliably. The theoretical line-of-sight direction labeled on the final effective calibration image meets the preset matching conditions with the estimated line-of-sight direction. Based on this, the accuracy of line-of-sight direction estimation based on the labeled line-of-sight direction of the effective calibration image and the estimated line-of-sight difference between the effective calibration image and the target image is well guaranteed.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram of a calibration panel provided in an embodiment of this disclosure;

[0016] Figure 2 A schematic flowchart illustrating a method for screening calibration images provided in an embodiment of this disclosure;

[0017] Figure 3 A schematic diagram of an error matrix provided in an embodiment of this disclosure;

[0018] Figure 4 A schematic flowchart illustrating a line-of-sight estimation method provided in an embodiment of this disclosure;

[0019] Figure 5 A schematic diagram of a line-of-sight estimation process provided in an embodiment of this disclosure;

[0020] Figure 6 A schematic diagram of the structure of a calibration image screening device provided in an embodiment of this disclosure;

[0021] Figure 7 This is a schematic diagram of the structure of a line-of-sight estimation device provided in an embodiment of the present disclosure;

[0022] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0024] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0025] In related technologies, to accurately track eye movements based on the user's eye activity, it's necessary to pre-guide the user to gaze at designated calibration points. This process captures images of the user's eyes as calibration images, and the theoretical gaze direction (i.e., the gaze direction the user should have when gazing at the designated calibration points) is labeled. This allows for subsequent gaze estimation and other operations based on the acquired calibration images. Specifically, users can wear devices such as VR glasses, smart glasses, or eye trackers, and gaze at the calibration points provided by these devices. The cameras on these devices then capture images of the user's eyes. Typically, multiple calibration points are provided to the user; for example, refer to... Figure 1 The diagram shows a calibration panel with nine calibration points displayed simultaneously. It should be noted that these nine points are not shown to the user at the same time; only one point is shown at a time. After the user focuses on that point and a corresponding calibration image is captured, the next calibration point is shown, and so on, allowing multiple calibration images to be captured. Figure 1 The number on each ring represents the offset angle of the calibration point on that ring in a specified direction. Figure 1 This is just one example; in practical applications, calibration points can be displayed in other ways, and there are no restrictions here.

[0026] However, the inventors discovered that some users might not be looking at the designated calibration points as required. In this case, the theoretical viewing direction marked on the calibration image corresponding to that calibration point is incorrect, and the actual viewing direction of the captured calibration image does not match the marked theoretical viewing direction. This means the calibration image is invalid, leading to inaccurate results from subsequent processing using that image. While some related technologies consider this problem and filter the acquired calibration images, the filtering method is unreliable. Taking calibration points 1-9 as an example, related technologies select one of the nine calibration points (assuming calibration point 1) as the test point. Based on the calibration images corresponding to the other eight calibration points, a network model is used to estimate the viewing direction of the calibration image corresponding to calibration point 1. Specifically, the network model can be used to estimate the viewing difference between the calibration image of calibration point 1 and the calibration images of other calibration points. Based on this, combined with the theoretical viewing directions of other calibration points, the estimated viewing direction of calibration point 1 can be obtained. Furthermore, the difference between the estimated viewing direction of calibration point 1 and its theoretical viewing direction can be obtained. The remaining calibration points are processed in the same way, with each calibration point used sequentially as a test point for line-of-sight direction estimation. This yields the difference between the estimated line-of-sight direction for each of the nine calibration points and the theoretical line-of-sight direction. Calibration images with differences exceeding a preset threshold are then considered invalid calibration points; in other words, the calibration images corresponding to invalid calibration points are ineffective. However, the inventors found that the main problem with this method is that estimating the line-of-sight direction for a test point based on the calibration image corresponding to an invalid calibration point leads to a large difference for that test point. In other words, invalid calibration points interfere with the line-of-sight direction estimation results of other calibration points. If multiple invalid calibration points exist simultaneously, it becomes even more difficult to effectively calculate the line-of-sight direction estimation results for other points, resulting in inaccurate differences between the estimated line-of-sight direction and the theoretical line-of-sight direction. This also makes it difficult to reasonably determine the threshold, and the comparison between the threshold and the difference makes it difficult to reliably select valid calibration images for subsequent applications.

[0027] In summary, the screening methods used in related technologies are unreliable. To improve this problem, this disclosure provides a method for screening calibration images, a line-of-sight estimation method, an apparatus, a device, and a medium, which will be described in detail below.

[0028] Figure 2 This is a flowchart illustrating a method for screening calibration images according to an embodiment of the present disclosure. This method can be executed by a device for screening valid calibration images, wherein the device can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 2 As shown, the method mainly includes the following steps S202 to S206:

[0029] Step S202: Acquire multiple candidate calibration images; wherein, the candidate calibration images are eye images taken when the target eye is required to gaze at a specified calibration point, different calibration images correspond to different specified calibration points, and the candidate calibration images are marked with the theoretical line of sight when the target eye gazes at the specified calibration point. Any method of acquiring candidate calibration images is acceptable and is not limited here.

[0030] Step S204: Obtain the estimated line-of-sight difference and the labeled line-of-sight difference between each pair of multiple candidate calibration images; wherein, the estimated line-of-sight difference is obtained by estimating using a pre-trained network model, and the labeled line-of-sight difference is obtained based on the theoretical line-of-sight direction of the candidate calibration images.

[0031] The network model can be a neural network model, and in practical applications, it can also be called a calibration model. This disclosure does not limit the structure of the network model; any network model capable of estimating the gaze difference between two eye images is acceptable. For example, the network model can be a difference model.

[0032] Multiple candidate calibration images are paired to obtain multiple image groups. These image groups are then input into a network model to obtain the gaze difference estimate for each image group, which is the pairwise gaze difference estimate between the candidate calibration images. Since the gaze difference estimate is obtained using a pre-trained network model, it is generally quite accurate and can well reflect the true gaze difference. Furthermore, based on the difference between the theoretical gaze directions of two candidate calibration images in each image group, the gaze difference annotation value for each image group can be obtained, which is the pairwise gaze difference annotation value between the candidate calibration images. However, since the theoretical gaze directions of the candidate calibration images may be incorrect, the gaze difference annotation value may also be inaccurate.

[0033] Step S206: Based on the estimated and labeled line-of-sight differences between each pair of candidate calibration images, a valid calibration image is determined from the candidate calibration images. The estimated line-of-sight direction of the valid calibration image and the labeled theoretical line-of-sight direction of the valid calibration image must satisfy a preset matching condition. For example, the direction difference between the estimated and labeled theoretical line-of-sight directions of the valid calibration image is no greater than a preset threshold. This preset threshold can be flexibly set according to requirements and can be determined based on empirical values. That is, for a candidate calibration image, it can only be considered a valid calibration image if the difference between its estimated and theoretical line-of-sight directions is small (i.e., no greater than the preset threshold). The estimated line-of-sight direction of the candidate calibration image is relatively accurate and can be predicted by a pre-trained network model or obtained through a line-of-sight direction estimation tool. Any method that can obtain a relatively accurate actual line-of-sight direction is acceptable and is not limited here. The estimated line-of-sight direction of the candidate calibration image can be used to characterize the actual line-of-sight direction. If the difference between the theoretical and estimated line-of-sight directions is small, it indicates that the labeled theoretical line-of-sight direction is also relatively accurate and reliable.

[0034] In this embodiment, the network model is pre-trained and can output relatively accurate results. Therefore, the estimated gaze difference can be used to characterize the relatively true gaze difference between two candidate calibration images. However, considering that the user may not be looking at the specified calibration point as required, the theoretical gaze direction labeled in the candidate calibration image may not match the actual gaze direction of the eyes in the candidate calibration image. In other words, the theoretical gaze direction corresponding to the candidate calibration image may not be accurate. Therefore, the gaze difference label value between two candidate calibration images obtained based on the theoretical gaze direction of the candidate calibration image may not be accurate. Based on this, the above method can analyze the estimated (relatively accurate, also known as the actual value) and labeled value of the gaze difference between different candidate calibration images, which helps to find a reasonable and reliable effective calibration image. Finally, the theoretical gaze direction labeled in the effective calibration image matches the actual gaze direction.

[0035] In some implementations, step S206 above, that is, determining the valid calibration image from multiple candidate calibration images based on the estimated and labeled line-of-sight differences between each pair of candidate calibration images, can be performed with reference to steps A and B below:

[0036] Step A: Based on the estimated line-of-sight difference and labeled line-of-sight difference values ​​between each pair of candidate calibration images, the pairwise error between the candidate calibration images is obtained. The error is determined based on the difference between the estimated line-of-sight difference and the labeled line-of-sight difference. For example, a specified loss function can be used to determine the error based on the difference between the estimated line-of-sight difference and the labeled line-of-sight difference (i.e., the loss value obtained based on the loss function). This embodiment of the disclosure does not limit the loss function. Taking candidate calibration image A1 and candidate calibration image A2 as an example, the error err_12 between candidate calibration image A1 and candidate calibration image A2 is: err_12 = Loss(F(A1,A2),label1-label2). Where F(A1,A2) represents the estimated line-of-sight difference obtained after inputting candidate calibration images A1 and A2 into the network model, label1 and label2 represent the theoretical line-of-sight directions corresponding to candidate calibration images A1 and A2, respectively, label1-label2 represent the line-of-sight difference label values ​​between candidate calibration images A1 and A2, and Loss represents the loss value calculated using a preset loss function based on the difference between the estimated line-of-sight difference and the labeled line-of-sight difference.

[0037] Step B involves determining the valid calibration image from the multiple candidate calibration images based on the pairwise errors between them. In some implementations, the valid calibration image can be determined directly from the multiple candidate calibration images based on the error magnitude. In other implementations, invalid calibration images can be determined first, followed by the determination of valid calibration images. Specifically, considering that if an invalid calibration image exists, the line-of-sight difference labeling value between it and other candidate calibration images will be generally inaccurate, and the corresponding error will be relatively significant, invalid calibration images can be determined first, and the images other than the invalid calibration images from the multiple candidate calibration images can be used as valid calibration images.

[0038] To further ensure the accurate identification of valid calibration images, in some specific implementation examples, the number of candidate calibration images is no less than three; based on this, step B can be performed as follows: steps B1 to B3:

[0039] Step B1 involves averaging the errors between the target calibration image and the remaining candidate calibration images from multiple candidate calibration images to obtain a first mean. Each candidate calibration image is sequentially used as the target calibration image; that is, steps B1 to B3 are performed for each candidate calibration image to determine whether each candidate calibration image is a valid calibration image. The "sequentially" order can be arbitrary and can be flexibly set. In other words, each candidate calibration image can be used as a target calibration image, and steps B1 to B3 are used to determine whether it is a valid calibration image. The order in which each candidate calibration image is used as a target calibration image is not restricted.

[0040] Step B2: Average the errors between each pair of candidate calibration images (excluding the target calibration image) among multiple candidate calibration images to obtain a second mean.

[0041] Step B3: Determine whether the target calibration image is a valid calibration image based on the first mean and the second mean. In practical applications, the difference between the first mean and the second mean can be used to determine whether the target calibration image is a valid calibration image. For example, step B3 can be performed with reference to steps B3.1 and B3.2 below:

[0042] Step B3.1: Determine whether the difference between the first mean and the second mean is not greater than a preset threshold. The preset threshold can be flexibly set based on experience and is not restricted here.

[0043] Step B3.2, if yes, determine that the target calibration image is a valid calibration image.

[0044] Understandably, invalid calibration images generally have larger errors than other candidate calibration images, meaning their first mean is usually larger. This leads to a correspondingly larger difference between the first and second means, allowing them to be accurately and reliably identified as invalid calibration images. Conversely, valid calibration images usually have relatively smaller errors than other candidate calibration images, meaning their first mean is usually smaller, and the difference between the first and second means is also smaller, unlikely to exceed a preset threshold. This allows them to be identified as valid calibration images with relatively high accuracy and reliability. Furthermore, for ease of understanding, the following will combine... Figure 3 The diagram shown illustrates an error matrix, further illustrating steps B1 to B3 provided in the embodiments of this disclosure:

[0045] Figure 3A 9x9 matrix is ​​illustrated, where each row and column represents a corresponding calibration point. The square AiAj in the i-th row and j-th column represents the error between the candidate calibration image Ai corresponding to calibration point Ai and the candidate calibration image Aj corresponding to calibration point Aj. For example, the square A1A2 in the first row and second column represents the error between the candidate calibration image A1 corresponding to calibration point A1 and the candidate calibration image A2 corresponding to calibration point A2. If i = j, the error is zero, such as A1A1, which has an error of 0 and can be directly eliminated in practical applications, but can also be included in the calculation (since the error is 0, it does not affect the final result). Candidate calibration images A1 to A9 are analyzed one by one as the target calibration images mentioned above. Figure 3 Taking the analysis of candidate calibration image A2 as the target calibration image as an example, the average error of all squares in the second row and second column can be calculated to obtain Err_this (corresponding to the aforementioned first mean). Then, the second row and second column are deleted, and the average error of the remaining squares is calculated to obtain Err_other (corresponding to the aforementioned second mean). If the difference between Err_this and Err_other is greater than a preset threshold, it indicates that candidate calibration image A2 is an invalid calibration image. In other words, if the difference between Err_this and Err_other is not greater than the preset threshold, it indicates that candidate calibration image A2 is a valid calibration image. It should be noted that... Figure 3 This is just a simple example for ease of understanding. The errors between AiAj and AjAi are the same. Similarly, the errors between A1A2 and A2A1 are the same. In practical applications, only one calculation is needed. That is, for A2, the errors between A2 and A1, and between A3 to A9 are respectively obtained and averaged to obtain the first mean. The errors between A1 and A3 to A9 (that is, the candidate calibration images remaining after removing A2) are also obtained and averaged to obtain the second mean.

[0046] Combination Figure 3 Taking the analysis of the target calibration image as candidate calibration image A2 as an example, the following four cases fully illustrate the reliability of the above-mentioned method for determining the effective calibration image provided by the embodiments of this disclosure:

[0047] Scenario 1: Among multiple candidate calibration images A1 to A9, only one candidate calibration image is invalid (let's say A2). When performing the above operation with candidate calibration image A2 as the target calibration image, the error between candidate calibration image A2 and the other candidate calibration images is generally large, that is, the first mean is usually large. Meanwhile, the other candidate calibration images (A1, A3 to A9) are normal, and their corresponding errors are generally small, that is, the second mean is usually small. Therefore, the difference between the first mean and the second mean will be large, so it can be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0048] Scenario 2: Among multiple candidate calibration images A1 to A9, only one candidate calibration image is invalid (let's say A3). When performing the above operation using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1, A4 to A9 are all small, only the error between it and A3 is large. The first mean will be affected by the errors of A2 and A3. However, the errors between A3 and A1, A4 to A9 are also large, and the second mean is affected by the errors of A3 between A3 and A1, A4 to A9 respectively. The difference between the first mean and the second mean can, to some extent, offset the influence of A3. Therefore, the difference between the first mean and the second mean will not be large, and the current candidate calibration image A2 can be accurately determined as a valid calibration image.

[0049] Scenario 3: Among multiple candidate calibration images A1 to A9, at least two candidate calibration images are invalid (let's say A2 and A3). When performing the above operation using candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and the other candidate calibration images (A1, A4 to A9) are generally large, causing the first mean to be too high. Since candidate calibration image A3 is also invalid, the theoretical line-of-sight direction it labels is inaccurate, so the error between A2 and A3 will also affect the first mean. However, when calculating the second mean of the errors between each pair of candidate calibration images A1 and A3 to A9, the errors between A3 and A1 and A4 to A9 are all large, which will also affect the second mean to some extent. When the difference between the first mean and the second mean is calculated, the error between A2 and A3 can be considered to offset the errors between A3 and A1, A4 to A9 to a certain extent. The remaining error is mainly caused by the invalidity of A2 itself. Therefore, the difference between the first mean and the second mean is still relatively large. For this reason, it can still be accurately determined that the current candidate calibration image A2 is an invalid calibration image.

[0050] Scenario 4: Among multiple candidate calibration images A1 to A9, at least two candidate calibration images are invalid (let's say A3 and A4). When performing the above operation with candidate calibration image A2 as the target calibration image, the errors between candidate calibration image A2 and A1, A5 to A9 are relatively small, while the errors between it and A3 and A4 are relatively large. The first mean will be affected by the errors of A2A3 and A2A4. However, the errors between A3 and A1, A4 to A9 are also relatively large, and the errors between A4 and A1, A3, and A5 to A9 are also relatively large. The second mean is affected by these errors. However, the difference between the first mean and the second mean can, to some extent, offset the error effects of A3 and A4. Therefore, the difference between the first mean and the second mean will not increase, and it can still be accurately determined that the current candidate calibration image A2 is a calibrated image.

[0051] As can be seen from the above analysis, the method for determining the effective calibration image provided in this embodiment can reasonably and reliably find the effective calibration image from multiple candidate calibration images. Compared with the method used in related technologies, it can effectively avoid invalid calibration images interfering with the line-of-sight direction estimation results of other calibration images. Furthermore, when multiple invalid calibration images exist simultaneously, it also avoids the problem in related technologies where the line-of-sight direction estimation results cannot be effectively calculated, resulting in inaccurate differences between the line-of-sight direction estimation results and the theoretical line-of-sight direction.

[0052] In some implementations, the method further includes prompting the user to reacquire calibration images when none of the candidate calibration images are valid. In practical applications, if a valid calibration image exists among the candidate calibration images, subsequent operations can be performed directly using the valid calibration image, such as using the valid calibration image for user gaze estimation. If none of the candidate calibration images are valid, a prompt to reacquire calibration images needs to be sent to the user in a specified format (such as voice prompts, text prompts, etc.). After obtaining the user's authorization, a valid calibration image can be reacquired, thereby ensuring the reliability of subsequent gaze estimation.

[0053] Based on the foregoing, this disclosure also provides a line-of-sight estimation method. Figure 4 This is a flowchart illustrating a gaze estimation method provided in an embodiment of the present disclosure. The method can be executed by a gaze estimation device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 4 As shown, the method mainly includes the following steps S402 to S408:

[0054] Step S402: Acquire multiple candidate calibration images, and use the aforementioned calibration image screening method to select valid calibration images from the multiple candidate calibration images. For details, please refer to the aforementioned related content, which will not be repeated here.

[0055] Step S404: In response to the screening result that the number of valid calibration images is not zero, the acquisition of candidate calibration images ends.

[0056] In addition, in response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images can be acquired until a valid calibration image is selected from the continuously acquired candidate calibration images.

[0057] In practical applications, multiple candidate calibration images can be acquired each time, and the aforementioned filtering method can be used to select valid calibration images. If the number of selected valid calibration images is not zero, the acquisition of candidate calibration images can be stopped. If the number of valid calibration images is zero, multiple candidate calibration images need to be acquired again until a valid calibration image is selected, at which point the acquisition operation stops. The number of candidate calibration images acquired in different sessions can be the same or different; there is no restriction on this. This method ensures that a valid calibration image is ultimately obtained.

[0058] Step S406: In response to the gaze estimation request, acquire the target image to be estimated and obtain the effective calibration image. Obtain the gaze difference estimate between the effective calibration image and the target image through the pre-trained network model.

[0059] The gaze estimation request can be issued directly by the user, triggered by the user when performing a target operation, or configured in the backend; no restrictions are imposed here. Additionally, before acquiring the target image to be estimated, a prompt message can be sent to the user to obtain their authorization. The network model can be a neural network model; in practical applications, a network model can also be called a calibration model. This disclosure does not limit the structure of the network model; any network model capable of estimating the gaze difference between two eye images is acceptable. For example, the network model can be a difference model.

[0060] Step S408: Estimate the gaze direction corresponding to the target image based on the theoretical gaze direction of the effectively calibrated image and the gaze difference estimate.

[0061] Specifically, the theoretical gaze direction of the effectively calibrated image can be summed with the gaze difference estimate to obtain the gaze direction corresponding to the target image. Since the theoretical gaze direction of the effectively calibrated image is accurate, and the trained network model can output accurate gaze difference estimates, the gaze direction corresponding to the final target image is also accurate and reliable.

[0062] When there are multiple valid calibration images, multiple line-of-sight direction estimates corresponding to the target image can be obtained based on the theoretical line-of-sight directions annotated in the multiple valid calibration images and the estimated line-of-sight differences between the multiple valid calibration images and the target image. These multiple line-of-sight direction estimates are then weighted, and the line-of-sight direction corresponding to the target image is estimated based on the weighted result. By combining the line-of-sight directions obtained in this way, the accuracy of the line-of-sight direction corresponding to the final target image can be effectively guaranteed.

[0063] Since the calibration image screening method provided in this embodiment can obtain accurate and reliable effective calibration images, and the theoretical line-of-sight direction marked on the effective calibration image matches the actual line-of-sight direction, the accuracy of line-of-sight direction estimation based on the theoretical line-of-sight direction of the effective calibration image and the line-of-sight difference estimate between the effective calibration image and the target image is well guaranteed.

[0064] For example, one can refer to Figure 5 The diagram illustrates a gaze estimation process, showing how a valid calibration image labeled with the theoretical gaze direction and a target image are simultaneously input into a pre-trained network model. This network model outputs a gaze difference estimate. By summing this estimate with the theoretical gaze direction labeled on the valid calibration image, the gaze direction corresponding to the target image can be obtained. Furthermore, the network model can be a difference model. Figure 5 The diagram also illustrates that the network model includes a feature extraction network and a line-of-sight difference prediction network. The feature extraction network extracts feature vectors from both the valid calibration image and the target image. The line-of-sight difference prediction network analyzes and processes these feature vectors to generate a line-of-sight difference estimate. In practice, the feature vectors from the valid calibration image and the target image can be concatenated to obtain a concatenated feature vector. This concatenated feature vector is then analyzed and processed by the line-of-sight difference prediction network to obtain the line-of-sight difference estimate. This is merely one implementation example of the network model and should not be considered a limitation.

[0065] In practical applications, post-processing such as smoothing can be performed on the output of the network model to further optimize the estimation results of the line of sight direction, and there are no restrictions on this.

[0066] In summary, the above-described methods provided in the embodiments of this disclosure can yield more accurate and reliable line-of-sight direction estimation results.

[0067] Corresponding to the aforementioned method for selecting calibration images, Figure 6 This is a schematic diagram of a calibration image screening device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 6 As shown, the image calibration screening device includes:

[0068] The calibration image acquisition module 602 is used to acquire multiple candidate calibration images; wherein, the candidate calibration images are eye images taken when the target eye is required to gaze at a specified calibration point, the specified calibration point is different for different calibration images, and the calibration images are marked with the theoretical line of sight when the target eye gazes at the specified calibration point;

[0069] The line-of-sight difference acquisition module 604 is used to acquire the line-of-sight difference estimates and line-of-sight difference annotations between each pair of multiple candidate calibration images; wherein, the line-of-sight difference estimates are obtained by estimating using a pre-trained network model, and the line-of-sight difference annotations are obtained based on the theoretical line-of-sight directions of the candidate calibration images;

[0070] The calibration image filtering module 606 is used to determine the effective calibration image from multiple candidate calibration images based on the estimated line-of-sight difference and the labeled line-of-sight difference between each pair of multiple candidate calibration images; wherein the estimated line-of-sight direction of the effective calibration image and the theoretical line-of-sight direction labeled in the effective calibration image satisfy a preset matching condition.

[0071] The aforementioned device can analyze the estimated and labeled values ​​of the line-of-sight difference between different candidate calibration images, which helps to find effective calibration images reasonably and reliably. Finally, the theoretical line-of-sight direction labeled on the effective calibration image matches the estimated line-of-sight direction that can be used to characterize the actual situation.

[0072] In some embodiments, the calibration image screening module 606 is specifically used to: obtain the pairwise error between the multiple candidate calibration images based on the estimated line-of-sight difference and the labeled line-of-sight difference between each pair of the multiple candidate calibration images; wherein the error is determined based on the difference between the estimated line-of-sight difference and the labeled line-of-sight difference; and determine the valid calibration image from the multiple candidate calibration images based on the pairwise error between each pair of the multiple candidate calibration images.

[0073] In some embodiments, the number of candidate calibration images is no less than three; the calibration image screening module 606 is specifically used to: average the error between the target calibration image and the other candidate calibration images in the plurality of candidate calibration images to obtain a first mean; wherein, each candidate calibration image in the plurality of candidate calibration images is sequentially used as the target calibration image; average the error between each pair of candidate calibration images in the plurality of candidate calibration images other than the target calibration image to obtain a second mean; and determine whether the target calibration image is a valid calibration image based on the first mean and the second mean.

[0074] In some implementations, the calibration image filtering module 606 is specifically used to: determine whether the difference between the first mean and the second mean is not greater than a preset threshold; if so, determine that the target calibration image is a valid calibration image.

[0075] In some embodiments, the device further includes an information initiation module, used to initiate a prompt message to reacquire calibration images when none of the multiple candidate calibration images are valid calibration images.

[0076] The calibration image screening device provided in this disclosure can execute the calibration image screening method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0078] Corresponding to the aforementioned line-of-sight estimation method, Figure 7 This is a schematic diagram of a line-of-sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 7 As shown, the line-of-sight estimation device includes:

[0079] The image acquisition module 702 is used to acquire multiple candidate calibration images and to select valid calibration images from the multiple candidate calibration images using the calibration image filtering method described above.

[0080] The acquisition module 704 is used to terminate the acquisition of candidate calibration images in response to the screening result that the number of valid calibration images is not zero;

[0081] The line-of-sight difference estimation module 706 is used to respond to the line-of-sight estimation request, acquire the target image to be estimated, obtain the effective calibration image, and obtain the line-of-sight difference estimate between the effective calibration image and the target image through a pre-trained network model;

[0082] The gaze direction estimation module 708 is used to estimate the gaze direction corresponding to the target image based on the theoretical gaze direction of the effectively calibrated image and the gaze difference estimate.

[0083] Since the calibration image screening method provided in this embodiment can obtain accurate and reliable effective calibration images, and the theoretical line-of-sight direction marked on the effective calibration image matches the actual line-of-sight direction, the accuracy of line-of-sight direction estimation based on the theoretical line-of-sight direction of the effective calibration image and the line-of-sight difference estimate between the effective calibration image and the target image is well guaranteed.

[0084] In some embodiments, the apparatus further includes: in response to a screening result that the number of valid calibration images is zero, continuing to acquire multiple candidate calibration images until a valid calibration image is selected from the continuously acquired candidate calibration images.

[0085] The line-of-sight direction estimation device provided in this disclosure can execute the line-of-sight direction estimation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0087] This disclosure also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-described calibration image screening method or the above-described line-of-sight estimation method.

[0088] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 800 includes one or more processors 801 and memory 802.

[0089] The processor 801 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0090] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the calibration image screening method, gaze estimation method, and / or other desired functions described in the embodiments of this disclosure above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0091] In one example, the electronic device 800 may also include an input device 803 and an output device 804, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0092] In addition, the input device 803 may also include, for example, a keyboard, a mouse, etc.

[0093] The output device 804 can output various information to the outside, including determined distance information, direction information, etc. The output device 804 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0094] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 800 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 800 may include any other suitable components depending on the specific application.

[0095] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the calibration image screening method or line-of-sight estimation method provided in the embodiments of this disclosure.

[0096] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0097] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the calibration image screening method or line-of-sight estimation method provided in embodiments of this disclosure.

[0098] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the calibration image screening method or gaze estimation method in this disclosure.

[0100] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0101] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. For instance, when collecting an image of a user's eyes, a prompt message is sent to the user regarding the collection of the eye image, and the message may further inform the user of the purpose of the image collection. This allows the user to independently choose whether to provide personal information to the software or hardware such as the electronic device, application program, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message, and the operations of this disclosed technical solution are performed only after obtaining the user's authorization.

[0102] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0103] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for screening calibration images, characterized in that, include: Multiple candidate calibration images are acquired; wherein, the candidate calibration images are eye images taken when the target eye is required to gaze at a specified calibration point, and the specified calibration point is different for different calibration images, and the candidate calibration images are marked with the theoretical line of sight when the target eye gazes at the specified calibration point; The estimated and labeled line-of-sight difference values ​​between each pair of the multiple candidate calibration images are obtained; wherein, the estimated line-of-sight difference value is obtained by estimating using a pre-trained network model, and the labeled line-of-sight difference value is obtained based on the theoretical line-of-sight direction of the candidate calibration images; Based on the estimated and labeled line-of-sight differences between each pair of the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images; wherein, the estimated line-of-sight direction of the valid calibration image and the labeled theoretical line-of-sight direction of the valid calibration image satisfy a preset matching condition.

2. The method according to claim 1, characterized in that, The step of determining the valid calibration image from the multiple candidate calibration images based on the estimated and labeled line-of-sight differences between each pair of candidate calibration images includes: The error between each pair of candidate calibration images is obtained based on the estimated line-of-sight difference and the labeled line-of-sight difference between each pair of the candidate calibration images; wherein the error is determined based on the difference between the estimated line-of-sight difference and the labeled line-of-sight difference. Based on the pairwise errors between the multiple candidate calibration images, a valid calibration image is determined from the multiple candidate calibration images.

3. The method according to claim 2, characterized in that, The number of candidate calibration images is no less than three; the step of determining the valid calibration image from the candidate calibration images based on the pairwise errors between the candidate calibration images includes: The error between the target calibration image and the other candidate calibration images in the plurality of candidate calibration images is averaged to obtain a first mean value; wherein, each candidate calibration image in the plurality of candidate calibration images is used as the target calibration image in turn; The errors between each pair of candidate calibration images (excluding the target calibration image) are averaged to obtain a second mean. Based on the first mean and the second mean, determine whether the target calibration image is a valid calibration image.

4. The method according to claim 3, characterized in that, The step of determining whether the target calibration image is a valid calibration image based on the first mean and the second mean includes: Determine whether the difference between the first mean and the second mean is not greater than a preset threshold; If so, the target calibration image is determined to be a valid calibration image.

5. The method according to claim 1, characterized in that, The method further includes: If none of the candidate calibration images are valid calibration images, a prompt message will be sent to initiate the reacquisition of calibration images.

6. A line-of-sight estimation method, characterized in that, include: Multiple candidate calibration images are acquired, and the calibration image screening method according to any one of claims 1 to 5 is used to screen valid calibration images from the multiple candidate calibration images; In response to the screening result that the number of valid calibration images is not zero, the acquisition of candidate calibration images ends; In response to the gaze estimation request, the target image to be estimated is acquired, and the effective calibration image is obtained. The gaze difference estimate between the effective calibration image and the target image is obtained through a pre-trained network model. Based on the theoretical gaze direction of the effective calibrated image and the gaze difference estimate, the gaze direction corresponding to the target image is estimated.

7. The method according to claim 6, characterized in that, The method further includes: In response to the screening result that the number of valid calibration images is zero, multiple candidate calibration images are continuously acquired until a valid calibration image is selected from the continuously acquired candidate calibration images.

8. A screening device for calibrated images, characterized in that, include: The calibration image acquisition module is used to acquire multiple candidate calibration images; wherein, the candidate calibration images are eye images taken when the target eye is required to gaze at a specified calibration point, different calibration images correspond to different specified calibration points, and the calibration images are marked with the theoretical line of sight when the target eye gazes at the specified calibration point; The line-of-sight difference acquisition module is used to acquire the line-of-sight difference estimates and line-of-sight difference annotations between each pair of the multiple candidate calibration images; wherein, the line-of-sight difference estimates are obtained by estimating using a pre-trained network model, and the line-of-sight difference annotations are obtained based on the theoretical line-of-sight directions of the candidate calibration images; The calibration image filtering module is used to determine the effective calibration image from the multiple candidate calibration images based on the estimated line-of-sight difference and the labeled line-of-sight difference between each pair of the multiple candidate calibration images; wherein the estimated line-of-sight direction of the effective calibration image and the theoretical line-of-sight direction labeled in the effective calibration image satisfy a preset matching condition.

9. A line-of-sight estimation device, characterized in that, include: An image acquisition module is used to acquire multiple candidate calibration images and to select valid calibration images from the multiple candidate calibration images using the calibration image screening method described in any one of claims 1 to 5. The acquisition module is terminated in response to the screening result that the number of valid calibration images is not zero, thereby ending the acquisition of candidate calibration images. The line-of-sight difference estimation module is used to respond to the line-of-sight estimation request, acquire the target image to be estimated, obtain the effective calibration image, and obtain the line-of-sight difference estimate between the effective calibration image and the target image through a pre-trained network model; The gaze direction estimation module is used to estimate the gaze direction corresponding to the target image based on the theoretical gaze direction of the effective calibration image and the gaze difference estimation value.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the calibration image screening method according to any one of claims 1-5 or the gaze estimation method according to any one of claims 6-7.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image screening method according to any one of claims 1-5 or the line-of-sight estimation method according to any one of claims 6-7.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image screening method of any one of claims 1-5 or the gaze estimation method of any one of claims 6-7.

Images