Sight line estimation method, device, equipment and medium

Abstract

The embodiment of the invention relates to a sight line estimation method and device, equipment and a medium. The method comprises the following steps: acquiring calibration information corresponding to a target eye image of a to-be-estimated sight line; wherein the calibration information comprises coordinate information corresponding to pixels of the target eye image in an equipment coordinate system, and the equipment coordinate system is determined based on a camera used for collecting the target eye image; acquiring a camera position corresponding to the camera and a light source position corresponding to a target light source; and based on the target eye image, the calibration information, the camera position and the light source position, utilizing a preset sight line estimation model to estimate a sight line direction corresponding to the target eye image. The sight line direction estimated by the embodiment of the invention is more accurate and reliable.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a line of sight estimation method, device, equipment and medium. Background Art

[0002] In many scenarios such as gaming, medical, and intelligent control, it is necessary to identify the gaze direction of the human eye so that corresponding strategies can be adopted based on the gaze direction estimation results. However, the accuracy of the current gaze direction estimation is likely to have adverse effects on the application scenarios of the gaze estimation results, so it is urgent to improve the accuracy of the gaze estimation results. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a line of sight estimation method, device, equipment and medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a line of sight estimation method, the method comprising: obtaining calibration information corresponding to a target eye image of which a line of sight is to be estimated; wherein the calibration information comprises coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to acquire the target eye image; obtaining a camera position corresponding to the camera and a light source position corresponding to a target light source; and based on the target eye image, the calibration information, the camera position and the light source position, estimating a line of sight direction corresponding to the target eye image using a preset line of sight estimation model.

[0005] In a second aspect, an embodiment of the present disclosure further provides a line of sight estimation device, comprising: a calibration information acquisition module, used to acquire calibration information corresponding to a target eye image of a line of sight to be estimated; wherein the calibration information includes coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to acquire the target eye image; a position acquisition module, used to acquire a camera position corresponding to the camera and a light source position corresponding to a target light source; and a line of sight estimation module, used to estimate a line of sight direction corresponding to the target eye image based on the target eye image, the calibration information, the camera position, and the light source position using a preset line of sight estimation model.

[0006] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a line of sight estimation method as provided in an embodiment of the present disclosure.

[0007] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method provided by the embodiment of the present disclosure.

[0008] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the line of sight estimation method provided in the embodiment of the present disclosure.

[0009] The above technical solution provided by the embodiment of the present disclosure fully considers the influence of the camera on the imaging result of the eye image, and therefore introduces calibration information when estimating the line of sight of the eye image, and can further obtain information such as the light source position and camera position corresponding to the target eye image, and estimate the line of sight direction based on the acquired target eye image and its corresponding calibration information, light source position, camera position and other information with the help of the line of sight estimation model. The line of sight direction estimated by the above solution method is more accurate and reliable.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] Figure 1 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a position relationship provided in an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram of training a sight line estimation model provided in an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of the structure of a sight line estimation device provided by an embodiment of the present disclosure;

[0017] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0020] Figure 1 The present invention provides a flow chart of a line of sight estimation method provided in an embodiment of the present invention. The method can be performed by a line of sight estimation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0021] Step S102, obtaining calibration information corresponding to the target eye image of the line of sight to be estimated; wherein the calibration information includes coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to capture the target eye image.

[0022] The target eye image can be the eye of the target object to be estimated (such as a user). For example, the target user can wear electronic devices such as VR glasses, smart glasses, eye trackers, or other electronic devices that need to estimate the direction of the user's eye line of sight. The camera in the electronic device can collect the target user's eye image as the target eye image; in addition, the user can also directly collect the target user's eye image through a camera at a preset position without wearing the device. The method of collecting the target eye image is not limited here. It should be noted that before collecting the eye image, a prompt message for collecting the eye image can be sent to the user, and the user's eye image can be collected after obtaining the user's authorization.

[0023] In the embodiment of the present disclosure, the influence of the camera parameters on the imaging result of the eye image is fully considered. The embodiment of the present disclosure will additionally obtain calibration information. The calibration information includes the coordinate information corresponding to the pixels of the target eye image in the device coordinate system of the camera. In the specific implementation, each pixel in the target eye image can be projected to the 3D space of the device coordinate system to obtain the X-axis coordinate and Y-axis coordinate of each pixel of the target eye image in the device coordinate system, and the Z-axis coordinate of each pixel in the device coordinate system is set to a preset value, such as a uniform value. The embodiment of the present disclosure does not limit the preset value corresponding to the Z-axis coordinate. The above-mentioned method of obtaining calibration information is equivalent to projecting the 2D eye image into the 3D space, which is helpful for further estimating the line of sight direction corresponding to the eye image in the 3D space.

[0024] Step S104, obtaining a camera position corresponding to the camera and a light source position corresponding to the target light source.

[0025] In practical applications, the number of target light sources may be one or more, including but not limited to the use of light sources such as LED (Light Emitting Diode). In the embodiments of the present disclosure, in addition to configuring the camera, one or more light sources are also configured to assist in estimating the line of sight of the eye based on the light spot formed by the light source on the eye. In practical applications, the target light source and the camera can be integrated in electronic devices such as the aforementioned VR glasses, eye trackers, etc. that need to estimate the user's line of sight, such as being integrated in the lens barrel of the electronic device. In addition, an external light source can also be directly used as the target light source, which is not limited here.

[0026] Step S106, based on the target eye image, calibration information, camera position and light source position, a preset sight line estimation model is used to estimate the sight line direction corresponding to the target eye image. Compared with the sight line estimation model obtained by directly training the eye image with the sight line direction marked in the related art, in the embodiment of the present disclosure, the sight line direction can be estimated based on the target eye image, calibration information, camera position and light source position with the help of the sight line estimation model. Specifically, the model input of the sight line estimation model can be determined based on the target eye image, calibration information, camera position and light source position, and then the sight line direction corresponding to the target eye image can be determined based on the model output. The embodiment of the present disclosure introduces additional calibration information into the input of the sight line estimation model, which helps to improve the estimation accuracy of the sight line direction. The main reason is that the inventor has found through research that the camera parameters of different devices or the same device may be different when worn at different times, and the relative position between the camera and the eye may also be different. All of the above can be regarded as camera state changes, which may have a certain degree of impact on the imaging result, and then affect the sight line direction represented by the eye image collected by the camera. However, under normal circumstances, the eye image samples used for training the gaze estimation model are basically difficult to fully cover the eye images taken under different camera states. As a result, the trained gaze estimation model is difficult to predict the impact of different camera state changes on the gaze direction. For this reason, the inventors additionally use the calibration information as the input of the gaze estimation model and directly introduce it into the estimation process of the gaze estimation model, which not only helps to improve the gaze estimation accuracy of the model, but also eliminates the need to spend a lot of cost to collect eye images corresponding to different camera states when training the model.

[0027] The disclosed embodiments provide an implementation example of estimating the line of sight direction corresponding to a target eye image using a preset line of sight estimation model. Specifically, the preset line of sight estimation model can be used to determine the corneal center position and the pupil position of the eye in the target eye image, and based on the corneal center position and the pupil position, the line of sight direction corresponding to the target eye image is estimated. The line of sight direction is the line between the corneal center position and the pupil position. In actual applications, the line of sight estimation model can output the corneal center position and the pupil position, so as to further estimate the line of sight direction based on the corneal center position and the pupil position; the step of estimating the line of sight direction based on the corneal center position and the pupil position can also be built into the line of sight estimation model, and the line of sight estimation model directly outputs the line of sight direction.

[0028] For example, the corneal center position and pupil position obtained by the line of sight estimation model may be 3D positions, and the corresponding line of sight direction may be a 3D direction. In addition, 3D may be converted to 2D as required, which is not limited here.

[0029] In some embodiments, when determining the corneal center position and the pupil position of the eye in the target eye image, the line of sight estimation model may refer to the following steps A to B:

[0030] Step A, merge the target eye image and the calibration information to obtain merged information. Exemplarily, based on a preset channel dimension, the target eye image and the calibration information are merged, wherein the merge process includes but is not limited to a splicing method. That is, the target eye image, the X-axis coordinates of each pixel in the target eye image, the Y-axis coordinates and other coordinate information in the device coordinate system can be spliced ​​in the channel dimension to obtain the merged information. The above method can also be regarded as converting a two-dimensional image into an image in a three-dimensional space, where the image in the three-dimensional space is encoded in the form of coordinates and represented by the merged information, which presents the target eye image information itself and the spatial information of mapping the target eye image to the three-dimensional space, which is more convenient for subsequent analysis and processing.

[0031] Step B, based on the merged information, the camera position and the light source position, determine the light spot position on the eye, the corneal center position of the eye and the pupil position of the eye corresponding to the target light source; wherein the light spot position and the corneal center position satisfy the preset relationship constraint conditions, and the pupil position is related to the light spot position and the corneal center position.

[0032] The above-mentioned preset relationship constraints are also the solution logic that the line of sight estimation model needs to follow, so that the relationship between the spot position and the corneal center position output by the line of sight estimation model can conform to the preset solution logic, thereby effectively ensuring the accuracy of the model output results. In some specific implementation examples, the preset relationship constraints include: the angle between the target line and the angle bisector of the target angle is less than a preset angle threshold, wherein the target line is the line between the spot position and the corneal center position, and the target angle is determined based on the angle between the first line and the second line, the first line is the line between the spot position and the light source position, and the second line is the line between the spot position and the camera position. For ease of understanding, you can refer to Figure 2 A schematic diagram of the position relationship shown in FIG. 1 illustrates the eyeball, the position of the camera used to capture the eye image, and the target light source. It should be noted that Figure 2 In the figure, only one target light source is shown. In practical applications, there may be multiple target light sources. Figure 2In , the center point of the eyeball is regarded as the corneal center position C, the target light source M forms a spot Q on the eyeball, the line connecting the camera position O and the spot Q is referred to as L1, and the line connecting the target light source M and the spot Q is marked as L2. Then the line connecting the corneal center C and the spot Q (marked as L3) is theoretically the angle bisector of L1 and L2. In other words, the angle between the aforementioned target line (i.e., L3) and the angle bisector of the aforementioned target angle (i.e., the angle between L1 and L2) should be zero in theory, but considering the calculation error and other reasons, a small range of fluctuations can be allowed. Therefore, a smaller angle threshold can be set based on the allowed fluctuation range. The angle between the target line and the angle bisector of the target angle is less than the preset angle threshold, which can be regarded as satisfying the preset relationship constraint condition. By setting the above-mentioned preset relationship constraint condition, it is equivalent to introducing the solution logic of eye-related information in the line of sight estimation process of the line of sight estimation model, so as to further reduce the negative impact of the camera state change on the accuracy of the line of sight estimation result as mentioned above.

[0033] In some specific implementation examples, the above step B can be performed with reference to the following steps B1 to B3:

[0034] Step B1, based on the merged information, the camera position and the light source position, determine the spot position of the target light source on the eye and the center position of the cornea of ​​the eye. In practical applications, feature extraction can be performed on the merged information, the camera position and the light source position, and then the spot position and the center position of the cornea formed by the light source on the eye can be determined based on the corresponding features. For ease of understanding, in some specific implementation examples, the above step B1 can be performed with reference to the following steps B1.1 to B1.4:

[0035] Step B1.1, extracting features from the merged information to obtain merged features. The disclosed embodiment does not limit the feature extraction algorithm or the feature extraction network used.

[0036] Step B1.2, the camera position is subjected to a first encoding process to obtain a camera position feature, and the light source position is subjected to a second encoding process to obtain a light source position feature. If there are multiple target light sources, each light source position is encoded separately to obtain multiple light source position features. The dimensions of the camera position feature and the light source position feature obtained by the above encoding method are consistent with the dimensions of the aforementioned merged feature.

[0037] Step B1.3, based on the merged feature, the camera position feature and the light source position feature, the light source associated feature corresponding to the target light source is obtained. The light source associated feature obtained by integrating the merged feature, the camera position feature and the light source position feature can present information related to the target light source. For example, the light source position feature and the merged feature are multiplied, and the light source associated feature is obtained based on the difference between the product result and the camera position feature. Assuming that there are N target light sources in total, the light source associated feature feature_i corresponding to the i-th target light source can be expressed as:

[0038] feature_i=Ledi_PE*F-Camera_PE

[0039] Among them, Ledi_PE represents the light source position feature corresponding to the i-th target light source Led, F represents the merged feature, and Camera_PE represents the camera position feature.

[0040] Step B1.4, obtain the corneal center feature, and determine the spot position of the target light source on the eye and the corneal center position of the eye based on the corneal center feature and the light source association feature corresponding to the target light source. The corneal center feature can be a preset feature vector, such as using a randomly initialized feature vector as a feature vector for characterizing the corneal center, and the dimension of the corneal center feature is consistent with the dimension of the aforementioned merged feature, camera position feature, and light source position feature.

[0041] In order to accurately determine the spot position and the corneal center position, the disclosed embodiment can use a self-attention mechanism to determine the spot position of the target light source on the eye and the corneal center position of the eye based on the corneal center feature and the light source associated feature corresponding to the target light source. It can be understood that the self-attention mechanism can effectively establish associations between different features. On the basis of the known corneal center feature and the light source associated feature, the self-attention mechanism can be used to perform association analysis on different features and extract effective information, so that the corneal center position and the spot position can be accurately and reliably determined. In practical applications, the number of target light sources can be multiple, and the light source associated features corresponding to the multiple target light sources can be firstly processed by the first self-attention to obtain the spot information corresponding to each target light source; then, based on the spot information, the corneal center feature and the light source associated features corresponding to the multiple target light sources are processed by the second self-attention to obtain the spot position of the target light source on the eye and the corneal center position of the eye. Specifically, multiple light source associated features (assuming N light source associated features) can first be processed by the first attention to each other. For each target light source, based on its own light source associated features, combined with the light source associated features of other target light sources, self-attention processing can more effectively extract the light spot information. On this basis, the corneal center feature can be introduced, combined with the aforementioned N light source associated features, and a total of N+1 features can be self-attention processed in pairs, and the light spot information obtained by the aforementioned first attention processing can be further optimized to determine more reliable N light spot positions and a corneal center position.

[0042] Step B2, based on the merged information, determine the pupil direction corresponding to the eye in the device coordinate system. Specifically, a merged feature corresponding to the merged information can be obtained, and based on the merged feature, the pupil direction corresponding to the eye in the device coordinate system is predicted. The pupil direction can be a straight line passing through the pupil in the three-dimensional space represented by the device coordinate system.

[0043] Step B3, determine the pupil position of the eye according to the light spot position, the corneal center position and the pupil direction. When the above information is determined, the pupil position can be further obtained by logical solution. In some specific implementation examples, the pupil position can be determined by referring to the following steps B3.1 to B3.2:

[0044] Step B3.1, determining the corneal radius corresponding to the eye according to the light spot position and the corneal center position. Exemplarily, the corneal radius R can be calculated according to the Euler distance between the light spot position and the corneal center position.

[0045] Step B3.2, determine the pupil position of the eye according to the corneal center position, corneal radius and pupil direction. It can be understood that the corneal center and corneal radius can form a sphere, and the straight line in the pupil direction intersects with the sphere, and the intersection point can be regarded as the 3D position of the pupil.

[0046] The above-mentioned method provided by the embodiment of the present disclosure can be implemented with the help of a sight line estimation model. The embodiment of the present disclosure does not limit the structure of the sight line estimation model. In order to ensure that the sight line estimation model can output reliable results, on the basis of the above, the embodiment of the present disclosure provides a training method for the sight line estimation model, that is, the sight line estimation model is obtained through the following steps 1 to 4:

[0047] Step 1, obtaining an eye sample image; wherein the eye sample image carries calibration information, camera position, light source position and theoretical line of sight direction corresponding to the eye sample image.

[0048] Step 2, based on the eye sample image and the calibration information, camera position, and light source position corresponding to the eye sample image, use the initial estimation model to determine the target information corresponding to the eye sample image; the target information corresponding to the eye sample image includes the corneal center position and pupil position corresponding to the eye sample image.

[0049] Step 3, according to the corneal center position and pupil position corresponding to the eye sample image, determine the estimated sight line direction corresponding to the eye sample image, and determine the first loss based on the difference between the estimated sight line direction and the theoretical sight line direction. For example, based on the difference between the estimated sight line direction and the theoretical sight line direction, a preset loss function is used to determine the first loss. In practical applications, the estimated sight line direction can be a three-dimensional direction. In this case, the three-dimensional sight line direction can be converted into a two-dimensional sight line direction, which is more convenient for calculating the difference between the sight line directions and determining the first loss.

[0050] Step 4: train the initial estimation model based on the first loss to obtain a sight line estimation model based on the trained initial estimation model. Adjust the parameters of the initial estimation model in the direction of reducing the first loss until the preset training end condition is reached to end the training, obtain the trained initial estimation model, and use the trained initial estimation model as the sight line estimation model.

[0051] In order to ensure the model training effect and the accuracy of the output results of the line of sight estimation model, in some embodiments, the target information corresponding to the eye sample image also includes the spot position corresponding to the eye sample image; on this basis, the above step 4 can be performed with reference to the following steps 4.1 to 4.3:

[0052] Step 4.1, obtain the correlation between the spot position and the corneal center position corresponding to the eye sample image, and determine the second loss based on the difference between the correlation and the preset relationship. In practical applications, if there are multiple target light sources, the corresponding second loss can be calculated for each target light source, and then the multiple second losses can be fused by means such as averaging.

[0053] In some embodiments, the preset relationship may include: the line between the spot position and the corneal center position is the angle bisector of the target angle, the target angle is determined based on the angle between the first line and the second line, the first line is the line between the spot position and the light source position, and the second line is the line between the spot position and the camera position. On this basis, the difference between the association relationship and the preset relationship can be measured by the angle between the line between the spot position and the corneal center position corresponding to the eye sample image output by the initial estimation model and the angle bisector of the target angle, and the smaller the angle is, the smaller the difference between the association relationship and the preset relationship is.

[0054] On the basis of the above implementation, for the convenience of calculation, the logic that the line between the spot position and the corneal center position is theoretically the angular bisector of the target angle can be represented by vector operation. Specifically, the first unit vector is determined based on the line between the spot position and the light source position, the second unit vector is determined based on the line between the spot position and the camera position, the vector obtained by subtracting the second unit vector from the first unit vector can be regarded as the eyeball tangent vector, and the third vector is determined based on the line between the spot position and the corneal center position. The third vector is perpendicular to the eyeball tangent vector, and the product of the third vector and the eyeball tangent vector is zero. On this basis, the difference between the association relationship and the preset relationship can be measured by the difference between the product of the third vector calculated based on the camera position, the spot position and the corneal center position output by the initial estimation model and the eyeball switching vector and zero.

[0055] The above methods can all reasonably obtain the second loss, but they are all exemplary descriptions. In actual applications, other methods can also be used to determine the second loss, which is not limited here.

[0056] Step 4.2: weight the first loss and the second loss to obtain the total loss. The weights of the first loss and the second loss in the embodiment of the present disclosure can be flexibly set according to the requirements, and are not limited here.

[0057] Step 4.3, train the initial estimation model based on the total loss. In practical applications, the training of the initial estimation model can be stopped when the total loss converges to a preset threshold.

[0058] For ease of understanding, the present disclosure also provides Figure 3The training diagram of a line of sight estimation model shown in the figure simply illustrates the camera and four target light sources, LED1 to LED4, and obtains the corresponding encoding vectors based on the position encoding method, specifically the camera position feature, the first light source position feature, the second light source position feature, the third light source position feature and the fourth light source position feature. The merged information obtained by the eye image and the calibration information is used for feature extraction to obtain the merged feature. Based on the camera position feature, the first to fourth light source position features and the merged feature, the first light source associated feature, the second light source associated feature, the third light source associated feature and the fourth light source associated feature can be obtained. Figure 3 It is simply illustrated in the figure that the line of sight estimation model includes a first neural network and a second neural network. The first light source associated feature to the fourth light source associated feature can be input into the first neural network, and the first neural network can output the first light spot position, the second light spot position, the third light spot position, the fourth light spot position and the corneal center position. The combined features are input into the second neural network, and the second neural network can output the pupil direction. Based on the first light spot position to the fourth light spot position and the corneal center position, the solution loss (that is, the aforementioned second loss) can be obtained. The specific calculation method can refer to the aforementioned related content, and based on the pupil direction and the corneal center position, the line of sight direction loss (that is, the aforementioned first loss) can be obtained. Specifically, the pupil position can be determined based on the pupil direction, and the pupil position and the corneal center position can obtain the estimated line of sight direction, so that the line of sight direction loss can be determined in combination with the theoretical line of sight direction. Finally, the model training can be performed based on the solution loss and the line of sight direction loss.

[0059] The present disclosure embodiments Figure 3 The structures of the first neural network and the second neural network are not limited. Exemplarily, the first neural network can be a Transformer network. The aforementioned step of performing a first self-attention process on the light source-related features corresponding to the multiple target light sources to obtain the light spot information corresponding to each of the target light sources can be implemented with the aid of the encoder in the Transformer network. The aforementioned step of performing a second self-attention process on the corneal center features and the light source-related features corresponding to the multiple target light sources based on the light spot information to obtain the light spot position corresponding to the target light source on the eye and the corneal center position of the eye can be implemented with the aid of the decoder in the Transformer network. It should be noted that the above is only an exemplary description. In actual applications, other network structures can also be used, which are not limited here. In addition, Figure 3 The entire network structure of the line of sight estimation model is not fully illustrated in the figure. For example, the network used for position encoding can also be considered as part of the line of sight estimation model, but in Figure 3 It is not indicated.

[0060] To sum up, compared with the related art in which the gaze estimation model is directly trained using eye images marked with gaze directions to perform gaze estimation, the gaze estimation model obtained in the embodiment of the present disclosure can perform gaze estimation by means of logical solution, and the obtained gaze estimation result is more accurate and reliable.

[0061] Corresponding to the aforementioned sight line estimation method, the embodiment of the present disclosure provides a sight line estimation device, Figure 4 FIG. 1 is a schematic diagram of a structure of a sight line estimation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 4 As shown, the sight line estimation device includes:

[0062] A calibration information acquisition module 402 is used to acquire calibration information corresponding to the target eye image of the line of sight to be estimated; wherein the calibration information includes coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to capture the target eye image;

[0063] A position acquisition module 404 is used to acquire a camera position corresponding to the camera and a light source position corresponding to the target light source;

[0064] The sight line estimation module 406 is used to estimate the sight line direction corresponding to the target eye image using a preset sight line estimation model based on the target eye image, calibration information, camera position, and light source position.

[0065] The above device provided by the embodiment of the present disclosure fully considers the influence of the camera on the imaging result of the eye image, and therefore introduces calibration information when estimating the line of sight of the eye image, and can further obtain information such as the light source position and camera position corresponding to the target eye image, and estimate the line of sight direction based on the acquired target eye image and its corresponding calibration information, light source position, camera position and other information with the help of the line of sight estimation model. The line of sight direction estimated by the above solution method is more accurate and reliable.

[0066] In some embodiments, the line of sight estimation module 406 is specifically used to: determine the corneal center position and the pupil position of the eye in the target eye image using a preset line of sight estimation model, and estimate the line of sight direction corresponding to the target eye image based on the corneal center position and the pupil position.

[0067] In some embodiments, the line of sight estimation module 406 is specifically used to: merge the target eye image and the calibration information to obtain merged information; determine the light spot position on the eye, the corneal center position of the eye and the pupil position of the eye corresponding to the target light source based on the merged information, the camera position and the light source position; wherein the light spot position and the corneal center position satisfy a preset relationship constraint; and the pupil position is related to the light spot position and the corneal center position.

[0068] In some embodiments, the preset relationship constraint condition includes: the angle between the target line and the angle bisector of the target angle is less than a preset angle threshold, wherein the target line is the line between the light spot position and the corneal center position, and the target angle is determined based on the angle between a first line and a second line, the first line is the line between the light spot position and the light source position, and the second line is the line between the light spot position and the camera position.

[0069] In some embodiments, the line of sight estimation module 406 is specifically used to: determine the light spot position of the target light source on the eye and the corneal center position of the eye based on the merged information, the camera position, and the light source position; determine the pupil direction corresponding to the eye in the device coordinate system based on the merged information; determine the pupil position of the eye according to the light spot position, the corneal center position, and the pupil direction.

[0070] In some embodiments, the line of sight estimation module 406 is specifically used to: perform feature extraction on the merged information to obtain a merged feature; perform a first encoding process on the camera position to obtain a camera position feature, and perform a second encoding process on the light source position to obtain a light source position feature; obtain a light source associated feature corresponding to the target light source based on the merged feature, the camera position feature, and the light source position feature; obtain a corneal center feature, and determine the light spot position on the eye corresponding to the target light source and the corneal center position of the eye based on the corneal center feature and the light source associated feature corresponding to the target light source.

[0071] In some embodiments, the line of sight estimation module 406 is specifically used to: based on the corneal center feature and the light source associated feature corresponding to the target light source, use a self-attention mechanism to determine the light spot position corresponding to the target light source on the eye and the corneal center position of the eye.

[0072] In some embodiments, there are multiple target light sources, and the position acquisition module 404 is specifically used to: perform a first self-attention processing on the light source associated features corresponding to the multiple target light sources to obtain the light spot information corresponding to each of the target light sources; based on the light spot information, perform a second self-attention processing on the corneal center feature and the light source associated features corresponding to the multiple target light sources to obtain the light spot position of the target light source on the eye and the corneal center position of the eye.

[0073] In some embodiments, the line of sight estimation module 406 is specifically used to: determine the corneal radius corresponding to the eye based on the light spot position and the corneal center position; determine the pupil position of the eye based on the corneal center position, the corneal radius and the pupil direction.

[0074] In some embodiments, the device also includes a model training module, which is used to obtain the line of sight estimation model through the following steps: obtaining the eye sample image; wherein the eye sample image carries a calibration information sample, a camera position sample, a light source position sample and a theoretical line of sight direction corresponding to the eye sample image; based on the eye sample image, and the calibration information, camera position, and light source position corresponding to the eye sample image, using an initial estimation model to determine the target information corresponding to the eye sample image; the target information corresponding to the eye sample image includes a corneal center position and a pupil position corresponding to the eye sample image; according to the corneal center position and the pupil position corresponding to the eye sample image, determining the estimated line of sight direction corresponding to the eye sample image, and determining a first loss based on the difference between the estimated line of sight direction and the theoretical line of sight direction; training the initial estimation model based on the first loss to obtain a line of sight estimation model based on the trained initial estimation model.

[0075] In some embodiments, the model training module is specifically used to: obtain the correlation relationship between the spot position corresponding to the eye sample image and the corneal center position, and determine the second loss based on the difference between the correlation relationship and a preset relationship; perform weighted processing on the first loss and the second loss to obtain a total loss; and train the initial estimation model based on the total loss.

[0076] The line of sight estimation device provided in the embodiments of the present disclosure can execute the line of sight estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0078] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0079] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0081] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0082] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0083] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0084] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the line of sight estimation method provided by the embodiment of the present disclosure.

[0085] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0086] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the line of sight estimation method in the embodiments of the present disclosure when executed by a processor.

[0087] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0088] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, such as collecting the user's eye image. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message. The present embodiment of the present disclosure will only obtain the user's eye image if the user grants permission.

[0089] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0090] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0091] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0092] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight estimation method, characterized in that: include: Acquire calibration information corresponding to the target eye image of the line of sight to be estimated; wherein the calibration information includes coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to capture the target eye image; Obtaining a camera position corresponding to the camera and a light source position corresponding to a target light source; Based on the target eye image, the calibration information, the camera position and the light source position, a preset line of sight estimation model is used to estimate the line of sight direction corresponding to the target eye image.

2. The method according to claim 1, characterized in that The estimating the sight line direction corresponding to the target eye image by using a preset sight line estimation model includes: The corneal center position and the pupil position of the eye in the target eye image are determined using a preset sight line estimation model, and based on the corneal center position and the pupil position, the sight line direction corresponding to the target eye image is estimated.

3. The method according to claim 2, characterized in that Determining the corneal center position of the eye and the pupil position of the eye in the target eye image comprises: Merging the target eye image and the calibration information to obtain merged information; Based on the merged information, the camera position and the light source position, determine the light spot position on the eye, the cornea center position of the eye and the pupil position of the eye corresponding to the target light source; wherein the light spot position and the cornea center position satisfy a preset relationship constraint, and the pupil position is related to the light spot position and the cornea center position.

4. The method according to claim 3, characterized in that The preset relationship constraint condition includes: the angle between the target line and the angle bisector of the target angle is less than a preset angle threshold, wherein the target line is the line between the light spot position and the corneal center position, and the target angle is determined based on the angle between a first line and a second line, the first line is the line between the light spot position and the light source position, and the second line is the line between the light spot position and the camera position.

5. The method according to claim 3, characterized in that: The determining, based on the merged information, the camera position and the light source position, the light spot position of the target light source corresponding to the eye, the corneal center position of the eye and the pupil position of the eye comprises: Based on the merged information, the camera position and the light source position, determining the spot position of the target light source on the eye and the corneal center position of the eye; Based on the combined information, determining a pupil direction corresponding to the eye in the device coordinate system; The pupil position of the eye is determined according to the light spot position, the corneal center position and the pupil direction.

6. The method according to claim 5, characterized in that The step of determining the spot position of the target light source corresponding to the eye and the cornea center position of the eye based on the merged information, the camera position and the light source position comprises: Performing feature extraction on the merged information to obtain merged features; Performing a first encoding process on the camera position to obtain a camera position feature, and performing a second encoding process on the light source position to obtain a light source position feature; Obtaining a light source associated feature corresponding to the target light source according to the combined feature, the camera position feature and the light source position feature; A cornea center feature is acquired, and based on the cornea center feature and a light source association feature corresponding to the target light source, a light spot position corresponding to the target light source on the eye and a cornea center position of the eye are determined.

7. The method according to claim 6, characterized in that The step of determining the light spot position of the target light source on the eye and the cornea center position of the eye based on the cornea center feature and the light source association feature corresponding to the target light source comprises: Based on the cornea center feature and the light source associated feature corresponding to the target light source, a self-attention mechanism is used to determine the light spot position corresponding to the target light source on the eye and the cornea center position of the eye.

8. The method according to claim 7, characterized in that The number of the target light sources is multiple, and the self-attention mechanism is used to determine the light spot position of the target light source on the eye and the cornea center position of the eye based on the cornea center feature and the light source association feature corresponding to the target light source, including: Performing a first self-attention process on the light source associated features corresponding to the plurality of target light sources to obtain light spot information corresponding to each of the target light sources; Based on the light spot information, a second self-attention process is performed on the cornea center feature and the light source associated features corresponding to the multiple target light sources to obtain the light spot position corresponding to the target light source on the eye and the cornea center position of the eye.

9. The method according to claim 5, characterized in that Determining the pupil position of the eye according to the light spot position, the corneal center position and the pupil direction includes: Determining the corneal radius corresponding to the eye according to the light spot position and the corneal center position; The pupil position of the eye is determined according to the corneal center position, the corneal radius and the pupil direction.

10. The method according to claim 1, characterized in that The line of sight estimation model is obtained by the following steps: Acquire the eye sample image; wherein the eye sample image carries calibration information, camera position, light source position and theoretical sight line direction corresponding to the eye sample image; Based on the eye sample image and the calibration information, camera position, and light source position corresponding to the eye sample image, the target information corresponding to the eye sample image is determined using an initial estimation model; the target information corresponding to the eye sample image includes the corneal center position and the pupil position corresponding to the eye sample image; Determine an estimated sight line direction corresponding to the eye sample image according to a corneal center position and a pupil position corresponding to the eye sample image, and determine a first loss based on a difference between the estimated sight line direction and the theoretical sight line direction; The initial estimation model is trained based on the first loss to obtain a line of sight estimation model based on the trained initial estimation model.

11. The method according to claim 10, characterized in that The target information corresponding to the eye sample image also includes the light spot position corresponding to the eye sample image; the step of training the initial estimation model based on the first loss includes: Acquire a correlation relationship between the spot position corresponding to the eye sample image and the cornea center position, and determine a second loss based on a difference between the correlation relationship and a preset relationship; Performing weighted processing on the first loss and the second loss to obtain a total loss; The initial estimation model is trained based on the total loss.

12. A sight line estimation device, characterized in that: include: A calibration information acquisition module, used to acquire calibration information corresponding to the target eye image of the line of sight to be estimated; wherein the calibration information includes coordinate information corresponding to pixels of the target eye image in a device coordinate system, and the device coordinate system is determined based on a camera used to capture the target eye image; A position acquisition module, used to acquire a camera position corresponding to the camera and a light source position corresponding to a target light source; A sight line estimation module is used to estimate the sight line direction corresponding to the target eye image using a preset sight line estimation model based on the target eye image, the calibration information, the camera position, and the light source position.

13. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the line of sight estimation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method described in any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the line of sight estimation method according to any one of claims 1 to 11.

Images