Sight line estimation method, device, equipment and medium

Abstract

The embodiment of the invention relates to a sight line estimation method and device, equipment and a medium, and the method comprises the steps: obtaining a target image feature corresponding to a target eye image in a to-be-estimated sight line direction, and obtaining a calibration image feature corresponding to a preset sight line direction; selecting one feature from the target image features and the calibration image features as an initial first feature and taking the other feature as a fixed second feature, and performing at least one time of rotation processing based on the first feature until the sight line direction corresponding to the rotated first feature is consistent with the sight line direction corresponding to the second feature; obtaining rotation information corresponding to each rotation processing, and determining a sight line direction conversion relation between the target image feature and the calibration image feature based on the rotation information corresponding to at least one rotation processing; and estimating the sight line direction corresponding to the target eye image based on the sight line direction conversion relation and a preset sight line direction. According to the embodiment of the invention, the accuracy of line-of-sight estimation can be well guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a line of sight estimation method, device, equipment and medium. Background Art

[0002] In many scenarios such as gaming, medical, and intelligent control, it is necessary to identify the gaze direction of the human eye so that corresponding strategies can be adopted based on the gaze direction estimation results. However, the current gaze direction estimation is not accurate enough, which can easily have adverse effects on the application scenarios of the gaze estimation results. Therefore, it is urgent to improve the accuracy of the gaze estimation results. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a line of sight estimation method, device, equipment and medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: obtaining a target image feature corresponding to a target eye image whose line of sight direction is to be estimated, and obtaining a calibration image feature corresponding to a preset line of sight direction; selecting one feature from the target image feature and the calibration image feature as an initial first feature and another feature as a fixed second feature, and performing at least one rotation processing based on the first feature until the line of sight direction corresponding to the first feature after rotation is consistent with the line of sight direction corresponding to the second feature; obtaining rotation information corresponding to each of the rotation processing to determine a line of sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to each of the at least one rotation processing; estimating the line of sight direction corresponding to the target eye image based on the line of sight direction transformation relationship and the preset line of sight direction corresponding to the calibration image feature.

[0005] In the second aspect, the embodiments of the present disclosure also provide a line of sight estimation device, including: a feature acquisition module, used to acquire target image features corresponding to a target eye image whose line of sight direction is to be estimated, and to acquire calibration image features corresponding to a preset line of sight direction; a rotation processing module, used to select one feature from the target image features and the calibration image features as an initial first feature and another feature as a fixed second feature, and perform at least one rotation processing based on the first feature until the line of sight direction corresponding to the first feature after rotation is consistent with the line of sight direction corresponding to the second feature; a relationship determination module, used to acquire rotation information corresponding to each of the rotation processing, so as to determine the line of sight direction transformation relationship between the target image features and the calibration image features based on the rotation information corresponding to the at least one rotation processing; a line of sight estimation module, used to estimate the line of sight direction corresponding to the target eye image based on the line of sight direction transformation relationship and the preset line of sight direction corresponding to the calibration image features.

[0006] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a line of sight estimation method as provided in an embodiment of the present disclosure.

[0007] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method provided by the embodiment of the present disclosure.

[0008] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the line of sight estimation method provided in the embodiment of the present disclosure.

[0009] The above technical solution provided by the embodiment of the present disclosure, based on the target image features corresponding to the target eye image to be estimated and the calibration image features corresponding to the preset line of sight direction, causes one of the features to be rotated at least once until the line of sight direction corresponding to it is rotated to be consistent with the line of sight direction corresponding to another fixed feature, thereby determining the line of sight direction transformation relationship between the target image features and the calibration image features based on the rotation information, and then combining the preset line of sight direction to estimate the line of sight direction corresponding to the target eye image. Through one or more rotation processes, the line of sight direction of one feature gradually approaches the other line of sight direction until the two line of sight directions are consistent, and the line of sight direction transformation relationship obtained based on the rotation information corresponding to each of the at least one rotation processes is more accurate, thereby further ensuring the accuracy of the line of sight estimation result corresponding to the target eye image.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] Figure 1 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0014] Figure 2 An angle schematic diagram provided for an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of the structure of a sight line estimation device provided by an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0021] Figure 1The flowchart of a line of sight estimation method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a line of sight estimation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0022] Step S102, obtaining target image features corresponding to the target eye image of the to-be-estimated sight line direction, and obtaining calibration image features corresponding to the preset sight line direction.

[0023] The target eye image can be the eye of the target object to be estimated (such as a user). For example, the target user can wear electronic devices such as VR glasses, smart glasses, eye trackers, or other electronic devices that need to estimate the direction of the user's eye line of sight. The electronic device can collect the target user's eye image as the target eye image; in addition, the user can also directly collect the target user's eye image through a camera at a preset position without wearing the device. The collection method of the target eye image is not limited here. It should be noted that before collecting the eye image, a prompt message for collecting the eye image can be sent to the user, and the user's eye image can be collected after obtaining the user's authorization.

[0024] After acquiring the target eye image whose sight direction is to be estimated, the disclosed embodiment can extract features of the target eye image through a feature extraction network, thereby obtaining target image features corresponding to the target eye image. In addition, in order to facilitate the estimation of the sight direction of the target eye image, the disclosed embodiment additionally introduces calibration image features corresponding to the preset sight direction.

[0025] The disclosed embodiments do not limit the acquisition method of the calibration image features corresponding to the preset sight line direction. In some implementation examples, a preset calibration eye image can be acquired, the sight line direction corresponding to the calibration eye image is used as the preset sight line direction, and the calibration image features corresponding to the calibration eye image are extracted. The calibration image features corresponding to the calibration eye image are the calibration image features corresponding to the preset sight line direction. Exemplarily, the calibration eye image can be a user eye image acquired by pre-assigning the target user to gaze at a specified calibration point, and the calibration eye image is collected in this way. At this time, the sight line direction of the target user when gazing at the calibration point is the aforementioned preset sight line direction. The preset sight line direction is known and can be used to assist in estimating the sight line direction of the target eye image in the subsequent process. In other implementation examples, multiple eye images with preset sight lines can be acquired, and the image features corresponding to each of the multiple eye images can be extracted, and the image features corresponding to each of the multiple eye images can be fused to obtain the calibration image features corresponding to the preset sight line direction. For example, the preset sight line direction is the normal sight line direction. At this time, multiple eye images with normal sight line directions (which may not be the eye images of the target user themselves) are acquired, and then the image features corresponding to each of the multiple eye images are averaged and fused to obtain the calibration image features corresponding to the preset sight line direction.

[0026] The embodiments of the present disclosure do not limit the method for extracting the above-mentioned image features, and a network or algorithm capable of extracting image features can be selected according to actual needs.

[0027] Step S104, selecting one feature from the target image feature and the calibration image feature as the initial first feature and the other feature as the fixed second feature, and performing at least one rotation process based on the first feature until the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature.

[0028] In some embodiments, a feature may be randomly selected from the target image feature and the calibration image feature as the initial first feature. For example, if the target image feature is selected as the first feature, the calibration image feature is used as the second feature. For another example, if the calibration image feature is selected as the first feature, the target image feature is used as the second feature. Specifically, the first feature and the second feature may be flexibly selected from the target image feature and the calibration image feature according to the needs, and are not limited here.

[0029] The disclosed embodiment may perform at least one rotation process on the first feature, so that the sight direction corresponding to the rotated first feature may gradually approach the sight direction corresponding to the second feature, until the sight directions corresponding to the two features coincide.

[0030] Step S106, obtaining rotation information corresponding to each rotation process, so as to determine the sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to at least one rotation process.

[0031] Exemplarily, the rotation information can be represented by the rotation angles corresponding to the XYZ axes of the line of sight in three-dimensional space. For ease of processing, the rotation matrix can be represented by a rotation matrix composed of rotation angle parameters. The disclosed embodiment can integrate the rotation information corresponding to multiple rotation processes to obtain the rotation angle of the line of sight direction between the initial first feature and the second feature, that is, to obtain the line of sight direction transformation relationship between the target image feature and the calibration image feature.

[0032] Step S108 , estimating the gaze direction corresponding to the target eye image based on the gaze direction transformation relationship and the preset gaze direction corresponding to the calibration image feature.

[0033] Based on the known relationship between the sight direction transformation of the calibration image features and the target eye image, the preset sight direction is transformed to obtain the sight direction corresponding to the target eye image.

[0034] Through one or more rotation processes, the sight direction of one feature is gradually approached to the other sight direction until the two sight directions are consistent. The sight direction transformation relationship obtained based on the rotation information corresponding to at least one rotation process is more accurate, thereby further ensuring the accuracy of the sight estimation result corresponding to the target eye image.

[0035] The embodiment of the present disclosure further provides a specific implementation of the rotation process. Exemplarily, the rotation process at least includes the following steps A to B:

[0036] Step A, obtaining a high-dimensional rotation matrix between the current first feature and the second feature; wherein the current first feature corresponding to the first rotation process is the initial first feature, and the current first feature corresponding to the non-first rotation process is the first feature after the last rotation process; the high-dimensional rotation matrix is ​​used to characterize the transformation relationship between the current first feature and the second feature in a preset high-dimensional space, and the dimension of the high-dimensional rotation matrix is ​​consistent with the dimension of the target image feature and the dimension of the calibration image feature. For example, the dimension of the high-dimensional rotation matrix can be 128 dimensions, and the size of the high-dimensional rotation matrix is ​​128*128.

[0037] In practical applications, the high-dimensional rotation matrix between the current first feature and the second feature can be predicted by a preset first neural network, such as merging the current first feature and the second feature and inputting them into the first neural network to obtain the high-dimensional rotation matrix between the current first feature and the second feature output by the first neural network. The disclosed embodiment does not limit the implementation method of the first neural network, such as an MLP (Multi-Layer Perceptron) network. By obtaining the high-dimensional rotation matrix between the current first feature and the second feature, the transformation relationship between the current first feature and the second feature can be more accurately characterized. The main reason is that compared with conventional three-dimensional angle parameters, the high-dimensional rotation matrix has more parameters, so it can more finely express the transformation relationship between the current first feature and the second feature, which helps to improve the accuracy of line of sight estimation later.

[0038] Step B: performing a rotation operation on the current first feature based on the high-dimensional rotation matrix to obtain a rotated first feature.

[0039] For example, a specific method of performing the rotation operation may be: multiplying the current first feature by a high-dimensional rotation matrix, thereby obtaining the rotated first feature based on the product result of the current first feature and the high-dimensional rotation matrix.

[0040] Further, the rotation process also includes step C: mapping the high-dimensional rotation matrix to a three-dimensional space to obtain a three-dimensional rotation matrix; wherein the three-dimensional rotation matrix is ​​used to characterize the transformation relationship between the line of sight direction corresponding to the current first feature and the line of sight direction corresponding to the second feature in the three-dimensional space, which can be specifically understood as characterizing the difference angles corresponding to the X-axis, Y-axis and Z-axis of the above two line of sight directions in the three-dimensional space. In this way, the difference between the line of sight direction of the first feature after rotation and the line of sight direction of the second feature can be objectively presented, so that it can be objectively and accurately evaluated whether the line of sight direction of the first feature after rotation coincides with the line of sight direction of the second feature. Moreover, as mentioned above, the high-dimensional rotation matrix can express the transformation relationship between the current first feature and the second feature more finely based on multi-dimensional parameters, so that the three-dimensional rotation matrix characterizing the difference in line of sight direction obtained based on the high-dimensional rotation matrix is ​​also more accurate. Compared with directly predicting the line of sight direction transformation relationship between the first feature and the second feature in the three-dimensional space, the method of first predicting the high-dimensional rotation matrix and then converting it into a three-dimensional rotation matrix, the final three-dimensional rotation matrix has a higher accuracy. Especially when the sight direction corresponding to the first feature is too different from the sight direction corresponding to the second feature, if the three-dimensional rotation matrix corresponding to the two is directly predicted, the accuracy is usually not high. However, the method of first predicting the high-dimensional rotation matrix and then converting it into a three-dimensional rotation matrix provided by the embodiment of the present disclosure can significantly improve the accuracy of the final three-dimensional rotation matrix.

[0041] In practical applications, the high-dimensional rotation matrix can be mapped to a three-dimensional space through a preset second neural network, and a three-dimensional rotation matrix can be output. In other words, the second neural network can be converted into a low-dimensional rotation matrix based on the high-dimensional rotation matrix. The embodiment of the present disclosure does not limit the implementation method of the second neural network, such as an MLP network.

[0042] On the basis of the aforementioned step C, the rotation information includes a three-dimensional rotation matrix; in a specific implementation example of determining the sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to at least one rotation process, the three-dimensional rotation matrix corresponding to at least one rotation process can be multiplied to determine the sight direction transformation relationship between the target image feature and the calibration image feature based on the product result. Specifically, the three-dimensional rotation matrix corresponding to each rotation process is multiplied, which is equivalent to superimposing each rotation angle, and the final product result is also the superposition result of each rotation angle, and the angle superposition result can be used to characterize the sight direction transformation relationship between the target image feature and the calibration image feature.

[0043] On the basis of the aforementioned step C, at least one rotation process is performed based on the first feature until the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature, including: performing at least one rotation process based on the first feature until the obtained three-dimensional rotation matrix is ​​a three-dimensional unit matrix, determining that the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature, and stopping the rotation process. That is, the difference between the sight direction of the rotated first feature and the sight direction of the second feature can be measured by the three-dimensional rotation matrix finally obtained by each rotation process, until the three-dimensional rotation matrix becomes a unit matrix, which means that the sight direction of the rotated first feature coincides with the sight direction of the second feature.

[0044] To facilitate the understanding of the three-dimensional rotation matrix, we can first refer to Figure 2 The angle diagram shown in the figure shows that the X-axis, Y-axis and Z-axis are perpendicular to each other in the three-dimensional space, which can convert the eye's sight direction into the pitch angle (which can be represented by β) rotating around the X-axis, the head angle (which can be represented by α) rotating around the Y-axis, and the roll angle (which can be represented by γ) rotating around the Z-axis. On this basis, the three-dimensional rotation matrix R (α, β, γ) can be expressed by referring to the following formula:

[0045]

[0046] Among them, cosα=c1, sinα=s1, cosβ=c2, sinβ=s2, cosγ=c3, sinγ=s3. In practical applications, the sight direction can be mainly determined by the α angle corresponding to the Y axis (which can be simply understood as the left and right direction) and the β angle corresponding to the X axis (which can be simply understood as the up and down direction), and the γ angle corresponding to the Z axis can be a preset value, such as 0.

[0047] To facilitate the understanding of the aforementioned line of sight estimation method, please refer to Figure 3 The flowchart of a line of sight estimation method shown in FIG. 1 mainly includes the following steps S302 to S318:

[0048] Step S302, obtaining a target eye image of a sight line direction to be estimated and a preset calibration eye image, and taking the sight line direction corresponding to the calibration eye image as the preset sight line direction.

[0049] Step S304, feature extraction is performed on the target eye image and the calibration eye image respectively to obtain target image features corresponding to the target eye image and calibration image features corresponding to the calibration eye image. The sight line direction corresponding to the calibration eye image is the aforementioned preset sight line direction, and the calibration image features corresponding to the calibration eye image are the calibration image features corresponding to the preset sight line direction.

[0050] Step S306 , selecting one feature from the target image feature and the calibration image feature as an initial first feature and the other feature as a fixed second feature.

[0051] Step S308, rotating the current first feature so that the rotated first feature is close to the second feature, and obtaining a three-dimensional rotation matrix corresponding to the rotation processing. The rotation processing step includes:

[0052] Step S3080, obtaining a high-dimensional rotation matrix between the current first feature and the second feature.

[0053] Step S3082: Perform a rotation operation on the current first feature based on the high-dimensional rotation matrix to obtain a rotated first feature.

[0054] Step S3084, mapping the high-dimensional rotation matrix to a three-dimensional space to obtain a three-dimensional rotation matrix.

[0055] Step S310, determining whether the three-dimensional rotation matrix obtained by the current rotation processing is a three-dimensional unit matrix; if not, executing step S312, if yes, executing step S314.

[0056] Step S312, update the current first feature based on the rotated first feature, and return to step S308. That is, the current first feature corresponding to the first rotation process is the initial first feature, and the current first feature corresponding to the non-first rotation process is the rotated first feature obtained by the last rotation process.

[0057] Step S314, performing product processing on the three-dimensional rotation matrix obtained from each rotation process.

[0058] Step S316, determining the sight direction transformation relationship between the target image feature and the calibration image feature based on the product result. The product result obtained by multiplying the three-dimensional rotation matrix obtained by each rotation process is essentially the target rotation matrix used to finally characterize the rotation angle of the sight direction between the target image feature and the calibration image feature. Based on the product result, the sight direction transformation relationship between the target image feature and the calibration image feature can be directly obtained.

[0059] Step S318, estimating the gaze direction corresponding to the target eye image based on the gaze direction transformation relationship and the preset gaze direction corresponding to the calibration eye image.

[0060] Through the above method, a cyclic adjustment method of one or more rotations can be used to make the sight direction corresponding to one of the target image features and the calibration image features approach the sight direction corresponding to the other feature until the two sight directions are consistent. Based on this, the three-dimensional rotation matrix obtained from each rotation process is multiplied, that is, the rotation angles are superimposed, and finally a more accurate sight direction change relationship between the sight directions corresponding to the target image feature and the calibration image feature can be obtained, thereby fully ensuring the accuracy of the sight direction corresponding to the target eye image.

[0061] In the aforementioned Figure 3 On the basis of Figure 4 A schematic diagram of the line of sight estimation process is shown in FIG. Figure 4In the example, the feature extracted by the feature extraction network of the target eye image is used as the first feature, and the feature extracted by the feature extraction network of the calibration eye image is used as the second feature. In practical applications, the feature extracted by the feature extraction network of the target eye image can also be used as the second feature, and the feature extracted by the feature extraction network of the calibration eye image can be used as the first feature, which is not limited here. The first feature and the second feature can be combined and input into the first neural network in a splicing manner. The high-dimensional rotation matrix between the first feature and the second feature is predicted by the first neural network, and its size is illustrated by taking 128*128 as an example. The high-dimensional rotation matrix can be converted into a three-dimensional rotation matrix (that is, the size is 3*3) by the second neural network. In addition, if the three-dimensional rotation matrix is ​​not a unit matrix, it means that the sight direction corresponding to the first feature after the rotation operation based on the high-dimensional rotation matrix is ​​still different from the sight direction corresponding to the second feature. Therefore, the rotated first feature can be further rotated again to perform cyclic fine-tuning on the first feature. The rotated first feature can be represented by the product of the high-dimensional rotation matrix and the first feature before rotation, and the rotated first feature is used as the updated first feature to further calculate the updated first feature ( Figure 4 The high-dimensional rotation matrix between the first feature and the second feature (indicated by the gray box in the figure) and the original fixed second feature, and so on, each rotation process corresponds to a three-dimensional rotation matrix between the first feature and the second feature after the rotation. Assuming that the three-dimensional rotation matrix obtained by the Nth rotation process is the unit matrix, the three-dimensional rotation matrices corresponding to the first, second, and Nth times are multiplied, and the output rotation matrix obtained is the cumulative result of N rotation angles, which can be used to accurately characterize the sight direction transformation relationship between the target eye image and the calibration eye image, so as to obtain an accurate sight evaluation result of the target eye image when the sight direction of the calibration eye image itself is known. It should be noted that Figure 4It is only a schematic illustration for ease of understanding, and is only for intuitively showing the process of using the network to perform multiple processing operations. In actual applications, only one feature extraction network, a first neural network and a second neural network can be set up, and the high-dimensional rotation matrix output by the first neural network is used to update the first feature previously input to the first neural network. The updated first feature is merged with the fixed second feature and then input to the first neural network to obtain a new high-dimensional rotation matrix and a new three-dimensional rotation matrix is ​​obtained using the second neural network. The above neural networks can be used repeatedly without setting up multiple repeated networks. In actual applications, the above-mentioned feature extraction network, the first neural network and the second neural network can be combined and trained to prompt the first neural network to generate a more accurate high-dimensional rotation matrix, and to prompt the second neural network to generate a more accurate three-dimensional rotation matrix; in addition, the existing feature extraction network can also be used, and only the first neural network and the second neural network can be combined and trained. The specific settings can be flexible and are not limited here.

[0062] In summary, the line of sight estimation method provided by the embodiment of the present invention adopts one or more rotation processing methods to make the line of sight direction of one feature gradually approach the other line of sight direction until the two line of sight directions are consistent. The line of sight direction transformation relationship obtained based on the rotation information corresponding to each of the at least one rotation processing is more accurate, thereby further ensuring the accuracy of the line of sight estimation result corresponding to the target eye image.

[0063] Corresponding to the aforementioned sight line estimation method, the embodiment of the present disclosure provides a sight line estimation device, Figure 5 The structure diagram of a sight line estimation device provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be implemented by software and / or hardware, and can generally be integrated in an electronic device, and can be used to execute a sight line estimation method. Figure 5 As shown, the sight line estimation device includes:

[0064] A feature acquisition module 502 is used to acquire target image features corresponding to a target eye image of a to-be-estimated sight line direction, and to acquire calibration image features corresponding to a preset sight line direction;

[0065] A rotation processing module 504 is used to select one feature from the target image feature and the calibration image feature as an initial first feature and another feature as a fixed second feature, and perform at least one rotation processing based on the first feature until the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature;

[0066] A relationship determination module 506 is used to obtain rotation information corresponding to each rotation process, so as to determine the sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to at least one rotation process;

[0067] The sight line estimation module 508 is used to estimate the sight line direction corresponding to the target eye image based on the sight line direction transformation relationship and the preset sight line direction corresponding to the calibration image feature.

[0068] Through one or more rotation processes, the sight direction of one feature is gradually approached to the other sight direction until the two sight directions are consistent. The sight direction transformation relationship obtained based on the rotation information corresponding to at least one rotation process is more accurate, thereby further ensuring the accuracy of the sight estimation result corresponding to the target eye image.

[0069] In some embodiments, the feature acquisition module 502 is specifically used to: obtain a preset calibration eye image, take the sight direction corresponding to the calibration eye image as the preset sight direction, and extract the calibration image features corresponding to the calibration eye image; or, obtain multiple eye images with preset sight directions, extract the image features corresponding to each of the multiple eye images, and fuse the image features corresponding to each of the multiple eye images to obtain the calibration image features corresponding to the preset sight direction.

[0070] In some embodiments, the rotation processing includes the following steps: obtaining a high-dimensional rotation matrix between the current first feature and the second feature; wherein the current first feature corresponding to the first rotation processing is the initial first feature, and the current first feature corresponding to the non-first rotation processing is the first feature after the last rotation processing; the high-dimensional rotation matrix is ​​used to characterize the transformation relationship between the line of sight direction corresponding to the current first feature and the line of sight direction corresponding to the second feature in a preset high-dimensional space, and the dimension of the high-dimensional rotation matrix is ​​consistent with the dimension of the target image feature and the dimension of the calibration image feature; performing a rotation operation on the current first feature based on the high-dimensional rotation matrix to obtain the rotated first feature.

[0071] In some embodiments, the rotation processing also includes: mapping the high-dimensional rotation matrix to a three-dimensional space to obtain a three-dimensional rotation matrix; wherein the three-dimensional rotation matrix is ​​used to characterize the transformation relationship between the line of sight direction corresponding to the current first feature and the line of sight direction corresponding to the second feature in the three-dimensional space.

[0072] In some embodiments, the rotation information includes the three-dimensional rotation matrix; the relationship determination module 506 is specifically used to: multiply the three-dimensional rotation matrices corresponding to the at least one rotation processing to determine the line of sight direction transformation relationship between the target image feature and the calibration image feature based on the product result.

[0073] In some embodiments, the rotation processing module 504 is specifically used to: perform at least one rotation processing based on the first feature, until the obtained three-dimensional rotation matrix is ​​a three-dimensional unit matrix, determine that the line of sight direction corresponding to the rotated first feature is consistent with the line of sight direction corresponding to the second feature, and stop the rotation processing.

[0074] In some implementations, the rotation processing module 504 is specifically configured to randomly select a feature from the target image feature and the calibration image feature as an initial first feature.

[0075] The line of sight estimation device provided in the embodiments of the present disclosure can execute the line of sight estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0077] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0078] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0081] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0082] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0083] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the line of sight estimation method provided by the embodiment of the present disclosure.

[0084] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0085] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the line of sight estimation method in the embodiments of the present disclosure when executed by a processor.

[0086] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0087] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, such as prompting the user to collect an eye image. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message. For example, the embodiment of the present disclosure will obtain the user's eye image and estimate the line of sight direction only under the premise of user authorization.

[0088] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0089] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0090] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0091] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight estimation method, characterized in that: include: Obtaining target image features corresponding to a target eye image of a to-be-estimated sight line direction, and obtaining calibration image features corresponding to a preset sight line direction; Selecting one feature from the target image feature and the calibration image feature as an initial first feature and another feature as a fixed second feature, and performing at least one rotation process based on the first feature until a sight direction corresponding to the rotated first feature is consistent with a sight direction corresponding to the second feature; Obtaining rotation information corresponding to each rotation process, so as to determine a sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to each of the at least one rotation processes; The gaze direction corresponding to the target eye image is estimated based on the gaze direction transformation relationship and the preset gaze direction corresponding to the calibration image feature.

2. The method according to claim 1, characterized in that The obtaining of calibration image features corresponding to a preset sight line direction includes: Acquire a preset calibration eye image, use the sight line direction corresponding to the calibration eye image as the preset sight line direction, and extract calibration image features corresponding to the calibration eye image; or, A plurality of eye images with preset sight lines are obtained, and image features corresponding to each of the plurality of eye images are extracted, and the image features corresponding to each of the plurality of eye images are fused to obtain calibration image features corresponding to the preset sight lines.

3. The method according to claim 1, characterized in that The rotation process comprises the following steps: Obtain a high-dimensional rotation matrix between the current first feature and the second feature; wherein the current first feature corresponding to the first rotation process is the initial first feature, and the current first feature corresponding to the non-first rotation process is the first feature after the last rotation process; the high-dimensional rotation matrix is ​​used to characterize the transformation relationship between the current first feature and the second feature in a preset high-dimensional space, and the dimension of the high-dimensional rotation matrix is ​​consistent with the dimension of the target image feature and the dimension of the calibration image feature; A rotation operation is performed on the current first feature based on the high-dimensional rotation matrix to obtain a rotated first feature.

4. The method according to claim 3, characterized in that The rotation process further includes: The high-dimensional rotation matrix is ​​mapped to a three-dimensional space to obtain a three-dimensional rotation matrix; wherein the three-dimensional rotation matrix is ​​used to characterize the transformation relationship between the line of sight direction corresponding to the current first feature and the line of sight direction corresponding to the second feature in the three-dimensional space.

5. The method according to claim 4, characterized in that The rotation information includes the three-dimensional rotation matrix; and determining the sight direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to each of the at least one rotation processing includes: The three-dimensional rotation matrices corresponding to the at least one rotation process are multiplied to determine the sight direction transformation relationship between the target image feature and the calibration image feature based on the product result.

6. The method according to claim 4, characterized in that The performing at least one rotation process based on the first feature until the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature, includes: Based on the first feature, at least one rotation process is performed until the obtained three-dimensional rotation matrix is ​​a three-dimensional unit matrix, and it is determined that the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature, and the rotation process is stopped.

7. The method according to claim 1, characterized in that The step of selecting a feature from the target image feature and the calibration image feature as an initial first feature comprises: A feature is randomly selected from the target image feature and the calibration image feature as an initial first feature.

8. A sight line estimation device, characterized in that: include: A feature acquisition module, used to acquire target image features corresponding to a target eye image of a to-be-estimated sight line direction, and to acquire calibration image features corresponding to a preset sight line direction; A rotation processing module, used to select one feature from the target image feature and the calibration image feature as an initial first feature and another feature as a fixed second feature, and perform at least one rotation processing based on the first feature until the sight direction corresponding to the rotated first feature is consistent with the sight direction corresponding to the second feature; a relationship determination module, configured to obtain rotation information corresponding to each rotation process, so as to determine a sight line direction transformation relationship between the target image feature and the calibration image feature based on the rotation information corresponding to each of the at least one rotation processes; A sight line estimation module is used to estimate the sight line direction corresponding to the target eye image based on the sight line direction transformation relationship and the preset sight line direction corresponding to the calibration image feature.

9. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the line of sight estimation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the line of sight estimation method described in any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the line of sight estimation method according to any one of claims 1 to 7.

Images