Self-calibration method, device and equipment of human eye sight deviation parameter and medium

Abstract

This application relates to the field of human eye gaze estimation technology, and provides a self-calibration method, apparatus, device, and medium for human eye gaze deviation parameters. Since the final gaze predicted in this application's embodiments is based on a gaze unrelated to the user's identity (i.e., the initial gaze) and a gaze related to the user's identity (i.e., the human eye gaze deviation parameters), and these parameters can be continuously optimized during user use, this application's embodiments can improve the accuracy of gaze estimation. Furthermore, since this application's embodiments perform gradient backpropagation based on intersection point error loss, i.e., using the geometric constraints of the gaze as a supervision signal, this application's embodiments do not require separate collection of the user's gaze label samples before the user uses the gaze estimation system. Instead, the human eye gaze deviation parameters can be continuously learned during user use, and as the number of samples increases, the human eye gaze deviation parameters will become increasingly accurate.

Description

Technical Field

[0001] This application relates to the field of human eye line estimation technology, and in particular to a self-calibration method, apparatus, equipment and medium for human eye line deviation parameters. Background Technology

[0002] The goal of 3D gaze estimation is to infer the direction of a person's gaze from an eye image or face image. Typically, the gaze direction can be represented using two angles, such as... Figure 1 As shown, the pitch angle and yaw angle are respectively, which can be used to estimate the gaze point of the human eye.

[0003] Currently, methods for estimating 3D gaze can be divided into two types. One type is based on the eye model, called the model-based method. This method mainly involves building an eye model and solving for the two angles of gaze using pupil features and geometric relationships. The limitation of this type of method is that a simplified eye model is usually used when building the eye model. These simplified assumptions may be difficult to meet in actual use cases, and high-resolution images of the eye are required. However, the images obtained in most use cases have low resolution.

[0004] Another approach is image-based, called appearance-based methods. These methods primarily use neural networks to extract features from face or eye images to predict pitch and yaw angles. This approach has lower requirements for the camera and is widely applicable. However, its limitation lies in the assumption that gaze is independent of identity, which results in the accuracy of appearance-based methods being only around 3°-4°.

[0005] Figure 2 This is a simplified diagram of an eyeball model, where v represents the line of sight, o represents the line connecting the center of the pupil and the center of the eyeball. Fovea is the point on the retina with the highest light sensitivity, and N represents a node at a distance d from the center of the eyeball. Intuitively, the line of sight v should coincide with o, but in reality, there is a fixed angle K between v and o, and the size of this angle varies from person to person, determined by internal parameters of the human eye, and cannot be learned from images.

[0006] To further improve the accuracy of appearance-based methods in estimating line of sight, personalized calibration is usually required for each user. Currently, most personalized estimation methods require a certain number of personalized calibration samples, which means that users need to cooperate in collecting certain samples before using the line of sight estimation system, which degrades the user experience. Summary of the Invention

[0007] To address the aforementioned technical problems, the purpose of this application is to provide a self-calibration method, apparatus, and device storage medium for human eye gaze deviation parameters. This aims to solve the technical problem that users need to cooperate in collecting certain samples before using the gaze estimation system, which impairs the user experience, and to improve the accuracy of gaze estimation.

[0008] In a first aspect, embodiments of this application provide a self-calibration method for human eye line deviation parameters, including:

[0009] Obtain N images of the user's left and right eyes when the user is looking at a point on the screen; where N is greater than or equal to 1.

[0010] The left-eye image is input into a pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left-eye image;

[0011] Obtain the left eye gaze deviation parameter corresponding to the user and the head pose corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector;

[0012] The left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image are stitched together to obtain the stitched feature corresponding to the left eye image.

[0013] The stitched features corresponding to the left eye image are input into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image.

[0014] Calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image;

[0015] The right eye image corresponding to the left eye image is input into a pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right eye image;

[0016] Obtain the right eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector;

[0017] The right eye gaze deviation parameter and the initial gaze corresponding to the right eye image are stitched together to obtain the stitching feature corresponding to the right eye image.

[0018] The stitched features corresponding to the right eye image are input into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image.

[0019] Calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image;

[0020] The intersection point error loss is calculated based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image.

[0021] Gradient backpropagation is performed based on the intersection point error loss to update the user's left-eye gaze deviation parameters and right-eye gaze deviation parameters.

[0022] Furthermore, the self-calibration method for human eye line deviation parameters further includes:

[0023] Obtain the first training data, which includes N left-eye sample images, gaze labels corresponding to the left-eye sample images, and M user identity information.

[0024] Obtain the second training data, which includes N right-eye sample images, gaze labels corresponding to the right-eye sample images, and M user identity information; wherein the left-eye sample images and the right-eye sample images are in one-to-one correspondence, and the left-eye sample images and the right-eye sample images are collected when the user looks at a point on the screen; wherein N and M are positive integers, and N is greater than or equal to M;

[0025] The left-eye sample image is input into the first convolutional neural network to be trained, and the convolutional layer outputs the initial gaze corresponding to the left-eye sample image;

[0026] Based on the user identity information corresponding to the left eye sample image, select the corresponding left eye gaze deviation parameter;

[0027] The left eye gaze deviation parameter corresponding to the left eye sample image, the initial gaze corresponding to the left eye sample image, and the head posture corresponding to the left eye sample image are stitched together to obtain the stitched feature corresponding to the left eye sample image.

[0028] The stitched features corresponding to the left eye sample image are input into the fully connected layer of the first convolutional neural network to be trained to obtain the final gaze corresponding to the left eye sample image.

[0029] The right eye sample image corresponding to the left eye sample image is input into the second convolutional neural network to be trained, and the initial gaze corresponding to the right eye sample image is output by the convolutional layer; wherein, the network parameters of the first convolutional neural network to be trained and the second convolutional neural network to be trained are shared;

[0030] Based on the user identity information corresponding to the right eye sample image, select the corresponding right eye gaze deviation parameter;

[0031] The right eye gaze deviation parameter corresponding to the right eye sample image, the initial gaze corresponding to the right eye sample image, and the head posture corresponding to the right eye sample image are stitched together to obtain the stitched feature corresponding to the right eye sample image.

[0032] The stitched features corresponding to the right eye sample image are input into the fully connected layer of the second convolutional neural network to be trained to obtain the final gaze corresponding to the right eye sample image.

[0033] The left-eye gaze error loss is calculated based on the final gaze corresponding to each left-eye sample image and the gaze label corresponding to each left-eye sample image.

[0034] The right-eye gaze error loss is calculated based on the final gaze corresponding to each right-eye sample image and the gaze label corresponding to each right-eye sample image.

[0035] Calculate the intersection point between the final line of sight corresponding to the left eye sample image and the screen based on the final line of sight corresponding to the left eye sample image;

[0036] Calculate the intersection point between the final line of sight corresponding to the right eye sample image and the screen based on the final line of sight corresponding to the right eye sample image;

[0037] The intersection point error loss is calculated based on the intersection point of the final line of sight with the screen for each left-eye sample image and the intersection point of the final line of sight with the screen for each right-eye sample image.

[0038] Gradient backpropagation is performed based on the left eye gaze error loss, the right eye gaze error loss, and the intersection point error loss to update the parameters of the first convolutional neural network, the parameters of the second convolutional neural network, and the left eye gaze deviation parameters and right eye gaze deviation parameters corresponding to each user, until the preset loss function converges, thus obtaining the pre-trained first convolutional neural network and the second convolutional neural network.

[0039] Furthermore, let the plane containing the screen be represented as ax + by + cz = d, where a, b, c, and d are known, the center of the eyeball is (o1, o2, o3), and the actual line-of-sight direction vector is (d1, d2, d3). Assuming the intersection point is (l, m, n), the intersection point of the line of sight and the screen is determined by the formula...

[0040]

[0041] Calculated.

[0042] Furthermore, the intersection point error loss is calculated according to the formula... The calculation yields the result; where N represents the total number of samples, and i represents the distinct samples. This represents the intersection of the final line of sight corresponding to the left-eye image i and the screen; This represents the intersection of the final line of sight corresponding to the right eye image i and the screen.

[0043] Furthermore, the line-of-sight error loss is calculated according to the following formula:

[0044]

[0045] Where N represents the total number of samples, v represents the final line of sight, and g represents the line of sight label.

[0046] Furthermore, the preset loss function is:

[0047] L = L rightgaze +L rightgaze +λL POG

[0048] Among them, L leftgaze L represents the visual error loss in the left eye. rightgaze L represents the visual error loss of the right eye. POG To represent the intersection point loss, λ is a parameter for adjusting the loss weight.

[0049] Furthermore, the step of calculating the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image specifically includes:

[0050] The validity of the left-eye image is determined based on the intersection of the final line of sight corresponding to the left-eye image and the screen.

[0051] The validity of the right-eye image is determined based on the intersection of the final line of sight corresponding to the right-eye image and the screen; wherein the right-eye image corresponds to the left-eye image;

[0052] If both the left-eye image and the right-eye image are valid, then the intersection point error loss is calculated based on the intersection point of the final line of sight with the screen corresponding to each valid left-eye image and the intersection point of the final line of sight with the screen corresponding to each valid right-eye image.

[0053] Furthermore, the validity of a single-eye image is determined as follows:

[0054] The single-eye image is a valid image if the intersection of the final line of sight corresponding to the single eye image and the screen satisfies any of the following conditions;

[0055] The intersection of the final line of sight corresponding to the single eye image and the screen is within the screen area;

[0056] The intersection of the final line of sight corresponding to the single eye image and the screen is within P times the standard deviation of the historical prediction mean; where P is a positive number.

[0057] Furthermore, the human eye line deviation parameter is a three-dimensional vector or a two-dimensional vector.

[0058] Secondly, embodiments of this application provide a self-calibration device for human eye line deviation parameters, comprising:

[0059] The first acquisition unit is used to acquire N images of the user's left eye and right eye when the user is looking at a point on the screen; where N is greater than or equal to 1.

[0060] The first initial gaze prediction unit is used to input the left eye image into a pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left eye image;

[0061] The second acquisition unit is used to acquire the left eye gaze deviation parameter corresponding to the user and the head posture corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector;

[0062] The first stitching unit is used to stitch together the left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image to obtain the stitching feature corresponding to the left eye image.

[0063] The first final gaze prediction unit is used to input the stitched features corresponding to the left eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image.

[0064] The first intersection point calculation unit is used to calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image.

[0065] The second initial gaze prediction unit is used to input the right eye image corresponding to the left eye image into the pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right eye image;

[0066] The third acquisition unit is used to acquire the right eye gaze deviation parameter corresponding to the user and the head posture corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector;

[0067] The second stitching unit is used to stitch together the right eye gaze deviation parameter, the initial gaze corresponding to the right eye image, and the head posture corresponding to the right eye image to obtain the stitching feature corresponding to the right eye image.

[0068] The second final gaze prediction unit is used to input the stitched features corresponding to the right eye image into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image.

[0069] The second intersection point calculation unit is used to calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image.

[0070] The intersection point error loss unit is used to calculate the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image.

[0071] The gradient backpropagation and update unit is used to perform gradient backpropagation based on the intersection point error loss and update the user's left-eye gaze deviation parameters and right-eye gaze deviation parameters.

[0072] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0073] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0074] This application provides a self-calibration method for human eye gaze deviation parameters, including: acquiring N left-eye and right-eye images when a user looks at a point on the screen; wherein N is greater than or equal to 1; inputting the left-eye images into a pre-trained first convolutional neural network, and having the convolutional layer output the initial gaze corresponding to the left-eye image; acquiring the left-eye gaze deviation parameters corresponding to the user and the head pose corresponding to the left-eye image; wherein the initial left-eye gaze deviation parameters are a 0 vector; concatenating the left-eye gaze deviation parameters, the initial gaze corresponding to the left-eye image, and the head pose corresponding to the left-eye image to obtain the concatenated features corresponding to the left-eye image; inputting the concatenated features corresponding to the left-eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left-eye image; calculating the intersection point between the final gaze corresponding to the left-eye image and the screen based on the final gaze corresponding to the left-eye image; inputting the right-eye image corresponding to the left-eye image into the pre-trained first convolutional neural network... In the second convolutional neural network, the initial gaze corresponding to the right-eye image is output by the convolutional layer; the right-eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right-eye image are obtained; wherein, the initial right-eye gaze deviation parameter is a 0 vector; the right-eye gaze deviation parameter, the initial gaze corresponding to the right-eye image, and the head pose corresponding to the right-eye image are concatenated to obtain the concatenated feature corresponding to the right-eye image; the concatenated feature corresponding to the right-eye image is input into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right-eye image; the intersection point of the final gaze corresponding to the right-eye image and the screen is calculated based on the final gaze corresponding to the right-eye image; the intersection point error loss is calculated based on the intersection point of the final gaze corresponding to each left-eye image and the screen, and the intersection point of the final gaze corresponding to each right-eye image and the screen; gradient backpropagation is performed based on the intersection point error loss to update the user's left-eye gaze deviation parameter and right-eye gaze deviation parameter. Since the final gaze predicted in this embodiment is based on a gaze independent of the user's identity (i.e., the initial gaze) and a gaze related to the user's identity (i.e., the human eye gaze deviation parameter), and the human eye gaze deviation parameter can be continuously optimized during user use, this embodiment can improve the accuracy of gaze estimation. Furthermore, since this embodiment performs gradient backpropagation based on intersection point error loss, i.e., uses the geometric constraints of the gaze as a supervision signal, this embodiment does not require separately collecting gaze label samples from the user before using the gaze estimation system. Instead, it can continuously learn the human eye gaze deviation parameter as the user uses the system; as the number of samples increases, the human eye gaze deviation parameter will become increasingly accurate. Attached Figure Description

[0075] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0076] Figure 1 This is a diagram of the pitch angle and yaw angle;

[0077] Figure 2 This is a schematic diagram of an eyeball model;

[0078] Figure 3 This is a flowchart of a self-calibration method for human eye line deviation parameters provided in the first embodiment of this application;

[0079] Figure 4 It is a simplified structural diagram of an eye;

[0080] Figure 5 This is a simplified flowchart of the training process of the first pre-trained convolutional neural network and the second pre-trained convolutional neural network provided in the first embodiment of this application;

[0081] Figure 6 This is a schematic diagram of a self-calibration device for human eye line deviation parameters provided in the second embodiment of this application;

[0082] Figure 7 A schematic block diagram of the structure of a computer device provided in the third embodiment of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] Example 1:

[0085] Please see Figure 3 This application provides a self-calibration method for human eye line deviation parameters, including:

[0086] S1. Obtain N images of the user's left eye and right eye when the user is looking at a point on the screen; where N is greater than or equal to 1.

[0087] S2. Input the left-eye image into the pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left-eye image;

[0088] S3. Obtain the left eye gaze deviation parameter corresponding to the user and the head pose corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector;

[0089] S4. The left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image are spliced ​​together to obtain the spliced ​​feature corresponding to the left eye image.

[0090] S5. Input the stitched features corresponding to the left eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image;

[0091] S6. Calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image;

[0092] S7. Input the right eye image corresponding to the left eye image into the pre-trained second convolutional neural network, and output the initial gaze corresponding to the right eye image from the convolutional layer;

[0093] S8. Obtain the right eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector;

[0094] S9. The right eye gaze deviation parameter, the initial gaze corresponding to the right eye image, and the head posture corresponding to the right eye image are spliced ​​together to obtain the splicing feature corresponding to the right eye image.

[0095] S10. Input the stitched features corresponding to the right eye image into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image;

[0096] S11. Calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image;

[0097] S12. Calculate the intersection point error loss based on the intersection point of the final line of sight corresponding to each left-eye image and the screen, and the intersection point of the final line of sight corresponding to each right-eye image and the screen.

[0098] S13. Perform gradient backpropagation based on the intersection point error loss to update the user's left eye gaze deviation parameter and right eye gaze deviation parameter.

[0099] For ease of understanding, the principles of the embodiments of this application will be explained below:

[0100] Figure 4This is a simplified diagram of the eye's structure. d is the center point of the eyeball, c is the center point of the cornea, p is the center of the pupil, and the optical axis is the line connecting the center of the eyeball and the center of the pupil, passing through the center of the cornea. The visual axis is the line connecting the fovea point and the center point of the cornea. Angles α and β are the components of the angle between the visual axis and the horizontal line in two directions, respectively.

[0101] The gaze direction obtained by person-independent deep learning methods can be considered as the direction of the optical axis. To obtain the actual gaze direction (visual axis), it is necessary to obtain the α and β angles for each person.

[0102] To estimate the angle between the line of sight and the horizontal line, we first need to define two right-handed coordinate systems: the World Coordinate System (WCS) and the Eye Coordinate System (ECS). In the Eye Coordinate System, the Z-axis coincides with the optical axis, the X-axis points horizontally, and the Y-axis points vertically. Therefore, in the Eye Coordinate System (ECS), the actual line of sight can be represented as...

[0103]

[0104] From the above, we can see that when using angles α and β to represent the actual line of sight v... ECS When viewing the image, there is a three-dimensional vector, but this is in the eye coordinate system. The network in this embodiment is in the camera coordinate system (i.e., the world coordinate system). There is a transformation relationship between them. The actual line of sight in the world coordinate system can be expressed as v(α,β)=Rv ECS (α,β), where R is the rotation matrix from the eye coordinate system to the world coordinate system, which can be calculated from the head pose and the estimated initial gaze.

[0105] Since unsupervised gaze calibration processes cannot obtain calibration samples with gaze labels, gaze cannot be used as supervision when learning human eye gaze deviation parameters. Therefore, we use the geometric constraint of gaze as the supervision signal, meaning that when a user gazes at a point on the screen, the intersection of the actual gaze lines of both eyes with the screen is the same point. This constraint is described as follows:

[0106] First, in the world coordinate system, we represent the plane containing the screen as ax + by + cz = d, where a, b, c, and d are known, d represents the intercept, and (a, b, c) represents the normal to the plane. Determining the four parameters a, b, c, and d defines a plane. The center of the eyeball is (o1, o2, o3), and the actual line of sight is (d1, d2, d3). We can represent the line containing the line of sight as (x... g ,y g ,z gThe expression is: ) = (o1 + d1*t, o2 + d2*t, o3 + d3*t), where t is a parameter in the linear expression. When t takes a certain value, a line segment can be determined using (o1, o2, o3) and the direction vector (d1, d2, d3). When t takes all values, a straight line is formed. Assuming the intersection point is (l, m, n), the following formula holds:

[0107] al+bm+cn=d

[0108] o1+d1*t0=l

[0109] o2+d2*t0=m

[0110] o3+d3*t0=n

[0111] Where t0 determines the line segment between the intersection point (l,m,n) and the point (o1,o2,o3);

[0112] Combining the above formulas, it can be expressed as:

[0113]

[0114] The intersection point can then be obtained by solving for it, and we use ψ to represent the obtained intersection point.

[0115] Let A and B represent the points of gaze (POG) between the two eyes and the screen, respectively. When the user is looking at a point on the screen, the two points of gaze (POG) should coincide. Therefore, our constraint can be written as:

[0116]

[0117] Where N represents the total number of samples, and i represents the distinct samples. This indicates the point where the left eye's final line of sight intersects with the screen; This indicates the point where the right eye's final line of sight intersects with the screen.

[0118] Since the calculated rotation matrix is ​​often noisy in real-world scenarios, leading to inaccurate POG calculations, we opted to let the network learn the relationship between the human eye gaze deviation parameter and head and eye poses. We concatenated the human eye gaze deviation parameter with the initial gaze and head pose features, connecting this to a fully connected layer (the fully connected layers of the two eye branches share parameters) to output the final gaze. The human eye gaze deviation parameter is jointly learned and optimized with the network parameters; therefore, we directly set a three-dimensional vector, the human eye gaze deviation parameter, and let the network learn it automatically. It should be noted that the head pose is calculated by detecting facial landmarks.

[0119] Since the final gaze predicted in this embodiment is based on a gaze independent of the user's identity (i.e., the initial gaze) and a gaze related to the user's identity (i.e., the human eye gaze deviation parameter), and the human eye gaze deviation parameter can be continuously optimized during user use, this embodiment can improve the accuracy of gaze estimation. Furthermore, since this embodiment performs gradient backpropagation based on intersection point error loss, i.e., uses the geometric constraints of the gaze as a supervision signal, this embodiment does not require separately collecting gaze label samples from the user before using the gaze estimation system. Instead, it can continuously learn the human eye gaze deviation parameter as the user uses the system; as the number of samples increases, the human eye gaze deviation parameter will become increasingly accurate.

[0120] Please see Figure 5 In one embodiment, the self-calibration method for human eye line deviation parameters further includes:

[0121] Obtain the first training data, which includes N left-eye sample images, gaze labels corresponding to the left-eye sample images, and M user identity information.

[0122] Obtain the second training data, which includes N right-eye sample images, gaze labels corresponding to the right-eye sample images, and M user identity information; wherein the left-eye sample images and the right-eye sample images are in one-to-one correspondence, and the left-eye sample images and the right-eye sample images are collected when the user looks at a point on the screen; wherein N and M are positive integers, and N is greater than or equal to M;

[0123] The left-eye sample image is input into the first convolutional neural network to be trained, and the convolutional layer outputs the initial gaze corresponding to the left-eye sample image;

[0124] Based on the user identity information corresponding to the left eye sample image, select the corresponding left eye gaze deviation parameter;

[0125] The left eye gaze deviation parameter corresponding to the left eye sample image, the initial gaze corresponding to the left eye sample image, and the head posture corresponding to the left eye sample image are stitched together to obtain the stitched feature corresponding to the left eye sample image.

[0126] The stitched features corresponding to the left eye sample image are input into the fully connected layer of the first convolutional neural network to be trained to obtain the final gaze corresponding to the left eye sample image.

[0127] The right eye sample image corresponding to the left eye sample image is input into the second convolutional neural network to be trained, and the initial gaze corresponding to the right eye sample image is output by the convolutional layer; wherein, the network parameters of the first convolutional neural network to be trained and the second convolutional neural network to be trained are shared;

[0128] Based on the user identity information corresponding to the right eye sample image, select the corresponding right eye gaze deviation parameter;

[0129] The right eye gaze deviation parameter corresponding to the right eye sample image, the initial gaze corresponding to the right eye sample image, and the head posture corresponding to the right eye sample image are stitched together to obtain the stitched feature corresponding to the right eye sample image.

[0130] The stitched features corresponding to the right eye sample image are input into the fully connected layer of the second convolutional neural network to be trained to obtain the final gaze corresponding to the right eye sample image.

[0131] The left-eye gaze error loss is calculated based on the final gaze corresponding to each left-eye sample image and the gaze label corresponding to each left-eye sample image.

[0132] The right-eye gaze error loss is calculated based on the final gaze corresponding to each right-eye sample image and the gaze label corresponding to each right-eye sample image.

[0133] Calculate the intersection point between the final line of sight corresponding to the left eye sample image and the screen based on the final line of sight corresponding to the left eye sample image;

[0134] Calculate the intersection point between the final line of sight corresponding to the right eye sample image and the screen based on the final line of sight corresponding to the right eye sample image;

[0135] The intersection point error loss is calculated based on the intersection point of the final line of sight with the screen for each left-eye sample image and the intersection point of the final line of sight with the screen for each right-eye sample image.

[0136] Gradient backpropagation is performed based on the left eye gaze error loss, the right eye gaze error loss, and the intersection point error loss to update the parameters of the first convolutional neural network, the parameters of the second convolutional neural network, and the left eye gaze deviation parameters and right eye gaze deviation parameters corresponding to each user, until the preset loss function converges, thus obtaining the pre-trained first convolutional neural network and the second convolutional neural network.

[0137] In this embodiment, it should be noted that since human eye gaze deviation is independent of image vision, we cannot learn it from the image itself. However, we can indirectly learn and estimate this deviation using other information. Because human eye gaze deviation is human-related, we incorporate user identity information, i.e., sample identity ID information, into the training process to learn the deviation. Furthermore, it should be noted that the two eyes analyzed in parallel must be captured at the same time. This can be achieved by capturing a single image including both eyes and then segmenting that image into left-eye and right-eye images, or by capturing images of each eye simultaneously.

[0138] In this embodiment, it should also be noted that the first and second convolutional neural networks are affected by left-eye viewing error, right-eye viewing error, and intersection point loss error. Similarly, the left-eye and right-eye viewing deviation parameters are also affected by left-eye viewing error, right-eye viewing error, and intersection point loss. Furthermore, since the network parameters of the first and second convolutional neural networks are shared, the trained first and second convolutional neural networks are identical. The convolutional neural networks to be trained can employ different network structures, such as the ResNet series.

[0139] It's important to note that after training the convolutional neural network (gaze estimation model), during testing, all network parameters except for the human eye gaze deviation parameter need to be fixed, and the human eye gaze deviation parameter should be initialized to a zero vector. The reason for this design is that we don't know the human eye gaze deviation parameter for the test samples. Therefore, during testing, the human eye gaze deviation parameter needs to be initialized to a zero vector. As the user uses the device, we continuously acquire images of both eyes, using POG loss to update the human eye gaze deviation parameter. The updated human eye gaze deviation parameter is then applied to the gaze estimation model, making the human eye gaze deviation parameter increasingly accurate.

[0140] In one embodiment, let the plane containing the screen be represented as ax + by + cz = d, where a, b, c, and d are known, the center of the eyeball is (o1, o2, o3), the actual gaze direction vector is (d1, d2, d3), and the intersection point is assumed to be (l, m, n). Then, the intersection point between the gaze and the screen is determined according to the formula...

[0141]

[0142] Calculated.

[0143] In one embodiment, the intersection point error loss is calculated according to the formula... The calculation yields the result; where N represents the total number of samples, and i represents the distinct samples. This represents the intersection of the final line of sight corresponding to the left-eye image i and the screen; This represents the intersection of the final line of sight corresponding to the right eye image i and the screen.

[0144] In one embodiment, the line-of-sight error loss is calculated according to the following formula:

[0145]

[0146] Where N represents the total number of samples, v represents the final line of sight, and g represents the line of sight label.

[0147] Furthermore, the preset loss function is:

[0148] L = L rightgaze +L rightgaze +λL POG

[0149] Among them, L leftgaze L represents the visual error loss in the left eye. rightgaze L represents the visual error loss of the right eye. POG This represents the intersection point loss, where λ is a parameter for adjusting the loss weights.

[0150] In one embodiment, calculating the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image specifically includes:

[0151] The validity of the left-eye image is determined based on the intersection of the final line of sight corresponding to the left-eye image and the screen.

[0152] The validity of the right-eye image is determined based on the intersection of the final line of sight corresponding to the right-eye image and the screen; wherein the right-eye image corresponds to the left-eye image;

[0153] If both the left-eye image and the right-eye image are valid, then the intersection point error loss is calculated based on the intersection point of the final line of sight with the screen corresponding to each valid left-eye image and the intersection point of the final line of sight with the screen corresponding to each valid right-eye image.

[0154] In the embodiments of this application, it should be noted that since the acquired images may be invalid, in order to reduce the impact of abnormal images on the optimization process, it is necessary to determine the validity of the two eye images before updating the human eye deviation parameters.

[0155] In one embodiment, the validity of a single-eye image is determined as follows:

[0156] The single-eye image is a valid image if the intersection of the final line of sight corresponding to the single eye image and the screen satisfies any of the following conditions;

[0157] The intersection of the final line of sight corresponding to the single eye image and the screen is within the screen area;

[0158] The intersection of the final line of sight corresponding to the single eye image and the screen is within P times the standard deviation of the historical prediction mean; where P is a positive number.

[0159] In this embodiment, it should be noted that the historical prediction mean refers to the average of the calculated intersection points. P is preferably 3; when P is selected as 3, the effect may be better compared to other values.

[0160] In one embodiment, the human eye gaze deviation parameter is a three-dimensional vector or a two-dimensional vector. When the human eye gaze deviation parameter is a two-dimensional vector, angles α and β are used as the human eye gaze deviation parameter. When the human eye gaze deviation parameter is a three-dimensional vector, some parameters related to the two angles α and β are used as the human eye gaze deviation parameter.

[0161] Example 2:

[0162] Please see Figure 6 This application provides a self-calibration device for human eye line deviation parameters, comprising:

[0163] The first acquisition unit 1 is used to acquire N images of the left eye and right eye when the user looks at a point on the screen; where N is greater than or equal to 1.

[0164] The first initial gaze prediction unit 2 is used to input the left eye image into a pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left eye image;

[0165] The second acquisition unit 3 is used to acquire the left eye gaze deviation parameter corresponding to the user and the head posture corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector;

[0166] The first stitching unit 4 is used to stitch together the left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image to obtain the stitching feature corresponding to the left eye image.

[0167] The first final gaze prediction unit 5 is used to input the spliced ​​features corresponding to the left eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image.

[0168] The first intersection point calculation unit 6 is used to calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image.

[0169] The second initial gaze prediction unit 7 is used to input the right eye image corresponding to the left eye image into the pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right eye image;

[0170] The third acquisition unit 8 is used to acquire the right eye gaze deviation parameter corresponding to the user and the head posture corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector;

[0171] The second stitching unit 9 is used to stitch together the right eye gaze deviation parameter, the initial gaze corresponding to the right eye image, and the head posture corresponding to the right eye image to obtain the stitching feature corresponding to the right eye image.

[0172] The second final gaze prediction unit 10 is used to input the stitched features corresponding to the right eye image into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image.

[0173] The second intersection point calculation unit 11 is used to calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image.

[0174] The intersection point error loss unit 12 is used to calculate the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image.

[0175] The gradient backpropagation and update unit 13 is used to perform gradient backpropagation based on the intersection point error loss and update the user's left eye gaze deviation parameter and right eye gaze deviation parameter.

[0176] Example 3:

[0177] Reference Figure 7 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 7As shown, this computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as a self-calibration method for a human eye gaze deviation parameter. The network interface is used for communication with external terminals via a network connection. When executed by a processor, this computer program implements a self-calibration method for human eye gaze deviation parameters, including: acquiring N left-eye and right-eye images of a user looking at a point on the screen; wherein N is greater than or equal to 1; inputting the left-eye images into a pre-trained first convolutional neural network, with the convolutional layer outputting the initial gaze corresponding to the left-eye image; acquiring the left-eye gaze deviation parameters corresponding to the user and the head pose corresponding to the left-eye image; wherein the initial left-eye gaze deviation parameters are a zero vector; concatenating the left-eye gaze deviation parameters, the initial gaze corresponding to the left-eye image, and the head pose corresponding to the left-eye image to obtain the concatenated features corresponding to the left-eye image; inputting the concatenated features corresponding to the left-eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left-eye image; calculating the intersection point between the final gaze corresponding to the left-eye image and the screen based on the final gaze corresponding to the left-eye image; and inputting the right-eye image corresponding to the left-eye image into the pre-trained... In the optimized second convolutional neural network, the initial gaze corresponding to the right-eye image is output by the convolutional layer; the right-eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right-eye image are obtained; wherein, the initial right-eye gaze deviation parameter is a 0 vector; the right-eye gaze deviation parameter, the initial gaze corresponding to the right-eye image, and the head pose corresponding to the right-eye image are concatenated to obtain the concatenated feature corresponding to the right-eye image; the concatenated feature corresponding to the right-eye image is input into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right-eye image; the intersection point of the final gaze corresponding to the right-eye image and the screen is calculated based on the final gaze corresponding to the right-eye image; the intersection point error loss is calculated based on the intersection point of the final gaze corresponding to each left-eye image and the screen, and the intersection point of the final gaze corresponding to each right-eye image and the screen; gradient backpropagation is performed based on the intersection point error loss to update the user's left-eye gaze deviation parameter and right-eye gaze deviation parameter.

[0178] Since the final gaze predicted in this embodiment is based on a gaze independent of the user's identity (i.e., the initial gaze) and a gaze related to the user's identity (i.e., the human eye gaze deviation parameter), and the human eye gaze deviation parameter can be continuously optimized during user use, this embodiment can improve the accuracy of gaze estimation. Furthermore, since this embodiment performs gradient backpropagation based on intersection point error loss, i.e., uses the geometric constraints of the gaze as a supervision signal, this embodiment does not require separately collecting gaze label samples from the user before using the gaze estimation system. Instead, it can continuously learn the human eye gaze deviation parameter as the user uses the system; as the number of samples increases, the human eye gaze deviation parameter will become increasingly accurate.

[0179] Example 4:

[0180] This application embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements a self-calibration method for human eye gaze deviation parameters, including the following steps: acquiring N left-eye and right-eye images when a user looks at a point on the screen; wherein, N is greater than or equal to 1; inputting the left-eye image into a pre-trained first convolutional neural network, and having the convolutional layer output the initial gaze corresponding to the left-eye image; acquiring the left-eye gaze deviation parameter corresponding to the user and the head pose corresponding to the left-eye image; wherein, the initial left-eye gaze deviation parameter is a 0 vector; concatenating the left-eye gaze deviation parameter, the initial gaze corresponding to the left-eye image, and the head pose corresponding to the left-eye image to obtain the concatenated feature corresponding to the left-eye image; inputting the concatenated feature corresponding to the left-eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left-eye image; calculating the intersection point between the final gaze corresponding to the left-eye image and the screen based on the final gaze corresponding to the left-eye image; and inputting the left-eye gaze deviation parameter into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left-eye image. The right-eye image corresponding to the left-eye image is input into a pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right-eye image. The right-eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right-eye image are obtained; wherein, the initial right-eye gaze deviation parameter is a zero vector. The right-eye gaze deviation parameter, the initial gaze corresponding to the right-eye image, and the head pose corresponding to the right-eye image are concatenated to obtain the concatenated feature corresponding to the right-eye image. The concatenated feature corresponding to the right-eye image is input into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right-eye image. The intersection point between the final gaze corresponding to the right-eye image and the screen is calculated based on the final gaze corresponding to the right-eye image. The intersection point error loss is calculated based on the intersection point between the final gaze corresponding to each left-eye image and the screen, and the intersection point between the final gaze corresponding to each right-eye image and the screen. Gradient backpropagation is performed based on the intersection point error loss to update the user's left-eye gaze deviation parameter and right-eye gaze deviation parameter.

[0181] The aforementioned self-calibration method for human eye gaze deviation parameters improves accuracy because the predicted final gaze distance in this embodiment is based on both a gaze distance unrelated to the user's identity (i.e., the initial gaze distance) and a gaze distance related to the user's identity (i.e., the human eye gaze deviation parameter). Furthermore, since this embodiment performs gradient backpropagation based on intersection point error loss (i.e., using the geometric constraints of the gaze distance as a monitoring signal), it does not require separate collection of the user's gaze label samples before the system is used. Instead, the human eye gaze deviation parameter is continuously learned as the user uses the system, and with an increasing number of samples, the parameter becomes more and more accurate.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0184] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A self-calibration method for human eye line deviation parameters, characterized in that, include: Obtain N images of the user's left and right eyes when the user is looking at a point on the screen; where N is greater than or equal to 1. The left-eye image is input into a pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left-eye image; Obtain the left eye gaze deviation parameter corresponding to the user and the head pose corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector; The left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image are stitched together to obtain the stitched feature corresponding to the left eye image. The stitched features corresponding to the left eye image are input into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image. Calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image; The right eye image corresponding to the left eye image is input into a pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right eye image; Obtain the right eye gaze deviation parameter corresponding to the user and the head pose corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector; The right eye gaze deviation parameter, the initial gaze corresponding to the right eye image, and the head posture corresponding to the right eye image are stitched together to obtain the stitching feature corresponding to the right eye image. The stitched features corresponding to the right eye image are input into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image. Calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image; The intersection point error loss is calculated based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image. Gradient backpropagation is performed based on the intersection point error loss to update the user's left-eye gaze deviation parameters and right-eye gaze deviation parameters.

2. The self-calibration method for human eye line deviation parameters according to claim 1, characterized in that, Also includes: Obtain the first training data, which includes N left-eye sample images, gaze labels corresponding to the left-eye sample images, and M user identity information. Obtain the second training data, which includes N right-eye sample images, gaze labels corresponding to the right-eye sample images, and M user identity information; wherein the left-eye sample images and the right-eye sample images are in one-to-one correspondence, and the left-eye sample images and the right-eye sample images are collected when the user looks at a point on the screen; wherein N and M are positive integers, and N is greater than or equal to M; The left-eye sample image is input into the first convolutional neural network to be trained, and the convolutional layer outputs the initial gaze corresponding to the left-eye sample image; Based on the user identity information corresponding to the left eye sample image, select the corresponding left eye gaze deviation parameter; The left eye gaze deviation parameter corresponding to the left eye sample image, the initial gaze corresponding to the left eye sample image, and the head posture corresponding to the left eye sample image are stitched together to obtain the stitched feature corresponding to the left eye sample image. The stitched features corresponding to the left eye sample image are input into the fully connected layer of the first convolutional neural network to be trained to obtain the final gaze corresponding to the left eye sample image. The right eye sample image corresponding to the left eye sample image is input into the second convolutional neural network to be trained, and the initial gaze corresponding to the right eye sample image is output by the convolutional layer; wherein, the network parameters of the first convolutional neural network to be trained and the second convolutional neural network to be trained are shared; Based on the user identity information corresponding to the right eye sample image, select the corresponding right eye gaze deviation parameter; The right eye gaze deviation parameter corresponding to the right eye sample image, the initial gaze corresponding to the right eye sample image, and the head posture corresponding to the right eye sample image are stitched together to obtain the stitched feature corresponding to the right eye sample image. The stitched features corresponding to the right eye sample image are input into the fully connected layer of the second convolutional neural network to be trained to obtain the final gaze corresponding to the right eye sample image. The left-eye gaze error loss is calculated based on the final gaze corresponding to each left-eye sample image and the gaze label corresponding to each left-eye sample image. The right-eye gaze error loss is calculated based on the final gaze corresponding to each right-eye sample image and the gaze label corresponding to each right-eye sample image. Calculate the intersection point between the final line of sight corresponding to the left eye sample image and the screen based on the final line of sight corresponding to the left eye sample image; Calculate the intersection point between the final line of sight corresponding to the right eye sample image and the screen based on the final line of sight corresponding to the right eye sample image; The intersection point error loss is calculated based on the intersection point of the final line of sight with the screen for each left-eye sample image and the intersection point of the final line of sight with the screen for each right-eye sample image. Gradient backpropagation is performed based on the left eye gaze error loss, the right eye gaze error loss, and the intersection point error loss to update the parameters of the first convolutional neural network, the parameters of the second convolutional neural network, and the left eye gaze deviation parameters and right eye gaze deviation parameters corresponding to each user, until the preset loss function converges, thus obtaining the pre-trained first convolutional neural network and the second convolutional neural network.

3. The self-calibration method for human eye line deviation parameters according to claim 1, characterized in that, ... The plane containing the screen is represented as ax + by + cz = d, where a, b, c, and d are known. The center of the eyeball is (o1, o2, o3), and the actual line-of-sight direction vector is (d1, d2, d3). Assuming the intersection point is (l, m, n), the intersection point of the line of sight and the screen is determined by the formula... Calculated.

4. The self-calibration method for human eye line deviation parameters according to claim 1, characterized in that, The intersection point error loss is calculated according to the formula... The calculation yields the result; where N represents the total number of samples, and i represents the distinct samples. This represents the intersection of the final line of sight corresponding to the left-eye image i and the screen; This represents the intersection of the final line of sight corresponding to the right eye image i and the screen.

5. The self-calibration method for human eye line deviation parameters according to claim 2, characterized in that, The line-of-sight error loss is calculated according to the following formula: Where N represents the total number of samples, v represents the final line of sight, and g represents the line of sight label.

6. The self-calibration method for human eye line deviation parameters according to claim 2, characterized in that, The preset loss function is L=L rightgaze +L rightgaze +λL POG Among them, L leftgaze L represents the visual error loss in the left eye. rightgaze L represents the visual error loss of the right eye. POG To represent the intersection point loss, λ is a parameter for adjusting the loss weight.

7. The self-calibration method for human eye line deviation parameters according to claim 1, characterized in that, The step of calculating the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image specifically includes: The validity of the left-eye image is determined based on the intersection of the final line of sight corresponding to the left-eye image and the screen. The validity of the right-eye image is determined based on the intersection of the final line of sight corresponding to the right-eye image and the screen; wherein the right-eye image corresponds to the left-eye image; If both the left-eye image and the right-eye image are valid, then the intersection point error loss is calculated based on the intersection point of the final line of sight with the screen corresponding to each valid left-eye image and the intersection point of the final line of sight with the screen corresponding to each valid right-eye image.

8. The self-calibration method for human eye line deviation parameters according to claim 7, characterized in that, The validity of a single-eye image is determined as follows: The single-eye image is a valid image if the intersection of the final line of sight corresponding to the single eye image and the screen satisfies any of the following conditions; The intersection of the final line of sight corresponding to the single eye image and the screen is within the screen area; The intersection of the final line of sight corresponding to the single eye image and the screen is within P times the standard deviation of the historical prediction mean; where P is a positive number.

9. The self-calibration method for human eye line deviation parameters according to claim 1, characterized in that, The human eye line deviation parameter is a three-dimensional vector or a two-dimensional vector.

10. A self-calibration device for human eye line deviation parameters, characterized in that, include: The first acquisition unit is used to acquire N images of the user's left eye and right eye when the user is looking at a point on the screen; where N is greater than or equal to 1. The first initial gaze prediction unit is used to input the left eye image into a pre-trained first convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the left eye image; The second acquisition unit is used to acquire the left eye gaze deviation parameter corresponding to the user and the head posture corresponding to the left eye image; wherein, the initial left eye gaze deviation parameter is a 0 vector; The first stitching unit is used to stitch together the left eye gaze deviation parameter, the initial gaze corresponding to the left eye image, and the head posture corresponding to the left eye image to obtain the stitching feature corresponding to the left eye image. The first final gaze prediction unit is used to input the stitched features corresponding to the left eye image into the fully connected layer of the pre-trained first convolutional neural network to obtain the final gaze corresponding to the left eye image. The first intersection point calculation unit is used to calculate the intersection point between the final line of sight corresponding to the left eye image and the screen based on the final line of sight corresponding to the left eye image. The second initial gaze prediction unit is used to input the right eye image corresponding to the left eye image into the pre-trained second convolutional neural network, and the convolutional layer outputs the initial gaze corresponding to the right eye image; The third acquisition unit is used to acquire the right eye gaze deviation parameter corresponding to the user and the head posture corresponding to the right eye image; wherein, the initial right eye gaze deviation parameter is a 0 vector; The second stitching unit is used to stitch together the right eye gaze deviation parameter, the initial gaze corresponding to the right eye image, and the head posture corresponding to the right eye image to obtain the stitching feature corresponding to the right eye image. The second final gaze prediction unit is used to input the stitched features corresponding to the right eye image into the fully connected layer of the pre-trained second convolutional neural network to obtain the final gaze corresponding to the right eye image. The second intersection point calculation unit is used to calculate the intersection point between the final line of sight corresponding to the right eye image and the screen based on the final line of sight corresponding to the right eye image. The intersection point error loss unit is used to calculate the intersection point error loss based on the intersection point of the final line of sight with the screen for each left-eye image and the intersection point of the final line of sight with the screen for each right-eye image. The gradient backpropagation and update unit is used to perform gradient backpropagation based on the intersection point error loss and update the user's left-eye gaze deviation parameters and right-eye gaze deviation parameters.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images