Gaze estimation method and device based on Gaussian geometric primitives, equipment and medium

Abstract

The invention relates to a gauss geometric primitive-based sight line estimation method, device and equipment and a medium. The gauss geometric primitive-based sight line estimation method comprises the following steps: acquiring image sequences continuously acquired when left and right eyeballs move; performing three-dimensional reconstruction on the left eyeball and the right eyeball according to the image sequence to generate a left eyeball model and a right eyeball model which are at least one group of dynamically changing Gaussian geometric primitives; performing motion analysis on Gaussian geometric primitives in the left and right eyeball models, and determining rotation information of the left and right eyeballs; according to eyeball coordinate systems of the left eyeball and the right eyeball, sight line intersection points of the left eyeball and the right eyeball in a world coordinate system are calculated, and the eyeball coordinate systems are constructed based on rotation information with eyeball centers of the left eyeball and the right eyeball as original points; and estimating the sight direction by using the sight intersection to obtain a sight estimation result. According to the method provided by the invention, high-precision and low-delay sight line estimation is realized, and the limitation of an eye movement tracking technology in real-time application is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a line of sight estimation method, device, equipment and medium based on Gaussian geometric primitives. Background Art

[0002] Eye tracking technology has a wide range of applications in virtual reality, augmented reality, human-computer interaction, and medical diagnosis. However, existing technologies still have problems in balancing computational complexity, accuracy, and efficiency. For example, complex image processing algorithms in eye tracking increase the demand for computing resources, while the pursuit of high accuracy may lead to increased latency, which in turn affects real-time performance. This balance problem also limits its development in real-time applications. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a line of sight estimation method, apparatus, device and medium based on Gaussian geometric primitives.

[0004] In a first aspect, an embodiment of the present disclosure provides a line of sight estimation method based on Gaussian geometric primitives, comprising:

[0005] Obtain a sequence of images continuously collected when the left and right eyeballs move;

[0006] Performing three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives;

[0007] Perform motion analysis on Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs;

[0008] According to the eyeball coordinate system of the left and right eyeballs, the intersection point of the sight lines of the left and right eyeballs in the world coordinate system is calculated, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin;

[0009] The line of sight direction is estimated using the line of sight intersection point to obtain the line of sight estimation result.

[0010] In a second aspect, an embodiment of the present disclosure provides a sight line calculation device based on Gaussian geometric primitives, including:

[0011] An acquisition unit, used for acquiring a sequence of images continuously acquired when the left and right eyeballs move;

[0012] A three-dimensional reconstruction unit, used for performing three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives;

[0013] A motion decomposition unit, used for performing motion analysis on Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs;

[0014] A calculation unit, used for calculating the intersection of the sight lines of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin;

[0015] The estimation unit is used to estimate the sight line direction by using the sight line intersection point to obtain a sight line estimation result.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0017] Memory;

[0018] Processor; and

[0019] Computer programs;

[0020] The computer program is stored in the memory and is configured to be executed by the processor to implement the method of the first aspect as described above.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0022] The present disclosure provides a line of sight estimation method based on Gaussian geometric primitives, including: obtaining a sequence of images continuously collected when the left and right eyeballs move; performing three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives; performing motion analysis on the Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs; calculating the line of sight intersection of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin; using the line of sight intersection to estimate the line of sight direction and obtain a line of sight estimation result. The method provided by the present application realizes high-precision, low-latency line of sight estimation and reduces the limitations of eye tracking technology in real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A schematic diagram of a flow chart of a line of sight estimation method based on Gaussian geometric primitives provided in an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of an eye image provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of an eyeball model provided in an embodiment of the present disclosure;

[0028] Figure 4 for Figure 1 A schematic diagram of a refinement process of S103 in a line of sight estimation method based on Gaussian geometric primitives is shown;

[0029] Figure 5 for Figure 1 A schematic diagram of a refinement process of S104 in a line of sight estimation method based on Gaussian geometric primitives is shown;

[0030] Figure 6 A schematic diagram of the structure of a sight line estimation device based on Gaussian geometric primitives provided in an embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0034] Specifically, eye tracking technology usually uses image processing-based methods or deep learning methods to estimate the gaze point. Image processing-based methods require manual feature design and have limited generalization capabilities. Deep learning-based methods require a large amount of labeled data for training, which has high data preparation costs and poor adaptability. In addition, existing technologies generally face the problems of high computational complexity and difficulty in balancing accuracy and efficiency, and cannot meet the needs of real-time applications. Among them:

[0035] 1) Image processing based methods

[0036] Usually, useful feature information (such as feature points of pupil and corneal reflection) is extracted from the image first, and then the gaze point is estimated based on the feature information and the pre-established model. This method is relatively simple in design and can achieve high efficiency in specific scenarios. However, it requires manual selection of key feature information, has poor generalization ability, and is difficult to adapt to complex scenarios and individual differences.

[0037] 2) Deep learning-based methods

[0038] Deep learning models such as convolutional neural networks are usually used to learn feature representations from images and directly predict the gaze point. This method has strong learning and generalization capabilities and can adapt to complex scenarios and data distributions. However, it requires a large amount of labeled data for training, which is costly to prepare, and the model complexity may also be high, affecting real-time performance.

[0039] In view of the above technical problems, the embodiments of the present disclosure provide a line of sight estimation method based on Gaussian geometric primitives, which is described in detail through one or more of the following embodiments.

[0040] Before describing in detail the sight line estimation method based on Gaussian geometric primitives provided in the embodiment of the present disclosure, the related terms are described first, where:

[0041] Gaussian Splatting: A differentiable rendering technique that represents a 3D scene as a set of sparse weighted Gaussian functions, called "Gaussian primitives". Each Gaussian primitive is parameterized by its position, scale, color, and other potential properties such as surface normal or density. The rendering process consists of projecting the Gaussian primitives onto the image plane, convolving their Gaussian kernels with the image, and accumulating the contributions of all visible Gaussian primitives to produce the final image. The differentiability of the rendering process stems from the explicit mathematical formulation of the Gaussian primitive contributions and their mixing, which enables the parameters of the Gaussian primitives to be learned from input data such as point clouds, meshes, or images using gradient-based optimization methods. It also allows complex scenes to be reconstructed and rendered from relatively sparse representations, providing a compelling alternative to traditional polygon- or point-based rendering methods. Efficiency is achieved through careful culling and rendering algorithms, which prioritize Gaussian primitives that contribute significantly to the final image. The sparsity inherent in the Gaussian primitive representation results in significant memory savings and speed improvements compared to denser representations.

[0042] 3D Gaussian Geometric Primitives: In the context of 3D reconstruction, 3D Gaussian Geometric Primitives is a method to represent the geometry and appearance of a scene as a set of weighted anisotropic Gaussian Probability Density Functions (PDFs) in 3D space. Each Gaussian sphere includes parameters such as centroid, covariance matrix, color / reflectivity, and weight / opacity.

[0043] Rendering: This is achieved by projecting Gaussian spheres onto the image plane and blending their contributions. When rendering each pixel color, the eye tracking system first generates a ray from the camera for that pixel and detects the intersection of this ray with all Gaussian spheres in the scene. Then, the Gaussian spheres are sorted according to the distance of the intersection from the camera, giving priority to those that are closer. Next, the system stacks the colors and opacities of these Gaussian spheres in depth order, accumulating the contribution of each Gaussian sphere to the final pixel color through alpha blending, while taking into account the occlusion effect of the previous Gaussian spheres. In this way, nearby Gaussian spheres can preferentially affect the pixel color, ensuring that the rendering result has realistic occlusion and transparency effects. The entire process is implemented through efficient sorting and blending algorithms to meet the requirements of real-time eye tracking systems for low latency and high accuracy. In the parameter optimization process, the system first defines a loss function that measures the difference between the rendered image and the input data. The gradient of this loss function relative to the parameters of each Gaussian sphere is calculated through the error backpropagation algorithm. Then, the parameters are gradually adjusted based on this gradient information to minimize the overall loss using optimization methods such as gradient descent. This iterative process continues until the loss function converges to a preset threshold or reaches the maximum number of iterations. Through this error backpropagation optimization process, the system can accurately adjust the parameters of the Gaussian geometric primitive model.

[0044] The line of sight estimation method based on Gaussian geometric primitives provided in the embodiment of the present disclosure can be applied to the eyeball line of sight estimation scenario. The method can be performed by a line of sight estimation device based on Gaussian geometric primitives, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device. Among them, the electronic device can include but is not limited to mobile terminals such as smart phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Tablet Personal Computers, Tablet PCs), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, smart home devices, etc.

[0045] Figure 1 A schematic flow chart of a line of sight estimation method based on Gaussian geometric primitives provided in an embodiment of the present disclosure, specifically comprising the following steps: Figure 1 The following steps are shown:

[0046] S101, obtaining a sequence of images continuously collected when the left and right eyeballs move.

[0047] The image sequence includes multiple frames of images.

[0048] It is understandable that, during the movement of the left and right eyeballs, the sight line changes with the movement of the eyeballs. In this case, images are continuously collected when the left and right eyeballs move to generate an image sequence. The image sequence includes multiple frames of images, which reflect the dynamic changes of the eyeballs, and each frame of the image contains the facial area of ​​the face. Figure 2 A schematic diagram of an eye image provided by an embodiment of the present disclosure.

[0049] It is understandable that the image sequences of the left and right eyeballs can be collected simultaneously by one acquisition device, or separately by two acquisition devices, that is, monocular acquisition, and the specific acquisition process is not limited. Monocular acquisition can reduce hardware costs and system complexity, is easier to apply in practice, and does not require complex calibration, only camera internal parameters are required.

[0050] It is understandable that after the image sequence is acquired, the image sequence may be subjected to denoising, normalization, and / or extraction of the region of interest of the human eye, etc. Subsequently, the Gaussian sphere may be reconstructed using the processed image sequence.

[0051] S102, performing three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models.

[0052] The left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives.

[0053] It is understandable that, based on the above S101, the left and right eyeballs are reconstructed in three dimensions according to the image sequence. The dynamic Gaussian geometric primitive method can be used to model the eyeball as a set of Gaussian geometric primitives with Gaussian parameters such as color and transparency. The left and right eyeball models can also be understood as a set of Gaussian balls. The Gaussian geometric primitives can be Gaussian balls or Gaussian patches. The following embodiments are explained in detail using the Gaussian geometric primitives as Gaussian balls. The dynamic Gaussian geometric primitive method is used to perform real-time three-dimensional reconstruction of the eyeball, avoiding the need for neural network training on large data sets.

[0054] Optionally, three-dimensional reconstruction of the left and right eyeballs is performed according to the image sequence to generate left and right eyeball models, which can be specifically achieved through the following steps:

[0055] The left and right eyeballs in each frame image are represented as a group of Gaussian geometric primitives, wherein each Gaussian geometric primitive represents an area on the surface of the eyeball and has at least one parameter of position, size, posture, color and transparency; multiple groups of Gaussian geometric primitives corresponding to multiple frame images are set to share the same color and the same transparency; wherein the left and right eyeball models include multiple groups of Gaussian geometric primitives.

[0056] It is understandable that a set of Gaussian balls can be obtained for each frame image, and multiple sets of Gaussian balls can be obtained for multiple frames of images. The dynamically changing sets of Gaussian balls can be understood as a dynamically changing eyeball model. Among them, each Gaussian ball represents a tiny area on the surface of the eyeball and has at least one Gaussian parameter such as position, size, color and transparency, that is, each Gaussian ball has attribute information in the corresponding image frame. Multiple sets of Gaussian balls share color and transparency parameters at different times (different image frames), but their positions and postures change over time (image frames), so as to track the movement of each Gaussian ball in continuous image frames, and then estimate the overall movement and deformation of the eyeball. Compared with the solution that requires accurate extraction of feature points such as pupils and corneal reflections, the gaze direction is indirectly estimated by modeling and tracking the entire surface of the eyeball, and the characteristics of the Gaussian function can easily handle problems such as occlusion and blur on the surface of the eyeball. In addition, the differentiability of the eyeball model can also be used to optimize the primitive parameters using optimization methods such as gradient descent to improve tracking accuracy.

[0057] For example, Figure 3 A schematic diagram of an eyeball model provided in an embodiment of the present disclosure is the above Figure 2 A set of Gaussian spheres of the eye image. Different structures of the eye correspond to multiple Gaussian spheres.

[0058] Optionally, after the left and right eyeballs in each frame of the image are represented as a group of Gaussian geometric primitives, for each group of Gaussian geometric primitives, the method further includes:

[0059] Rendering each group of Gaussian geometric primitives according to pre-set initial parameters to generate a rendered image; wherein the initial parameters are set randomly, set according to historical parameters of historical Gaussian geometric primitives, or set according to a depth image estimated by a neural network model; calculating the difference between the rendered image and each frame image to obtain an error loss; updating the parameters of each group of Gaussian geometric primitives according to the error loss to obtain each group of updated Gaussian geometric primitives.

[0060] It can be understood that after the left and right eyeballs are represented as a group of Gaussian balls, the Gaussian parameters of the Gaussian balls are optimized. Specifically: the initial parameters of the Gaussian distribution are set, and the Gaussian balls are rendered according to the initial parameters to render an image. The rendering process can refer to the above description, wherein the initial parameters can be a randomly set initial Gaussian distribution, or the mean of the Gaussian distribution in the current scene can be set as the initial parameters according to empirical data. The empirical data refers to historical parameters, and the initial parameters can also be inferred based on the depth image estimated by a general monocular deep neural network model as a reference; then, the error loss is calculated based on the difference between the rendered image and the captured image, and the Gaussian parameters are optimized through the reverse transfer of the error loss. Among them, a frame of captured image can be represented as a group of Gaussian balls, and a group of Gaussian balls can in turn render an image. According to the comparison between the captured image and the rendered image, the optimization of Gaussian parameters is achieved, which can improve the tracking accuracy.

[0061] It is understandable that in subsequent image frames, the position, size, direction, etc. of each Gaussian ball can be updated by tracking the changes of the same Gaussian ball in the image sequence. Figure 2 When the eye is imaged, the distribution of multiple groups of Gaussian balls changing over time naturally includes: the reciprocating motion of the Gaussian balls related to the gaze target caused by the rotation of the eyeball, the reciprocating motion of the eyelids and eyelashes along the up and down direction caused by the blinking of the eye, and the overall motion of all Gaussian balls caused by the movement of the head relative to the acquisition device. During the eyeball movement, at least one reciprocating motion will occur.

[0062] S103, performing motion analysis on the Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs.

[0063] It can be understood that, based on the above S102, multiple groups of Gaussian balls are subjected to motion decomposition. By representing the motion information of the Gaussian balls in different areas of the eyeball, the motion information of the eyeball can be further decomposed. The motion information includes the rotation information of the eyeball. The line of sight direction can be further estimated based on the rotation information.

[0064] S104, calculating the sight line intersection point of the left and right eyeballs in the world coordinate system according to the eye coordinate systems of the left and right eyeballs.

[0065] Among them, the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin.

[0066] It is understandable that, based on the above S103, the eyeball center is determined according to the rotation information, and the determination process is described in the following embodiment. Subsequently, the eyeball coordinate system is constructed with the eyeball center of each of the left and right eyeballs as the origin. The eyeball coordinate system has angle information in the world coordinate system, that is, the eyeball coordinate system has spatial information in the world coordinate system. The world coordinate system can also be the camera coordinate system of the acquisition device.

[0067] S105. Estimate the sight line direction using the sight line intersection point to obtain a sight line estimation result.

[0068] It is understandable that, based on the above S104, after obtaining the eye center and the continuous angle information of the eye center through a set of continuous image sequences, the line of sight direction is calculated based on the intersection of the lines of sight of both eyes to obtain the line of sight estimation result without the need for additional calibration points.

[0069] It can be understood that before the line of sight estimation is performed, an acquisition module, a Gaussian geometric primitive module, a calculation module, a calibration module and an application program interface are constructed. Among them, the acquisition module is used to acquire continuous image sequences; the Gaussian geometric primitive module is used to build an eyeball model based on the image sequence; the calculation module is used to simultaneously process the optimization algorithm of multiple Gaussian distributions, efficiently perform Gaussian geometric primitive operations, and use effective primitive combination technology to ensure that the sum of overlapping Gaussian balls can accurately reflect the probability distribution of the pupil position; the calibration module is used to record reference points in the initial stage and establish the association between the eyeball model and the gaze position. The calibration module can also be adjusted according to the difference in the gaze target to improve the tracking accuracy and system adaptability; the application program interface (API) is used to provide real-time eye tracking data to various applications, such as VR / AR (virtual display / virtual enhancement) systems or user interface devices. The API can ensure data streaming at a smooth interactive rate to achieve responsive gaze-based control. The application program interface can be understood as a seamless integration interface that can achieve smooth gaze-based control and minimal delay.

[0070] The embodiment of the present disclosure provides a line of sight estimation method based on Gaussian geometric primitives, which effectively reduces the computational complexity, improves the tracking accuracy and efficiency, and enhances the adaptability to different users through Gaussian geometric primitive technology. In addition, the acquisition of monocular video can also reduce hardware costs and system complexity, and is easy to apply in practice.

[0071] Figure 4 for Figure 1FIG. 1 is a schematic diagram of a detailed flow chart of S103 in a line of sight estimation method based on Gaussian geometric primitives. Optionally, a motion analysis is performed on the Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs, specifically including the following steps: Figure 4 The following steps are shown:

[0072] S401, constructing an adjacency matrix of multiple frames of images in an image sequence based on the spatial distance and relative position relationship between Gaussian geometric primitives in the left and right eyeball models.

[0073] The adjacency matrix reflects the adjacency relationship between Gaussian geometric primitives in each frame of the image.

[0074] It can be understood that for the Gaussian ball set of each frame image, the adjacency relationship between the Gaussian balls is determined based on the spatial distance and relative position relationship between the Gaussian balls. Specifically, the Euclidean distance between two Gaussian balls is calculated, and the Euclidean distance is compared with the set distance threshold. If the Euclidean distance is less than the distance threshold, it is determined that there is an adjacency relationship between the two Gaussian balls, that is, the two Gaussian balls are adjacent. After determining the adjacency relationship between every two Gaussian balls in the Gaussian ball set, an adjacency matrix is ​​constructed. For example, in the adjacency matrix A, A[i][j]=1 indicates that Gaussian ball i and Gaussian ball j are adjacent, and A[i][j]=0 indicates that Gaussian ball i and Gaussian ball j are not adjacent. The adjacency matrix of each frame image can then be stored for subsequent processing.

[0075] S402. According to the adjacency matrix, every two adjacent Gaussian geometric primitives are taken as an initial group to obtain multiple initial groups.

[0076] There are no repeated Gaussian geometric primitives in the multiple initial groups.

[0077] It can be understood that, based on the above S402, an initial grouping method is adopted to form an initial group of adjacent Gaussian balls in pairs based on the adjacency matrix, and ensure that each Gaussian ball belongs to only one initial group to avoid repeated grouping of Gaussian balls.

[0078] Optionally, according to the adjacency matrix, each two adjacent Gaussian geometric primitives are used as an initial group, including:

[0079] For each Gaussian geometric primitive, the adjacent Gaussian geometric primitives of each Gaussian geometric primitive are determined through an adjacency matrix; the adjacent Gaussian geometric primitives and each Gaussian geometric primitive are formed into an initial group.

[0080] It can be understood that for each Gaussian ball, its adjacent Gaussian balls are found based on the adjacency matrix and recorded as adjacent Gaussian balls. The Gaussian ball and its adjacent Gaussian balls are promoted to an initial group. After the Gaussian ball set completes the initial group division, multiple initial groups will be obtained, and there is no identical Gaussian ball in each initial group.

[0081] S403: Perform stability detection on multiple initial groups, and merge the multiple initial groups according to the obtained detection results to obtain multiple rigid body groups.

[0082] It can be understood that, based on the above S402, after the initial grouping is completed, stability detection of multiple initial groups is performed. The stability detection refers to the stability of the relative distance between two Gaussian balls in the detection group to ensure the consistency of the movement of all Gaussian balls included in each rigid body group. The rigid body group can be understood as a group of Gaussian balls that remain relatively still.

[0083] Optionally, stability testing is performed on multiple initial groups, which can be achieved through the following steps:

[0084] For each initial group, the relative distance change value of two Gaussian geometric primitives in multiple frames is calculated; if the relative distance change value is less than the set stability threshold, it is determined that each initial group maintains relative distance stability in motion, and a detection result with consistent relative distance change is generated.

[0085] It can be understood that for each initial group, the relative distance change between the two Gaussian balls in the group in the continuous image frames is calculated. If the relative distance change between the Gaussian balls in the group is within the stability threshold, it is considered that the relative distance between the Gaussian balls in the group is stable in motion, and can be merged into a stable sub-rigid body group, recorded as a stable group, where the initial group includes two Gaussian balls and the stable group includes at least two Gaussian balls. At the same time, a detection result with consistent relative distance change is generated. In addition, for the initial groups that do not meet the stability condition, they can be regrouped or marked as noise.

[0086] Optionally, multiple initial groups are merged according to the obtained detection results to obtain multiple rigid body groups, which can be specifically achieved through the following steps:

[0087] All initial groups with consistent relative distance changes in the detection results are merged into stable groups to obtain multiple stable groups; for each stable group, the process of adjacency matrix construction, initial group division and stability detection is repeated until multiple rigid body groups are obtained; wherein the multiple rigid body groups include at least an eye-periocular rigid body group and an eyeball rigid body group.

[0088] It can be understood that the above-mentioned process of initial group division, stability detection and sub-rigid body group division is performed recursively, and each sub-rigid body group is again subjected to initial grouping and stability detection, and a new adjacency matrix is ​​constructed, and only the adjacency relationship between groups is considered, and the sub-rigid body groups that maintain a stable relative distance in motion are gradually merged to obtain larger rigid body groups, until three larger rigid body groups are finally formed, corresponding to the eyeball, eyelashes and periocular skin of the eye, etc., among which the recursive merging is terminated when the periocular rigid body group and the eyeball rigid body group are obtained.

[0089] Understandably, after obtaining multiple rigid body groups, multiple rigid body groups are classified to determine the eyeball rigid body group and the eyeball rigid body group. Specifically, the motion law characteristics of each rigid body group are extracted, such as rotation angle change, translation amplitude, motion frequency, etc., wherein the eyeball is mainly rotational motion and a small amount of translational motion, the eyelashes are periodically reciprocating up and down, with a high motion frequency, and the skin around the eye is mainly caused by the slight translation and rotation of the head movement. Subsequently, the relative spatial position and distribution relationship between the rigid body groups are analyzed, and the relative spatial distribution characteristics of multiple rigid body groups are extracted, such as the eyeball is located inside the eyeball and the center position is stable, the eyelashes are located above the eyeball and are densely distributed, and the skin around the eye covers the entire eye area and surrounds the eyeball. Subsequently, according to the extracted motion law characteristics and relative spatial distribution characteristics, multiple rigid body groups are classified and identified, such as using predefined feature matching rules or training classifiers (such as support vector machines, decision trees) for classification. Subsequently, according to the classification results of multiple rigid body groups, each Gaussian ball is assigned an initial category label (such as eyeball, eyelashes, eyeball, etc.). In continuous image frames, the category labels are dynamically adjusted based on the temporal consistency and adjacency between Gaussian balls. If the motion characteristics or adjacency of a Gaussian ball change abnormally, its category label can be re-evaluated and divided into other rigid body groups. Subsequently, temporal smoothing techniques (such as Kalman filtering and moving average) are applied, combined with the adjacency between Gaussian balls, to ensure the temporal continuity and stability of the category labels. Combined with the characteristics of motion laws and relative spatial distribution characteristics, multiple rigid body groups are classified to improve the accuracy and robustness of classification. In addition, the dynamic adjustment mechanism provided maintains the stability of the category labels through temporal consistency and adjacency information, combined with filtering technology, and can adapt to eye changes and environmental interference.

[0090] S404: Perform motion analysis on multiple rigid body groups to determine rotation information of left and right eyeballs.

[0091] It is understandable that, based on the above S403, in the motion analysis of the eye rigid body group, the motion components caused by head movement need to be removed to obtain pure eye movement (such as eye rotation or translation). Specifically, this can be achieved by removing the motion components related to head movement from the motion trajectory of the eye rigid body group.

[0092] Optionally, motion analysis is performed on multiple rigid body groups to determine the rotation information of the left and right eyeballs, which can be specifically achieved through the following steps:

[0093] The center of mass position of the eye rigid body group in each frame image is calculated, and the first motion information of the eye rigid body group is determined according to the center of mass position; wherein the first motion information represents the head motion and the eye motion, the head motion includes the head translation, and the eye motion includes the eye rotation; the center of mass position of the periocular rigid body group in each frame image is calculated, and the second motion information of the periocular rigid body group is determined according to the center of mass position; wherein the second motion information represents the head motion; according to the first motion information and the second motion information, the motion difference between the eye rigid body group and the periocular rigid body group is calculated to obtain the third motion information of the eye rigid body group; wherein the third motion information represents the eye motion, and the third motion information reflects the rotation information of the left and right eyeballs.

[0094] It can be understood that the centroid of all Gaussian balls in the eyeball rigid body group in each frame image is extracted, as shown in formula (1).

[0095]

[0096] Where C1 is the center of mass position, P ti is the Gaussian sphere g ti The position in the t-th frame image.

[0097] It can be understood that the overall motion of the periocular rigid body group is extracted, and for each Gaussian sphere in the periocular rigid body group, its center of mass in multiple frame images is calculated. The specific calculation is shown in formula (2).

[0098]

[0099] Where C2 is the center of mass position, P si is the Gaussian sphere g si The position in the t-th frame image.

[0100] It is understandable that the first motion information is obtained by performing motion analysis based on the center of mass position of each Gaussian ball in the eyeball rigid body group, wherein the first motion information reflects the head movement and the eyeball movement, that is, the eyeball will not only move with the head but also rotate. The second motion information is obtained by performing motion analysis based on the center of mass position of each Gaussian ball in the eyeball rigid body group, wherein the second motion information reflects the head movement, that is, the eyeball will not rotate but only move with the head. The second motion information only includes the relevant information of the head movement, and the first motion information includes the relevant information of the head movement and the eyeball rotation. In this case, the relevant information of the head movement in the eyeball rigid body group is removed, and the relevant information of the eyeball rotation is retained. Specifically: the motion component of the eyeball rigid body group is removed from the center of mass motion of the eyeball rigid body group. In each frame of the image, the motion of the eyeball rigid body group can be expressed as the change of the center of mass, as shown in formula (3).

[0101] ΔC1=C1(t)-C1(t-1) Formula (3)

[0102] It can be understood that the movement of the periorbital rigid body group can also be expressed as the change of the center of mass, as shown in formula (4).

[0103] ΔC2=C2(t)-C2(t-1) Formula (4)

[0104] It is understandable that since the movement of the periorbital rigid body group represents pure head movement, the pure movement of the eyeball rigid body group should be the movement component that needs to be removed from the eyeball rigid body group. By calculating the difference between the eyeball movement and the periorbital movement, the rotational movement of the eyeball can be obtained, and then the third movement information can be obtained, as shown in formula (5).

[0105] C3(t)=C1(t)-C2(t) Formula (5)

[0106] Where C3(t) represents the remaining eye rotation movement after removing the head movement component from the eye rigid body group.

[0107] It can be understood that after removing the head movement in the eyeball rigid body group, the position of each Gaussian ball in the eyeball rigid body group is corrected, as shown in formula (6).

[0108] P(t)=P ti -(C2(t)-C2(t-1)) Formula (6)

[0109] Where P(t) is the position of the Gaussian ball after removing the head motion component, P ti is the position of the Gaussian ball in the t-th frame image.

[0110] It can be understood that, in an image sequence, by repeating the above steps, the influence of the eye periorbital rigid body group on the movement of the eyeball rigid body group can be removed frame by frame.

[0111] The line of sight estimation method based on Gaussian geometric primitives provided by the embodiment of the present disclosure calculates the center of mass of the eyeball rigid body group and the eye-periorbital rigid body group, removes the movement information of the head in the eyeball rigid body group, retains the movement information of the eyeball rotation, can accurately track the eyeball rotation movement, and avoids the interference of the head movement on the eyeball movement. In addition, after removing the head movement part in the eyeball rigid body group, the real eyeball movement trajectory can be obtained by correcting the position of each Gaussian ball. High-precision classification and tracking of eye Gaussian balls is achieved, while improving classification accuracy and robustness, maintaining good real-time and adaptability.

[0112] Based on the above embodiments, Figure 5 for Figure 1 FIG. 1 is a schematic diagram of a detailed flow chart of S104 in a line of sight estimation method based on Gaussian geometric primitives. Optionally, according to the eye coordinate system of the left and right eyeballs, the line of sight intersection of the left and right eyeballs in the world coordinate system is calculated, specifically including the following steps: Figure 5The following steps are shown:

[0113] S501 . For each eyeball, determine the spatial information of the sight line starting point in the eyeball coordinate system of each eyeball, and transform the spatial information from the eyeball coordinate system to the world coordinate system according to a preset transformation matrix.

[0114] The spatial information includes the relative position and direction vector of the starting point of the line of sight relative to the center of the eyeball.

[0115] It is understandable that for each eyeball (left eyeball and right eyeball), the spatial information of the starting point of the line of sight in the eyeball coordinate system is determined. The starting point of the line of sight refers to the center of the pupil. The eyeball coordinate system is constructed with the center of the eyeball as the origin. The starting point of the line of sight has a relative position and posture in the eyeball coordinate system. Among them, the spatial information includes the relative position and direction vector of the starting point of the line of sight relative to the center of the eyeball.

[0116] Optionally, the spatial information is transformed from the eye coordinate system to the world coordinate system according to a preset transformation matrix, which can be specifically achieved through the following steps:

[0117] The product of a preset transformation matrix and a relative position is calculated to obtain a transformation position; the product of a rotated transformation matrix and a direction vector is calculated to obtain a transformation vector; wherein the transformed spatial information includes a transformation position and a transformation vector.

[0118] It can be understood that the transformation matrix refers to the transformation matrix from the eye coordinate system to the world coordinate system. The conversion of spatial information between coordinate systems is as follows:

[0119] Left eye: PL = TL*pL, VL = RL*vL

[0120] Right eye: PR = TR*pR, VR = RR*vR Formula (7)

[0121] Where PL is the transformation position of the left eye, TL is the transformation matrix of the left eye, pL is the relative position of the left eye, VL is the transformation vector of the left eye, RL is the transformation matrix of the left eye after rotation, and vL is the direction vector of the left eye. PR is the transformation position of the right eye, TR is the transformation matrix of the right eye, pR is the relative position of the right eye, VR is the transformation vector of the right eye, RR is the transformation matrix of the right eye after rotation, and vR is the direction vector of the right eye.

[0122] S502: construct parameter equations of the left and right eyeballs in the world coordinate system according to the transformed spatial information, and calculate the intersection point of the sight lines of the left and right eyeballs in the world coordinate system.

[0123] It can be understood that, based on the above S501, the parametric equation refers to the parametric equation of the left and right eye sight lines in the world coordinate system. When the parametric equation of the left eye is equal to the parametric equation of the right eye, the intersection of the left and right eye sight lines is solved.

[0124] Optionally, parameter equations of the left and right eyeballs in the world coordinate system are constructed according to the transformed spatial information, and the intersection of the sight lines of the left and right eyeballs in the world coordinate system is calculated. This can be specifically achieved through the following steps:

[0125] For the left eyeball, a first parameter equation is constructed based on the sum of the product of the transformation vector of the left eyeball and the first parameter and the transformation position of the left eyeball; for the right eyeball, a second parameter equation is constructed based on the sum of the product of the transformation vector of the right eyeball and the second parameter and the transformation position of the right eyeball; the first parameter and the second parameter are optimized so that the first parameter equation and the second parameter equation are equal to calculate the line of sight intersection.

[0126] Understandably, the parametric equations for the left and right eyes are as follows.

[0127] Left eye: XL(t)=PL+t*VL

[0128] Right eye: XR(s) = PR + s*VR Formula (8)

[0129] Wherein, XL(t) is the first parameter equation, t is the first parameter, XR(s) is the second parameter equation, s is the second parameter.

[0130] It is understandable that the line of sight intersection satisfies the overdetermined equation set XL(t)=XR(s), and the least square method or other optimization methods can be used to solve t and s, thereby obtaining the intersection position of the line of sight intersection X=XR(s)=XL(t). In addition, in the process of calculating the line of sight intersection, the spatial information of each eyeball can also be optimized so that it can meet the conditions of line of sight intersection at every moment. Among them, the optimization can use an iterative optimization algorithm to optimize the relative position of the line of sight starting point, the direction vector, and the intersection position of the line of sight intersection.

[0131] The disclosed embodiments provide a line of sight estimation method based on Gaussian geometric primitives, which directly estimates the line of sight direction from an eyeball model, avoiding the extraction of complex eyeball features (such as pupils, eye corners, etc.). The line of sight estimation has high accuracy and is robust to factors such as occlusion and illumination. There is no need to consider the impact of occlusion of eyeball features on line of sight estimation, and high-precision, low-latency real-time eye tracking can be achieved, providing a more natural interactive experience for virtual applications.

[0132] Figure 6The structure diagram of a sight line estimation device based on Gaussian geometric primitives provided in an embodiment of the present disclosure is shown in FIG. The sight line estimation device based on Gaussian geometric primitives provided in an embodiment of the present disclosure can execute the processing flow provided in an embodiment of a sight line estimation method based on Gaussian geometric primitives, such as Figure 6 As shown, the sight line estimation device 600 based on Gaussian geometric primitives includes an acquisition unit 601, a three-dimensional reconstruction unit 602, a motion decomposition unit 603, a calculation unit 604 and an estimation unit 605, wherein:

[0133] An acquisition unit 601 is used to acquire a sequence of images continuously acquired when the left and right eyeballs move;

[0134] A three-dimensional reconstruction unit 602 is used to perform three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives;

[0135] A motion decomposition unit 603 is used to perform motion analysis on Gaussian geometric primitives in the left and right eyeball models to determine rotation information of the left and right eyeballs;

[0136] A calculation unit 604 is used to calculate the sight line intersection point of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin;

[0137] The estimation unit 605 is used to estimate the sight line direction by using the sight line intersection point to obtain a sight line estimation result.

[0138] Optionally, the 3D reconstruction unit 602 is used for:

[0139] Representing the left and right eyeballs in each frame of image as a set of Gaussian geometric primitives, wherein each Gaussian geometric primitive represents an area on the surface of the eyeball and has at least one parameter of position, size, posture, color and transparency;

[0140] Set multiple groups of Gaussian geometric primitives corresponding to multiple frame images to share the same color and the same transparency;

[0141] The left and right eyeball models include multiple groups of Gaussian geometric primitives.

[0142] Optionally, the device 600 is further used for:

[0143] Rendering each group of Gaussian geometric primitives according to pre-set initial parameters to generate a rendered image; wherein the initial parameters are set randomly, according to historical parameters, or according to a depth image estimated by a neural network model;

[0144] Calculate the difference between the rendered image and each frame image to get the error loss;

[0145] The parameters of each group of Gaussian geometric primitives are updated according to the error loss to obtain each group of updated Gaussian geometric primitives.

[0146] Optionally, the motion decomposition unit 603 is used to:

[0147] Based on the spatial distance and relative position relationship between each Gaussian geometric primitive in the left and right eyeball models, an adjacency matrix of multiple frames of images in the image sequence is constructed; wherein the adjacency matrix reflects the adjacency relationship between each Gaussian geometric primitive in each frame of the image;

[0148] According to the adjacency matrix, every two adjacent Gaussian geometric primitives are divided into initial groups to obtain a plurality of initial groups; wherein the plurality of initial groups do not have repeated Gaussian geometric primitives;

[0149] Performing stability tests on multiple initial groups, and merging the multiple initial groups according to the obtained test results to obtain multiple rigid body groups;

[0150] The motion of multiple rigid body groups is decomposed to determine the rotation information of the left and right eyeballs.

[0151] Optionally, the motion decomposition unit 603 is used to:

[0152] All initial groups with consistent relative distance changes in detection results are merged into stable groups to obtain multiple stable groups;

[0153] For each stable group, the process of adjacency matrix construction, initial group division and stability detection is repeated until multiple rigid body groups are obtained; wherein the multiple rigid body groups at least include an eye periorbital rigid body group and an eyeball rigid body group.

[0154] Optionally, the motion decomposition unit 603 is used to:

[0155] Calculating the center of mass position of the eyeball rigid body group in each frame image, and determining the first motion information of the eyeball rigid body group according to the center of mass position; wherein the first motion information represents the head motion and the eyeball motion, the head motion includes the head translation, and the eyeball motion includes the eyeball rotation;

[0156] Calculating the center of mass position of the periocular rigid body group in each frame of the image, and determining the second motion information of the periocular rigid body group according to the center of mass position; wherein the second motion information represents the head motion;

[0157] According to the first motion information and the second motion information, the motion difference between the eyeball rigid body group and the eye periorbital rigid body group is calculated to obtain the third motion information of the eyeball rigid body group; wherein the third motion information represents the eyeball movement, and the third motion information reflects the rotation information of the left and right eyeballs.

[0158] Optionally, the device 600 is further used for:

[0159] Determine the rotation center of the eyeball rigid body group according to the motion information of each Gaussian geometric primitive in the eyeball rigid body group, and use the rotation center as the eyeball center;

[0160] Taking the center of the eyeball as the origin, establish the eyeball coordinate system of the left and right eyeballs.

[0161] Optionally, the computing unit 604:

[0162] For each eyeball, determine the spatial information of the sight starting point in the eyeball coordinate system of each eyeball, and transform the spatial information from the eyeball coordinate system to the world coordinate system according to a preset transformation matrix;

[0163] The parametric equations of the left and right eyeballs in the world coordinate system are constructed according to the transformed spatial information, and the intersection point of the sight lines of the left and right eyeballs in the world coordinate system is calculated.

[0164] The spatial information includes the relative position and direction vector of the starting point of the line of sight relative to the center of the eyeball.

[0165] Optionally, the computing unit 604:

[0166] Calculate the product of the preset transformation matrix and the relative position to obtain the transformation position;

[0167] Calculate the product of the transformed matrix after rotation and the direction vector to obtain the transformed vector;

[0168] The transformed spatial information includes a transformed position and a transformed vector.

[0169] Optionally, the computing unit 604:

[0170] For the left eyeball, a first parameter equation is constructed based on the sum of the product of the transformation vector of the left eyeball and the first parameter and the transformation position of the left eyeball;

[0171] For the right eyeball, construct a second parameter equation based on the sum of the product of the transformation vector of the right eyeball and the second parameter and the transformation position of the right eyeball;

[0172] The first parameter and the second parameter are optimized so that the first parameter equation and the second parameter equation are equal to each other, so as to calculate the line of sight intersection point.

[0173] Figure 6 The sight line estimation device based on Gaussian geometric primitives of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0174] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7, which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The electronic device 700 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0175] like Figure 7 As shown, the electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703 to implement a line of sight estimation method based on Gaussian geometric primitives as in an embodiment of the present disclosure. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0176] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0177] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the line of sight estimation method based on Gaussian geometric primitives as above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0178] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0179] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0180] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0181] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps of the above embodiments.

[0182] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0185] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0186] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "comprises one..." do not exclude the presence of other identical elements in the process, method, article or gateway that includes the elements.

[0188] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight estimation method based on Gaussian geometric primitives, characterized in that: include: Obtain a sequence of images continuously collected when the left and right eyeballs move; Performing three-dimensional reconstruction on the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives; Performing motion analysis on Gaussian geometric primitives in the left and right eyeball models to determine rotation information of the left and right eyeballs; Calculate the sight line intersection point of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin; The line of sight direction is estimated using the line of sight intersection point to obtain a line of sight estimation result.

2. The method according to claim 1, characterized in that The image sequence includes a plurality of frames of images, and the three-dimensional reconstruction of the left and right eyeballs according to the image sequence to generate left and right eyeball models includes: Representing the left and right eyeballs in each frame of image as a set of Gaussian geometric primitives, wherein each Gaussian geometric primitive represents an area on the surface of the eyeball and has at least one parameter of position, size, posture, color and transparency; Set the multiple groups of Gaussian geometric primitives corresponding to the multiple frames of images to share the same color and the same transparency; Wherein, the left and right eyeball models include the multiple groups of Gaussian geometric primitives.

3. The method according to claim 2, characterized in that After representing the left and right eyeballs in each frame of image as a group of Gaussian geometric primitives, for each group of Gaussian geometric primitives, the method further includes: Rendering each group of Gaussian geometric primitives according to pre-set initial parameters to generate a rendered image; wherein the initial parameters are set randomly, according to historical parameters, or according to a depth image estimated by a neural network model; Calculating the difference between the rendered image and each frame image to obtain an error loss; The parameters of each group of Gaussian geometric primitives are updated according to the error loss to obtain the updated parameters of each group of Gaussian geometric primitives.

4. The method according to claim 1, characterized in that: The performing motion analysis on the Gaussian geometric primitives in the left and right eyeball models to determine the rotation information of the left and right eyeballs includes: Based on the spatial distance and relative position relationship between each Gaussian geometric primitive in the left and right eyeball models, construct an adjacency matrix of multiple frames of images in the image sequence; wherein the adjacency matrix reflects the adjacency relationship between each Gaussian geometric primitive in each frame of the image; According to the adjacency matrix, every two adjacent Gaussian geometric primitives are divided into initial groups to obtain a plurality of initial groups; wherein the plurality of initial groups do not contain repeated Gaussian geometric primitives; Performing stability detection on the multiple initial groups, and merging the multiple initial groups according to the obtained detection results to obtain multiple rigid body groups; Perform motion decomposition on the multiple rigid body groups to determine rotation information of the left and right eyeballs.

5. The method according to claim 4, characterized in that The multiple initial groups are merged according to the obtained detection results to obtain multiple rigid body groups, including: Merge all initial groups whose detection results show consistent relative distance changes into stable groups to obtain multiple stable groups; For each stable group, the process of adjacency matrix construction, initial group division and stability detection is repeated until multiple rigid body groups are obtained; wherein the multiple rigid body groups at least include an eye periorbital rigid body group and an eyeball rigid body group.

6. The method according to claim 5, characterized in that The performing motion analysis on the plurality of rigid body groups to determine the rotation information of the left and right eyeballs includes: Calculating the center of mass position of the eyeball rigid body group in each frame image, and determining first motion information of the eyeball rigid body group according to the center of mass position; wherein the first motion information represents head motion and eyeball motion, the head motion includes head translation, and the eyeball motion includes eyeball rotation; Calculating the center of mass position of the periocular rigid body group in each frame of the image, and determining second motion information of the periocular rigid body group according to the center of mass position; wherein the second motion information represents the head motion; According to the first motion information and the second motion information, the motion difference between the eyeball rigid body group and the periorbital rigid body group is calculated to obtain the third motion information of the eyeball rigid body group; wherein the third motion information represents the eyeball movement, and the third motion information reflects the rotation information of the left and right eyeballs.

7. The method according to claim 6, characterized in that Before calculating the sight line intersection point of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, the method further includes: Determine the rotation center of the eyeball rigid body group according to the third motion information, and use the rotation center as the eyeball center; An eyeball coordinate system of the left and right eyeballs is established with the eyeball center as the origin.

8. The method according to claim 1, characterized in that The calculating, according to the eyeball coordinate systems of the left and right eyeballs, the sight line intersection point of the left and right eyeballs in the world coordinate system comprises: For each eyeball, determine the spatial information of the sight line starting point in the eyeball coordinate system of each eyeball, and transform the spatial information from the eyeball coordinate system to the world coordinate system according to a preset transformation matrix; The parametric equations of the left and right eyeballs in the world coordinate system are constructed according to the transformed spatial information, and the intersection point of the sight lines of the left and right eyeballs in the world coordinate system is calculated.

9. The method according to claim 8, characterized in that The spatial information includes the relative position and direction vector of the sight starting point relative to the center of the eyeball, and the spatial information is transformed from the eyeball coordinate system to the world coordinate system according to a preset transformation matrix, including: Calculating the product of a preset transformation matrix and the relative position to obtain a transformation position; Calculate the product of the transformed matrix after rotation and the direction vector to obtain a transformed vector; The transformed spatial information includes the transformed position and the transformed vector.

10. The method according to claim 9, characterized in that The step of constructing parameter equations of the left and right eyeballs in the world coordinate system according to the transformed spatial information and calculating the sight line intersection point of the left and right eyeballs in the world coordinate system comprises: For the left eyeball, construct a first parameter equation based on the sum of the product of the transformation vector of the left eyeball and the first parameter and the transformation position of the left eyeball; For the right eyeball, construct a second parameter equation based on the sum of the product of the transformation vector of the right eyeball and the second parameter and the transformation position of the right eyeball; The first parameter and the second parameter are optimized so that the first parameter equation and the second parameter equation are equal to each other, so as to calculate the line of sight intersection point.

11. A sight line calculation device based on Gaussian geometric primitives, characterized in that: include: An acquisition unit, used for acquiring a sequence of images continuously acquired when the left and right eyeballs move; A three-dimensional reconstruction unit, used for performing three-dimensional reconstruction on the left and right eyeballs according to the image sequence to generate left and right eyeball models, wherein the left and right eyeball models are at least one set of dynamically changing Gaussian geometric primitives; A motion decomposition unit, used for performing motion analysis on Gaussian geometric primitives in the left and right eyeball models to determine rotation information of the left and right eyeballs; A calculation unit, configured to calculate the sight line intersection point of the left and right eyeballs in the world coordinate system according to the eyeball coordinate system of the left and right eyeballs, wherein the eyeball coordinate system is constructed based on the rotation information with the eyeball center of the left and right eyeballs as the origin; The estimation unit is used to estimate the sight line direction by using the sight line intersection point to obtain a sight line estimation result.

12. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the line of sight calculation method based on Gaussian geometric primitives as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the line of sight calculation method based on Gaussian geometric primitives as described in any one of claims 1 to 10 are implemented.

Images