Sight line calculation method and device, equipment and storage medium

Abstract

The invention relates to a sight line calculation method and device, equipment and a storage medium. The sight line calculation method comprises the following steps: acquiring an eye picture sequence; the eye picture sequence comprises multiple frames of pictures which are continuously collected; performing region classification on the multiple frames of pictures, and dividing each frame of picture into multiple types of regions; modeling the surfaces of the various types of regions through a two-dimensional Gaussian function to generate an eye Gaussian model; the eye Gaussian model comprises multiple groups of Gaussian primitives reflecting eye movement; and estimating a motion center of the multiple groups of Gaussian primitives, and performing sight line calculation according to the motion center. According to the method provided by the invention, the motion center of the Gaussian primitive is estimated by constructing the eye Gaussian model, so that the sight line direction of the user is obtained, training of a neural network and extraction and regression of complex eyeball features are avoided, and universality and flexibility of sight line calculation are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a line of sight calculation method, device, equipment and storage medium. Background Art

[0002] Gaze computing has a wide range of applications. By calculating the user's gaze direction, more natural and intuitive human-computer interaction can be achieved. For example, in games, virtual reality or augmented reality applications, the system can adjust the scene, provide information or trigger specific events based on the user's gaze direction, thereby improving user experience and immersion.

[0003] In the prior art, classic image processing techniques or convolutional neural network-related algorithms are usually used to extract useful feature information from eye images, and then a correspondence between the feature information and the user's line of sight in the scene is constructed, and the user's line of sight is calculated based on the constructed correspondence.

[0004] However, the above methods are limited in the breadth and generalization ability of feature selection, and rely on the feature design experience of algorithm designers or a large amount of labeled data for training, and their universality and flexibility are poor. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present disclosure provide a line of sight calculation method, device, equipment and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a line of sight calculation method, including:

[0007] Acquire an eye image sequence; wherein the eye image sequence includes multiple frames of images collected continuously;

[0008] Performing region classification on the multiple frames of images, dividing each frame of the image into multiple types of regions;

[0009] Modeling the surfaces of the multiple types of regions by two-dimensional Gaussian functions to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements;

[0010] The motion centers of the plurality of groups of Gaussian primitives are estimated, and line of sight calculation is performed based on the motion centers.

[0011] Optionally, the performing region classification on the multiple frames of images to divide each frame of images into multiple types of regions includes:

[0012] Determine the optical flow information of the multiple frames of images; wherein the optical flow information represents the position change of each pixel point in the time dimension, and the time dimension refers to the time sequence of the multiple frames of images;

[0013] Each frame of the image is divided into multiple types of regions according to the optical flow information; wherein the multiple types of regions represent image information of the eye including at least one structure.

[0014] Optionally, the optical flow information includes an optical flow value and a moving direction, and for each frame of the image, dividing each frame of the image into multiple types of regions according to the optical flow information includes:

[0015] Divide the area consisting of the first pixel points whose optical flow value is less than the first set threshold into a first type of area;

[0016] Divide the area consisting of the second pixel points whose optical flow value is greater than the first set threshold and less than the second set threshold and whose moving direction is the up-down direction into the second type of area;

[0017] Classify the region composed of the third pixel points whose optical flow values ​​are greater than the second set threshold and whose moving directions are consistent as a third type of region;

[0018] The multiple types of areas include the first type of areas, the second type of areas and the third type of areas.

[0019] Optionally, the surface modeling of the multiple types of regions by using a two-dimensional Gaussian function to generate an eye Gaussian model includes:

[0020] The pre-constructed two-dimensional Gaussian model is optimized according to the image information of each type of area to obtain an initial Gaussian model; wherein the two-dimensional Gaussian model includes a plurality of two-dimensional Gaussian patches, and the optimization of the two-dimensional Gaussian model refers to optimizing the tangent vector at the center position and the scale factor on the tangent vector based on the center position of each two-dimensional Gaussian patch;

[0021] Based on the initial Gaussian model, the pre-constructed surface model is optimized to obtain an eye Gaussian model; wherein the surface model is used to represent the curved surface of the two-dimensional Gaussian patch.

[0022] Optionally, the step of optimizing the pre-built surface model to obtain the eye Gaussian model includes:

[0023] The center position, the normal vector at the center position, and the curvature parameters are optimized to obtain a Gaussian model of the eye.

[0024] Optionally, when the two-dimensional Gaussian patch is a Gaussian ellipsoid patch, optimizing the pre-constructed surface model to obtain the eye Gaussian model includes:

[0025] Based on the radius of the Gaussian ellipsoid patch, the center position and the scale factor are optimized to obtain a Gaussian model of the eye.

[0026] Optionally, estimating the motion centers of the multiple groups of Gaussian primitives includes:

[0027] Determine a plurality of rigid body groups according to the correspondence between the pixel points in the plurality of types of regions and the plurality of groups of Gaussian basis elements; wherein the plurality of rigid body groups include an eye skin rigid body group and an eyeball rigid body group, the at least one structure includes an eye skin and an eyeball, and the rigid body group refers to a group of Gaussian basis elements that remain relatively still;

[0028] Based on the periocular skin rigid body group and the eyeball rigid body group, the motion changes caused by the head movement are stripped off, and the motion changes caused by the eyeball movement are retained to obtain the motion information of the eyeball rigid body group;

[0029] The movement center of the eyeball rigid body group is determined according to the movement information.

[0030] In a second aspect, an embodiment of the present disclosure provides a sight line calculation device, including:

[0031] An acquisition unit, configured to acquire an eye image sequence; wherein the eye image sequence includes a plurality of frames of images acquired continuously;

[0032] A region classification unit, used to perform region classification on the multiple frames of images, and divide each frame of images into multiple types of regions;

[0033] A model building unit, used for modeling the surfaces of the multiple types of regions by using a two-dimensional Gaussian function to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements;

[0034] The sight line calculation unit is used to estimate the motion centers of the multiple groups of Gaussian basis elements and perform sight line calculation based on the motion centers.

[0035] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0036] Memory;

[0037] Processor; and

[0038] Computer programs;

[0039] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0040] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0041] The method disclosed herein provides a line of sight calculation method, comprising: obtaining an eye picture sequence; wherein the eye picture sequence includes multiple frames of pictures collected continuously; performing regional classification on the multiple frames of pictures, dividing each frame of pictures into multiple categories of regions; modeling the surfaces of the multiple categories of regions through a two-dimensional Gaussian function to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis units that reflect eye movements; estimating the motion centers of the multiple groups of Gaussian basis units, and performing line of sight calculation based on the motion centers. The method provided in the present application estimates the motion centers of Gaussian basis units by constructing an eye Gaussian model, and then obtains the user's line of sight direction, thereby avoiding the training of neural networks and the extraction and regression of complex eyeball features, and improving the universality and flexibility of line of sight calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of a line of sight calculation method provided by an embodiment of the present disclosure;

[0045] Figure 2 for Figure 1 A schematic diagram of a detailed flow of S102 in a line of sight calculation method is shown;

[0046] Figure 3 for Figure 1 A schematic diagram of a detailed flow chart of S103 in a line of sight calculation method is shown;

[0047] Figure 4 A schematic diagram of the structure of a sight line calculation device provided in an embodiment of the present disclosure;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0051] For line of sight calculation, one type of algorithm in the prior art uses classic image processing technology to extract useful feature information from eye images, and then constructs a correspondence between the feature information and the user's line of sight in the scene, and estimates the line of sight point used based on the constructed correspondence. The advantage of this method is that, because of the algorithm designer's prior knowledge, it is possible to manually and accurately select the more critical feature information from the collected data, and then design a suitable method for extraction. With hardware design that is more specifically optimized for the algorithm, it can quickly adapt to a small number of uses on a device. However, this algorithm requires the algorithm designer to have rich domain knowledge to manually design features, and its feature selection is limited in breadth and generalization ability, making it difficult to adapt to more complex scenarios and data distributions.

[0052] Another type of algorithm uses commonly used convolutional neural network-related algorithms to extract feature information from images, construct a correspondence between feature information and the user's line of sight in the scene, and estimate the line of sight based on the constructed correspondence. The advantage of this method is that it can automatically learn feature representations from data without manual design; it has strong adaptability and can adapt to complex scenes and data distribution; as the data increases, the performance can be continuously improved, but this algorithm requires a large amount of labeled data for training, and the data preparation cost is high.

[0053] In view of the above technical problems, the present disclosure provides a line of sight calculation method, which is described in detail through one or more of the following embodiments.

[0054] The line of sight calculation method provided by the embodiment of the present disclosure can be applied to the eyeball line of sight calculation scenario. The method can be performed by a line of sight calculation device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device. Among them, the electronic device can include but is not limited to mobile terminals such as smart phones, laptops, digital broadcast receivers, personal digital assistants (PDA), tablet computers (Tablet Personal Computer, Tablet PC), PMP (portable multimedia player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, smart home devices, etc.

[0055] Figure 1A flow chart of a line of sight calculation method provided in an embodiment of the present disclosure, specifically including the following steps: Figure 1 The following steps are shown:

[0056] S101. Obtain an eye image sequence; wherein the eye image sequence includes a plurality of frames of images collected continuously.

[0057] It is understandable that the user's eyes are continuously captured by the image capture device to obtain a sequence of user's eye images, wherein the eye image sequence includes multiple frames of images. Specifically, continuous frame images of a single eye of the user can be captured, and the eye includes multiple structures, such as eyelids, eyebrows, eyelashes, pupils, irises, corneas, etc.

[0058] In an application scenario, in virtual reality (VR) glasses, a camera capable of taking pictures of the user's eyes is arranged around an imaging component that the user is looking at, and the camera is used to collect a single eye picture of the user.

[0059] Optionally, the specific process of collecting the eye image sequence is: displaying a preset picture or playing a preset video on the screen, and collecting continuous frame images when the user watches the screen. The preset video can be a movie that can guide the user to perform a certain eye rotation movement within the field of vision.

[0060] S102: performing region classification on multiple frames of images, and dividing each frame of images into multiple types of regions.

[0061] It is understandable that, based on the above S101, multiple frames of images are classified by regions, that is, each frame of image is divided into multiple types of regions, each region represents the image information of at least one structure of the eye, such as dividing a region where eyelashes are located, a region where eyeballs are located, and a region where eyebrows are located in a frame of image. Among them, a frame of image refers to a 2D image, and regional classification of 2D images can improve the accuracy and efficiency of subsequent eye Gaussian model reconstruction.

[0062] S103, modeling the surfaces of multiple types of regions using a two-dimensional Gaussian function to generate an eye Gaussian model.

[0063] The eye Gaussian model includes multiple groups of Gaussian basis units that reflect eye movements.

[0064] It is understandable that, based on the above S102, for each type of area, the surface of the area is modeled by a two-dimensional Gaussian function, and a Gaussian model of the eye is generated by modeling multiple types of areas. For multiple frames of images, a modeling order can be set for each type of area to improve calculation flexibility and reduce the amount of calculation. For example, a type of area where the eyeball is located is modeled first, and then another type of area where the eyelid is located is modeled. Using a two-dimensional Gaussian function to perform real-time three-dimensional reconstruction of the eyeball avoids the need for neural network training on large data sets.

[0065] Specifically, according to the spatiotemporal continuity of the eyeball in multiple frames of images, the eyeball Gaussian model is obtained by representing the change of the two-dimensional position information of the eyeball over time through a 2D Gaussian function.

[0066] S104, estimating the motion centers of the multiple groups of Gaussian basis elements, and performing line of sight calculations based on the motion centers.

[0067] It is understandable that, based on the above S103, multiple groups of Gaussian primitives reflect the dynamic changes of the eye, such as a group of Gaussian primitives of the eyeball rotates synchronously with the eyeball, so the center of motion can be the rotation center of multiple Gaussian primitives, and the rotation center is also the center of the eyeball. Subsequently, the rotation information of the eyeball is converted into line of sight information. Specifically, the line of sight can be calculated through a calibration process or by using the method of the intersection of binocular lines of sight, and the specific implementation process is not described in detail.

[0068] Optionally, the above-mentioned estimation of the motion centers of multiple groups of Gaussian basis elements can be specifically implemented through the following steps:

[0069] According to the correspondence between pixel points and multiple groups of Gaussian primitives in multiple types of regions, multiple rigid body groups are determined; wherein the multiple rigid body groups include an eyeball skin rigid body group and an eyeball rigid body group, at least one structure includes the eyeball skin and the eyeball, and the rigid body group refers to a group of Gaussian primitives that remain relatively still; based on the eyeball skin rigid body group and the eyeball rigid body group, the motion changes caused by head movement are stripped off, and the motion changes caused by eyeball movement are retained to obtain the motion information of the eyeball rigid body group; the motion center of the eyeball rigid body group is determined according to the motion information.

[0070] It can be understood that according to the correspondence between pixels and Gaussian primitives, the Gaussian primitives corresponding to each structure of the eye are determined, and all Gaussian primitives corresponding to each structure constitute a rigid body group of the structure. The rigid body group refers to a group of Gaussian primitives that remain relatively still. Among them, for a frame of an image, a certain surface of the image is represented as multiple Gaussian primitives, and a certain surface is composed of multiple pixels. One Gaussian primitive corresponds to at least one pixel. Therefore, on the same surface, there is a corresponding relationship between pixels and Gaussian primitives. In addition, each frame of the image is divided into multiple regions according to the structure of the eye. Subsequently, multiple Gaussian primitives that constitute the rigid body group of each structure of the eye can be further identified according to the region, which improves the recognition speed of Gaussian primitives and further reduces the amount of calculation. The identified rigid body groups are divided into the periocular skin rigid body group and the eyeball rigid body group. The motion model of the periocular skin rigid body group represents the head movement, and the motion model of the eyeball rigid body group represents the head movement and the eyeball movement. On this basis, the motion changes caused by the head movement in the motion information of the eyeball rigid body group are stripped off by subtracting the motion information of the periocular skin rigid body group, and only the rotation information of the motion changes caused by the eyeball movement in the motion information of the eyeball rigid body group is retained. Subsequently, the rotation center is calculated based on the continuous angle distribution information (rotation information) of multiple Gaussian primitives in the eyeball rigid body group.

[0071] It is understandable that one Gaussian primitive corresponds to multiple pixel points, and multiple pixel points can represent an image area, such as the eye area. In this case, the pixel points are classified, and then the Gaussian balls are classified according to the correspondence between the Gaussian balls and the pixel points, completing the preliminary division of the rigid body group and obtaining multiple initial groups, that is, the Gaussian primitives are roughly classified.

[0072] Optionally, multiple initial groups are subjected to motion decomposition to determine multiple rigid body groups, which can be specifically achieved through the following steps:

[0073] The relative motion of each Gaussian geometric primitive in each initial group is analyzed to construct a motion graph; and at least one rigid body group is identified based on the motion graph.

[0074] It is understandable that for each initial group, the relative motion of the Gaussian primitives is analyzed and a motion graph is constructed, wherein the motion graph reflects the dynamic changes of each Gaussian primitive over time. Subsequently, a clustering algorithm (k-means or DBSCAN) is used to continue to identify at least one rigid body group from the Gaussian primitives included in the initial group, that is, each initial group is finely classified, multiple initial groups are updated, and multiple rigid body groups are obtained. Among them, a rigid body group refers to a group of Gaussian primitives that remain relatively still in adjacent image frames. The eye includes a variety of structures, such as eyeballs, eyelids, eyelashes, eye corners, and eyebrows, etc. Each structure has a corresponding rigid body group. During the eye movement process, the multiple Gaussian primitives representing the eyebrows are relatively still, that is, the overall motion of the multiple Gaussian primitives is consistent. Subsequently, a motion model is established for each rigid body group, and the motion model of at least one rigid body group reflects the head movement and / or eyeball movement.

[0075] Optionally, based on the periorbital skin rigid body group and the eyeball rigid body group, the motion changes caused by the head movement are stripped off, and the motion changes caused by the eyeball movement are retained to obtain the motion information of the eyeball rigid body group, which can be specifically achieved by the following steps:

[0076] A motion model is established for each rigid body group; based on the motion model of the periocular skin rigid body group and the motion model of the eyeball rigid body group, the motion changes caused by head movement are stripped away by subtraction to estimate the motion information of the eyeball rigid body group; wherein, head movement includes head translation, eye movement includes eye rotation, the motion model of the periocular skin rigid body group represents the head movement, and the motion model of the eyeball rigid body group represents the head movement and eye movement.

[0077] It is understandable that the eyeball rigid body group is mainly affected by the movement of the head and the eyeball, and can be represented by a rotation matrix or a quaternion, that is, the eyeball will not only move with the head but also rotate. The periocular skin rigid body group is mainly affected by the movement of the head, and can be represented by a translation, that is, the periocular skin (such as eyebrows) will not rotate, but will only move with the head. Therefore, the motion information of the periocular skin rigid body group includes relevant information of the head movement, and the motion information of the eyeball rigid body group includes relevant information of the head movement and the eyeball rotation. Subsequently, parameters such as head movement and eyeball rotation are estimated by minimizing the error function distribution, and the error function is minimized such as minimizing the sum of squared displacements between the internal points of each rigid body group. Specifically, after determining the Gaussian basis element corresponding to the eyeball, the motion changes caused by the head movement in the motion information of the eyeball rigid body group can be stripped off by subtracting the motion information of the periocular skin rigid body group, and only the rotational motion information of the eyeball rigid body group is retained.

[0078] It is understandable that, based on the motion information decomposed from each rigid body group and the distribution of the motion components occurring at each continuous moment on each Gaussian primitive, it is possible to infer the eyeball area or the skin around the eye area to which each Gaussian primitive belongs. For subsequent line of sight tracking, the motion information related to the eyeball area is mainly needed. After determining the Gaussian primitive to which the eyeball area belongs, the motion information caused by the head movement in the motion information of the eyeball rigid body group can be stripped off by subtraction, leaving only the rotation information of the eyeball rigid body group. The motion information can then be used to calculate the line of sight direction, and the accuracy of the system can be improved through calibration and verification.

[0079] The line of sight calculation method provided by the embodiment of the present disclosure performs regional classification on the collected continuous frame images, and then reconstructs them in different regions to improve the reconstruction accuracy and efficiency. Subsequently, based on the correspondence between the pixel points and the Gaussian primitives, a group of Gaussian primitives corresponding to the eyeballs is found, and the eye movement is decomposed by identifying the rigid body group in the Gaussian primitives, and the rotation information of the eyeballs is accurately extracted, thereby achieving high-precision eye tracking. Subsequently, the line of sight information is calculated based on the rotation information, which not only reduces the computational complexity, but also achieves high-precision eye tracking.

[0080] Based on the above embodiments, Figure 2 for Figure 1 The detailed flow diagram of S102 in a line of sight calculation method shown in FIG. 1 is a flow diagram of S102 in which the multiple frames of images are classified into multiple regions, specifically including the following: Figure 2 The following steps are shown:

[0081] It is understandable that region classification can be performed using an optical flow method, a neural network model and / or a machine vision algorithm, and the optical flow method is used to track the correspondence between each pixel between adjacent frames.

[0082] S201: Determine optical flow information of multiple frames of images.

[0083] Among them, the optical flow information represents the position change of each pixel in the time dimension, and the time dimension refers to the time series of multi-frame image acquisition.

[0084] It can be understood that the optical flow information of multiple frames of images is determined by the optical flow algorithm, wherein the optical flow information represents the temporal changes of each pixel in the multiple frames of images, that is, the position changes of each pixel in the multiple frames of images, such as the position of pixel 1 in the first frame of the image is (1, 2), and the position in the second frame of the image is (1, 5).

[0085] Preferably, several frames with optical flow values ​​lower than a preset threshold are selected from multiple frames of images to construct the eyeball Gaussian model. Several frames with optical flow values ​​lower than the preset threshold are images obtained when the user is looking at a certain point, and the pixel changes between adjacent frames are small.

[0086] S202: Divide each frame of the image into multiple types of regions according to the optical flow information.

[0087] The multiple types of regions represent image information of the eye including at least one structure.

[0088] It is understandable that, based on the above S201, each frame of the picture is divided into multiple types of regions according to the optical flow information, and each type of region represents the picture information of at least one structure of the eye. The following embodiment takes a frame of the picture as an example to explain the classification of the picture regions in detail.

[0089] The optical flow information includes the optical flow value and the moving direction.

[0090] Optionally, for each frame of the image, each frame of the image is divided into multiple types of regions according to the optical flow information, which can be specifically achieved through the following steps:

[0091] The area composed of first pixel points whose optical flow value is less than a first set threshold is divided into a first type of area; the area composed of second pixel points whose optical flow value is greater than the first set threshold and less than the second set threshold and whose moving direction is up and down is divided into a second type of area; the area composed of third pixel points whose optical flow value is greater than the second set threshold and whose moving direction is consistent is divided into a third type of area; wherein the multiple types of areas include the first type of area, the second type of area and the third type of area.

[0092] Understandably, in the eye tracking scene, each frame of the picture can be divided into three types of areas. Specifically: the first type of area is the edge area, which is mainly the area corresponding to the eyelids, eyebrows and other eye structures far away from the eyelashes. The first pixel in this area has almost no movement or a very small movement, and the basis for determining that a pixel in a picture belongs to this area is that in the continuous pictures collected, the optical flow corresponding to the first pixel is mainly in the peripheral area of ​​the entire picture, and the amplitude of the optical flow data is very small, that is, the optical flow value is less than the first set threshold, and the first pixel has almost no movement or only moves within a small range. The second type of area is the transition area, which is mainly the area corresponding to the eyelashes and the eyelids near the eyelashes. The second pixel in this area mainly reciprocates in the up and down directions. The judgment basis is that during the entire collection process, the optical flow value corresponding to the second pixel is relatively small, that is, greater than the first set threshold and less than the second set threshold, and the movement direction continues to move up and down, there is a relatively large interval in the time when the movement occurs, and the duration of the movement is also relatively short. The third type of area is the core area, which is mainly the area corresponding to the pupil, iris, cornea and other eye structures. The third pixel in this area will move greatly because the eye rotates with the change of sight. The judgment is based on the fact that in the entire image sequence, the optical flow corresponding to the movement of the third pixel has a large value, that is, the optical flow value is greater than the second set threshold, and the moving direction is relatively consistent. Especially when the calibration process is set, the moving direction of the area in the image is correlated with the relative direction of the setting position of the calibration point. Each pixel in the image is marked by regional classification, and the pixels in the third type of area can be given priority in the subsequent reconstruction process, that is, calculation is performed area by area to improve the calculation speed and reconstruction accuracy.

[0093] The line of sight calculation method provided by the embodiment of the present disclosure uses an optical flow algorithm to perform region classification on each frame of the image, thereby improving reconstruction accuracy and efficiency.

[0094] Based on the above embodiments, Figure 3 for Figure 1 FIG. 1 is a schematic diagram of a detailed flow chart of S103 in a line of sight calculation method. Optionally, a surface model of multiple types of regions is modeled by a two-dimensional Gaussian function to generate an eye Gaussian model, specifically including the following steps: Figure 3 The following steps are shown:

[0095] S301 , optimizing a pre-built two-dimensional Gaussian model according to the image information of each type of region to obtain an initial Gaussian model.

[0096] The two-dimensional Gaussian model includes a plurality of two-dimensional Gaussian patches, and the optimization of the two-dimensional Gaussian model refers to optimizing the tangent vector at the center position and the scale factor on the tangent vector based on the center position of each two-dimensional Gaussian patch.

[0097] It can be understood that for multiple frames of images, surface modeling is performed by dividing the regions by two-dimensional Gaussian functions. Specifically, the pre-constructed two-dimensional Gaussian model is optimized according to the image information of each type of region to obtain an initial Gaussian model, wherein the two-dimensional Gaussian model can be understood as a plane Gaussian function, which includes multiple two-dimensional Gaussian patches with initial parameters. The initial Gaussian model includes multiple two-dimensional Gaussian patches after parameter optimization. The parameterized representation of the initial Gaussian model is specifically shown in formula (1).

[0098] P(u,v)=P k +s u t u u+s v t v vFormula (1)

[0099] Where P(u,v) is the initial Gaussian model on the plane, (u,v) is the first dimension information and the second dimension information, P k is the center position of the two-dimensional Gaussian patch, t u and t v is the normal vector, s u and v is the scale factor.

[0100] Understandably, the two-dimensional Gaussian function can be a two-dimensional Gaussian scatter method (2D Gaussian Sphere, 2DGS), which is used to model and reconstruct geometrically accurate radiation fields from multi-view images. The core idea is to compress the three-dimensional volume into a set of two-dimensional oriented plane Gaussian disks. Unlike the three-dimensional Gaussian function, the two-dimensional Gaussian function essentially models the surface while providing geometric information consistent with the perspective. In order to accurately restore the thin surface and achieve stable optimization, a perspective-accurate two-dimensional scatter process is introduced, using ray-scatter intersection and rasterization. In addition, depth distortion and normal consistency terms are added to further improve the reconstruction quality. Based on the two-dimensional Gaussian scatter method, noise-free and fine geometric reconstruction can be achieved while maintaining competitive appearance quality, fast training speed and real-time rendering. 2DGS is used in the embodiment of the present disclosure to more effectively represent the surface of the eyeball, reduce the amount of calculation, and improve the reconstruction accuracy. In addition, 2DGS is essentially modeling the surface of the eyeball and providing geometric information consistent with the perspective, which is particularly beneficial for relatively regular surfaces such as the eyeball, and can more effectively utilize image information and reduce redundant calculations.

[0101] S302: Based on the initial Gaussian model, the pre-built surface model is optimized to obtain an eye Gaussian model.

[0102] The surface model is used to represent the curved surface of a two-dimensional Gaussian patch.

[0103] It is understandable that, based on the above S301, there is also surface information on the surface of the eyeball. In order to simulate the eyeball surface more accurately, a surface model is introduced so that it can represent the curved surface. The two-dimensional Gaussian function mentioned earlier cannot accurately describe the curvature of the eyeball. Therefore, by introducing the surface model into the optimized initial Gaussian model, a mathematical model that can simultaneously represent curvature and non-curvature is constructed, so that the constructed eyeball Gaussian model includes multiple groups of Gaussian primitives that have not only dimensional information on the plane, but also dimensional information on the surface.

[0104] Optionally, the pre-built surface model is optimized to obtain the eye Gaussian model, which can be implemented by the following steps:

[0105] The center position, the normal vector at the center position, and the curvature parameters are optimized to obtain the eye Gaussian model.

[0106] It can be understood that a form of eyeball surface representation is a parameterized representation of the eyeball Gaussian model after adding the surface model to the initial Gaussian model as shown in formula (2).

[0107]

[0108] In the formula, k u and k v is the curvature parameter, n is the normal vector, and the parameterized representation of the surface model is

[0109]

[0110] It is understandable that when the curvature parameter dimension is 0, the model degenerates into a plane. By learning and optimizing the model parameters, that is, the curvature parameters, the curved surface shape of the eyeball can be better fitted, thereby improving the accuracy of eye tracking. In addition, the Newton iteration method can be used to solve the intersection of the ray and the surface, so that the model fits the actual shape of the eyeball better and improves the accuracy of reconstruction.

[0111] Optionally, when the two-dimensional Gaussian patch is a Gaussian ellipsoid patch, the pre-constructed surface model is optimized to obtain the eye Gaussian model, which can be specifically achieved by the following steps:

[0112] Based on the radius of the Gaussian ellipsoid patch, the center position and scale factor are optimized to obtain the eye Gaussian model.

[0113] It is understandable that the two-dimensional Gaussian patch can be set to a Gaussian ellipsoid patch. The Gaussian ellipsoid patch refers to a bendable Gaussian basis element. By combining various element forms, the geometric primitives are expanded to improve the model expression ability. When the Gaussian patch is bent by adding quadratic terms, there is a problem of local fitting for the Gaussian ellipsoid patch. Considering that there are structural surfaces in the eyeball that are relatively close to the ellipsoid, the expression form of the Gaussian patch can be further expanded, and the Gaussian patch can be parameterized as a part of a sphere or ellipsoid. The spherical parameterization of the eyeball Gaussian model is shown in formula (3).

[0114]

[0115] Where R is the radius of the sphere and the normal vector n = t u ·t v , scale factor / scaling factor s u and v is controlled at t u and t v The scale in the direction allows the Gaussian patch to fit the surface of the eye more accurately. The spherical parameterization of the surface model is expressed as In addition, it can also be extended to ellipsoidal parameterization to better adapt to the shape of the eyeball.

[0116] A line of sight calculation method provided by an embodiment of the present disclosure uses a two-dimensional Gaussian function to reduce calculation complexity, introduces a surface model into the two-dimensional Gaussian model to more accurately simulate the eyeball surface, and can more effectively utilize image information and reduce redundant calculations.

[0117] Figure 4 The structure diagram of the sight line calculation device provided in the embodiment of the present disclosure is as follows. The sight line calculation device provided in the embodiment of the present disclosure can execute the processing flow provided in the sight line calculation method embodiment, such as Figure 4 As shown, the sight line calculation device 400 includes an acquisition unit 401, a region classification unit 402, a model construction unit 403 and a sight line calculation unit 404, wherein:

[0118] An acquisition unit 401 is used to acquire an eye image sequence; wherein the eye image sequence includes a plurality of frames of images collected continuously;

[0119] The region classification unit 402 is used to classify the regions of the multiple frames of images and divide each frame of images into multiple regions;

[0120] The model building unit 403 is used to model the surfaces of multiple types of regions by using a two-dimensional Gaussian function to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements;

[0121] The sight line calculation unit 404 is used to estimate the motion centers of the multiple groups of Gaussian basis cells and perform sight line calculation according to the motion centers.

[0122] Optionally, the acquiring unit 401 is used to:

[0123] Acquire an eye image sequence; wherein the eye image sequence includes multiple frames of images collected continuously;

[0124] Perform regional classification on multiple frames of images and divide each frame into multiple types of regions;

[0125] The surface of multiple regions is modeled by a two-dimensional Gaussian function to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements;

[0126] Estimate the centers of motion of multiple groups of Gaussian primitives and perform line-of-sight calculations based on the centers of motion.

[0127] Optionally, the region classification unit 402 is used to:

[0128] Determine the optical flow information of multiple frames of images; wherein the optical flow information represents the position change of each pixel point in the time dimension, and the time dimension refers to the time sequence of the acquisition of multiple frames of images;

[0129] Each frame of the image is divided into multiple types of regions according to the optical flow information; wherein the multiple types of regions represent image information of the eye including at least one structure.

[0130] The optical flow information includes the optical flow value and the moving direction.

[0131] Optionally, the region classification unit 402 is used to:

[0132] Divide the area consisting of first pixel points whose optical flow values ​​are less than a first set threshold into a first type of area;

[0133] The area consisting of the second pixel points whose optical flow value is greater than the first set threshold value and less than the second set threshold value and whose moving direction is the up-down direction is divided into the second type of area;

[0134] The region composed of third pixel points whose optical flow values ​​are greater than the second set threshold and whose moving directions are consistent is divided into a third type of region;

[0135] Among them, the multiple types of areas include first type areas, second type areas and third type areas.

[0136] Optionally, the model building unit 403 is used to:

[0137] The pre-constructed two-dimensional Gaussian model is optimized according to the image information of each type of area to obtain an initial Gaussian model; wherein the two-dimensional Gaussian model includes multiple two-dimensional Gaussian patches, and the optimization of the two-dimensional Gaussian model refers to optimizing the tangent vector at the center position and the scale factor on the tangent vector based on the center position of each two-dimensional Gaussian patch;

[0138] Based on the initial Gaussian model, the pre-constructed surface model is optimized to obtain the eye Gaussian model; wherein the surface model is used to represent the curved surface of the two-dimensional Gaussian patch.

[0139] Optionally, the model building unit 403 is used to:

[0140] The center position, the normal vector at the center position, and the curvature parameters are optimized to obtain the eye Gaussian model.

[0141] Optionally, the model building unit 403 is used to:

[0142] Based on the radius of the Gaussian ellipsoid patch, the center position and scale factor are optimized to obtain the eye Gaussian model.

[0143] Optionally, the sight line calculation unit 404:

[0144] Determine multiple rigid body groups according to the correspondence between the pixel points and the multiple groups of Gaussian basis elements in the multiple types of regions; wherein the multiple rigid body groups include an eye skin rigid body group and an eyeball rigid body group, at least one structure includes the eye skin and the eyeball, and the rigid body group refers to a group of Gaussian basis elements that remain relatively still;

[0145] Based on the rigid body group of the periocular skin and the rigid body group of the eyeball, the motion changes caused by the head movement are stripped off, and the motion changes caused by the eyeball movement are retained to obtain the motion information of the rigid body group of the eyeball;

[0146] Determine the motion center of the eyeball rigid body group based on the motion information.

[0147] Figure 4 The line of sight calculation device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0148] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5, which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0149] like Figure 5 As shown, the electronic device 500 may include a processing device 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503 to implement the line of sight calculation method of the embodiment described in the present disclosure. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0150] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0151] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the line of sight calculation method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0152] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0153] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0154] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0155] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0156] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0159] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0160] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or gateway that includes the elements.

[0162] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight calculation method, characterized in that: The method comprises: Acquire an eye image sequence; wherein the eye image sequence includes multiple frames of images collected continuously; Performing region classification on the multiple frames of images, dividing each frame of the image into multiple types of regions; Modeling the surfaces of the multiple types of regions by two-dimensional Gaussian functions to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements; The motion centers of the plurality of groups of Gaussian primitives are estimated, and line of sight calculation is performed based on the motion centers.

2. The method according to claim 1, characterized in that The performing region classification on the multiple frames of images, dividing each frame of images into multiple types of regions, includes: Determine the optical flow information of the multiple frames of images; wherein the optical flow information represents the position change of each pixel point in the time dimension, and the time dimension refers to the time sequence of the multiple frames of images; Each frame of the image is divided into multiple types of regions according to the optical flow information; wherein the multiple types of regions represent image information of at least one structure included in the eye.

3. The method according to claim 2, characterized in that The optical flow information includes an optical flow value and a moving direction. For each frame of the picture, each frame of the picture is divided into multiple types of areas according to the optical flow information, including: Divide the area consisting of the first pixel points whose optical flow value is less than the first set threshold into a first type of area; Divide the area consisting of the second pixel points whose optical flow value is greater than the first set threshold and less than the second set threshold and whose moving direction is the up-down direction into the second type of area; Classify the region composed of the third pixel points whose optical flow values ​​are greater than the second set threshold and whose moving directions are consistent as a third type of region; The multiple types of areas include the first type of areas, the second type of areas and the third type of areas.

4. The method according to claim 1, characterized in that: The surface modeling of the multiple types of regions by using a two-dimensional Gaussian function to generate an eye Gaussian model includes: The pre-constructed two-dimensional Gaussian model is optimized according to the multiple types of regions to obtain an initial Gaussian model; wherein the two-dimensional Gaussian model includes a plurality of two-dimensional Gaussian patches, and the optimization of the two-dimensional Gaussian model refers to optimizing the tangent vector at the center position and the scale factor on the tangent vector based on the center position of each two-dimensional Gaussian patch; Based on the initial Gaussian model, the pre-constructed surface model is optimized to obtain an eye Gaussian model; wherein the surface model is used to represent the curved surface of the two-dimensional Gaussian patch.

5. The method according to claim 4, characterized in that The pre-built surface model is optimized to obtain the eye Gaussian model, including: The center position, the normal vector at the center position, and the curvature parameters are optimized to obtain a Gaussian model of the eye.

6. The method according to claim 4, characterized in that In the case where the two-dimensional Gaussian patch is a Gaussian ellipsoid patch, the pre-constructed surface model is optimized to obtain the eye Gaussian model, including: Based on the radius of the Gaussian ellipsoid patch, the center position and the scale factor are optimized to obtain a Gaussian model of the eye.

7. The method according to claim 1, characterized in that The estimating the motion centers of the plurality of groups of Gaussian primitives comprises: Determine a plurality of rigid body groups according to the correspondence between the pixel points in the plurality of types of regions and the plurality of groups of Gaussian basis elements; wherein the plurality of rigid body groups include an eye skin rigid body group and an eyeball rigid body group, the eye includes the eye skin and the eyeball, and the rigid body group refers to a group of Gaussian basis elements that remain relatively still; Based on the periocular skin rigid body group and the eyeball rigid body group, the motion changes caused by the head movement are stripped off, and the motion changes caused by the eyeball movement are retained to obtain the motion information of the eyeball rigid body group; The movement center of the eyeball rigid body group is determined according to the movement information.

8. A sight line calculation device, characterized in that: The device comprises: An acquisition unit, configured to acquire an eye image sequence; wherein the eye image sequence includes a plurality of frames of images acquired continuously; A region classification unit, used to perform region classification on the multiple frames of images, and divide each frame of images into multiple types of regions; A model building unit, used for modeling the surfaces of the multiple types of regions by using a two-dimensional Gaussian function to generate an eye Gaussian model; wherein the eye Gaussian model includes multiple groups of Gaussian basis elements that reflect eye movements; The sight line calculation unit is used to estimate the motion centers of the multiple groups of Gaussian basis elements and perform sight line calculation based on the motion centers.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the line of sight calculation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sight line calculation method according to any one of claims 1 to 7 are implemented.

Images