A line-of-sight position positioning method, device and terminal equipment control method

Abstract

The present application provides a line-of-sight positioning method, device and terminal equipment control method, wherein the positioning method comprises: generating a virtual space including a user model and a terminal equipment model according to a user facial image; generating at least N actual planes perpendicular to the actual iris / pupil edge reflection surface of the user according to the user facial image and the terminal equipment model in the virtual space; adjusting the simulated landing point position of the line-of-sight of the user eyeball model on the terminal equipment model in the virtual space according to a preset rule, and determining a plurality of groups of at least N simulated planes perpendicular to the simulated iris / pupil edge reflection surface of the user for each simulated landing point position. Comparing each group of simulated planes with actual planes, and taking the simulated landing point position when the simulated planes are parallel to the actual planes as the actual landing point position of the line-of-sight of the user on the terminal equipment. The present application can improve the calculation speed and accuracy of the actual landing point position of the line-of-sight of the user on the terminal equipment.

Description

Technical Field

[0001] This article relates to the field of eye-tracking, and in particular to an eye-tracking position positioning method, device, and terminal equipment control method. Background Technology

[0002] In existing technologies, one method for gaze tracking and positioning involves: pre-determining multiple second vector maps, including iris / pupil edge curves, representing different positions of the user's gaze towards the terminal device model in a virtual space; determining a first vector map, including the iris / pupil edge curves, based on real-time captured user images; and comparing the first and second vector maps to determine the location of the user's gaze on the terminal device. This method lacks a definitive basis for comparing the first and second vector maps, resulting in low recognition accuracy. Furthermore, the first vector map itself is based on the difference in grayscale values ​​between pixels representing the iris and sclera in the user image's eye region, with a threshold set to determine this difference. A smooth curve is then used to outline the threshold values. However, the threshold setting directly affects the shape of the iris / pupil edge curve in the first vector map. Changes in the threshold setting can be seen as a shift (expansion or reduction) in the direction perpendicular to the tangent line at each point on the iris / pupil edge curve, introducing uncertainty into the shape of the iris / pupil edge curve and affecting the comparison results.

[0003] In addition, rendering multiple second vector graphics will place a huge computational burden on the computer. Summary of the Invention

[0004] This paper addresses the issues of low accuracy and slow speed in the positioning of the user's gaze on the terminal device in existing technologies.

[0005] To address the aforementioned technical problems, the first aspect of this paper provides a line-of-sight positioning method, including:

[0006] Capture user facial images;

[0007] Based on the user's facial image or the principle of structured light, a virtual space is generated, including a user model and a terminal device model, wherein the user model includes at least a user's eyeball model and user's facial feature points;

[0008] Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface.

[0009] In the virtual space, the simulated landing point position of the user's eyeball model's line of sight on the terminal device model is adjusted according to the pre-set rules. For each simulated landing point position, multiple sets of simulated planes are determined. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface.

[0010] Compare each set of simulated planes with the actual planes, and take the simulated landing point position when the simulated plane is parallel to the actual plane as the actual landing point position of the user's line of sight on the terminal device.

[0011] In a further embodiment of this document, a virtual space including a user model and a terminal device model is generated based on the user's facial image, including:

[0012] The user's facial image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance;

[0013] Based on the user's facial image within the rectangular frame, determine the user's facial feature points within the rectangular frame;

[0014] Based on the user's facial feature points within the rectangular frame and the camera focus in the terminal device model, establish the reflection lines of the user's facial feature points within the rectangular frame;

[0015] Adjust the position of the user model so that the reflection lines of the facial feature points in the user model coincide with the reflection lines of the user feature points within the rectangular frame, thereby obtaining the spatial positional relationship between the user model and the terminal device model;

[0016] Based on the spatial relationship between the user model and the terminal device model, a virtual space including the user model and the terminal device model is generated.

[0017] In a further embodiment of this document, at least N actual planes are generated based on the user's facial image and the terminal device model in the virtual space, including:

[0018] Based on the user's facial image and the terminal device model in the virtual space, determine the user's actual iris / pupil edge reflection surface;

[0019] Determine at least N actual cross-sections of the actual iris / pupil edge reflection surface of the user, and the line segments intersecting each actual cross-section with the actual iris / pupil edge reflection surface of the user;

[0020] Based on each actual cut surface and its associated intersecting line segments, determine the actual plane that is perpendicular to each actual cut surface and contains the associated intersecting line segments.

[0021] In a further embodiment of this document, determining the actual iris / pupil edge reflection surface of the user based on the user's facial image and the terminal device model in the virtual space includes:

[0022] The user's facial image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance;

[0023] Based on the user's facial image within the rectangular frame, determine the user's actual iris / pupil edge curve;

[0024] Using the camera focus point in the terminal device model as the vertex and the actual iris / pupil edge curve of the user as the guideline, a conical surface is obtained, which is the actual iris / pupil edge reflection surface of the user.

[0025] In a further embodiment of this document, at least N actual planes are generated based on the user's facial image and the terminal device model in the virtual space, including:

[0026] The user's facial image or facial vector image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance;

[0027] Determine the actual iris / pupil edge curve of the user based on the user's facial image or facial vector image within the rectangular frame;

[0028] Establish at least N actual tangent lines and tangent points for the actual iris / pupil edge curve of the user;

[0029] Connect the tangent point to the focal point of the camera in the terminal device model to obtain the actual connection line between the tangent point and the focal point of the camera in the terminal device model;

[0030] Determine the actual tangent based on each actual connecting line and its associated actual tangent;

[0031] Based on each actual cut surface and its associated actual connecting line, an actual plane perpendicular to each actual cut surface and containing the associated actual connecting line is determined. In a further embodiment of this document, for any simulated landing point location, at least N simulated planes are determined, including:

[0032] For any simulated landing point, based on the simulated iris / pupil edge curve in the user's eyeball model at that simulated landing point, establish the simulated iris / pupil edge reflection surface at that simulated landing point.

[0033] Determine at least N simulated cross-sections of the user-simulated iris / pupil edge reflection surface and the line segments intersecting each simulated cross-section with the user-simulated iris / pupil edge reflection surface;

[0034] Based on each simulated cross section and its associated intersecting line segments, determine a simulated plane that is perpendicular to each simulated cross section and contains the associated intersecting line segments.

[0035] In a further embodiment of this paper, for any simulated landing point location, at least N simulated planes are determined, including:

[0036] For any simulated landing point location, the simulated iris / pupil edge curve of the user at that simulated landing point location is determined based on the user's eyeball model at that simulated landing point location.

[0037] Establish at least N simulated tangents and tangent points for the simulated iris / pupil edge curve of the user;

[0038] Connect the tangent point to the focal point of the camera in the terminal device model to obtain a simulated connection line between the tangent point and the focal point of the camera in the terminal device model;

[0039] Determine the simulated tangent based on each connecting line and its associated simulated tangent;

[0040] Based on each simulated section and its associated simulated connecting lines, determine a simulated plane that is perpendicular to each simulated section and contains the associated simulated connecting lines.

[0041] In a further embodiment of this paper, the actual plane and the simulated plane are represented by the normal vectors of their respective planes. The normal vector of the actual plane is a known quantity, and the normal vector of each simulated plane i includes point C on the user-simulated iris / pupil edge curve. i The unknown quantity of the coordinates, i, has a range of 1-N;

[0042] Comparing each set of simulated planes with the actual planes, the simulated landing point position when the simulated plane is parallel to the actual plane is taken as the actual landing point position of the user's line of sight on the terminal device, including:

[0043] By making the normal vector of the actual plane parallel to the normal vector of the simulated plane, point C on the simulated iris / pupil edge curve of the user is calculated. i ,…,C N The coordinates;

[0044] According to point C i ,…,C N The coordinates and equations of the user-simulated iris / pupil edge curve motion trajectory are used to calculate the exact equation of the user-simulated iris / pupil edge curve; wherein, the equations of the user-simulated iris / pupil edge curve motion trajectory include: variable equations that determine the spatial position of the user-simulated iris / pupil edge curve, and equations that determine point C on the user-simulated iris / pupil edge curve. i The equation of variables;

[0045] The position of the visual axis of the user's eye model is determined based on the exact equation of the simulated iris / pupil edge curve.

[0046] Based on the position of the visual axis of the user's eyeball model, the actual landing point of the user's line of sight on the terminal device is determined.

[0047] In a further embodiment of this document, the user eye module includes: an iris / pupil edge curve and a visual axis.

[0048] The second aspect of this paper also provides a method for locating the line of sight, including:

[0049] Capture user facial images;

[0050] Based on the user's facial image or the principle of structured light, a virtual space is generated, including a user model and a terminal device model, wherein the user model includes at least a user's eyeball model and user's facial feature points;

[0051] Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface.

[0052] In the virtual space, at least N simulated planes passing through the focal point of the camera in the terminal device model are established; taking the focal point of the camera in the terminal device model as a fixed point, at least N simulated planes are rotated until they are parallel to at least N actual planes;

[0053] The simulated landing point of the user's eyeball model's line of sight on the terminal device model is changed, such that at least N simulated cross-sections of the simulated iris / pupil edge reflection surface are perpendicular to the N simulated planes, wherein the simulated iris / pupil edge reflection surface is determined by the simulated iris / pupil edge curve in the user's eyeball model;

[0054] The simulated landing point of the user's eyeball model's gaze on the terminal device model is taken as the landing point of the user's gaze on the terminal device.

[0055] The third aspect of this document provides a terminal device control method, including:

[0056] Using the line-of-sight positioning method described in any of the foregoing embodiments, the actual landing point of the user's line of sight on the terminal device is determined;

[0057] The operation instructions for the terminal device are generated based on the actual position of the user's line of sight on the terminal device.

[0058] The fourth aspect of this article provides a line-of-sight positioning device, comprising:

[0059] The acquisition module is used to acquire the user's facial images;

[0060] The modeling module is used to generate a virtual space including a user model and a terminal device model based on the user's facial image or the principle of structured light. The user model includes at least a user's eyeball model and user's facial feature points.

[0061] The calculation module is used to generate at least N actual planes based on the user's facial image and the terminal device model in the virtual space, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface;

[0062] The simulation module is used to adjust the simulated landing point position of the user's eyeball model's line of sight on the terminal device model in a virtual space according to pre-set rules. For each simulated landing point position, multiple sets of simulated planes are determined. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface.

[0063] The comparison module is used to compare each group of simulated planes with the actual planes, and to take the simulated landing point position when the simulated planes are parallel to the actual planes as the actual landing point position of the user's line of sight on the terminal device.

[0064] A fifth aspect of this document provides a computer device including a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, performs instructions for the line-of-sight positioning method according to any of the foregoing embodiments.

[0065] A sixth aspect of this document provides a computer storage medium having a computer program stored thereon, which, when executed by a processor of a computer device, executes instructions for the line-of-sight positioning method according to any of the foregoing embodiments.

[0066] The inventors of this paper discovered that although the sizes of the iris / pupil edge curves differ under different thresholds, the perpendicular planes of the tangents of the iris / pupil edge reflection surfaces under different thresholds are only translated, exhibiting a parallel characteristic. Based on this characteristic, the inventors proposed a scheme to determine the actual landing point position of the user's gaze on the terminal device by comparing the actual plane and the simulated plane (the actual plane is perpendicular to the actual tangent of the user's actual iris / pupil edge reflection surface, and the simulated plane is perpendicular to the simulated tangent of the user's simulated iris / pupil edge reflection surface). This scheme avoids errors in determining the actual landing point position of the user's gaze on the terminal device caused by differences in the size of the iris / pupil edge curves, thereby increasing the accuracy of the calculation.

[0067] In addition, this paper compares the actual plane representing the three-dimensional features with the simulated plane instead of using rendered images for comparison. This reduces the amount of computation and increases the speed of calculating the actual position of the user's gaze on the terminal device.

[0068] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart of the line-of-sight positioning method in the embodiments of this article is shown;

[0071] Figure 2 A schematic diagram of the grid pattern in the embodiments of this article is shown;

[0072] Figure 3 A schematic diagram of an image captured by a terminal device in an embodiment of this paper is shown;

[0073] Figure 4 A schematic diagram of a user's eyeball model from an embodiment of this paper is shown;

[0074] Figure 5 This document illustrates a schematic diagram of a user's eyes looking directly at a known point in an embodiment of the invention.

[0075] Figure 6A A first flowchart illustrating the process of generating a virtual space in an embodiment of this paper is shown;

[0076] Figure 6B A second flowchart illustrating the process of generating a virtual space in the embodiments of this paper is shown;

[0077] Figure 7A A schematic diagram of a rectangular wireframe loading image is shown in the embodiment of this article;

[0078] Figure 7B for Figure 7A A magnified view of the image within the rectangular frame;

[0079] Figure 8A A schematic diagram of an embodiment of this article with graduated paper is shown;

[0080] Figure 8B A schematic diagram showing the placement of the camera and paper in an embodiment of this paper is provided.

[0081] Figure 8C This diagram illustrates an image of paper captured by a camera in an embodiment of this paper.

[0082] Figure 8D This diagram illustrates the relationship between the terminal device model constructed in the virtual space of this embodiment and the arrangement of the paper.

[0083] Figure 8E A schematic diagram of a rectangular wireframe of an embodiment of this paper is shown;

[0084] Figure 9 A first flowchart of the actual plane generation process in the embodiments of this paper is shown;

[0085] Figure 10A A schematic diagram of the iris edge curve in the embodiments described herein is shown;

[0086] Figure 10B A schematic diagram of the actual iris edge reflection surface of the user in the embodiments described herein is shown;

[0087] Figure 11A A schematic diagram of line segments on the reflective surface of the iris edge, as shown in the embodiments of this paper, is presented.

[0088] Figure 11B A schematic diagram illustrating the actual section determination process in the embodiments described herein is shown;

[0089] Figure 11C A schematic diagram of the actual cross-section of the embodiments described herein is shown;

[0090] Figure 11D A schematic diagram of the actual plane in the embodiments described herein is shown;

[0091] Figure 12 A second flowchart illustrating the actual plane generation process in the embodiments described herein is shown;

[0092] Figure 13 This diagram illustrates the connection line between any tangent point on the iris / pupil edge curve in the embodiment of this paper and the focal point of the camera in the terminal device model;

[0093] Figure 14 A schematic diagram of the coordinate system in the virtual space of the embodiments described herein is shown;

[0094] Figure 15 A first flowchart illustrating the simulation plane determination process in this embodiment is shown;

[0095] Figure 16 A second flowchart illustrating the simulation plane determination process in the embodiments of this paper is shown;

[0096] Figure 17 A flowchart of another embodiment of the line-of-sight positioning method is shown;

[0097] Figure 18 A first flowchart illustrating the process of determining the simulated cross section in an embodiment of this paper is shown;

[0098] Figure 19 A second flowchart illustrating the process of determining the simulated cross section in the embodiments of this article is shown;

[0099] Figure 20 A flowchart illustrating the process of determining the actual landing point location in the embodiments described herein is shown.

[0100] Figure 21 A flowchart of the terminal device control method of the embodiments of this article is shown;

[0101] Figure 22 A structural diagram of the line-of-sight positioning device according to an embodiment of this article is shown;

[0102] Figure 23 A flowchart of the terminal device control device of the embodiment of this article is shown;

[0103] Figure 24 A structural diagram of the computer device described in this embodiment is shown.

[0104] Explanation of symbols in the attached drawings:

[0105] 401. Iris / Pupil Edge Curve;

[0106] 402. Line of sight;

[0107] 403. Eye rotation point;

[0108] X, a known point;

[0109] 700. Camera in the terminal device model;

[0110] 710. Rectangular wireframe;

[0111] 720. Image;

[0112] 730. User facial image;

[0113] 7201, Marker point at the left outer corner of the eye; 7202, Marker point at the right outer corner of the eye;

[0114] 7203, Nasal tip marker;

[0115] 7204, Marker point at the end of the left nasal wing contour;

[0116] 800. Paper;

[0117] 810. Cameras on terminal devices;

[0118] 820. Camera model;

[0119] 830. Paper models;

[0120] 840. Rectangular wireframe;

[0121] 1000. User's facial image within a rectangular frame;

[0122] 1010. The iris margin curve of the left eye;

[0123] 1020. Right eye iris edge curve;

[0124] 1030. The actual iris edge reflection surface of the user;

[0125] 1100. Reflective surface of the left eye's iris margin;

[0126] 1110. First line segment;

[0127] 1120. The second line segment;

[0128] P1, P2, plane;

[0129] L1, L2, intersecting line segments;

[0130] ASP1, First Actual Section;

[0131] ASP2, the second actual cross-section;

[0132] AP1, the first actual plane;

[0133] AP2, the second actual plane;

[0134] 1000. User facial image;

[0135] 1310. Tangent point;

[0136] 1320. The focal point of the camera in the terminal device model;

[0137] 1330. Connecting cable;

[0138] 2210. Data Acquisition Module;

[0139] 2220. Modeling Module;

[0140] 2230. Calculation Module;

[0141] 2240. Simulation module;

[0142] 2250. Comparison Module;

[0143] 2310. Landing point location determination module;

[0144] 2320. Instruction generation module;

[0145] 2402. Computer equipment;

[0146] 2404, Processor;

[0147] 2406. Memory;

[0148] 2408. Drive mechanism;

[0149] 2410. Input / Output Module;

[0150] 2412. Input devices;

[0151] 2414. Output devices;

[0152] 2416. Presentation device;

[0153] 2418. Graphical User Interface;

[0154] 2420. Network interface;

[0155] 2422. Communication link;

[0156] 2424. Communication bus. Detailed Implementation

[0157] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0158] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0159] In existing technologies, there are schemes that determine the location of a user's gaze on a terminal device by comparing a first vector image with multiple second vector images. The first vector image is determined by analyzing real-time captured images of the user, resulting in a vector image including the iris / pupil edge curves. The second vector image is determined by simulating the user's gaze at various positions on the terminal device model in a virtual space containing the user and the terminal device model, capturing simulated user images, and determining vector images including the iris / pupil edge curves based on these simulated images. However, in this scheme, the determination of the vector image including the iris / pupil edge curves based on the simulated user images is problematic. Different thresholds for judging the pixel grayscale difference at the iris-sclera junction / pupil-iris junction in the simulated user images can cause variations in the size of the iris / pupil edge curves, directly affecting the accuracy of determining the location of the user's gaze on the terminal device. Furthermore, rendering multiple second vector images also results in low computational efficiency.

[0160] To address the aforementioned technical problems in the prior art, one embodiment of this paper provides a line-of-sight positioning method. This method can run in a terminal device or be implemented by the terminal device in conjunction with a server. Specifically, as shown in... Figure 1 As shown, the line-of-sight positioning methods include:

[0161] Step 110: Collect the user's facial image.

[0162] In this step, the user's facial image is captured by the terminal device while the user is looking at the screen. The capture frequency can be set according to the actual situation, such as 1 second, but this article does not limit it. The user's facial image can be an image, which can be captured by the camera on a regular terminal device, or it can be the user's facial features, such as eyes, nose, ears, mouth, face, etc., which can be obtained by scanning by the structured light sensor or TOF sensor in the terminal device.

[0163] Step 120: Based on the user's facial image, generate a virtual space including a user model and a terminal device model, wherein the user model includes at least a user eyeball model.

[0164] In this step, the principle of structured light can also be used to generate a virtual space that includes user models and terminal device models.

[0165] The user's facial image is a distortion-free image, and the camera on the terminal device is a distortion-free camera. The distortion-free image described in this article needs to meet the following conditions: the camera captures a plane with a grid pattern of uniformly sized squares, and during the capture, the central axis of the camera is perpendicular to the grid pattern, such as... Figure 2 As shown, each square in the captured or restored image remains a square, and all squares are the same size. Figure 3As shown, the square in the image must satisfy the following conditions: all four sides are of equal length and perpendicular to the adjacent sides.

[0166] The terminal devices described herein include, but are not limited to, mobile terminals, tablets, desktop computers, and laptops. These terminal devices may include a screen, a camera, and may also include an information processor, an information storage medium, pre-installed image recognition software, and 3D spatial calculation software.

[0167] User models can be pre-collected and stored on terminal devices or servers. The user model also includes facial feature points (e.g., left outer corner of the eye feature point, right outer corner of the eye feature point, nose tip feature point, left nasal wing contour end feature point). Figure 4 As shown, the user's eye model includes features such as the iris / pupil edge curve 401 and the visual axis 402. In specific implementation, the user's eye model also includes an eye rotation point 403. The user's eye model includes a left eye model and a right eye model, and correspondingly, the eye rotation point also includes a left eye rotation point and a right eye rotation point.

[0168] Specifically, the iris / pupil edge curve is a model of the eyeball that reconstructs the boundary between the iris and sclera in the user's eyeball by calculating and modeling the boundary between the iris and sclera in the image obtained from the camera capturing the user's eyes.

[0169] The pupil edge curve is a model of the eyeball that reconstructs the boundary between the pupil and iris in an image captured by a camera and obtained from the user's eyes.

[0170] While maintaining a constant relative position with the facial model, the entire eyeball model rotates around the eye rotation point 403 to simulate the rotation of a real user's eyeball within its socket. The facial model is composed of user facial feature points.

[0171] like Figure 5 As shown, the visual axis 402 is formed by taking a picture of the user's eyes with a camera when the user's eyes are looking directly at a known point X. Based on the image of the user's eyes obtained from the picture, the virtual space including the eyeball model and the known point is reconstructed in the virtual space through reasoning. The point at the center of the pupil in the user's eyeball model and the known point are connected by a straight line, which is the visual axis 402. The relative positional relationship between the visual axis and all three-dimensional shape features in the entire user's eyeball model remains unchanged.

[0172] The user eye model is used to determine the direction of the object being looked at by the real user's eye by using the direction pointed to by the visual axis in the eye model when the user's eye is captured by the camera of the terminal device and the virtual space including the camera and the user eye model is constructed.

[0173] In some implementations, a 3D scanner can be used to scan the face, and a user model can be obtained by modeling the feature points obtained from the scan. In other implementations, multiple optical sensors at different locations can be used to capture images of the user's face, and the user model can be obtained by modeling the facial images.

[0174] In practice, a laser transmitter and a laser receiver can be installed on the terminal device. By transmitting and receiving signals through the laser transmitter and laser receiver, the spatial relationship between the user and the terminal device can be determined. Then, based on the spatial relationship between the user and the terminal device, a virtual space including a user model and a terminal device model can be generated.

[0175] Step 130: Based on the user's facial image and the terminal device model in the virtual space, generate at least N actual planes, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface.

[0176] In this article, the iris / pupil edge reflection surface refers to either the iris edge reflection surface or the pupil edge reflection surface; either one is acceptable.

[0177] The actual iris / pupil edge reflection surface of a user refers to the surface formed by all the light rays that enter the camera lens and create the user image during image capture, specifically those rays that form the iris-sclera boundary or pupil edge features in the user image. In other words, it is a conical surface constructed using the iris / pupil edge curve established from the real-time captured user facial image as the guideline and the focal point of the terminal device model camera as the endpoint.

[0178] Definition of a conical surface: A conical surface is the surface formed by a moving straight line L passing through a fixed point M1 moving along a defined curve C. The straight line L is called the generating line (genesis) of the conical surface, the curve C is called the directrix, and the fixed point M1 is called a vertex of the conical surface.

[0179] The number of actual planes N is a positive integer greater than or equal to 2. This paper does not limit the number of actual planes.

[0180] Step 140: In the virtual space, adjust the simulated landing point position of the user's eyeball model's line of sight on the terminal device model according to the preset rules. When adjusting to a simulated landing point position, determine multiple sets of simulated planes. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface.

[0181] In this step, once the rotation point in the user's eyeball model overlaps with the rotation point in the face model, the user's eyeball model can rotate around the rotation point in the face model.

[0182] The screen of the terminal device model is divided into multiple small areas. The user's eye model looks at different small areas according to pre-set rules, such as from left to right and from top to bottom. This article does not specify these rules. The number of simulated planes in each group of simulated planes is the same as the number of actual planes determined in step 130.

[0183] The user's virtual iris / pupil edge reflection surface refers to a conical surface created using the iris / pupil edge curve in the user's eyeball model as the guideline and the focal point of the camera in the terminal device model as the endpoint.

[0184] Step 150: Compare each group of simulated planes with the actual planes, and take the simulated landing point position when the simulated plane is parallel to the actual plane as the actual landing point position of the user's line of sight on the terminal device.

[0185] This embodiment takes into account that, under a certain line of sight, although the sizes of the iris / pupil edge curves differ at different thresholds, the perpendicular planes of the tangents of the iris / pupil edge reflection surfaces at different thresholds are only translated and remain parallel to each other. By comparing the perpendicular lines of the tangents of the iris edge surfaces, errors in determining the actual landing point of the user's gaze on the terminal device caused by differences in the size of the iris / pupil edge curves can be avoided, thus increasing the accuracy of the calculation. Furthermore, this paper uses the perpendicular lines of the tangents of the iris edge surfaces for comparison instead of using rendered images, which reduces the computational load and improves the calculation speed of the actual landing point of the user's gaze on the terminal device.

[0186] The iris / pupil edge curves and iris / pupil edge reflection surfaces mentioned in this article refer to the fact that the iris edge curves and iris edge reflection surfaces can be interchanged with the pupil edge curves and pupil edge reflection surfaces, respectively, and only one of them needs to be chosen.

[0187] In one embodiment of this article, such as Figure 6A As shown, step 120 above, which generates a virtual space including a user model and a terminal device model based on the user's facial image, includes:

[0188] Step 610: Determine the spatial relationship between the user model and the terminal device based on the user's facial image.

[0189] Step 620: Generate a virtual space including the user model and the terminal device model based on the spatial relationship between the user model and the terminal device model.

[0190] When implementing step 610, it can also be replaced by using the laser emitter and laser receiver installed on the terminal device to determine the spatial positional relationship between the user and the terminal device through the structured light principle or the TOF principle, that is, the spatial positional relationship between the user model and the terminal device.

[0191] Structured light is a system consisting of a projector and a camera. Specific light information is projected onto the surface and background of an object by the projector, and then captured by the camera. The position and depth information of the object are calculated based on the changes in the light signal caused by the object, thus reconstructing the entire three-dimensional space. By using the system structure consisting of the projector and camera installed on the terminal device to reconstruct the three-dimensional space, including the real user's face, the spatial relationship between the user and the terminal device can be determined.

[0192] TOF stands for Time of Flight, a technology that uses a sensor to emit modulated near-infrared light. This light is reflected from an object, and the sensor calculates the distance to the object by measuring the time difference or phase difference between the emission and reflection, thus generating depth information. By using a TOF sensor system installed on a terminal device, a three-dimensional space, including the user's face, can be reconstructed, thereby determining the spatial relationship between the user and the terminal device.

[0193] In one embodiment of this paper, to avoid dependence on terminal device hardware, the spatial relationship between the user model and the terminal device model can be established through software methods. Specifically, for example... Figure 6B As shown, step 610 above, based on the user's facial image, determines the spatial relationship between the user model and the terminal device model, including:

[0194] Step 611: Load the user's facial image into the rectangular wireframe 710 in front of the terminal device model.

[0195] Wherein, the rectangular frame 710 is used to represent the image size of the camera 700 of the terminal device model at a predetermined distance, such as Figure 7AAs shown, the line connecting the feature points on image 720 in the rectangular frame 710 to the focal point of the terminal device model camera is collinear with the line connecting the feature points on user face image 730 to the focal point of the terminal device model camera. The rectangular frame can be determined at the factory of the terminal device or it can be calibrated and determined before the terminal device is used. The distance between the rectangular frame and the terminal device model is a known quantity, and its value range can be determined according to the usual distance from the terminal device camera, for example, 20 cm to 30 cm. This paper does not make a specific limitation on this.

[0196] In this embodiment, the user's facial image is loaded into the rectangular frame in front of the terminal device model. That is, by scaling down or scaling up, the pixels around the user's facial image are overlapped with the rectangular frame.

[0197] Step 612: Determine the user's facial feature points within the rectangular frame based on the user's facial image within the rectangular frame.

[0198] like Figure 7B As shown, Figure 7B for Figure 7A Enlarged view of image 720 within the rectangular wireframe 710. The identified feature points include, for example, the left outer corner of the eye marker 7201, the right outer corner of the eye marker 7202, the tip of the nose marker 7203, and the end of the left nasal wing contour marker 7204.

[0199] Step 613: Based on the user's facial feature points within the rectangular frame and the camera focus in the terminal device model, establish the reflection line of the user's facial feature points within the rectangular frame, that is, the line connecting the user's facial feature points within the rectangular frame and the camera focus in the terminal device model.

[0200] Step 614: Adjust the position of the user model so that the reflection lines of the facial feature points in the user model coincide with the reflection lines of the user feature points within the rectangular frame, such as... Figure 7A As shown, the spatial relationship between the user model and the terminal device model is obtained.

[0201] In step 611, the rectangular wireframe in front of the terminal device model can be predetermined when the terminal device leaves the factory, or predetermined before the terminal device enables the gaze control terminal device function. The specific determination process is as follows:

[0202] (1) Print a piece of paper 800 with a Cartesian coordinate system. The Cartesian coordinate system on the paper has an origin, a horizontal axis, and a vertical axis, and the horizontal and vertical axes have scales, such as... Figure 8AAs shown. On the horizontal axis, the scale readings to the right of the origin are positive, and the scale readings to the left are negative, with the absolute value of the scale readings increasing as they are further away from the origin. On the vertical axis, the scale readings above the origin are positive, and the scale readings below the origin are negative, with the absolute value of the scale readings increasing as they are further away from the origin. The length of each smallest scale unit on both the horizontal and vertical axes is equivalent to 1 cm, 1 mm, 0.1 mm, 0.01 mm, or 1 μm.

[0203] (2) Use the camera 810 of the terminal device to photograph the paper 800 with the plane rectangular coordinate system, such as Figure 8B As shown, before taking the picture, the camera 810 of the terminal device must be perfectly aligned with the table on the paper 800. That is, the central axis of the camera 810 intersects the origin of the coordinate system on the image, and the central axis of the camera 810 is perpendicular to the plane where the paper 800 is located. Furthermore, the horizontal pixels of the image captured by the camera 810 should be parallel to the horizontal axis of the coordinate system on the paper 800. Then, adjust the focus of the camera 810 and the distance between the plane where the paper 800 is located to a known fixed value, such as 1 meter. Next, photograph the paper to obtain a rectangular coordinate system image, as shown below. Figure 8C As shown, the shape of the captured rectangular coordinate system image is rectangular.

[0204] (3) In the virtual space, recreate the scene of the camera of the terminal device shooting the paper, ensuring that the size of the paper model and the Cartesian coordinate system on the paper model are the same as the real paper, and that the relative positional relationship between the camera model 820 and the paper model 830 is the same as the relative positional relationship between the camera and the paper of the real terminal device, such as... Figure 8D As shown.

[0205] (4) In the rectangular coordinate system image of the paper obtained by the camera of the terminal device, find the four sides of the rectangular coordinate system image of the rectangle, and read the horizontal and vertical scale values ​​of the points where each side intersects on the horizontal or vertical axis of the rectangular coordinate system.

[0206] (5) In the virtual space, construct a rectangular frame 840 using four points corresponding to the scale values ​​on the horizontal and vertical axes of the rectangular coordinate system in the paper model 830, such that the four sides of the rectangular frame 840 intersect at these four points. Figure 8E As shown.

[0207] The rectangular frame area is the region within the field of view of the camera at a predetermined distance, which can be captured in the image. The boundary of the rectangular frame is the boundary of the camera's field of view. The rectangular frame has four endpoints, representing the upper left, upper right, lower left, and lower right corners of the rectangular image obtained from the photographed paper.

[0208] Using the four points of the rectangular frame as endpoints and the camera's focal point as the other endpoint, draw four straight lines. These four lines represent the light rays reflected from and entering the lens at a point in the image corresponding to the top left, top right, bottom left, and bottom right corners when the image is captured. The three-dimensional region enclosed by these four lines forms a pyramid shape, which is the camera's field of view.

[0209] A rectangular Cartesian coordinate system image captured by the terminal device's camera is imported into a virtual space scene depicting a piece of paper with a Cartesian coordinate system captured by a camera. The Cartesian coordinate system image is scaled proportionally so that its four sides coincide with the four sides of the rectangular frame. Furthermore, the orientation of the Cartesian coordinate system in the image is ensured to be consistent with that in the paper model; that is, the horizontal axis and vertical axis of the Cartesian coordinate system in both images point in the same direction. At this point, it can be observed that the features of the Cartesian coordinate system in the image almost perfectly match those in the paper model. This is because the image captured by the distortion-free camera is distortion-free, meaning the Cartesian coordinate system in the image is free from any stretching or distortion.

[0210] In a scenario where a camera captures images of paper in virtual space, any point in the Cartesian coordinate system image is connected to the camera's focal point by a straight line; this line is called line one. Similarly, a point in the paper model that is identical to this point is connected to the focal point of the camera's 3D model by a straight line; this line is called line two. It can be observed that line one and line two almost coincide, meaning the positional relationship between line one and the camera's 3D model is almost identical to that of line two.

[0211] The straight line represents a method of creating a rectangular wireframe in front of the camera model based on the camera model and the image captured by the camera. The method involves creating a camera model and a rectangular wireframe in virtual space, placing the image captured by the camera corresponding to the camera model within the rectangular wireframe, and connecting any point on the captured image to the focal point of the camera model.

[0212] Line 2 simulates the light rays that enter the camera lens at any point in the real space corresponding to that point in the image captured by the camera.

[0213] In one embodiment of this article, such as Figure 9 As shown, step 130 above generates at least N actual planes based on the user's facial image and the terminal device model in the virtual space, including:

[0214] Step 910: Determine the actual iris / pupil edge reflection surface of the user based on the user's facial image and the terminal device model in the virtual space.

[0215] In practical implementation, to improve the calculation speed of the actual iris / pupil edge reflection surface of the user, this paper provides a calculation process for the actual iris / pupil edge reflection surface of the user. Specifically, the calculation process for the actual iris / pupil edge reflection surface of the user includes:

[0216] (1) Load the user's facial image into the rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the camera of the terminal device model at a predetermined distance.

[0217] Specifically, the user's facial image is loaded into the rectangular frame in front of the terminal device model, which means the user's facial image is enlarged or reduced until the four sides of the processed image coincide with the four sides of the rectangular frame.

[0218] The line connecting any point on the user's facial image within the rectangular frame to the focal point of the camera on the terminal device model is the same as the line of light that forms that point on the user's facial image when the camera on the real terminal device captures the user's facial image.

[0219] (2) Determine the actual iris / pupil edge curve of the user based on the user's facial image within the rectangular frame.

[0220] Taking the iris edge curve as an example, the features at the junction of the iris and sclera of the left and right eyes in the rectangular frame user face image 1000 are outlined using non-uniform rational B-spline curves (NURBS), resulting in the left eye iris edge curve 1010 and the right eye iris edge curve 1020, as shown below. Figure 10A As shown.

[0221] In practice, the user's facial image can be vectorized beforehand. Specifically, the features at the junction of the iris and sclera in the user's facial image can be outlined using non-uniform rational B-spline curves to obtain the left and right eye iris edge curves, respectively. The user image boundary can then be outlined using vector rectangles. The left and right eye iris edge curves, along with the vector rectangles, together form the user's vector image.

[0222] (3) Using the camera focus point in the terminal device model as the vertex and the actual iris edge curve of the user as the guideline, a conical surface is obtained. This conical surface is the actual iris edge reflection surface 1030. Figure 10B As shown.

[0223] The actual iris edge reflection surface of the user includes: the actual left eye iris edge reflection surface of the user and the actual right eye iris edge reflection surface of the user.

[0224] When a person's eyeballs turn upwards, the boundary between the iris and sclera in the eye is only partially obscured by the upper eyelid and is not obscured at all by the lower eyelid. At this time, the boundary between the iris and sclera in the eye features of the user's face image obtained by the camera is a continuous line. The iris / pupil edge curve obtained by outlining with a non-uniform rational B-spline curve (NURBS) is also a continuous line. With the focal point of the virtual camera as the vertex and the iris / pupil edge curve as the guideline, the conical surface made is a continuous surface. That is, the obtained iris edge reflection surface is the iris edge reflection surface corresponding to one eye.

[0225] When a person's eyes look straight ahead, the boundary between the iris and sclera is simultaneously obscured by both the upper and lower eyelids, resulting in two broken boundary lines between the iris and sclera. Using non-uniform rational B-spline curves (NURBS), these two boundary lines are outlined, resulting in two iris / pupil edge curves. Using the focal point of the virtual camera as the vertex and the two iris / pupil edge curves as the guide lines, two conical surfaces are created. In this example, these two conical surfaces together form the iris edge reflection surface corresponding to one eye.

[0226] Step 920: Determine at least N actual cross-sections of the actual iris / pupil edge reflection surface of the user, and the line segments intersecting each actual cross-section with the actual iris / pupil edge reflection surface of the user.

[0227] In detail, the higher the perpendicularity between the actual cut surfaces, the better; all actual cut surfaces must not be coplanar.

[0228] The following example, using the reflective surface of the left eye's iris edge and N=2, illustrates the process of determining the actual cross-section:

[0229] (1) The conical surface containing the iris edge reflection surface of the left eye is obtained by constructing a conical surface with the focal point of the camera model as the vertex and the iris / pupil edge curve of the left eye as the guideline. That is, every point on the iris / pupil edge curve is connected to the focal point of the camera model by a line segment, and the conical surface formed by all the line segments is the iris edge reflection surface. Take any two line segments that make up the iris edge reflection surface, that is, any two generatrices that make up the iris edge reflection surface, and name them as the first line segment 1110 and the second line segment 1120, respectively. Figure 11A As shown.

[0230] (2) Construct the line segment L1, which intersects the plane P1 intersecting the first line segment 1110 and the reflective surface 1100 of the left iris edge, and the line segment L2, which intersects the plane P2 intersecting the second line segment and the reflective surface 1100 of the left iris edge, as follows: Figure 11A As shown.

[0231] (3) Construct the first actual cross-section ASP1 and the second actual cross-section ASP2, ensuring that the first line segment 1110 and the second line segment 1120 are respectively within the first actual cross-section ASP1 and the second actual cross-section ASP2. This is achieved by ensuring that the first actual cross-section ASP1 and the second actual cross-section ASP2 are tangent to the intersection line segment L1 of the plane intersecting the first line segment and the reflection surface of the left iris edge, and the intersection line segment L2 of the plane intersecting the second line segment and the reflection surface of the left iris edge, respectively. Figure 11B As shown, this ensures that both the first actual cross-section ASP1 and the second actual cross-section ASP2 are tangent to the reflective surface of the left eye's iris margin, as... Figure 11C As shown.

[0232] Step 930, based on each actual cross-section (e.g.) Figure 11C The first actual cross plane ASP1 and the second actual cross plane ASP2) and their related intersecting line segments (such as Figure 11C The first line segment 1110 and the second line segment 1120 in the figure determine the actual plane that is perpendicular to each actual tangent and contains the relevant intersecting line segments (e.g., Figure 11D The first actual plane AP1 and the second actual plane AP2 are shown in the figure.

[0233] In this step, for each actual cross-section and its intersection with the user's actual iris / pupil edge reflection surface, an actual plane containing that intersection segment can be determined. The number of actual planes is the same as the number of actual cross-sections.

[0234] In one embodiment of this article, in order to save the computational load of the computing device (terminal device or server), such as Figure 12 As shown, step 130 above generates at least N actual planes perpendicular to the actual iris / pupil edge reflection surface based on the user's facial image and the terminal device model in virtual space, including:

[0235] Step 1210: Load the user's facial image into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance.

[0236] The aspect ratio of the rectangular frame is the same as that of the user's facial image. The position of the rectangular frame ensures that the straight line connecting each pixel in the user's facial image to the focal point of the camera model in the terminal device model is the same as the light rays that enter the terminal device camera and form that pixel when the user's facial image is captured.

[0237] In practice, by scaling and moving the image proportionally, the four sides of the scaled user face image are made to coincide with the four sides of the rectangular frame.

[0238] Step 1220: Determine the actual iris / pupil edge curve of the user based on the user's facial image within the rectangular frame.

[0239] In practice, the iris edge curve is determined by outlining the features at the junction of the iris and sclera using a non-uniform rational B-spline curve (NURBS). The method for determining the pupil edge curve is similar and will not be detailed here.

[0240] Step 1230: Determine at least N actual tangent lines and tangent points of the user's actual iris / pupil edge curve.

[0241] In practice, any point can be selected as the tangent point on the iris / pupil edge curve, and the actual tangent line at that point can be created.

[0242] Step 1240: Connect the tangent point 1310 to the focal point 1320 of the camera in the terminal device model to obtain the actual connection line 1330 between the tangent point 1310 and the focal point 1320 of the camera in the terminal device model, as shown below. Figure 13 As shown.

[0243] Step 1250: Determine the actual tangent based on each actual connecting line and its associated actual tangent.

[0244] Step 1260: Determine the actual plane that is perpendicular to each actual cut surface and contains the relevant actual connecting lines, based on each actual cut surface and its associated actual connecting lines.

[0245] In this step, the actual tangent related to the connecting line refers to the connecting line and the actual tangent that share the same tangent point.

[0246] In this step, firstly, an actual tangent plane is created, passing through the focal point of the camera in the terminal device model and the actual tangent line. Then, a plane containing connecting lines and perpendicular to the actual tangent plane containing the connecting lines is created. This plane consists of at least N actual planes perpendicular to the actual iris / pupil edge reflection surface of the user.

[0247] The proof of the above conclusion is given below: The actual iris edge reflection surface is formed by taking the focal point of the camera model as the vertex and the iris / pupil edge curve as the directrix. According to the definition of a conical surface: the surface generated by a moving straight line L passing through a fixed point M1 moving along a defined curve C is called a conical surface. The straight line L is called the generating line (generatrix) of the conical surface, the curve C is called the directrix, and the fixed point M1 is called a vertex of the conical surface. In the constructed conical surface, the actual connecting line can be regarded as a generatrix within the actual iris edge reflection surface, and the actual iris / pupil edge curve can be regarded as the directrix of the actual iris edge reflection surface.

[0248] That is, the actual connecting line and the actual iris edge curve are both lines within the actual iris edge reflection surface.

[0249] Because the actual tangent plane passes through the camera focus point in the terminal device model and the actual tangent line, and the actual tangent point and the camera focus point in the terminal device model are within the connecting line, the actual connecting line is within the actual tangent plane, and therefore the actual connecting line and the actual tangent plane are tangent.

[0250] Because the actual tangent line is tangent to the actual iris edge curve, and the actual tangent line is within the actual tangent plane, the actual iris edge curve is tangent to the actual tangent plane.

[0251] Because the actual connecting line intersects but is not tangent to the actual iris edge curve, the actual tangent plane is tangent to the actual iris edge reflection surface. Furthermore, the actual tangent plane and the actual iris edge reflection surface intersect at the actual connecting line.

[0252] Because the actual plane and the actual tangent are perpendicular and intersect at the same connecting line, the actual plane is perpendicular to the actual iris edge reflection surface.

[0253] This embodiment eliminates the need to calculate the actual iris / pupil edge reflection surface of the user. By using the actual iris / pupil edge curve of the user and the camera focus in the terminal device model, at least N actual planes perpendicular to the actual iris / pupil edge reflection surface can be determined. This embodiment improves the calculation speed.

[0254] In one embodiment of this paper, step 140 above can control the relative position of the user's eyeball model with respect to other feature points of the user model in the virtual space according to a preset rule.

[0255] Specifically, in virtual space, the relative positions of the terminal device model and the user's facial model remain unchanged, and a coordinate system is placed within the user's facial model, which is called the facial coordinate system.

[0256] like Figure 14 As shown, in this example, the facial coordinate system placed within the user's facial model is a three-dimensional Cartesian coordinate system. The three mutually perpendicular coordinate axes of this three-dimensional Cartesian coordinate system are the x-axis, y-axis, and z-axis. The plane where the x-axis and y-axis intersect is the xy-plane, the plane where the x-axis and z-axis intersect is the xz-plane, and the plane where the y-axis and z-axis intersect is the yz-plane.

[0257] The z-axis of the three-dimensional rectangular coordinate system points in the direction the user is facing, and the x-axis points in the direction the user travels from the left corner of the eye to the right corner of the eye. The origin of the facial coordinate system is placed at the center of the facial model. The purpose of placing the facial coordinate system in the facial model is to measure the angular relationship between the visual axis in the eye model and the facial model.

[0258] The angular relationship between the visual axis in the eye model and the user's facial model is determined by the angular relationship between the visual axis in the eye model and the facial coordinate system.

[0259] The angle between the orthographic projection of the visual axis onto the xz plane of the facial coordinate system and the visual axis is α. If the y-coordinate value of the point corresponding to the point on the visual axis in the facial coordinate system increases as the z-value increases, then the angle α is positive. If the y-coordinate value decreases as the z-value increases, then the angle α is negative. If the y-coordinate value remains unchanged as the z-value changes, then the angle α is 0.

[0260] The angle between the orthographic projection of the visual axis onto the yz plane of the facial coordinate system and the visual axis is β. If the x-coordinate value of a point on the visual axis in the facial coordinate system increases as the z-value increases, then the angle β is positive. If the x-coordinate value decreases as the z-value increases, then the angle β is negative. If the y-coordinate value remains unchanged as the z-value changes, then the angle β is 0.

[0261] In this example, the angular relationship between the visual axis and the eye coordinate system is expressed in the format (α, β) of the orthographic projection of the visual axis onto the xz plane of the facial coordinate system and the visual axis.

[0262] This allows the visual axis within the eyeball model to be in an arbitrary angular relationship with the facial coordinate system within the facial model.

[0263] In one embodiment of this article, such as Figure 15 As shown, in step 140 above, for any simulated landing point location, at least N simulated planes are determined, including:

[0264] Step 1510: For any simulated landing point, based on the simulated iris / pupil edge curve in the user's eyeball model at that simulated landing point, establish the simulated iris / pupil edge reflection surface at that simulated landing point.

[0265] The user's eye model contains iris / pupil edge curves. In this step, a conical surface is created using the focal point of the terminal device model's camera as the endpoint and the iris / pupil edge curve as the guideline. This conical surface is the iris / pupil edge reflection surface.

[0266] Step 1520: Determine at least N simulated cross-sections of the user-simulated iris / pupil edge reflection surface, and the line segments intersecting the simulated cross-sections with the user-simulated iris / pupil edge reflection surface.

[0267] Specifically, when implementing step 1520, at least N intersecting line segments passing through the camera focus of the terminal model are first established within the simulated iris / pupil edge reflection surface; then at least N simulated cross-sections, each containing at least N intersecting line segments and tangent to the user simulated iris / pupil edge reflection surface, are established.

[0268] The specific implementation process for establishing intersecting line segments can be found in step 920, which describes the process for establishing the first or second line segment. It will not be detailed here.

[0269] The specific implementation process for establishing the simulated cross-section can be found in step 920, which describes the process for establishing the first or second actual cross-section. It will not be detailed here.

[0270] Step 1530: Based on each simulated cross section and its associated intersecting line segments, determine a simulated plane that is perpendicular to each simulated cross section and contains the associated intersecting line segments.

[0271] The specific implementation process for establishing the simulated plane can be found in step 920, which describes the implementation process for establishing the actual plane. It will not be detailed here.

[0272] In one embodiment of this paper, in order to reduce the amount of computation, such as Figure 16 As shown, in step 140 above, for any simulated landing point location, at least N simulated planes are determined, including:

[0273] Step 1610: Based on the user's eyeball model at the arbitrary simulated landing point position, determine the simulated iris / pupil edge curve at the simulated landing point position.

[0274] Step 1620: Establish at least N simulated tangents and tangent points for the simulated iris / pupil edge curve of the user;

[0275] Step 1630: Connect the tangent point to the focal point of the camera in the terminal device model to obtain a simulated connection line between the tangent point and the focal point of the camera in the terminal device model;

[0276] Step 1640: Determine the simulation cross-section based on each simulation connection line and its associated simulation tangent line, that is, create a simulation cross-section containing the simulation connection line and the simulation tangent line.

[0277] Step 1650: Based on each simulated section and its associated simulated connecting lines, determine a simulated plane that is perpendicular to each simulated section and contains the associated simulated connecting lines.

[0278] The specific implementation process of steps 1610 to 1650 can be referred to the implementation process of steps 1220 to 1260, which will not be detailed here.

[0279] In one embodiment of this paper, to improve computational efficiency, another line-of-sight positioning method is also provided, such as... Figure 17 As shown, specifically, including:

[0280] Step 1710: Acquire user's facial image;

[0281] Step 1720: Based on the user's facial image or the principle of structured light, generate a virtual space including a user model and a terminal device model, wherein the user model includes at least a user's eyeball model and user's facial feature points.

[0282] Step 1730: Based on the user's facial image and the terminal device model in the virtual space, generate at least N actual planes, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface;

[0283] Step 1740: In the virtual space, establish at least N simulated planes passing through the focal point of the camera in the terminal device model; using the focal point of the camera in the terminal device model as a fixed point, rotate at least N simulated planes until they are parallel to at least N actual planes;

[0284] When the user's eyeball is in any position under the pre-set rules, at least N simulated planes of the camera focus in the terminal device model are established in the virtual space according to steps 1510-1530.

[0285] The proof of the focal point of the camera in the simulated planar terminal device model is as follows:

[0286] At least N simulated planes intersect at least N simulated cross-sections on the simulated iris / pupil edge reflection surface, and at least N simulated cross-sections are tangent to the simulated iris / pupil edge reflection surface, and at least N simulated planes are perpendicular to at least N simulated cross-sections.

[0287] Since both the simulated cross section and the simulated plane are planes, the simulated intersection line obtained by the intersection of the simulated cross section and the simulated plane is a straight line.

[0288] Because the simulated intersection line lies within the reflective surface of the pseudo-iris / pupil edge, it is a generatrix within the cone surface containing the reflective surface of the pseudo-iris / pupil edge. Therefore, the simulated intersection line must pass through the vertex of the cone surface containing the reflective surface of the pseudo-iris / pupil edge, and thus, the simulated intersection line must pass through the camera focus in the terminal device model.

[0289] Since the simulated intersection line is the intersection of the simulated tangent and the simulated plane, the simulated intersection line lies within the simulated plane, and therefore the simulated plane passes through the focal point of the camera in the terminal device model.

[0290] The focal point of the camera in the terminal device model is fixed. Rotate N simulated planes until they are parallel to the corresponding N actual planes. The proof that the rotated simulated planes still pass through the focal point of the camera in the terminal device model and are coplanar with the actual planes is as follows:

[0291] Using the same proof method as above, based on the actual iris / pupil edge reflection surface, the actual intersection line, the actual tangent, the actual plane, and the focal point of the camera model in the terminal device model, it is proved that the actual plane passes through the focal point of the camera in the terminal device model.

[0292] Since both the actual plane and the simulated plane pass through the focal point of the camera model in the terminal device model, if the simulated plane is rotated to be parallel to the actual plane, then the simulated plane and the actual plane will be coplanar.

[0293] Step 1750: Change the simulated landing point position of the user's eyeball model's line of sight on the terminal device model until at least N simulated cross-sections of the simulated iris / pupil edge reflection surface are perpendicular to the N simulated planes, wherein the simulated iris / pupil edge reflection surface is determined by the simulated iris / pupil edge curve in the user's eyeball model.

[0294] In practice, the positions of the user's eye model and facial model can be changed under pre-set rules to adjust the simulated focal point of the user's eye model's gaze on the terminal device model. For example... Figure 14 As shown above, an eye model and a face model, along with rules governing their mutual movement, are provided. Inputting a set of α and β values ​​will result in different relative positions between the eye model and the face model. For example, multiple sets of input α and β values ​​could be (15, 15) / (0, 15) / (-15, 15) / (15, 0) / (0, 0) / (-15, 0) / (15, -15) / (0, -15) / (-15, -15).

[0295] The rules governing the mutual movement of the user's eye model and facial model described above are only used to introduce one method for determining the point where the user's gaze falls on the terminal device. Other rules governing the mutual movement of the user's eye model and facial model are also applicable to the method mentioned in this article for determining the point where the user's gaze falls on the terminal device.

[0296] Changing the position of the user's eye model causes a change in the shape and position of the simulated iris / pupil edge reflection surface, which is built based on the simulated iris / pupil edge curve in the eye model:

[0297] According to specific rules, a set of α and β values ​​are input. For example, if the input α value is 15 and the input β value is 15, expressed as (15, 15), the eye model moves to a new specific relative position relationship with the face model according to the pre-selected rules. Because the relative position relationship between the face model and the terminal device model is pre-calculated based on the operator's image or structured light principle, it can be assumed that it is fixed before determining the user's gaze point on the terminal device. Therefore, because of the input set of α and β values, expressed as (15, 15), the focus of the camera model in the eye model and the simulated iris / pupil edge curve in the eye model and the focus of the camera in the terminal device model are in a new relative position relationship, and the relative position relationship has changed.

[0298] Because the simulated iris / pupil edge reflection surface is a conical surface constructed with the camera focus in the terminal device model as the vertex and the simulated iris / pupil edge curve in the eyeball model as the guideline, when the relative positional relationship between the simulated iris / pupil edge curve and the camera focus in the terminal device model changes, it is equivalent to the relative positional relationship between the vertex of the conical surface and the guideline of the conical surface changing. Therefore, the shape and position of the simulated iris / pupil edge reflection surface change.

[0299] The shape and position of the simulated iris / pupil edge reflective surface change, and the spatial position of at least N simulated intersection lines obtained by intersecting with at least N simulated planes change.

[0300] Because at least N virtual planes are always coplanar with at least N real planes before and after rotating the eyeball model, the spatial positions of at least N virtual planes do not change.

[0301] Before and after rotating the eyeball model, the shape and position of the simulated iris / pupil edge reflection surface change, so the position of the intersection line between the simulated iris / pupil edge reflection surface and at least N virtual planes changes.

[0302] Changing the spatial position of at least N simulated cross-sections so that the intersection lines of at least N simulated cross-sections with the corresponding at least N simulated planes coincide with the simulated intersections of at least N simulated planes with the simulated iris / pupil edge reflection surface can also be seen as changing the spatial position of at least N simulated cross-sections so that the corresponding at least N simulated intersection lines contain at least N simulated cross-sections, and ensuring that at least N simulated planes are tangent to the simulated iris / pupil edge reflection surface.

[0303] Step 1760: The simulated landing point of the user's eyeball model's line of sight on the terminal device model is taken as the landing point of the user's line of sight on the terminal device.

[0304] In one embodiment of this article, such as Figure 18 As shown, the implementation process of step 1750 above includes:

[0305] Step 1751: Change the position of the user's eye model relative to the facial model;

[0306] Step 1752: Based on the simulated iris / pupil edge curve of the user's eyeball model, establish a simulated iris / pupil edge reflection surface;

[0307] Step 1753: Determine at least N simulated intersection lines between the simulated iris / pupil edge reflection surface and at least N simulated planes;

[0308] Step 1754: Based on at least N simulated intersection lines, determine at least N simulated cross-sections that contain the simulated intersection lines and are tangent to the simulated iris / pupil edge reflection surface;

[0309] In practice, a simulated cross-section of the camera focus in the terminal device model can be randomly generated. The simulated cross-section is rotated according to the simulated intersection line and the simulated iris / pupil edge reflection surface, so that at least N simulated cross-sections after rotation contain the simulated intersection line and are tangent to the simulated iris / pupil edge reflection surface.

[0310] Step 1755: Determine whether at least N simulated cross-sections are perpendicular to at least N simulated planes respectively. If the determination result is yes, proceed to step 1760. If the determination result is no, return to step 1751.

[0311] In a specific embodiment of this article, such as Figure 14 As shown above, an eyeball model and a face model are provided, and the rules for their mutual movement are followed according to specific rules. Multiple sets of different α and β values ​​are input, for example, the multiple sets of input α and β values ​​are (15, 15) / (0, 15) / (-15, 15) / (15, 0) / (0, 0) / (-15, 0) / (15, -15) / (0, -15) / (-15, -15).

[0312] Each input of a set of α and β values ​​changes the position of the user's eye model, alters the shape and position of the simulated iris / pupil edge reflection surface, and changes the spatial position of the simulated intersection lines with at least N simulated planes. Therefore, the spatial position of the simulated sectional planes also changes. This satisfies the premise that the line segments intersecting at least N simulated planes with at least N simulated sectional planes lie within the simulated iris / pupil edge reflection surface corresponding to the simulated iris / pupil edge curve in the repositioned eye model.

[0313] The intersection lines of at least N simulated planes and at least N simulated sections corresponding to the eyeball model position under multiple different α and β values ​​are compared. These intersections are then compared to the perpendicularity of the simulated planes and sections on the simulated iris / pupil edge reflection surface. The perpendicularity is the angle between the simulated plane and the simulated section; a closer angle to 90 degrees is considered higher. Finally, based on the at least N perpendicularities corresponding to the eyeball model position under multiple different α and β values, the set with the highest average perpendicularity among the at least N perpendicularities corresponding to the eyeball model position for each α and β value is selected as the final determined eyeball position.

[0314] The perpendicularity mentioned in this article is a generalized perpendicularity, which can be considered as the value of the included angle within the range of 90°, such as 89° to 91°, etc. This article does not limit the specific range of the included angle.

[0315] In this example, nine different α and β values ​​were set. More sets of different α and β values ​​can be set to improve the accuracy of the measurement and judgment of the user's line of sight on the terminal device. Alternatively, four different α and β values ​​can be selected from the nine sets, and then these four sets can be considered as a range. Specifically, the largest α value is x1, the smallest α value is x2, the largest β value is y1, and the smallest β value is y2. Multiple sets of α and β values ​​can then be set within this range, where the α value is between x1 and x2, and the β value is between y1 and y2. The process of determining the position of the eyeball model using the above steps is repeated multiple times until the position of the user's eyeball model is changed so that the line segments intersecting at least N simulated planes and at least N simulated cross-sections lie within the simulated iris / pupil edge reflection surface corresponding to the simulated iris / pupil edge curve in the repositioned eyeball model, and at least N simulated cross-sections are perpendicular to at least N simulated planes. This allows for a more accurate determination of the eyeball model's position.

[0316] In one embodiment of this paper, in order to simplify the calculation process and improve calculation efficiency, such as Figure 19 As shown, the implementation process of step 1750 above includes:

[0317] Step 1751': Change the position of the user's eye model relative to the face model;

[0318] Step 1752': Determine the intersection points of the iris / pupil edge curves with at least N simulation planes in the user's eye model;

[0319] Step 1753': Take the at least N intersection points determined in step 1752' as simulated tangent points, and determine at least N simulated tangent lines that each contain at least N simulated tangent points and are tangent to the iris / pupil edge curve;

[0320] In practice, simulated tangents can be generated randomly in advance. In this step, the simulated tangent plane is rotated so that it contains the simulated tangent point and is tangent to the edge curve of the iris / pupil.

[0321] Step 1754': Based on the at least N simulated tangents determined in step 1753' and the camera focus in the terminal device model, determine at least N simulated tangent planes;

[0322] Step 1755': Determine whether at least N simulated cross-sections are perpendicular to at least N simulated planes respectively. If the determination result is yes, proceed to step 1760. If the determination result is no, return to step 1751'.

[0323] The method in this example can omit the step of creating the iris edge reflection surface, thereby saving the computational load on the terminal device.

[0324] In one embodiment of this paper, a specific embodiment of the above-described location positioning method is also provided. In this embodiment, the actual plane and the simulated plane are respectively represented by the normal vectors of their respective planes. The normal vector of the actual plane is a known quantity, and the normal vector of each simulated plane i includes point C on the user-simulated iris / pupil edge curve. i The unknown quantity of the coordinates, C i Simulate any point on the iris / pupil edge curve for the user, where i is a positive integer from 1 to N.

[0325] Specifically, such as Figure 20 As shown, step 150 compares each group of simulated planes with the actual planes, and takes the simulated landing point position when the simulated plane and the actual plane are parallel as the actual landing point position of the user's line of sight on the terminal device, including:

[0326] Step 2010: Make the normal vector of the actual plane parallel to the normal vector of the simulated plane, and calculate point C on the simulated iris / pupil edge curve. i ,…,C N The coordinates.

[0327] Step 2020, based on point C i ,…,C N By using the coordinates and the equations of the motion trajectory of the simulated iris / pupil edge curve, the exact equation of the simulated iris / pupil edge curve can be calculated.

[0328] The equation set for the motion trajectory of the user-simulated iris / pupil edge curve includes: variables and equations that determine the spatial position of the user-simulated iris / pupil edge curve, and equations that determine point C on the user-simulated iris / pupil edge curve. i The variables and equations.

[0329] Step 2030: Determine the position of the visual axis of the user's eyeball model based on the exact equation of the user's simulated iris / pupil edge curve.

[0330] Step 2040: Determine the actual landing point of the user's line of sight on the terminal device based on the position of the visual axis of the user's eyeball model.

[0331] To more clearly illustrate the implementation process of this embodiment, the following explanation uses N=2 as an example. When N=2, the left and right eyes will respectively obtain two actual cross-sections, two actual planes, two simulated cross-sections, and two simulated planes. The following steps are required when implementing this embodiment:

[0332] 1) First, establish a coordinate system.

[0333] Using the focal point of the camera in the terminal device model as the origin of the absolute coordinate system, a three-dimensional rectangular coordinate system is established, including the x, y, and z axes, with the x, y, and z axes being perpendicular to each other. The x-axis of the three-dimensional rectangular coordinate system is parallel to the long side of the rectangular frame, and the z-axis is parallel to the short side of the rectangular frame. The coordinates of the origin are (0, 0, 0).

[0334] Assume the distance from the plane containing the rectangular frame to the focal point is w, where w can take any value. The center point of the rectangular frame is the line passing through the camera's focal point and perpendicular to the plane containing the rectangular frame, intersecting with the rectangular frame itself. Assume the length of the longer side of the rectangular frame is 2U, and the length of the shorter side is 2V, where U and V are known quantities and are both positive non-zero numbers. Then, the data expression for the rectangular frame is:

[0335]

[0336] 2) Establish the equation for the iris / pupil edge curve within the rectangular frame.

[0337] The user image is placed within a rectangular frame, and the equation for the iris / pupil edge curve is established using non-uniform rational B-splines. The specific implementation process of this embodiment is illustrated below using the left eye iris / pupil edge curve as an example. The implementation process of the right eye iris / pupil edge curve is the same as that of the left eye iris / pupil edge curve, and will not be described in detail here.

[0338] Assuming (x0, y0, z0) represents a point on the edge curve of the left iris / pupil, where x0, y0, and z0 are unknowns, the expression for the equation of the left iris / pupil edge curve is:

[0339]

[0340] Where f() is a function of the left eye iris / pupil edge curve. Take any point on the left eye iris / pupil edge curve and name it point A. Let x0 = a, then we have y0 = w and z0 = ∫(a), where a is a known value.

[0341] The coordinates of point A on the edge curve of the left eye's iris / pupil are (a, w, ∫(a)).

[0342] 3) Represent the actual plane using its normal vector, which is a known quantity. Taking the left eye as an example, the process of determining the normal vector of the actual plane includes:

[0343] (3.1) First determine the first actual tangent.

[0344] Based on the left eye iris / pupil edge curve Find the derivative function ∫'(x0)=K that is tangent to the curve at the edge of the left iris / pupil, where K is the value of the partial derivative with respect to x at x0.

[0345] Based on the derivative of the left eye iris / pupil edge curve, the tangent line to the left eye iris / pupil edge curve at point A is determined, which can be called the first actual tangent line, and its expression is:

[0346]

[0347] Specifically, the above equation can be calculated using the linear calculation formula based on the derivative function of the left eye iris / pupil edge curve and the coordinates of point A.

[0348] Take any point B on the first actual tangent line, and let x0 = b. Then we have y0 = w and z0 = ∫'(a)×(ba) + ∫(a), where b is a known value and b ≠ a.

[0349] The coordinates of any point B on the first actual tangent are known, and are (b, w, ∫'(a)×(ba)+∫(a)).

[0350] The coordinates of the camera focus O in the terminal device model are known values ​​(0, 0, 0).

[0351] (3.2) Calculate the normal vector of the first actual tangent and the camera focus O of the terminal device model.

[0352] Because the camera focus O of the terminal device model and any point A on the edge curve of the left iris / pupil are both within the first actual cross-section, the camera focus O of the terminal device model points to any point A on the edge reflection curve of the left iris / pupil, i.e., the vector... Parallel to the first actual cross-section.

[0353] The coordinates of the camera focus O in the terminal device model are known values ​​(0, 0, 0), and the coordinates of any point A on the edge curve of the left iris / pupil are known values ​​(a, w, ∫(a)). Then, the vector... for

[0354] Furthermore, since the first actual tangent plane is tangent to the edge curve of the left eye's iris / pupil, the tangent line of the left eye's iris / pupil edge curve at the point of tangency with the first actual tangent plane is on the first actual tangent plane, that is, the first actual tangent line is within the first actual tangent plane.

[0355] Because the first actual tangent is within the first actual tangent plane, and point A on the left iris / pupil edge curve (the point of tangency between the left iris / pupil edge curve and the first actual tangent) and any point B on the first actual tangent are both points on the first actual tangent plane, point A on the left iris / pupil edge curve and point B on the first actual tangent plane are both within the first actual tangent plane.

[0356] Therefore, the vector pointing from point A on the left eye's iris / pupil edge curve to point B on the first actual tangent line is the vector... Parallel to the first actual cross-section.

[0357] The coordinates of point A on the iris / pupil edge curve of the left eye are known values ​​(a, w, ∫(a)). The coordinates of any point B on the iris / pupil edge curve of the left eye are known values ​​(b, w, ∫'(a)×(ba)+∫(a)). Therefore, the vector... It is known that: Simplified to:

[0358] and The cross product is obtained perpendicular to and The vector, that is, the normal vector of the plane containing OAB (i.e., the first actual tangent plane), is:

[0359]

[0360] The principle of the cross product has the following formula:

[0361]

[0362] Where i = (1, 0, 0), j = (0, 1, 0), k = (0, 0, 1),

[0363] Based on the relationship between i, j, and k, we have:

[0364]

[0365] (3.3) Based on the normal vector of the first actual tangent and the vectors in the first actual tangent The normal vector of the first actual plane is calculated.

[0366] Because the first actual tangent plane is perpendicular to the first actual plane, the normal vector of the first actual tangent plane is, i.e. Parallel to the first actual plane.

[0367] Because the first actual line segment lies in the first actual plane, the vector containing the first actual line segment... Parallel to the first actual plane, and the normal vector of the first tangent plane, i.e. and the vector containing the first actual line segment Not parallel.

[0368] so and Cross products can be performed. The cross product yields the normal vector of the first actual plane.

[0369] in,

[0370]

[0371] (3.4) Using the principles of (3.1) to (3.3) above, the normal vector of the second actual plane is calculated.

[0372] Specifically, assuming there is any point A' on the left eye iris / pupil edge curve that does not coincide with any point A, then the coordinates of any point A' on the left eye iris / pupil edge curve are known, and are (a', w, ∫(a')).

[0373] Using the same method as above, the expression for the second actual tangent line of the iris / pupil edge curve is obtained as follows:

[0374]

[0375] Using the same method as above, take any point B' on the second actual tangent line, and let x0 = b'. Then we have y0 = w and z0 = ∫'(a')×(b'-a')+∫(a').

[0376] The coordinates of any point B' on the first actual tangent are known, and are (b', w, ∫'(a')×(b'-a')+∫(a')).

[0377] Finally, using the same method as above, the normal vector of the second actual plane is obtained as follows:

[0378]

[0379] 4) Represent each simulation plane with its normal vector. The normal vector of each simulation plane contains the unknowns of the coordinates of any point C1 and C2 on the user-simulated iris / pupil edge curve.

[0380] The following example, using the left eye, illustrates the process of determining the normal vector of each simulated plane i:

[0381] (4.1) Establish the spherical equation of the center of the simulated iris / pupil edge curve, i.e. the first sphere.

[0382] In this example, the focus of the camera in the user terminal device model is a known quantity, so the relative positional relationship between the left eye rotation point and the camera focus is fixed and known. Let the coordinates of the left eye rotation point be (G, H, I), where G, H, and I are known values.

[0383] In this example, the left eye iris / pupil edge curve in the left eyeball model is represented by a circle. Within the left eyeball model, the line connecting the center of the circle containing the left eye iris / pupil edge curve and the left eye rotation point is perpendicular to the plane containing the left eye iris / pupil edge curve. Let the distance between the center of the circle containing the left eye iris / pupil edge curve and the left eye rotation point be a known value L.

[0384] Assume the user's eyeball can rotate arbitrarily around the left eye rotation point. Therefore, taking the left eye as an example, the surface shape formed by the set of points that may exist at the center of the left eye iris / pupil edge curve is a sphere. That is, the sphere of the center of the left eye iris / pupil edge curve relative to the left eye rotation point of the user's eyeball model is called the first sphere. The coordinates of the center of the left eye iris / pupil edge curve on the x-axis, y-axis, and z-axis of the three-dimensional coordinate system are x1, y1, and z1, respectively, where x1, y1, and z1 are unknowns.

[0385] The equation of the first sphere is:

[0386] (x1-G) 2 +(y1-H) 2 +(z1-I) 2 =L 2 .

[0387] x1, y1, z1 are variables that determine the spatial position of the simulated iris / pupil edge curve, and the first spherical equation is the equation that determines the spatial position of the simulated iris / pupil edge curve.

[0388] (4.2) Establish the equation for the iris / pupil edge curve.

[0389] Because the shape of the iris / pupil edge curve in this paper is a circular curve in a three-dimensional coordinate system, the iris / pupil edge curve can be expressed by the intersection of a spherical equation and a planar equation.

[0390] Establish the spherical equation of point C1 on the simulated iris / pupil edge curve relative to its center. Let the radius of the circle containing the iris / pupil edge curve be a known value R. Construct a sphere with the center of the iris / pupil edge curve as the center and the radius R of the circle containing the iris / pupil edge curve as the radius; this sphere is called the second sphere. Construct a plane with the vector pointing from the center of the iris / pupil edge curve to the point of rotation as the normal vector; this plane is called the iris / pupil edge curve section. The intersection of the second sphere and the iris / pupil edge curve section is the iris / pupil edge curve itself.

[0391] Specifically, let the coordinates of any point C1 on the iris / pupil edge curve be (x2, y2, z2), and let the coordinates of any other point on the iris / pupil edge curve that does not coincide with any point C1 be C2, which is expressed as (x4, y4, z4).

[0392] (4.2.1) Calculate the second sphere.

[0393] The equation of the second sphere at any point C1 on the iris / pupil edge curve is (x2-x1). 2 +(y2-y1) 2 +(z2-z1) 2 =R 2 .

[0394] (4.2.2) Calculate the equation of the iris / pupil edge curve section. Starting from the center of the circle (x1, y1, z1) where the iris / pupil edge curve is located, and using the endpoint of the rotation point as the starting point, construct a vector... This vector is perpendicular to the curved section of the iris / aperture edge. The formula for calculating the vector is as follows:

[0395] According to the definition of the point normal form equation in the equation of a plane in a three-dimensional rectangular coordinate system, the point normal form equation of a plane is: A(X-X0) + B(Y-Y0) + C(Z-Z0) = 0, where (A, B, C) are tangent vectors perpendicular to the plane. Based on this principle, according to vector... User simulates point A on the iris / pupil edge curve. i Based on the user-simulated center points x1, y1, z1 of the iris / pupil edge curve, the equation for the cross-section of the iris / pupil edge curve can be calculated as follows:

[0396] (x1-G)(x2-x1)+(y1-H)(y2-y1)+(z1-I)(z2-z1)=0. Integrating the equations of the second sphere and the iris / pupil edge curve section, the intersection of the two yields the C1 equation system for any point on the iris / pupil edge curve:

[0397]

[0398] (x2, y2, z2) is used to determine point C on the simulated iris / pupil edge curve of the user. i variables, To determine point C on the simulated iris / pupil edge curve of the user i The variable equation.

[0399] Similarly, in this example, the system of equations for the iris / pupil edge curve at any point C2 on the iris / pupil edge curve is as follows:

[0400]

[0401] Combining this with the equation of the center of the circle containing the iris / pupil edge curve, i.e., the equation of the first sphere, we obtain the set of equations for the trajectory of the iris / pupil edge curve at any point C1 on the iris / pupil edge curve in this example:

[0402]

[0403] Similarly, in this example, the system of equations for the trajectory of the iris / pupil edge curve at any point C2 on the iris / pupil edge curve is as follows:

[0404]

[0405] These two sets of equations respectively express the point sets where the iris / pupil edge curves can exist in an eyeball model that can rotate arbitrarily around the eye's rotation point.

[0406] In this example, the virtual iris edge curve of the user's left eye is represented by a circle. The rule for the relative motion between the user's eye model and facial model is that a point in the user's eye model can rotate arbitrarily around the left eye rotation point. This article uses this rule for the relative motion between the user's eye model and facial model to deduce the point where the user's gaze lands on the terminal screen, based on the method mentioned in this article. There are other rules for the relative motion between the user's eye model and facial model, which can also be used to determine the point where the user's gaze lands on the terminal device based on the method and rules mentioned in this example.

[0407] In this example, the rules governing the relative motion of other user eye and facial models are specifically: the equations for the trajectory of the iris / pupil edge curve: consisting of variables and equations determining the spatial position of the simulated iris / pupil edge curve, and variables and equations determining point Ci on the simulated iris / pupil edge curve; which can be divided into ∫ 1 (x1,y1)=z1, (i takes a positive integer greater than or equal to 1), expressed in a general expression.

[0408] (4.3) Calculate the equation of the simulated tangent plane of the iris / pupil edge curve. The equation of the simulated tangent line is represented by the normal vector of the simulated tangent plane. Differentiate the system of equations for the motion trajectory of the iris / pupil edge curve obtained in (4.2), i.e.

[0409]

[0410] The point-slope form of the equation describing a straight line in a three-dimensional rectangular coordinate system:

[0411]

[0412] Let x3, y3, z3 be any point on the first simulated tangent line of the iris / pupil edge curve. Then, taking any point C1 of the iris / pupil edge curve as the tangent point, based on the point-slope form equation, the expression for the first simulated tangent line of the iris / pupil edge curve can be obtained as follows:

[0413]

[0414] Let D be any point on the first simulated tangent line, and let x3 = d (where d is a specific known value). Then we have

[0415] The coordinates of any point D on the first simulated tangent line are expressed as follows:

[0416]

[0417] The first simulated tangent and the plane containing the focal point O of the terminal device model camera form the first simulated tangent plane. Therefore, the vector pointing from the focal point O of the terminal device model camera to any point C1 on the iris / pupil edge curve is parallel to the first simulated tangent plane.

[0418] The coordinates of the focal point O of the terminal device model camera are known values ​​(0, 0, 0). The coordinates of any point C1 on the iris / pupil edge curve are expressed as (x2, y2, z2).

[0419] in:

[0420] so The expression is: (x2, y2, z2).

[0421] in:

[0422] Because the first simulated tangent line is within the first simulated tangent plane, and any point C1 (the point of tangency between the iris / pupil edge curve and the first simulated tangent line) on the iris / pupil edge curve and any point D on the first simulated tangent line are both points on the first simulated tangent line, therefore any point C1 on the iris / pupil edge curve and any point D on the first simulated tangent line are both within the first simulated tangent plane.

[0423] Any point C1 on the iris / pupil edge curve points to any point D on the first simulated tangent line, i.e. Parallel to the first simulated cross-section. The coordinates of any point C1 on the iris / pupil edge curve are expressed as: (x2, y2, z2).

[0424] The coordinates of any point D on the first simulated tangent line are expressed as:

[0425]

[0426] Therefore, vector Expressed as:

[0427] Right now:

[0428] in:

[0429] According to the cross product principle and The cross product of the two elements yields the product perpendicular to the line. and The vector, that is: the normal vector of the first simulated tangent plane is:

[0430]

[0431] Similarly, the normal vector of the second simulated tangent can be calculated based on C2.

[0432] (4.4) Calculate the equation of the simulated plane, which is represented by the normal vector of the simulated plane.

[0433] Because the first simulated tangent is perpendicular to the first simulated plane, the normal vector of the first simulated tangent is... Parallel to the simulated plane.

[0434] so Let be the normal vector of the first simulated plane.

[0435]

[0436] Similarly, based on the second simulated sectional plane, the normal vector of the second simulated plane can be calculated as follows:

[0437]

[0438] Solving (x1, y1, z1) yields the relative positional relationship between the center of the iris / pupil edge curve and the left eye rotation point. In this example, both the center of the iris / pupil edge curve and the left eye rotation point are on the visual axis, and the relative positional relationship of the left eye rotation point with respect to the camera focus of the user's face model, eyeball model, and terminal device model is determined and known in the entire virtual space.

[0439] Therefore, obtaining the specific values ​​of (x1, y1, z1) yields the relative positional relationship between the axis and the focal point of the camera model, thus completing the solution for the direction pointed by the user.

[0440] 5) Find the center (x1, y1, z1) of the iris / pupil edge curve of the left eye.

[0441] By solving for (x1, y1, z1), the relative positional relationship between the center of the left eye iris / pupil edge curve and the left eye rotation point is obtained. In this example, both the center of the left eye iris / pupil edge curve and the left eye rotation point are on the visual axis, and the relative positional relationship of the left eye rotation point with respect to the camera focus in the user's face model and the terminal device model is determined and known throughout the virtual space. Therefore, obtaining the specific values ​​of (x1, y1, z1) yields the relative positional relationship between the visual axis and the camera in the terminal device model, thus completing the solution for the direction pointed by the user.

[0442] The specific solution steps include:

[0443] Let the first simulated plane be parallel to the first actual plane, and the second simulated plane be parallel to the second actual plane. That is, the normal vector of the first simulated plane needs to be parallel to the normal vector of the first actual plane, and the normal vector of the second simulated plane needs to be parallel to the normal vector of the second actual plane.

[0444] The normal vector of the first simulated plane is:

[0445]

[0446] The normal vector of the first actual plane is:

[0447]

[0448] Since two normal vectors are parallel, their ratio is a constant. Therefore:

[0449]

[0450]

[0451]

[0452] The solution value of (x2, y2, z2) can be obtained through the above three equations. Then, (x2, y2, z2) can be substituted into the system of equations for the motion trajectory of the iris / pupil edge curve at any point C1 on the iris / pupil edge curve:

[0453]

[0454] The solution value of (x4, y4, z4) is obtained using the same method as above, and (x4, y4, z4) is substituted into the system of equations for the motion trajectory of the iris / pupil edge curve at any point C2 on the iris / pupil edge curve:

[0455]

[0456] Combining the equations of the iris / pupil edge curve at any point C1 on the iris / pupil edge curve with the equations of the iris / pupil edge curve at any point C2 on the iris / pupil edge curve, we obtain:

[0457]

[0458] The equations (x2, y2, z2, x4, y4, z4) have been solved.

[0459] By using the above equations, we can finally obtain the solutions for x1, y1, and z1. Substituting these solutions into the equations for the iris / pupil edge curve trajectory:

[0460]

[0461] The exact set of equations for the iris / pupil edge curves is obtained:

[0462]

[0463] The precise position of the user's eye model can be determined by moving the simulated iris / pupil edge curve in the user's eye model until it coincides with the curve expressed by the exact equation of the iris / pupil edge curve. Then, based on the intersection of the visual axis in the user's eye model and the terminal device model, the position of the user's gaze on the terminal device can be determined.

[0464] For the motion rules between other eyeball models and terminal device models, the solved (x2, y2, z2, x4, y4, z4) are substituted into the corresponding system of equations for the motion trajectory of the iris / pupil edge curve ∫ 1 (x1,y1)=z1, Similarly, the exact equations for the iris / pupil edge curves can be obtained:

[0465] The above method is used to determine the location where the user's gaze falls on the terminal device.

[0466] In the specific solution, x1, y1, z1 will yield two sets of solutions, representing two spatial locations of the center point of the left eye's iris / pupil edge curve, with these two points located on either side of the left eye's rotation point. Based on human physiology, the iris can only move within a limited area during eye movement; therefore, only one of the two spatial locations of the center point of the left eye's iris / pupil edge curve corresponds to reality.

[0467] In this example, the visual axis always passes through the center of the left eye iris / pupil edge curve and the left eye rotation point, and these two points do not coincide. The verticality of the left eye rotation point is known. Therefore, by finding the position of the center of the left eye iris / pupil edge curve, the spatial position of the visual axis can be obtained, which means the point where the user's gaze falls on the terminal device screen can be calculated.

[0468] The gaze position localization method provided in this paper can be simulated and verified in a computer. A three-dimensional simulation model of the left eyeball, a three-dimensional simulation model of the right eyeball, the left eye visual axis, the right eye visual axis, and a laptop computer model are established in the computer. Table 1 below shows the computer simulation results. From the simulation results, it can be confirmed that this paper can accurately determine the landing point of the user's gaze on the terminal device screen.

[0469] Table 1

[0470]

[0471] The angle and distance differences generated in this simulation experiment are obtained by comparing the visual axis of the simulated eye model with the visual axis in the known conditions, which are input into the calculation method mentioned in this paper. That is, the calculation error of the operator's eye landing point on the terminal screen is calculated by this method, and all other possible error factors are eliminated, such as the error between the relative position relationship between the camera model in the wireframe model of the laptop and the face model determined by taking a picture of the user's face and the actual relative position relationship between the laptop and the operator's face at the moment the picture is taken. The experiment estimates that if the error of the actual relative position relationship between the laptop and the operator's face exists, the error in judging the landing point of the operator's gaze on the terminal device will be 20 to 30 times the error in Table 1.

[0472] In one embodiment of this document, a terminal device control method is also provided, such as... Figure 21 As shown, it includes:

[0473] Step 2110: Execute the line-of-sight positioning method described in the foregoing embodiment to determine the actual landing point of the user's line of sight on the terminal device;

[0474] Step 2120: Generate operation instructions for the terminal device based on the actual position of the user's gaze on the terminal device. In detail, the operation instructions are often related to a specific operation corresponding to the actual gaze position; however, this article does not specify the exact relationship.

[0475] Based on the same inventive concept, this article also provides a line-of-sight positioning device, as described in the following embodiments. Since the principle by which the line-of-sight positioning device solves the problem is similar to that of the line-of-sight positioning method, the implementation of the line-of-sight positioning device can refer to the line-of-sight positioning method, and repeated details will not be elaborated further.

[0476] The line-of-sight positioning device provided in this embodiment includes multiple functional modules, all of which can be implemented by dedicated or general-purpose chips, or by software programs. This document does not limit the implementation of these modules.

[0477] Specifically, such as Figure 22 As shown, the line-of-sight positioning device includes:

[0478] The acquisition module 2210 is used to acquire facial images of the user.

[0479] The modeling module 2220 is used to generate a virtual space including a user model and a terminal device model based on the user's facial image or the principle of structured light, wherein the user model includes at least a user's eyeball model and user's facial feature points.

[0480] The calculation module 2230 is used to generate at least N actual planes based on the user's facial image and the terminal device model in the virtual space, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface.

[0481] The simulation module 2240 is used to adjust the simulated landing point position of the user's eyeball model's line of sight on the terminal device model in a virtual space according to a preset rule. For each simulated landing point position, multiple sets of simulated planes are determined. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface.

[0482] The comparison module 2250 is used to compare each group of simulated planes with the actual planes, and to take the simulated landing point position when the simulated planes are parallel to the actual planes as the actual landing point position of the user's line of sight on the terminal device.

[0483] In one embodiment of this article, such as Figure 23 As shown, a terminal device control device is also provided, comprising:

[0484] The landing point determination module 2310 is used to control the line of sight positioning device to execute the line of sight positioning method described in the foregoing embodiment, so as to determine the actual landing point of the user's line of sight on the terminal device.

[0485] The instruction generation module 2320 is used to generate operation instructions for the terminal device based on the actual position of the user's gaze on the terminal device. More specifically, the operation instructions are often related to a pair of operations corresponding to the actual gaze position; however, this article does not specify the exact relationship.

[0486] In one embodiment of this article, such as Figure 24 As shown, a computer device is also provided for executing the line-of-sight positioning method or terminal device control method described in any of the foregoing embodiments. Specifically, the computer device 2402 may include one or more processors 2404, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 2402 may also include any memory 2406 for storing any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 2406 may include any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Furthermore, any memory can provide volatile or non-volatile retention of information. Furthermore, any memory can represent a fixed or removable component of the computer device 2402. In one case, when the processor 2404 executes associated instructions stored in any memory or combination of memories, the computer device 2402 can perform any operation of the associated instructions. The computer device 2402 also includes one or more drive mechanisms 2408 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0487] Computer device 2402 may also include an input / output module 2410 (I / O) for receiving various inputs (via input device 2412) and providing various outputs (via output device 2414). A specific output mechanism may include a presentation device 2416 and an associated graphical user interface 2418 (GUI). In other embodiments, the input / output module 2410 (I / O), input device 2412, and output device 2414 may be omitted, and the device may function solely as a computer device within a network. Computer device 2402 may also include one or more network interfaces 2420 for exchanging data with other devices via one or more communication links 2422. One or more communication buses 2424 couple the components described above together.

[0488] Communication link 2422 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 2422 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0489] Corresponding to Figure 1 , Figure 17 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0490] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figure 1 , Figure 17 The method shown.

[0491] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0492] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0493] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0494] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0495] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0496] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0497] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0498] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0499] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A method for locating a line-of-sight position, characterized in that, include: Capture user facial images; Based on the user's facial image or the principle of structured light, a virtual space including a user model and a terminal device model is generated. The structured light system includes a projector and a camera. The projector projects specific light information onto the user's surface, which is then captured by the camera within the system. The user's position and depth information are calculated based on changes in the light signal caused by the user, reconstructing a three-dimensional space including the real user's face. The spatial relationship between the user and the terminal device is determined, and based on this relationship, a virtual space including the user model and the terminal device model is generated. The user model includes at least a user eye model and user facial feature points. Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface. In the virtual space, the simulated landing point position of the user's eyeball model's line of sight on the terminal device model is adjusted according to the pre-set rules. For each simulated landing point position, multiple sets of simulated planes are determined. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface. Compare each set of simulated planes with the actual planes, and take the simulated landing point position when the simulated plane is parallel to the actual plane as the actual landing point position of the user's line of sight on the terminal device.

2. The line-of-sight positioning method as described in claim 1, characterized in that, Generate a virtual space that includes user models and terminal device models, including: The user's facial image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance; Based on the user's facial image within the rectangular frame, determine the user's facial feature points within the rectangular frame; Based on the user's facial feature points within the rectangular frame and the camera focus in the terminal device model, establish the reflection lines of the user's facial feature points within the rectangular frame; Adjust the position of the user model so that the reflection lines of the facial feature points in the user model coincide with the reflection lines of the user feature points within the rectangular frame, thereby obtaining the spatial positional relationship between the user model and the terminal device model; Based on the spatial relationship between the user model and the terminal device model, a virtual space including the user model and the terminal device model is generated.

3. The line-of-sight positioning method as described in claim 1, characterized in that, Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, including: Based on the user's facial image and the terminal device model in the virtual space, determine the user's actual iris / pupil edge reflection surface; Determine at least N actual cross-sections of the actual iris / pupil edge reflection surface of the user, and the line segments intersecting each actual cross-section with the actual iris / pupil edge reflection surface of the user; Based on each actual cut surface and its associated intersecting line segments, determine the actual plane that is perpendicular to each actual cut surface and contains the associated intersecting line segments.

4. The line-of-sight positioning method as described in claim 3, characterized in that, Based on the user's facial image and the terminal device model in the virtual space, the actual iris / pupil edge reflection surface of the user is determined, including: The user's facial image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance; Based on the user's facial image within the rectangular frame, determine the user's actual iris / pupil edge curve; Using the camera focus point in the terminal device model as the vertex and the actual iris / pupil edge curve of the user as the guideline, a conical surface is obtained, which is the actual iris / pupil edge reflection surface of the user.

5. The line-of-sight positioning method as described in claim 1, characterized in that, Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, including: The user's facial image or facial vector image is loaded into a rectangular frame in front of the terminal device model, wherein the rectangular frame is used to represent the image size of the terminal device model's camera at a predetermined distance; Determine the actual iris / pupil edge curve of the user based on the user's facial image or facial vector image within the rectangular frame; Establish at least N actual tangent lines and tangent points for the actual iris / pupil edge curve of the user; Connect the tangent point to the focal point of the camera in the terminal device model to obtain the actual connection line between the tangent point and the focal point of the camera in the terminal device model; Determine the actual tangent based on each actual connecting line and its associated actual tangent; Based on each actual cut surface and its associated actual connecting lines, determine the actual plane that is perpendicular to each actual cut surface and contains the associated actual connecting lines.

6. The line-of-sight positioning method as described in claim 1, characterized in that, For any simulated landing point location, at least N simulated planes are determined, including: For any simulated landing point, based on the simulated iris / pupil edge curve in the user's eyeball model at that simulated landing point, establish the simulated iris / pupil edge reflection surface at that simulated landing point. Determine at least N simulated cross-sections of the user-simulated iris / pupil edge reflection surface and the line segments intersecting each simulated cross-section with the user-simulated iris / pupil edge reflection surface; Based on each simulated cross section and its associated intersecting line segments, determine a simulated plane that is perpendicular to each simulated cross section and contains the associated intersecting line segments.

7. The line-of-sight positioning method as described in claim 1, characterized in that, For any simulated landing point location, at least N simulated planes are determined, including: For any simulated landing point location, the simulated iris / pupil edge curve of the user at that simulated landing point location is determined based on the user's eyeball model at that simulated landing point location. Establish at least N simulated tangents and tangent points for the simulated iris / pupil edge curve of the user; Connect the tangent point to the focal point of the camera in the terminal device model to obtain a simulated connection line between the tangent point and the focal point of the camera in the terminal device model; Determine the simulated tangent based on each connecting line and its associated simulated tangent; Based on each simulated section and its associated simulated connecting lines, determine a simulated plane that is perpendicular to each simulated section and contains the associated simulated connecting lines.

8. The line-of-sight positioning method as described in claim 1, characterized in that, The actual plane and the simulated plane are represented by their respective normal vectors. The normal vector of the actual plane is a known quantity, and the normal vector of each simulated plane i includes point C on the user-simulated iris / pupil edge curve. i The unknown quantity of the coordinates, i, has a range of 1-N; Comparing each set of simulated planes with the actual planes, the simulated landing point position when the simulated planes are parallel to the actual planes is taken as the actual landing point position of the user's line of sight on the terminal device, including: By making the normal vector of the actual plane parallel to the normal vector of the simulated plane, the points C1,…C on the simulated iris / pupil edge curve of the user are calculated. i ,… ,C N The coordinates; Based on the points C1,…C i ,… ,C N The coordinates and equations of the user-simulated iris / pupil edge curve motion trajectory are used to calculate the exact equation of the user-simulated iris / pupil edge curve; wherein, the equations of the user-simulated iris / pupil edge curve motion trajectory include: variable equations that determine the spatial position of the user-simulated iris / pupil edge curve, and equations that determine point C on the user-simulated iris / pupil edge curve. i The equation of variables; Based on the exact equation of the simulated iris / pupil edge curve, determine the position of the visual axis of the user's eyeball model; Based on the position of the visual axis of the user's eyeball model, the actual landing point of the user's line of sight on the terminal device is determined.

9. A method for locating a line of sight, characterized in that, include: Capture user facial images; Based on the user's facial image or the principle of structured light, a virtual space including a user model and a terminal device model is generated. The structured light system includes a projector and a camera. The projector projects specific light information onto the object's surface, which is then captured by the camera. The position and depth information of the object are calculated based on the changes in the light signal caused by the object, thus reconstructing the entire three-dimensional space. The system structure consisting of the projector and camera installed on the terminal device reconstructs the three-dimensional space, including the real user's face, to determine the spatial relationship between the user and the terminal device. Based on this spatial relationship, a virtual space including the user model and the terminal device model is generated. The user model includes at least a user eye model and user facial feature points. Based on the user's facial image and the terminal device model in the virtual space, at least N actual planes are generated, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface. In the virtual space, at least N simulated planes passing through the focal point of the camera in the terminal device model are established; taking the focal point of the camera in the terminal device model as a fixed point, at least N simulated planes are rotated until they are parallel to at least N actual planes; The simulated landing point of the user's eye model's line of sight on the terminal device model is changed so that at least N simulated cross-sections of the simulated iris / pupil edge reflection surface are perpendicular to the N simulated planes, wherein the simulated iris / pupil edge reflection surface is determined by the simulated iris / pupil edge curve in the user's eye model; The simulated landing point of the user's eyeball model's gaze on the terminal device model is taken as the landing point of the user's gaze on the terminal device.

10. A terminal device control method, characterized in that, include: Using the method of any one of claims 1 to 8, determine the actual location where the user's line of sight falls on the terminal device; The operation instructions for the terminal device are generated based on the actual position of the user's line of sight on the terminal device.

11. A line-of-sight positioning device, characterized in that, include: The acquisition module is used to acquire the user's facial images; The modeling module is used to generate a virtual space including a user model and a terminal device model based on the user's facial image or the principle of structured light. The structured light system includes a projector and a camera. The projector projects specific light information onto the surface of an object, which is then captured by the camera. The position and depth information of the object are calculated based on the changes in the light signal caused by the object, thus reconstructing the entire three-dimensional space. The system structure consisting of the projector and camera installed on the terminal device reconstructs the three-dimensional space, including the real user's face, to determine the spatial relationship between the user and the terminal device. Based on this spatial relationship, a virtual space including the user model and the terminal device model is generated. The user model includes at least a user eye model and user facial feature points. The calculation module is used to generate at least N actual planes based on the user's facial image and the terminal device model in the virtual space, wherein the actual planes are perpendicular to the actual cross-section of the user's actual iris / pupil edge reflection surface; The simulation module is used to adjust the simulated landing point position of the user's eyeball model's line of sight on the terminal device model in a virtual space according to pre-set rules. For each simulated landing point position, multiple sets of simulated planes are determined. The number of simulated planes in each set of simulated planes is the same as the actual planes, and the simulated planes are perpendicular to the simulated cross-section of the user's simulated iris / pupil edge reflection surface. The comparison module is used to compare each group of simulated planes with the actual planes, and to take the simulated landing point position when the simulated planes are parallel to the actual planes as the actual landing point position of the user's line of sight on the terminal device.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the line-of-sight positioning method according to any one of claims 1-9.

13. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the line-of-sight positioning method according to any one of claims 1-9.

Images