A real-time pupil detection and tracking method

Abstract

The present invention relates to the field of human-computer interaction technology, and specifically discloses a real-time pupil detection and tracking method, comprising: collecting image information of a subject's pupil area through a visual sensor and performing preprocessing to obtain pupil feature parameters; extracting the pupil feature parameters to construct a sight estimation model and obtain sight direction information; constructing a sight direction mapping model based on the pupil feature parameters and sight landing point feature parameters, and outputting three-dimensional sight space parameters; obtaining a position deviation coefficient by performing gaze pattern analysis on the three-dimensional sight space parameters, judging the gaze pattern deviation according to the size of the position deviation coefficient and generating a calibration adjustment instruction; obtaining a position dynamic change curve of the three-dimensional sight space parameters within a preset time period, and judging the pupil state of the gaze pattern; and inputting a data set of the calibrated position deviation coefficient within the preset time period and preset visual scene position information into a preset three-dimensional model for training, constructing a three-dimensional space visual model, and performing verification and update.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a real-time pupil detection and tracking method. Background Art

[0002] Real-time pupil detection and tracking is an important research topic in computer vision, with widespread applications in fields such as face detection, human-computer interaction, and driver behavior monitoring. This method typically involves three stages: face detection, pupil detection, and pupil tracking. In recent years, scientists have conducted extensive research on pupil detection and tracking technologies to improve their accuracy and efficiency.

[0003] There are many ways to detect and track the human pupil, which can be divided into invasive and non-invasive methods based on the human eye's environmental perception. The invasive method refers to attaching certain auxiliary equipment to the subject's eyes to measure or estimate the line of sight, which may cause certain damage to the eyes. The non-invasive method mainly uses camera technology, infrared technology, and computer technology to estimate the direction of human eye gaze, which is harmless to the subject. Based on the detection conditions of the subjects and the goal of achieving tracking technology, non-invasive eye movement methods based on images and videos have become one of the current mainstream methods.

[0004] However, existing non-invasive methods are limited by the location of cameras and sensors, making it difficult to directly obtain the subject's pupil line of sight. In addition, the spatial range of pupil detection and tracking is limited, so the tolerance for perspective changes caused by head movement is small for the subject. In actual use, due to the differences in professional equipment and technologies used, the types of scenarios in which pupil detection and tracking methods are actually applied are mostly incompatible, making it difficult to achieve the high-precision requirements for eye line of sight estimation. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time pupil detection and tracking method to solve the following technical problems:

[0006] How to enhance the pupil sight interaction experience of visual participants and improve the spatial range and accuracy of pupil sight detection and tracking.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A real-time pupil detection and tracking method, comprising:

[0009] Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters;

[0010] Step 2: Extract pupil feature parameters to build a gaze estimation model and obtain gaze direction information; the gaze direction information includes gaze landing point feature parameters and gaze direction image;

[0011] Step 3: construct a sight direction mapping model based on pupil feature parameters and sight point feature parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters;

[0012] Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the visual space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction;

[0013] Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the visual space within a preset time period and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional spatial visual model;

[0014] Step 6: Input the real-time line of sight direction information into the three-dimensional space vision model for verification and update the current model to update the target visual scene position information.

[0015] Preferably, the pupil characteristic parameters in step 1 are obtained as follows:

[0016] Performing regional cropping on the pupil area image according to the matching template;

[0017] Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour;

[0018] Determine the pupil center coordinate point according to the pupil edge contour;

[0019] Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

[0020] Preferably, step 2 includes:

[0021] S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head;

[0022] S2. Determine the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left, and right of the subject;

[0023] S3, analyzing the angles between the pupil image within the target frame number range and the four orientation vision sensors to determine the sight-space relationship;

[0024] S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

[0025] Preferably, the method for extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is:

[0026] S41, obtaining pupil area features and determining the sight direction of the pupil corneal reflection point;

[0027] S42, using triangulation to calculate the linear sight point in the sight direction to generate sight point position information;

[0028] S43, performing coordinate conversion and vector operation on the sight point position information to obtain characteristic parameters of the sight point;

[0029] S44: Input the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

[0030] Preferably, step four includes:

[0031] Get the coordinates of the sight angle mapping direction point, which includes the position coordinate vectors on the x-axis, y-axis, and z-axis. By analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis, the first position parameter P is obtained. axi , the second position parameter P ayi , the third position parameter P azi ;

[0032] Gaze pattern analysis includes:

[0033] By formula Calculate the position deviation coefficient Co p ; Where f is the preset sight offset influence function; α1, α2, α3 are the first position parameters P axi , the second position parameter P ayi , the third position parameter P azi The preset weight coefficient of the influence on the standard landing point position; n is the total number of sight points in continuous time, and i∈[1,n]; P ax0 is the standard first position parameter; P ay0 is the standard second position parameter; P az0 It is the standard third position parameter.

[0034] Preferably, the position deviation coefficient Co p The threshold value interval of the coefficient of deviation from the preset standard position [Co A ,Co B ] for comparison:

[0035] If Co p <Co A , then the deviation of the gaze pattern is judged to be low;

[0036] If Co A ≤Co p ≤Co B , then the deviation of the gaze pattern is judged to be normal;

[0037] If Co p >Co B , then it is judged that the deviation of the gaze pattern is high, and a calibration adjustment instruction is issued.

[0038] Preferably, in step five:

[0039] Get the dynamic change curve E of the position of the three-dimensional parameters of the sight space within the preset time period p ;

[0040] Obtain the standard position dynamic change curve E of the three-dimensional parameters of the sight space in the historical database within the same preset time period p0 ;

[0041] Construct curve E in the same coordinate system p and curve E p0 , and respectively for the curve E of i in the target period p and curve E p0 Take the derivative and get E p ' and E p0 ';

[0042] Get E p ' and E p0 'Similarity value I E :

[0043]

[0044] Among them, θ i is the pupil state time influence function, and i is the target time period.

[0045] Preferably, the similarity value I E and the preset similarity threshold interval [I E1 , I E2 ] for comparison:

[0046] If I E ∈[I E1 , I E2 ], then it is judged to be consistent with the gaze pattern;

[0047] Otherwise, it is determined that it does not conform to the gaze mode, and the pupil state of the gaze mode is determined.

[0048] Preferably, the pupil states of the gaze pattern include gaze focus, saccadic movement and tracking movement.

[0049] Beneficial effects of the present invention:

[0050] (1) The present invention inputs the real-time acquired line of sight direction information into the three-dimensional space visual model to verify and update the current model, thereby achieving real-time update of the target visual scene position information and in-depth understanding of the scene visual changes. This construction not only enhances the system's real-time response capability to the user's visual behavior, but also improves the accurate tracking and analysis of scene information.

[0051] (2) The present invention receives and integrates the gaze direction information extracted from pupil features and gaze point feature parameters in real time, inputs the real-time gaze direction information into the reconstruction of the three-dimensional space based on the environment, and then verifies the three-dimensional space visual model to update the spatial information and visual object information; by constructing a gaze direction mapping model to output the three-dimensional parameters of the gaze space, and by converting the pupil feature parameters and the gaze point feature parameters into specific spatial coordinate information, a specific data basis is provided for accurate gaze direction analysis and application.

[0052] (3) The present invention analyzes the relationship changes among the first position parameter, the second position parameter, and the third position parameter, and mobilizes the influence of the number of sight points to obtain the position deviation coefficient, thereby realizing the analysis of the gaze pattern deviation, obtaining the calibration adjustment process, and realizing the flexible adjustment of visual tracking and pupil detection. In conjunction with the scene changes, the present invention realizes the construction of the subject and interactive information, as well as the visual tracking and scene interaction process of the subsequent experience personnel, thereby ensuring the rapid calibration process of further visual scene changes and realizing the upgrade of the interactive experience.

[0053] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 A diagram showing the steps of a real-time pupil detection and tracking method according to the present invention;

[0056] Figure 2 This is a diagram of the steps of the method for extracting pupil feature parameters and constructing a sight line estimation model in step 2 of the present invention;

[0057] Figure 3 This is a step diagram of the method for obtaining line of sight direction information according to the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] However, existing non-invasive methods are limited by the location of cameras and sensors, making it difficult to directly obtain the subject's pupil line of sight. In addition, the spatial range of pupil detection and tracking is limited, so the tolerance for perspective changes caused by head movement is small for the subject. In actual use, due to the differences in professional equipment and technologies used, the types of scenarios in which pupil detection and tracking methods are actually applied are mostly incompatible, making it difficult to achieve the high-precision requirements for eye line of sight estimation.

[0060] To solve the above technical problems, please refer to Figure 1 As shown, the present invention is a real-time pupil detection and tracking method, comprising:

[0061] Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters;

[0062] Step 2: Extract pupil feature parameters to build a gaze estimation model and obtain gaze direction information; the gaze direction information includes gaze landing point feature parameters and gaze direction image;

[0063] Step 3: construct a sight direction mapping model based on pupil feature parameters and sight point feature parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters;

[0064] Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the visual space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction;

[0065] Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the visual space within a preset time period and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional spatial visual model;

[0066] Step 6: Input the real-time line of sight direction information into the three-dimensional space vision model for verification and update the current model to update the target visual scene position information.

[0067] In the above technical solution, the present invention realizes the process of information collection, analysis, model construction and verification of real-time pupil detection and tracking through the above steps, realizes accurate prediction of actual user visual behavior and scene construction, and ensures that it provides strong technical support for virtual reality, human-computer interaction, driving safety and other fields.

[0068] Specifically, in step one, the image information of the pupil area of ​​the subject is collected through a visual sensor, and the image information of the pupil area is preprocessed to obtain pupil feature parameters; the eye image information of the subject and the subsequent real-time detection of the user is realized to obtain the pupil area, and the specific information of the pupil area is obtained through image preprocessing and feature selection technology. The specific pupil information includes pupil feature information, such as the pupil center position, diameter, shape area size, etc.

[0069] As an embodiment of the present invention, the pupil characteristic parameters in step 1 are obtained as follows:

[0070] Performing regional cropping on the pupil area image according to the matching template;

[0071] Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour;

[0072] Determine the pupil center coordinate point according to the pupil edge contour;

[0073] Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

[0074] In the above technical solution, in this embodiment, first, the input pupil area image is located in a specific area by using a matching template, and the template is usually matched according to the external contour shape of the pupil area image, such as a circle or an ellipse; of course, it can also be based on machine learning methods, such as a convolutional neural network model (CNN) to perform feature learning and position prediction on historical pupil area images, and the pupil area can be cropped by a machine algorithm of area cropping to remove unnecessary background interference information such as eyelashes, hair and other objects at the edge of the pupil, thereby improving the efficiency and accuracy of subsequent processing.

[0075] Image preprocessing methods include grayscale processing and edge detection, which involves converting the cropped image to a grayscale image while retaining accurate information for feature extraction. After grayscale processing, an edge detection algorithm (such as the Canny algorithm, Sobel operator, or Laplacian operator) is used to obtain the circular or elliptical edge outline of the pupil, obtain edge information from the image, and extract the pupil outline from the line of sight. Based on the edge outline, a geometric method such as least squares fitting is used to determine the geometric center of the pupil.

[0076] Finally, the pupil's characteristic parameters are calculated by combining the pupil center coordinate point and edge contour information. The pupil's characteristic parameters include the pupil's geometric center. Usually, the geometric center is not much different from the visual center. In subsequent calculations, the geometric center is used as a reference for the visual starting point coordinates for simulation. The pupil's diameter, area, circularity or irregularity index are also included, which is of great significance for iris recognition, pupil dynamic analysis and eye status assessment.

[0077] Step 2: Extract pupil feature parameters to build a gaze estimation model and obtain gaze direction information; gaze direction information includes gaze landing point feature parameters and gaze direction image; the pupil feature parameters obtained in step 1 are used to build a gaze estimation model using machine learning methods, and gaze direction information is obtained through the gaze estimation model, thereby ensuring the confirmation of the gaze origin point (pupil geometric center) and the gaze landing point (target object) as well as the gaze information connecting the two.

[0078] See also Figure 2 As shown, as an embodiment of the present invention, step 2 includes:

[0079] S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head;

[0080] S2. Determine the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left, and right of the subject;

[0081] S3, analyzing the angles between the pupil image within the target frame number range and the four orientation vision sensors to determine the sight-space relationship;

[0082] S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

[0083] In the above technical solution, this embodiment achieves precise tracking of the subject's gaze direction by collecting pupil feature parameters from multi-directional images, determining the installation orientation information of the visual sensor, analyzing the line of sight-space relationship, and intelligently identifying target features. Specifically, pupil feature parameters are first collected from continuous time frames of eye images based on the four main orientations of the subject's head. Based on the relative position of the subject's head, the subject's eye images are continuously collected from four different angles (up, down, left, and right), focusing on extracting pupil feature parameters. Digital high-precision visual sensors and fast image processing technology are used to ensure the real-time and accuracy of the collected pupil feature parameter data. Pupil feature parameters generally include pupil size, position, shape, etc., ensuring the expansion of the scope and accuracy of gaze direction analysis.

[0084] Then, determine the installation angle information and installation position coordinate information of the four azimuth visual sensors arranged above, below, left and right of the subject; by accurately determining the installation angle and position coordinate information of the four azimuth visual sensors, and using these information involving the acquisition orientation and equipment installation position as the basis to input the basic model for machine training, and then establish a mathematical model of line of sight and space, to ensure that the subsequent analysis of the line of sight direction can more accurately reflect the real situation; the installation of the sensor needs to take into account ergonomics and the natural distribution of line of sight angles to reduce errors.

[0085] Next, the angle between the eye pupil image within the target frame range and the four orientation vision sensors is analyzed to determine the line of sight-space relationship; by analyzing the angle difference between the eye pupil image within a specific frame range and the four orientation vision sensors, a mathematical model of line of sight and space is established, and the relationship between the line of sight direction and the spatial coordinates is determined through geometric calculation and angle conversion. The current mathematical model can analyze the tiny movements of the eyeball by comparing the changes in pupil images captured by vision sensors in different orientations, thereby inferring the spatial relationship between the line of sight direction and vision sensors in different orientations.

[0086] Finally, target feature recognition methods are used to extract gaze direction information from the gaze-space relationship. Common target feature recognition methods, such as deep learning-based algorithms, are used to extract specific gaze direction information from the gaze-space relationship, including gaze direction angle and gaze inclination parameters. Furthermore, the implementation involves combining multiple algorithms to improve the accuracy and robustness of the current recognition method.

[0087] See also Figure 3 As shown, as an embodiment of the present invention, the method of extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is:

[0088] S41, obtaining pupil area features and determining the sight direction of the pupil corneal reflection point;

[0089] S42, using triangulation to calculate the linear sight point in the sight direction to generate sight point position information;

[0090] S43, performing coordinate conversion and vector operation on the sight point position information to obtain characteristic parameters of the sight point;

[0091] S44: Input the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

[0092] This embodiment further enhances the aforementioned technical solution by first acquiring pupil region features and determining the direction of sight from the pupil corneal reflection. This is accomplished by accurately identifying and extracting pupil region feature parameters from the image, including pupil size, position, and shape. Furthermore, the position of the pupil corneal reflection, typically formed on the corneal surface by an external light source, is determined. By analyzing the relative position of the pupil and the corneal reflection, the direction of sight can be preliminarily inferred.

[0093] Then, triangulation is used to calculate the linear landing point of the line of sight in the direction of the line of sight to generate the line of sight landing point position information; using the principle of triangulation and based on the positional relationship between the pupil center, the corneal reflection point and the subject's eyes, the theoretical landing point of the line of sight in space is calculated to accurately determine the direction and landing point of the line of sight in space.

[0094] Next, the coordinate conversion and vector operation are performed on the line of sight landing point position information to obtain the characteristic parameters of the line of sight landing point; and after the line of sight landing point position is determined, the coordinate system is converted to realize the conversion from image coordinates to world geographic coordinates, and its current position information is obtained through vector operation to determine the characteristic parameters of the line of sight landing point, such as landing point coordinates, line of sight direction vector, etc.

[0095] Finally, the preset target space image and the characteristic parameters of the gaze landing point are input into the three-dimensional model for matching training to output the gaze direction image; through deep learning technology, the gaze direction image is trained and learned in the preset three-dimensional model, which intuitively displays the direction of the subject's gaze in three-dimensional space, providing an intuitive visual experience for subsequent data analysis and interactive applications.

[0096] The specific steps in step S4 above, from basic image feature extraction to complex triangulation and coordinate transformation, and then to deep learning model matching, demonstrate the entire process of high-precision gaze tracking technology.

[0097] Step 3: Construct a gaze direction mapping model based on pupil feature parameters and gaze landing point feature parameters, and output gaze space three-dimensional parameters; the gaze space three-dimensional parameters include gaze angle mapping direction point coordinates and mapping distance parameters.

[0098] This embodiment implements eye tracking technology by constructing a gaze direction mapping model to output three-dimensional gaze spatial parameters (including gaze angle mapping direction point coordinates and mapping distance parameters). This process converts pupil characteristic parameters and gaze landing point characteristic parameters into specific spatial coordinate information, providing a specific data foundation for subsequent gaze direction analysis and application. The gaze direction mapping model is constructed based on mathematical model data conversion, and the accuracy of the calculation is ensured by comprehensively considering factors such as the geometric structure of the eyeball, pupil position, and head posture.

[0099] Through the line of sight direction mapping model, the angular information of the line of sight in space, that is, the line of sight direction angle and inclination angle, is calculated; at the same time, the coordinates of the landing point of the line of sight in space, that is, the point where the line of sight intersects with the target object, are determined. This coordinate is usually expressed as the x, y, and z values ​​relative to the three-dimensional coordinate system; the distance between the landing point of the line of sight and the eyes is calculated through the mapping distance parameter, reflecting the understanding of the subject's gaze depth and distance perception during the interaction process; calculating the mapping distance usually requires combining the line of sight angle and landing point coordinate information, and is obtained through trigonometric functions or distance formulas.

[0100] Step 4: Analyze the gaze pattern of the three-dimensional parameters of the gaze space to obtain the position deviation coefficient. Based on the magnitude of the position deviation coefficient, determine the gaze pattern deviation and generate calibration adjustment instructions. Gaze calibration in any scenario is achieved by aligning the gaze point with the known gaze pattern through translation mapping and scaling. Using this information, a two-dimensional or three-dimensional gaze estimation model is established. This involves establishing a geometric equation or mathematical function that corresponds to the extracted two-dimensional eye movement features and the three-dimensional gaze point position, and then inversely calculating the estimated gaze point. Gaze calibration in any scenario is achieved by aligning the gaze point with the known gaze pattern through translation mapping and scaling.

[0101] Specifically, as an embodiment of the present invention, the method of analyzing the gaze pattern by the three-dimensional parameters of the line of sight space in step 4 is as follows: obtaining the position deviation coefficient; obtaining the coordinates of the line of sight angle mapping direction point, whose coordinates include the position coordinate vectors on the x-axis, y-axis, and z-axis, and performing parameter conversion by analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis respectively, to obtain the first position parameter P respectively. axi , the second position parameter P ayi , the third position parameter P azi ;

[0102] Gaze pattern analysis includes:

[0103] By formula Calculate the position deviation coefficient Co p ; Where f is the preset sight offset influence function; α1, α2, α3 are the first position parameters P axi , the second position parameter P ayi , the third position parameter P azi The preset weight coefficient of the influence on the standard landing point position; n is the total number of sight points in continuous time, and i∈[1,n]; P ax0 is the standard first position parameter; P ay0 is the standard second position parameter; P az0 It is the standard third position parameter.

[0104] In the above technical solution, in this embodiment, by analyzing the first position parameter Paxi , the second position parameter P ayi , the third position parameter P azi The relationship between the two is changed, the influence of the number of sight points is mobilized to obtain the position deviation coefficient, and then the gaze pattern deviation is analyzed, the calibration adjustment process is obtained, and the flexible adjustment of visual tracking and pupil detection is realized. The construction of interactive information of the subjects and the subsequent visual tracking and scene interaction process of the experience personnel are realized in conjunction with the scene changes, ensuring the rapid calibration process of further visual scene changes and realizing the upgrade of interactive experience.

[0105] It needs to be further explained that the preset sight offset influence function f is a deviation function set according to historical data to ensure that the result of the position deviation coefficient is within a specific reasonable range; ax0 、P ay0 、P az0 These are standard data values ​​obtained based on historical experience and will not be described in detail here.

[0106] As an embodiment of the present invention, the position deviation coefficient Co p The threshold value interval of the coefficient of deviation from the preset standard position [Co A ,Co B ] for comparison:

[0107] If Co p <Co A , then the deviation of the gaze pattern is judged to be low;

[0108] If Co A ≤Co p ≤Co B , then the deviation of the gaze pattern is judged to be normal;

[0109] If Co p >Co B , then it is judged that the deviation of the gaze pattern is high, and a calibration adjustment instruction is issued.

[0110] In the above technical solution, by analyzing the position deviation coefficient Co p The size of the deviation in the gaze mode is further analyzed, and then the calibration adjustment instruction is started according to the specific deviation change, specifically by comparing the position deviation coefficient Co p The threshold value interval of the coefficient of deviation from the preset standard position [Co A ,Co B ], if it falls within the standard range, the deviation is normal; if it is larger than the standard range, the deviation in the gaze mode is too high and calibration adjustment is required.

[0111] Step 5. Obtain the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period, and judge the pupil state of the gaze pattern by analyzing the changes in the three-dimensional parameters of the line of sight space; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional space vision model. The calibration and adjustment process is carried out through the output of the position deviation coefficient, specifically the calibration and adjustment of the gaze pattern, to ensure that a more accurate three-dimensional space vision model is obtained through the training and optimization of the current model.

[0112] Specifically, as an embodiment of the present invention, the pupil changes in the gaze mode are determined in step 5, and the specific adjustment process is as follows:

[0113] Get the dynamic change curve E of the position of the three-dimensional parameters of the sight space within the preset time period p ;

[0114] Obtain the standard position dynamic change curve E of the three-dimensional parameters of the sight space in the historical database within the same preset time period p0 ;

[0115] Construct curve E in the same coordinate system p and curve E p0 , and respectively for the curve E of i in the target period p and curve E p0 Take the derivative and get E p ' and E p0 ';

[0116] Get E p ' and E p0 'Similarity value I E :

[0117]

[0118] Among them, θ i is the pupil state time influence function, and i is the target time period.

[0119] In the above technical solution, in this embodiment, the position dynamic change curve E of the three-dimensional parameters of the visual space within the preset time period is first obtained. p ; and obtain the standard position dynamic change curve E of the three-dimensional parameters of the sight space in the historical database within the same preset time period p0 , using it as a control group to dynamically realize the characteristic comparison of spatial three-dimensional parameters, that is, constructing the curve E in the same coordinate system p and curve E p0 , and respectively for the curve E of i in the target period p and curve E p0 Take the derivative and get E p' and E p0 '; By taking the derivative of the curve, we can get the slope change in a specific time interval, and by getting E p ' and E p0 'Similarity value I E To judge the gaze mode, the specific calculation formula is: Among them, θ i is the pupil state time influence function, which is a similarity limiting function set according to historical data to ensure that the obtained ratio size conforms to the normal range of similarity values. i is the target time period, which is determined according to the gaze mode time period selected by the subject or the experience time period set by the initiator and is determined according to the specific implementation situation.

[0120] As an embodiment of the present invention, the similarity value I E and the preset similarity threshold interval [I E1 , I E2 ] for comparison:

[0121] If I E ∈[I E1 , I E2 ], then it is judged to be consistent with the gaze pattern;

[0122] Otherwise, it is determined that it does not conform to the gaze mode, and the pupil state of the gaze mode is determined.

[0123] In the above technical solution, in this embodiment, by obtaining the similarity value I E The size is used to judge whether the gaze pattern is consistent with the pupil change. Specifically, the similarity value I E and the preset similarity threshold interval [I E1 , I E2 ] to compare the size and determine if I E ∈[I E1 , I E2 ], it is judged to be consistent with the gaze mode; otherwise, it is judged not to be consistent with the gaze mode, and the pupil state of the gaze mode is determined.

[0124] As an embodiment of the present invention, the pupil state of the gaze mode includes gaze focus, saccadic movement and tracking movement.

[0125] In the above technical solution, pupil states in the gaze mode include fixation, saccadic eye movements, and tracking movements. These states reflect the different dynamic reactions of the eye when processing visual information. Fixation refers to the eye's stable fixation on a single point for a period of time to clearly see details; saccadic eye movements refer to the eye's rapid movement between two fixations, which is usually not accompanied by clear vision; and tracking movement refers to the eye's smooth tracking of a moving target to maintain clear vision of the target.

[0126] Step 6. Input the real-time gaze direction information into the three-dimensional space vision model for verification and update the current model to update the target visual scene position information; by verifying the acquired information in real time and updating the model according to the verification results, the construction of the three-dimensional space vision model based on the subject's guarantee and further deeper understanding of visual behavior and scene information tracking process are achieved.

[0127] In the above technical solution, the gaze direction information obtained in real time is input into the three-dimensional space visual model for verification and updating of the current model, further realizing the real-time update of the target visual scene position information and the deep understanding of the scene visual changes. Through construction, not only the real-time response capability of the system to the user's visual behavior is enhanced, but also the accurate tracking and analysis of scene information is improved. Specifically, by receiving and integrating the gaze direction information extracted from the pupil features and gaze landing point feature parameters in real time; including the gaze angle, direction point coordinates and mapping distance parameters, real-time data is provided for subsequent model verification and update. Then, the real-time gaze direction information is input into the reconstruction of the three-dimensional space based on the environment, and then the three-dimensional space visual model is verified to realize the update of spatial information and visual object information.

[0128] Furthermore, by comparing gaze direction information with the position and orientation of objects in the model, it is possible to verify whether the gaze is accurately directed at the target, as well as the relative relationship between the gaze and the object. This design provides a deeper understanding of user visual behavior, including attention allocation, point of interest analysis, and cognitive processes, and is of great value in various fields such as human-computer interaction design, virtual reality scene optimization, and advertising effectiveness evaluation.

[0129] This design also provides accurate gaze tracking and scene information tracking through real-time verification and model updates. It can also optimize user experience and enhance application effects based on a deep understanding of user visual behavior, which is conducive to promoting the further development of human-computer interaction and virtual reality technology.

[0130] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0131] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] The above content is merely an example and explanation of the concept of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined in this application, they should all fall within the scope of protection of the present invention.

Claims

1. A real-time pupil detection and tracking method, characterized in that: The method comprises: Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters; Step 2: Extract pupil feature parameters to build a sight line estimation model and obtain sight line direction information; the sight line direction information includes sight line landing point feature parameters and a sight line direction image; Step 3: construct a sight direction mapping model based on pupil characteristic parameters and sight point characteristic parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters; Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the visual space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction; Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the visual space within a preset time period and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional spatial visual model; Step 6: Input the real-time line of sight direction information into the three-dimensional space visual model for verification and update the current model to update the target visual scene position information; The step 4 includes: Get the coordinates of the sight angle mapping direction point, which includes the position coordinate vectors on the x-axis, y-axis, and z-axis. By analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis, the first position parameters are obtained respectively. , second position parameter , the third position parameter ; The gaze pattern analysis includes: By formula Calculate the position deviation coefficient ;in, It is the preset sight offset influence function; 、 、 The first position parameters , second position parameter , the third position parameter Preset weight coefficient of the influence on the standard landing point position; is the total number of sight points in continuous time, and ∈ ; is the standard first position parameter; is the standard second position parameter; It is the standard third position parameter.

2. A real-time pupil detection and tracking method according to claim 1, characterized in that: The pupil characteristic parameters in step 1 are obtained as follows: Performing regional cropping on the pupil area image according to the matching template; Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour; Determine the pupil center coordinate point according to the pupil edge contour; Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

3. A real-time pupil detection and tracking method according to claim 2, characterized in that: The second step includes: S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head; S2. Determine the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left, and right of the subject; S3, analyzing the angles between the pupil image within the target frame number range and the four orientation vision sensors to determine the sight-space relationship; S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

4. A real-time pupil detection and tracking method according to claim 3, characterized in that: The method for extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is: S41, obtaining pupil area features and determining the sight direction of the pupil corneal reflection point; S42, using triangulation to calculate the linear sight point in the sight direction to generate sight point position information; S43, performing coordinate conversion and vector operation on the sight point position information to obtain characteristic parameters of the sight point; S44: Input the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

5. A real-time pupil detection and tracking method according to claim 1, characterized in that: The position deviation coefficient The threshold value of the deviation coefficient from the preset standard position To compare: like < , then the deviation of the gaze pattern is judged to be low; like ≤ ≤ , then the deviation of the gaze pattern is judged to be normal; like > , then it is judged that the deviation of the gaze pattern is high, and a calibration adjustment instruction is issued.

6. A real-time pupil detection and tracking method according to claim 1, characterized in that: In the step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the sight space within a preset time period ; Obtain the standard position dynamic change curve of the three-dimensional parameters of the sight space in the historical database within the same preset time period ; Constructing curves in the same coordinate system and curves , and respectively for the target period curve and curves Take the derivatives and get and ; Get and Similarity value of : in, is the pupil state time influence function, The target time period.

7. A real-time pupil detection and tracking method according to claim 6, characterized in that: The similarity value Similarity threshold interval with the preset value To compare: like ∈ , then it is judged to be consistent with the current gaze mode; Otherwise, it is determined that it does not conform to the current gaze mode, and the pupil state of the gaze mode is determined.

8. A real-time pupil detection and tracking method according to claim 7, characterized in that: The pupil states of the gaze pattern include gaze focus, saccadic eye movement and tracking movement.

Images