Real-time pupil detection and tracking method

Abstract

The invention relates to the technical field of human-computer interaction, and particularly discloses a real-time pupil detection and tracking method, which comprises the following steps of: acquiring pupil region image information of a subject through a visual sensor, and preprocessing to obtain pupil characteristic parameters; pupil feature parameters are extracted to construct a sight line estimation model, and sight line direction information is obtained; constructing a sight line direction mapping model according to the pupil feature parameters and the sight line drop point feature parameters, and outputting sight line space three-dimensional parameters; performing gaze mode analysis on the sight space three-dimensional parameters to obtain a position deviation coefficient, judging a gaze mode deviation condition according to the size of the position deviation coefficient, and generating a calibration adjustment instruction; obtaining a position dynamic change curve of the sight space three-dimensional parameters in a preset time period, and judging the pupil state of the gazing mode; and inputting the calibrated data set of the position deviation coefficients in the preset time period and the preset visual scene position information into a preset three-dimensional model for training, constructing a three-dimensional space visual model, and performing verification and updating.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a real-time pupil detection and tracking method. Background Art

[0002] Real-time pupil detection and tracking is an important research topic in the field of computer vision. It has wide applications in many fields such as face detection, human-computer interaction, and driver behavior monitoring. This method usually includes three stages: face detection, pupil detection, and pupil tracking. In recent years, scientists have conducted a lot of research on pupil detection and tracking technology to improve the accuracy and efficiency of detection and tracking.

[0003] There are many ways to detect and track the human pupil, which can be divided into invasive and non-invasive methods based on the human eye's perception of the environment. The invasive method refers to attaching certain auxiliary equipment to the subject's eyes to measure or estimate the line of sight, which is harmful to the eyes. The non-invasive method mainly uses camera technology, infrared technology and computer technology to estimate the direction of human eye gaze, which is harmless to the subject. Based on the test conditions of the subjects and the goal of achieving tracking technology, non-invasive eye movement methods based on images and videos have become one of the mainstream methods.

[0004] However, existing non-invasive methods are limited by the location of cameras and sensors, and it is difficult to directly obtain the subject's pupil line of sight. In addition, the spatial range of pupil detection and tracking is limited. Therefore, for the subject, the tolerance for perspective changes caused by head movement is relatively small. In actual use, due to the differences in professional equipment and technologies used, the types of scenarios in which pupil detection and tracking methods are actually applied are mostly incompatible, making it difficult to achieve high-precision requirements for eye line of sight estimation. Summary of the invention

[0005] The object of the present invention is to provide a real-time pupil detection and tracking method to solve the following technical problems: How to enhance the pupil sight interaction experience of visual participants and improve the spatial range and accuracy of pupil sight detection and tracking.

[0006] The purpose of the present invention can be achieved by the following technical solutions: A real-time pupil detection and tracking method, comprising: Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters; Step 2: Extract pupil feature parameters to build a sight line estimation model and obtain sight line direction information; the sight line direction information includes sight line landing point feature parameters and sight line direction image; Step 3: construct a sight direction mapping model according to pupil characteristic parameters and sight point characteristic parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters; Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the line of sight space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction; Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period, and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional space visual model; Step 6: Input the real-time line of sight direction information into the three-dimensional space visual model for verification and update the current model to update the target visual scene position information.

[0007] Preferably, the pupil characteristic parameters in step 1 are obtained by: Performing regional cropping on the pupil area image according to the matching template; Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour; Determine the pupil center coordinate point according to the pupil edge contour; Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

[0008] Preferably, step 2 includes: S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head; S2, determining the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left and right of the subject; S3, analyzing the angles between the pupil image of the eye within the target frame number range and the four orientation vision sensors to determine the line of sight-space relationship; S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

[0009] Preferably, the method for extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is: S41, acquiring pupil area features and determining the sight direction of the pupil cornea reflection point; S42, using triangulation to calculate the linear sight point in the sight direction to generate the sight point position information; S43, performing coordinate conversion and vector operation on the sight point position information to obtain the sight point characteristic parameters; S44, inputting the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

[0010] Preferably, step 4 includes: Get the coordinates of the direction point of the sight angle mapping, which includes the position coordinate vectors on the x-axis, y-axis, and z-axis. By analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis, the first position parameters are obtained. , second position parameter , the third position parameter ; Gaze pattern analysis includes: By formula Calculate the position deviation coefficient ;in, It is the preset sight offset influence function; , , The first position parameters are , second position parameter , the third position parameter Preset weight coefficients for the magnitude of the impact on the standard drop point position; is the total number of sight points in continuous time, and ∈ ; is the standard first position parameter; is the standard second position parameter; It is the standard third position parameter.

[0011] Preferably, the position deviation coefficient The threshold range of the coefficient of deviation from the preset standard position To compare: like < , then the bias in judging the gaze pattern is low; like ≤ ≤ , then the deviation of the gaze pattern is judged to be normal; like > , then the deviation of the gaze pattern is judged to be high, and a calibration adjustment instruction is issued.

[0012] Preferably, in step five: Get the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period ; Obtain the standard position dynamic change curve of the three-dimensional parameters of the line of sight space in the historical database within the same preset time period ; Constructing curves in the same coordinate system and curve , and respectively for the target period The curve and curve Take the derivative and get and ; Get and Similarity value :

[0013] in, is the pupil state time influence function, The target time period.

[0014] Preferably, the similarity value Similarity threshold interval To compare: like ∈ , then it is judged to be consistent with the current gaze mode; Otherwise, it is determined that it does not conform to the current gaze mode, and the gaze mode pupil state is determined.

[0015] Preferably, the pupil states of the gaze pattern include gaze focus, saccadic eye movement and tracking movement.

[0016] Beneficial effects of the present invention: (1) The present invention inputs the line of sight direction information obtained in real time into the three-dimensional space visual model to verify and update the current model, thereby achieving real-time update of the target visual scene position information and in-depth understanding of the scene visual changes. This construction not only enhances the system's real-time response capability to the user's visual behavior, but also improves the accurate tracking and analysis of scene information.

[0017] (2) The present invention receives and integrates the gaze direction information extracted from pupil features and gaze point feature parameters in real time, inputs the real-time gaze direction information into the reconstruction of the three-dimensional space based on the environment, and then verifies the three-dimensional space visual model to update the spatial information and visual object information; constructs a gaze direction mapping model to output the three-dimensional parameters of the gaze space, and converts the pupil feature parameters and the gaze point feature parameters into specific spatial coordinate information, thereby providing a specific data basis for accurate gaze direction analysis and application.

[0018] (3) The present invention analyzes the relationship changes among the first position parameter, the second position parameter, and the third position parameter, and obtains the position deviation coefficient by taking into account the influence of the number of sight points, thereby realizing the analysis of the gaze pattern deviation, obtaining the calibration adjustment process, and realizing the flexible adjustment of visual tracking and pupil detection. In conjunction with the scene changes, the present invention realizes the construction of the subject's interactive information and the subsequent visual tracking and scene interaction process of the experiencer, thereby ensuring the rapid calibration process of further visual scene changes and realizing the upgrade of the interactive experience.

[0019] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A step diagram of a real-time pupil detection and tracking method of the present invention; Figure 2 This is a step diagram of the method for extracting pupil feature parameters and constructing a sight line estimation model in step 2 of the present invention; Figure 3 This is a step diagram of the method for obtaining line of sight direction information of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] However, existing non-invasive methods are limited by the location of cameras and sensors, and it is difficult to directly obtain the subject's pupil line of sight. In addition, the spatial range of pupil detection and tracking is limited. Therefore, for the subject, the tolerance for perspective changes caused by head movement is relatively small. In actual use, due to the differences in professional equipment and technologies used, the types of scenarios in which pupil detection and tracking methods are actually applied are mostly incompatible, making it difficult to achieve high-precision requirements for eye line of sight estimation.

[0024] To solve the above technical problems, please refer to Figure 1 As shown, the present invention is a real-time pupil detection and tracking method, comprising: Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters; Step 2: Extract pupil feature parameters to build a sight line estimation model and obtain sight line direction information; the sight line direction information includes sight line landing point feature parameters and sight line direction image; Step 3: construct a sight direction mapping model according to pupil characteristic parameters and sight point characteristic parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters; Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the line of sight space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction; Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period, and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional space visual model; Step 6: Input the real-time line of sight direction information into the three-dimensional space visual model for verification and update the current model to update the target visual scene position information.

[0025] In the above technical scheme, the present invention realizes the process of information collection, analysis, model construction and verification of real-time pupil detection and tracking through the above steps, and the accurate prediction and scene construction of the visual behavior of the actual user by sight, ensuring that it provides strong technical support for the fields of virtual reality, human-computer interaction, driving safety, etc.

[0026] Specifically, in step one, the image information of the pupil area of ​​the subject is collected by a visual sensor, and the image information of the pupil area is preprocessed to obtain pupil feature parameters; the eye image information of the subject and the user who is subsequently detected in real time is obtained to obtain the pupil area, and the specific information of the pupil area is obtained through image preprocessing and feature selection technology. The specific pupil information includes pupil feature information, such as the pupil center position, diameter, shape area size, etc.

[0027] As an implementation mode of the present invention, the pupil characteristic parameters in step 1 are obtained as follows: Performing regional cropping on the pupil area image according to the matching template; Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour; Determine the pupil center coordinate point according to the pupil edge contour; Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

[0028] In the above technical scheme, in this embodiment, first, the input pupil area image is located in a specific area by using a matching template, and the template is usually matched according to the external contour shape of the pupil area image, such as a circle or an ellipse; of course, it can also be based on machine learning methods, such as a convolutional neural network model (CNN) to perform feature learning and position prediction on historical pupil area images, and the pupil area can be cropped by a machine algorithm of area cropping to remove unnecessary background interference information such as eyelashes, hair and other objects at the edge of the pupil, thereby improving the efficiency and accuracy of subsequent processing.

[0029] Then, the image preprocessing methods involved include grayscale processing and edge detection methods, including converting the cropped image into a grayscale image and retaining accurate information for feature extraction. After grayscale processing, an edge detection algorithm (such as the Canny algorithm, Sobel operator, or Laplacian operator) is used to obtain the circular or elliptical edge contour of the pupil, obtain edge information in the image, and extract the pupil contour by sight; and based on the edge contour, a geometric method such as least squares fitting is used to determine the geometric center of the pupil.

[0030] Finally, the pupil's characteristic parameters are calculated by combining the pupil center coordinate point and edge contour information. The pupil's characteristic parameters include the pupil's geometric center. Usually, the geometric center is not much different from the visual center. In subsequent calculations, the geometric center is used as a reference for the visual starting point coordinates for simulation. The pupil's diameter, area, circularity or irregularity index are also included, which is of great significance for iris recognition, pupil dynamic analysis and eye status assessment.

[0031] Step 2: Extract pupil feature parameters to build a line of sight estimation model and obtain line of sight direction information; the line of sight direction information includes line of sight landing point feature parameters and line of sight direction image; the pupil feature parameters obtained in step 1 are used to build a line of sight estimation model using an extreme learning method, and the line of sight direction information is obtained through the line of sight estimation model, thereby ensuring the confirmation of the line of sight originating point (pupil geometric center) and the line of sight landing point (target object) as well as the line of sight information connecting the two.

[0032] See also Figure 2 As shown, as an implementation mode of the present invention, step 2 includes: S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head; S2, determining the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left and right of the subject; S3, analyzing the angles between the pupil image of the eye within the target frame number range and the four orientation vision sensors to determine the line of sight-space relationship; S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

[0033] In the above technical scheme, this embodiment realizes accurate tracking of the subject's line of sight direction through the collection of pupil feature parameters of multi-directional images, determination of the installation orientation information of the visual sensor, line of sight-space relationship analysis, and intelligent recognition of target features. Specifically, first, the pupil feature parameters of the eye images of the four main orientations of the subject's head are collected in continuous time frames; according to the relative position of the subject's head, the subject's eye images are continuously collected from four different angles (up, down, left, and right), and the pupil feature parameters are extracted in detail. The digital high-precision visual sensor and fast image processing technology are used to ensure the real-time and accuracy of the collected pupil feature parameter data. The pupil feature parameters usually include the size, position, shape, etc. of the pupil, which ensures the expansion of the scope and accuracy of the line of sight direction analysis.

[0034] Then, determine the installation angle information and installation position coordinate information of the four azimuth visual sensors arranged above, below, left and right of the subject; accurately determine the installation angle and position coordinate information of the four azimuth visual sensors, and use these acquisition orientation and equipment installation position information as the basis to input the basic model for machine training and then establish a mathematical model of line of sight and space, to ensure that the subsequent analysis of the line of sight direction can more accurately reflect the real situation; the installation of the sensor needs to take into account ergonomics and the natural distribution of line of sight angles to reduce errors.

[0035] Next, the angle between the eye pupil image within the target frame number range and the four orientation vision sensors is analyzed to determine the line of sight-space relationship; by analyzing the angle difference between the eye pupil image within a specific frame number range and the four orientation vision sensors, a mathematical model of line of sight and space is established, and the relationship between the line of sight direction and the spatial coordinates is determined through geometric calculation and angle conversion. The current mathematical model can analyze the tiny movements of the eyeball by comparing the changes in pupil images captured by vision sensors in different orientations, thereby inferring the spatial relationship between the line of sight direction and vision sensors in different orientations.

[0036] Finally, the sight direction information in the sight-space relationship is extracted by using the target feature recognition method; by using the common target feature recognition method, such as the deep learning-based algorithm, the specific sight direction information is extracted from the sight-space relationship, including the sight direction angle, sight inclination parameters, etc. In addition, the specific implementation also involves the combined use of multiple algorithms to improve the accuracy and robustness of the current method recognition.

[0037] See also Figure 3As shown, as an embodiment of the present invention, the method of extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is: S41, acquiring pupil area features and determining the sight direction of the pupil cornea reflection point; S42, using triangulation to calculate the linear sight point in the sight direction to generate the sight point position information; S43, performing coordinate conversion and vector operation on the sight point position information to obtain the sight point characteristic parameters; S44, inputting the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

[0038] In the above technical solution, this embodiment further, first, obtain the pupil area characteristics and determine the sight direction of the pupil corneal reflection point; accurately identify and extract the characteristic parameters of the pupil area from the image, including the pupil size, position and shape; and determine the position of the pupil corneal reflection point, which is usually formed by an external light source on the corneal surface. By analyzing the relative position of the pupil and the corneal reflection point, the direction of the sight can be preliminarily inferred.

[0039] Then, triangulation is used to calculate the linear landing point of the line of sight direction to generate the line of sight landing point position information; the triangulation principle is used, and based on the positional relationship between the pupil center, the corneal reflection point and the subject's eye, the theoretical landing point of the line of sight in space is calculated to accurately determine the direction and landing point of the line of sight in space.

[0040] Next, the line of sight landing point position information is subjected to coordinate conversion and vector operation to obtain the line of sight landing point characteristic parameters; and after the line of sight landing point position is determined, the coordinate system is converted to realize the conversion from image coordinates to world geographic coordinates, and its current position information is obtained through vector operation to determine the characteristic parameters of the line of sight landing point, such as landing point coordinates, line of sight direction vector, etc.

[0041] Finally, the preset target space image and the characteristic parameters of the line of sight point are input into the three-dimensional model for matching training to output the line of sight direction image; through deep learning technology, the direction image is acquired by training and learning in the preset three-dimensional model, which intuitively displays the direction of the subject's line of sight in three-dimensional space, providing an intuitive visual experience for subsequent data analysis and interactive applications.

[0042] The specific steps in the above step S4, from basic image feature extraction, to complex triangulation and coordinate transformation, and then to deep learning model matching, demonstrate the entire process of high-precision gaze tracking technology.

[0043] Step 3: construct a sight direction mapping model based on pupil characteristic parameters and sight point characteristic parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters.

[0044] In this embodiment, the eye tracking technology process is implemented: by constructing a gaze direction mapping model to output the three-dimensional parameters of the gaze space (including the coordinates of the gaze angle mapping direction point and the mapping distance parameters), and by converting the pupil characteristic parameters and the gaze landing point characteristic parameters into specific spatial coordinate information, a specific data basis is provided for subsequent gaze direction analysis and application. The gaze direction mapping model is constructed by data conversion based on a mathematical model, and the accuracy of the calculation is ensured by comprehensively considering factors such as the geometric structure of the eyeball, the pupil position, and the head posture.

[0045] The line of sight direction mapping model is used to calculate the angle information of the line of sight in space, that is, the line of sight direction angle and inclination angle. At the same time, the coordinates of the point where the line of sight falls in space are determined, that is, the point where the line of sight intersects with the target object. This coordinate is usually expressed as x, y, and z values ​​relative to a three-dimensional coordinate system. The mapping distance parameter is used to calculate the distance between the line of sight landing point and the eyes, reflecting the understanding of the subject's gaze depth and distance perception during the interaction process. Calculating the mapping distance usually requires combining the line of sight angle and the landing point coordinate information, and is obtained through trigonometric functions or distance formulas.

[0046] Step 4: Obtain the position deviation coefficient by analyzing the gaze pattern of the three-dimensional parameters of the line of sight space, determine the deviation of the gaze pattern according to the size of the position deviation coefficient and generate a calibration adjustment instruction; realize gaze calibration in any scene by aligning the translation mapping and scale transformation of the gaze point with the known gaze pattern. A two-dimensional or three-dimensional line of sight estimation model is established through the obtained line of sight parameter information, that is, a geometric equation or a mathematical function of the corresponding relationship is established between the extracted two-dimensional eye movement features and the three-dimensional gaze point position, and the estimated gaze point is obtained by reverse calculation. Gaze calibration in any scene is realized by aligning the translation mapping and scale transformation of the gaze point with the known gaze pattern.

[0047] Specifically, as an implementation of the present invention, the method of analyzing the gaze pattern through the three-dimensional parameters of the line of sight space in step 4 is: obtaining the position deviation coefficient; obtaining the coordinates of the line of sight angle mapping direction point, whose coordinates include the position coordinate vectors on the x-axis, y-axis, and z-axis, and performing parameterization conversion by analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis respectively, to obtain the first position parameters respectively. , second position parameter , the third position parameter ; Gaze pattern analysis includes: By formula Calculate the position deviation coefficient ;in, It is the preset sight offset influence function; , , The first position parameters are , second position parameter , the third position parameter Preset weight coefficients for the magnitude of the impact on the standard drop point position; is the total number of sight points in continuous time, and ∈ ; is the standard first position parameter; is the standard second position parameter; It is the standard third position parameter.

[0048] In the above technical solution, in this embodiment, by analyzing the first position parameter , second position parameter , the third position parameter The relationship between the two is changed, and the influence of the number of sight points is mobilized to obtain the position deviation coefficient, thereby realizing the analysis of the gaze pattern deviation, obtaining the calibration adjustment process, realizing the flexible adjustment of visual tracking and pupil detection, and coordinating with the scene changes to realize the construction of the subject, the construction of interactive information and the subsequent visual tracking and scene interaction process of the experience personnel, ensuring the rapid calibration process of further visual scene changes and realizing the upgrade of interactive experience.

[0049] What needs further explanation is that the preset sight offset affects the function It is a deviation function set according to historical data to ensure that the result of the position deviation coefficient is within a specific reasonable range; , , These are standard data values ​​obtained based on historical experience and will not be described in detail here.

[0050] As an embodiment of the present invention, the position deviation coefficient The threshold range of the coefficient of deviation from the preset standard position To compare: like < , then the bias in judging the gaze pattern is low; like ≤ ≤ , then the deviation of the gaze pattern is judged to be normal; like > , then the deviation of the gaze pattern is judged to be high, and a calibration adjustment instruction is issued.

[0051] In the above technical solution, by analyzing the position deviation coefficient The size of the deviation in the gaze mode is further analyzed, and then the calibration adjustment instruction is started according to the specific deviation change. Specifically, the position deviation coefficient is compared The threshold range of the coefficient of deviation from the preset standard position If the size of is within the standard range, the deviation is normal. If it is larger than the standard range, the deviation in the current gaze mode is too high and needs to be calibrated.

[0052] Step 5. Obtain the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period, and judge the pupil state of the gaze mode by analyzing the changes in the three-dimensional parameters of the line of sight space; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional space visual model, and perform calibration and adjustment through the output of the position deviation coefficient, specifically calibrating and adjusting the gaze mode, to ensure that a more accurate three-dimensional space visual model is obtained through training and optimization of the current model.

[0053] Specifically, as an implementation of the present invention, in step 5, the pupil change in the current gaze mode is determined, and the specific adjustment process is as follows: Get the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period ; Obtain the standard position dynamic change curve of the three-dimensional parameters of the line of sight space in the historical database within the same preset time period ; Constructing curves in the same coordinate system and curve , and respectively for the target period The curve and curve Take the derivative and get and ; Get and Similarity value :

[0054] in, is the pupil state time influence function, The target time period.

[0055] In the above technical solution, in this embodiment, the position dynamic change curve of the three-dimensional parameters of the line of sight space within a preset time period is first obtained. ; and obtain the standard position dynamic change curve of the three-dimensional parameters of the line of sight space in the historical database within the same preset time period , using this as a control group to dynamically compare the characteristics of spatial three-dimensional parameters, that is, to construct a curve in the same coordinate system and curve , and respectively for the target period The curve and curve Take the derivative and get and ; By taking the derivative of the curve, we can get the slope change in a specific time interval, and by getting and Similarity value To judge the current gaze mode, the specific calculation formula is: in, is the pupil state time influence function, which is a similarity limiting function set according to historical data to ensure that the ratio obtained is in line with the normal range of similarity values. It is the target time period, which is determined based on the gaze pattern time period selected by the subject or the experience time period set by the initiator, and is determined according to the specific implementation situation.

[0056] As an implementation mode of the present invention, the similarity value Similarity threshold interval To compare: like ∈ , then it is judged to be consistent with the current gaze mode; Otherwise, it is determined that it does not conform to the current gaze mode, and the gaze mode pupil state is determined.

[0057] In the above technical solution, in this embodiment, by obtaining the similarity value size to determine whether the current gaze mode is consistent with pupil changes. Similarity threshold interval Compare the sizes and determine if ∈ , it is judged to be consistent with the current gaze mode; otherwise, it is judged not to be consistent with the current gaze mode, and the pupil state of the gaze mode is determined.

[0058] As an embodiment of the present invention, the pupil state of the gaze mode includes gaze focus, saccadic movement and tracking movement.

[0059] In the above technical solution, the pupil states in the gaze mode include gaze focus, eye saccade and tracking movement, which reflect the different dynamic response processes of the eyes when processing visual information. Among them, gaze focus refers to the eyes stably fixating on one point for a period of time in order to clearly see the details; eye saccade refers to the process of the eyes moving quickly between two gaze points, which is usually not accompanied by clear vision; tracking movement refers to the eyes smoothly tracking the moving target to maintain clear vision of the target.

[0060] Step 6. Input the line of sight direction information obtained in real time into the three-dimensional space vision model for verification and update the current model to update the target visual scene position information; by verifying the obtained information in real time and updating the model according to the verification results, the construction of the three-dimensional space vision model based on the subject's guarantee and further understanding of the visual behavior and tracking of the scene information can be achieved.

[0061] In the above technical solution, the line of sight direction information acquired in real time is input into the three-dimensional space visual model for verification and updating the current model, further realizing the real-time update of the target visual scene position information and the deep understanding of the scene visual changes. Through construction, not only the real-time response capability of the system to the user's visual behavior is enhanced, but also the accurate tracking and analysis of scene information is improved. Specifically, by real-time receiving and integrating the line of sight direction information extracted from the pupil features and line of sight landing point feature parameters; including line of sight angle, direction point coordinates and mapping distance parameters, real-time data is provided for subsequent model verification and update. Then, the real-time line of sight direction information is input into the reconstruction of the three-dimensional space based on the environment, and then the three-dimensional space visual model is verified to realize the update of spatial information and visual object information.

[0062] Furthermore, by comparing the line of sight direction information with the position and direction of the object in the model, it is possible to verify whether the line of sight is accurately pointing to the target, as well as the relative relationship between the line of sight and the object. This design ensures a deeper understanding of the user's visual behavior, including attention allocation, interest point analysis, cognitive process, etc., which is of great value in various fields such as human-computer interaction design, virtual reality scene optimization, and advertising effect evaluation.

[0063] Through real-time verification and model updating, this design can not only provide accurate gaze tracking and scene information tracking, but also optimize user experience and improve application effects based on a deep understanding of user visual behavior, which is conducive to promoting the further development of human-computer interaction and virtual reality technology.

[0064] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0065] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0066] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined in this application, they shall all fall within the protection scope of the present invention.

Claims

1. A real-time pupil detection and tracking method, characterized in that: The method comprises: Step 1: Collecting pupil area image information of the subject through a visual sensor, and preprocessing the pupil area image information to obtain pupil feature parameters; Step 2: extract pupil feature parameters to build a sight line estimation model and obtain sight line direction information; the sight line direction information includes sight line landing point feature parameters and a sight line direction image; Step 3: construct a sight direction mapping model according to pupil characteristic parameters and sight point characteristic parameters, and output sight space three-dimensional parameters; the sight space three-dimensional parameters include sight angle mapping direction point coordinates and mapping distance parameters; Step 4: Obtain a position deviation coefficient by performing gaze pattern analysis on the three-dimensional parameters of the line of sight space, determine the gaze pattern deviation according to the size of the position deviation coefficient, and generate a calibration adjustment instruction; Step 5: Obtain the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period, and determine the pupil state of the gaze mode; and input the calibrated data set of the position deviation coefficient within the preset time period and the preset visual scene position information into the preset three-dimensional model for training and constructing a three-dimensional space visual model; Step 6: Input the real-time line of sight direction information into the three-dimensional space visual model for verification and update the current model to update the target visual scene position information.

2. A real-time pupil detection and tracking method according to claim 1, characterized in that: The pupil characteristic parameters in step 1 are obtained as follows: Performing regional cropping on the pupil area image according to the matching template; Perform grayscale processing on the cropped image area and perform edge detection to obtain the pupil edge contour; Determine the pupil center coordinate point according to the pupil edge contour; Output pupil feature parameters based on pupil center coordinate point parameters and pupil edge contour area.

3. A real-time pupil detection and tracking method according to claim 2, characterized in that: The step 2 includes: S1, collecting pupil feature parameters of eye images of continuous time frames based on four main orientations of the subject's head; S2, determining the installation angle information and installation position coordinate information of four azimuth vision sensors arranged above, below, left and right of the subject; S3, analyzing the angles between the pupil image of the eye within the target frame number range and the four orientation vision sensors to determine the line of sight-space relationship; S4. Extract the sight direction information in the sight-space relationship through the target feature recognition method.

4. A real-time pupil detection and tracking method according to claim 3, characterized in that: The method for extracting the sight direction information in the sight-space relationship by the target feature recognition method in step S4 is: S41, acquiring pupil area features and determining the sight direction of the pupil cornea reflection point; S42, using triangulation to calculate the linear sight point in the sight direction to generate the sight point position information; S43, performing coordinate conversion and vector operation on the sight point position information to obtain the sight point characteristic parameters; S44, inputting the preset target space image and the sight point feature parameters into the three-dimensional model for matching training to output the sight direction image.

5. A real-time pupil detection and tracking method according to claim 1, characterized in that: The step 4 includes: Get the coordinates of the direction point of the sight angle mapping, which includes the position coordinate vectors on the x-axis, y-axis, and z-axis. By analyzing the values ​​of the coordinate mapping distance on the x-axis, y-axis, and z-axis, the first position parameters are obtained. , second position parameter , the third position parameter ; The gaze pattern analysis includes: By formula Calculate the position deviation coefficient ;in, It is the preset sight offset influence function; , , The first position parameters are , second position parameter , the third position parameter Preset weight coefficients for the magnitude of the impact on the standard drop point position; is the total number of sight points in continuous time, and ∈ ; is the standard first position parameter; is the standard second position parameter; It is the standard third position parameter.

6. A real-time pupil detection and tracking method according to claim 5, characterized in that: The position deviation coefficient The threshold range of the coefficient of deviation from the preset standard position To compare: like < , then the bias in judging the gaze pattern is low; like ≤ ≤ , then the deviation of the gaze pattern is judged to be normal; like > , then the deviation of the gaze pattern is judged to be high, and a calibration adjustment instruction is issued.

7. A real-time pupil detection and tracking method according to claim 1, characterized in that: In the step 5: Get the dynamic change curve of the position of the three-dimensional parameters of the line of sight space within a preset time period ; Obtain the standard position dynamic change curve of the three-dimensional parameters of the line of sight space in the historical database within the same preset time period ; Constructing curves in the same coordinate system and curve , and respectively for the target period The curve and curve Take the derivative and get and ; Get and Similarity value : in, is the pupil state time influence function, The target time period.

8. A real-time pupil detection and tracking method according to claim 7, characterized in that: Similarity value Similarity threshold interval To compare: like ∈ , then it is judged to be consistent with the current gaze mode; Otherwise, it is determined that it does not conform to the current gaze mode, and the gaze mode pupil state is determined.

9. A real-time pupil detection and tracking method according to claim 8, characterized in that: The pupil states of the gaze pattern include gaze focus, saccadic eye movement and tracking movement.

Images