Eye tracking method and system

Abstract

The present invention provides an eye tracking method and system, including obtaining a facial image to be processed, normalizing the facial image to be processed; locating the eye position in the normalized facial image, determining the eye range; binarizing the color values ​​of the pupil and the surrounding color within the eye range, using a predefined RGB color model, and finding the position information of the image with noise near the pupil removed by combining an image extraction method based on the RGB color model with an image binarization method; thereby obtaining the position coordinates of the pupil relative to the eye. The tracking method and system reduce the requirements for hardware equipment, can record the movement of people's eyeballs when observing information in different scenes in real time and accurately, and has broad application prospects in the fields of psychology, education, etc.

Description

Technical Field

[0001] The present invention belongs to the field of facial feature recognition, and in particular relates to an eye tracking method and system. Background Art

[0002] Eye-tracking can collect important information about the subjects' visual cognition by recording features such as eye movement trajectories and pupil changes when people process visual information. Abnormal eye movements are not only related to pathological structural changes in brain functional areas, but also reflect the decline of patients' cognitive functions. It is currently believed that characteristic eye movement changes are consistent with their pathological changes. However, among existing methods, most tracking technologies require professional equipment such as eye trackers to be implemented, which are costly, inconvenient to operate, and can only identify eye positions with slow response speeds.

[0003] Therefore, there is an urgent need for an eye tracking solution that can assist in the diagnosis and clinical differential diagnosis of autism. Summary of the invention

[0004] In order to achieve the above needs, the present invention provides an eye tracking method and system, which aims to solve the current lack of scientific and applicable means and technical methods for eye tracking, and can accurately obtain the position changes of the pupil, providing technical support for assisting the diagnosis and clinical differential diagnosis of autism.

[0005] The purpose of the present invention is achieved by adopting the following technical solutions:

[0006] An eye tracking method, comprising:

[0007] Acquire a facial image to be processed, and perform normalization processing on the facial image to be processed;

[0008] Locate the eye position in the normalized facial image and determine the eye range;

[0009] Binarize the color values ​​of the pupil and surrounding colors within the eye range, and use a predefined RGB color model to find the location information of the image near the pupil to remove the noise image by combining an image extraction method based on the RGB color model with an image binarization method;

[0010] Get the position coordinates of the pupil relative to the eye.

[0011] Preferably, the normalizing process of the facial image to be processed includes:

[0012] Acquire an original facial image and scale the size of the original facial image to a preset specific value to obtain a scaled image;

[0013] Calculating the mean and variance of pixels of the scaled image;

[0014] Obtaining each pixel value of the scaled image, subtracting each pixel value of the scaled image from its mean, and dividing by the variance to obtain a normalized facial image;

[0015] The collected original facial image is obtained by shooting with a terminal device equipped with a camera, and a label is added to the collected original facial image, and the image is stored according to the corresponding label.

[0016] Preferably, the step of locating the eye position in the facial image and determining the eye range comprises:

[0017] The normalized facial image is input into a pre-established deep learning model to extract eye image features;

[0018] Scaling the dimension of the feature map generated by the eye image features to the dimension obtained by calculation;

[0019] The scaled eye image features are input into the mean square error loss function for calculation, and the error coefficient between the detection box and the label value is defined as p1, the confidence loss is p2, and the error coefficient of the sum of the key point losses is p3;

[0020] Among them, p1>p2>p3;

[0021] Get the loss value based on p1, p2 and p3;

[0022] Obtain the loss value gradient based on the loss value and perform back propagation to obtain the overall eye positioning result and single-point accurate position information;

[0023] Determine the eye range based on the overall eye positioning results and the single point accurate position information.

[0024] Furthermore, the step of inputting the normalized facial image into a pre-established deep learning model to extract eye image features includes:

[0025] Inputting the normalized facial image into a pre-established deep learning model, and performing deep learning on the input normalized image information through a convolutional neural network in the deep learning model;

[0026] Extracting the eye image features according to the result of the deep learning;

[0027] Wherein, the deep learning model is obtained through training based on facial image data.

[0028] Furthermore, scaling the dimension of the feature map generated by the eye image feature to the dimension obtained by calculation includes:

[0029] Scaling the dimension of the feature map to (5+c+2*k) dimensions;

[0030] Among them, 5 is the coordinates x, y, w, h and confidence of the detection box, c is the object category, and k is the number of key points you want to obtain.

[0031] Preferably, the RGB color model is defined by the following formula:

[0032] (R(x,y)-G(x,y))+(R(x,y)-B(x,y))>T

[0033] Where R, G, and B represent red, green, and blue, respectively, i.e., the three primary colors; R(x, y) is the value of the red component, G(x, y) is the value of the green component, B(x, y) is the value of the blue component, and T is the selected threshold.

[0034] Preferably, the method of finding the position information of the image near the pupil to remove the noise by combining the image extraction method based on the RGB color model with the image binarization comprises:

[0035] The color values ​​of the pupil and the surrounding non-pupil area within the eye range are subjected to noise reduction and binarization processing to obtain a binarized image after grayscale enhancement;

[0036] Extract the position information matrix A of the pupil within the eye range in the binary image; in the matrix A, the color value of the pupil area is 1, and the color value of the non-pupil area is 0;

[0037] make:

[0038]

[0039] Where, the IMR matrix is ​​the value of each pixel in the red primary color layer of the original RGB image of the RGB color model; the IMG matrix refers to the value of each pixel in the green primary color layer of the original RGB image of the RGB color model; the IMB matrix refers to the value of each pixel in the blue primary color layer of the original RGB image of the RGB color model;

[0040] The noise images in the areas other than the pupil area in different color layers are removed by the above formula.

[0041] An eye tracking system, comprising:

[0042] A processing module, used for acquiring a facial image to be processed and performing normalization processing on the facial image to be processed;

[0043] A range determination module, used to locate the eye position in the normalized facial image and determine the eye range;

[0044] A tracking module is used to binarize the color values ​​of the pupil and surrounding colors within the eye range, and find the position information of the image near the pupil to remove the noise image by combining the image extraction method based on the RGB color model with the binarization of the image using a predefined RGB color model;

[0045] The positioning module is used to obtain the position coordinates of the pupil relative to the eye.

[0046] Compared with the closest prior art, the beneficial effects of the present invention are:

[0047] The present invention discloses a tracking method and system, which binarizes the color values ​​of the pupil and surrounding colors within the eye range, uses a predefined RGB color model, and finds the position information of the noise image near the pupil by combining an image extraction method based on the RGB color model with an image binarization method, thereby obtaining the position coordinates of the pupil relative to the eye. This effectively reduces the requirements for hardware equipment and solves the defects that most tracking technologies require professional equipment such as eye trackers to be implemented, are costly, inconvenient to operate, and can only identify the position of the eyes with a slow response speed.

[0048] The present invention can record in real time and accurately the eye movements of people when they observe information in different scenarios, can assist in the diagnosis and clinical differential diagnosis of autism, and has broad application prospects in the fields of psychology, education, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of an eye tracking process provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0051] In order to specifically understand the technical solution provided by the present invention, the technical solution of the present invention will be described and illustrated in detail in the following embodiments. Obviously, the embodiments provided by the present invention are not limited to the special details familiar to those skilled in the art. The preferred embodiments of the present invention are described in detail as follows. In addition to these descriptions, the present invention may also have other embodiments.

[0052] Embodiment 1:

[0053] like Figure 1 As shown, the present invention provides an eye tracking method, comprising the following steps:

[0054] S1: obtaining a facial image to be processed, and performing normalization processing on the facial image to be processed;

[0055] S2 locates the eye position in the normalized facial image and determines the eye range;

[0056] S3 binarizes the color values ​​of the pupil and surrounding colors within the eye range, and uses a predefined RGB color model to find the position information of the image near the pupil for removing noise by combining an image extraction method based on the RGB color model with the binarization of the image;

[0057] S4 obtains the position coordinates of the pupil relative to the eye.

[0058] In step S1, normalizing the facial image to be processed includes:

[0059] Acquire an original facial image and scale the size of the original facial image to a preset specific value to obtain a scaled image;

[0060] Calculating the mean and variance of pixels of the scaled image;

[0061] Obtaining each pixel value of the scaled image, subtracting each pixel value of the scaled image from its mean, and dividing by the variance to obtain a normalized facial image;

[0062] The collected original facial image is obtained by shooting with a terminal device equipped with a camera, and a label is added to the collected original facial image, and the image is stored according to the corresponding label.

[0063] In step S2, locating the eye position in the facial image and determining the eye range includes:

[0064] First, the normalized facial image is input into a pre-established deep learning model, and the input normalized image information is deep-learned by a convolutional neural network in the deep learning model; and the eye image features are extracted according to the result of the deep learning;

[0065] Wherein, the deep learning model is obtained through training based on facial image data.

[0066] The pre-establishment of deep learning models specifically includes:

[0067] First, take out the facial images and manually mark them as the training data set of the model. Assume Represents the shapes of all feature points in the image and builds a model using cascade regression.

[0068]

[0069] in, represents the feature shape predicted before iteration (a vector of coordinates), represents the predicted feature shape after update, t represents the cascade level, I represents the facial image, r t is the residual between the current shape and the true shape, (I,S (t) ) is the residual shape or coordinates of the current cascade layer.

[0070] Learning r through cascading t , find the model with the best effect, the algorithm steps are as follows:

[0071] (1) Initialize r0 first, and set the training data to Learning rate 0<λ<1(λ∈R 2P ), use the following formula to Initialize.

[0072]

[0073] (2) Iterative update r t :

[0074]

[0075] (3) Through r ik Get the regression equation After iteration, update as follows:

[0076]

[0077] (4) Loop update parameter r t and Until convergence or reaching the set number of iterations, the final output is:

[0078] After the model is trained, the images are punctuated to obtain the eye image features.

[0079] Then, the dimension of the feature map generated by the eye image features is scaled to the dimension obtained by calculation;

[0080] Scaling the dimension of the feature map to (5+c+2*k) dimensions;

[0081] Among them, 5 is the coordinates x, y, w, h and confidence of the detection box, c is the object category, and k is the number of key points you want to obtain.

[0082] Specifically, the convolutional layer is used to scale the feature map dimension to 5+c+2*k, where c is the object category (usually 1, indicating that only key points of a certain type of object are obtained, such as eyes), and k is the number of key points to be obtained. Taking the two key points of the eye as an example, c is 1 and k is 2, then the convolutional layer needs to scale the feature dimension to 16 dimensions, where 2 is the coordinates x, y, w, h and confidence score of the eye detection frame, and the coordinates kx and ky of the two key points in 2*k; x, y, w, h, kx, ky are all the ratios of the coordinates to the width and height of the face image.

[0083] Again, the scaled eye image features are input into the mean square error loss function for calculation, and the error coefficient between the detection box and the label value is defined as p1, the confidence loss is p2, and the error coefficient of the sum of the key point losses is p3;

[0084] Among them, p1>p2>p3;

[0085] Get the loss value based on p1, p2 and p3;

[0086] Obtain the loss value gradient based on the loss value and perform back propagation to obtain the overall eye positioning result and single-point accurate position information;

[0087] Determine the eye range based on the overall eye positioning results and the accurate position information of a single point.

[0088] In step S3, the RGB color model is defined by the following formula:

[0089] (R(x,y)-G(x,y))+(R(x,y)-B(x,y))>T

[0090] Where R, G, and B represent red, green, and blue, respectively, i.e., the three primary colors; R(x, y) is the value of the red component, G(x, y) is the value of the green component, B(x, y) is the value of the blue component, and T is the selected threshold.

[0091] By combining the image extraction method based on the RGB color model with the image binarization method, the position information of the image near the pupil for removing noise is found, including:

[0092] 1. Perform noise reduction and binarization on the color values ​​of the pupil and the surrounding non-pupil area within the eye range to obtain a binary image after grayscale enhancement; the image grayscale level ranges from 0 to 255, with a total of 256 levels. The binarization of the image is to select a value from the 256 levels as the threshold T, then the pixels greater than this threshold will be set to 1, and the pixels less than this threshold will be set to 0. If the pixel is 1, it means the grayscale is 255; if the pixel is 0, it means the grayscale is 0. After the image grayscale is enhanced first and then binarized, the image is clearer than the image directly binarized with the original image.

[0093] 2. Extract the position information matrix A of the pupil within the eye range in the binary image; in the matrix A, the color value of the pupil area is 1, and the color value of the non-pupil area is 0;

[0094] make:

[0095]

[0096] Where, the IMR matrix is ​​the value of each pixel in the red primary color layer of the original RGB image of the RGB color model; the IMG matrix refers to the value of each pixel in the green primary color layer of the original RGB image of the RGB color model; the IMB matrix refers to the value of each pixel in the blue primary color layer of the original RGB image of the RGB color model;

[0097] 3. The above formula is used to remove the noise images in the different color layers except the pupil area. That is, these three layers are merged into one matrix to obtain the final extracted pupil image. In this way, the position coordinates of the pupil relative to the eye are calculated.

[0098] Embodiment 2:

[0099] Based on the same technical concept, the present invention also provides an eye tracking system, comprising:

[0100] A processing module, used for acquiring a facial image to be processed and performing normalization processing on the facial image to be processed;

[0101] A range determination module, used to locate the eye position in the normalized facial image and determine the eye range;

[0102] A tracking module is used to binarize the color values ​​of the pupil and surrounding colors within the eye range, and find the position information of the image near the pupil to remove the noise image by combining the image extraction method based on the RGB color model with the binarization of the image using a predefined RGB color model;

[0103] The positioning module is used to obtain the position coordinates of the pupil relative to the eye.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than to limit its protection scope. Although the present application has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present application, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, and these changes, modifications or equivalent substitutions are all within the scope of the claims to be approved by the application.

Claims

1. An eye tracking method, characterized in that: include: Acquire a facial image to be processed, and perform normalization processing on the facial image to be processed; Locate the eye position in the normalized facial image and determine the eye range; The color values ​​of the pupil and surrounding colors within the eye range are binarized, and a predefined RGB color model is used. By combining an image extraction method based on the RGB color model with an image binarization method, the position information of the image near the pupil for removing noise is found, including: The color values ​​of the pupil and the surrounding non-pupil area within the eye range are subjected to noise reduction and binarization processing to obtain a binarized image after grayscale enhancement; Extract the position information matrix A of the pupil within the eye range in the binary image; in the matrix A, the color value of the pupil area is 1, and the color value of the non-pupil area is 0; make: Where, the IMR matrix is ​​the value of each pixel in the red primary color layer of the original RGB image of the RGB color model; the IMG matrix refers to the value of each pixel in the green primary color layer of the original RGB image of the RGB color model; the matrix refers to the value of each pixel in the blue primary color layer of the original RGB image of the RGB color model; The above formulas are used to remove noise images in areas other than the pupil area in different color layers; that is, the three layers are merged into one matrix to obtain the final extracted pupil image, thereby calculating the position coordinates of the pupil relative to the eye; Get the position coordinates of the pupil relative to the eye; The normalization process of the facial image to be processed comprises: Acquire an original facial image and scale the size of the original facial image to a preset specific value to obtain a scaled image; Calculating the mean and variance of pixels of the scaled image; Obtaining each pixel value of the scaled image, subtracting each pixel value of the scaled image from its mean, and dividing by the variance to obtain a normalized facial image; The collected original facial image is obtained by shooting with a terminal device equipped with a camera, and a label is added to the collected original facial image, and the image is stored according to the corresponding label; The RGB color model is defined by the following formula: (R (x,y) -G(x,y)) + (R (x,y) -B(x,y))>T Where R, G, and B represent red, green, and blue, respectively, i.e., the three primary colors; R(x, y) is the value of the red component, G(x, y) is the value of the green component, B(x, y) is the value of the blue component, and t is the selected threshold.

2. The method according to claim 1, characterized in that The method of locating the eye position in the facial image and determining the eye range comprises: The normalized facial image is input into a pre-established deep learning model to extract eye image features; Scaling the dimension of the feature map generated by the eye image features to the dimension obtained by calculation; The scaled eye image features are input into the mean square error loss function for calculation, and the error coefficient between the detection box and the label value is defined as p1, the confidence loss is p2, and the error coefficient of the sum of the key point losses is p3; Among them, p1>p2>p3; Get the loss value based on p1, p2 and p3; Obtain the loss value gradient based on the loss value and perform back propagation to obtain the overall eye positioning result and single-point accurate position information; Determine the eye range based on the overall eye positioning results and the single point accurate position information.

3. The method according to claim 2, characterized in that The step of inputting the normalized facial image into a pre-established deep learning model to extract eye image features comprises: Inputting the normalized facial image into a pre-established deep learning model, and performing deep learning on the input normalized image information through a convolutional neural network in the deep learning model; Extracting the eye image features according to the result of the deep learning; Wherein, the deep learning model is obtained through training based on facial image data.

4. The method according to claim 2, characterized in that: The step of scaling the dimension of the feature graph generated by the eye image feature to the dimension obtained by calculation comprises: Scaling the dimension of the feature map to (5+c+2*k) dimensions; Among them, 5 is the coordinates x, y, w, h and confidence of the detection box, c is the object category, and k is the number of key points you want to obtain.

5. An eye tracking system, characterized in that: include: A processing module, used for acquiring a facial image to be processed and performing normalization processing on the facial image to be processed; A range determination module, used to locate the eye position in the normalized facial image and determine the eye range; The tracking module is used to binarize the color values ​​of the pupil and surrounding colors within the eye range, using a predefined RGB color model, and combining an image extraction method based on the RGB color model with an image binarization method to find the position information of the image near the pupil to remove the noise, including: The color values ​​of the pupil and the surrounding non-pupil area within the eye range are subjected to noise reduction and binarization processing to obtain a binarized image after grayscale enhancement; Extract the position information matrix A of the pupil within the eye range in the binary image; in the matrix A, the color value of the pupil area is 1, and the color value of the non-pupil area is 0; make: Where, the IMR matrix is ​​the value of each pixel in the red primary color layer of the original RGB image of the RGB color model; the IMG matrix refers to the value of each pixel in the green primary color layer of the original RGB image of the RGB color model; the matrix refers to the value of each pixel in the blue primary color layer of the original RGB image of the RGB color model; The above formulas are used to remove noise images in areas other than the pupil area in different color layers; that is, the three layers are merged into one matrix to obtain the final extracted pupil image, thereby calculating the position coordinates of the pupil relative to the eye; A positioning module, used to obtain the position coordinates of the pupil relative to the eye; The normalization process of the facial image to be processed comprises: Acquire an original facial image and scale the size of the original facial image to a preset specific value to obtain a scaled image; Calculating the mean and variance of pixels of the scaled image; Obtaining each pixel value of the scaled image, subtracting each pixel value of the scaled image from its mean, and dividing by the variance to obtain a normalized facial image; The collected original facial image is obtained by shooting with a terminal device equipped with a camera, and a label is added to the collected original facial image, and the image is stored according to the corresponding label; The RGB color model is defined by the following formula: (R (x,y) -G(x,y)) + (R (x,y) -B(x,y))>T Where R, G, and B represent red, green, and blue, respectively, i.e., the three primary colors; R(x, y) is the value of the red component, G(x, y) is the value of the green component, B(x, y) is the value of the blue component, and t is the selected threshold.

Images