YOLOv5-based sight line tracking method in free motion state of head

Abstract

The invention provides a YOLOv5-based sight tracking method in a head free motion state, and belongs to the technical field of sight tracking, and the method comprises the steps: recognizing two eye regions from a face image through employing a YOLOv5 network model; using another YOLOv5 network model to identify pupils and Purkinje spots of each eye area; for each eye area, selecting a midpoint of a connecting line of two Purkinje spots as a reference point, constructing a feature vector of the pupil center and the reference point, and performing normalization; based on calibration points uniformly distributed on a screen, establishing a mapping relation between the feature vectors and the calibration points for each eye region, and forming a sight line mapping model of each eye region; and calibrating the sight mapping model of each eye area through calibration points on the screen, and tracking the sight after calibration. According to the invention, full-scene, efficient and accurate pupil and Purkinje spot position detection is realized, and the precision of a sight tracking result is improved.

Description

Technical Field

[0001] The present application relates to the field of gaze tracking technology, and in particular to a gaze tracking method in a free head movement state based on YOLOv5. Background Art

[0002] Gaze tracking is mainly based on digital video analysis technology. It uses a camera to capture facial images and analyze the gaze direction of the eyes based on the image information of the eyes. At present, gaze tracking technology is mainly divided into two categories: feature-based gaze tracking and appearance-based gaze tracking. Compared with the latter, feature-based gaze tracking technology has great advantages in improving gaze accuracy and handling the problem of free head movement, so it has received more widespread attention. The most prominent method is the pupil-corneal reflection method. This method enhances eye features by adding an infrared light source to the system, effectively reducing the difficulty of eye feature extraction. This method also has many other advantages such as simple system configuration and non-invasiveness.

[0003] The basic principle of the gaze tracking technology based on pupil-corneal reflection is: using the Purkinje spot formed by the corneal reflection of the infrared light source as the reference point of the eye movement, the change of the sight direction will cause the vector pointing from the pupil center to the Purkinje spot center to change accordingly, and the corresponding relationship between the pupil center-Purkinje spot center vector and the sight line is constructed through a two-dimensional mapping model, so as to realize the tracking of eye movement according to the change of the pupil center-Purkinje spot center vector. The gaze tracking technology based on pupil-corneal reflection is usually divided into two parts: pupil and Purkinje spot positioning and mapping function construction. In practical application, both have many limitations. For example, the invention patent CN116612523A-a 2D gaze point estimation method based on pupil-corneal reflection vector method uses a traditional algorithm to extract eye features to realize the positioning of pupil and Purkinje spot. This method is inefficient and limited by the change of lighting conditions. This method may also have the problem of insufficient positioning accuracy. Furthermore, this method does not consider the influence of head movement in the process of constructing the mapping function, which leads to its limitations in practical application.

[0004] In recent years, thanks to the development of deep learning, emerging target detection algorithms represented by the YOLO (You Only Look Once) algorithm have been widely used. Among them, the YOLOv5 algorithm has many advantages such as lightweight model, efficient feature extraction, high detection accuracy and easy expansion. Under the premise of effective training, it can meet the needs of real-time, efficient and accurate detection of pupils and Purkinje spots under various lighting conditions. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a gaze tracking method in a free head movement state based on YOLOv5, which can alleviate the problem of decreased gaze tracking accuracy caused by free head movement to a certain extent while ensuring high efficiency and robustness.

[0006] The present application embodiment provides a method for tracking eye contact in a head free motion state based on YOLOv5, the method comprising:

[0007] Use the first YOLOv5 network model to identify two eye regions from the face image;

[0008] Using a second YOLOv5 network model, identifying the pupil and two Purkinje spots in each of the eye regions, and outputting the pupil center coordinates and the Purkinje spot center coordinates;

[0009] For each eye region, the midpoint of the line connecting two Purkinje spots is selected as a reference point, a feature vector of the pupil center and the reference point is constructed, and the feature vector is normalized;

[0010] Based on a plurality of calibration points evenly distributed on the screen, respectively establish a mapping relationship between the feature vector and the calibration point for each eye region, and form a sightline mapping model for each eye region based on the mapping relationship;

[0011] Calibrate the sight mapping model of each eye area respectively through a plurality of calibration points on the screen;

[0012] The calibrated gaze mapping model is used to perform gaze tracking.

[0013] According to a specific implementation of an embodiment of the present application, the method of using the first YOLOv5 network model to identify two eye areas from a face image includes:

[0014] Collecting facial image samples of different individuals looking at different positions of the screen under different lighting conditions, and marking the eye regions in the facial image samples to form a facial data set;

[0015] Using the face data set to train a first YOLOv5s lightweight network model to obtain the first YOLOv5 network model;

[0016] The first YOLOv5 network model is used to identify two eye areas from the face image.

[0017] According to a specific implementation of an embodiment of the present application, the identifying the pupil and two Purkinje spots in each of the eye regions using the second YOLOv5 network model includes:

[0018] The eye regions in the face dataset are cropped to the same size, and the pupils and Purkinje spots in the cropped images are labeled to form a human eye dataset;

[0019] Using the labeled human eye data set to train a second YOLOv5s lightweight network model to obtain the second YOLOv5 network model;

[0020] The second YOLOv5 network model is used to identify the pupil and two Purkinje spots in each of the eye regions.

[0021] According to a specific implementation of an embodiment of the present application, the labeling is implemented by using the Labelme tool.

[0022] According to a specific implementation of the embodiment of the present application, constructing a feature vector of the pupil center and the reference point and normalizing the feature vector includes:

[0023] A feature vector of the pupil center and the reference point is constructed based on each coordinate, and the feature vector is normalized using the Euclidean distance between two Purkinje spots.

[0024] According to a specific implementation method of the embodiment of the present application, the coordinates are set as follows: the pupil center coordinates of the left eye are The center coordinates of the two Purkinje spots of the left eye are and The pupil center coordinates of the right eye are The center coordinates of the two Purkinje spots of the left eye are and

[0025] The normalized feature vectors of the left and right eye regions are:

[0026]

[0027]

[0028] is the normalized left eye feature vector, is the normalized right eye feature vector, L left is the Euclidean distance between the two Purkinje spots of the left eye, L right is the Euclidean distance between the two Purkinje spots of the right eye.

[0029] According to a specific implementation of the embodiment of the present application, the mapping relationship between the feature vector and the calibration point is established for each eye region based on a plurality of calibration points evenly distributed on the screen, including:

[0030] The user gazes at each of the calibration points for a first preset time period;

[0031] For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the first preset time period;

[0032] For each frame of face image at each calibration point, construct the feature vector of each eye area;

[0033] For each eye region, the feature vectors corresponding to all frames of human eye images of each calibration point are averaged to obtain the feature vector of each calibration point;

[0034] A mapping relationship between the feature vector and the calibration point is established for each eye region respectively.

[0035] According to a specific implementation of the embodiment of the present application, the expression form of the sight line mapping model is:

[0036]

[0037] Among them, s x 、s y are the horizontal and vertical coordinates of the point of sight on the screen, v x 、v y are the horizontal and vertical coordinates of the eigenvector respectively, and a0~a7 and b0~b7 are the coefficients of the line of sight mapping model.

[0038] According to a specific implementation manner of an embodiment of the present application, the coefficients of the line of sight mapping model are solved by a least squares method.

[0039] According to a specific implementation of the embodiment of the present application, calibrating the sight line mapping model of each eye area respectively through multiple calibration points on the screen includes:

[0040] The user gazes at each of the calibration points for a second preset time period;

[0041] For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the second preset time period;

[0042] Based on the sight mapping model, for each frame of the face image, the screen gaze point corresponding to each eye area when the user gazes at each calibration point is obtained respectively;

[0043] For each eye region, the screen gaze points corresponding to all frames of human eye images of each calibration point are averaged to obtain the screen gaze point estimation result corresponding to each calibration point;

[0044] For each eye region, based on the coordinates of each calibration point and the coordinates of the screen gaze point estimation result corresponding to the calibration point, obtain an error vector corresponding to each calibration point;

[0045] For each eye region, the sight line mapping model is calibrated using the error vector corresponding to each calibration point.

[0046] Beneficial effects:

[0047] The gaze tracking method based on YOLOv5 in the embodiment of the present application in a state of free head movement adopts YOLOv5 as the target detection algorithm, and can achieve full-scene, efficient and accurate pupil and Purkinje spot position detection under the premise of sufficient training; when constructing the mapping model, the distance between the two Purkinje spots is used to normalize the feature vector, which reduces the sensitivity of the mapping model to head movement to a certain extent; the gaze tracking process is improved through the calibration process and the averaging of the binocular mapping results, thereby further improving the accuracy of the gaze tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is an overall flow chart of a method for tracking gaze in a head free motion state based on YOLOv5 according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a configuration of an eye tracking system according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of eye region detection according to an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of pupil and Purkinje spot detection according to an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of setting the calibration points and calibration points according to an embodiment of the present invention;

[0054] Figure 6 FIG. 4 is a schematic diagram of a mapping equation calibration process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0056] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0057] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0058] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0059] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0060] The present application embodiment provides a method for tracking the line of sight in a free-movement state of the head based on YOLOv5. Figures 1 to 6 Describe in detail.

[0061] In one embodiment, the gaze tracking method in a head free motion state based on YOLOv5 includes the following steps:

[0062] Step 1: Use the first YOLOv5 network model to identify two eye areas from the face image;

[0063] Step 2: using a second YOLOv5 network model to identify the pupil and two Purkinje spots in each of the eye regions, and outputting the pupil center coordinates and the Purkinje spot center coordinates;

[0064] Step 3: For each eye region, select the midpoint of the line connecting two Purkinje spots as a reference point, construct a feature vector of the pupil center and the reference point, and normalize the feature vector;

[0065] Step 4: Based on a plurality of calibration points evenly distributed on the screen, a mapping relationship between the feature vector and the calibration point is established for each eye region, and a sightline mapping model for each eye region is formed based on the mapping relationship;

[0066] Step 5: calibrate the sight line mapping model of each eye area respectively through multiple calibration points on the screen;

[0067] Step 6: Use the calibrated gaze mapping model to perform gaze tracking.

[0068] In this embodiment, YOLOv5 is used as the target detection algorithm, which can achieve full-scene, efficient and accurate pupil and Purkinje spot position detection under the premise of sufficient training; when constructing the mapping model, the feature vector is normalized, which reduces the sensitivity of the mapping model to head movement to a certain extent; the gaze tracking process is improved through the calibration process and the averaging of the binocular mapping results, thereby further improving the accuracy of the gaze tracking results.

[0069] Furthermore, the method of using the first YOLOv5 network model to identify two eye regions from the face image includes:

[0070] Collecting facial image samples of different individuals looking at different positions of the screen under different lighting conditions, and marking the eye regions in the facial image samples to form a facial data set;

[0071] Using the face data set to train a first YOLOv5s lightweight network model to obtain the first YOLOv5 network model;

[0072] The first YOLOv5 network model is used to identify two eye areas from the face image.

[0073] Further, the identifying the pupil and two Purkinje spots in each of the eye regions using the second YOLOv5 network model includes:

[0074] The eye regions in the face dataset are cropped to the same size, and the pupils and Purkinje spots in the cropped images are labeled to form a human eye dataset;

[0075] Using the labeled human eye data set to train a second YOLOv5s lightweight network model to obtain the second YOLOv5 network model;

[0076] The second YOLOv5 network model is used to identify the pupil and two Purkinje spots in each of the eye regions.

[0077] In this embodiment, data travel data sets of different environments, different individuals, and different locations are used to meet the application requirements of different devices and improve the recognition accuracy of the trained first YOLOv5 network model and the second YOLOv5 network model.

[0078] In one embodiment, the labeling is implemented by using the Labelme tool.

[0079] In one embodiment, constructing a feature vector of the pupil center and the reference point and normalizing the feature vector comprises:

[0080] A feature vector of the pupil center and the reference point is constructed based on each coordinate, and the feature vector is normalized using the Euclidean distance between two Purkinje spots.

[0081] According to a specific implementation method of the embodiment of the present application, the coordinates are set as follows: the pupil center coordinates of the left eye are The center coordinates of the two Purkinje spots of the left eye are and The pupil center coordinates of the right eye are The center coordinates of the two Purkinje spots of the left eye are and

[0082] The normalized feature vectors of the left and right eye regions are:

[0083]

[0084]

[0085] in,

[0086] is the normalized left eye feature vector, is the normalized right eye feature vector, L left is the Euclidean distance between the two Purkinje spots of the left eye, L right is the Euclidean distance between the two Purkinje spots of the right eye.

[0087] In this embodiment, the Euclidean distance between the double Purkinje spots is used to normalize the feature vector, which reduces the sensitivity of the mapping model to head movement to a certain extent.

[0088] In one embodiment, the step of establishing a mapping relationship between the feature vector and the calibration point for each eye region based on a plurality of calibration points evenly distributed on the screen includes:

[0089] The user gazes at each of the calibration points for a first preset time period;

[0090] For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the first preset time period;

[0091] For each frame of face image at each calibration point, construct the feature vector of each eye area;

[0092] For each eye region, the feature vectors corresponding to all frames of human eye images of each calibration point are averaged to obtain the feature vector of each calibration point;

[0093] A mapping relationship between the feature vector and the calibration point is established for each eye region respectively.

[0094] According to a specific implementation of the embodiment of the present application, the expression form of the sight line mapping model is:

[0095]

[0096] Among them, s x 、s y are the horizontal and vertical coordinates of the point of sight on the screen, v x 、v y are the horizontal and vertical coordinates of the eigenvector respectively, and a0~a7 and b0~b7 are the coefficients of the line of sight mapping model.

[0097] Specifically, the coefficients of the sight line mapping model are solved by the least squares method.

[0098] In the specific implementation, considering the differences in eye structure of each user, the conversion relationship between eye features and sight points is determined through the user calibration process. For the selection of calibration points, 4*4 calibration points can be evenly selected on the screen, referring to Figure 5As shown, there are 16 calibration points. The selection of calibration points here is only exemplary and has no limitations. The user gazes at each calibration point. For example, starting from the user's gaze point, 30 frames of images are collected. These 30 frames of images are all facial images of this user. A total of 10 frames of images from the 11th to the 20th frame are used as the basic data for subsequent construction of feature vectors. The main reason for selecting data in the middle time period is that the user's gaze in the middle time period is relatively stable, which can improve the validity of the data.

[0099] Furthermore, the sight line mapping model of each eye area is calibrated respectively through a plurality of calibration points on the screen, including:

[0100] The user gazes at each of the calibration points for a second preset time period;

[0101] For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the second preset time period;

[0102] Based on the sight mapping model, for each frame of the face image, the screen gaze point corresponding to each eye area when the user gazes at each calibration point is obtained respectively;

[0103] For each eye region, the screen gaze points corresponding to all frames of human eye images of each calibration point are averaged to obtain the screen gaze point estimation result corresponding to each calibration point;

[0104] For each eye region, based on the coordinates of each calibration point and the coordinates of the screen gaze point estimation result corresponding to the calibration point, obtain an error vector corresponding to each calibration point;

[0105] For each eye region, the sight line mapping model is calibrated using the error vector corresponding to each calibration point.

[0106] In specific implementation, for the selection of calibration points, the calibration points can be selected from the calibration points, for example Figure 5 As shown in the figure, the calibration points are the four corner points of the calibration points. The selection of calibration points can also be selected according to the actual situation. The user looks at the four calibration points at the four corners of the screen in turn, such as Figure 6 Each calibration point collects 30 frames of face images, and the above steps are used to calculate the screen gaze points corresponding to the left and right eye regions for 10 frames of images from the 11th frame to the 20th frame, and the average of the 10 calculation results for each eye region is used as the final screen gaze point estimation result.

[0107] Taking a single eye region as an example, the screen gaze point calculated according to its mapping model is s i , i=1,2,3,4, 4 calibration points are p i, i=1,2,3,4, the error vector is e i =(p i -s i ), i = 1, 2, 3, 4. For the eye region, four error vectors are used to calibrate the mapping model.

[0108] The calibration process in this embodiment is to estimate the gaze point according to the mapping equation in step 4, and use the mapping equation estimation result and the coordinates of the actual calibration point as the error vector for the subsequent correction of the line of sight estimation.

[0109] In one embodiment, the overall flow chart of the gaze tracking method in the free head movement state based on YOLOv5 is as follows: Figure 1 As shown, the specific steps include:

[0110] Step 1: In Figure 2 In the system environment shown in the figure, face images of different individuals looking at different positions of the screen under different lighting conditions are collected (covering the entire screen area as much as possible), and Labelme is used to mark the eyes in the image to form a face data set. Then, a YOLOv5s lightweight network model is trained using the labeled face data set to meet the application requirements of different devices. The trained model is referred to as Model1. Using the trained YOLOv5 network model Model1, the eye area is detected from the face image, as shown in Figure 3 shown.

[0111] Step 2: Crop the eye regions in the face dataset to the same size, and use Labelme to annotate the pupils and Purkinje spots in the cropped images to form a human eye dataset. Then use the annotated human eye dataset to train another YOLOv5s lightweight network model, and the trained model is called Model2. For a face image to be detected, first use Model1 to detect the two eye regions and crop them to the same size as the image in the human eye dataset. Then use Model2 to detect the pupil and two Purkinje spots from the two eye regions, respectively, as shown in Figure 2. Figure 4 As shown, the coordinates of the pupil center of each eye area and the coordinates of the two Purkinje spots are output. The coordinates of the pupil and the two Purkinje spots of the left eye are and The coordinates of the pupil and two Purkinje spots of the right eye are and

[0112] Step 3: For each eye region, select the midpoint of the line connecting the two Purchin spots as the reference point, construct the feature vector of the pupil center-Purchin spot line center, and use the Euclidean distance between the two Purchin spots to normalize the feature vector. The feature vectors of the left and right eye regions are:

[0113]

[0114] in,

[0115] Step 4: Considering the differences in eye structure of each user, the conversion relationship between eye features and sight points is determined through user calibration. The user looks at 16 calibration points evenly distributed on the screen in turn, such as Figure 5 As shown. Each calibration point collects 30 frames of face images, and for the 10 frames from the 11th to the 20th frame, the above steps are used to construct the feature vectors of the two eye regions respectively, and the 10 feature vectors of each eye region are averaged as the feature vector of the calibration point. Then, the sightline mapping model is constructed for the left and right eye regions respectively, and the mapping relationship between the feature vector and the screen gaze point is established. The basic form of the mapping model is:

[0116]

[0117] Among them, s x 、s y are the horizontal and vertical coordinates of the point of sight on the screen, v x 、v y are the horizontal and vertical coordinates of the feature vector, respectively. a0~a7 and b0~b7 are the coefficients of the sightline mapping model. The left and right eye regions correspond to a mapping model respectively, and the 14 coefficients in the model are different. Through the user calibration process, 16 equations are formed for each eye region, and the least squares method is used to solve the 14 unknown coefficients.

[0118] Step 5: The user looks at the four calibration points in the four corners of the screen in turn, such as Figure 6 As shown. Each calibration point collects 30 frames of face images, and the above step 4 is used to calculate the screen gaze points corresponding to the left and right eye regions for 10 frames from the 11th to the 20th frame, and the average of the 10 calculation results for each eye region is used as the final screen gaze point estimation result. Taking a single eye region as an example, the screen gaze point calculated according to its mapping model is s i , i=1,2,3,4, 4 calibration points are p i , i=1,2,3,4, the error vector is e i =(p i -s i ), i = 1, 2, 3, 4. For the eye region, four error vectors are used to calibrate the mapping model, such as Figure 6 When a new face image is collected, the screen gaze point calculated according to the eye area mapping model is recorded as Scomputed , Figure 6 Medium i =‖S computed -s i ‖,i=1,2,3,4, record the calibration weight Then the calibrated screen gaze point s calibrated =s computed +(∑ i＝1，2，3，4 w i ·e i ). When d i = 0, w i =1, and the other weights are 0.

[0119] After completing the above five steps, eye tracking can be achieved. Specifically, the infrared camera captures the face image in real time, maps the feature vectors of the left and right eye areas in each frame of the image to the screen gaze point respectively, and averages the mapping results of the left and right eye areas in each frame of the image as the final mapping result.

[0120] The embodiment provided by the present invention adopts YOLOv5 as the target detection algorithm, which can realize full-scene, efficient and accurate pupil and Purkinje spot position detection under the premise of sufficient training; when constructing the mapping model, the distance between the two Purkinje spots is used to normalize the feature vector, which reduces the sensitivity of the mapping model to head movement to a certain extent; the gaze tracking process is improved through the calibration process and the averaging of the binocular mapping results, thereby further improving the accuracy of the gaze tracking result.

[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A gaze tracking method in a head free motion state based on YOLOv5, characterized in that: The method comprises: Use the first YOLOv5 network model to identify two eye regions from the face image; Using a second YOLOv5 network model, identifying the pupil and two Purkinje spots in each of the eye regions, and outputting the pupil center coordinates and the Purkinje spot center coordinates; For each eye region, the midpoint of the line connecting two Purkinje spots is selected as a reference point, a feature vector of the pupil center and the reference point is constructed, and the feature vector is normalized; Based on a plurality of calibration points evenly distributed on the screen, respectively establish a mapping relationship between the feature vector and the calibration point for each eye region, and form a sightline mapping model for each eye region based on the mapping relationship; Calibrate the sight mapping model of each eye area respectively through a plurality of calibration points on the screen; The calibrated gaze mapping model is used to perform gaze tracking.

2. The method for tracking the sight line in a head free motion state based on YOLOv5 according to claim 1, characterized in that: The method of using the first YOLOv5 network model to identify two eye regions from the face image includes: Collecting facial image samples of different individuals looking at different positions of the screen under different lighting conditions, and marking the eye regions in the facial image samples to form a facial data set; Using the face data set to train a first YOLOv5s lightweight network model to obtain the first YOLOv5 network model; The first YOLOv5 network model is used to identify two eye areas from the face image.

3. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 2, characterized in that: The method of using the second YOLOv5 network model to identify the pupil and two Purkinje spots in each eye region includes: The eye regions in the face dataset are cropped to the same size, and the pupils and Purkinje spots in the cropped images are labeled to form a human eye dataset; Using the labeled human eye data set to train a second YOLOv5s lightweight network model to obtain the second YOLOv5 network model; The second YOLOv5 network model is used to identify the pupil and two Purkinje spots in each of the eye regions.

4. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 3, characterized in that: The labeling is achieved through the Labelme tool.

5. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 1, characterized in that: The step of constructing a feature vector of the pupil center and the reference point and normalizing the feature vector comprises: A feature vector of the pupil center and the reference point is constructed based on each coordinate, and the feature vector is normalized using the Euclidean distance between two Purkinje spots.

6. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 5, characterized in that: The coordinates are set as follows: the pupil center coordinates of the left eye are The center coordinates of the two Purkinje spots of the left eye are and The pupil center coordinates of the right eye are The center coordinates of the two Purkinje spots of the left eye are and The normalized feature vectors of the left and right eye regions are: in, is the normalized left eye feature vector, is the normalized right eye feature vector, L left is the Euclidean distance between the two Purkinje spots of the left eye, L right is the Euclidean distance between the two Purkinje spots of the right eye.

7. The method for tracking the sight line in a head free motion state based on YOLOv5 according to claim 1, characterized in that: The step of establishing a mapping relationship between the feature vector and the calibration point for each eye region based on a plurality of calibration points evenly distributed on the screen includes: The user gazes at each of the calibration points for a first preset time period; For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the first preset time period; For each frame of face image at each calibration point, construct the feature vector of each eye area; For each eye region, the feature vectors corresponding to all frames of human eye images of each calibration point are averaged to obtain the feature vector of each calibration point; A mapping relationship between the feature vector and the calibration point is established for each eye region respectively.

8. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 1, characterized in that: The expression form of the sightline mapping model is: Among them, s x 、s y are the horizontal and vertical coordinates of the point of sight on the screen, v x 、v y are the horizontal and vertical coordinates of the eigenvector respectively, and a0~a7 and b0~b7 are the coefficients of the line of sight mapping model.

9. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 8, characterized in that: The coefficients of the sight line mapping model are solved by the least squares method.

10. The method for tracking the sight line in a free head motion state based on YOLOv5 according to claim 1, characterized in that: The step of calibrating the sight line mapping model of each eye area respectively through a plurality of calibration points on the screen includes: The user gazes at each of the calibration points for a second preset time period; For each calibration point, selecting a number of frames of face images corresponding to the middle time period within the second preset time period; Based on the sight mapping model, for each frame of the face image, the screen gaze point corresponding to each eye area when the user gazes at each calibration point is obtained respectively; For each eye region, the screen gaze points corresponding to all frames of human eye images of each calibration point are averaged to obtain the screen gaze point estimation result corresponding to each calibration point; For each eye region, based on the coordinates of each calibration point and the coordinates of the screen gaze point estimation result corresponding to the calibration point, obtain an error vector corresponding to each calibration point; For each eye region, the sight line mapping model is calibrated using the error vector corresponding to each calibration point.

Images