A real-time gaze inference method integrating head pose and eye tracking

Abstract

The present invention relates to a real-time sight line inference method integrating head posture and eye tracking, which sequentially determines the face area, head posture, eye tracking point, and face reference point, obtains the eye direction according to the distance ratio and angle change between the eye and the face reference point, and makes corrections in combination with the change of head posture, so as to infer the sight line direction under the current posture in real time. The method of the present invention takes into account the influence of the movement of the human head and the change of posture on the inferred sight line direction, and the inference method has high real-time performance and is easy to control errors. The embodiment is implemented by using a machine learning algorithm, and an ordinary visible light camera is used to obtain images. The required hardware requirements are low and the versatility is good, and the person can be placed in a natural state during the inference process.

Description

Technical Field

[0001] The invention relates to a real-time sight line inference method integrating head posture and eye direction tracking, and belongs to the fields of computer vision and machine learning. Background Art

[0002] During interrogation, it is very important to grasp the psychological changes of the interrogated person. During the conversation, the changes in a person's line of sight reflect part of the psychological activities. Recording the changes in the line of sight of the interrogated person and storing them as quantitative data will provide the possibility of finding the flaws of the interrogated person. However, it is too complicated to record these data manually, and how to use machines to naturally monitor the interrogated person becomes more critical. During the interrogation, the direction of eye movement is determined by detecting the face area, and then the line of sight changes of the interrogated person are inferred. In traditional line of sight inference, infrared cameras or wearable cameras are mostly required, and the direction of eye movement needs to be detected at close range. However, the hardware requirements required for the above are high, which can easily put people in an unnatural state. In addition, the traditional method of only using pupil coordinates to determine is easily disturbed by changes in the posture of the human head, and it is difficult to apply to a wide range of interrogation processes. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a real-time line of sight inference method that integrates head posture and eye tracking.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A real-time line of sight inference method integrating head posture and eye tracking, characterized in that it comprises the following steps:

[0006] Step 1: Detect the face area and eye area of ​​the image, and determine the positions of the left eye and right eye areas in the face area;

[0007] Step 2: Determine the coordinates of the eyeball center and the reference point, and perform optical flow tracking on the feature coordinates;

[0008] Step 3: Detect and record the current head posture features and head movement trajectory;

[0009] Step 4: Infer the real-time line of sight direction based on the distance ratio and angle information between the current feature tracking points, combined with the head posture and trajectory changes.

[0010] Furthermore, the step 1 of determining the position of the eyeball at the initial moment includes the following process:

[0011] Step 1.1: Obtain image information of the human eye in the normal gaze state at the initial moment;

[0012] Step 1.2: Use the face cascade detection algorithm to determine the face ROI area;

[0013] Step 1.3: Divide the left eye ROI area and the right eye ROI area by biometric features;

[0014] Step 1.4: Use the eye cascade detection algorithm in the left eye ROI area and the right eye ROI area to determine the actual positions of the left and right eyes.

[0015] Furthermore, the process of determining the face reference point in step 2 and tracking the left and right eyeballs and the face reference point is as follows:

[0016] Step 2.1: According to the positions of the left and right eyes, the centers of the left and right eyes are taken as eye tracking points respectively, and real-time tracking is performed using optical flow;

[0017] Step 2.2: Take the center of the two eye tracking points in the same plane in the face area as the center point C, and use the optical flow tracking algorithm to track.

[0018] Furthermore, the step 3 of detecting the current head posture feature and movement trajectory includes the following process:

[0019] Step 3.1: Segment the skin color in the image through the HSV H component feature;

[0020] Step 3.2: Perform face cascade detection on the segmented area, crop the head area, and perform ellipse fitting on the face skin color area;

[0021] Step 3.3: Perform PCA principal component analysis on the fitted ellipse, calculate the major axis A1 and the minor axis A2 of the fitted ellipse, obtain the ratio λ=A1 / A2, and construct the face coordinate system with the center of the fitted ellipse as the origin of the coordinate system, the major axis of the ellipse as the y-axis, and the minor axis as the x-axis;

[0022] Step 3.4: In the face area, perform optical flow tracking on the origin of the coordinate system in the initial state, set it as the tracking point o, and set the error range of radius r = A1 / 10;

[0023] Step 3.5: Infer the head posture and trajectory based on the rotation angle of the fitted ellipse, the displacement of the tracking point, and the ratio of the major and minor axes;

[0024] The specific process of step 3.5 inferring the head posture and trajectory according to the rotation angle of the fitted ellipse, the displacement of the tracking point, and the ratio of the major and minor axes includes:

[0025] Step 3.5.1: Use PCA principal component analysis to calculate the rotation angle ∠1 for the fitted ellipse, infer the current head tilt state, and set the range angle to a positive value θ. When the rotation angle ∠1 is less than -θ, the posture is left-tilted. When the angle ∠1 is greater than θ, the posture is right-tilted. If ∠1 is between -θ and θ, it is a normal upright state.

[0026] Step 3.5.2: Re-divide the face coordinate system into four regions, and infer the change in the head turning posture based on the change in the major-minor axis ratio λ and the change in the coordinates of the tracking point o in the face coordinate system:

[0027] When λ increases and the tracking point o is located in the first area, the posture is head up; when λ increases and the tracking point o is located in the third area, the posture is head down; when λ decreases and the tracking point o is located in the second area, the posture is left head turn; when λ decreases and the tracking point o is located in the fourth area, the posture is right head turn;

[0028] Step 3.5.3: Determine the center coordinates of the head fitting ellipse, and draw and judge the head movement trajectory in real time based on the relative displacement of the previous and next frames.

[0029] Step 3.5.4: Generate the head posture according to the combination of head tilt posture, head turn posture and movement direction.

[0030] Furthermore, the step 4 infers the real-time sight line direction according to the distance ratio and angle information between the current feature tracking points, combined with the head posture and trajectory changes, and includes the following process:

[0031] Step 4.1: The distance ratio and angle information between the current feature tracking points mentioned above, combined with Figure 6 , that is, the distance ratio is the line L between the center of the left eyeball and the center point C in the current frame i Line R connecting the center point of the right eyeball with the center point C i The ratio of γ i , angle information is the line S between the eyes in the current frame i The line L connecting the center of the left eyeball and the center point C i The generated angle α i , i=1, 2, 3... corresponds to the current i-th frame image information;

[0032] Set the allowable error value of the distance ratio change to n, the allowable error value of the angle change to m, and set γ to the line L connecting the center point of the left eyeball and the center point C in the initial state. i Connect the line R with the center point of the right eyeball and the center point C i The ratio of α is set to the initial state of the eye connection line S i The line L connecting the center point of the left eye and the center point C i The generated angle;

[0033] The tracking points are placed on the same plane, and the direction of eye movement is inferred based on the angle information and distance ratio between the tracking points of both eyes, including nine directions: center, up, down, left, right, upper left, lower left, upper right, and lower right;

[0034] By recording the distance ratio γ of each frame during tracking i and the angle αi , substitute into formula (1) and (2) to obtain the current value result. When result is equal to 0, 1, 2, 3, 4, 5, 6, 7, 8, the corresponding inferred eye directions are center, right, left, top, bottom, bottom left, top left, bottom right, top right, respectively.

[0035] γ i =L i / R i (1)

[0036]

[0037] When in the "straight-on" state, the angle α ranges from -m to m degrees, and the distance ratio γ i The range is from γ-n to γ+n; when in the "look left" and "look right" states, the range of the angle α remains unchanged, from -m to m degrees, and the distance ratio γ i changes, when "looking left", γ i is less than γ–n, when “looking right”, γ i Greater than γ+n; when in the "looking up" and "looking down" states, the range of the angle α changes. When "looking up", it is greater than m, and the distance ratio is γ i From γ-n to γ+n, when “looking down” it is less than -m, and the distance ratio is γ i γ-n to γ+n; when "looking at the upper left", "looking at the lower left", "looking at the upper right", and "looking at the lower right", the angle range of angle α and the distance ratio γ i The range of has changed compared with the initial moment. When looking from the upper left, the angle α is greater than m, and the distance ratio γ i is greater than γ+n, and when “looking from the lower left”, the angle α range is less than -m, and the distance ratio γ i is greater than γ+n, when “looking from the upper right”, the angle α range is greater than m, and the distance ratio γ i When the angle α is less than γ-n and the distance ratio is γ, when looking from the lower right, the angle α is less than -m. i is smaller than γ-n.

[0038] Step 4.2: Infer the current actual line of sight direction by combining the head posture direction with the eye movement direction:

[0039] If the sight direction is forward, it may be 1) the head posture is turned left, the eye movement is looking right, 2) the head posture is turned right, the eye direction is looking left, 3) the head posture is raised, the eye movement is looking down, 4) the head posture is lowered, the eye direction is looking up, etc. By analogy, the head posture and sight direction are combined, and the interaction between the head posture and the eye movement direction is used to divide the actual sight direction into 9 categories, namely forward, above, below, left, right, upper left, lower left, upper right, and lower right.

[0040] Beneficial effects of the present invention: The method of the present invention can be implemented using an ordinary monocular camera, does not require the wearing of special equipment, has low requirements on the hardware resources, and is convenient for more extensive application. The method of the present invention does not require the use of wearable equipment, and can put the person being interrogated in a state of natural conversation. The method of the present invention adopts a method combining the distance ratio between tracking points and the angle quadrant to detect the direction of the eyeball, which can better control the error range for different scenes and has high real-time performance. The method of the present invention adopts a method combining the detection of the eyeball direction with the head posture, which can control the impact of changes in the head posture on the inferred line of sight direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a basic flow chart of the method of the present invention.

[0042] Figure 2 It is a side schematic diagram of the application scenario of the method of the present invention.

[0043] Figure 3 It is a flowchart for determining the eye position and tracking at the initial moment.

[0044] Figure 4a It is a virtual schematic diagram of determining the facial coordinate system of the present invention.

[0045] Figure 4b It is the present invention that determines the facial coordinate system partitions.

[0046] Figure 5 It is a flow chart of inferring head posture of the present invention.

[0047] Figure 6 a is a virtual schematic diagram of the present invention in the "direct viewing" state.

[0048] Figure 6 b is a virtual schematic diagram of the present invention in the "looking up" state.

[0049] Figure 6 c is a virtual schematic diagram of the present invention in the "bottom right looking" state.

[0050] Figure 7a It is a "direct view" schematic diagram of the actual test of the present invention.

[0051] Figure 7b It is a schematic diagram of "looking up" in actual testing of the present invention.

[0052] Figure 7c It is a "left-looking" schematic diagram of the actual test of the present invention.

[0053] Figure 7d It is a schematic diagram of "looking from the upper left" in actual testing of the present invention. DETAILED DESCRIPTION

[0054] The method of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments to further understand the purpose, scheme and effect of the method of the present invention.

[0055] The method of the present invention is specifically divided into four modules from the implementation process: 1) cascade detection, 2) eye direction inference, 3) head posture inference, 4) real-time line of sight direction inference;

[0056] The method of the present invention mainly involves "direction inference", and mainly provides a method for inferring the line of sight by combining the various directions of eye movement with the change of head posture after eye tracking is realized;

[0057] like Figure 1 As shown in the figure, it is the basic process of implementing the present invention. At the initial moment, the camera needs to be faced straightly. The side view of the scene arrangement is as follows Figure 2 As shown;

[0058] 1. Detect the face and eye areas in the image, and determine the left and right eye areas in the face area:

[0059] Use an ordinary visible light camera to obtain image information, reset the image size and preprocess it, load face cascade detection into the image, and use Adaboost+Haar features to divide the face area ROI;

[0060] In the divided face area, the left eye area and the right eye area are roughly determined by using biological features, and Adaboost+Haar features are used in the left eye area and the right eye area respectively to perform cascade detection to determine the initial left and right eye positions;

[0061] At the initial moment, it is necessary to collect complete binocular information. If the face cascade detection in the current image fails or the binocular position is not fully acquired, it is necessary to reacquire the image for detection. The specific process is as follows: Figure 3 shown.

[0062] 2. Determine the coordinates of the eyeball center and the facial reference point, and perform optical flow tracking on the feature coordinates.

[0063] After obtaining the binocular detection frame, the center coordinates are inferred to infer the eye position when looking straight, and the coordinates of the left and right eyeballs are used to determine the reference point coordinates of the face. The reference point C (x, y) relative to the eye is determined in the face area, and this reference point is the center position of the binocular coordinates.

[0064]

[0065] When the coordinates of the required tracking points have been determined, the three points are placed on the same plane. In an environment with good lighting conditions, the Lucas-Kandade optical flow tracking algorithm is used to track the three points separately and return real-time coordinate information. During the tracking process, if the deviation between the tracking value and the detection value is too large, the tracking value and the detection value will compete and the correction process will be repeated again.

[0066] 3. Detect and record the current head posture changes and movement trajectory, such as Figure 5 shown.

[0067] The face area in the image is segmented by the skin color threshold, and the HSV model is used to adjust the H component to segment the skin color area in the foreground. The face cascade detection frame is combined to eliminate the influence of the neck area on the segmentation of the head area.

[0068] By determining the maximum inscribed ellipse of the minimum circumscribed matrix of the segmented area, the segmented area is fitted with an ellipse, and the edge point set of the ellipse is extracted. The edge point set is processed using PCA principal component analysis. Processing only the edge point set effectively improves the processing speed. Then, the length of the major axis and the length of the minor axis of the fitted ellipse are obtained, and the face coordinate system is constructed with the center of the fitted ellipse as the origin of the coordinate system, the major axis of the ellipse as the y-axis, and the minor axis as the x-axis, as shown in Figure 4a shown.

[0069] The rotation angle of the fitted ellipse is used to infer the current left and right head tilt posture change. The PCA principal component analysis is used to calculate the rotation angle ∠1 of the fitted ellipse to infer the current head tilt state. The range angle is set to a positive value of θ. When the rotation angle ∠1 is less than -θ, the posture is left tilted. When the angle ∠1 is greater than θ, the posture is right tilted. If ∠1 is between -θ and θ, it is a normal upright state.

[0070] Moreover, in the previous and next frames, the real-time angle change of the head will not change drastically, thereby monitoring whether the system is normal.

[0071] The center point of the initial state of the fitted ellipse is tracked using the Lucas-Kandade optical flow tracking algorithm, and the face coordinate system is divided into four areas, such as Figure 4b As shown, an allowable error range of radius r=A1 / 10 is set with the coordinate origin as the center.

[0072] By using the change in the ratio λ of the major axis A1 and the minor axis A2 of the fitted ellipse and combining it with the change in the area where the tracking point o is located in the face coordinate system, the changes in the posture of turning the head left and right, and raising and lowering the head are inferred.

[0073] When λ increases and the tracking point o is located in the first area, the posture is head up; when λ increases and the tracking point o is located in the third area, the posture is head down; when λ decreases and the tracking point o is located in the second area, the posture is left head turn; when λ decreases and the tracking point o is located in the fourth area, the posture is right head turn;

[0074] In the real-time acquisition process, the current moving direction is obtained by judging the relative position of the center of the fitted ellipse in the previous and next frames, and the trajectory curve is drawn according to the center change.

[0075] The head posture is generated based on the combination of head tilt posture, head turn posture and movement direction.

[0076] The inference of eye movement direction is mainly based on two key features: one is the change in the ratio of the distances between the two eyes relative to the central reference point, and the other is the change in the sign of the angle information between the line between the two eyes and the line between the two eyes and the reference point.

[0077] The distance ratio is the line L between the center of the left eyeball and the center point C in the current frame. i Line R connecting the center point of the right eyeball with the center point C i The ratio of γ i , the angle information is the line S between the eyes in the current frame i The line L connecting the center of the left eyeball and the center point C i The generated angle α i , i = 1, 2, 3... corresponds to the current i-th frame image information, such as Figure 6 a to Figure 6 As shown in c;

[0078] In order to control the jitter problem of tracking points in actual applications, the allowable error value of the distance ratio change is set to n, and the allowable error value of the angle change is set to m. At the initial moment, it is necessary to look straight at the camera to collect relevant parameters, and set γ to the line L connecting the center point of the left eyeball and the center point C in the initial state. i Line R connecting the center point of the right eyeball with the center point C i The ratio of α is set to the initial state of the eye connection line S i The line L connecting the center point of the left eye and the center point C i The generated angle;

[0079] The tracking points are placed on the same plane, and the direction of eye movement is inferred based on the angle information and distance ratio between the tracking points of both eyes, including nine directions: center, up, down, left, right, upper left, lower left, upper right, and lower right;

[0080] By recording the distance ratio γ of each frame during tracking i and the angle α i, substitute into formula (1) and (2) to obtain the current value result. When result is equal to 0, 1, 2, 3, 4, 5, 6, 7, 8, the corresponding inferred eye directions are center, right, left, top, bottom, bottom left, top left, bottom right, top right, respectively.

[0081] γ i =L i / R i (1)

[0082]

[0083] When in the "straight-on" state, the angle α ranges from -m to m degrees, and the distance ratio γ i The range is from γ-n to γ+n; as shown in Figure 6a;

[0084] When in the "look left" and "look right" states, the range of the angle α remains unchanged, from -m to m degrees, and the distance ratio γ i changes, when "looking left", γ i is less than γ–n, when “looking right”, γ i Greater than γ+n;

[0085] When in the "looking up" and "looking down" states, the range of the angle α changes. When "looking up", it is greater than m, and the distance ratio γ i From γ-n to γ+n, such as Figure 6 b, when “looking down”, it is less than -m, and the distance ratio is γ i γ-n to γ+n;

[0086] When "looking up left", "looking down left", "looking up right", "looking down right", the angle range of angle α and the distance ratio γ i The range of has changed compared with the initial moment. When looking from the upper left, the angle α is greater than m, and the distance ratio γ i is greater than γ+n, and when “looking from the lower left”, the angle α range is less than -m, and the distance ratio γ i is greater than γ+n, when looking from the upper right, the angle α range is greater than m, and the distance ratio γ i When the angle α is less than γ-n and the distance ratio is γ, when looking from the lower right, the angle α is less than -m. i is less than γ-n, such as Figure 6 c.

[0087] Then, the actual line of sight is inferred by combining the head posture direction with the eye movement direction:

[0088] If the sight direction is forward, it may be 1) the head posture is turned left, the eye movement is looking right, 2) the head posture is turned right, the eye direction is looking left, 3) the head posture is raised, the eye movement is looking down, 4) the head posture is lowered, the eye direction is looking up, etc. By analogy, the head posture and sight direction are combined, and the actual sight direction can be divided into 9 categories, namely forward, above, below, left, right, upper left, lower left, upper right, and lower right, to infer the real-time sight.

[0089] Through the above implementation modes, the representative examples of the methods of the present invention are as follows: Figures 7a to 7d As shown, Figure 7a This is an example of "direct viewing". Figure 7b This is an example of "looking up". Figure 7c This is an example of "looking from the left". Figure 7d This is an example of "viewing from the upper left", which basically covers the format angles.

[0090] The direction of a person's gaze under various head postures can be fed back stably and in real time in the image.

[0091] The above is a detailed introduction to the method for inferring the real-time line of sight direction of the present invention. The scheme of the above embodiment is used to help understand and verify the idea of ​​the present method; at the same time, the present invention may also have many other embodiments. According to the spirit and essence of the present invention, for technical personnel in the field, various corresponding changes and deformations can be legally made according to the present invention, and the content of the present invention should not be understood as a limitation on the present invention.

Claims

1. A real-time line of sight inference method integrating head posture and eye tracking, characterized by: The following steps are involved: Step 1: Detect the face area and eye area in the image, and determine the positions of the left eye and right eye areas in the face area; Step 2: Determine the coordinates of the eyeball center and the reference point, and perform optical flow tracking on the feature coordinates; Step 3: Detect and record the current head posture features and head movement trajectory; Step 4: Infer the real-time sight direction based on the distance ratio and angle information between the current feature tracking points, combined with the head posture and trajectory changes; The specific process of determining the eyeball center coordinates and the facial reference coordinate points and performing optical flow tracking on the feature coordinates in step 2 is as follows: Step 2.1: According to the positions of the left and right eyes, the centers of the left and right eyes are taken as eye tracking points respectively, and real-time tracking is performed using optical flow; Step 2.2: Take the center of the two eye tracking points in the same plane in the face area as the center point C, and use the optical flow tracking algorithm to track; The step 3 of detecting the current head posture features and movement trajectory includes the following processes: Step 3.1: Segment the skin color in the image through the HSV H component feature; Step 3.2: Perform face cascade detection on the segmented area, crop the head area, and perform ellipse fitting on the face skin color area; Step 3.3: Perform PCA principal component analysis on the fitted ellipse, calculate the major axis A1 and the minor axis A2 of the fitted ellipse, obtain the ratio λ=A1 / A2, and construct the face coordinate system with the center of the fitted ellipse as the origin of the coordinate system, the major axis of the ellipse as the y-axis, and the minor axis as the x-axis; Step 3.4: In the face area, perform optical flow tracking on the origin of the coordinate system in the initial state, set it as the tracking point o, and set the error range of radius r = A1 / 10; Step 3.5: Infer the head posture and trajectory based on the rotation angle of the fitted ellipse, the displacement of the tracking point, and the ratio of the major and minor axes.

2. The real-time line of sight inference method integrating head posture and eye tracking according to claim 1, characterized in that: In step 1, determining the left eye and right eye regions and positions in the face region includes the following processes: Step 1.1: Obtain image information of the human eye in the normal gaze state at the initial moment; Step 1.2: Use the face cascade detection algorithm to determine the face ROI area; Step 1.3: Divide the left eye ROI area and the right eye ROI area by biometric features; Step 1.4: Use the eye cascade detection algorithm in the left eye ROI area and the right eye ROI area to determine the actual positions of the left and right eyes.

3. The real-time line of sight inference method integrating head posture and eye tracking according to claim 1, characterized in that: The specific process of step 3.5 for inferring the head posture according to the rotation angle of the fitted ellipse, the displacement of the tracking point, and the ratio of the major and minor axes is as follows: Step 3.5.1: Calculate the rotation angle ∠1 for the fitted ellipse using PCA principal component analysis, infer the current head tilt state, set the range angle to a positive value of θ. When the rotation angle ∠1 is less than -θ, the posture is left-tilted. When the angle ∠1 is greater than θ, the posture is right-tilted. If ∠1 is between -θ and θ, it is a normal upright state. Step 3.5.2: Re-divide the face coordinate system into four regions, and infer the change in the head turning posture based on the change in the major-minor axis ratio λ and the change in the coordinates of the tracking point o in the face coordinate system: When λ increases and the tracking point o is located in the first area, the posture is head up; when λ increases and the tracking point o is located in the third area, the posture is head down; when λ decreases and the tracking point o is located in the second area, the posture is left head turn; when λ decreases and the tracking point o is located in the fourth area, the posture is right head turn; Step 3.5.3: Determine the center coordinates of the head fitting ellipse, and draw and judge the head movement trajectory in real time based on the relative displacement of the previous and next frames; Step 3.5.4: Generate the head posture according to the combination of head tilt posture, head turn posture and movement direction.

4. The real-time line of sight inference method integrating head posture and eye tracking according to claim 1, characterized in that: The step 4 infers the real-time sight direction according to the distance ratio and angle information between the current feature tracking points, combined with the head posture and trajectory changes, and includes the following process: Step 4.1: The distance ratio and angle information between the above current feature tracking points, that is, the distance ratio is the line L between the center of the left eyeball and the center point C in the current frame i Line R connecting the center point of the right eyeball with the center point C i The ratio of γ i , the angle information is the line S between the eyes in the current frame i The line L connecting the center of the left eyeball and the center point C i The generated angle α i , i=1, 2, 3... corresponds to the current i-th frame image information; Set the allowable error value of the distance ratio change to n, the allowable error value of the angle change to m, and set γ to the line L connecting the center point of the left eyeball and the center point C in the initial state. i Connect the line R with the center point of the right eyeball and the center point C i The ratio of α is set to the initial state of the eye connection line S i The line L connecting the center point of the left eye and the center point C i The generated angle; The tracking points are placed on the same plane, and the direction of eye movement is inferred based on the angle information and distance ratio between the tracking points of both eyes, including nine directions: center, up, down, left, right, upper left, lower left, upper right, and lower right; By recording the distance ratio γ of each frame during tracking i and the angle α i , Substitute into formula (1) and (2) to obtain the current value result. When result is equal to 0, 1, 2, 3, 4, 5, 6, 7, 8, the corresponding inferred eye directions are center, right, left, top, bottom, bottom left, top left, bottom right, top right; When in the "straight-on" state, the angle α ranges from -m to m degrees, and the distance ratio γ i The range is from γ-n to γ+n; when in the "look left" and "look right" states, the range of the angle α remains unchanged, from -m to m degrees, and the distance ratio γ i changes, when "looking left", γ i is less than γ–n, when “looking right”, γ i Greater than γ+n; when in the "looking up" and "looking down" states, the range of the angle α changes. When "looking up", it is greater than m, and the distance ratio is γ i From γ-n to γ+n, "looking down" is less than -m, the distance ratio is γ i γ-n to γ+n; when "looking at the upper left", "looking at the lower left", "looking at the upper right", and "looking at the lower right", the angle range of angle α and the distance ratio γ i The range of has changed compared with the initial moment. When "looking from the upper left", the angle α is greater than m, and the distance ratio γ i is greater than γ+n, when "looking from the lower left", the angle α range is less than -m, and the distance ratio is γ i is greater than γ+n, when "looking from the upper right", the angle α range is greater than m, and the distance ratio γ i is less than γ-n, when "looking down to the right", the angle α range is less than -m, and the distance ratio is γ i is less than γ-n; Step 4.2: Infer the current actual line of sight direction by combining the head posture direction with the eye movement direction: If the line of sight is forward, it may be 1) the head posture is turned left, and the eye movement is looking right, 2) the head posture is turned right, and the eye direction is looking left, 3) the head posture is raised, and the eye movement is looking down, 4) the head posture is lowered, and the eye direction is looking up; and so on, by combining the head posture and the line of sight direction, the actual line of sight direction can be divided into 9 categories, namely front, top, bottom, left, right, upper left, lower left, upper right, and lower right, to infer the real-time line of sight.

Images