Driver attention distribution detection method based on eye movement tracking

Abstract

The invention discloses a driver attention distribution detection method based on eye movement tracking, and the method comprises the steps: collecting a driver face image and a driving external environment image, obtaining eye key points, and extracting eye movement data; establishing a mathematical model for a real eyeball structure according to the physiological structure of the eyeballs, and fixing the position relation of the two eyeballs in the space; based on the three-dimensional eyeball model, modeling human eyes and a gazing target to obtain coordinates of a gazing point of the driver; a PNP algorithm is used to realize concern coordinate conversion, and the line-of-sight direction is fused to the external environment; and according to the obtained watching information, carrying out attention analysis on the driver. The method can effectively obtain the attention distribution of the driver, timely monitors the distraction driving of the driver and the like, and plays an important role in improving the driving safety.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving, and in particular relates to a method for detecting driver attention distribution based on eye tracking. Background Art

[0002] Traffic driving scenarios are dynamic and contain multiple information sources, many of which attract the driver's attention. For example, when turning at an intersection, the driver's attention to different visual fields will change when facing vehicles and pedestrians coming from the side and behind.

[0003] The father of eye movement vision believes that humans are visual animals. Although eye movement is a physical measurement, it can directly provide insight into your cognitive process. The development of eye tracking technology provides strong support for real-time detection of the driver's line of sight distribution during driving. The present invention adopts a driver attention distribution detection method based on non-invasive eye tracking, which detects and analyzes the driver's line of sight attention distribution without interference to the driver. This method is helpful in the fields of driving safety, driver training, and optimized design of road facilities. Summary of the invention

[0004] The present invention proposes a driver attention distribution detection method based on eye tracking. Combining this method with the existing intelligent driving or assisted driving system can help improve traffic safety, improve driving efficiency, promote the development of autonomous driving technology, and promote the construction of smart city transportation.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for detecting driver attention distribution based on eye tracking comprises the following steps:

[0007] Step 1: Collect the driver's facial image and the external driving environment image, obtain the eye key points, and extract the eye movement data;

[0008] Step 2: Based on the physiological structure of the eyeball, a mathematical model is established for the real eyeball structure to fix the positional relationship between the two eyeballs in space;

[0009] Step 3: Based on the 3D eyeball model, model the human eye and gaze target to obtain the coordinates of the driver's gaze point;

[0010] Step 4: Use the PNP algorithm to realize the coordinate transformation of the focus point and integrate the sight direction into the external environment;

[0011] Step 5: Analyze the driver's attention based on the obtained gaze information.

[0012] Preferably, the eye key points of the driver's facial image are obtained, and the driver's facial grid is extracted using the Mediapipe algorithm, thereby determining the location of the driver's eyes and extracting eye movement data including iris features. The Mediapipe algorithm is a powerful computer vision tool that can efficiently process image data and accurately identify various facial features. Based on a machine learning visual algorithm, a fine facial grid is extracted from the driver's facial image, with a total of 468 points, including key facial feature points and key points that can be identified, thereby performing a comprehensive analysis of the face. In addition, the algorithm can also detect the key points of the eye including the iris, and using the detected iris features to extract eye movement data can ensure a certain degree of accuracy and real-time performance.

[0013] Preferably, a mathematical model is established for the real eyeball structure, and the model consists of two spheres, namely the pupil sphere and the eyeball sphere. The light path generated by the line of sight can be regarded as the line connecting the two sphere centers. By calculating its relative position and size in the three-dimensional Cartesian coordinate system, the optical axis vector of the eyeball is determined. When projected onto a two-dimensional image, the three-dimensional optical axis vector will pass through the center of the eyeball and the center of the pupil in the two-dimensional image, and the direction of the vector is the same as the direction of the short axis of the pupil fitting ellipse. According to the rotational invariance of the spherical shape, the above two features are always valid when the pupil moves, and the coordinates and radius of the eyeball remain unchanged.

[0014] By calculating the coordinates of the center point (x e ,y e ) and radius R, let the center of the eyeball be located on the XOY plane of the three-dimensional coordinate system, and determine the three-dimensional coordinate equation of the eyeball (xx e ) 2 +(yy e ) 2 +z 2 =R 2 . Define the three-dimensional coordinate equation of the pupil sphere (xi) 2 +(yj) 2 +(zk) 2 =r 2 , combined eyeball and pupil sphere equations:

[0015]

[0016] Solve the equations to get the plane where the two spheres intersect, substitute it into any equation, and get the projection equation of the intersection line on the XOY plane. And the projection of the center of the pupil sphere on the XOY plane falls on the minor axis l: y = kx + b of the ellipse fitting equation. Two constraint equations can be obtained:

[0017] j=ki+b

[0018] Ax2 +Bxy+Cy 2 +D x +Ey+F=0

[0019] By solving the above equation, we can find the center of the pupil sphere (x p ,y p , z p ), and the center of the eyeball solved in the previous article, the optical axis of the double-ball model is calculated, and the modeling of the double-ball model of the eyeball is completed.

[0020]

[0021] Preferably, the human eye and the fixation target are modeled, and based on the positional relationship of the two eyeballs in a fixed space, the visual axis vector and the three-dimensional coordinates of its intersection in space are calculated, thereby obtaining the coordinates of the driver's fixation point. In physiology, the angle between the optical axis and the visual axis is called kappa (k). The kappa angle of an adult is fixed, and the horizontal and vertical rotation angles are about 5° and 1.5° respectively. The visual axis is used to measure the current fixation point direction of the subject, that is, the fixation point is defined as the intersection of the binocular visual axes, rather than the intersection of the optical axes, so the positional relationship between the optical axis and the visual axis is further modeled.

[0022] Preferably, the line of sight direction is integrated into the external environment, and the conversion relationship between the two coordinate systems of the driver's facial image and the external environment image and the world coordinate system is obtained through the PNP algorithm, and the pairwise conversion relationship between the three coordinate systems is obtained, thereby integrating the driver's line of sight direction into the external environment image. Since the positional relationship between the two cameras that collect the image is fixed, the rotation R and translation T vectors are known, and the three-dimensional coordinates of the line of sight vector in its camera coordinate system can be converted to the external environment perception camera coordinate system through a certain rotation and translation relationship, so as to realize the fusion of the line of sight vector and the external environment in the same coordinate system, and convert the three-dimensional point represented in the world frame (another camera, understood as w here) into the camera frame:

[0023]

[0024]

[0025] Point X expressed in 3D world coordinates w is projected into the image plane [u, v] using the perspective projection model Π and the camera intrinsic parameter matrix A:

[0026]

[0027]

[0028] Equations of the above form can be solved using some algebraic derivation by using a method called Direct Linear Transformation (DLT).

[0029] The "gold standard" solution for PNP is to estimate the six parameters of the transformation by minimizing the norm of the reprojection error using a nonlinear minimization method. Minimizing this reprojection error provides the maximum likelihood estimate when assuming Gaussian noise in the measurements. The problem can then be stated as:

[0030]

[0031] Where d is the Euclidean distance between two points. Solving the above equation involves minimizing the cost function, which is defined as follows:

[0032] E(q)=e(q) T e(q)

[0033] e(q)=x(q)-x

[0034] This coordinate system is converted into image coordinates based on the camera's intrinsic parameter relationship, so that both are in the same picture.

[0035] Preferably, the driver attention analysis, through the gaze point statistics method and the collected gaze point data, counts the distribution of the driver's gaze points while driving frame by frame to obtain the driver's attention distribution result. This project uses the gaze point statistics method to process the driver's line of sight distribution, collects gaze point data through an eye tracker, and counts the distribution of the driver's gaze points while driving frame by frame. After statistics, the driver's gaze points during driving are divided into eight areas, named A, B, C, D, E, F, G, and H areas, which are far away from the road, far away from the left, far away from the right, left side of the road, right side of the road, inside the cab, left side of the vehicle, and right side of the vehicle.

[0036] First, the number and duration of the driver's gaze points in each area are counted by combining the collected eye movement data and gaze point information. The total number of frames and cumulative gaze duration of the driver's gaze in a certain area over a period of time can be counted. In addition, the attention percentage of each area is calculated, that is, the proportion of the number of gaze points in the area to the total number of gaze points, or the proportion of the gaze duration in the area to the total gaze duration, so as to judge the driver's attention to the area.

[0037] Secondly, the targets in each area are analyzed in detail. By analyzing the distribution of sight lines, the driver's attention to the targets in a certain area during driving is analyzed, such as road signs, distant vehicles or other specific targets. The frequency and duration of the driver's gaze on different targets are recorded. If the driver frequently looks at a specific target, such as an obstacle on the road, it means that the target may have attracted the driver's high attention. Such data can provide a basis for the subsequent optimization of road signs and markings.

[0038] Based on the above analysis results, the driver's attention distribution during driving is judged from multiple aspects. Combined with the on-board driving assistance system, when the system background detects that the driver's attention to a certain target in a certain area during driving exceeds the set threshold, the system will issue an alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The overall flow chart of the method of the present invention is as follows;

[0040] Figure 2 It is a schematic diagram of coordinate conversion to the front-end screen;

[0041] Figure 3 Provide a distribution map of the driver's attention area; DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0043] like Figure 1 As shown, a driver attention distribution detection method based on eye tracking includes the following steps:

[0044] Step 1: Collect the driver's facial image and the external driving environment image, obtain the eye key points, and extract the eye movement data;

[0045] The MediaPipe algorithm is mainly used to build machine learning pipelines, which can simplify complex multimedia processing processes, such as human posture estimation, facial recognition, gesture recognition and other tasks in video streams. Taking facial recognition as an example, the key feature points of the face can be accurately detected. Usually it can identify multiple feature points of the face contour, eyes (including eye corners, pupils, etc.), eyebrows, nose (nose tip, nose wings, etc.), mouth (lip contour, mouth corners, etc.). The MediaPipe algorithm can also estimate the posture of the head, including pitch (nodding), yaw (shaking head) and roll (head tilt) angles. After knowing the facial orientation, we can extract the driver's eye orientation, which is good for the subsequent extraction of eye movement data, analysis of driver attention, and auxiliary driving safety in this project.

[0046] The Mediapipe algorithm is used to extract the driver's facial mesh to determine the location of the driver's eyes and extract eye movement data including iris features. The Mediapipe algorithm is a powerful computer vision tool that can efficiently process image data and accurately identify various facial features. Based on the machine learning vision algorithm, a fine facial mesh is extracted from the driver's facial image. The facial mesh is composed of a series of points, which are distributed in various parts of the driver's face and together form a comprehensive facial model. The facial mesh extracted in this project has a total of 468 points. This facial mesh contains a lot of information, including key facial feature points and key points that can be identified, so as to conduct a comprehensive analysis of the face and accurately determine the location of the driver's eyes. In addition, the algorithm can also detect the key points of the eyes including the iris. Using the detected iris features to extract eye movement data can ensure a certain degree of accuracy and real-time performance.

[0047] Step 2: Based on the physiological structure of the eyeball, a double-ball mathematical model is established for the real eyeball structure to determine the driver's eyeball position;

[0048] In order to establish a mathematical model for the real eyeball structure, the model consists of two spheres, the pupil sphere and the eyeball sphere. The light path generated by the line of sight can be regarded as the line connecting the centers of the two spheres. By calculating their relative position and size in the three-dimensional Cartesian coordinate system, the optical axis vector of the eyeball is determined.

[0049] Specifically, the pupil sphere represents the actual structure of the pupil, while the eyeball sphere simulates the shape of the entire eyeball. In a three-dimensional Cartesian coordinate system, we can determine the relative position relationship between the two spheres by measuring and calculating the center coordinates of the two spheres. Based on these data, we can simulate the light path generated by the line of sight along the line connecting the two sphere centers. The optical axis vector is an important parameter that describes the optical direction of the eyeball. It determines the propagation path of light after entering the eyeball and the imaging position on the retina.

[0050] In the two-sphere model, the optical axis is represented as the line connecting the centers of the two spheres. However, for the pupil fitting ellipse in the two-dimensional image, the optical axis is calculated. It is necessary to first obtain the projection of the optical axis on a two-dimensional plane. Pupil image as input data.

[0051] When projected onto a two-dimensional image, the three-dimensional optical axis vector passes through the center of the eyeball and the center of the pupil in the two-dimensional image, and the direction of the vector is the same as the direction of the minor axis of the pupil fitting ellipse. According to the rotational invariance of the sphere, the above two features always hold true when the pupil moves, and the eyeball coordinates and radius remain unchanged.

[0052] By calculating the coordinates of the center point (xe ,y e ) and radius R, let the center of the eyeball be located on the XOY plane of the three-dimensional coordinate system, and determine the three-dimensional coordinate equation of the eyeball (xx e ) 2 +(yy e ) 2 +z 2 =R 2 . Define the three-dimensional coordinate equation of the pupil sphere (xi) 2 +(yj) 2 +(zk) 2 =r 2 , combined eyeball and pupil sphere equations:

[0053]

[0054] Solve the equations to get the plane where the two spheres intersect, substitute it into any equation, and get the projection equation of the intersection line on the XOY plane. And the projection of the center of the pupil sphere on the XOY plane falls on the minor axis l: y = kx + b of the ellipse fitting equation. Two constraint equations can be obtained:

[0055] j=ki+b

[0056] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0

[0057] By solving the above equation, we can find the center of the pupil sphere (x p ,y p , z p ), and the center of the eyeball solved in the previous article, the optical axis of the double-ball model is calculated, and the modeling of the double-ball model of the eyeball is completed.

[0058]

[0059] Step 3: Based on the 3D eyeball model, model the human eye and gaze target to obtain the coordinates of the driver's gaze point;

[0060] In order to model the human eyes and the gaze target, the visual axis vector and the three-dimensional coordinates of its intersection point in space are calculated based on the positional relationship of the two eyeballs in a fixed space, thereby obtaining the coordinates of the driver's gaze point.

[0061] In physiology, the angle between the optical axis and the visual axis is called kappa (k). In the application scenario of this project, the optical axis direction of the camera may not be consistent with the visual axis direction of the driver's eyes, so the size of the kappa (k) angle is used to reflect the degree of inconsistency. If the kappa (k) angle is large, it may cause a large visual deviation between the position of the object seen by the driver and the position of the object recorded by the camera. This project has high accuracy requirements for extracting the driver's gaze point. Therefore, we calibrate this angle to ensure that the visual perception of the camera and the driver is as consistent as possible. The kappa angle of adults is fixed, and the horizontal and vertical rotation angles are about 5° and 1.5° respectively. The visual axis is used to measure the current gaze direction of the subject, that is, the gaze point is defined as the intersection of the binocular visual axes, not the intersection of the optical axes, so it is necessary to further model the positional relationship between the optical axis and the visual axis.

[0062] First, define a simple visual axis-optical axis angle model, where the unit vector of the optical axis is v o =[0 0 1] T , the angle between the visual axis and the optical axis is expressed as κ = (α, β), and the visual axis vectors of the left and right eyes can be expressed as:

[0063]

[0064] Decomposing the optical axis into horizontal and vertical angles is expressed as but:

[0065]

[0066] Finally, the intersection of the left and right eye visual axis vectors in space is calculated to estimate the subject's current gaze point. The three-dimensional coordinates of the gaze point are defined as g(o, κ) = (g x , g y , g z ), the three-dimensional coordinate of the eyeball is o=(o x , o y , o z ), the fixation point g(o,κ) is determined by the following formula:

[0067]

[0068] Different subjects have different kappa angles. During the personal calibration process, the subjects are asked to look at N specific calibration points on the screen: g i , i = 1, 2, ..., N, and then optimize the model parameters by minimizing the distance between the estimated predicted gaze point and the actual gaze point, as follows:

[0069]

[0070] Among them, gi are the actual coordinates of the calibration points, is the coordinate of the driver's current gaze point.

[0071] Step 4: Use the PNP algorithm to realize the coordinate transformation of the focus point and integrate the sight direction into the external environment;

[0072] like Figure 2 As shown in the figure, in order to integrate the sight direction into the external environment, the conversion relationship between the two coordinate systems of the driver's facial image and the external environment image and the world coordinate system is calculated through the PNP algorithm, and the pairwise conversion relationship between the three coordinate systems is obtained, so that the driver's sight direction is integrated into the external environment image; the PNP (Perspective-n-Point) algorithm is an algorithm used in the field of computer vision to solve the position and posture (position and attitude) of an object. In essence, it calculates the position and attitude of the camera (or observer) relative to the object through the projection positions of n known points in the three-dimensional space (in the object coordinate system) on the image plane.

[0073] Since the positional relationship between the two cameras that collect images is fixed, the rotation R and translation T vectors are known, the three-dimensional coordinates of the sight vector in its camera coordinate system can be converted to the external environment perception camera coordinate system through a certain rotation and translation relationship, realizing the fusion of the sight vector and the external environment in the same coordinate system, and converting the three-dimensional point represented in the world frame (the other camera, understood as w here) into the camera frame:

[0074]

[0075]

[0076] Point X expressed in 3D world coordinates w is projected into the image plane [u, v] using the perspective projection model Π and the camera intrinsic parameter matrix A:

[0077]

[0078]

[0079] Equations of the above form can be solved using some algebraic derivation by using a method called Direct Linear Transformation (DLT).

[0080] The "gold standard" solution for PNP is to estimate the six parameters of the transformation by minimizing the norm of the reprojection error using a nonlinear minimization method. Minimizing this reprojection error provides the maximum likelihood estimate when assuming Gaussian noise in the measurements. The problem can then be stated as:

[0081]

[0082] Where d is the Euclidean distance between two points. Solving the above equation involves minimizing the cost function, which is defined as follows:

[0083] E(q)=e(q) T e(q)

[0084] e(q)=x(q)-x

[0085] This coordinate system is converted into image coordinates based on the camera's intrinsic parameter relationship, so that both are in the same picture.

[0086] Step 5: Analyze the driver's attention based on the obtained gaze information.

[0087] In order to analyze the driver's attention, the gaze point distribution of the driver while driving was counted frame by frame through the gaze point statistics method and the collected gaze point data to obtain the driver's attention distribution result.

[0088] Gaze usually refers to the state when the driver's line of sight stays in the 2-5° area of ​​central vision for a minimum of more than 80-100ms. The length of gaze time indicates the length of time the driver focuses on a certain area, reflecting the driver's attention to the area, and can indirectly reflect the driver's energy used to obtain information from the area and the difficulty of processing information.

[0089] This project uses the gaze point statistics method to process the driver's line of sight distribution. Figure 3 As shown. With the help of the camera that collects eye movement data, we collect the driver's gaze point data during driving and count the distribution of their gaze points frame by frame. After detailed statistical analysis, we divide the driver's gaze points during driving into eight different areas, named A, B, C, D, E, F, G, and H. The corresponding areas are far away from the road, far away on the left, far away on the right, left side of the road, right side of the road, inside the cab, left side of the vehicle, and right side of the vehicle.

[0090] First, combining the collected eye movement data and fixation point information, we counted the number and duration of the driver's fixation points in each area, that is, by counting the total number of frames and cumulative fixation duration of the driver's fixation on a certain area within a specific period of time. In addition, the driver's attention to the area is judged by calculating the percentage of attention in each area. The percentage of attention can be determined by the ratio of the number of fixations in the area to the total number of fixations, or it can be calculated by the ratio of the fixation duration in the area to the total fixation duration.

[0091] Secondly, conduct an in-depth and detailed analysis of the targets in each area. By studying the distribution of sight lines, analyze the driver's attention to the targets in a certain area during driving. For example, analyze the driver's attention to road signs, distant vehicles, or other specific targets. At the same time, record the driver's gaze frequency and duration of different targets. If the driver frequently looks at a specific target, such as an obstacle on the road, then this means that the target may have attracted the driver's high attention. This type of data can provide an important basis for the subsequent optimization of road signs and markings.

[0092] Based on the above analysis results, we can judge the driver's attention distribution during driving from multiple aspects. Combined with the in-vehicle driving assistance system, when the system background detects that the driver's attention to a certain target in a certain area during driving exceeds the preset threshold, the system will issue an alarm to remind the driver to pay attention to potential dangers or poor driving conditions.

[0093] The above describes the specific implementation methods of the present invention in conjunction with the accompanying drawings, but does not limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solution of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A driver attention distribution detection method based on eye tracking, characterized in that: The following steps are involved: Step 1: Collect the driver's facial image and the external driving environment image, obtain the eye key points, and extract the eye movement data; Step 2: Based on the physiological structure of the eyeball, a mathematical model is established for the real eyeball structure to fix the positional relationship between the two eyeballs in space; Step 3: Based on the 3D eyeball model, model the human eye and gaze target to obtain the coordinates of the driver's gaze point; Step 4: Use the Perspective-n-Point algorithm, namely the PNP algorithm, to achieve the coordinate transformation of the focus point and integrate the sight direction into the external environment; Step 5: Analyze the driver's attention based on the obtained gaze information.

2. The method for detecting driver attention distribution based on eye tracking according to claim 1, characterized in that: The step 1 obtains the eye key points of the driver's facial image, uses the Mediapipe algorithm to extract the driver's facial mesh, thereby determining the location of the driver's eyes and extracting eye movement data including iris features.

3. The method for detecting driver attention distribution based on eye tracking according to claim 1, characterized in that: The step 2 establishes a mathematical model for the real eyeball structure. The model consists of two spheres, namely the pupil sphere and the eyeball sphere. The light path generated by the line of sight can be regarded as the line connecting the centers of the two spheres. By calculating their relative positions and sizes in the three-dimensional Cartesian coordinate system, the optical axis vector of the eyeball is determined.

4. The method for detecting driver attention distribution based on eye tracking according to claim 1, characterized in that: The step 3 models the human eye and the gaze target, and based on the positional relationship between the two eyeballs in a fixed space, calculates the visual axis vector and the three-dimensional coordinates of its intersection point in space, thereby obtaining the coordinates of the driver's gaze point.

5. The method for detecting driver attention distribution based on eye tracking according to claim 1, characterized in that: In step 4, the process of integrating the sight direction into the external environment is as follows: First, the conversion relationship between the two coordinate systems of the driver's facial image and the external environment image and the world coordinate system is determined by the PNP algorithm. Specifically, the algorithm uses the known position information of specific points in different coordinate systems, and through complex geometric calculations and optimization processes, solves the rotation and translation parameters of the two coordinate systems relative to the world coordinate system. After determining these conversion relationships, the conversion relationships between the three coordinate systems can be obtained, thereby obtaining the conversion matrix from the driver's facial image coordinate system to the world coordinate system, and the conversion matrix from the external environment image coordinate system to the world coordinate system, and then through matrix operations, the conversion relationship between the driver's facial image coordinate system and the external environment image coordinate system can be obtained. Finally, the driver's line of sight direction is integrated into the external environment image, which more intuitively shows the driver's line of sight focus in the external environment, providing more accurate and comprehensive information for analyzing the driver's attention distribution.

6. The method for detecting driver attention distribution based on eye tracking according to claim 1, characterized in that: The step 5 driver attention analysis, this project uses the gaze point statistics method to process the driver's line of sight distribution, collects the gaze point data through the camera, and performs frame-by-frame statistics on the driver's gaze point distribution during driving. After statistics, the driver's gaze points during driving are divided into eight areas, named A, B, C, D, E, F, G, and H areas, corresponding to the far side of the road, the far side of the left side, the far side of the right side, the left side of the road, the right side of the road, the cab, the left side of the vehicle, and the right side of the vehicle. Combined with the collected eye movement data and gaze point information, the number and duration of the driver's gaze points in each area are counted respectively, and the percentage of attention in each area is calculated, so as to judge the driver's attention to the area. At the same time, the target objects in each area are analyzed in detail. Through the line of sight distribution, the driver's attention to the target objects in a certain area during driving is analyzed, and the driver's gaze frequency and duration of different targets are recorded. According to the above analysis results, the driver's attention distribution during driving is judged in many aspects. Preferably, the driver's attention analysis, through the gaze point statistics method and the collected gaze point data, the driver's gaze point distribution during driving is counted frame by frame, and the driver's attention distribution result is obtained, so as to judge the driver's attention distribution during driving in many aspects. Combined with the on-board driving assistance system, when the system background detects that the driver's attention to a target object in a certain area during driving lasts for a period of time or the degree of attention exceeds the set threshold, the system will issue an alarm.

Images