Sight tracking test method
By acquiring the line of sight direction in the target image and converting it to the body coordinate system, combining 3D model and deep learning model, the problem of insufficient accuracy and stability of traditional line of sight tracing systems is solved, and higher line of sight detection accuracy and stability are achieved .
Patent Information
- Application Number
- CN202411111958.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-10
AI Technical Summary
The traditional line-of-view tracing system is affected by lighting conditions and changes in the head posture, resulting in insufficient accuracy and stability. Manual calibration consumes a lot of manpower and time, has low reproducibility and low test accuracy.
By obtaining the starting point and direction of the line of sight in the target image, converting to the body coordinate system, combining the 3D model and deep learning model, determining the gaze direction and the landing area, and using multi-source information fusion to improve the accuracy of the driver's line of sight detection and stability.
It improves the accuracy and stability of line of sight detection, reduces the impact on lighting conditions and head posture changes, reduces the complexity and time cost of manual calibration, and improves the reproducibility and accuracy of the test.
Smart Images
Figure CN119107625A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an automobile safety testing method, in particular to a testing method for sight tracking in driver fatigue detection, and belongs to the technical field of advanced driver assistance control in intelligent vehicle control. Background Art
[0002] In the monitoring system of the driver recognition device, cameras are usually used to monitor the driver's status information, such as the degree of distraction, abnormal behavior, head deviation angle, and gaze area. However, the traditional gaze tracking system is affected by lighting conditions and changes in head posture, resulting in insufficient accuracy and stability. Usually, related technologies require manual arrangement of different scenes and calculation of calibration gaze data based on manually measured data, which consumes a lot of manpower and time, has low reproducibility, and low test accuracy. Summary of the invention
[0003] In order to solve the above problems, the present invention proposes a gaze tracking test method to improve overall driving safety by fusing multi-source information.
[0004] In order to achieve the above objectives, the specific technical solution of the present invention is as follows: A test method for eye tracking comprises the following specific steps: Step 1, obtaining the sight line starting point and sight line method in the target image in the camera coordinate system; Step 2, obtain the 3D model in the cockpit and convert the camera coordinate system into the vehicle body coordinate system; Step 3, calculate the starting point and direction of the sight line of the vehicle body coordinate system; Step 4: Determine the main viewing method and the area of vision based on the partitions inside the vehicle.
[0005] Furthermore, in step 1, the target image is a 2D image captured by a camera, and the line of sight is a spatial vector, and the starting point and direction of the spatial vector are the starting point and direction of the line of sight in the camera coordinate system.
[0006] Furthermore, the sight line starting point and sight line direction of the camera coordinate system can be converted to the camera coordinate system through the camera's intrinsic parameters, thereby calculating the 3D coordinates of the eye in the camera coordinate system, thereby determining the starting coordinates of the coordinate sight line. The image coordinate system is calculated from the camera coordinate system using a standard formula, and with the help of Matlab's calibration tool, the distortion parameters and intrinsic parameters can be calibrated at one time.
[0007] Furthermore, in step 2, the 3D model of the cockpit includes 3D sparse point modeling inside the vehicle, template library creation, and real vehicle dynamic calibration.
[0008] The 3D sparse point modeling inside the car includes: using the same camera to take pictures of the car at different angles, in particular, pasting drawings of different pictures on the glass inside the car to increase the number of feature points; using the same camera to take pictures of the car under different lighting conditions, and selecting typical and useful samples from them. Using the same camera to take at least 20 pictures of the car at different angles, the minimum angle unit is 7-8° to dedistort the samples, using opencv to extract the feature points of the samples, and arranging them into corresponding sift files. Using software tools to reconstruct 3D point clouds for sift. Selecting the z-axis and x-axis points and the origin of the coordinate system that can represent the body coordinate system from the point cloud, and converting the point cloud into the body coordinate system. Transfer the 3D model to the body coordinate system, the body coordinate system is defined as x pointing to the right of the driver, y upward, z to the front of the driver, and the origin of the coordinate is selected as a point on the handle above the main driver. The coordinate point is used as the anchor point of the body coordinate system, and can also be selected arbitrarily.
[0009] Furthermore, the template library is created by organizing 20 to 40 pictures under different light angles and corresponding feature information into a template library; considering that in actual scenes, the sunroof, main driving window and driver's activity area cannot be matched with the features, these areas on each template image are excluded through masks.
[0010] Furthermore, the real vehicle dynamic calibration uses a camera to take a picture, and then extracts feature points of the entire picture through the opencv tool; the feature points of the picture are matched with each template in the template library, and the matching results are further screened and filtered by the algorithm to organize useful matching information; the useful information includes the pixel coordinates of the feature points in the camera picture that can participate in the matching, the corresponding points in the 3D point cloud that can participate in the matching, and their corresponding coordinates in the vehicle body coordinate system.
[0011] Furthermore, the sight starting point and sight direction in the vehicle body coordinate system are obtained; the vehicle body coordinate system can calculate the relationship between the camera coordinate system and the vehicle body coordinate system through steps such as PNP, thereby obtaining the sight starting point and sight direction in the vehicle body coordinate system.
[0012] Furthermore, the direction and area where the line of sight is fixed on are determined based on the partitions inside the vehicle. The partitions inside the vehicle are 3D coordinate areas in the vehicle body coordinate system, and each area is surrounded by 4 points. If the intersection of the line of sight and the area falls within the area, then the line of sight is fixed on the area. Combined with the starting point and direction of the line of sight in the vehicle body coordinate system, the intersection of the line of sight and the plane where each area is located is finally calculated.
[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention obtains the sight line direction in the target image, obtains the 3D model in the cockpit, obtains the sight line starting point and sight line direction in the vehicle body coordinate system, and determines the sight line gaze direction and the area where the sight line falls according to the partitions in the vehicle. Based on the depth model, the present invention collects a large number of sight line data of different people, obtains a large number of training sets of training sight line directions, and thus obtains the depth information of the target image. Since the present invention reconstructs the 3D cockpit based on different scenes, the obtained sight line starting point and sight line direction are highly accurate, not easily affected by lighting conditions and changes in head posture, and have high stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic diagram of the process of the present invention.
[0015] Figure 2 It is a schematic diagram summarizing the situation of the plane corner points where multiple frames of sight lines and the left front windshield partition are located in an embodiment of the present invention. DETAILED DESCRIPTION
[0016] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present invention, the examples of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0017] This embodiment proposes a gaze tracking test method to obtain the gaze direction in the target image, obtain the 3D model in the cockpit, obtain the gaze starting point and gaze direction in the vehicle body coordinate system, and determine the gaze direction and the area where the gaze falls according to the partitions in the vehicle.
[0018] Optionally, the sight line direction in the target image is obtained, the target image is a 2D image taken by a camera, the sight line is a spatial vector, and the sight line starting point and sight line direction in the camera coordinate system need to be determined.
[0019] Optionally, the starting point and direction of the line of sight of the camera coordinate system can be converted into the camera coordinate system through the intrinsic and extrinsic parameters of the camera, thereby calculating the 3D coordinates of the eye in the camera coordinate system, thereby determining the starting coordinates of the coordinate line of sight.
[0020] Optionally, the sight direction is obtained by a deep learning model. The deep model is based on multi-camera acquisition to train sight and head posture. The deep model can predict the depth of facial pixels relative to the camera, thereby obtaining the sight direction of the sight.
[0021] Optionally, a 3D model of the cockpit is obtained, which includes 3D sparse point modeling inside the vehicle, template library creation, and real vehicle dynamic calibration.
[0022] Optionally, the 3D sparse point modeling inside the vehicle includes: Use the same camera to take pictures of the interior of the car from different angles. In particular, paste drawings of different pictures on the glass inside the car to increase the number of feature points. Use the same camera to take pictures of the interior of the car under different lighting conditions, and select typical and useful samples from them. Dedistort the samples, use opencv to extract the feature points of the samples, and organize them into corresponding sift files. Use software tools to reconstruct the 3D point cloud of sift. Select the z-axis and x-axis points and the origin of the coordinate system that can represent the body coordinate system from the point cloud, and convert the point cloud into the body coordinate system. Transfer the 3D model to the body coordinate system. The body coordinate system is defined as x pointing to the right of the driver, y upward, and z to the front of the driver. The origin of the coordinate is selected as a point on the handle above the main frame.
[0023] Optionally, the template library is created by organizing dozens of pictures under different light angles and corresponding feature information into a template library. Considering that in actual scenes, the skylight, the main driving window and the driver's activity area cannot match the features, these areas on each template image are excluded by masking.
[0024] Optionally, the real vehicle dynamic calibration uses a camera to take a picture and extracts the feature points of the whole picture through the opencv tool. The feature points of the picture are matched with each template in the template library, and the matching results are further screened and filtered by the algorithm to organize useful matching information. The useful information includes the pixel coordinates of the feature points in the camera picture that can participate in the matching, the corresponding points in the 3D point cloud that can participate in the matching, and their corresponding coordinates in the vehicle body coordinate system.
[0025] Optionally, the sight starting point and sight direction in the vehicle body coordinate system are obtained. The vehicle body coordinate system can calculate the relationship between the camera coordinate system and the vehicle body coordinate system through steps such as PNP, so as to obtain the sight starting point and sight direction in the vehicle body coordinate system.
[0026] Optionally, the direction and area of gaze are determined according to the partitions in the vehicle, and the partitions in the vehicle are 3D coordinate areas in the vehicle body coordinate system, each area is surrounded by 4 points. If the intersection of the line of sight and the area falls within the area, then the line of sight is gazing at the area. Combined with the starting point of the line of sight and the direction of the line of sight in the vehicle body coordinate system in claim 9, the intersection of the line of sight and the plane where each area is located can be calculated.
[0027] Optionally, the partitions in the car are generally the front windshield, left and right rearview mirrors, interior rearview mirrors, instrument panel, etc. Further, the specific area of the partition can be customized, but due to the error accuracy of the line of sight, the area should not be too small.
[0028] Furthermore, the error accuracy of the line of sight is related to the placement of the camera. Within the 30° inner cone in front of the camera, the error is less than 5°, and beyond the 30° inner cone in front of the camera, the error is about 10°.
[0029] In this embodiment, the sight line partition is output by caching multiple frames and other logic. Figure 2 As shown, the multi-frame sight lines and the corner points of the plane where the left front windshield partition is located are summarized. The gray area is the left front windshield partition range, the gray point is the intersection of the sight line and the plane where the front windshield is located, and the points within the gray line range indicate that these frame sight lines are looking at the left front windshield partition.
[0030] Those skilled in the art should understand that the present invention is not limited to the above-mentioned specific embodiments, and the present invention is subject to various changes and improvements, all of which fall within the scope of the present invention claimed for protection.
Claims
1. A method for testing gaze tracking, characterized in that: The specific steps include: Step 1, obtaining the sight line starting point and sight line method in the target image in the camera coordinate system; Step 2, obtain the 3D model in the cockpit and convert the camera coordinate system into the vehicle body coordinate system; Step 3, calculate the starting point and direction of the sight line of the vehicle body coordinate system; Step 4: Determine the main viewing method and the area of vision based on the partitions inside the vehicle.
2. The sight tracking test method according to claim 1, characterized in that: In step 1, the target image is a 2D image captured by a camera, and the line of sight is a space vector, and the starting point and direction of the space vector are the starting point and direction of the line of sight in the camera coordinate system.
3. The sight tracking test method according to claim 2, characterized in that: The starting point and direction of the line of sight of the camera coordinate system can be obtained through the intrinsic parameters of the camera, and the image coordinate system can be converted into the camera coordinate system, so as to calculate the 3D coordinates of the eye in the camera coordinate system, thereby determining the starting coordinates of the coordinate line of sight.
4. The eye tracking test method according to claim 1, characterized in that: In step 2, the 3D model of the cockpit includes 3D sparse point modeling inside the vehicle, template library creation, and real vehicle dynamic calibration.
5. The sight tracking test method according to claim 4, characterized in that: The 3D sparse point modeling inside the car includes: using the same camera to take pictures of the car from different angles, pasting drawings of different pictures on the glass inside the car, using the same camera to take at least 20 pictures of the car from different angles, the minimum angle unit is 7-8°, and selecting typical useful samples; dedistorting the samples, extracting the feature points of the samples using opencv, and arranging them into corresponding sift files; using software tools to reconstruct 3D point clouds from sift, selecting the z-axis and x-axis points and the origin of the coordinate system that can represent the body coordinate system from the point cloud, and converting the point cloud into the body coordinate system. The 3D model is transferred to the body coordinate system, and the body coordinate system is defined as x pointing to the right of the driver, y upward, and z to the front of the driver, and the origin of the coordinates is selected as any point on the main driver's steering wheel handle.
6. The sight tracking test method according to claim 4, characterized in that: The template library is created by organizing 20 to 40 pictures under different light angles and corresponding feature information into a template library; considering that in actual scenes, the sunroof, main driving window and driver's activity area cannot be matched with the features, these areas on each template image are excluded through masks; the feature information refers to the feature points calibrated in the car and the left and right rearview mirrors, except for the driver's activity area and deformable objects, including corner points, edges and spots in the captured pictures that can be extracted by opencv.
7. The sight tracking test method according to claim 4, characterized in that: The real vehicle dynamic calibration uses a camera to take a picture, and then extracts the feature points of the whole picture through the opencv tool; the feature points of the picture are matched with each template in the template library, and the matching results are further screened and filtered by the algorithm to organize useful matching information; the useful information includes the pixel coordinates of the feature points in the camera picture that can participate in the matching, the corresponding points in the 3D point cloud that can participate in the matching, and their corresponding coordinates in the vehicle body coordinate system.
8. The method according to claim 1, characterized in that The starting point and direction of the sight line in the vehicle body coordinate system are obtained; the vehicle body coordinate system can calculate the relationship between the camera coordinate system and the vehicle body coordinate system through steps such as PNP, so as to obtain the starting point and direction of the sight line in the vehicle body coordinate system.
9. The method according to claim 1, characterized in that: The direction and area where the line of sight is fixed on are determined based on the partitions inside the vehicle. The partitions inside the vehicle are 3D coordinate areas in the vehicle body coordinate system, and each area is surrounded by 4 points. If the intersection of the line of sight and the area falls within the area, then the line of sight is fixed on the area. Combined with the starting point and direction of the line of sight in the vehicle body coordinate system, the intersection of the line of sight and the plane where each area is located is finally calculated.