Sight tracking data acquisition method, device and system

Abstract

The invention provides a gaze tracking data acquisition method, device and system, the method is applied to a control device, and the method comprises the following steps: controlling a display screen to display a set first gaze drop point; based on the input of the tester on the display screen, after the display screen displays a second sight drop point, whether the first sight drop point and the second sight drop point meet a preset distance condition is identified, the second sight drop point is determined based on the guidance of the first sight drop point, and the first sight drop point and the second sight drop point form a sight drop point group; and in response to the condition that the first sight line drop point and the second sight line drop point meet a preset distance condition, controlling the IR acquisition camera to perform face acquisition on the tester, obtaining a face image carrying sight line tracking data, and storing target coordinates corresponding to the face image and the first sight line drop point in a camera coordinate system of the IR acquisition camera. According to the application, the cost and complexity of data acquisition can be reduced, the stability and reliability of data acquisition are improved, and the demand of manual intervention is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and system for collecting gaze tracking data. Background Art

[0002] On highways and urban roads, the driver's attention and state directly affect driving safety. In order to improve road safety, DMS (Driver Monitoring System) evaluates the driver's attention level and alertness in real time by analyzing the driver's facial features, eye movements and head posture. The DMS system is usually integrated on the front panel of the vehicle and uses IR cameras and visual algorithms for monitoring. IR cameras (also known as infrared cameras) are cameras that can capture infrared light (usually invisible light). Infrared cameras focus on detecting light in the infrared band, which gives them obvious advantages in low-light environments or at night. IR cameras can capture the driver's facial features and eye movements, and can provide clear images even in low-light environments, which helps to accurately assess the driver's fatigue and distraction status and ensure driving safety.

[0003] Distraction monitoring is an important component of DMS, which aims to improve driving safety. With the advancement of autonomous driving technology and the popularization of intelligent driving assistance systems, driver status monitoring has become increasingly important. Distraction monitoring is mainly achieved through eye tracking technology, which is a technology used to track and record the position and movement of a person's eyes. It can help understand people's gaze points, areas of attention, and visual behaviors, and requires special hardware and software systems to achieve.

[0004] Eye tracking mainly includes two main parts: data collection and visual algorithm data analysis. The data collection part is completed by an IR camera, which can collect clear facial images under different lighting conditions at different times, different head postures, and different eye movement states. By analyzing this data, the visual algorithm can determine whether the driver is paying attention. The visual algorithm usually includes algorithms based on deep learning and traditional computer vision. By learning and counting the image data obtained from data collection, it can accurately judge the driver's attention and behavior status during actual vehicle use.

[0005] Data collection is the basis for the successful application of visual algorithms, affecting the accuracy, robustness and generalization ability of the algorithms. High-quality, diverse and accurately labeled data is essential for training and validating visual algorithms. Eye tracking technology requires high-precision data collection due to the high complexity of the algorithm, so that eye movement behavior can be effectively monitored. High-precision data collection for eye tracking needs to meet the following key requirements:

[0006] 1. High-precision sight point measurement: Accurately measuring the sight point is one of the core tasks of sight tracking data acquisition. The accuracy of the sight point directly affects the effectiveness and reliability of the sight tracking algorithm.

[0007] 2. Consistent lighting conditions: IR cameras may exhibit different photosensitivity characteristics under different lighting conditions. Especially when the driver wears glasses, the reflection and shadow of the glasses may interfere with the results of the gaze tracking algorithm. Therefore, the data collection process and the actual use process need to maintain consistent lighting conditions.

[0008] 3. Low-cost calibration method: High-cost calibration equipment and complex calibration process may lead to problems with actual data collection time and economy. Therefore, it is particularly important to develop a low-cost high-precision calibration method.

[0009] In order to obtain high-precision gaze tracking data, an eye tracker is usually used to collect gaze tracking data. However, the cost of eye trackers is very high, which leads to a surge in data collection costs. In actual applications, eye trackers are usually not used, but low-cost IR cameras are used for data collection.

[0010] When collecting gaze tracking data based on an IR camera, the following key steps are included:

[0011] S1. Collection environment setup: Install collection equipment, including cameras (such as IR cameras or visible light cameras), light sources, locations of people being collected, etc.

[0012] S2. Camera calibration: Prepare a standard calibration chart (such as a chessboard chart) for camera calibration to correct the camera's internal parameters (such as focal length, principal point) and distortion parameters. Use the camera to shoot the calibration chart to obtain image data, and perform calibration processing to obtain the camera's geometric parameters.

[0013] S3. Data collection: Arrange the person being collected to perform eye tracking related actions at the designated location, and use the camera to record the eye movement data of the person being collected (including eye movement trajectory, pupil position and line of sight).

[0014] S4. Data verification and storage: Use known reference points to verify the accuracy of the line-of-sight tracking data and compare the actual measurement results with the expected results. Store the collected data in a database or file system to ensure data integrity and availability.

[0015] At present, when collecting gaze tracking data based on IR cameras, the system is usually built inside the car, with an IR camera set behind the steering wheel, directly aimed at the driver's head, and a gaze point set inside the car. The tester is instructed to look at a specific point. Finally, the inspector subjectively checks the collected images and eliminates erroneous images.

[0016] In the prior art, when collecting sight tracking data based on IR cameras, there are the following problems:

[0017] 1. Construction cost: The cost of building the system in a real car environment is relatively high.

[0018] 2. Accuracy of camera calibration: The camera calibration needs to be as consistent as possible with the actual in-car environment to ensure the accuracy of the sight label. However, the current calibration process often involves complex settings of multiple cameras, which is prone to problems such as exposure and focal length, which affects the accuracy and consistency of the data.

[0019] 3. Reasonableness of the sight point plane setting: Existing technologies usually use a single sight point plane and sparse sight point positions, but these settings are often not completely consistent with the actual driving environment, which may affect the range and accuracy of the sight point. In addition, if the sight point position is not designed properly, it may easily lead to insufficient data diversity in data collection, affecting the generalization of the subsequently developed sight tracking algorithm in actual application scenarios.

[0020] 4. Lighting condition setting: In actual driving environments, the reflections and shadows of glasses have a significant impact on the image acquisition of IR cameras. Existing systems often use fixed lighting conditions when processing these reflections and shadows, which cannot effectively adapt to the changing lighting environment in actual driving; and if the position of the light source is not appropriate, it may cause the gaze tracking data to be inconsistent with the actual vehicle usage scenario, affecting the effectiveness of the gaze tracking algorithm developed later in actual application scenarios.

[0021] 5. Data collection action settings: Different action settings are required for different driver eye movement states (such as head movement and eye movement) to ensure data accuracy. The existing system is not flexible enough in this regard, resulting in the collected data not fully reflecting the actual driving state.

[0022] 6. Difficulty in data review: When performing operations, testers are easily disturbed by the environment and may behave unconsciously, resulting in label errors. Existing data review methods mainly rely on manual inspection of images for problems such as overexposure, darkness, and failure to perform specified actions. This is not only inefficient, but also prone to subjective errors. Therefore, improving the accuracy and efficiency of data review has also become a key issue of concern. Summary of the invention

[0023] In view of the above problems, the embodiments of the present application provide a gaze tracking data collection method, device and system that overcome the above problems or at least partially solve the above problems.

[0024] In a first aspect, an embodiment of the present application provides a gaze tracking data collection method, which is applied to a control device, wherein the control device is connected to an IR collection camera and a display screen, wherein the IR collection camera is arranged on a simulated steering wheel, and the display screen is located in front of the simulated steering wheel, and the method comprises:

[0025] Controlling the display screen to display a set first sight point;

[0026] After the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, identifying whether the first sight point and the second sight point meet a preset distance condition, the second sight point is determined based on the guidance of the first sight point, and the first sight point and the second sight point form a sight point group;

[0027] In response to the first sight point and the second sight point satisfying a preset distance condition, controlling the IR acquisition camera to perform facial acquisition on the tester, obtaining a facial image carrying sight tracking data provided by the IR acquisition camera, and storing the facial image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera;

[0028] The control device obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

[0029] In a second aspect, an embodiment of the present application provides a gaze tracking data acquisition device, which is applied to a control device, wherein the control device is connected to an IR acquisition camera and a display screen, wherein the IR acquisition camera is arranged on a simulated steering wheel, and the display screen is located in front of the simulated steering wheel, and the device comprises:

[0030] A first control module, used for controlling the display screen to display a first sight point set;

[0031] an identification module, configured to identify whether the first sight point and the second sight point meet a preset distance condition after the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, the second sight point being determined based on the guidance of the first sight point, and the first sight point and the second sight point forming a sight point group;

[0032] a processing module, configured to control the IR acquisition camera to perform facial acquisition on the tester in response to the first sight point and the second sight point satisfying a preset distance condition, obtain a face image carrying sight tracking data provided by the IR acquisition camera, and store the face image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera;

[0033] The control device obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

[0034] In a third aspect, an embodiment of the present application provides a gaze tracking data acquisition system, comprising:

[0035] Simulated steering wheel;

[0036] A display screen, arranged in front of the simulated steering wheel, for displaying the sight point;

[0037] An IR acquisition camera, arranged on the simulated steering wheel, for acquiring a facial image of a tester behind the simulated steering wheel;

[0038] A control device is connected to the display screen and the IR acquisition camera respectively, and is used to execute the method described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method described in the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method described in the first aspect.

[0041] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed, is used to execute the method described in the first aspect above.

[0042] The technical solution of the embodiment of the present application can reduce the cost and complexity of data collection while collecting line of sight tracking data through the mutual cooperation of the IR acquisition camera, the display screen and the control device; by controlling the display screen as the line of sight landing plane to display the reference line of sight landing point, guiding the tester to perform input actions on the display screen based on the reference line of sight landing point to record the actual line of sight landing point, it can improve the stability and reliability of data collection based on the interactive acquisition verification method and reduce the need for manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a sight tracking data acquisition system provided in an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram showing a sight tracking data acquisition system built in a target space provided by an embodiment of the present application;

[0045] Figure 3 An example of a world coordinate system and a camera coordinate system provided in an embodiment of the present application is shown;

[0046] Figure 4 A flowchart showing the processing of sight point provided by an embodiment of the present application;

[0047] Figure 5 A schematic diagram showing a method for collecting gaze tracking data provided by an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a sight tracking data acquisition device provided in an embodiment of the present application is shown;

[0049] Figure 7 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0052] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0053] The present application embodiment provides a gaze tracking data acquisition system, such as Figure 1 As shown, it includes: a simulated steering wheel 10, a display screen 20 located in front of the simulated steering wheel 10, an IR acquisition camera 30 set on the simulated steering wheel 10, and a control device 40 connected to the display screen 20 and the IR acquisition camera 30;

[0054] The control device 40 controls the display screen 20 to display the set first sight point;

[0055] In response to the input of the tester behind the simulated steering wheel 10 on the display screen 20, the display screen 20 records the tester's guidance based on the first sight point and the corresponding second sight point on the display screen 20, and the first sight point and the second sight point form a sight point group;

[0056] In response to the first sight point and the second sight point satisfying the preset distance condition, the control device 40 controls the IR acquisition camera 30 to perform facial acquisition on the tester, obtains a face image carrying sight tracking data provided by the IR acquisition camera 30, and stores the face image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera 30;

[0057] The control device 40 obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

[0058] The eye tracking data acquisition system provided in the embodiment of the present application includes: a simulated steering wheel 10, a display screen 20, an IR acquisition camera 30 and a control device 40. Among them, the display screen 20 serves as a sight point plane for displaying content, and the control device 40 is connected to the display screen 20 to control the display screen 20 to display the set first sight point. The display screen 20 can receive clicks from an input device (such as a mouse) and can also receive touch inputs from a user (such as a tester). At this time, the display screen 20 has a touch function. The display screen 20 in this embodiment is equivalent to the windshield of the vehicle. The display screen 20 is in a flat shape and its size can be the same as the size of the windshield of the vehicle.

[0059] The display screen 20 is located in front of the simulated steering wheel 10, and the tester is located behind the simulated steering wheel 10. To ensure authenticity, the distance between the display screen 20 and the simulated steering wheel 10 is consistent with the horizontal distance from the steering wheel of the real car to the center point of the windshield, and the positional relationship between the tester and the simulated steering wheel 10 is similar to that of the real car. The display screen 20 faces the tester's face, ensuring that the tester can browse the plane of sight point.

[0060] The control device 40 sets a first sight point on the sight point plane, controls the display screen 20 to display the first sight point, and the tester faces the display screen 20. Based on the guidance of the first sight point, the tester performs an input action on the display screen 20, such as clicking the display screen 20 with a mouse or manually touching the display screen 20. The display screen 20 records the second sight point corresponding to the input action performed by the tester on the display screen 20 based on the guidance of the first sight point. The second sight point is the sight point corresponding to the input action performed on the display screen 20 by the tester after seeing the first sight point based on the need to click or touch the first sight point.

[0061] The control device 40 obtains the second sight point recorded by the display screen 20 based on the connection with the display screen 20. Since the first sight point is set by the control device 40, after obtaining the second sight point matching the first sight point, the control device 40 identifies whether the distance between the first sight point and the second sight point meets the preset distance condition. If the distance between the first sight point and the second sight point meets the preset distance condition, it is determined that the second sight point is within the error allowable range of the first sight point, and the control device 40 sends a control instruction to the IR acquisition camera 30 connected thereto. The IR acquisition camera 30 is set on the simulated steering wheel 10, specifically, for example, at the 12 o'clock position of the simulated steering wheel 10, and is used to collect the facial image of the test person. After the control device 40 sends the control instruction to the IR acquisition camera 30, the IR acquisition camera 30 performs facial acquisition on the test person to obtain a facial image carrying sight tracking data.

[0062] The control device 40 obtains the face image carrying the sight tracking data provided by the IR acquisition camera 30 based on the connection with the IR acquisition camera 30. Since the face image taken by the IR acquisition camera 30 is the face image corresponding to the set first sight point, the control device 40 calculates the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera 30, and stores the face image provided by the IR acquisition camera 30 and the calculated target coordinates as a set of data to realize the recording of the sight tracking data.

[0063] The first sight point is the reference sight point set by the control device 40, and the second sight point is the actual sight point corresponding to the tester on the sight point plane based on the guidance of the first sight point. The first sight point and the second sight point constitute a sight point group. The control device 40 can identify the sight point group that meets the preset distance condition, and for each sight point group that meets the preset distance condition, obtain the collected face image and calculate the target coordinates corresponding to the reference sight point for storage, so as to record the data corresponding to the multiple sight point groups that meet the preset distance condition, so as to ensure that effective data is provided for the subsequent analysis of the visual algorithm data, thereby ensuring the successful application of the visual algorithm.

[0064] The above-mentioned implementation scheme of the present application, by deploying a gaze tracking data acquisition system including a simulated steering wheel, an IR acquisition camera, a display screen and a control device, can reduce the cost and complexity of data acquisition while performing gaze tracking data acquisition; by controlling the display screen as the gaze landing plane to display the reference gaze landing point, guiding the tester to perform input actions on the display screen based on the reference gaze landing point to record the actual gaze landing point, it can improve the stability and reliability of data acquisition based on an interactive acquisition verification method and reduce the need for manual intervention.

[0065] As an optional embodiment, Figure 2 As shown, the sight tracking data acquisition system further includes: an IR relay camera 50 disposed on the display screen 20, wherein the imaging plane of the IR relay camera 50 is parallel to the plane of the display screen 20, and the display screen 20 is perpendicular to the ground;

[0066] The control device 40 calculates the first world coordinates of the first sight point in the world coordinate system of the display screen 20 based on the pixel coordinates corresponding to the first sight point and the resolution and width and height information of the display screen 20;

[0067] The control device 40 calculates the target coordinates corresponding to the first sight point based on the first world coordinates, the second world coordinates of the IR relay camera 50 in the world coordinate system where the display screen 20 is located, and the external parameters of the IR acquisition camera 30 relative to the IR relay camera 50.

[0068] When calculating the target coordinates corresponding to the first sight point, the control device 40 first determines the pixel coordinates corresponding to the first sight point on the display screen 20. Since the first sight point is set by the control device 40, the control device 40 can directly obtain the pixel coordinates corresponding to the first sight point according to the setting of the first sight point.

[0069] After determining the pixel coordinates (x i ,y i ) then the control device 40 calculates the pixel coordinates (xi ,y i ), the resolution and width and height information of the display screen 20, and calculate the first world coordinate (x iw ,y iw , z iw ). The control device 40 calculates the first world coordinates of the first sight point based on the following formula:

[0070] x iw =x i *w / H

[0071] y iw =y i *h / V

[0072] z iw =0

[0073] x i Indicates the pixel horizontal coordinate of the first sight point, y i The pixel vertical coordinate of the first sight point is represented, H represents the horizontal pixel number of the display screen 20, V represents the vertical pixel number of the display screen 20, w represents the width information of the display screen 20, and h represents the height information of the display screen 20.

[0074] Since the pixel abscissa x of the display screen 20 i With the world horizontal coordinate x iw The ratio is equal to the ratio of the horizontal pixel number H of the display screen 20 to the width information w. Therefore, based on the calculation x i *w / H determines the horizontal coordinate of the display screen 20 in the world coordinate system. i With the world ordinate y iw The ratio is equal to the ratio of the number of vertical pixels V of the display screen 20 to the height information h. Therefore, based on the calculation y i *h / V determines the vertical coordinate of the display screen 20 in the world coordinate system. In this embodiment, the display screen 20 is perpendicular to the ground, and the coordinate of the display screen 20 in the Z-axis direction in the world coordinate system is 0.

[0075] After determining the first world coordinate, the control device 40 calculates the target coordinates corresponding to the first line of sight point based on the first world coordinate, the second world coordinate of the IR relay camera 50 in the world coordinate system where the display screen 20 is located, and the external parameters of the IR acquisition camera 30 relative to the IR relay camera 50.

[0076] The IR relay camera 50 is used as a relay device to convert points on the display screen 20 into the camera coordinate system of the IR acquisition camera 30. The external parameters of the IR acquisition camera 30 relative to the IR relay camera 50 are determined based on the calibration process, such as using the IR acquisition camera 30 and the IR relay camera 50 to simultaneously shoot a calibration plate for camera parameter calibration to obtain the external parameters of the IR acquisition camera 30 relative to the IR relay camera 50. Since the imaging plane of the IR relay camera 50 is parallel to the plane of the display screen 20, the second world coordinate of the IR relay camera 50 in the world coordinate system where the display screen 20 is located can be determined.

[0077] The control device 40 uses the following formula when calculating the target coordinates of the first sight point:

[0078] M=R·(BZ)+T

[0079] B represents the first world coordinate, Z represents the second world coordinate of the camera coordinate origin of the IR relay camera 50 in the world coordinate system where the display screen 20 is located, (BZ) represents the camera coordinate of the first line of sight in the camera coordinate system where the IR relay camera 50 is located, the second world coordinate is determined based on the distance between the camera coordinate origin of the IR relay camera 50 and the world coordinate origin of the display screen 20, R represents the rotation parameter of the IR acquisition camera 30 relative to the IR relay camera 50, and T represents the translation parameter of the IR acquisition camera 30 relative to the IR relay camera 50.

[0080] Since the imaging plane of the IR relay camera 50 is parallel to the plane of the display screen 20, the world coordinate system of the display screen 20 is parallel to the coordinate axis corresponding to the camera coordinate system of the IR relay camera 50, and since the IR relay camera 50 is set on the display screen 20, the distance between the IR relay camera 50 and the simulated steering wheel 10 is the same as the distance between the display screen 20 and the simulated steering wheel 10. In order to ensure that the IR relay camera 50 is perpendicular to the ground and the camera coordinate system is parallel to the length and width of the display screen 20, a laser level can be used to check the horizontal state of the IR relay camera 50; in order to ensure that the display screen 20 is perpendicular to the ground, a laser level can be used to check the horizontal state of the display screen 20.

[0081] The height of the IR relay camera 50 can be kept consistent with the height of the IR acquisition camera 30. Optionally, the IR relay camera 50 is disposed at the left edge of the display surface of the display screen 20. Figure 3 As shown, the imaging plane of the IR relay camera 50 is parallel to the plane of the display screen 20, the world coordinate system of the display screen 20 is parallel to the coordinate axis corresponding to the camera coordinate system of the IR relay camera 50, and the IR relay camera 50 is set at the left edge of the display screen 20. Figure 3In the figure, the lower left corner of the display screen 20 is used as the origin of the world coordinate system to construct the world coordinate system, and the camera optical center of the IR transfer camera 50 is used as the origin of the camera coordinate system to construct the camera coordinate system.

[0082] Among them, since the world coordinate system of the display screen 20 is parallel to the coordinate axis corresponding to the camera coordinate system of the IR relay camera 50, when the camera coordinate origin of the IR relay camera 50 is known, the second world coordinate of the IR relay camera 50 in the world coordinate system where the display screen 20 is located can be determined based on the distance between the camera coordinate origin of the IR relay camera 50 and the world coordinate origin of the display screen 20. For example, the distance between the optical center of the IR relay camera 50 and the origin of the world coordinate system is measured by a rangefinder to determine the coordinate (x ) of the optical center of the IR relay camera 50 in the world coordinate system where the display screen 20 is located. w ,y w , z w ).

[0083] When the first world coordinate B, the second world coordinate Z, and the rotation parameter R and translation parameter T of the IR acquisition camera 30 relative to the IR relay camera 50 are determined, the camera coordinates of the first line of sight in the camera coordinate system where the IR relay camera 50 is located are determined based on (BZ), and then based on the rotation parameter R, translation parameter T and (BZ) of the IR acquisition camera 30 relative to the IR relay camera 50, the camera coordinates of the first line of sight in the camera coordinate system where the IR relay camera 50 is located are converted into the target coordinates in the camera coordinate system where the IR acquisition camera 30 is located.

[0084] In the above implementation scheme, by setting a display screen perpendicular to the ground and setting the imaging plane of the IR relay camera parallel to the plane of the display screen, the sight point plane setting can be optimized, the accuracy of sight point measurement is improved, and the error caused by different viewing angles is reduced; and by using an IR acquisition camera to collect data and using the IR relay camera to convert the points on the display screen to the camera coordinate system of the IR acquisition camera, it is possible to reduce the reliance on high-precision tools such as eye trackers while ensuring the provision of the required sight tracking data, thereby reducing the cost and complexity of data collection.

[0085] In the embodiments of the present application, Figure 2 As shown, the gaze tracking data acquisition system is built in the target space, and the target space is provided with a light-transmitting area for allowing natural light to pass through; the display screen 20 is located between the light-transmitting area and the simulated steering wheel 10, and natural light passes through the light-transmitting area and crosses the display screen 20 to illuminate the face of the tester.

[0086] In this embodiment, an open room with windows that can be illuminated by natural light is selected as a laboratory (or a laboratory simulating such conditions), which is the target space for building a sight tracking data acquisition system. The simulated steering wheel 10 is placed in front of the window (light-transmitting area), and the tester sits behind the simulated steering wheel 10 facing the window. The positions of the simulated steering wheel 10 and the tester are determined and cannot be moved.

[0087] A certain space is reserved between the window and the simulated steering wheel 10, and then an IR acquisition camera 30 is set on the simulated steering wheel 10. A display screen 20 is set between the window and the simulated steering wheel 10. The display screen 20 is perpendicular to the ground, and the display screen side of the display screen 20 faces the face of the test person, ensuring that natural light can pass through the display screen 20 and shine on the face opposite the display screen 20. The lower edge of the display screen 20 coincides with the lower edge of the simulated steering wheel 10, and the display screen 20 can move horizontally left and right but cannot move forward and backward in the direction between the simulated steering wheel 10 and the test person. The initial position of the display screen 20 can be: the center point of the simulated steering wheel 10 is located on the central axis of the upper and lower edges of the display screen 20. After the display screen 20 is set between the window and the simulated steering wheel 10, an IR transfer camera 50 is set on the display screen 20 with an imaging plane parallel to the plane of the display screen.

[0088] like Figure 2 As shown, the eye tracking data acquisition system also includes: an IR fill light 60 arranged at the periphery of the IR acquisition camera 30 and connected to the control device 40. When the natural light source in the acquisition environment is insufficient, the control device 40 controls the IR fill light 60 to turn on to provide light source for the acquisition environment, so that the IR acquisition camera 30 can capture clear images.

[0089] In a collection environment with sufficient sunlight during the day, natural light is mainly relied on; at night or during the day when natural light is insufficient, the control device 40 controls the IR fill light 60 to turn on, and relies on the IR fill light 60 to provide an auxiliary light source to ensure that the IR collection camera 30 can capture clear images under limited lighting conditions to provide high-quality line of sight tracking data.

[0090] Among them, natural light shines directly onto the tester's face through the window. If the tester wears glasses, a reflection and light that are consistent with the real car can be formed on the lens during the collection process to ensure the authenticity of the collected data. When collecting data, you can choose to collect data in different light intensities such as cloudy days, sunny days, early mornings, noon, and evenings. That is, the collection process needs to cover different times of the day and different weather conditions to ensure data diversity.

[0091] By using natural light and IR fill light for data collection, the lighting conditions during the day and night in the actual driving environment are effectively simulated, and data on glasses reflections and reflections and lighting conditions in most actual driving processes are successfully obtained.

[0092] During the data collection phase of the embodiment of the present application, after the equipment is ready, relevant personnel explain to the tester the required action requirements, and the action requirements are mainly designed according to the driver's behavioral actions during driving, including owl style and lizard style.

[0093] For the owl-style action requirement, the tester keeps his head facing forward, holds the simulated steering wheel tightly with both hands, and is in a generally focused driving posture. He keeps his head steady and does not move, and his eyes look at the sight point set on the display screen. After his eyes are fixed on the sight point, he clicks the sight point with the mouse. For the lizard-style action requirement, the tester keeps his head facing forward, holds the simulated steering wheel tightly with both hands, and is in a generally focused driving posture. He turns his head and his eyes look at the sight point set on the display screen. After his eyes are fixed on the sight point, he clicks the sight point with the mouse. By making the tester perform eye movements according to the above two action requirements, it can be ensured that data with different sight angles are collected for the same tester.

[0094] During the acquisition process of the embodiment of the present application, the display screen supports left and right translation, and the multiple sight point groups that meet the preset distance conditions recorded by the control device include sight point groups corresponding to different sight angles. When the display screen does not move, the first sight point set at different positions of the display screen corresponds to the sight point groups at different sight angles; when the display screen moves, the first sight point set at the same position of the display screen corresponds to the sight point groups at different sight angles.

[0095] That is, Figure 2 As shown, in the initial state, the center point of the simulated steering wheel 10 is located on the central axis of the upper and lower edges of the display screen 20. At this time, the display screen 20 does not move left or right. By setting the first sight point at different positions of the display screen 20, a sight point group corresponding to different sight angles can be determined. After a period of data collection in the current state, the display screen 20 is moved left or right, and the data collection process is repeated. Since the display screen 20 moves left or right, the first sight point set at the same position of the display screen 20 corresponds to a sight point group with different sight angles before and after the display screen 20 moves. By moving the display screen 20 left or right, the diversity of the sight angles of the collected data can be guaranteed.

[0096] In another optional embodiment of the present application, when the first sight point and the second sight point do not meet the preset distance condition, the control device outputs prompt information prompting the tester to adjust the action; wherein the prompt information includes prompt content guiding the tester to move the line of sight according to the prompted action.

[0097] After obtaining the first sight point as the reference sight point and the second sight point as the matched real sight point, the control device identifies whether the distance between the first sight point and the second sight point meets the preset distance condition. If the distance between the first sight point and the second sight point does not meet the preset distance condition, it is determined that the second sight point is not within the error allowable range of the first sight point. At this time, there is no need to control the IR acquisition camera to collect data. The control device outputs prompt information to prompt the tester to adjust the action. The prompt information is output in the form of audio and / or in the form of content display. For example, the control device includes an audio output module, outputs a prompt voice based on the audio output module, and / or the control device controls the display screen to output the prompt information, so that the tester can understand that the real sight point is not within the error allowable range of the reference sight point by browsing the display screen.

[0098] The prompt information may include prompt content for guiding the tester to move the line of sight according to the prompt action, so as to assist the tester in adjusting the action so that the actual line of sight falls within the error tolerance range of the reference line of sight, and then the IR acquisition camera performs image acquisition. For example, when it is determined that the second line of sight is not within the error tolerance range of the first line of sight, an offset angle corresponding to the pixel point deviation between the second line of sight and the first line of sight is output to guide the tester to make the actual line of sight fall within the error tolerance range of the reference line of sight after the action adjustment.

[0099] The following is an example to introduce the process of comparing the reference sight point and the real sight point for processing. Figure 4 As shown:

[0100] Step 401: The control device controls the display screen to display the reference sight point P1.

[0101] Step 402: The tester performs a click operation on the display screen based on the guidance of the reference sight point P1, and controls the device to obtain the actual sight point P2 recorded by the display screen.

[0102] Step 403: The control device determines whether the distance between the reference sight point P1 and the real sight point P2 satisfies a preset distance condition. If the distance between the reference sight point P1 and the real sight point P2 is less than a threshold value (e.g., 10 pixels), it is determined that the preset distance condition is met, and step 404 is executed; otherwise, step 405 is executed.

[0103] Step 404: The control device controls the IR acquisition camera to capture the face of the tester, obtain the face image and record the coordinates corresponding to the reference sight point P1 in the camera coordinate system of the IR acquisition camera.

[0104] Step 405 , the control device outputs prompt information to enable the tester to adjust the action, update the actual sight point P2 , and then execute step 403 .

[0105] During the above implementation process, the distance between the reference sight point P1 and the real sight point P2 is compared to see whether it is less than a preset number of pixels. When it is less than the preset number of pixels, it is determined that the real sight point is within the allowable error range of the reference sight point, and facial image acquisition can be performed and the coordinate information of the reference sight point P1 can be stored. When it is not less than the preset number of pixels, it is determined that the real sight point is not within the allowable error range of the reference sight point, and a prompt message is output to ensure the provision of high-quality sight tracking data.

[0106] The above is the overall implementation scheme of the gaze tracking data acquisition system provided in the embodiment of the present application. By deploying the gaze tracking data acquisition system including a simulated steering wheel, an IR acquisition camera, an IR relay camera, a display screen and a control device, the IR acquisition camera can be used for data acquisition, and the IR relay camera can be used to convert the points on the display screen into the camera coordinate system of the IR acquisition camera. While ensuring the provision of the required gaze tracking data, the dependence on high-precision tools such as eye trackers is reduced, thereby reducing the cost and complexity of data acquisition. By setting a display screen perpendicular to the ground and setting the imaging plane of the IR relay camera parallel to the plane of the display screen, the sight point plane setting can be optimized, the accuracy of the sight point measurement is improved, and the error caused by different viewing angles is reduced. By guiding the test personnel to perform input actions on the display screen based on the reference sight point to record the actual sight point, the stability and reliability of data acquisition can be improved based on an interactive acquisition verification method, and the need for manual intervention can be reduced. By using natural light and IR fill light for data acquisition, the lighting conditions in the actual driving environment are effectively simulated, and the sight tracking data under different lighting environments are successfully obtained. By collecting data at different times, in different weather conditions, and at different sight angles, diverse data can be provided.

[0107] The embodiment of the present application also provides a method for collecting gaze tracking data applied to a control device, wherein the control device is connected to an IR collection camera and a display screen, the IR collection camera is arranged on a simulated steering wheel, and the display screen is located in front of the simulated steering wheel. Figure 5 As shown, the method includes:

[0108] Step 501: Control the display screen to display the set first sight point.

[0109] The display screen is used as a sight point plane for displaying content, and the control device is connected to the display screen to control the display screen to display the set first sight point. For the relevant introduction of the display screen, the simulated steering wheel and the control device, please refer to the description of the system embodiment, which will not be repeated here.

[0110] Step 502: After the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, identify whether the first sight point and the second sight point meet the preset distance condition, and the second sight point is determined based on the guidance of the first sight point. The first sight point and the second sight point form a sight point group.

[0111] The tester located behind the simulated steering wheel faces the display screen. Based on the guidance of the first sight point, the tester performs input actions on the display screen, such as clicking the display screen with a mouse or manually touching the display screen. The display screen records the second sight point corresponding to the input action performed by the tester on the display screen based on the guidance of the first sight point. The second sight point is a real sight point that matches the reference sight point (first sight point), and the first sight point and the second sight point constitute a sight point group. For any sight point group, it includes a reference sight point and a matching real sight point.

[0112] The control device obtains the second sight point recorded by the display screen based on the connection between the control device and the display screen. Since the first sight point is set by the control device, after obtaining the first sight point and the matching second sight point, the control device identifies whether the distance between the first sight point and the second sight point meets the preset distance condition.

[0113] Step 503, in response to the first sight point and the second sight point satisfying the preset distance condition, controlling the IR acquisition camera to perform facial acquisition on the tester, obtaining a face image carrying sight tracking data provided by the IR acquisition camera, and storing the face image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera.

[0114] When the distance between the first sight point and the second sight point meets the preset distance condition, such as when the distance between the first sight point and the second sight point is less than 10 pixels, it is determined that the second sight point is within the error allowable range of the first sight point. At this time, the control device sends a control instruction to the IR acquisition camera, and the IR acquisition camera set on the simulated steering wheel acquires the facial image of the test person to obtain a facial image carrying sight tracking data.

[0115] The control device obtains the face image carrying the sight tracking data provided by the IR acquisition camera. Since the face image taken by the IR acquisition camera is the face image corresponding to the set first sight point, the control device calculates the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera, and stores the face image provided by the IR acquisition camera and the calculated target coordinates as a set of data to realize the recording of the sight tracking data.

[0116] If the distance between the first sight point and the second sight point does not meet the preset distance condition, for example, the distance between the two is not less than 10 pixels, it is determined that the second sight point is not within the error range of the first sight point, and the IR acquisition camera is not required to collect data at this time, and the control device outputs prompt information to prompt the tester to adjust the action, and the prompt information is output in the form of audio and / or in the form of content display. The prompt information may include prompt content to guide the tester to move the line of sight according to the prompt action to assist the tester in adjusting the action. When the actual sight point falls within the error range of the reference sight point, the IR acquisition camera performs image acquisition.

[0117] For the control device, it identifies the sight point groups that meet the preset distance conditions, obtains the corresponding facial image for each sight point group that meets the preset distance conditions, calculates the target coordinates corresponding to the reference sight point and stores them, so as to record the data corresponding to multiple sight point groups that meet the preset distance conditions, thereby ensuring that effective data is provided for the subsequent analysis of visual algorithm data, thereby ensuring the successful application of the visual algorithm.

[0118] The above-mentioned gaze tracking data collection method can reduce the cost and complexity of data collection while performing gaze tracking data collection; by controlling the display screen as the gaze landing point plane to display the reference gaze landing point, guiding the tester to perform input actions on the display screen based on the reference gaze landing point to record the actual gaze landing point, it can improve the stability and reliability of data collection based on an interactive collection and verification method, and reduce the need for manual intervention.

[0119] The following is an introduction to the process of calculating the target coordinates corresponding to the first sight point. An IR relay camera is arranged on the display screen, the imaging plane of the IR relay camera is parallel to the plane of the display screen, and the display screen is perpendicular to the ground; the method also includes:

[0120] Calculate the first world coordinates of the first sight point in the world coordinate system of the display screen based on the pixel coordinates corresponding to the first sight point, the resolution and width and height information of the display screen;

[0121] The target coordinates corresponding to the first sight point are calculated based on the first world coordinates, the second world coordinates of the IR relay camera in the world coordinate system where the display screen is located, and the external parameters of the IR acquisition camera relative to the IR relay camera.

[0122] The control device obtains the pixel coordinates (x i ,y i ) and then calculate the first world coordinates of the first sight point based on the following formula:

[0123] x iw =xi *w / H

[0124] y iw =y i *h / V

[0125] z iw =0

[0126] x i Indicates the pixel horizontal coordinate of the first sight point, y i The pixel vertical coordinate of the first sight point, H represents the horizontal pixel number of the display screen, V represents the vertical pixel number of the display screen, w represents the width information of the display screen, h represents the height information of the display screen, and the first world coordinate is expressed as (x iw ,y iw , z iw ).

[0127] In the pixel x i , when the horizontal pixel number H and width information w of the display are known, based on x i *w / H determines the horizontal coordinate of the display screen in the world coordinate system; in the pixel vertical coordinate y i , when the number of vertical pixels V and height information h of the display are known, based on y i *h / V determines the vertical coordinate of the display screen in the world coordinate system; the coordinate of the Z axis direction in the world coordinate system of the display screen is 0.

[0128] The IR relay camera is used as a relay device to convert points on the display screen into the camera coordinate system of the IR acquisition camera. The external parameters of the IR acquisition camera relative to the IR relay camera are determined based on a calibration process, such as using the IR acquisition camera and the IR relay camera to simultaneously shoot a calibration plate for camera parameter calibration to obtain the external parameters of the IR acquisition camera relative to the IR relay camera. Since the imaging plane of the IR relay camera is parallel to the plane of the display screen, the world coordinate system of the display screen is parallel to the coordinate axis corresponding to the camera coordinate system of the IR relay camera. The height of the IR relay camera can be kept consistent with the height of the IR acquisition camera. Optionally, the IR relay camera is set at the left edge of the display surface of the display screen.

[0129] The world coordinate system corresponding to the display screen is constructed, for example, with the lower left corner of the display screen as the origin of the world coordinate system, and the camera coordinate system corresponding to the IR relay camera is constructed with the camera optical center of the IR relay camera as the origin of the camera coordinate system. When the camera coordinate origin of the IR relay camera is known, the second world coordinate of the IR relay camera in the world coordinate system where the display screen is located is determined based on the distance between the camera coordinate origin of the IR relay camera and the world coordinate origin of the display screen, such as by measuring the distance between the optical center of the IR relay camera and the origin of the world coordinate system with a rangefinder to determine the coordinates of the optical center of the IR relay camera in the world coordinate system where the display screen is located, thereby determining the second world coordinate.

[0130] When the first world coordinate B, the second world coordinate Z, and the rotation parameter R and translation parameter T of the IR acquisition camera relative to the IR transfer camera are determined, the target coordinates of the first sight point are calculated using the following formula:

[0131] M=R·(BZ)+T

[0132] B represents the first world coordinate, Z represents the second world coordinate, (BZ) represents the camera coordinate of the first line of sight in the camera coordinate system where the IR relay camera is located, R represents the rotation parameter of the IR acquisition camera relative to the IR relay camera, and T represents the translation parameter of the IR acquisition camera relative to the IR relay camera.

[0133] In the above formula, the camera coordinates of the first line of sight in the camera coordinate system where the IR relay camera is located are determined based on (BZ), and then the camera coordinates of the first line of sight in the camera coordinate system where the IR relay camera is located are converted into the target coordinates in the camera coordinate system where the IR acquisition camera is located based on the rotation parameter R, translation parameter T and (BZ) of the IR acquisition camera relative to the IR relay camera.

[0134] In the above implementation scheme, the setting of the sight point plane can be optimized, the accuracy of the sight point measurement can be improved, and the error caused by different viewing angles can be reduced; and the IR acquisition camera is used for data acquisition, and the IR transfer camera is used to convert the points on the display screen into the camera coordinate system of the IR acquisition camera. While ensuring the provision of the required sight tracking data, the dependence on high-precision tools such as eye trackers can be reduced, thereby reducing the cost and complexity of data acquisition.

[0135] In an embodiment of the present application, a system consisting of a simulated steering wheel, a display screen, an IR acquisition camera, an IR transfer camera, and a control device is built in a target space, and the target space is an open laboratory with windows that can be illuminated by natural light or a laboratory that simulates such conditions. The simulated steering wheel is placed in front of the window, and the tester sits behind the simulated steering wheel facing the window. The position of the simulated steering wheel and the position of the tester are determined and cannot be moved. A certain space is reserved between the window and the simulated steering wheel, an IR acquisition camera is set on the simulated steering wheel, and a display screen is set between the window and the simulated steering wheel. The display screen is perpendicular to the ground, and the display screen side of the display screen faces the tester's face, ensuring that natural light can pass over the display screen and shine on the face opposite the display screen. The lower edge of the display screen coincides with the lower edge of the simulated steering wheel, and the display screen can move horizontally left and right but cannot move forward and backward in the direction between the simulated steering wheel and the tester. After the display screen is set between the window and the simulated steering wheel, an IR transfer camera with an imaging plane parallel to the plane of the display screen is set on the display screen.

[0136] The control device is connected to an IR fill light arranged around the IR acquisition camera. When the natural light source in the acquisition environment is insufficient, the control device controls the IR fill light to turn on to provide light source for the acquisition environment so that the IR acquisition camera can capture clear images.

[0137] In the daytime collection environment with sufficient sunlight, it mainly relies on natural light; at night or during the day when natural light is insufficient, the control device controls the IR fill light to turn on, and relies on the fill light to provide auxiliary light source to ensure that the IR collection camera can capture clear images under limited lighting conditions to provide high-quality sight tracking data.

[0138] Among them, natural light shines directly onto the tester's face through the window. If the tester wears glasses, a reflection and light that are consistent with the real car can be formed on the lens during the collection process to ensure the authenticity of the collected data. When collecting data, you can choose to collect data in different light intensities such as cloudy days, sunny days, early mornings, noon, and evenings. That is, the collection process needs to cover different times of the day and different weather conditions to ensure data diversity.

[0139] By using natural light and IR fill light for data collection, the lighting conditions during the day and night in the actual driving environment are effectively simulated, and data on glasses reflections and reflections and lighting conditions in most actual driving processes are successfully obtained.

[0140] During the acquisition process of the embodiment of the present application, the display screen supports left and right translation, and the multiple sight point groups that meet the preset distance conditions recorded by the control device include sight point groups corresponding to different sight angles. When the display screen does not move, the first sight point set at different positions of the display screen corresponds to the sight point groups at different sight angles; when the display screen moves, the first sight point set at the same position of the display screen corresponds to the sight point groups at different sight angles.

[0141] That is, in the initial situation, the center point of the simulated steering wheel is located on the central axis of the upper and lower edges of the display screen. At this time, the display screen does not move left and right. By setting the first sight point at different positions of the display screen, the sight point group corresponding to different sight angles can be determined. After a period of data collection in the current state, move the display screen to the left or right, and repeat the data collection process. Since the display screen moves left and right, the first sight point set at the same position of the display screen corresponds to the sight point group of different sight angles before and after the display screen moves. By moving the display screen left and right, the diversity of the sight angles of the collected data can be guaranteed.

[0142] The above is the overall implementation process of the gaze tracking data collection method provided in the embodiment of the present application. For the method embodiment, the parts that are related to the system embodiment and are not elaborated in detail can refer to the description of the system embodiment.

[0143] The embodiment of the present application also provides a sight tracking data acquisition device, which is applied to a control device, wherein the control device is connected to an IR acquisition camera and a display screen, wherein the IR acquisition camera is arranged on a simulated steering wheel, and the display screen is arranged in front of the simulated steering wheel. Figure 6 As shown, the device comprises:

[0144] A first control module 601, used to control the display screen to display a set first sight point;

[0145] The identification module 602 is used to identify whether the first sight point and the second sight point meet a preset distance condition after the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, the second sight point is determined based on the guidance of the first sight point, and the first sight point and the second sight point form a sight point group;

[0146] The processing module 603 is used for controlling the IR acquisition camera to perform facial acquisition on the tester in response to the first sight point and the second sight point satisfying a preset distance condition, obtaining a face image carrying sight tracking data provided by the IR acquisition camera, and storing the face image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera;

[0147] The control device obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

[0148] Optionally, an IR relay camera is arranged on the display screen, an imaging plane of the IR relay camera is parallel to a plane of the display screen, and the display screen is perpendicular to the ground; the device further comprises:

[0149] A first calculation module, configured to calculate a first world coordinate of the first sight point in a world coordinate system where the display screen is located based on the pixel coordinates corresponding to the first sight point, the resolution and width and height information of the display screen;

[0150] The second calculation module is used to calculate the target coordinates corresponding to the first line of sight landing point based on the first world coordinates, the second world coordinates of the IR relay camera in the world coordinate system where the display screen is located, and the external parameters of the IR acquisition camera relative to the IR relay camera.

[0151] Optionally, the first calculation module is further used for:

[0152] The first world coordinate is calculated based on the following formula:

[0153] x iw =x i *w / H

[0154] y iw =y i *h / V

[0155] z iw =0

[0156] Among them, x i The pixel horizontal coordinate of the first sight point, y i represents the pixel vertical coordinate of the first sight point, H represents the horizontal pixel number of the display screen, V represents the vertical pixel number of the display screen, w represents the width information of the display screen, h represents the height information of the display screen, and the first world coordinate is expressed as (x iw ,y iw , z iw ).

[0157] Optionally, the second calculation module is further used for:

[0158] The target coordinates corresponding to the first sight point are calculated based on the following formula:

[0159] M=R·(BZ)+T

[0160] Wherein, B represents the first world coordinate, Z represents the second world coordinate of the camera coordinate origin of the IR relay camera in the world coordinate system where the display screen is located, (BZ) represents the camera coordinate of the first sight point in the camera coordinate system where the IR relay camera is located, and the second world coordinate is determined based on the distance between the camera coordinate origin of the IR relay camera and the world coordinate origin of the display screen;

[0161] R represents a rotation parameter of the IR acquisition camera relative to the IR relay camera, and T represents a translation parameter of the IR acquisition camera relative to the IR relay camera.

[0162] Optionally, natural light passes through the display screen and shines onto the face of the tester, and the control device is connected to an IR fill light arranged at the periphery of the IR acquisition camera; the device comprises:

[0163] The second control module is used for controlling the IR fill light to turn on in response to insufficient natural light in the acquisition environment to provide light for the acquisition environment so that the IR acquisition camera can capture clear images.

[0164] Optionally, the multiple sight point groups include sight point groups corresponding to different sight angles, and the display screen supports left and right translation;

[0165] When the display screen does not move, the first sight points set at different positions of the display screen correspond to sight point groups at different sight angles;

[0166] When the display screen moves, the first sight point set at the same position of the display screen corresponds to a sight point group with different sight angles.

[0167] Optionally, the device further comprises:

[0168] An output module, configured to output prompting the tester to adjust an action in response to the first sight point and the second sight point not satisfying a preset distance condition;

[0169] The prompt information includes prompt content for guiding the tester to move his / her sight according to the prompt action.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] Please refer to Figure 7, which shows a block diagram of a computer device provided in one embodiment of the present application. The computer device can be used to implement the sight tracking data acquisition method provided in the above embodiment. The computer device can be a terminal device or a server, or other device with data processing and storage capabilities. Specifically:

[0172] The computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The computer device 700 also includes a basic input / output system (I / O system) 706 that helps to transfer information between various components in the computer, and a large-capacity storage device 707 for storing an operating system 713, application programs 714, and other program modules 715.

[0173] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 such as a mouse and a keyboard for user inputting information. The display 708 and the input device 709 are connected to the central processing unit 701 through an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include an input / output controller 710 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, a printer, or other types of output devices.

[0174] The mass storage device 707 is connected to the central processing unit 701 through a mass storage controller (not shown) connected to the system bus 705. The mass storage device 707 and its associated computer readable media provide non-volatile storage for the computer device 700. That is, the mass storage device 707 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM drive.

[0175] Without loss of generality, the computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above. The above-mentioned system memory 704 and mass storage device 707 can be collectively referred to as memory.

[0176] According to various embodiments of the present application, the computer device 700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 700 can be connected to the network 712 through the network interface unit 711 connected to the system bus 705, or the network interface unit 711 can be used to connect to other types of networks or remote computer systems (not shown).

[0177] The memory further includes one or more programs, which are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for executing the gaze tracking data collection method.

[0178] In an example embodiment, a computer device is also provided, the computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is configured to be executed by one or more processors to implement the above-mentioned gaze tracking data collection method.

[0179] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set implements the above-mentioned line of sight tracking data collection method when executed by a processor of a computer device.

[0180] Optionally, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0181] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed, it is used to implement the above-mentioned gaze tracking data collection method.

[0182] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0183] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A gaze tracking data acquisition method, applied to a control device, characterized in that: The control device is connected to an IR acquisition camera and a display screen, the IR acquisition camera is arranged on a simulated steering wheel, and the display screen is located in front of the simulated steering wheel. The method includes: Controlling the display screen to display a set first sight point; After the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, identifying whether the first sight point and the second sight point meet a preset distance condition, the second sight point is determined based on the guidance of the first sight point, and the first sight point and the second sight point form a sight point group; In response to the first sight point and the second sight point satisfying a preset distance condition, controlling the IR acquisition camera to perform facial acquisition on the tester, obtaining a facial image carrying sight tracking data provided by the IR acquisition camera, and storing the facial image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera; The control device obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

2. The method according to claim 1, characterized in that The display screen is provided with an IR relay camera, the imaging plane of the IR relay camera is parallel to the plane of the display screen, and the display screen is perpendicular to the ground; the method further comprises: Calculate the first world coordinates of the first sight point in the world coordinate system of the display screen based on the pixel coordinates corresponding to the first sight point, the resolution and width and height information of the display screen; Based on the first world coordinate, the second world coordinate of the IR relay camera in the world coordinate system where the display screen is located, and the external parameters of the IR acquisition camera relative to the IR relay camera, the target coordinate corresponding to the first line of sight landing point is calculated.

3. The method according to claim 2, characterized in that The calculating, based on the pixel coordinates corresponding to the first sight point, the resolution and width and height information of the display screen, the first world coordinates of the first sight point in the world coordinate system where the display screen is located, includes: The first world coordinate is calculated based on the following formula: x iw =x i *w / H y iw =y i *h / V z iw ＝0 Among them, x i The pixel horizontal coordinate of the first sight point, y i represents the pixel vertical coordinate of the first sight point, H represents the horizontal pixel number of the display screen, V represents the vertical pixel number of the display screen, w represents the width information of the display screen, h represents the height information of the display screen, and the first world coordinate is expressed as (x iw ,y iw , z iw ).

4. The method according to claim 3, characterized in that The calculating the target coordinates corresponding to the first sight point based on the first world coordinates, the second world coordinates of the IR relay camera in the world coordinate system where the display screen is located, and the external parameters of the IR acquisition camera relative to the IR relay camera includes: The target coordinates corresponding to the first sight point are calculated based on the following formula: M=R·(BZ)+T Wherein, B represents the first world coordinate, Z represents the second world coordinate of the camera coordinate origin of the IR relay camera in the world coordinate system where the display screen is located, (BZ) represents the camera coordinate of the first sight point in the camera coordinate system where the IR relay camera is located, and the second world coordinate is determined based on the distance between the camera coordinate origin of the IR relay camera and the world coordinate origin of the display screen; R represents a rotation parameter of the IR acquisition camera relative to the IR relay camera, and T represents a translation parameter of the IR acquisition camera relative to the IR relay camera.

5. The method according to any one of claims 1 to 4, characterized in that: Natural light passes through the display screen and shines onto the face of the tester, and the control device is connected to an IR fill light arranged at the periphery of the IR acquisition camera; the method comprises: In response to insufficient natural light in the acquisition environment, the IR fill light is controlled to turn on to provide light for the acquisition environment, so that the IR acquisition camera can capture clear images.

6. The method according to claim 5, characterized in that The multiple sight point groups include sight point groups corresponding to different sight angles, and the display screen supports left and right translation; When the display screen does not move, the first sight points set at different positions of the display screen correspond to sight point groups at different sight angles; When the display screen moves, the first sight point set at the same position of the display screen corresponds to a sight point group with different sight angles.

7. The method according to claim 1, characterized in that Also includes: In response to the first sight point and the second sight point not satisfying a preset distance condition, outputting prompt information prompting the tester to adjust the action; The prompt information includes prompt content for guiding the tester to move his / her sight according to the prompt action.

8. A sight tracking data acquisition device, applied to a control device, characterized in that: The control device is connected to an IR acquisition camera and a display screen, the IR acquisition camera is arranged on a simulated steering wheel, and the display screen is located in front of the simulated steering wheel. The device includes: A first control module, used for controlling the display screen to display a first sight point set; an identification module, configured to identify whether the first sight point and the second sight point meet a preset distance condition after the display screen displays the second sight point based on the input of the tester located behind the simulated steering wheel on the display screen, the second sight point being determined based on the guidance of the first sight point, and the first sight point and the second sight point forming a sight point group; a processing module, configured to control the IR acquisition camera to perform facial acquisition on the tester in response to the first sight point and the second sight point satisfying a preset distance condition, obtain a face image carrying sight tracking data provided by the IR acquisition camera, and store the face image and the target coordinates corresponding to the first sight point in the camera coordinate system of the IR acquisition camera; The control device obtains a plurality of face images according to a plurality of sight point groups satisfying a preset distance condition, and records the target coordinates respectively associated with the plurality of face images.

9. A gaze tracking data acquisition system, characterized in that: include: Simulated steering wheel; A display screen, arranged in front of the simulated steering wheel, for displaying the sight point; An IR acquisition camera, arranged on the simulated steering wheel, for acquiring a facial image of a tester behind the simulated steering wheel; A control device is connected to the display screen and the IR acquisition camera respectively, and is used to execute the sight tracking data acquisition method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images