Sight line data acquisition method, device, equipment and medium

Abstract

The invention discloses a sight line data acquisition method and device, equipment and a medium. The method comprises the following steps: acquiring a first pose, a second pose and a third pose; the first pose is the pose between the second auxiliary calibration plate and the first auxiliary calibration plate, the second pose is the pose between the third auxiliary calibration plate and the second auxiliary calibration plate, and the third pose is the pose between the first auxiliary calibration plate and a camera coordinate system; obtaining coordinates of a sight line end point in a camera coordinate system based on the first pose, the second pose and the third pose to obtain first three-dimensional coordinates; the sight line end point is a fixation point on the fixation plate; obtaining the coordinate of the sight line starting point in the camera coordinate system to obtain a second three-dimensional coordinate; marking a sight line direction corresponding to the sight line end point according to the first three-dimensional coordinate and the second three-dimensional coordinate; the method does not depend on a depth camera, the 3D coordinate of the sight line end point in the camera coordinate system can be calibrated and obtained only through three auxiliary calibration plates, so that the sight line data is obtained, and the specific cost is low.

Description

Technical Field

[0001] This application relates to the field of testing technologies, and particularly to a method, device, equipment and medium for collecting line-of-sight data. Background Art

[0002] A driver monitoring system (DMS, Driver Monitor System) is generally used to monitor whether a driver has dangerous behaviors such as fatigue, distraction, smoking, making phone calls, etc. during driving. Among them, distraction detection is mainly based on inferring the driver's line-of-sight direction from a trained line-of-sight estimation model, and then calculating the fixation area according to the 3D (three-dimensional) position of the human eye, so as to judge whether the line of sight is distracted. Training a line-of-sight estimation model requires a large amount of line-of-sight data (the line-of-sight data refers to the face image captured by the DMS camera and the line-of-sight direction in the DMS camera coordinate system corresponding to this face image, represented by a vector). Most of the collection of line-of-sight data relies on a depth camera to obtain the 3D position of the line-of-sight end point, and this method has a high cost. Summary of the Invention

[0003] In view of the above-mentioned disadvantages of the prior art, this application provides a method, device, equipment and medium for collecting line-of-sight data, which is used to solve at least one defect in the prior art.

[0004] To achieve the above and other purposes, this application provides a method for collecting line-of-sight data, which is applied to a line-of-sight data collection system. The line-of-sight data collection system includes a fixation board with multiple fixation points, a first auxiliary calibration board, a second auxiliary calibration board, and a third auxiliary calibration board. Among them, the fixation board is attached to the third auxiliary calibration board. The method for collecting line-of-sight data includes:

[0005] Obtain a first pose, a second pose, and a third pose; the first pose is the pose between the second auxiliary calibration board and the first auxiliary calibration board, the second pose is the pose between the third auxiliary calibration board and the second auxiliary calibration board, and the third pose is the pose between the first auxiliary calibration board and the camera coordinate system;

[0006] Based on the first pose, the second pose, and the third pose, obtain the coordinates of the line-of-sight end point in the camera coordinate system to obtain a first three-dimensional coordinate; where the line-of-sight end point is the fixation point on the fixation board;

[0007] Obtain the coordinates of the line-of-sight starting point in the camera coordinate system to obtain a second three-dimensional coordinate;

[0008] Mark the line-of-sight direction corresponding to the line-of-sight end point according to the first three-dimensional coordinate and the second three-dimensional coordinate to obtain line-of-sight data.

[0009] In an embodiment of the present application, the first pose includes a first rotation matrix and a first translation matrix. Obtaining the first pose includes:

[0010] Obtaining a first image, the first image including a first auxiliary calibration board and a second auxiliary calibration board;

[0011] Determining a first rotation matrix and a first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the first image and the camera imaging model; wherein, the camera imaging model represents the correlation between an image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0012] In an embodiment of the present application, determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the first image and the camera imaging model includes:

[0013] Performing corner detection on the first image to obtain a first corner point and a second corner point; the first corner point is a corner point in the first auxiliary calibration board, and the second corner point is a corner point in the second auxiliary calibration board;

[0014] Obtaining the pixel coordinates and world coordinates of the first corner point and the pixel coordinates and world coordinates of the second corner point;

[0015] Obtaining a rotation matrix and a translation matrix between the first auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the first corner point and the camera imaging model, and obtaining a rotation matrix and a translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the second corner point and the camera imaging model;

[0016] Determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the rotation matrix and translation matrix between the first auxiliary calibration board and the camera coordinate system and the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system.

[0017] In an embodiment of the present application, the second pose includes a second rotation matrix and a second translation matrix. Obtaining the second pose includes:

[0018] Obtaining a second image, the second image including a second auxiliary calibration board and a third auxiliary calibration board;

[0019] Determining a second rotation matrix and a second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the second image and the camera imaging model; wherein, the camera imaging model represents the correlation between an image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0020] In an embodiment of the present application, the determining the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the second image and the camera imaging model includes:

[0021] Perform corner detection on the second image to obtain a third corner point and a fourth corner point; the third corner point is a corner point in the second auxiliary calibration board, and the fourth corner point is a corner point in the third auxiliary calibration board;

[0022] Obtain the pixel coordinates and world coordinates of the third corner point, and the pixel coordinates and world coordinates of the fourth corner point;

[0023] Obtain the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the third corner point and the camera imaging model, and obtain the rotation matrix and translation matrix between the third auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the fourth corner point and the camera imaging model;

[0024] Determine the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system and the rotation matrix and translation matrix between the third auxiliary calibration board and the camera coordinate system.

[0025] In an embodiment of the present application, the obtaining the first three-dimensional coordinate of the line-of-sight end point in the camera coordinate system based on the first pose, the second pose, and the third pose includes:

[0026] Convert the line-of-sight end point from the gaze board coordinate system to the third auxiliary calibration board coordinate system to obtain the coordinate of the line-of-sight end point in the third auxiliary calibration board coordinate system;

[0027] Convert the coordinate of the line-of-sight end point in the third auxiliary calibration board coordinate system to the camera coordinate system based on the first pose, the second pose, and the third pose to obtain the first three-dimensional coordinate.

[0028] In an embodiment of the present application, the converting the line-of-sight start point to the camera coordinate system to obtain the second three-dimensional coordinate includes:

[0029] Obtain a face image, and perform key point detection on the face image to obtain face key points;

[0030] Obtain the pixel coordinates and world coordinates of the face key points;

[0031] Obtain the rotation matrix and translation matrix from the world coordinate system to the camera coordinate system based on the pixel coordinates, world coordinates of the facial key points, and the camera imaging model; wherein, the camera imaging model represents the correlation relationship among the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0032] Using the eye key points as the starting point of the line of sight, convert the starting point of the line of sight to the camera coordinate system according to the world coordinates of the starting point of the line of sight and the rotation matrix and translation matrix from the world coordinate system to the camera coordinate system, to obtain the second three-dimensional coordinates.

[0033] To achieve the above object and other related objects, the present application provides a line-of-sight data acquisition device, which is applied to a line-of-sight data acquisition system. The line-of-sight data acquisition system includes a fixation board with multiple fixation points, a first auxiliary calibration board, a second auxiliary calibration board, and a third auxiliary calibration board. Among them, the fixation board is attached to the third auxiliary calibration board; the line-of-sight data acquisition device includes:

[0034] An attitude acquisition module, configured to acquire a first attitude, a second attitude, and a third attitude; the first attitude is the attitude between the second auxiliary calibration board and the first auxiliary calibration board, the second attitude is the attitude between the third auxiliary calibration board and the second auxiliary calibration board, and the third attitude is the attitude between the first auxiliary calibration board and the camera coordinate system.

[0035] A first coordinate acquisition module, configured to use the fixation point as the end point of the line of sight, and based on the first attitude, the second attitude, and the third attitude, acquire the coordinates of the end point of the line of sight in the camera coordinate system, to obtain the first three-dimensional coordinates; wherein, the end point of the line of sight is the fixation point on the fixation board.

[0036] A second coordinate acquisition module, configured to acquire the coordinates of the starting point of the line of sight in the camera coordinate system, to obtain the second three-dimensional coordinates.

[0037] A line-of-sight annotation module, configured to annotate the line-of-sight direction corresponding to the end point of the line of sight according to the first three-dimensional coordinates and the second three-dimensional coordinates, to obtain line-of-sight data.

[0038] To achieve the above object and other related objects, the present application provides a line-of-sight data acquisition device, including:

[0039] One or more processors; and

[0040] A memory, configured to store one or more programs. When the one or more programs are executed by the one or more processors, the memory implements the line-of-sight data acquisition method.

[0041] To achieve the above and other related objectives, the present application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause the processors to execute the line-of-sight data acquisition method described above.

[0042] Advantages of the present application:

[0043] A line-of-sight data acquisition method of the present application includes: obtaining a first pose, a second pose, and a third pose; the first pose is the pose between the second auxiliary calibration board and the first auxiliary calibration board, the second pose is the pose between the third auxiliary calibration board and the second auxiliary calibration board, and the third pose is the pose between the first auxiliary calibration board and the camera coordinate system; obtaining the coordinates of the line-of-sight end point in the camera coordinate system based on the first pose, the second pose, and the third pose to obtain the first three-dimensional coordinates; where the line-of-sight end point is the fixation point on the fixation board; obtaining the coordinates of the line-of-sight start point in the camera coordinate system to obtain the second three-dimensional coordinates; marking the line-of-sight direction corresponding to the line-of-sight end point according to the first three-dimensional coordinates and the second three-dimensional coordinates to obtain the line-of-sight data; the present application does not rely on a depth camera and can calibrate the 3D coordinates of the line-of-sight end point in the camera coordinate system only through 3 auxiliary calibration boards, thereby obtaining the line-of-sight data, with the characteristic of low specific cost.

[0044] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In the drawings:

[0046] Figure 1 is a flowchart of the line-of-sight data acquisition method according to an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of the line-of-sight data acquisition system according to an embodiment of the present application;

[0048] Figure 3 is a flowchart of obtaining the first pose according to an embodiment of the present application;

[0049] Figure 4 is a flowchart of determining the first rotation matrix and the first translation matrix according to an embodiment of the present application;

[0050] Figure 5 is a flowchart of obtaining the second pose according to an embodiment of the present application;

[0051] Figure 6 Flow chart for determining the second rotation matrix and the second translation matrix according to an embodiment of the present application;

[0052] Figure 7 Flow chart for obtaining the third pose according to an embodiment of the present application;

[0053] Figure 8 Flow chart for converting the starting point of the line of sight into the camera coordinate system according to an embodiment of the present application;

[0054] Figure 9 Block diagram of the line of sight data acquisition device shown according to an embodiment of the present application

[0055] Figure 10 Shows a schematic structural diagram of a computer system of a memory suitable for implementing the embodiments of the present application. Detailed implementation manners

[0056] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0057] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0058] Although terms such as "first", "second", "A", and "B" can be used herein to describe various elements, these elements should not be limited by these terms and are only used to distinguish one element from another. For example, without departing from the scope of the following technology, the first element can be called the second element, and similarly, the second element can be called the first element. The term "and / or" includes combinations of multiple related items or any item among multiple related items.

[0059] As used herein, unless the context indicates otherwise, the singular form is also intended to include the plural form. It will be understood that the term "comprising" means the presence of the described features, quantities, steps, operations, elements, or combinations thereof, but does not exclude the presence or addition of one or more other features, quantities, steps, operations, elements, components, or combinations thereof.

[0060] Before the detailed description, it is intended to clarify that the division of components in this specification is only based on the main functions of each component. That is, two or more of the following-described components can be combined into one component, or can be divided into two or more components according to more detailed functions. In addition to the main functions of the components, each of the following-described components can also perform some or all of the functions of other components, and some of the main functions of each component can be specifically performed by other components.

[0061] First, each noun will be explained.

[0062] The first auxiliary calibration board coordinate system is a coordinate system established with the physical center or a preset reference point of the first auxiliary calibration board as the origin;

[0063] The second auxiliary calibration board coordinate system is a coordinate system established with the physical center or a preset reference point of the second auxiliary calibration board as the origin;

[0064] The third auxiliary calibration board coordinate system is a coordinate system established with the physical center or a preset reference point of the third auxiliary calibration board as the origin;

[0065] The fixation board coordinate system is a coordinate system established with the physical center or a preset reference point of the fixation board as the origin;

[0066] The world coordinate system is the absolute coordinate of an object in three-dimensional space, expressed as (X w , Y w , Z w ).

[0067] The camera coordinate system is a coordinate system with the camera optical center as the origin and the optical axis as the Z c axis, expressed as (X c , Y c , Z c ).

[0068] Embodiments of the present application respectively propose a method for collecting gaze data, a device for collecting gaze data, a device for collecting gaze data, and a computer-readable storage medium. The following will describe these embodiments in detail.

[0069] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for collecting gaze data according to an embodiment of the present application. This method for collecting gaze data can be applied to a gaze data collection system. As Figure 2 shown, the gaze data collection system includes:

[0070] At least one camera for capturing a face image and an image of an auxiliary calibration board; the at least one camera specifically includes cameras C1, C2, C3, and C4.

[0071] A fixation board 1, with a flat surface, having regularly arranged fixation points, each fixation point numbered in sequence; wherein, when collecting line-of-sight data, the fixation point is used as the end point of the line of sight;

[0072] An acquisition host, used to obtain the images captured by the DMS camera and record the numbers of the end points of the line of sight of the acquisition personnel at the corresponding moments; the acquisition host is connected to the camera, and the operator can click the button on the screen of the acquisition host to control the camera to take pictures and record the end point of the line of sight of the acquisition personnel at this time (on the auxiliary calibration board);

[0073] An auxiliary calibration board (including a first auxiliary calibration board Q1, a second auxiliary calibration board Q2, and a third auxiliary calibration board Q3), used to calibrate the pose between the fixation board coordinate system and the camera coordinate system to obtain the 3D position of the fixation point in the camera coordinate system; wherein, the first auxiliary calibration board and the second auxiliary calibration board are approximately perpendicular to each other, and can appear in the camera's field of view simultaneously when the camera takes pictures; the second auxiliary calibration board and the third auxiliary calibration board are approximately perpendicular to each other, and can appear in the camera's field of view simultaneously when the camera takes pictures, and the third auxiliary calibration board is attached to the fixation board. Two auxiliary calibration boards being approximately perpendicular to each other means the included angle between the two auxiliary calibration boards is within where x is a relatively small angle, for example, x is less than 5.

[0074] In one embodiment, the auxiliary calibration board can adopt a checkerboard pattern with black and white alternating patterns.

[0075] When collecting line-of-sight data, the acquisition personnel's eyes sequentially fixate on each fixation point on the fixation board, the camera C1 captures the face image, and at the same time records the number of the fixation point that the human eye is fixating on at this time. Then, the fixation points on the fixation board are converted into the coordinate system of the third auxiliary calibration board, and then based on the first pose, the second pose, and the third pose, the fixation points in the coordinate system of the third auxiliary calibration board are converted into the camera coordinate system to obtain the first 3D coordinates of the fixation points in the camera coordinate system. At the same time, the second 3D coordinates of the starting point of the line of sight in the camera coordinate system are obtained. Then, the first 3D coordinates are subtracted from the second 3D coordinates to obtain the line-of-sight direction. Since the third auxiliary calibration board is attached to the fixation board, when converting the fixation points on the fixation board into the coordinate system of the third auxiliary calibration board, only the coordinates of the fixation points need to be scaled to convert the fixation points into the coordinate system of the third auxiliary calibration board.

[0076] Similarly, for the face images captured by the cameras C2, C3, and C4, the above steps are repeated to obtain the corresponding line-of-sight directions.

[0077] As Figure 1 shown, the line-of-sight data acquisition method at least includes steps S110 - step S140:

[0078] Step S110, obtain the first pose, the second pose, and the third pose; the first pose is the pose between the second auxiliary calibration board and the first auxiliary calibration board, the second pose is the pose between the third auxiliary calibration board and the second auxiliary calibration board, and the third pose is the pose between the camera coordinate system and the first auxiliary calibration board;

[0079] Use the camera to simultaneously capture the first auxiliary calibration board Q1 and the second auxiliary calibration board Q2, so that the first auxiliary calibration board Q1 and the second auxiliary calibration board Q2 are in the same image, denoted as the first image CaliImageQ1Q2. According to the first image CaliImageQ1Q2 and the internal parameters of the camera, the pose between the second auxiliary calibration board Q2 and the first auxiliary calibration board Q1, that is, the first pose, can be calibrated. Among them, the first pose includes the first rotation matrix and the first translation matrix.

[0080] Use the camera to simultaneously capture the second auxiliary calibration board Q2 and the third auxiliary calibration board Q3, so that the second auxiliary calibration board Q2 and the third auxiliary calibration board Q3 are in the same image, denoted as the second image CaliImageQ2Q3. According to the second image CaliImageQ2Q3 and the camera imaging model, the pose between the third auxiliary calibration board Q3 and the second auxiliary calibration board Q2, that is, the second pose, can be calibrated. Among them, the second pose includes the second rotation matrix and the second translation matrix. Among them, the camera imaging model is established in advance to represent the correlation between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0081] Use the camera to capture the third auxiliary calibration board Q3 to obtain the third image, and then according to the third image and the camera imaging model, the pose between the camera coordinate system and the third auxiliary calibration board Q3, that is, the third pose, can be obtained. The third pose includes the third rotation matrix and the third translation matrix;

[0082] Step S120, based on the first pose, the second pose, and the third pose, obtain the coordinates of the line-of-sight end point in the camera coordinate system to obtain the first three-dimensional coordinates; among them, the line-of-sight end point is the fixation point on the fixation board;

[0083] After determining the first pose, the second pose, and the third pose, record the coordinates of the line-of-sight end point in the third auxiliary calibration board coordinate system, and then through the calibrated first rotation matrix and first translation matrix, second rotation matrix and second translation matrix, third rotation matrix and third translation matrix, convert the coordinates of the line-of-sight end point in the third auxiliary calibration board coordinate system to the camera coordinate system, so as to obtain the 3D position of the line-of-sight end point in the camera coordinate system, that is, the first three-dimensional coordinates.

[0084] Step S130, obtain the coordinates of the line-of-sight start point in the camera coordinate system to obtain the second three-dimensional coordinates;

[0085] Take a face image with a camera. By detecting the key points of the face, according to the 3D face model and the camera imaging model, the coordinates of the key points of the eyes in the camera coordinate system can be calculated, that is, the second 3D coordinates, and the second 3D coordinates are used as the coordinates of the starting point of the line of sight.

[0086] Step S140: Mark the line-of-sight direction corresponding to the end point of the line of sight according to the first 3D coordinates and the second 3D coordinates to obtain line-of-sight data.

[0087] Subtracting the first 3D coordinates from the second 3D coordinates can obtain the line-of-sight direction corresponding to the end point of the line of sight. For example, the second 3D coordinates are (x c1 , y c1 , z c1 ), and the first 3D coordinates are (x ce , y ce , z ce ). Subtracting the first 3D coordinates from the second 3D coordinates can obtain the line-of-sight direction (x c1 - x ce , y c1 - y ce , z c1 - z ce ) of the end point of the line of sight corresponding to the face image captured by the camera.

[0088] This application does not rely on a depth camera. Only through 3 auxiliary calibration plates, the 3D coordinates of the end point of the line of sight in the camera coordinate system can be calibrated, so as to obtain line-of-sight data, with the specific characteristic of low cost.

[0089] Please refer to Figure 3 , Figure 3 which is a flowchart for obtaining the first pose in an embodiment of this application. The first pose includes a first rotation matrix and a first translation matrix. As Figure 3 shown, obtaining the first pose includes:

[0090] Step S310: Obtain a first image, where the first image includes a first auxiliary calibration plate and a second auxiliary calibration plate;

[0091] Take images of the first auxiliary calibration plate and the second auxiliary calibration plate with a camera, where the first auxiliary calibration plate and the second auxiliary calibration plate are in the same image.

[0092] Step S320: Determine the first rotation matrix and the first translation matrix between the second auxiliary calibration plate and the first auxiliary calibration plate according to the first image and the camera imaging model; where the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0093] Please refer to Figure 4 , Figure 4The flowchart for determining the first rotation matrix and the first translation matrix according to an embodiment of the present application. In Figure 4 it, according to the first image and the camera imaging model, determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board includes:

[0094] Step S410, performing corner detection on the first image to obtain the first corner points and the second corner points; the first corner points are the corner points in the first auxiliary calibration board, and the second corner points are the corner points in the second auxiliary calibration board;

[0095] There are many corner points on the first image. By using the corner detection algorithm to perform corner detection on the first image, the first corner points corresponding to the first auxiliary calibration board and the second corner points corresponding to the second auxiliary calibration board can be obtained.

[0096] Step S420, obtaining the pixel coordinates, world coordinates of the first corner points, and the pixel coordinates, world coordinates of the second corner points;

[0097] For the corner points corresponding to the first auxiliary calibration board and the second auxiliary calibration board, each corner point provides two sets of information. One is the image pixel coordinates (u, v), which can be obtained through the image corner detection algorithm. The other is the world coordinates (x w , y w , z w ). By defining the center of the auxiliary calibration board as the origin, the world coordinates of the corner points can be obtained.

[0098] Step S430, obtaining the rotation matrix and the translation matrix between the first auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the first corner points and the camera imaging model, and obtaining the rotation matrix and the translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the second corner points and the camera imaging model;

[0099] The camera imaging model is as shown in Equation (1)

[0100]

[0101] where (u, v) are the image coordinates, (x w , y w , z w ) are the world coordinates, R is the rotation matrix, T is the translation matrix, M is the internal parameter matrix of the camera, which has been calibrated in advance; z c is the depth of the point in the camera coordinate system.

[0102] The internal parameters of a camera refer to the parameters that describe the characteristics of the camera itself and the imaging process, mainly including: focal length, principal point coordinates, distortion parameters, etc. The internal parameters of the camera can be represented by an internal parameter matrix. The calibration of the internal parameters is achieved by using the camera to capture images of multiple auxiliary calibration plates from different angular positions, detecting and extracting the feature points of the calibration objects in each image, and calculating the internal parameters through mathematical algorithms (such as Zhang Zhengyou calibration method) to ensure the accuracy and consistency of imaging.

[0103] In the aforementioned step S420, the image coordinates and world coordinates of the corner points have been obtained. Substituting the image coordinates and world coordinates of the corner points into the camera imaging model, an overdetermined system of equations regarding the rotation matrix R and the translation matrix T can be obtained. Solving this overdetermined system of equations yields the rotation matrix R and the translation matrix T.

[0104] Substituting the corner point coordinates and world coordinates of the first auxiliary calibration plate CornerQ1 into the camera imaging model, an overdetermined system of equations regarding the rotation matrix R and the translation matrix T is obtained. Solving the overdetermined system of equations gives the rotation matrix R between the first auxiliary calibration plate CornerQ1 and the camera coordinate system Ca Q1Ca and the translation matrix T Q1Ca ;

[0105] Similarly, substituting the corner point coordinates and world coordinates of the second auxiliary calibration plate CornerQ2 into the camera imaging model, an overdetermined system of equations regarding the rotation matrix R and the translation matrix T is obtained. Solving the overdetermined system of equations gives the rotation matrix R between the second auxiliary calibration plate CornerQ2 and the camera coordinate system Ca Q2Ca and the translation matrix T Q2Ca .

[0106] After determining the rotation matrix R Q1Ca and the translation matrix T Q1Ca between the first auxiliary calibration plate CornerQ1 and the camera coordinate system Ca, according to Equation (2), the point (x Q1 , y Q1 , z Q1 ) in the coordinate system of the first auxiliary calibration plate Q1 is transformed into the camera coordinate system Ca to obtain the point (x ca , y ca , z ca ).

[0107]

[0108] Similarly, after determining the rotation matrix R Q2Ca and the translation matrix T Q2Ca between the second auxiliary calibration plate CornerQ2 and the camera coordinate system Ca, according to Equation (3), the point (x Q2 , y Q2 , zQ2 ) is transformed into the camera coordinate system Ca to obtain the point (x ca , y ca , z ca ).

[0109]

[0110] Step S440: Determine the first rotation matrix and the first translation matrix between the second auxiliary calibration plate and the first auxiliary calibration plate according to the rotation matrix and the translation matrix between the first auxiliary calibration plate and the camera coordinate system, and the rotation matrix and the translation matrix between the second auxiliary calibration plate and the camera coordinate system.

[0111] Based on Equation (2) and Equation (3), Equation (4) can be obtained. By solving Equation (4), the first rotation matrix R Q2Q1 and the first translation matrix T Q2Q1 between the second auxiliary calibration plate Q2 and the first auxiliary calibration plate Q1 can be obtained.

[0112]

[0113]

[0114] Please refer to Figure 5 , Figure 5 which is the flowchart of obtaining the second pose in an embodiment of the present application. The second pose includes a second rotation matrix and a second translation matrix. As Figure 5 shown, obtaining the second pose includes:

[0115] Step S510: Obtain a second image, where the second image includes a second auxiliary calibration plate and a third auxiliary calibration plate;

[0116] Take images of the second auxiliary calibration plate and the third auxiliary calibration plate through a camera. Among them, the second auxiliary calibration plate and the third auxiliary calibration plate are in the same image.

[0117] Step S520: Determine the second rotation matrix and the second translation matrix between the third auxiliary calibration plate and the second auxiliary calibration plate according to the second image and the camera imaging model; where the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

[0118] Please refer to Figure 6 , Figure 6 which is the flowchart of determining the second rotation matrix and the second translation matrix in an embodiment of the present application. In Figure 6 , determining the second rotation matrix and the second translation matrix between the third auxiliary calibration plate and the second auxiliary calibration plate according to the second image and the camera imaging model includes:

[0119] Step S610: Perform corner detection on the second image to obtain the third corner point and the fourth corner point. The third corner point is the corner point on the second auxiliary calibration board, and the fourth corner point is the corner point on the third auxiliary calibration board.

[0120] There are many corner points on the second image. By using the corner detection algorithm to perform corner detection on the second image, the third corner point corresponding to the second auxiliary calibration board and the fourth corner point corresponding to the third auxiliary calibration board can be obtained.

[0121] Step S620: Obtain the pixel coordinates and world coordinates of the third corner point, and the pixel coordinates and world coordinates of the fourth corner point.

[0122] For the corner points on the second auxiliary calibration board and the third auxiliary calibration board, each corner point provides two sets of information. One is the image pixel coordinates (u, v), which can be obtained through the image corner detection algorithm. The other is the world coordinates (x w , y w , z w ). By defining the center of the auxiliary calibration board as the origin, the world coordinates of the corner point can be obtained.

[0123] Step S630: Obtain the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the third corner point and the camera imaging model, and obtain the rotation matrix and translation matrix between the third auxiliary calibration board and the camera coordinate system according to the pixel coordinates, world coordinates of the fourth corner point and the camera imaging model.

[0124] In the previous step S620, the image coordinates and world coordinates of the corner point have been obtained. Substituting the image coordinates and world coordinates of the corner point into the camera imaging model, an overdetermined system of equations about the rotation matrix R and the translation matrix T can be obtained. Solving this overdetermined system of equations can obtain the rotation matrix R and the translation matrix T.

[0125] Substitute the corner point coordinates and world coordinates of the second auxiliary calibration board CornerQ2 into the camera imaging model to obtain an overdetermined system of equations about the rotation matrix R and the translation matrix T. Solving the overdetermined system of equations can obtain the rotation matrix R Q2Ca and the translation matrix T Q2Ca ;

[0126] Similarly, substitute the corner point coordinates and world coordinates of the third auxiliary calibration board CornerQ3 into the camera imaging model to obtain an overdetermined system of equations about the rotation matrix R and the translation matrix T. Solving the overdetermined system of equations can obtain the rotation matrix R Q3Ca and the translation matrix T Q3Ca .

[0127] When determining the rotation matrix R Q2Ca and the translation matrix T Q2Ca between the second auxiliary calibration board CornerQ2 and the camera coordinate system Ca, according to Equation (5), the points (x Q2 , y Q2 , z Q2 ) in the second auxiliary calibration board coordinate system are transformed into the camera coordinate system Ca to obtain the points (x 2ca , y 2ca , z 2ca ).

[0128]

[0129] Similarly, when determining the rotation matrix R Q3Ca and the translation matrix T Q3Ca between the third auxiliary calibration board CornerQ3 and the camera coordinate system Ca, according to Equation (6), the points (x Q3 , y Q3 , z Q3 ) in the third auxiliary calibration board coordinate system are transformed into the camera coordinate system Ca to obtain the points (x 3ca , y 3ca , z 3ca ).

[0130]

[0131] Step S640: Determine the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the rotation matrix and the translation matrix between the second auxiliary calibration board and the camera coordinate system and the rotation matrix and the translation matrix between the third auxiliary calibration board and the camera coordinate system.

[0132] Based on Equation (5) and Equation (6), Equation (7) can be obtained. By solving Equation (7), the second rotation matrix R Q3Q2 and the second translation matrix T Q3Q2 between the third auxiliary calibration board Q3 and the second auxiliary calibration board Q2 can be obtained.

[0133]

[0134]

[0135] Please refer to Figure 7 , Figure 7 which is the flowchart of obtaining the third pose in an embodiment of the present application. The third pose includes a third rotation matrix and a third translation matrix. As Figure 7 shown, obtaining the third pose includes:

[0136] Step S710: Obtain the image of the first auxiliary calibration board;

[0137] The first auxiliary calibration board is photographed by a camera to obtain an image of the first auxiliary calibration board. After the camera finishes photographing, the relative positions of the camera and the first auxiliary calibration board remain unchanged.

[0138] Step S720: Detect corner points of the image of the first auxiliary calibration board to obtain the fifth corner point, and acquire the pixel coordinates and world coordinates of the fifth corner point.

[0139] There are many corner points on the image of the first auxiliary calibration board. By using a corner point detection algorithm to detect the corner points of the image of the first auxiliary calibration board, the fifth corner point corresponding to the first auxiliary calibration board can be obtained. For the corner points corresponding to the first auxiliary calibration board, each corner point provides two sets of information. One is the image pixel coordinates (u, v), which can be obtained through the image corner point detection algorithm. The other is the world coordinates (x w , y w , z w ). By defining the center of the auxiliary calibration board as the origin, the world coordinates of the corner points can be obtained.

[0140] Step S730: According to the pixel coordinates, world coordinates of the fifth corner point and the camera imaging model, obtain the third rotation matrix and the third translation matrix between the first auxiliary calibration board and the camera coordinate system.

[0141] In the foregoing step S420, the image coordinates and world coordinates of the corner points have been obtained. Substituting the image coordinates and world coordinates of the corner points into the camera imaging model, an overdetermined system of equations about the rotation matrix R and the translation matrix T can be obtained. Solving this overdetermined system of equations can obtain the rotation matrix R and the translation matrix T.

[0142] Substitute the corner point coordinates and world coordinates of the first auxiliary calibration board CornerQ1 into the camera imaging model to obtain an overdetermined system of equations about the rotation matrix R and the translation matrix T. Solving the overdetermined system of equations can obtain the rotation matrix R Q1C1 and the translation matrix T Q1C1 .

[0143] In one implementation, obtaining the first three-dimensional coordinates of the line-of-sight end point in the camera coordinate system based on the first pose, the second pose, and the third pose includes: converting the line-of-sight end point from the fixation board coordinate system to the third auxiliary calibration board coordinate system to obtain the coordinates of the line-of-sight end point in the third auxiliary calibration board coordinate system; based on the first pose, the second pose, and the third pose, converting the coordinates of the line-of-sight end point in the third auxiliary calibration board coordinate system to the camera coordinate system to obtain the first three-dimensional coordinates.

[0144] Since the third auxiliary calibration board is attached to the fixation board, when converting the fixation points in the fixation board to the coordinate system of the third auxiliary calibration board, the coordinates of the fixation points can be scaled to convert the fixation points to the coordinate system of the third auxiliary calibration board. Denote the point (x Q3 , y Q3 , z Q3 ) in the coordinate system of the third auxiliary calibration board, and based on Equation (8), convert it to the camera coordinate system to obtain the first three-dimensional coordinates, denoted as (x c1 , y c1 , z c1 ).

[0145]

[0146] Please refer to Figure 8 , Figure 8 , which is the flowchart of converting the starting point of the line of sight to the camera coordinate system in an embodiment of the present application. In Figure 8 , converting the starting point of the line of sight to the camera coordinate system to obtain the second three-dimensional coordinates includes:

[0147] Step S810: Obtain a face image and perform key point detection on the face image to obtain face key points;

[0148] Take a face image through the camera, and then input the face image into a face key point detection algorithm (such as a face key point detection algorithm based on yolov8) to detect the image coordinates of the face key points. According to the general face 3D face model, the world coordinates (x f , y f , z f ) corresponding to each face key point can be obtained.

[0149] Step S820: Obtain the pixel coordinates and world coordinates of the face key points;

[0150] Step S830: Obtain the rotation matrix and translation matrix from the world coordinate system to the camera coordinate system according to the image coordinates, world coordinates of the face key points and the camera imaging model; wherein, the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix;

[0151] Substitute the image coordinates and world coordinates of all face key points into the camera model to obtain an overdetermined system of equations about the rotation matrix and translation matrix. Solve the overdetermined system of equations to obtain the rotation matrix R FC1 and the translation matrix T FC1 .

[0152] Step S840: Use the eye key points as the starting point of the line of sight. According to the world coordinates of the starting point of the line of sight and the rotation matrix and translation matrix from the world coordinate system to the camera coordinate system, convert the starting point of the line of sight to the camera coordinate system to obtain the second three-dimensional coordinates.

[0153] Use the eye key points as the starting point of the line of sight. According to Equation (9), convert the starting point of the line of sight to the camera coordinate system to obtain the second three-dimensional coordinates, denoted as (x ce , y ce , z ce ).

[0154]

[0155] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0156] Figure 9 is a block diagram of a line-of-sight data acquisition device according to an embodiment of the present application. The line-of-sight data acquisition device is applied to a line-of-sight data acquisition system, and the line-of-sight data acquisition system includes a first auxiliary calibration board, a second auxiliary calibration board, and a third auxiliary calibration board. As Figure 9 shown, a line-of-sight data acquisition device includes:

[0157] An attitude acquisition module 910, configured to acquire a first attitude, a second attitude, and a third attitude; the first attitude is the attitude between the second auxiliary calibration board and the first auxiliary calibration board, the second attitude is the attitude between the third auxiliary calibration board and the second auxiliary calibration board, and the third attitude is the attitude between the first auxiliary calibration board and the camera coordinate system;

[0158] A first coordinate acquisition module 920, configured to acquire the coordinates of the end point of the line of sight in the camera coordinate system based on the first attitude, the second attitude, and the third attitude to obtain the first three-dimensional coordinates; wherein, the end point of the line of sight is the fixation point on the fixation board;

[0159] A second coordinate acquisition module 930, configured to acquire the coordinates of the starting point of the line of sight in the camera coordinate system to obtain the second three-dimensional coordinates;

[0160] A line-of-sight annotation module 940, configured to annotate the line-of-sight direction corresponding to the end point of the line of sight according to the first three-dimensional coordinates and the second three-dimensional coordinates to obtain line-of-sight data.

[0161] It should be noted that the line-of-sight data acquisition device provided in the above embodiment and the line-of-sight data acquisition method provided in the above embodiment belong to the same concept. The specific manners in which each module and unit perform operations have been described in detail in the method embodiment, and will not be elaborated herein. In practical applications, the line-of-sight data acquisition device provided in the above embodiment may, as needed, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited herein either.

[0162] An embodiment of the present application further provides a line-of-sight data acquisition device, including: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the memory to implement the line-of-sight data acquisition method in the above embodiment.

[0163] An embodiment of the present application further provides one or more machine-readable media, on which instructions are stored, which, when executed by one or more processors, cause the processors to execute the line-of-sight data acquisition method in the above embodiment.

[0164] Figure 10 The structural schematic diagram of a computer system suitable for implementing the memory in the embodiment of the present invention is shown. It should be noted that Figure 10 The computer system of the memory shown is only an example and should not bring any limitation to the functions and usage scope of the embodiment of the present invention.

[0165] As Figure 10 shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part into the random access memory (RAM) 1003, such as executing the method in the above embodiment. In the RAM, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other through a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0166] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as required. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as required so that a computer program read therefrom is installed into the storage section 1008 as required.

[0167] Specifically, according to an embodiment of the present invention, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for performing the foregoing implemented line-of-sight data acquisition method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section, and / or installed from the removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present invention are executed.

[0168] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM) 1003, a read-only memory (ROM) 1002, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. Among them, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0170] The units involved in the embodiments of the present invention can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, constitute a limitation on the units themselves.

[0171] Another aspect of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor of the computer, the computer is caused to execute the line-of-sight data acquisition method as described above. The computer-readable storage medium may be included in the memory described in the above embodiments, or may exist alone without being assembled into the memory.

[0172] Another aspect of the present invention also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the line-of-sight data acquisition method provided in the above various embodiments.

[0173] The above embodiments are only used to exemplarily illustrate the principles and effects of the present application, rather than to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present application should still be covered by the claims of the present application.

Claims

1. A method for collecting line-of-sight data, which is applied to a line-of-sight data collection system, and is characterized in that, The line-of-sight data acquisition system includes a fixation board with multiple fixation points, a first auxiliary calibration board, a second auxiliary calibration board, and a third auxiliary calibration board. Among them, the fixation board is attached to the third auxiliary calibration board. The line-of-sight data acquisition method includes: Obtaining a first pose, a second pose, and a third pose; the first pose is the pose between the second auxiliary calibration board and the first auxiliary calibration board, the second pose is the pose between the third auxiliary calibration board and the second auxiliary calibration board, and the third pose is the pose between the first auxiliary calibration board and the camera coordinate system; Based on the first pose, the second pose, and the third pose, obtaining the coordinates of the line-of-sight end point in the camera coordinate system to obtain a first three-dimensional coordinate; where the line-of-sight end point is the fixation point on the fixation board; Obtaining the coordinates of the line-of-sight starting point in the camera coordinate system to obtain a second three-dimensional coordinate; Marking the line-of-sight direction corresponding to the line-of-sight end point according to the first three-dimensional coordinate and the second three-dimensional coordinate to obtain line-of-sight data.

2. The method for collecting line-of-sight data according to claim 1, wherein The first pose includes a first rotation matrix and a first translation matrix. Obtaining the first pose includes: Obtaining a first image, which includes the first auxiliary calibration board and the second auxiliary calibration board; Determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the first image and the camera imaging model; where the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

3. The method for collecting line-of-sight data according to claim 2, wherein The determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the first image and the camera imaging model includes: Performing corner detection on the first image to obtain a first corner point and a second corner point; the first corner point is the corner point in the first auxiliary calibration board, and the second corner point is the corner point in the second auxiliary calibration board; Obtaining the pixel coordinates and world coordinates of the first corner point and the pixel coordinates and world coordinates of the second corner point; Obtaining the rotation matrix and translation matrix between the first auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the first corner point and the camera imaging model, and obtaining the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the second corner point and the camera imaging model; Determining the first rotation matrix and the first translation matrix between the second auxiliary calibration board and the first auxiliary calibration board according to the rotation matrix and translation matrix between the first auxiliary calibration board and the camera coordinate system and the rotation matrix and translation matrix between the second auxiliary calibration board and the camera coordinate system.

4. The line-of-sight data acquisition method according to claim 1, wherein The second pose includes a second rotation matrix and a second translation matrix. Obtaining the second pose includes: Obtaining a second image, which includes the second auxiliary calibration board and the third auxiliary calibration board; Determining the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the second image and the camera imaging model; where the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix.

5. The method for collecting line-of-sight data according to claim 4, wherein Determining the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the second image and the camera imaging model includes: Performing corner detection on the second image to obtain a third corner point and a fourth corner point; the third corner point is a corner point in the second auxiliary calibration board, and the fourth corner point is a corner point in the third auxiliary calibration board; Obtaining the pixel coordinates and world coordinates of the third corner point, and the pixel coordinates and world coordinates of the fourth corner point; Obtaining the rotation matrix and the translation matrix between the second auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the third corner point and the camera imaging model, and obtaining the rotation matrix and the translation matrix between the third auxiliary calibration board and the camera coordinate system according to the pixel coordinates and world coordinates of the fourth corner point and the camera imaging model; Determining the second rotation matrix and the second translation matrix between the third auxiliary calibration board and the second auxiliary calibration board according to the rotation matrix and the translation matrix between the second auxiliary calibration board and the camera coordinate system and the rotation matrix and the translation matrix between the third auxiliary calibration board and the camera coordinate system.

6. The method for collecting line-of-sight data according to claim 1, wherein Obtaining the first three-dimensional coordinate of the line-of-sight end point in the camera coordinate system based on the first pose, the second pose, and the third pose includes: Converting the line-of-sight end point from the fixation board coordinate system to the third auxiliary calibration board coordinate system to obtain the coordinate of the line-of-sight end point in the third auxiliary calibration board coordinate system; Converting the coordinate of the line-of-sight end point in the third auxiliary calibration board coordinate system to the camera coordinate system based on the first pose, the second pose, and the third pose to obtain the first three-dimensional coordinate.

7. The method for collecting line-of-sight data according to claim 1, wherein Converting the line-of-sight starting point to the camera coordinate system to obtain the second three-dimensional coordinate includes: Obtaining a face image and performing key point detection on the face image to obtain face key points; Obtaining the pixel coordinates and world coordinates of the face key points; Obtaining the rotation matrix and the translation matrix from the world coordinate system to the camera coordinate system according to the pixel coordinates and world coordinates of the face key points and the camera imaging model; wherein, the camera imaging model represents the correlation relationship between the image, the internal parameter matrix of the camera, the translation matrix, and the rotation matrix; Taking the eye key point as the line-of-sight starting point, and converting the line-of-sight starting point to the camera coordinate system according to the world coordinate of the line-of-sight starting point and the rotation matrix and the translation matrix from the world coordinate system to the camera coordinate system to obtain the second three-dimensional coordinate.

8. A line-of-sight data acquisition device, which is applied to a line-of-sight data acquisition system, is characterized in that The line-of-sight data acquisition system includes a fixation board with multiple fixation points, a first auxiliary calibration board, a second auxiliary calibration board, and a third auxiliary calibration board, wherein the fixation board is attached to the third auxiliary calibration board; the line-of-sight data acquisition device includes: An attitude acquisition module for acquiring a first pose, a second pose, and a third pose; the first pose is the pose between the second auxiliary calibration board and the first auxiliary calibration board, the second pose is the pose between the third auxiliary calibration board and the second auxiliary calibration board, and the third pose is the pose between the first auxiliary calibration board and the camera coordinate system; A first coordinate acquisition module, configured to use the fixation point as the end point of the line of sight, and acquire the coordinates of the end point of the line of sight in the camera coordinate system based on the first pose, the second pose, and the third pose, so as to obtain a first three-dimensional coordinate; wherein, the end point of the line of sight is the fixation point on the fixation board. A second coordinate acquisition module, configured to acquire the coordinates of the starting point of the line of sight in the camera coordinate system, so as to obtain a second three-dimensional coordinate. A line of sight annotation module, configured to annotate the line of sight direction corresponding to the end point of the line of sight according to the first three-dimensional coordinate and the second three-dimensional coordinate, so as to obtain line of sight data.

9. A line-of-sight data acquisition device, characterized in that, Comprising: One or more processors; And A memory, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the memory implements the line of sight data acquisition method according to any one of claims 1-7.

10. A machine-readable medium, characterized in that, Instructions are stored thereon, and when executed by one or more processors, cause the processors to execute the line of sight data acquisition method according to any one of claims 1-7.

Images