Sight line data acquisition method and device, storage medium and vehicle

Abstract

The invention provides a sight line data acquisition method and device, a storage medium and a vehicle, and belongs to the technical field of data processing. According to the embodiment of the invention, the fixation plate is connected with the vehicle through the plurality of fixed points, and the first position information of the plurality of fixation points on the fixation plate in the vehicle body coordinate system can be accurately calculated through the three-dimensional model of the vehicle. Determining fixation point position information of the target fixation point location under the camera coordinates based on first position information of the target fixation point location under the vehicle body coordinate system and a pre-calibrated first conversion relation between the vehicle body coordinate system and the camera coordinates, and determining the target fixation point location under the camera coordinates based on the eye position information and the fixation point position information. And obtaining the sight line data of the face image under the camera coordinates. Therefore, on the premise of not depending on a depth camera, high-precision and low-cost acquisition of the sight line data can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly to a method and device for collecting line-of-sight data, a storage medium, and a vehicle. Background Art

[0002] To ensure the driving safety of drivers, the current vehicle DMS (Driver Monitoring System) is usually configured with a distraction detection algorithm. The distraction detection algorithm can infer the line-of-sight direction of the driver based on a trained line-of-sight estimation model, and then calculate the fixation area according to the position of the human eye, so as to determine whether the driver is distracted.

[0003] The training of the line-of-sight estimation model depends on a large amount of line-of-sight data, and most of the existing line-of-sight data acquisition systems rely on high-cost depth cameras to obtain the position information of the fixation points, resulting in a high cost of collecting line-of-sight data. Summary of the Invention

[0004] The present application provides a method and device for collecting line-of-sight data, a storage medium, and a vehicle to solve the problem of high cost of collecting line-of-sight data at present.

[0005] To solve the above problems, the present application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for collecting line-of-sight data, the method including:

[0007] Based on the three-dimensional model of the vehicle and a plurality of fixed points connected to the vehicle on the fixation board, determine the first position information of a plurality of fixation points on the fixation board in the vehicle body coordinate system; wherein, the plurality of fixation points are arranged on the fixation board at a preset interval;

[0008] Obtain a face image of a target person collected by an image acquisition device, and determine the eye position information of the eye key points in the face image in the camera coordinate system of the image acquisition device; the target person is in a state of looking at a target fixation point on the fixation board;

[0009] Based on the first position information of the target fixation point in the vehicle body coordinate system and a first conversion relationship pre-calibrated between the vehicle body coordinate system and the camera coordinate system, determine the fixation point position information of the target fixation point in the camera coordinate system;

[0010] Based on the eye position information and the fixation point position information, obtain the line-of-sight data of the face image in the camera coordinate system.

[0011] In an embodiment of the present application, based on the three-dimensional model of the vehicle and multiple fixed points connected to the vehicle on the eye plate, determining the first position information of multiple eye positions on the eye plate in the vehicle body coordinate system includes:

[0012] Based on the three-dimensional model, determining the second position information of the multiple fixed points in the vehicle body coordinate system;

[0013] Based on the second position information of the multiple fixed points in the vehicle body coordinate system and the third position information of the multiple fixed points in the eye plate coordinate system, determining the second conversion relationship between the eye plate coordinate system and the vehicle body coordinate system;

[0014] Based on the third position information of the multiple fixed points in the eye plate coordinate system and the preset interval, determining the fourth position information of the multiple eye positions in the eye plate coordinate system;

[0015] Based on the second conversion relationship and the fourth position information of the multiple eye positions in the eye plate coordinate system, determining the first position information of the multiple eye positions in the vehicle body coordinate system.

[0016] In an embodiment of the present application, the method further includes:

[0017] Obtaining a number mapping relationship; the number mapping relationship is used to represent the correspondence between different numbers and the first position information of different eye positions in the vehicle body coordinate system;

[0018] In response to the target number input by the user for the target eye position, based on the target number and the number mapping relationship, determining the first position information of the target eye position in the vehicle body coordinate system.

[0019] In an embodiment of the present application, determining the eye position information of the eye key points in the face image in the camera coordinate of the image acquisition device includes:

[0020] Determining the fifth position information of the eye key points in the face image in the world coordinate system;

[0021] Based on the fifth position information and the third conversion relationship between the world coordinate system and the camera coordinate calibrated in advance, determining the eye position information of the eye key points in the camera coordinate.

[0022] In an embodiment of the present application, determining the fifth position information of the eye key points in the face image in the world coordinate system includes:

[0023] Determining the image coordinates of the eye key points in the face image;

[0024] Determine the fifth position information of the eye key points in the world coordinate system based on a preset three-dimensional face model and the image coordinates of the eye key points in the face image.

[0025] In an embodiment of the present application, the first conversion relationship is calibrated in the following manner:

[0026] Obtain an in-vehicle image collected by the image acquisition device; the in-vehicle image includes a plurality of marked points arranged inside the vehicle;

[0027] Based on the three-dimensional model, determine the marked point coordinates of the plurality of marked points in the vehicle body coordinate system;

[0028] Based on the camera internal parameter matrix of the image acquisition device, the image coordinates of the plurality of marked points in the in-vehicle image, and the marked point coordinates of the plurality of marked points in the vehicle body coordinate system, determine the first conversion relationship between the vehicle body coordinate system and the camera coordinate system.

[0029] In an embodiment of the present application, the third conversion relationship is calibrated in the following manner:

[0030] Obtain a sample face image collected by the image acquisition device, and determine the image coordinates of multiple face key points in the sample face image;

[0031] Based on a preset three-dimensional face model and the image coordinates of the multiple face key points, determine the sixth position information of the multiple face key points in the world coordinate system;

[0032] Based on the camera internal parameter matrix of the image acquisition device, the image coordinates of the multiple face key points in the sample face image, and the sixth position information of the multiple face key points in the world coordinate system, determine the third conversion relationship between the world coordinate system and the camera coordinate system.

[0033] In a second aspect, based on the same inventive concept, an embodiment of the present application provides a gaze data acquisition device, the device includes:

[0034] A first position determination module, configured to determine the first position information of a plurality of gaze points on the gaze board in the vehicle body coordinate system based on a three-dimensional model of the vehicle and a plurality of fixed points connected to the vehicle on the gaze board; wherein, the plurality of gaze points are arranged on the gaze board at a preset interval;

[0035] A second position determination module, configured to obtain a face image of a target person collected by an image acquisition device, and determine eye position information of eye key points in the face image in the camera coordinates of the image acquisition device; the target person is in a state of looking at a target gaze point on the gaze board;

[0036] A third position determination module, configured to determine gaze point position information of the target gaze point in the camera coordinates based on the first position information of the target gaze point in the vehicle body coordinate system and a first conversion relationship between the vehicle body coordinate system and the camera coordinates calibrated in advance;

[0037] A line-of-sight data determination module, configured to obtain line-of-sight data of the face image in the camera coordinates based on the eye position information and the gaze point position information.

[0038] In an embodiment of the present application, the first position determination module includes:

[0039] A first position determination sub-module, configured to determine second position information of the plurality of fixed points in the vehicle body coordinate system based on the three-dimensional model;

[0040] A first conversion relationship determination sub-module, configured to determine a second conversion relationship between the gaze board coordinate system and the vehicle body coordinate system based on the second position information of the plurality of fixed points in the vehicle body coordinate system and third position information of the plurality of fixed points in the gaze board coordinate system;

[0041] A second position determination sub-module, configured to determine fourth position information of the plurality of gaze points in the gaze board coordinate system based on the third position information of the plurality of fixed points in the gaze board coordinate system and the preset interval;

[0042] A third position determination sub-module, configured to determine first position information of the plurality of gaze points in the vehicle body coordinate system based on the second conversion relationship and the fourth position information of the plurality of gaze points in the gaze board coordinate system.

[0043] In an embodiment of the present application, the line-of-sight data acquisition device further includes:

[0044] A mapping relationship acquisition module, configured to acquire a number mapping relationship; the number mapping relationship is used to represent the corresponding relationship between different numbers and the first position information of different gaze points in the vehicle body coordinate system;

[0045] An information determination module, configured to, in response to a target number input by a user for the target gaze point, determine the first position information of the target gaze point in the vehicle body coordinate system based on the target number and the number mapping relationship.

[0046] In an embodiment of the present application, the second position determination module includes:

[0047] A fourth position determination sub-module, configured to determine fifth position information of the eye key points in the face image in the world coordinate system;

[0048] An eye position determination sub-module, configured to determine eye position information of the eye key points in the camera coordinate system based on the fifth position information and a third conversion relationship between the pre-calibrated world coordinate system and the camera coordinate.

[0049] In an embodiment of the present application, the fourth position determination sub-module includes:

[0050] An image coordinate determination unit, configured to determine the image coordinates of the eye key points in the face image;

[0051] An eye position determination unit, configured to determine fifth position information of the eye key points in the world coordinate system based on a preset three-dimensional face model and the image coordinates of the eye key points in the face image.

[0052] In an embodiment of the present application, the line-of-sight data acquisition device further includes:

[0053] An in-vehicle image acquisition module, configured to acquire an in-vehicle image acquired by the image acquisition device; the in-vehicle image includes a plurality of marked points arranged inside the vehicle;

[0054] A marked point coordinate determination module, configured to determine the marked point coordinates of the plurality of marked points in the vehicle body coordinate system based on the three-dimensional model;

[0055] A first calibration module, configured to determine a first conversion relationship between the vehicle body coordinate system and the camera coordinate based on the camera internal parameter matrix of the image acquisition device, the image coordinates of the plurality of marked points in the in-vehicle image, and the marked point coordinates of the plurality of marked points in the vehicle body coordinate system.

[0056] In an embodiment of the present application, the line-of-sight data acquisition device further includes:

[0057] A face image acquisition module, configured to acquire a sample face image acquired by the image acquisition device and determine the image coordinates of multiple face key points in the sample face image;

[0058] A key point position determination module, configured to determine sixth position information of multiple face key points in the world coordinate system based on a preset three-dimensional face model and the image coordinates of the multiple face key points;

[0059] A second calibration module, configured to determine a third conversion relationship between the world coordinate system and the camera coordinate system based on the camera internal parameter matrix of the image acquisition device, the image coordinates of multiple face key points in the sample face image, and the sixth position information of the multiple face key points in the world coordinate system.

[0060] In a third aspect, based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which an executable program is stored, and when the executable program is executed by a processor, the gaze data acquisition method proposed in the first aspect of the present application is implemented.

[0061] In a fourth aspect, based on the same inventive concept, an embodiment of the present application provides a vehicle, including:

[0062] A memory, configured to store an executable program;

[0063] A processor;

[0064] When the executable program is executed by the processor, the gaze data acquisition method proposed in the first aspect of the present application is implemented.

[0065] Compared with the prior art, the present application has the following advantages:

[0066] A gaze data acquisition method provided by an embodiment of the present application first determines the first position information of multiple gaze points on a gaze board in a vehicle body coordinate system based on a three-dimensional model of the vehicle and multiple fixed points connected to the vehicle on the gaze board; then acquires a face image of a target person collected by an image acquisition device, and determines the eye position information of the eye key points in the face image in the camera coordinate of the image acquisition device; subsequently, based on the first position information of the target gaze point in the vehicle body coordinate system and the first conversion relationship between the pre-calibrated vehicle body coordinate system and the camera coordinate, determines the gaze point position information of the target gaze point in the camera coordinate; finally, based on the eye position information and the gaze point position information, obtains the gaze data of the face image in the camera coordinate. By connecting the gaze board to the vehicle through multiple fixed points, the embodiment of the present application can use the three-dimensional model of the vehicle to accurately calculate the first position information of multiple gaze points on the gaze board in the vehicle body coordinate system. In this way, it is possible to achieve high-precision and low-cost acquisition of gaze data without relying on a depth camera. Description of the Drawings

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0068] Figure 1 It is a flowchart of the steps of a method for collecting line-of-sight data in an embodiment of the present application;

[0069] Figure 2 It is a schematic diagram of the functional modules of a line-of-sight data collection device in an embodiment of the present application;

[0070] Figure 3 It is a schematic diagram of the structure of a vehicle in an embodiment of the present application. Detailed implementation manners

[0071] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be completely conveyed to those skilled in the art.

[0072] It should be noted that the line-of-sight data used for training the line-of-sight estimation model usually appears as a vector from the starting point of the line of sight to the ending point of the line of sight. Therefore, the line-of-sight data requires the three-dimensional coordinates of the starting point and the ending point of the line of sight. Therefore, the method for collecting line-of-sight labels in the related art usually requires the support of complex devices. For example, a depth camera is used to obtain the depth information of the objects in the scene, and then the position information of the driver's fixation point is determined. However, the depth camera requires high-precision sensors, complex optical designs, and high-performance algorithm support. Therefore, this method has the problem of high cost for collecting line-of-sight data.

[0073] Aiming at the problem of high cost for collecting line-of-sight data at present, the present application aims to provide a method for collecting line-of-sight data. By connecting the fixation board to the vehicle through multiple fixed points, the first position information of multiple fixation points on the fixation board in the vehicle body coordinate system can be accurately calculated using the three-dimensional model of the vehicle. Then, based on the first position information of the target fixation point in the vehicle body coordinate system and the first conversion relationship between the vehicle body coordinate system and the camera coordinate system calibrated in advance, the fixation point position information of the target fixation point in the camera coordinate system is determined, and based on the eye position information and the fixation point position information, the line-of-sight data of the face image in the camera coordinate system is obtained. In this way, high-precision and low-cost collection of line-of-sight data can be realized without relying on a depth camera.

[0074] Referring to Figure 1 , a method for collecting line-of-sight data of the present application is shown, and this method may include the following steps:

[0075] S101: Determine the first position information of multiple fixation points on the fixation board in the vehicle body coordinate system based on the three-dimensional model of the vehicle and multiple fixed points connected to the vehicle on the fixation board.

[0076] In this embodiment, multiple fixed points and multiple fixation points are provided on the fixation board. Among them, the fixed points are used to connect to the vehicle, and each fixation point is provided with a point identifier. For example, for the convenience of the target person to fixate, the point identifier can be set as a red dot of a preset size.

[0077] In this embodiment, multiple fixation points are arranged on the fixation board at a preset interval. For example, multiple fixation points can be arranged on the fixation board at a fixed distance interval. In this way, there is a fixed relative position relationship between multiple fixed points and multiple fixation points.

[0078] In this embodiment, the surface of the fixation board is flat. In this way, it can be ensured that multiple fixed points and multiple fixation points can be located on the same plane, thereby ensuring the accuracy of the relative position.

[0079] In this embodiment, any corner point of the fixation board (such as the fixed point located at the lower left corner) can be used as the origin, and the horizontal and vertical axes are respectively parallel to the long side and the short side of the fixation board to construct the fixation board coordinate system. In this way, based on this fixation board coordinate system, the position information of multiple fixed points and multiple fixation points in the fixation board coordinate system can be obtained.

[0080] In specific implementation, for the convenience of installation and for the acquisition personnel to fixate, the fixation board can be fixed to the vehicle body A-pillar, and 4 fixed points are provided on the fixation board.

[0081] In this embodiment, since the three-dimensional model of the vehicle stores the coordinates of each point of the vehicle in the vehicle body coordinate system, therefore, based on the three-dimensional model of the vehicle, the coordinates of multiple fixed points of the fixation board in the vehicle body coordinate system can be obtained. And multiple fixed points are simultaneously located in the fixation board coordinate system and the vehicle body coordinate system. Therefore, according to the coordinates of multiple fixed points in the vehicle body coordinate system, the first position information of multiple fixation points on the fixation board in the vehicle body coordinate system can be obtained.

[0082] S102: Obtain the face image of the target person collected by the image acquisition device, and determine the eye position information of the eye key points in the face image in the camera coordinate system of the image acquisition device.

[0083] In this embodiment, the target person is in a state of looking at the target fixation point on the fixation board. Among them, the target person is the data acquisition personnel. When the data acquisition personnel perform line-of-sight data acquisition, they can fixate on multiple fixation points on the fixation board in turn, and then obtain the line-of-sight data corresponding to each fixation point.

[0084] It should be noted that the image acquisition device is a device without the function of collecting depth information. The image acquisition device can adopt the DMS camera configured by DMS, and this DMS camera can be an infrared camera. Therefore, compared with complex depth cameras, it has the advantage of low cost.

[0085] In this embodiment, multiple image acquisition devices can be set. Different image acquisition devices have different installation positions and installation angles. In this way, by capturing face images from different positions and angles, the collected line-of-sight data can be made more abundant, thereby improving the training effect of the line-of-sight estimation model.

[0086] In this embodiment, the face image can be input into the face key-point detection model to obtain the image coordinates of the eye key-points input by the face key-point detection model in the face image. Then, by converting the image coordinates of the eye key-points in the face image into the camera coordinate system, the eye position information of the eye key-points in the camera coordinate system of the image acquisition device can be obtained.

[0087] S103: Based on the first position information of the target fixation point in the vehicle body coordinate system and the first conversion relationship between the pre-calibrated vehicle body coordinate system and the camera coordinate system, determine the fixation point position information of the target fixation point in the camera coordinate system.

[0088] In this embodiment, the first conversion relationship between the vehicle body coordinate system and the camera coordinate system specifically includes the rotation matrix and the translation matrix between the vehicle body coordinate system and the camera coordinate system.

[0089] In this embodiment, after obtaining the first position information of the target fixation point in the vehicle body coordinate system, based on the first conversion relationship, the target fixation point can be converted from the vehicle body coordinate system to the camera coordinate system to obtain the fixation point position information in the camera coordinate system.

[0090] S104: Based on the eye position information and the fixation point position information, obtain the line-of-sight data of the face image in the camera coordinate system.

[0091] In this embodiment, both the eye position information and the fixation point position information are three-dimensional position information. Denote the eye position information as (x ce , y ce , z ce ), and the fixation point position information as (x c , y c , z c ), then by subtracting the eye position information (x ce , y ce , z ce ) from the fixation point position information (x c , y c , z c) to obtain the sight direction (x c -x ce ,y c -y ce ,z c -z ce ). In this way, based on the eye position information, the gaze point position information, and the gaze direction, a set of gaze data can be obtained. The gaze data is represented by a vector with the eye position information as the gaze starting point and pointing to the gaze point position information.

[0092] In this embodiment, the gaze data is used as the gaze label of the face image to obtain a training sample, which is used to train the gaze estimation model.

[0093] In this embodiment, by connecting multiple fixed points on the gaze board to the vehicle, a line of sight data acquisition system can be built simply and efficiently. At the same time, by cleverly utilizing the three-dimensional model of the vehicle, the position information of multiple fixed points of the gaze board in the vehicle body coordinate system can be quickly obtained, and the coordinates of all gaze points in the vehicle body coordinate system can be obtained, so that each gaze point has a high position accuracy. In this way, the accuracy of the line of sight data can be effectively improved while reducing the data acquisition cost.

[0094] In a feasible implementation, S101 may specifically include the following sub-steps:

[0095] S101 - 1 : Determine second position information of a plurality of fixed points in a vehicle body coordinate system based on the three-dimensional model.

[0096] In this embodiment, since the three-dimensional model of the vehicle stores the coordinates of each point of the vehicle in the vehicle body coordinate system, the coordinates of multiple fixed points of the gaze plate in the vehicle body coordinate system can be obtained based on the three-dimensional model of the vehicle.

[0097] In the specific implementation, after the collection personnel install the gaze board on multiple fixed points of the vehicle, they can open the three-dimensional model of the vehicle in the three-dimensional software, and by clicking the corresponding fixed points in the three-dimensional model, they can retrieve the second position information of multiple fixed points in the vehicle body coordinate system.

[0098] S101 - 2 : Determine a second conversion relationship between the gaze plate coordinate system and the vehicle body coordinate system based on second position information of the plurality of fixed points in the vehicle body coordinate system and third position information of the plurality of fixed points in the gaze plate coordinate system.

[0099] In this embodiment, since multiple fixed points are simultaneously located in both the sight board coordinate system and the vehicle body coordinate system, based on the second position information and the third position information, the rotation matrix and the translation matrix between the sight board coordinate system and the vehicle body coordinate system can be solved, and then, according to the rotation matrix and the translation matrix between the sight board coordinate system and the vehicle body coordinate system, the second conversion relationship between the sight board coordinate system and the vehicle body coordinate system can be obtained.

[0100] It should be noted that the coordinates (x g , y g , z g ) of the fixed points of the sight board in the sight board coordinate system and the coordinates (x w , y w , z w ) of the fixed points in the vehicle body coordinate system need to satisfy the following formula:

[0101]

[0102] where, R gw represents the rotation matrix between the sight board coordinate system and the vehicle body coordinate system; T gw represents the translation matrix between the sight board coordinate system and the vehicle body coordinate system.

[0103] In this embodiment, by substituting the second position information of multiple fixed points in the vehicle body coordinate system and the third position information of multiple fixed points in the sight board coordinate system into formula (1) and solving the equation, the rotation matrix R gw and the translation matrix T gw from the sight board coordinate system to the vehicle body coordinate system can be obtained.

[0104] S101-3: Based on the third position information of multiple fixed points in the sight board coordinate system and a preset interval, determine the fourth position information of multiple sight positions in the sight board coordinate system.

[0105] In this embodiment, since multiple fixed points and multiple sight positions are all located in the sight board coordinate system, based on the preset interval between multiple sight positions, the fourth position information of multiple sight positions in the sight board coordinate system can be deduced.

[0106] Exemplarily, taking the fixed point at the lower left of the fixation board as the origin (0, 0) of the fixation board coordinate system, a plurality of fixation points are arranged at a preset interval of 20 cm in the abscissa direction of the fixation board coordinate system and at a preset interval of 10 cm in the ordinate direction of the fixation board coordinate system. Then, the coordinates of the fixation points in the bottom row of the fixation board are (20, 0), (40, 0), (60, 0),... in sequence, and the coordinates of the fixation points in the second bottom row of the fixation board are (20, 10), (40, 10), (60, 10),... in sequence, and so on. Thus, the fourth position information of all fixation points in the fixation board coordinate system can be accurately obtained.

[0107] S101-4: Based on the second conversion relationship and the fourth position information of a plurality of fixation points in the fixation board coordinate system, determine the first position information of the plurality of fixation points in the vehicle body coordinate system.

[0108] In this embodiment, after determining the fourth position information of the plurality of fixation points in the fixation board coordinate system, by substituting the fourth position information into the solved second conversion relationship, the first position information of the plurality of fixation points on the fixation board in the vehicle body coordinate system can be obtained.

[0109] In this embodiment, to facilitate quickly determining the first position information of each target fixation point in the vehicle body coordinate system during the data acquisition stage, the line-of-sight data acquisition method may further include the following steps: obtain a number mapping relationship; in response to a target number input by the user for the target fixation point, based on the target number and the number mapping relationship, determine the first position information of the target fixation point in the vehicle body coordinate system. Wherein, the number mapping relationship is used to represent the corresponding relationship between different numbers and the first position information of different fixation points in the vehicle body coordinate system.

[0110] In this embodiment, each fixation point on the fixation board is numbered in sequence. After the acquisition personnel look at the target fixation point, by directly inputting the target number corresponding to the target fixation point, the first position information of the target fixation point in the vehicle body coordinate system can be quickly determined according to the number mapping relationship.

[0111] In a feasible embodiment, the first conversion relationship between the vehicle body coordinate system and the camera coordinate system can be calibrated through the following steps:

[0112] S201: Obtain the in-vehicle image collected by the image acquisition device.

[0113] In this embodiment, the in-vehicle image includes a plurality of marker points arranged inside the vehicle. That is to say, a plurality of marker points can be pasted inside the vehicle, and the features of each marker point are obvious and easy to distinguish and within the field of view of the image acquisition device. In this way, the in-vehicle image containing a plurality of marker points can be captured by the image acquisition device.

[0114] In this embodiment, the number of marked points can be set according to the accuracy requirement. The more the number of marked points, the higher the calibration accuracy of the first conversion relationship. For example, in order to obtain a higher calibration accuracy, the number of marked points can be set to be greater than 10.

[0115] S202: Based on the three-dimensional model, determine the marked point coordinates of multiple marked points in the vehicle body coordinate system.

[0116] In this embodiment, since multiple marked points are all set on the vehicle, therefore, by calling the coordinate data of the three-dimensional model, the marked point coordinates of multiple marked points in the vehicle body coordinate system can be obtained.

[0117] S203: Based on the camera internal parameter matrix of the image acquisition device, the image coordinates of multiple marked points in the in-vehicle image, and the marked point coordinates of multiple marked points in the vehicle body coordinate system, determine the first conversion relationship between the vehicle body coordinate system and the camera coordinate system.

[0118] It should be noted that the camera internal parameter matrix of the image acquisition device is pre-calibrated. Exemplarily, the image acquisition device can be used to capture multiple checkerboard images from different angular positions, and then based on a preset camera internal parameter calibration method, such as the Zhang Zhengyou camera internal parameter calibration method, the camera internal parameter matrix of the image acquisition device can be calibrated. As a mature existing technology, the specific calibration process of the camera internal parameter calibration method will not be elaborated here.

[0119] In this embodiment, image recognition can be performed on multiple marked points in the in-vehicle image to obtain the image coordinates of multiple marked points in the in-vehicle image; or, the marked points can be clicked on the in-vehicle image, and according to the click position coordinates, the image coordinates of the marked points can be obtained.

[0120] In this embodiment, by substituting the image coordinates of all marked points in the in-vehicle image and the corresponding marked point coordinates in the vehicle body coordinate system into the first camera imaging model, an overdetermined system of equations about the rotation matrix and the translation matrix can be obtained, and by solving the overdetermined system of equations, the rotation matrix and the translation matrix between the vehicle body coordinate system and the camera coordinate system can be obtained. Among them, the first camera imaging model refers to the following formula:

[0121]

[0122] Among them, (u, v) represents the image coordinates of the marked point, (x w , y w , z w ) represents the marked point coordinates of the marked point in the vehicle body coordinate system, R represents the rotation matrix between the vehicle body coordinate system and the camera coordinate system, T represents the translation matrix between the vehicle body coordinate system and the camera coordinate system, M is the camera internal parameter matrix of the image acquisition device, zc is the depth of the point in the camera coordinate system.

[0123] In this embodiment, since each marker point provides two sets of information simultaneously, one is the image coordinates (u, v), and the other is the marker point coordinates (x w , y w , z w ) in the vehicle body coordinate system. Therefore, substituting the image coordinates and vehicle body coordinates of all marker points into the camera model, an overdetermined system of equations about R and T can be obtained. Solving this overdetermined system of equations can obtain the rotation matrix R and the translation matrix T. It should be noted that to ensure obtaining the solution of the overdetermined system of equations, at least six marker points can be set, and the more the number of marker points, the higher the accuracy of the calibrated first conversion relationship.

[0124] In this embodiment, by pasting a large number of marker points inside the vehicle and cleverly using the three-dimensional model of the vehicle, the marker point coordinates of multiple marker points in the vehicle body coordinate system can be quickly obtained, and then the accurate calibration of the first conversion relationship between the vehicle body coordinate system and the camera coordinate can be realized.

[0125] In a feasible embodiment, the step of determining the eye position information of the eye key points in the face image in the camera coordinates of the image acquisition device in S102 may specifically include the following sub-steps:

[0126] S102-1: Determine the fifth position information of the eye key points in the face image in the world coordinate system.

[0127] In this embodiment, the face image collected by the image acquisition device is a two-dimensional image. Therefore, it is necessary to convert the eye key points in the face image from two-dimensional image coordinates to three-dimensional world coordinates, that is, the fifth position information of the eye key points in the world coordinate system.

[0128] In this embodiment, the conversion of the eye key points from two-dimensional image coordinates to three-dimensional world coordinates can be realized according to a general face three-dimensional model. In a specific implementation, S102-1 may specifically include the following sub-steps:

[0129] S102-1-1: Determine the image coordinates of the eye key points in the face image.

[0130] In this embodiment, a pre-trained key point detection model can be used to detect the eye key points in the face image to obtain the image coordinates of the eye key points. Among them, the key point detection model can be constructed based on yolov8 (a target detection algorithm).

[0131] S102-1-2: Based on the preset face three-dimensional model and the image coordinates of the eye key points in the face image, determine the fifth position information of the eye key points in the world coordinate system.

[0132] It should be noted that the 3D face model is a model containing a large amount of face data, which can describe the 3D structure of the face. The 3D face model is usually composed of a set of 3D coordinates, and these 3D coordinates correspond to the key feature points of the face, such as eyes, nose, mouth, etc.

[0133] In this embodiment, each eye key point has a 3D coordinate in the world coordinate system. Therefore, after extracting the image coordinates of the eye key points from the face image, based on the 3D face model, the fifth position information of the eye key points in the world coordinate system can be determined.

[0134] S102-2: Based on the fifth position information and the pre-calibrated third conversion relationship between the world coordinate system and the camera coordinate system, determine the eye position information of the eye key points in the camera coordinate system.

[0135] In this embodiment, the third conversion relationship between the world coordinate system and the camera coordinate system specifically includes the rotation matrix and the translation matrix between the world coordinate system and the camera coordinate system.

[0136] In this embodiment, after obtaining the fifth position information of the eye key points in the world coordinate system, based on the third conversion relationship, the eye key points can be converted from the world coordinate system to the camera coordinate system to obtain the eye position information in the camera coordinate system.

[0137] In this embodiment, by cleverly using the 3D face model, the fifth position information of the eye key points in the world coordinate system can be obtained more accurately, and then combined with the third conversion relationship, the eye key points can be converted from the world coordinate system to the camera coordinate system.

[0138] In a feasible embodiment, the third conversion relationship can be calibrated through the following steps:

[0139] S301: Obtain the sample face image collected by the image acquisition device and determine the image coordinates of multiple face key points in the sample face image.

[0140] In this embodiment, the sample face image collected by the image acquisition device is used, and the pre-trained key point detection model is used to detect the key points of the face image to obtain the image coordinates of multiple face key points. Among them, the face key points can include but are not limited to eye key points, mouth key points, and nose key points.

[0141] S302: Based on the preset 3D face model and the image coordinates of multiple face key points, determine the sixth position information of multiple face key points in the world coordinate system.

[0142] In this embodiment, after extracting the image coordinates of the facial key points from the facial image, the fifth position information of the facial key points in the world coordinate system can be determined based on the 3D facial model.

[0143] S303: Determine the third transformation relationship between the world coordinate system and the camera coordinate system based on the camera internal parameter matrix of the image acquisition device, the image coordinates of multiple facial key points in the sample facial image, and the sixth position information of the multiple facial key points in the world coordinate system.

[0144] In this embodiment, by substituting the image coordinates of all facial key points in the sample facial image and the corresponding sixth position information in the world coordinate system into the second camera imaging model, an overdetermined system of equations about the rotation matrix and the translation matrix can be obtained. Solving the overdetermined system of equations can obtain the rotation matrix and the translation matrix between the world coordinate system and the camera coordinate system. Among them, the second camera imaging model refers to the following formula:

[0145]

[0146] Among them, (u', v') represents the image coordinates of the facial key points in the sample facial image, (x w ', y w ', z w ') represents the six-position information of the facial key points in the world coordinate system, R FC represents the rotation matrix between the world coordinate system and the camera coordinate system, T FC represents the translation matrix between the world coordinate system and the camera coordinate system, M is the camera internal parameter matrix of the image acquisition device, and z c is the depth of the point in the camera coordinate system.

[0147] In this embodiment, after obtaining the rotation matrix and the translation matrix between the world coordinate system and the camera coordinate system, based on this, the eye position information (x ce , y ce , z ce ) can be calculated through the following formula:

[0148]

[0149] Among them, (x ce , y ce , z ce ) represents the eye position information of the eye key points in the camera coordinate system, (x f , y f , z f ) represents the fifth position information of the eye key points in the world coordinate system; R FC represents the rotation matrix between the world coordinate system and the camera coordinate system, T FCRepresents the translation matrix between the world coordinate system and the camera coordinate system.

[0150] In this embodiment, by using the sample face images captured by the image acquisition device and the position information provided by the three-dimensional face model, the accurate calibration of the third conversion relationship can be achieved.

[0151] Second, based on the same inventive concept, referring to Figure 2 , an embodiment of the present application provides a line-of-sight data acquisition device 200, which includes:

[0152] A first position determination module 201, configured to determine the first position information of a plurality of gaze points on the gaze board in the vehicle body coordinate system based on the three-dimensional model of the vehicle and a plurality of fixed points connected to the vehicle on the gaze board; wherein, the plurality of gaze points are arranged on the gaze board at a preset interval;

[0153] A second position determination module 202, configured to obtain the face image of the target person captured by the image acquisition device and determine the eye position information of the eye key points in the face image in the camera coordinate system of the image acquisition device; the target person is in a state of looking at the target gaze point on the gaze board;

[0154] A third position determination module 203, configured to determine the gaze point position information of the target gaze point in the camera coordinate system based on the first position information of the target gaze point in the vehicle body coordinate system and the first conversion relationship between the vehicle body coordinate system and the camera coordinate system calibrated in advance;

[0155] A line-of-sight data determination module 204, configured to obtain the line-of-sight data of the face image in the camera coordinate system based on the eye position information and the gaze point position information.

[0156] In an embodiment of the present application, the first position determination module 201 includes:

[0157] A first position determination sub-module, configured to determine the second position information of the plurality of fixed points in the vehicle body coordinate system based on the three-dimensional model;

[0158] A first conversion relationship determination sub-module, configured to determine the second conversion relationship between the gaze board coordinate system and the vehicle body coordinate system based on the second position information of the plurality of fixed points in the vehicle body coordinate system and the third position information of the plurality of fixed points in the gaze board coordinate system;

[0159] A second position determination sub-module, configured to determine the fourth position information of the plurality of gaze points in the gaze board coordinate system based on the third position information of the plurality of fixed points in the gaze board coordinate system and the preset interval;

[0160] A third position determination sub-module, configured to determine first position information of multiple fixation points in the vehicle body coordinate system based on the second conversion relationship and fourth position information of the multiple fixation points in the fixation board coordinate system.

[0161] In an embodiment of the present application, the line-of-sight data acquisition device 200 further includes:

[0162] A mapping relationship acquisition module, configured to acquire a number mapping relationship; the number mapping relationship is used to represent the corresponding relationship between different numbers and first position information of different fixation points in the vehicle body coordinate system;

[0163] An information determination module, configured to determine first position information of a target fixation point in the vehicle body coordinate system based on the target number and the number mapping relationship in response to a target number input by a user for the target fixation point.

[0164] In an embodiment of the present application, the second position determination module 202 includes:

[0165] A fourth position determination sub-module, configured to determine fifth position information of eye key points in the face image in the world coordinate system;

[0166] An eye position determination sub-module, configured to determine eye position information of the eye key points in the camera coordinate system based on the fifth position information and a pre-calibrated third conversion relationship between the world coordinate system and the camera coordinate.

[0167] In an embodiment of the present application, the fourth position determination sub-module includes:

[0168] An image coordinate determination unit, configured to determine image coordinates of eye key points in the face image;

[0169] An eye position determination unit, configured to determine fifth position information of eye key points in the world coordinate system based on a preset three-dimensional face model and image coordinates of eye key points in the face image.

[0170] In an embodiment of the present application, the line-of-sight data acquisition device 200 further includes:

[0171] An in-vehicle image acquisition module, configured to acquire an in-vehicle image acquired by an image acquisition device; the in-vehicle image includes multiple marker points arranged inside the vehicle;

[0172] A marker point coordinate determination module, configured to determine marker point coordinates of the multiple marker points in the vehicle body coordinate system based on a three-dimensional model;

[0173] The first calibration module is configured to determine a first conversion relationship between the vehicle body coordinate system and the camera coordinate system based on the camera internal parameter matrix of the image acquisition device, the image coordinates of multiple marker points in the in-vehicle image, and the marker point coordinates of the multiple marker points in the vehicle body coordinate system.

[0174] In an embodiment of the present application, the line-of-sight data acquisition device 200 further includes:

[0175] The face image acquisition module is configured to acquire a sample face image acquired by the image acquisition device and determine the image coordinates of multiple face key points in the sample face image;

[0176] The key point position determination module is configured to determine the sixth position information of multiple face key points in the world coordinate system based on a preset three-dimensional face model and the image coordinates of the multiple face key points;

[0177] The second calibration module is configured to determine a third conversion relationship between the world coordinate system and the camera coordinate system based on the camera internal parameter matrix of the image acquisition device, the image coordinates of multiple face key points in the sample face image, and the sixth position information of the multiple face key points in the world coordinate system.

[0178] It should be noted that the specific implementation manner of the line-of-sight data acquisition device 200 in the embodiment of the present application refers to the specific implementation manner of the line-of-sight data acquisition method proposed in the first aspect of the present application described above, and will not be elaborated here.

[0179] In a third aspect, based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which an executable program is stored, and when the executable program is executed by a processor, the line-of-sight data acquisition method proposed in the first aspect of the present application is implemented.

[0180] It should be noted that the specific implementation manner of the computer-readable storage medium in the embodiment of the present application refers to the specific implementation manner of the line-of-sight data acquisition method proposed in the first aspect of the present application described above, and will not be elaborated here.

[0181] In a fourth aspect, referring to Figure 3 , based on the same inventive concept, an embodiment of the present application provides a vehicle 300, including:

[0182] A memory 301 for storing an executable program;

[0183] A processor 302;

[0184] When the executable program is executed by the processor 302, the line-of-sight data acquisition method proposed in the first aspect of the present application is implemented.

[0185] It should be noted that the specific implementation of the vehicle 300 in the embodiments of the present application refers to the specific implementation of the line-of-sight data acquisition method proposed in the first aspect of the embodiments of the present application, which will not be elaborated here.

[0186] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0188] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks. [[ID=2)]

[0190] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0191] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0192] The above has introduced in detail a method, device, storage medium and vehicle for collecting line-of-sight data. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for collecting line-of-sight data, characterized in that The method includes: Based on a three-dimensional model of a vehicle and a plurality of fixed points connected to the vehicle on a sight board, determining first position information of a plurality of sight points on the sight board in a vehicle body coordinate system; wherein, the plurality of sight points are arranged on the sight board at a preset interval; Obtaining a face image of a target person collected by an image acquisition device, and determining eye position information of eye key points in the face image in a camera coordinate system of the image acquisition device; the target person is in a state of looking at a target sight point on the sight board; Based on the first position information of the target sight point in the vehicle body coordinate system and a first conversion relationship between the vehicle body coordinate system and the camera coordinate system calibrated in advance, determining sight point position information of the target sight point in the camera coordinate system; Based on the eye position information and the sight point position information, obtaining sight line data of the face image in the camera coordinate system.

2. The method for collecting line-of-sight data according to claim 1, wherein Determining the first position information of the plurality of sight points on the sight board in the vehicle body coordinate system based on the three-dimensional model of the vehicle and the plurality of fixed points connected to the vehicle on the sight board includes: Based on the three-dimensional model, determining second position information of the plurality of fixed points in the vehicle body coordinate system; Based on the second position information of the plurality of fixed points in the vehicle body coordinate system and third position information of the plurality of fixed points in a sight board coordinate system, determining a second conversion relationship between the sight board coordinate system and the vehicle body coordinate system; Based on the third position information of the plurality of fixed points in the sight board coordinate system and the preset interval, determining fourth position information of the plurality of sight points in the sight board coordinate system; Based on the second conversion relationship and the fourth position information of the plurality of sight points in the sight board coordinate system, determining the first position information of the plurality of sight points in the vehicle body coordinate system.

3. The method for collecting line-of-sight data according to claim 2, wherein The method further includes: Obtaining a number mapping relationship; the number mapping relationship is used to represent the corresponding relationship between different numbers and the first position information of different sight points in the vehicle body coordinate system; In response to a target number input by a user for the target sight point, based on the target number and the number mapping relationship, determining the first position information of the target sight point in the vehicle body coordinate system.

4. The method for collecting line-of-sight data according to claim 1, characterized in that Determining the eye position information of the eye key points in the face image in the camera coordinate system of the image acquisition device includes: Determining fifth position information of the eye key points in the face image in a world coordinate system; Based on the fifth position information and a third conversion relationship between the world coordinate system and the camera coordinate system calibrated in advance, determining the eye position information of the eye key points in the camera coordinate system.

5. The method for collecting line-of-sight data according to claim 4, wherein Determining the fifth position information of the eye key points in the face image in the world coordinate system includes: Determining the image coordinates of the eye key points in the face image; Based on a preset three-dimensional face model and the image coordinates of the eye key points in the face image, determining the fifth position information of the eye key points in the world coordinate system.

6. The method for collecting line-of-sight data according to claim 1, wherein The first conversion relationship is calibrated as follows: Obtain the in-vehicle image collected by the image acquisition device; the in-vehicle image includes a plurality of marker points arranged inside the vehicle; Based on the three-dimensional model, determine the marker point coordinates of the plurality of marker points in the vehicle body coordinate system; Based on the camera internal parameter matrix of the image acquisition device, the image coordinates of the plurality of marker points in the in-vehicle image, and the marker point coordinates of the plurality of marker points in the vehicle body coordinate system, determine the first conversion relationship between the vehicle body coordinate system and the camera coordinate system.

7. The method for collecting line-of-sight data according to claim 4, wherein The third conversion relationship is calibrated as follows: Obtain the sample face image collected by the image acquisition device, and determine the image coordinates of multiple face key points in the sample face image; Based on the preset three-dimensional face model and the image coordinates of the multiple face key points, determine the sixth position information of the multiple face key points in the world coordinate system; Based on the camera internal parameter matrix of the image acquisition device, the image coordinates of the multiple face key points in the sample face image, and the sixth position information of the multiple face key points in the world coordinate system, determine the third conversion relationship between the world coordinate system and the camera coordinate system.

8. A line-of-sight data acquisition device, characterized in that, The device includes: A first position determination module, configured to determine the first position information of a plurality of gaze points on the gaze board in the vehicle body coordinate system based on the three-dimensional model of the vehicle and a plurality of fixed points connected to the vehicle on the gaze board; wherein, the plurality of gaze points are arranged on the gaze board at a preset interval; A second position determination module, configured to obtain the face image of the target person collected by the image acquisition device, and determine the eye position information of the eye key points in the face image in the camera coordinate system of the image acquisition device; the target person is in a state of looking at the target gaze point on the gaze board; A third position determination module, configured to determine the gaze point position information of the target gaze point in the camera coordinate system based on the first position information of the target gaze point in the vehicle body coordinate system and the first conversion relationship between the vehicle body coordinate system and the camera coordinate system calibrated in advance; A gaze data determination module, configured to obtain the gaze data of the face image in the camera coordinate system based on the eye position information and the gaze point position information.

9. A computer-readable storage medium having an executable program stored thereon, characterized in that, When the executable program is executed by the processor, it implements the gaze data acquisition method according to any one of claims 1-7.

10. A vehicle, characterized in that, Including: A memory, configured to store an executable program; A processor; When the executable program is executed by the processor, it implements the gaze data acquisition method according to any one of claims 1-7.

Images