A method for collecting line of sight data, electronic device and computer-readable storage medium

Abstract

The present invention discloses a sight data acquisition method, an electronic device and a computer-readable storage medium, the method comprising: acquiring image data, the image data including the acquired image and the sight information of the target object in each acquired image, the sight information including the observation point of the target object, constructing a coordinate system according to the position information of the target object, and obtaining the vehicle coordinate system of the vehicle where the target object is located; then converting the coordinates of the observation point of the target object in each acquired image to the vehicle coordinate system, and obtaining the three-dimensional coordinates of each observation point in the vehicle coordinate system; and then determining the viewing angle information of the target object in the corresponding acquired image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system. Thus, the quality can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a line of sight data collection method, electronic equipment and computer-readable storage medium. Background Art

[0002] With the popularization of new energy vehicles and the upgrading of the intelligent automobile industry, the number of intelligent cockpits capable of intelligent interaction is gradually increasing. The intelligent cockpit can use a pre-trained line of sight estimation model to estimate the line of sight of the vehicle-related object, thereby determining whether the vehicle-related object is in a distracted state, which is of great significance in reducing vehicle accidents.

[0003] Training the gaze estimation model requires a large amount of gaze data. Currently, a gaze tracking module is used to track the trajectory of the gaze source area of ​​the object in the object image, and the gaze data is obtained according to the tracked trajectory direction. However, due to the possible amblyopia, strabismus and other problems in the gaze source area of ​​the object, the accuracy of trajectory tracking is affected, resulting in low quality of gaze data. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a line of sight data collection method, electronic device and computer-readable storage medium, which can improve data quality.

[0005] To solve the above technical problems, the present application provides a line of sight data acquisition method, the method comprising: acquiring image data, the image data including acquired images and line of sight information of target objects in each acquired image, the line of sight information including observation points of the target objects; constructing a coordinate system according to the position information of the target objects to obtain a vehicle coordinate system of the vehicle where the target objects are located; converting the coordinates of each observation point to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system; and determining the viewing angle information of the target object in the corresponding acquired image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0006] In one embodiment, the method also includes: constructing a coordinate system according to the position information of the target object to obtain a spatial coordinate system; the step of converting the coordinates of each observation point to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system includes: mapping the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system; mapping the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0007] In one embodiment, the step of mapping the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system includes: acquiring first conversion data; the first conversion data represents the mapping relationship between the coordinates of the observation point and the spatial coordinate system; and determining the ratio between the coordinates of each observation point and the first conversion data as the spatial coordinates of each observation point in the spatial coordinate system.

[0008] In one embodiment, the step of mapping the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system includes: acquiring second conversion data; the second conversion data represents the mapping relationship between the spatial coordinate system and the vehicle coordinate system; and determining the product of the spatial coordinates of each observation point in the spatial coordinate system and the second conversion data as the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0009] In one embodiment, the step of obtaining the second transformation data includes: obtaining posture information when the first acquisition device acquires a calibration image, and the acquisition direction of the calibration image is different from the acquisition direction of the acquisition image; determining a rotation matrix according to the posture information when the first acquisition device acquires the calibration image; obtaining a spatial translation vector according to the position information of the target object in the calibration image and the origin of the vehicle coordinate system; and obtaining the second transformation data according to the rotation matrix and the spatial translation vector.

[0010] In one embodiment, the step of determining the viewing angle information of the target object in the corresponding acquired image based on the three-dimensional coordinates of each observation point in the vehicle coordinate system includes: in response to the observation starting point of the target object being at the origin of the vehicle coordinate system, acquiring the sight angle corresponding to each observation point based on the three-dimensional coordinates of each observation point in the vehicle coordinate system; and determining the sight angle corresponding to each observation point as the viewing angle information of the target object in the corresponding acquired image.

[0011] In one embodiment, the step of determining the viewing angle information of the target object in the corresponding acquired image based on the three-dimensional coordinates of each observation point in the vehicle coordinate system also includes: in response to the observation starting point of the target object being at a point other than the origin of the vehicle coordinate system, obtaining the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system; obtaining the observation vector corresponding to each observation point based on the three-dimensional coordinates of each observation point in the vehicle coordinate system and the three-dimensional coordinates of the observation starting point of the target object; and determining the vector angle of each observation vector as the viewing angle information of the target object in the corresponding acquired image.

[0012] In one embodiment, the step of constructing a coordinate system based on the position information of the target object to obtain a vehicle coordinate system of the vehicle where the target object is located includes: acquiring calibration image information, the calibration image information including a calibration image and direction information in which the calibration image is acquired by a first acquisition device; constructing a coordinate system based on the direction information in which the calibration image is acquired by the first acquisition device and the position information of the first acquisition device to obtain the vehicle coordinate system of the vehicle where the target object is located.

[0013] In order to solve the above technical problems, the present application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above line of sight data collection method.

[0014] In order to solve the above technical problems, the present application provides a computer-readable storage medium including program data stored therein, and the program data is used to implement the above line of sight data collection method when executed by a processor.

[0015] The beneficial effects of the present application are as follows: collecting image data, constructing a coordinate system according to the position information of the target object of the collected image in the image data, and obtaining the vehicle coordinate system of the vehicle where the target object is located; then converting the coordinates of the observation points of the target object in each collected image to the vehicle coordinate system, and obtaining the three-dimensional coordinates of each observation point in the vehicle coordinate system; and then determining the perspective information of the target object in the corresponding collected image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system. Compared with obtaining line of sight data by tracking the trajectory of the source area of ​​the line of sight of the object, the present application converts the coordinates of the clear observation points, and then obtains the perspective information of the target object according to the three-dimensional coordinates of the observation points after the coordinate conversion, thereby avoiding the error caused by the line of sight source area, thereby improving the data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0017] Figure 1 is a flowchart of an exemplary embodiment of a line of sight data collection method shown in the present application;

[0018] Figure 2 It is a schematic diagram of the effect of an exemplary embodiment of a single-sided chessboard in the line of sight data collection method shown in the present application;

[0019] Figure 3It is a schematic diagram of the effect of an exemplary embodiment of a target object observation method in a line of sight data collection method shown in the present application;

[0020] Figure 4 It is a schematic diagram of the effect of an exemplary embodiment of a vehicle coordinate system in the line of sight data collection method shown in the present application;

[0021] Figure 5 is a flowchart of an exemplary embodiment of step S130 in the line of sight data collection method shown in the present application;

[0022] Figure 6 is a schematic diagram of an exemplary embodiment of the positional relationship between the second acquisition device and the target object in the line of sight data acquisition method shown in the present application;

[0023] Figure 7 is a schematic diagram of an exemplary embodiment of the positional relationship between a double-sided chessboard and a second acquisition device in a line of sight data acquisition method shown in the present application;

[0024] Figure 8 is a schematic diagram of an exemplary embodiment of the positional relationship between the space coordinate system and the vehicle coordinate system in the line of sight data acquisition method shown in the present application;

[0025] Fig. 9 is a flowchart of another exemplary embodiment of the line of sight data collection method shown in the present application;

[0026] Fig.10 is a block diagram of a sight line data acquisition device shown in an exemplary embodiment of the present application;

[0027] Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0028] Fig.12 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0030] First of all, it should be noted that with the popularization of new energy vehicles and the upgrading of the intelligent automobile industry, the number of intelligent cockpits capable of intelligent interaction has gradually increased. The intelligent cockpit can use a pre-trained line of sight estimation model to estimate the line of sight of the vehicle-related object, thereby determining whether the vehicle-related object is in a distracted state, which is of great significance in reducing vehicle accidents.

[0031] Training the gaze estimation model requires a large amount of gaze data. Currently, a gaze tracking module is used to track the trajectory of the gaze source area of ​​the object in the object image, and the gaze data is obtained according to the tracked trajectory direction. However, due to the possible amblyopia, strabismus and other problems in the gaze source area of ​​the object, the accuracy of trajectory tracking is affected, resulting in low quality of gaze data.

[0032] Based on this, the present application proposes a line of sight data collection method, electronic device and computer readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the line of sight data collection method shown in the present application.

[0033] The execution subject of the sight line data collection method may be a terminal device or a server or other processing device, wherein the terminal device may be a computer, a mobile device, a terminal, a computing device, a vehicle-mounted device, etc. The execution subject of the sight line data collection method may also be a sight line data collection device. In some possible implementations, the sight line data collection method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0034] The sight line data may include a captured image and the viewing angle information corresponding to the captured image. The captured image may be a frontal image of the target object; the target object may be a robot, an animal, or other object with sight line observation capability. The sight line information is the observation point information obtained when the target object observes the set end point. The observation point may be a set observation point.

[0035] Specifically, the line of sight data collection method of this embodiment includes the following steps:

[0036] Step S110 , collecting image data, where the image data includes collected images and sight line information of target objects in each collected image, where the sight line information includes observation points of the target objects.

[0037] The image data includes the captured images and the sight line information of the target object in each captured image. The sight line information includes the observation point observed by the target object.

[0038] The sight line data acquisition device acquires images. In one embodiment, the sight line data acquisition device may be a device with a shooting function, the sight line data acquisition device captures images, and uses the captured images as the captured images. For example, the sight line acquisition device may be a camera. In another embodiment, the sight line data acquisition device may be a device without a shooting function, the sight line data acquisition device is communicatively connected with a camera; the camera captures images, and sends the captured images to the sight line data acquisition device, and after receiving the captured images, the sight line data acquisition device uses the captured images as the captured images. In yet another embodiment, the sight line data acquisition device includes a data storage module and a processor, the processor sends an image call instruction to the data storage module in the sight line data acquisition device, the data storage module sends the stored images to the processor after receiving the image call instruction, and the processor uses the received stored images as the captured images.

[0039] Furthermore, when the sight line data acquisition device detects that the angle of the target area of ​​the target object changes, the front of the target object is photographed to obtain a captured image. The target area may be the robot eyeball, or other areas of the robot head except the eyeball.

[0040] The observation point can be a point on the device with a certain coordinate point. For example, the observation point is a square on a single-sided chessboard. Figure 2 As shown, the grids on the single-sided chessboard have certain coordinates, and the horizontal and vertical positions are represented by numbers 1, 2, 3, 4, 5, 6, 7, 8 and 9 respectively. For example, the observation point G has coordinates (2, 2); the observation point F has coordinates (7, 9).

[0041] The observation point information in the image data can be the coordinates of the marked area in the collected image, and the marked area can be a robot hand or a number card, etc. For a detailed explanation, please continue to refer to Figure 3 , the horizontal and vertical positions are indicated by reference numerals 1, 2, 3, 4, 5, 6, 7, 8 and 9 respectively. Figure 3 As shown in the left figure in , the observation point where the target object's sight end point is located on the grid with coordinates (2, 8) on the single-sided chessboard, and the observation point information is 2 and 8 indicated by the robot's hand in the captured image. Figure 3 As shown in the right figure in , the observation point where the target object's line of sight ends is located at the grid with coordinates (8, 4) on the single-sided chessboard, and the observation point information is 8 and 4 indicated by the robot's hand in the captured image.

[0042] Furthermore, the device where the observation point is located can set the position and posture to simulate the observation object in the real application scene. For example, the observation point is on a single-sided chessboard, and the display screen in the smart cockpit or smart home application scene is simulated by changing the position and posture of the single-sided chessboard. Therefore, the sight line data obtained by the sight line data collection method of the present application can be applied to model training in various scenarios, with a strong degree of freedom, and can provide important data support for the development of other intelligent functions.

[0043] Furthermore, the sight line data acquisition device also includes filtering the collected image according to the posture information of the target object in the collected image to obtain filtered image data. The posture information can be the head deflection angle of the target object. Specifically, the head deflection angle of the target object in the collected image is compared with a preset head deflection angle range, and the collected images and corresponding sight line information whose head deflection angle of the target object is not within the preset head deflection angle range are eliminated. Therefore, after the sight line data acquisition device filters the collected image, it obtains the collected image whose head deflection angle is within the preset head deflection angle range, thereby obtaining clean and effective image data, preventing the erroneous image data from polluting the overall image data, and is conducive to improving the quality of the sight line data obtained based on the image data.

[0044] The head deflection angle range can represent the rotation range of the simulated human head. The head deflection angle range can be obtained through a 3D standard target object head rigid body model test.

[0045] Step S120: construct a coordinate system according to the position information of the target object to obtain a vehicle coordinate system of the vehicle where the target object is located.

[0046] The location information may be the spatial location of the target object.

[0047] In one embodiment of constructing a vehicle coordinate system, the sight data acquisition device establishes a coordinate system with the driving direction of the vehicle where the target object is located as the Z axis and the vehicle plane where the target object is located as the XY plane to obtain the vehicle coordinate system. It should be noted that the Z axis is perpendicular to the XY plane.

[0048] In another embodiment of constructing a vehicle coordinate system, the sight line data acquisition device acquires calibration image information, the calibration image information includes the calibration image and the direction information of the calibration image acquired by the first acquisition device; the coordinate system is constructed based on the direction information of the calibration image acquired by the first acquisition device and the location information of the first acquisition device, and the vehicle coordinate system of the vehicle where the target object is located is obtained. The direction information can be the horizontal direction from the first acquisition device to the target object. Specifically, the sight line data acquisition device takes the location of the first acquisition device as the origin, the horizontal direction from the first acquisition device to the target object as the Z axis, and the plane where the first acquisition device is located as the XY plane to establish a coordinate system to obtain a vehicle coordinate system. Among them, the XY plane is perpendicular to the Z axis.

[0049] The first acquisition device may be an image acquisition device, such as a camera, a mobile phone, etc. The acquisition position of the first acquisition device may be in the opposite direction of the line of sight of the target object. Figure 4 As shown, the first acquisition device in the opposite direction of the target object's sight line is taken as the origin position , the horizontal direction of the first acquisition device toward the target object, that is, the sight direction of the target object, is the Zc axis, and the plane where the first acquisition device is located is the XcYc plane. The Zc axis is perpendicular to the XcYc plane.

[0050] The calibration image may be an image containing the target object captured by the first acquisition device. The calibration image may be obtained by capturing an image with a camera. The calibration image may also be obtained by capturing a video with a camera.

[0051] Step S130: convert the coordinates of each observation point into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0052] The sight line data acquisition device converts the coordinates of each observation point into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system. Exemplarily, the sight line data acquisition device maps the coordinates of each observation point directly into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system. In another exemplary embodiment, the sight line data acquisition device maps the coordinates of each observation point into the space coordinate system to obtain the three-dimensional coordinates of each observation point in the space coordinate system, and then maps the three-dimensional coordinates of each observation point in the space coordinate system into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0053] Step S140: determining the viewing angle information of the target object in the corresponding captured image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0054] The viewing angle information may be the angle information of the sight line of the target object, for example, the pitch angle and the yaw angle.

[0055] It can be seen that the line of sight data acquisition method of the embodiment of the present application acquires image data, constructs a coordinate system according to the position information of the target object of the acquired image in the image data, and obtains the vehicle coordinate system of the vehicle where the target object is located; then the coordinates of the observation points of the target object in each acquired image are converted to the vehicle coordinate system, and the three-dimensional coordinates of each observation point in the vehicle coordinate system are obtained; then the perspective information of the target object in the corresponding acquired image is determined according to the three-dimensional coordinates of each observation point in the vehicle coordinate system. On the one hand, since the coordinates of the clear observation points are converted, and then the perspective information of the target object is obtained according to the three-dimensional coordinates of the observation points after the coordinate conversion, the error caused by the line of sight source area is avoided, so that the data quality is higher. On the other hand, the present application avoids the use of high-cost equipment such as eye trackers to collect perspective information, thereby reducing the acquisition cost and being more conducive to practical promotion.

[0056] Based on the above examples, please continue to refer to Figure 5 , Figure 5 1 is a flowchart of an exemplary embodiment of step S130 in the line of sight data collection method of the present application. Specifically, step S130 converts the coordinates of each observation point to the vehicle coordinate system, and obtaining the three-dimensional coordinates of each observation point in the vehicle coordinate system includes the following steps:

[0057] Step S510: construct a coordinate system according to the position information of the target object to obtain a spatial coordinate system.

[0058] The sight line data acquisition device constructs a coordinate system according to the position information of the target object to obtain a spatial coordinate system. Specifically, the sight line data acquisition device obtains the target image and the direction information of the second acquisition device acquiring the target image; the direction information of the second acquisition device acquiring the target image can be the horizontal direction from the second acquisition device to the target object. The sight line data acquisition device uses the position of the second acquisition device as the origin, the horizontal direction from the second acquisition device to the target object as the Z axis, and the plane where the second acquisition device is located as the XY plane to establish a coordinate system to obtain a spatial coordinate system. Among them, the XY plane is perpendicular to the Z axis.

[0059] The second acquisition devices can be an even number, and the acquisition positions of every two second acquisition devices are symmetrical with the target object as the center, and there is a second acquisition device with the same acquisition position as the first acquisition device. The second acquisition device can be a camera, a mobile phone, etc. The timestamps of the second acquisition devices are unified, the resolution of the output screen is unified, and the models are the same. In one embodiment, the second acquisition devices are three groups. For details, please refer to Figure 6 , Figure 6 There are three groups of second acquisition devices in total, each group of second acquisition devices takes the target object as the symmetry center, the acquisition direction of each second acquisition device is toward the target object, and one of the second acquisition devices is located in the sight direction of the target object.

[0060] Step S520: Map the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system.

[0061] The sight line data acquisition device maps the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system. Specifically, the sight line data acquisition device obtains first conversion data; the first conversion data represents the mapping relationship between the coordinates of the observation point and the spatial coordinate system; and the ratio between the coordinates of each observation point and the first conversion data is determined as the spatial coordinates of each observation point in the spatial coordinate system.

[0062] The first conversion data may be the product of the camera intrinsic parameters and the camera extrinsic parameters. The first conversion data is obtained by:

[0063] The first step is to obtain a checkerboard image collected by the second acquisition device. The grids in the checkerboard image have fixed pixel coordinates. , the homogeneous coordinates of the pixel coordinates ; The chessboard in the chessboard image corresponds to the double-sided chessboard at the midpoint position of each group of second acquisition devices; each grid on the double-sided chessboard has a fixed coordinate , the homogeneous coordinates of the grid coordinates , where the chessboard is 0. The position of the double-sided chessboard and a set of symmetrical second acquisition devices can be found in Figure 7 .like Figure 7 As shown, the double-sided chessboard is located in the middle of the two second acquisition devices, and the acquisition directions of the two second acquisition devices are toward the grid surfaces of the double-sided chessboard.

[0064] Step 2: Homography Matrix H Satisfies the following formula, where the vector to is the homography matrix H The elements in:

[0065] = ;

[0066] Homography matrix H , homogeneous coordinates of pixel coordinates in a checkerboard image , the homogeneous coordinates of the grid coordinates on the chessboard The following homography relationship equation is satisfied:

[0067] ;

[0068] Substituting the homogeneous coordinates of the four pairs of chessboard coordinates and pixel coordinates into the above homography relationship equation, we get the following formula. Perform SVD (Singular Value Decomposition) on the following formula to get the homography matrix: H Each element vector in to ;

[0069] ;

[0070] Step 3: Elements and constraint vectors in the homography matrix Satisfies the following formula; where, The homography matrix is OK, The homography matrix is List;

[0071] ;

[0072] Step 4: Constraint Vector The following constraints are satisfied, where b is the feature matrix, to For each eigenvector in the feature matrix:

[0073] ;

[0074] At least three groups Substituting the value of into the above formula we get to .

[0075] Step 5: The camera intrinsic parameter matrix A satisfies the following equation: Representing images u Axis scale scaling factor, Representing images v Axis scale scaling factor, Characterizing images u Axis and v The degree of axis deviation, express u Axis origin, express v Axis origin;

[0076]

[0077] Eigenvector to , Each element in the camera intrinsic parameter matrix A satisfies the following formula:

[0078]

[0079] Step 6: Camera extrinsics T Satisfy the formula , represents the first rotation vector, represents the second rotation vector, represents the third rotation vector, Represents the translation vector; the elements in the camera extrinsic parameters, the camera intrinsic parameter matrix A and the above homography matrix elements 、 、 Satisfies the following formula:

[0080] ;

[0081] in, Represents the scaling factor, satisfying the following formula:

[0082] .

[0083] Step S530: Map the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0084] The sight line data acquisition device maps the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system. Specifically, the sight line data acquisition device obtains the second conversion data; the second conversion data represents the mapping relationship between the spatial coordinate system and the vehicle coordinate system; the product of the spatial coordinates of each observation point in each spatial coordinate system and the second conversion data is determined as the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0085] As an example, see Figure 8 , Figure 8 There are two spatial coordinate systems based on the second acquisition device. , , and a vehicle coordinate system constructed based on the first acquisition device, the origin of the vehicle coordinate system (0 0 0), the coordinate axes are Xc, Yc, Zc There is a double-sided chessboard between the two second acquisition devices and the first acquisition device. The sight data acquisition device obtains the mapping relationship between the actual coordinates of the chessboard and the pixel coordinates of the chessboard image based on the images of the double-sided chessboard taken by the second acquisition device and the first acquisition device, that is, the first conversion data. The first conversion data is then used to map the coordinates of the observation point to the spatial coordinate system, and then the three-dimensional coordinates of the observation point in each spatial coordinate system are mapped to the vehicle coordinate system, so as to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0086] The sight data acquisition device acquires the second conversion data. Specifically, the posture information when the first acquisition device acquires the calibration image is acquired, and the acquisition direction of the calibration image is different from the acquisition direction of the acquisition image; the rotation matrix is ​​determined according to the posture information when the first acquisition device acquires the calibration image; the space translation vector is acquired according to the position information of the target object in the calibration image and the origin of the vehicle coordinate system; and the second conversion data is acquired according to the rotation matrix and the space translation vector.

[0087] The attitude information is the spatial angle information of the first acquisition device. The attitude information may be the pitch angle and yaw angle of the first acquisition device.

[0088] The pitch angle of the first acquisition device pitch, Yaw angle yaw and the rotation matrix R Satisfies the following formula:

[0089] .

[0090] The sight line data acquisition device obtains the spatial translation vector according to the position information of the target object in the calibration image and the origin of the vehicle coordinate system. Specifically, the sight line data acquisition device obtains the spatial translation vector between the position of the target object and the origin of the vehicle coordinate system. Satisfies the following formula:

[0091] ;

[0092] in, Indicates the distance between the target object's position and the origin of the vehicle coordinate system in the x-axis direction. Indicates the distance between the target object's position and the origin of the vehicle coordinate system in the y-axis direction. Indicates the distance between the target object's position and the origin of the vehicle coordinate system in the z-axis direction.

[0093] Exemplarily, the spatial translation vector can be obtained by actually measuring the distance between the target object and the first acquisition device. In another exemplary embodiment, the spatial translation vector can be obtained by calculating the distance between the three-dimensional coordinates of the target object and the origin in the vehicle coordinate system. In another exemplary embodiment, the spatial translation vector can be obtained by constructing a digital model.

[0094] The sight data acquisition device obtains the second conversion data according to the rotation matrix and the spatial translation vector. For details, please refer to the following formula, where Q For the second converted data:

[0095] .

[0096] It can be seen that the coordinate system is constructed according to the position information of the target object to obtain the spatial coordinate system, the ratio between the coordinates of each observation point and the first conversion data is determined as the spatial coordinates of each observation point in the spatial coordinate system, and then the product of the spatial coordinates of each observation point in each spatial coordinate system and the second conversion data is determined as the three-dimensional coordinates of each observation point in the vehicle coordinate system, so that the spatial coordinates of each observation point in the spatial coordinate system are mapped to the vehicle coordinate system. In this way, the two-dimensional coordinates of the observation point of the target object are converted to the three-dimensional coordinates of the observation point in the vehicle coordinate system.

[0097] See also Fig. 9 , Fig. 9 FIG. 1 is a flow chart of another exemplary embodiment of the line of sight data collection method shown in the present application. Specifically, the line of sight data collection method of the embodiment of the present application includes the following steps:

[0098] Step S910: collecting image data, where the image data includes collected images and sight line information of target objects in each collected image, where the sight line information includes observation points of the target objects.

[0099] Step S920: construct a coordinate system according to the position information of the target object to obtain a vehicle coordinate system of the vehicle where the target object is located.

[0100] Step S930: convert the coordinates of each observation point into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0101] The detailed description of step S910 to step S930 in the embodiment of the present application can refer to step S110 to step S130 in the above embodiment, which will not be repeated here.

[0102] Step S940, determine whether the observation starting point of the target object is at the origin of the vehicle coordinate system, if so, execute step S950, if not, execute step S960.

[0103] Step S950, in response to the observation starting point of the target object being at the origin of the vehicle coordinate system, obtaining the sight angle corresponding to each observation point based on the three-dimensional coordinates of each observation point in the vehicle coordinate system, and determining the sight angle corresponding to each observation point as the viewing angle information of the target object in the corresponding captured image.

[0104] The observation starting point is the starting point of the sight line of the target object. The observation starting point can be the midpoint position of the target area of ​​the target object, for example, the observation starting point is the midpoint position of the robot's eyes.

[0105] The sight angle is the angle of the sight observed by the target object. The sight angle can be a pitch angle or a yaw angle.

[0106] The sight line data acquisition device obtains the sight line angle corresponding to each observation point based on the three-dimensional coordinates of each observation point in the vehicle coordinate system, and determines the sight line angle corresponding to each observation point as the viewing angle information of the target object in the corresponding collected image. For details, please refer to the following formula: Wherein, the three-dimensional coordinates of the observation point in the vehicle coordinate system are expressed as ( x, y, z ), Indicates the sight line pitch angle corresponding to the observation point, Indicates the line of sight yaw angle corresponding to the observation point;

[0107] .

[0108] Step S960: In response to the observation starting point of the target object being at a point other than the origin of the vehicle coordinate system, the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system are acquired.

[0109] The line of sight data acquisition device obtains the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system. Exemplarily, the line of sight data acquisition device includes a data storage module, and the line of sight data acquisition device retrieves the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system from the data storage module. Specifically, the target object is scanned in advance by a 3D scanner to obtain the three-dimensional coordinates of the midpoint position of the target area of ​​the target object. The 3D scanner sends the three-dimensional coordinates of the midpoint position of the target area of ​​the target object to the line of sight data acquisition device, and the line of sight data acquisition device stores the three-dimensional coordinates of the midpoint position of the target area as the three-dimensional coordinates of the observation starting point in the data storage module so that the line of sight data acquisition device can retrieve it later.

[0110] Step S970, obtaining the observation vector corresponding to each observation point according to the three-dimensional coordinates of each observation point in the vehicle coordinate system and the three-dimensional coordinates of the observation starting point of the target object, and determining the vector angle of each observation vector as the viewing angle information of the target object in the corresponding captured image.

[0111] The observation vector is the vector formed from the observation starting point to the observation point in the vehicle coordinate system.

[0112] The sight data acquisition device determines the vector obtained by subtracting the three-dimensional coordinates of the observation starting point of the target object from the three-dimensional coordinates of each observation point in the vehicle coordinate system as the observation vector corresponding to each observation point.

[0113] Specifically, the sight line data acquisition device calculates the vector angle of each observation vector by the following formula, and determines the vector angle of each observation vector obtained as the viewing angle information of the target object in the corresponding collected image; wherein the observation vector is expressed as , the viewing angle information is expressed as the pitch angle and yaw angle .

[0114]

[0115] It can be seen that in response to the observation starting point of the target object being at the origin of the vehicle coordinate system, the sight angle corresponding to each observation point is obtained based on the three-dimensional coordinates of each observation point in the vehicle coordinate system, and the sight angle corresponding to each observation point is determined as the viewing angle information of the target object in the corresponding collected image; in response to the observation starting point of the target object being at other points other than the origin of the vehicle coordinate system, the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system are obtained, and the observation vector corresponding to each observation point is obtained according to the three-dimensional coordinates of each observation point in the vehicle coordinate system and the three-dimensional coordinates of the observation starting point of the target object, and the vector angle of each observation vector is determined as the viewing angle information of the target object in the corresponding collected image. On the one hand, the viewing angle information of the target object in the corresponding collected image is obtained based on the three-dimensional coordinates of each observation point in the vehicle coordinate system and the three-dimensional coordinates of the observation starting point of the target object, so as to realize the collection of sight data. On the other hand, for the observation starting point of the target object being at the origin of the vehicle coordinate system, only the three-dimensional coordinates of the observation point are needed to obtain the viewing angle information of the target object in the corresponding collected image, and the process is simple. Thus, an efficient and high-precision sight data collection method is provided.

[0116] In one embodiment, the line of sight data collection method of the present application is applied to the training of the line of sight estimation network model, and the line of sight estimation network model can be applied to the distraction alarm or line of sight landing point detection of the intelligent driving cockpit. Specifically, the line of sight data collection method of the present application is applied to the training process of the line of sight estimation network model, which includes the following steps: obtaining the line of sight data set to be trained, the line of sight data set to be trained includes the collected image samples obtained by the line of sight data collection method of the present application and the sample labels of each collected image, and the sample labels represent the perspective information corresponding to the collected image; inputting the collected image samples into the line of sight estimation network model to be trained to obtain the perspective information estimation value output by the line of sight estimation network model to be trained; calculating the loss function based on the sample labels of the collected image samples and the corresponding perspective information estimation value; continuing to train the line of sight estimation network model to be trained with the goal of convergence of the loss function until a line of sight estimation network model that meets the requirements is obtained.

[0117] It can be seen that the line of sight estimation network model is obtained by using the line of sight data acquisition method of the present application to obtain the collected images and the view information corresponding to each collected image for model training. Therefore, the line of sight data acquisition method of the present application can provide a large number of low-cost data samples for model training, which facilitates the training of the line of sight estimation network model.

[0118] Fig.10 FIG. 1 is a block diagram of a sight line data acquisition device shown in an exemplary embodiment of the present application. Fig.10As shown, the exemplary sight line data acquisition device 1000 includes an acquisition module 1010, a vehicle coordinate system construction module 1020, a coordinate conversion module 1030 and a viewing angle information determination module 1040. Specifically:

[0119] The acquisition module 1010 is used to acquire image data, wherein the image data includes acquired images and sight line information of target objects in each acquired image, wherein the sight line information includes an observation point of the target object.

[0120] The vehicle coordinate system construction module 1020 is used to construct a coordinate system according to the position information of the target object to obtain the vehicle coordinate system of the vehicle where the target object is located.

[0121] The coordinate conversion module 1030 is used to convert the coordinates of each observation point into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0122] The viewing angle information determination module 1040 is used to determine the viewing angle information of the target object in the corresponding captured image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system.

[0123] In this exemplary sight data acquisition device, image data is acquired, and a coordinate system is constructed according to the position information of the target object of the acquired image in the image data to obtain the vehicle coordinate system of the vehicle where the target object is located; then the coordinates of the observation points of the target object in each acquired image are converted to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system; then the perspective information of the target object in the corresponding acquired image is determined according to the three-dimensional coordinates of each observation point in the vehicle coordinate system. Since the coordinates of the clear observation points are converted, and the perspective information of the target object is obtained according to the three-dimensional coordinates of the observation points after the coordinate conversion, the error caused by the sight source area is avoided, so that the data quality is higher.

[0124] Among them, the functions of each module can be found in the embodiment of the line of sight data collection method, which will not be repeated here.

[0125] In order to implement the sight data collection method of the above embodiment, the present application proposes another electronic device, which can be seen in detail in Fig.11 , Fig.11 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 1100 includes a memory 1101 and a processor 1102, wherein the memory 1101 and the processor 1102 are coupled. The memory 1101 is used to store program data, and the processor 1102 is used to execute the program data to implement the line of sight data collection method of the above embodiment.

[0126] In this embodiment, the processor 1102 may also be referred to as a CPU (Central Processing Unit). The processor 1102 may be an integrated circuit chip having signal processing capabilities. The processor 1102 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 1102 may also be any conventional processor, etc.

[0127] The present application also provides a computer-readable storage medium, such as Fig.12 As shown, the computer-readable storage medium 1200 is used to store program data 1201. When the program data 1201 is executed by the processor, it is used to implement the line of sight data collection method in the method embodiment of the present application.

[0128] The method involved in the embodiment of the line of sight data collection method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0129] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for collecting sight line data, characterized in that: The method comprises: Capture image data, the image data including the captured images and sight line information of the target object in each captured image, the sight line information including the observation point of the target object, the captured images also including a marked area where the hand of the target object is located, the marked area including the coordinates of the observation point represented by the gesture shown by the hand; Constructing a coordinate system according to the position information of the target object to obtain a vehicle coordinate system of the vehicle where the target object is located; Convert the coordinates of each observation point to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system; Determine the viewing angle information of the target object in the corresponding captured image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system; The method further comprises: Constructing a coordinate system according to the position information of the target object to obtain a spatial coordinate system; The step of converting the coordinates of each observation point into the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system includes: Mapping the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system; Mapping the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system; The driving direction of the vehicle where the target object is located is the Z axis in the vehicle coordinate system, and the vehicle plane where the target object is located is the XY plane in the vehicle coordinate system.

2. The sight line data collection method according to claim 1, characterized in that: The step of mapping the coordinates of each observation point to the spatial coordinate system to obtain the spatial coordinates of each observation point in the spatial coordinate system comprises: Acquire first conversion data; the first conversion data represents a mapping relationship between the observation point coordinates and the spatial coordinate system; The ratio between the coordinates of each observation point and the first conversion data is determined as the spatial coordinates of each observation point in the spatial coordinate system.

3. The sight line data collection method according to claim 1, characterized in that: The step of mapping the spatial coordinates of each observation point in the spatial coordinate system to the vehicle coordinate system to obtain the three-dimensional coordinates of each observation point in the vehicle coordinate system comprises: Acquire second conversion data; the second conversion data represents a mapping relationship between the space coordinate system and the vehicle coordinate system; The product of the spatial coordinates of each observation point in each of the spatial coordinate systems and the second conversion data is determined as the three-dimensional coordinates of each observation point in the vehicle coordinate system.

4. The sight line data collection method according to claim 3, characterized in that: The step of obtaining the second conversion data comprises: Acquiring posture information when the first acquisition device acquires a calibration image, wherein the acquisition direction of the calibration image is different from the acquisition direction of the acquired image; Determine a rotation matrix according to the posture information when the first acquisition device acquires the calibration image; Acquire a spatial translation vector according to the position information of the target object in the calibration image and the origin of the vehicle coordinate system; The second conversion data is acquired according to the rotation matrix and the spatial translation vector.

5. The sight line data collection method according to claim 1, characterized in that: The step of determining the viewing angle information of the target object in the corresponding captured image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system comprises: In response to the observation starting point of the target object being at the origin of the vehicle coordinate system, acquiring a sight angle corresponding to each observation point based on the three-dimensional coordinates of each observation point in the vehicle coordinate system; The sight angle corresponding to each observation point is determined as the viewing angle information of the target object in the corresponding captured image.

6. The sight line data collection method according to claim 1, characterized in that: The step of determining the viewing angle information of the target object in the corresponding captured image according to the three-dimensional coordinates of each observation point in the vehicle coordinate system also includes: In response to the observation starting point of the target object being at a point other than the origin of the vehicle coordinate system, acquiring the three-dimensional coordinates of the observation starting point of the target object in the vehicle coordinate system; Acquire an observation vector corresponding to each observation point according to the three-dimensional coordinates of each observation point in the vehicle coordinate system and the three-dimensional coordinates of the observation starting point of the target object; The vector angle of each observation vector is determined as the viewing angle information of the target object in the corresponding acquired image.

7. The sight line data collection method according to claim 1, characterized in that: The step of constructing a coordinate system according to the position information of the target object to obtain a vehicle coordinate system of the vehicle where the target object is located comprises: Acquire calibration image information, where the calibration image information includes the calibration image and direction information in which the first acquisition device acquires the calibration image; A coordinate system is constructed based on the direction information of the calibration image collected by the first acquisition device and the location information of the first acquisition device to obtain a vehicle coordinate system of the vehicle where the target object is located.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 7.

Images