Gaze direction data collection method, device, equipment, and storage medium

Abstract

This application discloses gaze direction data collection methods, devices, equipment, and storage media. [Solution] This method includes the steps of collecting an object image of a target object, which is an object that a user is gazing at, using a first deep camera, and determining the three-dimensional object coordinates of the target object on the first deep camera coordinate system based on the object image; collecting an image of the user's face using a second deep camera, and determining the three-dimensional eye coordinates of the user's eye on the second deep camera coordinate system based on the face image; determining the object coordinates and eye coordinates on each collecting camera coordinate system of the object three-dimensional coordinates and the eye three-dimensional coordinates; and determining the user's gaze direction data based on the object coordinates and eye coordinates on each collecting camera coordinate system.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application with application number 202211305842.9, filed on October 24, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of data collection technology, and in particular to a method, device, apparatus and storage medium for collecting gaze direction data. [Background technology]

[0003] Conventional gaze direction data collection methods generally use a single collection method, which can only collect one gaze direction data each time, resulting in low data collection efficiency and inability to meet the training needs of deep learning algorithms. Moreover, due to the influence of the viewing angle of the deep camera, there are significant limitations on the location of data collection, and the accuracy of the collected gaze direction data is low.

[0004] The above content is merely intended to aid in the technical understanding of the present application, and is not an admission that the above content is prior art. Summary of the Invention [Problem to be solved by the invention]

[0005] The main objective of the present application is to provide a gaze direction data collection method, device, equipment and storage medium to solve the technical problems in the prior art of low collection efficiency and accuracy of gaze direction data. [Means for solving the problem]

[0006] In order to achieve the above object, the present application provides a gaze direction data collection method, the gaze direction data collection method comprising: Collecting an object image of a target object, which is an object that a user is gazing at, by a first deep camera, and determining three-dimensional object coordinates of the target object on the first deep camera coordinate system based on the object image; Collecting a facial image of the user using a second deep camera, and determining three-dimensional eye coordinates of the user's eyes on a second deep camera coordinate system based on the facial image; determining object coordinates and eye coordinates on respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates; and determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system.

[0007] In one embodiment, the step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object 3D coordinates and the eye 3D coordinates comprises: When the first deep camera coordinate system and the second deep camera coordinate system are unified, determining object coordinates on each collection camera coordinate system of the object three-dimensional coordinates according to an extrinsic parameter matrix from the first deep camera coordinate system to each collection camera coordinate system matrix; and determining eye coordinates on each of the collecting camera coordinate systems of the three-dimensional eye coordinates based on an external parameter matrix from the second deep camera coordinate system to each of the collecting camera coordinate system matrices.

[0008] In one embodiment, the step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates comprises: When the first deep camera coordinate system and the second deep camera coordinate system are not unified, determining an object calibration three-dimensional coordinate of the object three-dimensional coordinate on the first preset calibration plate coordinate system based on an extrinsic parameter matrix from the first deep camera coordinate system to a first preset calibration plate coordinate system; determining eye calibration three-dimensional coordinates on the first preset calibration plate coordinate system of the eye three-dimensional coordinates based on an extrinsic parameter matrix from the second deep camera coordinate system to the first preset calibration plate coordinate system; The method further includes a step of determining object coordinates and eye coordinates on each collection camera coordinate system of the object calibration three-dimensional coordinates and the eye calibration three-dimensional coordinates based on an external parameter matrix from the first predetermined calibration plate coordinate system to each collection camera coordinate system matrix.

[0009] In one embodiment, before the step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates, determining a first extrinsic parameter matrix from the first deep camera coordinate system to a second preset calibration plate coordinate system, and determining a second extrinsic parameter matrix from the second deep camera coordinate system to the second preset calibration plate coordinate system; determining first three-dimensional object coordinates on the first deep camera coordinate system of a target reference object and determining second three-dimensional object coordinates on the second deep camera coordinate system of a target reference object; The method further includes determining whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first three-dimensional object coordinates, the second three-dimensional object coordinates, the first external parameter matrix, and the second external parameter matrix.

[0010] In one embodiment, the steps of determining first three-dimensional object coordinates on the first deep camera coordinate system of a target reference object and determining second three-dimensional object coordinates on the second deep camera coordinate system of a target reference object include: collecting a first reference object image of a target reference object by a first deep camera, and determining first three-dimensional object coordinates of the target reference object in the first deep camera coordinate system based on the first reference object image; The method includes a step of collecting a second reference object image of the target reference object by a second deep camera, and determining second three-dimensional object coordinates of the target reference object on the second deep camera coordinate system based on the second reference object image.

[0011] In one embodiment, the step of determining whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first three-dimensional object coordinates, the second three-dimensional object coordinates, the first extrinsic parameter matrix, and the second extrinsic parameter matrix includes: transforming the first three-dimensional object coordinates into first object coordinates on the second preset calibration plate coordinate system based on the first extrinsic parameter matrix; transforming the second three-dimensional object coordinates into second object coordinates on the second preset calibration plate coordinate system based on the second extrinsic parameter matrix; If the first object coordinates and the second object coordinates match, determining that the first deep camera coordinate system and the second deep camera coordinate system are unified; If the first object coordinates and the second object coordinates do not match, determining that the first deep camera coordinate system and the second deep camera coordinate system are not unified.

[0012] In one embodiment, the step of determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system comprises: determining a line-of-sight vector on each of the acquisition camera coordinate systems based on the object coordinates and the eye coordinates on each of the acquisition camera coordinate systems; and determining gaze direction data of the user based on a gaze vector on each collection camera coordinate system.

[0013] In order to achieve the above object, the present application provides a gaze direction data collection device, the gaze direction data collection device comprising: a first coordinate determination module for collecting an object image of a target object, which is an object that a user is gazing at, by a first deep camera, and determining three-dimensional object coordinates of the target object in a first deep camera coordinate system based on the object image; a second coordinate determination module for collecting a face image of the user by a second deep camera and determining three-dimensional eye coordinates of the user's eye on a second deep camera coordinate system based on the face image; a third coordinate determination module for determining object coordinates and eye coordinates on respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates; and a direction data determination module for determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system.

[0014] To achieve the above object, the present application further provides a gaze direction data collection device, the device including: a memory; a processor; and a gaze direction data collection program stored in the memory and executable on the processor, the gaze direction data collection program being configured to implement the steps of the gaze direction data collection method described above.

[0015] To achieve the above object, the present application further provides a storage medium having a gaze direction data collection program stored therein, and when the gaze direction data collection program is executed by a processor, the steps of the gaze direction data collection method described above are realized. [Effects of the Invention]

[0016] In the present application, an object image of a target object, which is an object that a user is gazing at, is collected by a first deep camera, and based on the object image, the three-dimensional object coordinates of the target object on the first deep camera coordinate system are determined, an image of the user's face is collected by a second deep camera, and based on the face image, the three-dimensional eye coordinates of the user's eye on the second deep camera coordinate system are determined, and the object coordinates and eye coordinates of the object three-dimensional coordinates and the eye three-dimensional coordinates on each collecting camera coordinate system are determined, and based on the object coordinates and eye coordinates on each collecting camera coordinate system, the user's gaze direction data is determined. In the present application, an object image of a target object and a face image of a user are collected by a first deep camera and a second deep camera, respectively; based on the object image and the face image, three-dimensional object coordinates and three-dimensional eye coordinates are determined; the three-dimensional object coordinates and three-dimensional eye coordinates are converted into the coordinate systems of each collecting camera to obtain the object coordinates and eye coordinates; and gaze direction data is determined based on the object coordinates and eye coordinates on the coordinate systems of each collecting camera. The collected gaze range is enlarged, making the gaze direction data more accurate; and multiple gaze direction data can be collected each time, improving the efficiency of gaze direction data collection. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a structural schematic diagram of a gaze direction data collection device of a hardware execution environment according to an embodiment of the present application; [Figure 2] 1 is a flowchart of a first embodiment of a gaze direction data collection method of the present application. [Figure 3] FIG. 2 is a schematic diagram of collecting gaze direction data in one embodiment of the gaze direction data collection method of the present application. [Figure 4] 10 is a flowchart of a second embodiment of the gaze direction data collection method of the present application. [Figure 5] FIG. 2 is a schematic diagram of a camera capturing an image of a first preset calibration plate in an embodiment of the gaze direction data collection method of the present application; [Figure 6] 10 is a flowchart of a third embodiment of the gaze direction data collection method of the present application. [Figure 7]FIG. 10 is a schematic diagram illustrating the second preset reference plate being photographed by the deep camera in an embodiment of the gaze direction data collection method of the present application; [Figure 8] 1 is a schematic diagram of a target reference object being photographed by a deep camera in an embodiment of the gaze direction data collection method of the present application; FIG. [Figure 9] 1 is a structural block diagram of a first embodiment of a gaze direction data collection device of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0018] The realization of the object, the functional features and advantages of the present application will be further explained in connection with the embodiments and with reference to the drawings. It should be understood that the specific embodiments described herein are used only to illustrate the present application and are not intended to limit the present application.

[0019] Please refer to FIG. 1, which is a schematic diagram of a gaze direction data collection device structure of a hardware execution environment according to an embodiment of the present application.

[0020] As shown in FIG. 1 , the gaze direction data collection device may include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display and an input unit, such as a keyboard. The user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may include a standard wired interface and a wireless interface (e.g., a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a magnetic disk memory. The memory 1005 may be a storage device independent of the processor 1001.

[0021] As will be appreciated by those skilled in the art, the structure shown in FIG. 1 does not constitute a limitation on gaze direction data collection devices, which may include more or fewer components than shown, or may include combinations of components or configurations with different component arrangements.

[0022] As shown in FIG. 1, the memory 1005 of the storage medium may include an operating system, a network communication module, a user interface module, and a gaze direction data collection program.

[0023] 1 , the network interface 1004 is mainly used for data communication with a network server. The user interface 1003 is mainly used for data interaction with a user. The processor 1001 and memory 1005 in the gaze direction data collection device of the present application may be provided in the gaze direction data collection device, and the gaze direction data collection device invokes the gaze direction data collection program stored in the memory 1005 by the processor 1001 to execute the gaze direction data collection method according to the embodiment of the present application.

[0024] An embodiment of the present application provides a gaze direction data collection method, and reference is made to FIG. 2, which is a flowchart of a first embodiment of the gaze direction data collection method of the present application.

[0025] In this embodiment, the gaze direction data collection method includes the following steps: Step S10: An object image of a target object, which is an object that a user is gazing at, is collected by a first deep camera, and based on the object image, the three-dimensional object coordinates of the target object in the first deep camera coordinate system are determined.

[0026] It should be noted that the execution entity of this embodiment may be a computing service device having data processing, network communication and program execution functions such as a tablet, a personal computer, a mobile phone, or an electronic device capable of realizing the above functions, a gaze direction data collection device, etc. Hereinafter, this embodiment and each of the following embodiments will be described using a gaze direction data collection device (referred to as a collection device) as an example.

[0027] As can be understood, the deep camera for collecting the object image of the target object is collectively referred to as the first deep camera, the number of the first deep camera may be one or more, the object three-dimensional coordinates may be the three-dimensional coordinates of the target object that the user gazes at in the object image on the first deep camera coordinate system, the first deep camera can automatically obtain the three-dimensional coordinates of the target object in this image after capturing the object image, the number of the target objects may be more than one, the target object may be an object provided on the background board, or may be a spot or other object irradiated on the background board by a laser device, and this embodiment is not limited thereto.

[0028] In this embodiment, the object on the background board for the user to gaze at may be referred to as a gaze point. The gaze point may be fixed on the background board, and a deep image of the background board may be collected by a first deep camera. Based on the deep image, the three-dimensional coordinates of each gaze point on the background board may be predetermined. When collecting gaze direction data, the gaze point of the object the user gazes at may be determined, and the three-dimensional coordinates of the object corresponding to the gaze point may be determined based on the predetermined three-dimensional coordinates of the object. Conventional collection solutions typically use a moving gaze point, which requires coordinate calibration for each collected picture, which takes a long time. By using a fixed gaze point, the coordinates of the gaze point in each picture collected by the first deep camera are fixed, and only one calibration of the gaze point coordinate is required. Subsequent data can be directly multiplexed, realizing automated coordinate marking and reducing the time required for data marking. In this embodiment, the gaze point may be a moving gaze point, and when collecting gaze direction data, the target gaze point is collected by the first deep camera, so that the collected gaze range is wider and the collected gaze direction data is more abundant, thereby improving the accuracy of subsequent deep learning algorithm training. Depending on the specific scene, it can be determined whether the object on the background board is fixed or moving, and this embodiment is not limited thereto.

[0029] Step S20: A face image of the user is collected by a second deep camera, and three-dimensional eye coordinates of the user's eyes on the second deep camera coordinate system are determined based on the face image.

[0030] As can be understood, the deep camera for collecting the user's facial image is collectively referred to as the second deep camera, and the number of the second deep cameras may be one or more, and the three-dimensional coordinates of the eyes may be the three-dimensional coordinates of the eyes in the user's facial image on the second deep camera coordinate system when the user gazes at the target object, and the second deep camera can automatically obtain the three-dimensional coordinates of the eyes in the image after capturing the user's facial image, and the three-dimensional coordinates of the eyes include the three-dimensional coordinates of the left eye and the three-dimensional coordinates of the right eye, and can be selected based on the specific scene, and this embodiment is not limited thereto.

[0031] Step S30: The object coordinates and eye coordinates on the respective acquisition camera coordinate systems are determined.

[0032] As can be understood, the collection camera may be a camera for collecting user pictures, including a red-green-blue camera (RGB camera) and an infrared camera. Determining the object coordinates and eye coordinates on each collection camera coordinate system of the object three-dimensional coordinates and the eye three-dimensional coordinates may be as follows: transform the object three-dimensional coordinates on the first deep camera coordinate system into each collection camera coordinate system to obtain the object coordinates on each collection camera coordinate system; transform the eye three-dimensional coordinates on the second deep camera coordinate system into each collection camera coordinate system to obtain the eye coordinates on each collection camera coordinate system; after the coordinate transformation, the object coordinates and eye coordinates on the collection camera coordinate system corresponding to the user pictures collected by each collection camera can be obtained, and the user pictures collected by the collection cameras may be user face pictures.

[0033] Step S40: Determine the gaze direction data of the user based on the object coordinates and eye coordinates on each collection camera coordinate system.

[0034] As can be understood, determining the user's gaze direction data based on the object coordinates and eye coordinates on each collection camera coordinate system may also be determining gaze direction data corresponding to user pictures captured by each collection camera based on the object coordinates and eye coordinates on each collection camera coordinate system, for example, if the number of collection cameras is N, gaze direction data corresponding to N user pictures can be obtained.

[0035] In a specific implementation, the collection device collects an object image of a target object gazed upon by a user using a first deep camera, and determines the object's three-dimensional object coordinates in the first deep camera coordinate system based on the target object image in the object image. A face image of the user gazing at the target object is collected using a second deep camera, and determines the eye's three-dimensional eye coordinates in the second deep camera coordinate system based on the eye image in the face image. After the object three-dimensional coordinates and eye three-dimensional coordinates are acquired by the deep camera, user face pictures are further collected by multiple collection cameras to acquire multiple user face pictures. The object three-dimensional coordinates in the first deep camera coordinate system are transformed into each collection camera coordinate system to acquire multiple object coordinates. The eye three-dimensional coordinates in the second deep camera coordinate system are transformed into each collection camera coordinate system to acquire multiple eye coordinates. Based on the object coordinates and eye coordinates in each collection camera coordinate system, gaze direction data corresponding to each user face picture is determined.

[0036] For example, referring to FIG. 3, FIG. 3 is a schematic diagram of collecting gaze direction data, in which the number of the first deep camera and the second deep camera are both 1 and are represented by A and B, respectively. The first deep camera A is disposed behind the user, and the second deep camera B is disposed in front of the user. A background wall is provided with a plurality of objects for the user to gaze at. The user is in front of the background wall. N collection cameras are disposed between the user and the background wall. The collection cameras include RGB cameras and infrared cameras, and are disposed in front of the user (for example, within a range of 0.5 m to 1 m in front of the user) and distributed within a preset angle range in front of the user (for example, distributed within a range of ±45 degrees around the user). The first deep camera A and the background wall maintain a first preset distance (for example, the first preset distance is set to 1.5 m to 3 m), and the second deep camera B and the user maintain a second preset distance (for example, the second preset distance is set to 0.5 m to 1 m). A deep image of the background wall is captured by a first deep camera, and the three-dimensional coordinates of each object on the background wall can be determined based on the captured deep image in the first deep camera coordinate system. The three-dimensional coordinates of each object can then be directly multiplexed, saving data marking time and further improving gaze direction data collection efficiency. If the object being gazed at by a user is object 1 on the background wall, the three-dimensional coordinates of object 1 on the first deep camera coordinate system are obtained, and an image of the user's face is captured by a second deep camera B. Based on the image of the eye in the face image, the three-dimensional coordinates of the eye in the second deep camera coordinate system are determined when the user gazes at object 1. When an image is collected by the deep camera, N user face pictures are further collected by the N collection cameras, and the three-dimensional coordinates of the object and the three-dimensional coordinates of the eye on the deep camera coordinate system are transformed into the coordinate systems of the N collection cameras to obtain the coordinates of the object and the coordinates of the eye on the coordinate systems of the N collection cameras. Based on the coordinates of the object and the coordinates of the eye on each collection camera coordinate system, gaze direction data corresponding to the N user face pictures is determined.

[0037] In one embodiment, in order to improve the accuracy of the collected gaze direction data, step S30 includes, when the first deep camera coordinate system and the second deep camera coordinate system are unified, a step of determining object coordinates on each collection camera coordinate system of the object's three-dimensional coordinates based on an external parameter matrix from the first deep camera coordinate system to each collection camera coordinate system matrix, and a step of determining eye coordinates on each collection camera coordinate system of the eye's three-dimensional coordinates based on an external parameter matrix from the second deep camera coordinate system to each collection camera coordinate system matrix.

[0038] As can be understood, the extrinsic parameter matrices from the first deep camera coordinate system and the second deep camera coordinate system to each collection camera coordinate system matrix may be obtained by extrinsic parameter calibration, and the parameter matrices from the first deep camera coordinate system and the second deep camera coordinate system to each collection camera coordinate system matrix are different.

[0039] In one embodiment, in order to improve the efficiency of collecting gaze direction data, step S40 includes a step of determining a gaze vector on each collection camera coordinate system based on object coordinates and eye coordinates on each collection camera coordinate system, and a step of determining the user's gaze direction data based on the gaze vector on each collection camera coordinate system.

[0040] In a specific implementation, for example, extrinsic parameter calibration is performed between the collection camera and the second deep camera to obtain an extrinsic parameter matrix (R2, T2) from the second deep camera coordinate system to each collection camera coordinate system matrix. Extrinsic parameter calibration is performed between the collection camera and the first deep camera to obtain an extrinsic parameter matrix (R1, T1) from the first deep camera coordinate system to each collection camera coordinate system matrix. A first deep camera A captures a deep image of a background wall to obtain three-dimensional object coordinates in the first deep camera coordinate system for all objects on the background wall. When a user gazes at object 1 on the background wall, the three-dimensional object coordinate of object 1 in the first deep camera coordinate system may be represented as deep_cam_point1. A second deep camera B captures a deep image of the user's face to obtain the three-dimensional left eye coordinate of the user's left eye: deep_cam_left_eye1 in the second deep camera coordinate system. The three-dimensional coordinates in the deep camera coordinate system can be converted to coordinates in the collection camera coordinate system based on Equation 1.

[0041]

number

[0042] In this embodiment, an object image of a target object, which is an object that a user is gazing at, is collected by a first deep camera, and based on the object image, the three-dimensional object coordinates of the target object on the first deep camera coordinate system are determined, an image of the user's face is collected by a second deep camera, and based on the face image, the three-dimensional eye coordinates of the user's eye on the second deep camera coordinate system are determined, and the object coordinates and eye coordinates of the object three-dimensional coordinates and the eye three-dimensional coordinates on each collection camera coordinate system are determined, and based on the object coordinates and eye coordinates on each collection camera coordinate system, the user's gaze direction data is determined. In this embodiment, an object image of the target object and a face image of the user are collected by a first deep camera and a second deep camera, respectively. Based on the object image and the face image, the three-dimensional coordinates of the object and the three-dimensional coordinates of the eye are determined. The three-dimensional coordinates of the object and the three-dimensional coordinates of the eye are converted into the coordinate systems of each collecting camera to obtain the object coordinates and the eye coordinates. Based on the object coordinates and the eye coordinates on the coordinate systems of each collecting camera, gaze direction data is determined. The collected gaze range is enlarged, making the gaze direction data more accurate. Moreover, multiple gaze direction data can be collected each time, improving the efficiency of gaze direction data collection.

[0043] Please refer to FIG. 4, which is a flowchart of a second embodiment of the gaze direction data collection method of the present application.

[0044] Based on the above first embodiment, in this embodiment, the step S30 includes the following steps:

[0045] Step S301: If the first deep camera coordinate system and the second deep camera coordinate system are not unified, determine the object calibration three-dimensional coordinates of the object three-dimensional coordinates on the first preset calibration plate coordinate system based on the external parameter matrix from the first deep camera coordinate system to a first preset calibration plate coordinate system.

[0046] As can be understood, if the first deep camera coordinate system and the second deep camera coordinate system are not unified, directly converting the three-dimensional coordinates on the deep camera coordinate system into the camera coordinate system of each collecting camera will result in the collected gaze direction data being inaccurate, and the first preset calibration plate coordinate system may be a coordinate system corresponding to the preset first calibration plate.

[0047] Step S302: Based on the external parameter matrix from the second deep camera coordinate system to the first preset calibration plate coordinate system, determine the eye calibration three-dimensional coordinates of the eye three-dimensional coordinates on the first preset calibration plate coordinate system.

[0048] Step S303: Based on the external parameter matrix from the first preset calibration plate coordinate system to each collection camera coordinate system matrix, determine the object coordinates and eye coordinates on each collection camera coordinate system of the object calibration three-dimensional coordinates and the eye calibration three-dimensional coordinates.

[0049] In a specific implementation, referring to FIG. 5, FIG. 5 is a schematic diagram of photographing a first preset calibration plate by a camera, where the preset calibration plate is a grid calibration plate. The grid calibration plate is photographed by the deep camera and each acquisition camera, and extrinsic parameter calibration is performed between the deep camera and the grid calibration plate to obtain an extrinsic parameter matrix from the deep camera coordinate system to the grid calibration plate coordinate system. External parameter calibration is performed between each acquisition camera and the grid calibration plate to obtain an extrinsic parameter matrix from each acquisition camera coordinate system to the grid calibration plate coordinate system. The three-dimensional coordinates of the object in the first deep camera coordinate system are transformed into the grid calibration plate coordinate system using Equation 2 to obtain the three-dimensional coordinates of the object calibration in the grid calibration plate coordinate system. If the three-dimensional coordinates of the eye are the three-dimensional coordinates of the left eye, the three-dimensional coordinates of the left eye in the second deep camera coordinate system are transformed into the grid calibration plate coordinate system using Equation 2 to obtain the three-dimensional coordinates of the left eye calibration in the grid calibration plate coordinate system. The object calibration 3D coordinates on the grid calibration plate coordinate system are transformed into the coordinates of each collecting camera according to Equation 1, thereby obtaining the object coordinates on each collecting camera coordinate system. The left eye 3D coordinates on the grid calibration plate coordinate system are transformed into the coordinates of each collecting camera according to Equation 1, thereby obtaining the left eye 3D coordinates on each collecting camera coordinate system. The left eye gaze direction vector can be determined based on the object coordinates and the left eye 3D coordinates on each collecting camera coordinate system. The process of determining the right eye gaze direction vector is similar to that of the left eye gaze direction vector, so no further description is given in this embodiment.

[0050]

number

[0051] In this embodiment, when the first deep camera coordinate system and the second deep camera coordinate system are not unified, the three-dimensional coordinates of the object on the first deep camera coordinate system are converted into the first preset calibration plate coordinate system, the three-dimensional coordinates of the eye on the second deep camera coordinate system are converted into the first preset calibration plate coordinate system, and the coordinates on the first preset calibration plate coordinate system are converted into each collecting camera coordinate system to obtain the object coordinates and the eye coordinates.When the deep camera coordinate systems are not unified, the deep camera coordinates are first unified and then the gaze direction data is determined, so that the accuracy of the collected gaze direction data can be improved.

[0052] Please refer to FIG. 6, which is a flowchart of a third embodiment of the gaze direction data collection method of the present application.

[0053] Based on the above embodiments, in this embodiment, before step S30, the method further includes the following steps:

[0054] Step S01: Determine a first extrinsic parameter matrix from the first deep camera coordinate system to a second preset calibration plate coordinate system, and determine a second extrinsic parameter matrix from the second deep camera coordinate system to the second preset calibration plate coordinate system.

[0055] As can be understood, the second preset calibration plate coordinate system may be a coordinate system corresponding to the second preset calibration plate, and the first extrinsic parameter matrix from the first deep camera coordinate system to the second preset calibration plate coordinate system and the second extrinsic parameter matrix from the second deep camera coordinate system to the second preset calibration plate coordinate system may be obtained by extrinsic parameter calibration.

[0056] Step S02: Determine a first three-dimensional object coordinate of a target reference object on the first deep camera coordinate system, and determine a second three-dimensional object coordinate of a target reference object on the second deep camera coordinate system.

[0057] As can be understood, the target reference object may be an object for determining whether the first deep camera coordinate system and the second deep camera coordinate system are unified, the first three-dimensional object coordinates may be coordinates on the first deep camera coordinate system of the target reference object, and the second three-dimensional object coordinates may be coordinates on the second deep camera coordinate system of the target reference object.

[0058] Step S03: Determine whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first 3D object coordinates, the second 3D object coordinates, the first external parameter matrix, and the second external parameter matrix.

[0059] In a specific implementation, the first and second extrinsic parameter matrices are determined from the first and second deep cameras to a second preset calibration plate coordinate system through extrinsic parameter calibration. The first and second 3D object coordinates of the target reference object are determined in the first and second deep camera coordinate systems, respectively. The first 3D object coordinates are transformed into the second preset calibration plate coordinate system based on the first extrinsic parameter matrix, and the second 3D object coordinates are transformed into the second preset calibration plate coordinate system based on the second extrinsic parameter matrix. Whether the first and second deep camera coordinate systems are consistent is determined based on the coordinate system under the second preset calibration plate.

[0060] In one embodiment, to determine whether the coordinate systems of the deep cameras are unified, step S02 includes the steps of collecting a first reference object image of the target reference object by a first deep camera and determining first three-dimensional object coordinates of the target reference object on the first deep camera coordinate system based on the first reference object image, and collecting a second reference object image of the target reference object by a second deep camera and determining second three-dimensional object coordinates of the target reference object on the second deep camera coordinate system based on the second reference object image.

[0061] In one embodiment, to determine whether the deep camera coordinate systems are unified, step S03 includes the steps of: transforming the first three-dimensional object coordinates into first object coordinates on the second preset calibration plate coordinate system based on the first extrinsic parameter matrix; transforming the second three-dimensional object coordinates into second object coordinates on the second preset calibration plate coordinate system based on the second extrinsic parameter matrix; determining that the first deep camera coordinate system and the second deep camera coordinate system are unified if the first object coordinates and the second object coordinates are consistent; and determining that the first deep camera coordinate system and the second deep camera coordinate system are not unified if the first object coordinates and the second object coordinates are not consistent.

[0062] As can be seen, the first three-dimensional object coordinates and the second three-dimensional object coordinates may be transformed into the first object coordinates and the second object coordinates by Equation 1.

[0063] In a specific implementation, referring to Figures 7 and 8, Figure 7 is a schematic diagram of photographing a second preset reference plate using a deep camera, and Figure 8 is a schematic diagram of photographing a target reference object using a deep camera. For example, when verifying the unification of coordinate systems for a first deep camera and a second deep camera, a second preset calibration plate is photographed using the first deep camera and the second deep camera, and a first extrinsic parameter matrix and a second extrinsic parameter matrix are obtained by extrinsic parameter calibration from the first deep camera coordinate system and the second deep camera coordinate system to the second preset calibration plate coordinate system. A target reference object is photographed using the first deep camera and the second deep camera, and a first reference object image and a second reference object image are obtained. Based on the first reference image and the second reference image, a first 3D object coordinate in the first deep camera coordinate system and a second 3D object coordinate in the second deep camera coordinate system are determined. The object coordinates in the deep camera coordinate system are transformed into the second preset calibration plate coordinate system using Equation 1 and the first extrinsic parameter matrix to obtain the first object coordinates and the second object coordinates. If the first object coordinates and the second object coordinates are consistent, it is determined that the coordinate systems of the two deep cameras are unified; otherwise, it is determined that the coordinate systems of the two deep cameras are not unified.

[0064] In this embodiment, an external parameter matrix from the first deep camera coordinate system and the second deep camera coordinate system to a second preset calibration plate coordinate system is determined, and the three-dimensional object coordinates on the first deep camera coordinate system and the second deep camera coordinate system are converted to the second preset calibration plate coordinate system based on the external parameter matrix. If the two object coordinates on the second preset calibration plate coordinate system are consistent, it is determined that the two deep camera coordinate systems are unified, which improves the accuracy of the unification judgment of the deep camera coordinate systems and also improves the accuracy of the gaze direction data collected later.

[0065] Note that the embodiments of the present application further propose a storage medium on which a gaze direction data collection program is stored, and when the gaze direction data collection program is executed by a processor, the steps of the gaze direction data collection method described above are realized.

[0066] Please refer to FIG. 9, which is a structural block diagram of a first embodiment of the gaze direction data collection device of the present application.

[0067] As shown in FIG. 9, the gaze direction data collection device proposed in the embodiment of the present application includes: a first coordinate determination module 10 for collecting an object image of a target object, which is an object that a user is gazing at, by a first deep camera, and determining, based on the object image, the object three-dimensional coordinates of the target object in the first deep camera coordinate system; a second coordinate determination module 20 for collecting a face image of the user by a second deep camera and determining three-dimensional eye coordinates of the user's eye on a second deep camera coordinate system based on the face image; a third coordinate determination module 30 for determining object coordinates and eye coordinates on respective acquisition camera coordinate systems of the object 3D coordinates and the eye 3D coordinates; and a direction data determination module 40 for determining gaze direction data of the user based on object coordinates and eye coordinates on each acquisition camera coordinate system.

[0068] In this embodiment, an object image of a target object, which is an object that a user is gazing at, is collected by a first deep camera, and based on the object image, the three-dimensional object coordinates of the target object on the first deep camera coordinate system are determined, an image of the user's face is collected by a second deep camera, and based on the face image, the three-dimensional eye coordinates of the user's eye on the second deep camera coordinate system are determined, and the object coordinates and eye coordinates of the object three-dimensional coordinates and the eye three-dimensional coordinates on each collection camera coordinate system are determined, and based on the object coordinates and eye coordinates on each collection camera coordinate system, the user's gaze direction data is determined. In this embodiment, an object image of the target object and a face image of the user are collected by a first deep camera and a second deep camera, respectively. Based on the object image and the face image, the three-dimensional coordinates of the object and the three-dimensional coordinates of the eye are determined. The three-dimensional coordinates of the object and the three-dimensional coordinates of the eye are converted into the coordinate systems of each collecting camera to obtain the object coordinates and the eye coordinates. Based on the object coordinates and the eye coordinates on the coordinate systems of each collecting camera, gaze direction data is determined. The collected gaze range is enlarged, making the gaze direction data more accurate. Moreover, multiple gaze direction data can be collected each time, improving the efficiency of gaze direction data collection.

[0069] Based on the above-described first embodiment of the gaze direction data collection device of the present application, a second embodiment of the gaze direction data collection device of the present application is proposed.

[0070] In this embodiment, when the first deep camera coordinate system and the second deep camera coordinate system are unified, the third coordinate determination module 30 is further used to determine object coordinates on each collection camera coordinate system of the object three-dimensional coordinates based on an external parameter matrix from the first deep camera coordinate system to each collection camera coordinate system matrix, and to determine eye coordinates on each collection camera coordinate system of the eye three-dimensional coordinates based on an external parameter matrix from the second deep camera coordinate system to each collection camera coordinate system matrix.

[0071] The third coordinate determination module 30 is further used to, when the first deep camera coordinate system and the second deep camera coordinate system are not unified, determine object calibration three-dimensional coordinates on the first preset calibration plate coordinate system of the object three-dimensional coordinates based on an external parameter matrix from the first deep camera coordinate system to a first preset calibration plate coordinate system, determine eye calibration three-dimensional coordinates on the first preset calibration plate coordinate system of the eye three-dimensional coordinates based on an external parameter matrix from the second deep camera coordinate system to the first preset calibration plate coordinate system, and determine object coordinates and eye coordinates on each collection camera coordinate system of the object calibration three-dimensional coordinates and the eye calibration three-dimensional coordinates based on an external parameter matrix from the first preset calibration plate coordinate system to each collection camera coordinate system matrix.

[0072] The third coordinate determination module 30 is further used to determine a first extrinsic parameter matrix from the first deep camera coordinate system to a second preset calibration plate coordinate system, determine a second extrinsic parameter matrix from the second deep camera coordinate system to the second preset calibration plate coordinate system, determine first three-dimensional object coordinates of a target reference object on the first deep camera coordinate system, and determine second three-dimensional object coordinates of a target reference object on the second deep camera coordinate system, and determine whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first three-dimensional object coordinates, the second three-dimensional object coordinates, the first extrinsic parameter matrix, and the second extrinsic parameter matrix.

[0073] The third coordinate determination module 30 is further used to collect a first reference object image of the target reference object by a first deep camera, and determine first three-dimensional object coordinates of the target reference object on the first deep camera coordinate system based on the first reference object image, and to collect a second reference object image of the target reference object by a second deep camera, and determine second three-dimensional object coordinates of the target reference object on the second deep camera coordinate system based on the second reference object image.

[0074] The third coordinate determination module 30 is further used to transform the first three-dimensional object coordinates into first object coordinates on the second preset calibration plate coordinate system based on the first extrinsic parameter matrix, transform the second three-dimensional object coordinates into second object coordinates on the second preset calibration plate coordinate system based on the second extrinsic parameter matrix, and determine that the first deep camera coordinate system and the second deep camera coordinate system are unified if the first object coordinates and the second object coordinates are consistent, and determine that the first deep camera coordinate system and the second deep camera coordinate system are not unified if the first object coordinates and the second object coordinates are not consistent.

[0075] The direction data determination module 40 is further used to determine a gaze vector on each collection camera coordinate system based on the object coordinates and eye coordinates on each collection camera coordinate system, and to determine the user's gaze direction data based on the gaze vector on each collection camera coordinate system.

[0076] Other examples or specific embodiments of the gaze direction data collection device of the present application may refer to the above-mentioned method examples, and will not be further described here.

[0077] It should be explained that, as used herein, the terms "comprises," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that includes a set of elements includes not only those elements but also other elements not expressly listed, or includes elements inherent in such a process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising one or more" does not exclude the presence of other identical elements in a process, method, article, or system that includes this element.

[0078] The numbering of the examples in the present application above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments.

[0079] From the above description of the embodiments, it will be clear to those skilled in the art that the methods of the above embodiments can be realized by adding a required general-purpose hardware platform to software, or of course by hardware, and in many cases, the former is a preferred embodiment. Based on this understanding, the technical proposal of the present application, or the portion that contributes to the prior art, may be expressed in the form of a software product. This computer software product is stored in a storage medium (e.g., read-only memory / random access memory, magnetic disk, optical disk) and includes a plurality of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, a network device, etc.) to execute the methods described in each embodiment of the present application.

[0080] The above are merely optional embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A method for collecting gaze direction data, comprising: Collecting an object image of a target object, which is an object that a user is gazing at, by a first deep camera, and determining three-dimensional object coordinates of the target object on the first deep camera coordinate system based on the object image; Collecting a facial image of the user using a second deep camera, and determining three-dimensional eye coordinates of the user's eyes on a second deep camera coordinate system based on the facial image; determining object coordinates and eye coordinates on respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates; determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system.

2. The step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates includes: When the first deep camera coordinate system and the second deep camera coordinate system are unified, determining object coordinates on each collection camera coordinate system of the object three-dimensional coordinates based on an extrinsic parameter matrix from the first deep camera coordinate system to each collection camera coordinate system matrix; The method of claim 1, further comprising: determining eye coordinates on each collecting camera coordinate system of the three-dimensional eye coordinates based on an external parameter matrix from the second deep camera coordinate system to each collecting camera coordinate system matrix.

3. The step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates includes: When the first deep camera coordinate system and the second deep camera coordinate system are not unified, determining an object calibration three-dimensional coordinate of the object three-dimensional coordinate on the first preset calibration plate coordinate system based on an extrinsic parameter matrix from the first deep camera coordinate system to a first preset calibration plate coordinate system; determining eye calibration three-dimensional coordinates on the first preset calibration plate coordinate system of the eye three-dimensional coordinates based on an extrinsic parameter matrix from the second deep camera coordinate system to the first preset calibration plate coordinate system; The method of claim 1, further comprising: determining object coordinates and eye coordinates on each of the collecting camera coordinate systems of the object calibration three-dimensional coordinates and the eye calibration three-dimensional coordinates based on an external parameter matrix from the first predetermined calibration plate coordinate system to each of the collecting camera coordinate system matrices.

4. before the step of determining the object coordinates and the eye coordinates on the respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates, determining a first extrinsic parameter matrix from the first deep camera coordinate system to a second preset calibration plate coordinate system, and determining a second extrinsic parameter matrix from the second deep camera coordinate system to the second preset calibration plate coordinate system; determining first three-dimensional object coordinates on the first deep camera coordinate system of a target reference object and determining second three-dimensional object coordinates on the second deep camera coordinate system of a target reference object; The method according to any one of claims 1 to 3, further comprising: determining whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first three-dimensional object coordinates, the second three-dimensional object coordinates, the first external parameter matrix, and the second external parameter matrix.

5. The step of determining first three-dimensional object coordinates of a target reference object on the first deep camera coordinate system and determining second three-dimensional object coordinates of a target reference object on the second deep camera coordinate system includes: collecting a first reference object image of a target reference object by a first deep camera, and determining first three-dimensional object coordinates of the target reference object in the first deep camera coordinate system based on the first reference object image; 5. The method of claim 4, further comprising: collecting a second reference object image of the target reference object by a second deep camera; and determining second three-dimensional object coordinates of the target reference object on the second deep camera coordinate system based on the second reference object image.

6. The step of determining whether the first deep camera coordinate system and the second deep camera coordinate system are unified based on the first three-dimensional object coordinates, the second three-dimensional object coordinates, the first extrinsic parameter matrix, and the second extrinsic parameter matrix includes: transforming the first three-dimensional object coordinates into first object coordinates on the second preset calibration plate coordinate system based on the first extrinsic parameter matrix; transforming the second three-dimensional object coordinates into second object coordinates on the second preset calibration plate coordinate system based on the second extrinsic parameter matrix; If the first object coordinates and the second object coordinates match, determining that the first deep camera coordinate system and the second deep camera coordinate system are unified; and determining that the first and second deep camera coordinate systems are not unified if the first and second object coordinates do not match.

7. The step of determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system includes: determining a line-of-sight vector on each of the acquisition camera coordinate systems based on the object coordinates and the eye coordinates on each of the acquisition camera coordinate systems; The method according to any one of claims 1 to 3, further comprising the step of: determining gaze direction data of the user based on gaze vectors on each collection camera coordinate system.

8. A gaze direction data collection device, a first coordinate determination module for collecting an object image of a target object, which is an object that a user is gazing at, by a first deep camera, and determining three-dimensional object coordinates of the target object in a first deep camera coordinate system based on the object image; a second coordinate determination module for collecting a face image of the user by a second deep camera and determining three-dimensional eye coordinates of the user's eye on a second deep camera coordinate system based on the face image; a third coordinate determination module for determining object coordinates and eye coordinates on respective acquisition camera coordinate systems of the object three-dimensional coordinates and the eye three-dimensional coordinates; a direction data determination module for determining gaze direction data of the user based on object coordinates and eye coordinates on each collection camera coordinate system.

9. Memory and a processor; a gaze direction data collection program stored in the memory and executable on the processor, A gaze direction data collection device, wherein the gaze direction data collection program is configured to implement the steps of the gaze direction data collection method according to any one of claims 1 to 7.

10. A storage medium on which a gaze direction data collection program is stored, the storage medium realizing the steps of the gaze direction data collection method according to any one of claims 1 to 7 when the gaze direction data collection program is executed by a processor.

Images