Line-of-sight direction marking method and system, electronic equipment and storage medium
By calibrating the external parameters relationship between the observation viewing angle coordinate system and the eye tracking camera coordinate system in the virtual reality device, and using multiple external parameters groups for hand-eye calibration, the accuracy and reusability of line-of-view direction labeling in virtual reality devices is solved, and fast and accurate line-of-view direction labeling is achieved.
Patent Information
- Application Number
- CN202411612047.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-08-01
AI Technical Summary
In the prior art, the visual direction labeling data obtained by virtual reality devices are inaccurate, and the coverage scene is not comprehensive and cannot be reused.
By modeling the virtual scene and real space of virtual reality devices, the external parameters relationship between the observation angle coordinate system and the eye tracking camera coordinate system is calibrated, and multiple external parameters groups are used for hand-eye calibration, simplifying the coordinate system conversion process and directly obtaining the line of sight direction annotations.
It realizes the fast and accurate acquisition of line of sight direction marking under different lighting environments, solves the poor accuracy and reusability of line of sight direction marking in the prior art, and simplifies the calibration process.
Smart Images
Figure CN120411250A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of eye tracking technology, and in particular to a gaze direction marking method, system, electronic device, and storage medium. Background Art
[0002] Eye tracking technology, which uses cameras to capture eye movement and gaze direction, is crucial for tasks such as human-computer interaction and gaze point rendering in virtual reality devices. Current methods for eye tracking can be categorized into traditional methods based on eye models and deep learning methods based on neural network models. To better evaluate the effectiveness of eye tracking in virtual reality devices, a large amount of annotated gaze direction data is required.
[0003] However, existing methods for obtaining gaze direction annotations for virtual reality devices do not produce accurate gaze direction annotation data. Therefore, how to obtain correct gaze direction annotation data is a technical problem that needs to be solved urgently. Summary of the Invention
[0004] In view of the above problems, embodiments of the present application provide a line of sight direction annotation method, system, electronic device, and storage medium to overcome the above problems or at least partially solve the above problems.
[0005] A first aspect of the embodiments of the present application discloses a method for marking a sight direction, comprising: Acquire multiple extrinsic parameter groups, each extrinsic parameter group including: a first extrinsic parameter and a second extrinsic parameter acquired when the virtual reality device is located at a position in a world coordinate system, the first extrinsic parameter being an extrinsic parameter between an observation view coordinate system and a world coordinate system, and the second extrinsic parameter being an extrinsic parameter between an eye tracking camera coordinate system and a calibration plate coordinate system; the observation view coordinate system being a coordinate system corresponding to a virtual scene of the virtual reality device; Performing hand-eye calibration based on the multiple extrinsic parameter groups to obtain a third extrinsic parameter between the observation view coordinate system and the eye tracking camera coordinate system; Based on the third extrinsic parameter, the pupil center and the gaze point are unified into the eye tracking camera coordinate system to achieve the annotation of the line of sight direction from the pupil center to the gaze point.
[0006] Optionally, based on the third extrinsic parameter, the pupil center and the gaze point are unified into the eye tracking camera coordinate system to implement the annotation of the gaze direction from the pupil center to the gaze point, including: Obtaining the position of the pupil center in the eye tracking camera coordinate system; Converting the first gaze point position in the observation perspective coordinate system to the second gaze point position in the eye tracking camera coordinate system according to the third external parameter; Calculate the marked line-of-sight direction according to the position of the second fixation point and the pupil center point in the coordinate system of the eye movement tracking camera.
[0007] Optionally, obtaining the position of the pupil center point in the coordinate system of the eye movement tracking camera includes: Based on the image of the pupil captured by the eye movement tracking camera, obtain the position of the pupil center point in the pixel coordinate system of the eye movement tracking camera; Convert the position of the pupil center point in the pixel coordinate system of the eye movement tracking camera to the coordinate system of the eye movement tracking camera to obtain the position of the pupil center point in the coordinate system of the eye movement tracking camera.
[0008] Optionally, calculating the marked line-of-sight direction according to the position of the second fixation point and the pupil center point in the coordinate system of the eye movement tracking camera includes: Using the position of the pupil center point in the coordinate system of the eye movement tracking camera as the starting point and the position of the second fixation point as the end point to obtain the line-of-sight vector; Based on the line-of-sight vector, obtain the marked information of the line-of-sight direction, and the marked information of the line-of-sight direction includes: the yaw angle of the line of sight and the pitch angle of the line of sight.
[0009] Optionally, performing hand-eye calibration according to the obtained multiple sets of external parameters to obtain the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system, including: Establish a world coordinate system with the robot base in the hand-eye calibration model as a reference, establish an observation perspective coordinate system with the robot manipulator in the hand-eye calibration model as a reference, establish an eye movement tracking camera coordinate system with the camera in the hand-eye calibration model as a reference, and establish a calibration board coordinate system with the target object in the hand-eye calibration model as a reference, and convert the calculation of the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task; Solve the hand-eye calibration task according to the multiple sets of external parameters to obtain the third external parameter.
[0010] Optionally, obtaining multiple sets of external parameters includes: When the virtual reality device is at the first position point in the world coordinate system, obtain the first external parameter based on the SLAM positioning system of the virtual reality device; When the virtual reality device is at the first position point in the world coordinate system, obtain the second external parameter based on the image of the calibration board captured by the eye movement tracking camera and the position of the calibration board in the world coordinate system; Adjust the position of the virtual reality device in the world coordinate system multiple times, and obtain the sets of external parameters when the virtual reality device is at different position points in the world coordinate system according to the above steps.
[0011] In a second aspect of the embodiments of the present application, a line-of-sight direction annotation system is disclosed, which is used to execute the line-of-sight direction annotation method described in the first aspect of the embodiments of the present application, and includes: A virtual reality device, which describes a fixation point in a virtual scene based on an observation perspective coordinate system. An eye movement tracking camera and a SLAM positioning system are configured in the virtual reality device. The eye movement tracking camera is used to collect pupil images to obtain the center point of the pupil; A calibration board, the position of which is fixed in the world coordinate system; When the virtual reality device is at a first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, a first external parameter is obtained. Based on the image collected by the eye movement tracking camera for the calibration board and the position of the calibration board in the world coordinate system, a second external parameter is obtained. The position of the virtual reality device in the world coordinate system is adjusted multiple times, and according to the above steps, an external parameter group composed of the first external parameter and the second external parameter is obtained when the virtual reality device is at different position points in the world coordinate system; Perform hand-eye calibration based on the obtained multiple external parameter groups to obtain a third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system; based on the third external parameter, unify the center point of the pupil and the fixation point in the eye movement tracking camera coordinate system to realize the line-of-sight direction annotation from the center point of the pupil to the fixation point.
[0012] In a third aspect of the embodiments of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the line-of-sight direction annotation method described in the first aspect of the embodiments of the present application are realized.
[0013] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the line-of-sight direction annotation method described in the first aspect of the embodiments of the present application are realized.
[0014] In a fifth aspect of the embodiments of the present application, a computer program product is disclosed, including a computer program. When the computer program is executed by a processor, the steps of the line-of-sight direction annotation method described in the first aspect of the embodiments of the present application are realized.
[0015] The embodiments of the present application include the following advantages: In the embodiments of the present application, by modeling the virtual scene and the real space of the virtual reality device, the external parameter relationship (i.e., the third external parameter) between the viewing angle coordinate system and the eye tracking camera coordinate system in the virtual scene is calibrated. Thus, based on the third external parameter, the pupil center obtained by the eye tracking camera and the fixation point of the virtual scene in the virtual reality device are unified into the same coordinate system, making the line-of-sight direction annotation under the virtual reality device simpler and more direct. Moreover, according to the multiple external parameter groups obtained, hand-eye calibration is performed, and the calculation of the third external parameter between the viewing angle coordinate system and the eye tracking camera coordinate system is converted into a hand-eye calibration task, simplifying the calibration process of the external parameter relationship between the viewing angle coordinate system and the eye tracking camera coordinate system, so as to quickly and directly obtain the line-of-sight direction annotation from the pupil center point to the fixation point. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0017] Figure 1 is a schematic structural diagram of a line-of-sight direction annotation system provided by an embodiment of the present application; Figure 2 is a flowchart of the steps of a line-of-sight direction annotation method provided by an embodiment of the present application; Figure 3 is a flowchart of collecting multiple external parameter groups for hand-eye calibration; Figure 4 is a schematic diagram of a hand-eye calibration model provided by an embodiment of the present application; Figure 5 is a flowchart of line-of-sight direction annotation according to the third external parameter provided by an embodiment of the present application; Figure 6 is a schematic structural diagram of a line-of-sight direction annotation device provided by an embodiment of the present application; Figure 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0018] To make the above objects, features, and advantages of the present application more apparent and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.
[0019] In the related art, methods for obtaining gaze direction annotations for virtual reality devices include: simulation data synthesis methods, eye tracker generation annotation methods, and methods for directly annotating the positions of fixation points in virtual space. Among them, simulation data synthesis methods generally use software such as Unity or image generation algorithms to simulate eye movement data, but the obtained data has a large gap from the actual image. The eye tracker generation annotation method irradiates the eyes with an infrared light source, and the camera captures the positions of the pupil and corneal reflection points, and determines the gaze direction by establishing a mapping model; however, this method is sensitive to ambient light, so there is a problem of poor generalization. The method of directly annotating the positions of fixation points in virtual space has the obtained gaze direction annotations related to the position of the eye movement tracking camera of the virtual reality device, and cannot be reused after the position of the eye movement tracking camera is adjusted. Therefore, the existing methods for obtaining gaze direction annotations for virtual reality devices have problems such as poor accuracy, incomplete coverage of scenarios, and inability to be reused.
[0020] To overcome the limitations of the related art, the embodiments of the present application provide a gaze direction annotation method. Considering that in a virtual reality device, the user's fixation position is in the virtual reality space, and the eye images obtained by the eye movement tracking camera exist in the real space; therefore, how to connect the two to obtain the correct annotation data of the gaze direction is the key technical means of the gaze direction annotation method in the embodiments of the present application.
[0021] Specifically, by modeling the virtual scene and the real space of the virtual reality device, the external parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system in the virtual scene is calibrated, so that the pupil center obtained by the eye movement tracking camera and the fixation point in the virtual scene of the virtual reality device are unified under the same coordinate system, making the gaze direction annotation under the virtual reality device more simple and direct. And, according to the obtained multiple groups of external parameters, hand-eye calibration is performed, and the calculation of the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system is converted into a hand-eye calibration task, simplifying the calibration process of the external parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system, so as to quickly and directly obtain the gaze direction annotation from the pupil center point to the fixation point. For specific descriptions, please refer to the following text.
[0022] To better understand the technical solutions of the embodiments of the present application, the relevant coordinate systems involved in the embodiments of the present application are first described as follows: (1) Observation perspective coordinate system: The observation perspective coordinate system is the coordinate system corresponding to the virtual scene of the virtual reality device. In the virtual reality device, the observation perspective coordinate system is equivalent to the coordinate system of the user's perspective or the camera position, and is used to describe the fixation point and the line-of-sight direction of the user (or the virtual reality device headset) in the virtual scene. The origin of the observation perspective coordinate system is usually located at the position of the user's eyes or the center position of the virtual reality device headset, and the observation perspective coordinate system changes dynamically with the movement and rotation of the user's head. The virtual reality device usually uses sensors (such as gyroscopes, accelerometers) and computer vision technologies to track the position and posture of the user's head. The observation perspective coordinate system is a three-dimensional rectangular coordinate system, which is the coordinate system corresponding to the constructed virtual scene. Based on this, the position of the fixation point in the virtual scene can be described, and the observation perspective coordinates are used to represent the coordinates of the fixation point or the fixation object in the virtual scene or the augmented reality world.
[0023] (2) World coordinate system: The reference coordinate system for SLAM (Simultaneous Localization and Mapping) positioning of the observation perspective coordinate system. The world coordinate system is a fixed reference framework, and its origin and direction remain unchanged throughout the virtual scene. The world coordinate system provides a unified reference point for all other coordinate systems (such as device coordinate systems, user coordinate systems, etc.).
[0024] (3) Eye-tracking camera coordinate system: In the virtual reality device, the eye-tracking camera is fixed on the virtual reality device and is used to collect the eye images (pupil images) of the user. The eye-tracking camera coordinate system is a three-dimensional rectangular coordinate system, and the origin is located at the optical center of the lens in the eye-tracking camera. The X-axis and Y-axis are respectively parallel to the two sides of the image plane of the eye-tracking camera, and the Z-axis is the optical axis of the corresponding lens, perpendicular to the image plane. Usually, in the virtual reality device, there are two eye-tracking cameras, and each eye-tracking camera has an eye-tracking camera coordinate system.
[0025] (4) Calibration board coordinate system: The calibration board is fixed at a certain position in the real world and does not move with the virtual reality device. During the process of obtaining multiple sets of external parameters, both the calibration board coordinate system and the world coordinate system remain relatively fixed. The calibration board coordinate system is a three-dimensional rectangular coordinate system, and the X-axis and Y-axis are respectively parallel to two adjacent perpendicular sides of the calibration board, and the Z-axis is perpendicular to the calibration board plane.
[0026] The embodiments of the present application provide a line-of-sight direction annotation system for implementing the line-of-sight direction annotation method provided by the embodiments of the present application. Refer to Figure 1 as shown Figure 1It is a schematic structural diagram of a line-of-sight direction annotation system provided by an embodiment of the present application. The line-of-sight direction annotation system includes: A virtual reality device, which describes a fixation point in a virtual scene based on an observation perspective coordinate system. An eye tracking camera and a SLAM positioning system are configured in the virtual reality device. The eye tracking camera is used to collect pupil images to obtain the center point of the pupil. A calibration board, whose position in the world coordinate system is fixed.
[0027] Specifically, when the virtual reality device is at a first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, a first external parameter is obtained. Based on the image collected by the eye tracking camera for the calibration board and the position of the calibration board in the world coordinate system, a second external parameter is obtained. The position of the virtual reality device in the world coordinate system is adjusted multiple times, and according to the above steps, an external parameter group composed of the first external parameter and the second external parameter is obtained when the virtual reality device is at different position points in the world coordinate system; hand-eye calibration is performed according to the obtained multiple external parameter groups to obtain a third external parameter between the observation perspective coordinate system and the eye tracking camera coordinate system; based on the third external parameter, the center point of the pupil and the fixation point are unified in the eye tracking camera coordinate system to achieve the line-of-sight direction annotation from the center point of the pupil to the fixation point.
[0028] Among them, the first external parameter is the external parameter between the observation perspective coordinate system and the world coordinate system, that is, the first external parameter represents the coordinate conversion relationship between the observation perspective coordinate system and the world coordinate system; the second external parameter is the external parameter between the eye tracking camera coordinate system and the calibration board coordinate system, that is, the second external parameter represents the coordinate conversion relationship between the eye tracking camera coordinate system and the calibration board coordinate system. The first external parameter and the second external parameter form an external parameter group; the third external parameter represents the coordinate conversion relationship between the observation perspective coordinate system and the eye tracking camera coordinate system. In some embodiments, the first external parameter, the second external parameter, and the third external parameter are represented in the form of matrices.
[0029] In the embodiment of the present application, the first position is any position in the world coordinate system. After moving the virtual reality device to the first position point in the world coordinate system, the first external parameter is obtained based on the SLAM positioning system of the virtual reality device. Specifically, obtaining the first external parameter based on the SLAM positioning system of the virtual reality device includes: obtaining the position and attitude of the observation perspective coordinate system of the virtual reality device in the world coordinate system based on the SLAM positioning system, and thus obtaining the first external parameter based on the position and attitude of the observation perspective coordinate system in the world coordinate system. The position of the calibration board is fixed in the world coordinate system, and each position on the calibration board is known. After moving the virtual reality device to the first position point in the world coordinate system, the calibration board image is collected by the eye movement tracking camera, and thus the external parameter calibration is performed based on the image collected by the eye movement tracking camera for the calibration board and the position of the calibration board in the world coordinate system to obtain the second external parameter. In this way, the parameter groups of the virtual reality device at different position points in the world coordinate system are obtained, and multiple external parameter groups are obtained by adjusting the position points of the virtual reality device in the world coordinate system multiple times.
[0030] Perform hand-eye calibration according to the obtained multiple external parameter groups, and convert the calculation of the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task to quickly calibrate the external parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system. And use the third external parameter to unify the pupil center obtained by the eye movement tracking camera and the fixation point of the virtual scene in the virtual reality device into the same coordinate system, so that the line-of-sight direction annotation under the virtual reality device is simpler and more direct, and thus the line-of-sight direction annotation from the pupil center point to the fixation point can be obtained quickly and directly.
[0031] The embodiment of the present application also provides a line-of-sight direction annotation method, referring to Figure 2 as shown, Figure 2 is the step flow chart of a line-of-sight direction annotation method provided by the embodiment of the present application. As Figure 2 shown, the line-of-sight direction annotation method may include step S210 and step S230: Step S210: Obtain multiple external parameter groups, each external parameter group including: the first external parameter and the second external parameter obtained when the virtual reality device is at a position point in the world coordinate system, the first external parameter being the external parameter between the observation perspective coordinate system and the world coordinate system, and the second external parameter being the external parameter between the eye movement tracking camera coordinate system and the calibration board coordinate system; the observation perspective coordinate system is the coordinate system corresponding to the virtual scene of the virtual reality device.
[0032] Step S220: Perform hand-eye calibration according to the obtained multiple external parameter groups to obtain the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system.
[0033] Step S230: Based on the third extrinsic parameter, unify the pupil center point and the fixation point in the eye movement tracking camera coordinate system to achieve the line-of-sight direction annotation from the pupil center point to the fixation point.
[0034] In the embodiments of the present application, multiple extrinsic parameter groups are extrinsic parameter groups obtained at different position points of the virtual reality device in the world coordinate system, and each extrinsic parameter group corresponds to the extrinsic parameters obtained at a position point of the virtual reality device in the world coordinate system. Among them, the first extrinsic parameter represents the coordinate system conversion relationship between the observation perspective coordinate system and the world coordinate system, and the second extrinsic parameter represents the coordinate system conversion relationship between the eye movement tracking camera coordinate system and the calibration board coordinate system.
[0035] The third extrinsic parameter represents the coordinate system conversion relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system. Perform hand-eye calibration based on the obtained multiple extrinsic parameter groups, and convert the calculation of the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task to quickly calibrate the extrinsic parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system.
[0036] The pupil center point exists in the real world and can be obtained by the eye movement tracking camera; the fixation point refers to the fixation point in the virtual scene of the virtual reality device. Therefore, the pupil center and the fixation point are in different coordinate systems. To obtain an accurate line-of-sight direction annotation, based on the third extrinsic parameter, unify the pupil center point and the fixation point in the eye movement tracking camera coordinate system, so as to perform line-of-sight direction annotation based on the position of the pupil center point in the eye movement tracking camera coordinate system and the position of the fixation point in the eye movement tracking camera coordinate system.
[0037] Through the above process, model the virtual scene and the real space of the virtual reality device, calibrate the extrinsic parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system in the virtual scene, so as to unify the pupil center obtained by the eye movement tracking camera and the fixation point in the virtual scene of the virtual reality device to the same coordinate system based on the third extrinsic parameter, making the line-of-sight direction annotation under the virtual reality device simpler and more direct. Moreover, perform hand-eye calibration based on the obtained multiple extrinsic parameter groups, and convert the calculation of the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task, simplifying the calibration process of the extrinsic parameter relationship between the observation perspective coordinate system and the eye movement tracking camera coordinate system, so as to quickly and directly obtain the line-of-sight direction annotation from the pupil center point to the fixation point.
[0038] In this way, the problem of difficult annotation of the line-of-sight direction of the eye image collected by the eye tracking camera under the virtual reality device in the actual use scenario is solved. After obtaining the third extrinsic parameter between the observation perspective coordinate system and the eye tracking camera coordinate system, the eye image in the actual use scenario can be collected, and the pupil center point and the fixation point can be unified into the eye tracking camera coordinate system through coordinate transformation, so as to realize the annotation of the line-of-sight direction from the pupil center point to the fixation point. Moreover, the problem of insufficient comprehensive coverage of the use scenario is solved. With this method, the line-of-sight direction can still be annotated under any external environment such as arbitrary illumination. In addition, the problem that the line-of-sight direction annotation cannot be reused after the position of the eye tracking camera is adjusted is solved. After calibrating the third extrinsic parameter between the eye tracking camera coordinate system and the observation perspective coordinate system, the line-of-sight direction of the current pupil center point can be annotated, ensuring that the annotation is still correct under different installation positions of the eye tracking camera, and avoiding repeated calibration.
[0039] Combined with the above embodiments, in one implementation, the embodiment of the present application further provides a method for annotating the line-of-sight direction. In this method, the "acquiring a plurality of extrinsic parameter groups" in the above step S210 may specifically include the following sub-steps S210-1 to step S210-3: Step S210-1: When the virtual reality device is at the first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, obtain the first extrinsic parameter.
[0040] Step S210-2: When the virtual reality device is at the first position point in the world coordinate system, based on the image collected by the eye tracking camera for the calibration board and the position of the calibration board in the world coordinate system, obtain the second extrinsic parameter.
[0041] Step S210-3: Adjust the position of the virtual reality device in the world coordinate system multiple times, and obtain the extrinsic parameter groups when the virtual reality device is at different position points in the world coordinate system according to the above steps.
[0042] Wherein, the first position is any position in the world coordinate system. After moving the virtual reality device to the first position point in the world coordinate system, the first extrinsic parameter is obtained based on the SLAM positioning system of the virtual reality device. Specifically, the position and attitude of the observation perspective coordinate system of the virtual reality device in the world coordinate system are obtained based on the SLAM positioning system, and thus the first extrinsic parameter is obtained based on the position and attitude of the observation perspective coordinate system in the world coordinate system.
[0043] The position of the calibration board is fixed in the world coordinate system, and each position on the calibration board is known; after moving the virtual reality device to the first position point in the world coordinate system, the eye-tracking camera is used to collect the calibration board image, so as to perform external parameter calibration based on the image of the calibration board collected by the eye-tracking camera and the position of the calibration board in the world coordinate system, and obtain the second external parameter.
[0044] The virtual reality device is located at the same position point in the world coordinate system, and the first external parameter and the second external parameter obtained are used as an external parameter group; the position of the virtual reality device in the world coordinate system is adjusted multiple times, and in the manner of the above steps S210-1 and S210-2, external parameter groups in the case of different positions of the virtual reality device in the world coordinate system are obtained, so as to obtain multiple external parameter groups.
[0045] After obtaining multiple external parameter groups, hand-eye calibration is performed according to the obtained multiple external parameter groups to obtain the third external parameter between the observation perspective coordinate system and the eye-tracking camera coordinate system. Exemplarily, Figure 3 It is a flowchart of collecting multiple external parameter groups for hand-eye calibration provided by an embodiment of the present application, Figure 3 It shows hand-eye calibration based on the obtained N external parameter groups. Specifically, when the virtual reality device is at the first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, the first external parameter Ai between the observation perspective coordinate system and the world coordinate system is obtained; and based on the image of the calibration board collected by the eye-tracking camera and the position of the calibration board in the world coordinate system, the second external parameter Bi between the eye-tracking camera coordinate system and the calibration board coordinate system is calculated. If the number of currently obtained external parameter groups is less than N, the position of the virtual reality device in the world coordinate system is adjusted; the next external parameter group is obtained until N external parameter groups are obtained. Then, hand-eye calibration is performed based on the N external parameter groups composed of the first external parameter Ai and the second external parameter Bi, and the third external parameter X between the observation perspective coordinate system and the eye-tracking camera coordinate system is calculated.
[0046] Combined with the above embodiments, in one implementation manner, the embodiment of the present application further provides a line-of-sight direction annotation method. In this method, "performing hand-eye calibration according to the obtained multiple external parameter groups to obtain the third external parameter between the observation perspective coordinate system and the eye-tracking camera coordinate system" in the above step S220 may specifically include the following sub-steps S220-1 to step S220-2: Step S220-1: Establish a world coordinate system with the robot base in the hand-eye calibration model as a reference, establish an observation perspective coordinate system with the robot manipulator in the hand-eye calibration model as a reference, establish an eye movement tracking camera coordinate system with the camera in the hand-eye calibration model as a reference, and establish a calibration board coordinate system with the target object in the hand-eye calibration model as a reference. Convert the calculation of the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task.
[0047] Step S220-2: Solve the hand-eye calibration task according to the multiple extrinsic parameter groups to obtain the third extrinsic parameter.
[0048] Exemplarily, Figure 4 FIG. 8 is a schematic diagram of a hand-eye calibration model provided by an embodiment of the present application. The hand-eye calibration model includes a robot base T, a robot manipulator (equivalent to a hand) H, a target object P, and a camera (equivalent to an eye) E. Among them, the robot manipulator H and the camera E are relatively fixed and can move freely as a whole. The extrinsic parameter X between the robot manipulator H and the camera E is a value to be solved. The positions of the robot base T and the target object P in the world coordinate system are fixed, and the extrinsic parameter Y between the robot base T and the target object P is unchanged but unknown. When the robot manipulator H and the camera E move, the extrinsic parameters Ai between the robot manipulator H and the robot base T, and the extrinsic parameter Bi between the target object P and the camera E are known. Therefore, the hand-eye calibration process is to solve the extrinsic parameter X when multiple groups of AiX = XBi are known.
[0049] In the embodiment of the present application, according to the method of step S220-1 above, the calculation of the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system is converted into a hand-eye calibration task to simplify the calibration process of the third extrinsic parameter. Then, according to the hand-eye calibration principle, the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system is calculated based on multiple extrinsic parameter groups.
[0050] Specifically, solving the hand-eye calibration task according to multiple extrinsic parameter groups can be expressed as solving according to multiple "first extrinsic parameters Third extrinsic parameter = Third extrinsic parameter Second extrinsic parameter" to obtain the third extrinsic parameter. In this way, by converting the calculation of the third extrinsic parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task, the calibration of the third extrinsic parameter is realized in a more simple and direct manner, making the subsequent eye movement tracking line-of-sight direction annotation more convenient and direct.
[0051] It should be noted that in the virtual reality device, there are two eye tracking cameras. When calibrating the third extrinsic parameter between the viewing angle coordinate system and the eye tracking camera coordinate system, the third extrinsic parameter between the viewing angle coordinate system and the two eye tracking camera coordinate systems can be calibrated respectively in the above manner; or the third extrinsic parameter between the viewing angle coordinate system and one eye tracking camera coordinate system can be calibrated first, and then according to the relative relationship between the two eye tracking cameras, the third extrinsic parameter between the viewing angle coordinate system and the other eye tracking camera coordinate system can be calculated.
[0052] Combined with the above embodiments, in one implementation manner, the embodiments of the present application also provide a method for marking the line of sight direction. In this method, the "unifying the pupil center point and the fixation point in the eye tracking camera coordinate system based on the third extrinsic parameter to realize the marking of the line of sight direction from the pupil center point to the fixation point" in the above step S230 may specifically include the following sub-steps S230-1 to step S230-3: Step S230-1: Obtain the position of the pupil center point in the eye tracking camera coordinate system.
[0053] In the embodiments of the present application, the pupil center point exists in the real world, and the position of the pupil center point in the eye tracking camera coordinate system can be obtained through the eye tracking camera.
[0054] Specifically, obtaining the position of the pupil center point in the eye tracking camera coordinate system includes: obtaining the position of the pupil center point in the pixel coordinate system of the eye tracking camera based on the image collected by the eye tracking camera for the pupil; converting the position of the pupil center point in the pixel coordinate system of the eye tracking camera to the eye tracking camera coordinate system to obtain the position of the pupil center point in the eye tracking camera coordinate system.
[0055] Among them, the pixel coordinate system of the eye tracking camera is the coordinate system where the imaging plane of the camera is located, and the position of the pupil center point in the pixel coordinate system of the eye tracking camera can be expressed as . The position of the pupil center point in the pixel coordinate system of the eye tracking camera can be converted to the eye tracking camera coordinate system according to the target depth d of the pupil center point and the intrinsic matrix of the eye tracking camera to obtain the position of the pupil center point in the eye tracking camera coordinate system . Among them, the target depth d is the average distance between the pupil center point and the eye tracking camera after the user wears the virtual reality device. Generally, the target depth d is a fixed value; the intrinsic matrix of the eye tracking camera is a matrix, which contains the focal length, principal point position of the eye tracking camera, and rotation and translation information of the image.
[0056] Exemplarily, the conversion of the position of the pupil center point in the pixel coordinate system of the eye movement tracking camera to the eye movement tracking camera coordinate system can be expressed as:
[0057] Step S230-2: According to the third extrinsic parameter, convert the position of the first fixation point in the viewing angle coordinate system to the position of the second fixation point in the eye movement tracking camera coordinate system.
[0058] Among them, the position of the first fixation point in the viewing angle coordinate system represents the position of the fixation point in the virtual scene, and the first fixation point position is a randomly specified fixation point position in the viewing angle coordinate system; the third extrinsic parameter represents the coordinate transformation relationship between the viewing angle coordinate system and the eye movement tracking camera coordinate system. Therefore, coordinate transformation is performed according to the third extrinsic parameter to convert the position of the first fixation point in the viewing angle coordinate system to the position of the second fixation point in the eye movement tracking camera coordinate system.
[0059] Step S230-3: Calculate the marked line-of-sight direction according to the position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system.
[0060] In the embodiment of the present application, the position of the second fixation point is described by the eye movement tracking camera coordinate system. The position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system belong to the same coordinate system. Therefore, the marked line-of-sight direction can be directly calculated according to the position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system.
[0061] Specifically, calculating the marked line-of-sight direction according to the position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system includes: taking the position of the pupil center point in the eye movement tracking camera coordinate system as the starting point and the position of the second fixation point as the ending point to obtain a line-of-sight vector; based on the line-of-sight vector, obtaining the marked information of the line-of-sight direction, and the marked information of the line-of-sight direction includes: the yaw angle of the line of sight and the pitch angle of the line of sight.
[0062] In the embodiments of the present application, based on the third extrinsic parameter, the position of the fixation point in the eye tracking camera coordinate system (i.e., the second fixation point position) and the position of the pupil center point in the eye tracking camera coordinate system can be obtained. Thus, based on the pupil center point and the fixation point in the same coordinate system, the gaze vector can be directly calculated. Among them, based on the gaze vector, the annotation information of the gaze direction is obtained, including: normalizing the gaze vector, and converting the normalized gaze vector into annotation information composed of the yaw angle yaw and the pitch angle pitch of the gaze. Since the gaze vector is described based on the eye tracking camera coordinate system, the yaw angle and the pitch angle of the gaze are also described based on the eye tracking camera coordinate system.
[0063] In this way, through the third extrinsic parameter, the position of the fixation point in the virtual scene (observation perspective coordinate system) is converted into the eye tracking camera coordinate system. Thus, based on the position of the pupil center point in the eye tracking camera coordinate system and the position of the fixation point in the eye tracking camera coordinate system, the annotation of the gaze direction from the pupil center point to the fixation point is realized.
[0064] It should be noted that when obtaining the gaze direction annotation data set, each gaze direction annotation data is calculated in the manner of the above steps S230-1 to S230-3, so as to obtain a gaze direction annotation data set composed of multiple gaze direction annotation data.
[0065] Exemplarily, Figure 5 is a flowchart of gaze direction annotation according to the third extrinsic parameter provided by the embodiments of the present application, Figure 5 which schematically shows the process of obtaining the gaze direction annotation data of multiple fixation points. Specifically, for the k-th fixation point, the first fixation point position in the observation perspective coordinate system is obtained ; and according to the third extrinsic parameter, the first fixation point position in the observation perspective coordinate system is converted into the second fixation point position in the eye tracking camera coordinate system . Then, based on the image of the pupil collected by the eye tracking camera, the position of the pupil center point in the pixel coordinate system of the eye tracking camera is obtained ; and according to the target depth d, the position of the pupil center point in the pixel coordinate system of the eye tracking camera is converted into the eye tracking camera coordinate system to obtain the position of the pupil center point in the eye tracking camera coordinate system . Finally, from the second fixation point position and the position of the pupil center point in the eye tracking camera coordinate system , the annotated gaze direction is calculated. If all the fixation points are annotated, the gaze direction annotation data of multiple fixation points are obtained.
[0066] An embodiment of the present application also provides a line-of-sight direction annotation device. Refer to Figure 6 as shown in Figure 6 which is a schematic structural diagram of a line-of-sight direction annotation device provided by an embodiment of the present application, including: A data acquisition module 610, configured to acquire a plurality of external parameter groups, each external parameter group including: a first external parameter and a second external parameter acquired when the virtual reality device is at a position point in the world coordinate system, where the first external parameter is the external parameter between the observation perspective coordinate system and the world coordinate system, and the second external parameter is the external parameter between the eye movement tracking camera coordinate system and the calibration plate coordinate system; the observation perspective coordinate system is the coordinate system corresponding to the virtual scene of the virtual reality device; An external parameter calibration module 620, configured to perform hand-eye calibration based on the acquired plurality of external parameter groups to obtain a third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system; A line-of-sight annotation module 630, configured to unify the pupil center point and the fixation point under the eye movement tracking camera coordinate system based on the third external parameter, so as to implement line-of-sight direction annotation from the pupil center point to the fixation point.
[0067] In an optional embodiment, the line-of-sight annotation module includes: A pupil position acquisition module, configured to acquire the position of the pupil center point under the eye movement tracking camera coordinate system; A fixation point position conversion module, configured to convert the first fixation point position under the observation perspective coordinate system to the second fixation point position under the eye movement tracking camera coordinate system according to the third external parameter; A line-of-sight calculation module, configured to calculate the annotated line-of-sight direction according to the second fixation point position and the position of the pupil center point under the eye movement tracking camera coordinate system.
[0068] In an optional embodiment, the pupil position acquisition module includes: A pupil position acquisition sub-module, configured to acquire the position of the pupil center point under the pixel coordinate system of the eye movement tracking camera based on the image acquired by the eye movement tracking camera for the pupil; A pupil center position conversion module, configured to convert the position of the pupil center point under the pixel coordinate system of the eye movement tracking camera to the position under the eye movement tracking camera coordinate system, so as to obtain the position of the pupil center point under the eye movement tracking camera coordinate system.
[0069] In an optional embodiment, the line-of-sight calculation module includes: A line-of-sight vector calculation module, configured to use the position of the pupil center point under the eye movement tracking camera coordinate system as the starting point and the second fixation point position as the ending point to obtain a line-of-sight vector; The annotation information module is used to obtain the annotation information of the line-of-sight direction based on the line-of-sight vector. The annotation information of the line-of-sight direction includes: the yaw angle of the line of sight and the pitch angle of the line of sight.
[0070] In an alternative embodiment, the external parameter calibration module includes: The task conversion module is used to establish a world coordinate system with reference to the robot base in the hand-eye calibration model, establish an observation perspective coordinate system with reference to the robot manipulator in the hand-eye calibration model, establish an eye movement tracking camera coordinate system with reference to the camera in the hand-eye calibration model, and establish a calibration board coordinate system with reference to the target object in the hand-eye calibration model, and convert the third external parameter calculated between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task; The task solving module is used to solve the hand-eye calibration task according to the multiple external parameter groups to obtain the third external parameter.
[0071] In an alternative embodiment, the data acquisition module includes: The first external parameter acquisition module is used to obtain the first external parameter based on the SLAM positioning system of the virtual reality device when the virtual reality device is at the first position point in the world coordinate system; The second external parameter acquisition module is used to obtain the second external parameter based on the image captured by the eye movement tracking camera for the calibration board and the position of the calibration board in the world coordinate system when the virtual reality device is at the first position point in the world coordinate system; The position adjustment module is used to adjust the position of the virtual reality device in the world coordinate system multiple times, and obtain the external parameter groups when the virtual reality device is at different position points in the world coordinate system according to the above steps.
[0072] The embodiment of the present application also provides an electronic device, referring to Figure 7 , Figure 7 is a schematic structural diagram of an electronic device provided by the embodiment of the present application. As Figure 7 shown, the electronic device 700 includes: a memory 710 and a processor 720. The memory 710 and the processor 720 are communicatively connected via a bus. A computer program is stored in the memory 710, and the computer program can run on the processor 720, thereby implementing the steps of the line-of-sight direction annotation method described in the embodiment of the present application.
[0073] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the line-of-sight direction annotation method described in the embodiment of the present application are implemented.
[0074] The embodiment of the present application also provides a computer program product, including a computer program, which when executed by a processor, implements the steps of the line-of-sight direction annotation method described in the embodiment of the present application.
[0075] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.
[0076] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0077] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0078] These computer program instructions can also be loaded onto the computer or other programmable data processing terminal devices, so that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0079] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.
[0080] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the said element.
[0081] The above has introduced in detail a method, system, electronic device and storage medium for marking the line-of-sight direction provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.
Claims
1. A method for annotating the line of sight direction, characterized in that, Including: Obtain a plurality of external parameter groups, each external parameter group including: a first external parameter and a second external parameter obtained when the virtual reality device is at a position point in the world coordinate system, the first external parameter being the external parameter between the observation perspective coordinate system and the world coordinate system, and the second external parameter being the external parameter between the eye movement tracking camera coordinate system and the calibration board coordinate system; the observation perspective coordinate system being the coordinate system corresponding to the virtual scene of the virtual reality device; Perform hand-eye calibration based on the obtained plurality of external parameter groups to obtain a third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system; Based on the third external parameter, unify the pupil center point and the fixation point in the eye movement tracking camera coordinate system to implement the annotation of the line-of-sight direction from the pupil center point to the fixation point.
2. The method according to claim 1, characterized in that, Based on the third external parameter, unify the pupil center point and the fixation point in the eye movement tracking camera coordinate system to implement the annotation of the line-of-sight direction from the pupil center point to the fixation point, including: Obtain the position of the pupil center point in the eye movement tracking camera coordinate system; According to the third external parameter, convert the position of the first fixation point in the observation perspective coordinate system to the position of the second fixation point in the eye movement tracking camera coordinate system; According to the position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system, calculate the annotated line-of-sight direction.
3. The method according to claim 2, wherein Obtain the position of the pupil center point in the eye movement tracking camera coordinate system, including: Based on the image of the pupil collected by the eye movement tracking camera, obtain the position of the pupil center point in the pixel coordinate system of the eye movement tracking camera; Convert the position of the pupil center point in the pixel coordinate system of the eye movement tracking camera to the eye movement tracking camera coordinate system to obtain the position of the pupil center point in the eye movement tracking camera coordinate system.
4. The method according to claim 2, wherein According to the position of the second fixation point and the position of the pupil center point in the eye movement tracking camera coordinate system, calculate the annotated line-of-sight direction, including: Taking the position of the pupil center point in the eye movement tracking camera coordinate system as the starting point and the position of the second fixation point as the ending point to obtain a line-of-sight vector; Based on the line-of-sight vector, obtain the annotation information of the line-of-sight direction, and the annotation information of the line-of-sight direction includes: the yaw angle of the line-of-sight and the pitch angle of the line-of-sight.
5. The method according to claim 1, characterized in that Perform hand-eye calibration based on the obtained plurality of external parameter groups to obtain a third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system, including: Establish a world coordinate system with reference to the robot base in the hand-eye calibration model, establish an observation perspective coordinate system with reference to the robot manipulator in the hand-eye calibration model, establish an eye movement tracking camera coordinate system with reference to the camera in the hand-eye calibration model, and establish a calibration board coordinate system with reference to the target object in the hand-eye calibration model, and convert the calculation of the third external parameter between the observation perspective coordinate system and the eye movement tracking camera coordinate system into a hand-eye calibration task; Solve the hand-eye calibration task according to the plurality of external parameter groups to obtain the third external parameter.
6. The method according to any one of claims 1-5, characterized in that, Obtain a plurality of external parameter groups, including: When the virtual reality device is at the first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, obtain the first external parameter; When the virtual reality device is at the first position point in the world coordinate system, based on the images captured by the eye-tracking camera for the calibration board and the position of the calibration board in the world coordinate system, a second extrinsic parameter is obtained. Adjust the position of the virtual reality device in the world coordinate system multiple times, and according to the above steps, obtain a set of extrinsic parameters when the virtual reality device is at different position points in the world coordinate system.
7. A line-of-sight direction annotation system, characterized in that Used to execute the line-of-sight direction annotation method described in any one of claims 1-6 above, including: A virtual reality device, which describes a fixation point in a virtual scene based on an observation perspective coordinate system. An eye-tracking camera and a SLAM positioning system are configured in the virtual reality device. The eye-tracking camera is used to capture pupil images to obtain the center point of the pupil. A calibration board, the position of which is fixed in the world coordinate system. When the virtual reality device is at the first position point in the world coordinate system, based on the SLAM positioning system of the virtual reality device, obtain a first extrinsic parameter. Based on the images captured by the eye-tracking camera for the calibration board and the position of the calibration board in the world coordinate system, obtain a second extrinsic parameter. Adjust the position of the virtual reality device in the world coordinate system multiple times, and according to the above steps, obtain a set of extrinsic parameters composed of the first extrinsic parameter and the second extrinsic parameter when the virtual reality device is at different position points in the world coordinate system. Perform hand-eye calibration based on the obtained multiple sets of extrinsic parameters to obtain a third extrinsic parameter between the observation perspective coordinate system and the eye-tracking camera coordinate system; based on the third extrinsic parameter, unify the center point of the pupil and the fixation point in the eye-tracking camera coordinate system to achieve the line-of-sight direction annotation from the center point of the pupil to the fixation point.
8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the line-of-sight direction annotation method described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the line-of-sight direction annotation method described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the line-of-sight direction annotation method as described in any one of claims 1-6.