Teleoperation target three-dimensional position estimation method based on human eye fixation interaction
Through human-eye gaze interaction technology, the sensors of binocular cameras and head-mounted displays calculate the direction of the human-eye line of sight in real time, and directly calculate the three-dimensional spatial position of the slave target, solving the dependence of cooperative target customization and pre-training models in the existing technology The problem is to achieve fast and accurate target positioning and is suitable for remote operations in complex environments.
Patent Information
- Application Number
- CN202410973809.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, cooperative target customization requirements are high, and the target detection network requires pre-trained data sets and a large number of manual annotations, which limits its application environment and efficiency.
Through human-eye gaze interaction, sensors of slave binocular cameras and head-mounted displays calculate the direction of human-eye vision in real time. Combined with the information of binocular stereo cameras, the three-dimensional spatial position of slave targets is directly calculated, avoiding the cooperation marker and the dependency of pre-trained models.
It realizes rapid identification and positioning of non-cooperation targets from the terminal, improves the operator's accuracy of target location, reduces the impact of the lighting environment, broadens the remote operation application scenarios, and enhances operation safety.
Smart Images

Figure CN119027490A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for estimating a target position in a mixed reality scenario, and in particular to a method for estimating a three-dimensional position of a remote-operated target based on human eye gaze interaction. Background Art
[0002] In the application fields of master-slave teleoperation, such as telemedicine, extreme environment exploration, anti-terrorism and explosion prevention, the master-end operator can immersively observe the view of the three-dimensional space environment obtained by the slave-end binocular camera through a head-mounted display. How to accurately locate the three-dimensional spatial position of the slave-end target is one of the key research technologies in the field of master-slave teleoperation. At present, cooperative markers and target detection networks are the main ways to locate slave-end targets. Among them, cooperative markers are suitable for structured scenes. The cooperative markers are fixed on the target surface, and the three-dimensional position of the target relative to the camera is calculated through classical image processing algorithms. The invention patent "A cooperative target and target identification method" (application number: 202010891587.5, patentee: China University of Metrology) provides a method for the design and identification of cooperative targets. The application scenarios of such methods are very limited, and the customization requirements of the markers are high. Target detection networks, such as YOLO and other intelligent image recognition algorithms, can quickly infer the three-dimensional position of the target in unstructured scenes through training data sets; the invention patent "Three-dimensional target detection method, device and system based on binocular vision" (publication number: CN112287824A, patentee: Hangzhou Hikvision Digital Technology Co., Ltd.) provides a three-dimensional target detection method, device and system based on binocular vision. This type of method relies on the coverage of the training data set and requires a lot of manual annotation. Summary of the invention
[0003] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and solve the problems in the prior art such as high requirements for cooperative target customization and the need for pre-training of target detection networks.
[0004] The purpose of the present invention is achieved through the following technical solutions:
[0005] In a first aspect, the present invention provides a method for estimating a three-dimensional position of a teleoperation target based on human eye gaze interaction, comprising:
[0006] The slave-side binocular camera images the three-dimensional space environment, which is then presented in a head-mounted display worn by the master-side operator;
[0007] The IMU in the head-mounted display calculates the operator's head posture information relative to the world coordinate system in real time (r h ,p h ,y h ), determine the posture rotation matrix of the head coordinate system;
[0008] The infrared camera in the head-mounted display is used to calculate the posture of the human eye relative to the head coordinate system (0, p e ,y e ), determine the posture rotation matrix of the eye coordinate system relative to the head coordinate system;
[0009] Determine the posture rotation matrix of the eye coordinate system relative to the world coordinate system according to the posture rotation matrix of the head coordinate system and the posture rotation matrix of the eye coordinate system relative to the head coordinate system, and further determine the eye sight vector;
[0010] Based on the fact that the line of sight of the human eye is consistent with the line connecting the target from the end and the optical center of the camera, the point M where the line of sight falls on the image is determined according to the line of sight direction. img That is, the coordinate point (u, v) of the target in the two-dimensional image;
[0011] According to the coordinate point (u, v) of the target in the two-dimensional image in the slave binocular camera, the left and right eye internal parameter matrix M L and M R , the rotation matrix R and translation vector T of the right eye center relative to the left eye center, and solve the three-dimensional spatial pose information of the target.
[0012] Based on the first aspect, in one embodiment of the present invention, the posture rotation matrix of the head coordinate system is:
[0013]
[0014] Wherein, c represents the cos() function, and s represents the sin() function.
[0015] Based on the first aspect, in one embodiment of the present invention, the posture rotation matrix of the eye coordinate system relative to the head coordinate system is:
[0016]
[0017] Wherein, c represents the cos() function, and s represents the sin() function.
[0018] Based on the first aspect, in one embodiment of the present invention, according to the following formula:
[0019]
[0020] Solve the three-dimensional spatial pose information of the target (x W ,y W ,z W ), where the coordinate point of the target in the left image is (u L ,v L ), the coordinate point of the right eye image is (u R ,v R ).
[0021] Based on the first aspect, in one embodiment of the present invention, the slave end imaging is transmitted to the master end after being distortion corrected.
[0022] In the second aspect, the present invention provides a remote operation system based on human eye gaze interaction, including a master end and a slave end. The operator of the master end wears a head-mounted display with head posture positioning and eye tracking positioning functions; the slave end is deployed with a binocular stereo camera; data communication is carried out between the master end and the slave end, and the three-dimensional position of the target is determined by the remote operation target three-dimensional position estimation method described in the first aspect.
[0023] Based on the second aspect, in one embodiment of the present invention, the posture rotation matrix of the head coordinate system is:
[0024]
[0025] Wherein, c represents the cos() function, and s represents the sin() function.
[0026] Based on the second aspect, in one embodiment of the present invention, the posture rotation matrix of the eye coordinate system relative to the head coordinate system is:
[0027]
[0028] Wherein, c represents the cos() function, and s represents the sin() function.
[0029] Based on the second aspect, in one embodiment of the present invention, according to the following formula:
[0030]
[0031] Solve the three-dimensional spatial pose information of the target (x W ,y W ,z W ), where the coordinate point of the target in the left image is (u L ,v L ), the coordinate point of the right eye image is (u R ,v R ).
[0032] Based on the second aspect, in one embodiment of the present invention, the slave end imaging is transmitted to the master end after being distortion corrected.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] (1) A method for estimating the three-dimensional position of a remote-controlled target based on human eye gaze interaction is proposed. A method for calculating the three-dimensional spatial position of a slave target based on the information of the human eye gaze point is proposed. This method does not rely on cooperative markers or pre-trained models deployed on the slave end, and can achieve rapid identification and positioning of non-cooperative targets on the slave end, thus solving the problems of limited application environment and complex algorithms of existing methods.
[0035] (2) A method for estimating the three-dimensional position of a remote-controlled target based on human eye gaze interaction is proposed. A real-time and rapid calculation method for the three-dimensional spatial information of the slave end is proposed, which provides the slave end position information to the operator at the master end in real time, improves the accuracy of the operator's perception of distance, greatly reduces the distortion caused by distortion of the head-mounted display, and ensures operational safety.
[0036] (3) A method for estimating the three-dimensional position of a remote-controlled target based on human eye gaze interaction. The three-dimensional coordinates of the target are calculated only through human eye information and are not affected by the lighting environment conditions at the slave end. This broadens the application scenarios of remote-controlled operations and is not affected by lighting conditions. Stable estimation can be performed in various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 Schematic diagram of the human eye's line of sight.
[0038] Figure 2 Schematic diagram of imaging inside the main head display.
[0039] Figure 3 Schematic diagram of imaging from the slave binocular camera. DETAILED DESCRIPTION
[0040] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0041] A method for estimating the three-dimensional position of a teleoperated target based on human eye gaze interaction calculates the three-dimensional spatial position of the target being gazed at the slave end through the information of the human eye gaze point, and can realize rapid identification and positioning of non-cooperative targets at the slave end.
[0042] A method for estimating the three-dimensional position of a remote-controlled target based on human eye gaze interaction requires the deployment of a binocular stereo camera at the slave end, and the operator at the master end should wear a head-mounted display with head posture positioning and eye tracking positioning functions.
[0043] The main steps are as follows:
[0044] (1) Imaging
[0045] The slave-end binocular camera images the three-dimensional space environment, and after distortion correction, the slave-end image is presented on a head-mounted display worn by the master-end operator.
[0046] (2) Calculate the binocular sight direction of the master end
[0047] The IMU and other sensors in the head-mounted display are used to calculate the operator's head posture information relative to the world coordinate system in real time (r h ,p h ,y h ), respectively represent the rotation angles of the head coordinate system around the X-axis, Y-axis, and Z-axis, then the posture rotation matrix of the head coordinate system is
[0048]
[0049] The posture of the human eye's line of sight relative to the head coordinate system (0, p e ,y e ),like Figure 1 As shown, p e ,y e Represent the pitch angle and yaw angle of the human eye respectively. Then the posture rotation matrix of the eye coordinate system relative to the head coordinate system is
[0050]
[0051] According to the above formula, the posture rotation matrix of the human eye coordinate system relative to the world coordinate system can be calculated:
[0052] R gaze =R head R eye
[0053] Then the eye sight vector Expressed as The formula is expanded to express
[0054]
[0055] In the formula, c represents the cos() function and s represents the sin() function. For example, cy h represents cos(y h ). The sight vectors of the left and right eyes are expressed as and
[0056] (3) Calculate the two-dimensional coordinates of the slave target in the binocular image
[0057] like Figure 2 As shown in the figure, the head-mounted display is equipped with an optical lens group and a display in front of the left and right eyes, respectively restoring the imaging effects of the left camera and the right camera in the slave environment. Therefore, the direction of the human eye's line of sight is consistent with the direction of the line connecting the slave target and the optical center of the camera, that is,
[0058]
[0059] In the formula, O SL , O SR They represent the origin of the optical center coordinate system of the left camera and the right camera at the slave end, M represents the target point in the three-dimensional space, Represent the vectors from the origin of the left eye and right eye centroid coordinate system to the target point M respectively. Figure 3 That is, the imaging effect of the left camera and the right camera at the end. When the origin of the coordinate system in the figure is O S Written as O SL When , it indicates the left camera; when the origin of the coordinate system in the figure is O S Written as O SR , it indicates the right camera.
[0060] like Figure 3 As shown, the vector of the extended sight direction and the point M where the sight falls on the image img That is, the coordinate point (u, v) of the target in the two-dimensional image. Define the coordinate point of the target in the left image as (u L ,v L ), the coordinate point of the right eye image is (u R ,v R ).
[0061] (4) Calculate the three-dimensional coordinates of the slave target in the three-dimensional space
[0062] Known calibration information of the slave binocular camera: left and right eye intrinsic parameter matrix M L 、M R , the rotation matrix R and translation vector T of the right eye center relative to the left eye center. Define the left eye center coordinate system as the measurement coordinate system. According to the limit geometry principle of binocular stereo vision, we have
[0063]
[0064]
[0065] The three-dimensional spatial pose information (x W ,y W ,z W ).
[0066] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.
[0067] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.
Claims
1. A method for estimating the three-dimensional position of a teleoperation target based on human eye gaze interaction, characterized in that: include: The slave-side binocular camera images the three-dimensional space environment, which is then presented in a head-mounted display worn by the master-side operator; The IMU in the head-mounted display calculates the operator's head posture information relative to the world coordinate system in real time (r h ,p h ,y h ), determine the posture rotation matrix of the head coordinate system; The infrared camera in the head-mounted display is used to calculate the posture of the human eye relative to the head coordinate system (0, p e ,y e ), determine the posture rotation matrix of the eye coordinate system relative to the head coordinate system; Determine the posture rotation matrix of the eye coordinate system relative to the world coordinate system according to the posture rotation matrix of the head coordinate system and the posture rotation matrix of the eye coordinate system relative to the head coordinate system, and further determine the eye sight vector; Based on the fact that the line of sight of the human eye is consistent with the line connecting the target from the end and the optical center of the camera, the point M where the line of sight falls on the image is determined according to the line of sight direction. img That is, the coordinate point (u, v) of the target in the two-dimensional image; According to the coordinate point (u, v) of the target in the two-dimensional image in the slave binocular camera, the left and right eye internal parameter matrix M L and M R , the rotation matrix R and translation vector T of the right eye center relative to the left eye center, and solve the three-dimensional spatial pose information of the target.
2. The method for estimating the three-dimensional position of a teleoperation target according to claim 1, characterized in that: The posture rotation matrix of the head coordinate system is: Wherein, c represents the cos() function, and s represents the sin() function.
3. The method for estimating the three-dimensional position of a teleoperation target according to claim 1, characterized in that: The posture rotation matrix of the eye coordinate system relative to the head coordinate system is: Wherein, c represents the cos() function, and s represents the sin() function.
4. The method for estimating the three-dimensional position of a teleoperation target according to claim 1, characterized in that: According to the following formula: Solve the three-dimensional spatial pose information of the target (x W ,y W ,z W ), where the coordinate point of the target in the left image is (u L ,v L ), the coordinate point of the right eye image is (u R ,v R ).
5. The method for estimating the three-dimensional position of a teleoperation target according to claim 1, characterized in that: The image from the slave is transmitted to the master after distortion correction.
6. A remote control system based on human eye gaze interaction, characterized in that: The invention comprises a master end and a slave end, wherein the operator of the master end wears a head mounted display with head posture positioning and eye tracking positioning functions; a binocular stereo camera is deployed at the slave end; data communication is performed between the master end and the slave end, and the three-dimensional position of the target is determined by the three-dimensional position estimation method of the remote operation target described in claim 1.
7. The teleoperation system according to claim 6, characterized in that: The posture rotation matrix of the head coordinate system is: Wherein, c represents the cos() function, and s represents the sin() function.
8. The teleoperation system according to claim 6, characterized in that: The posture rotation matrix of the eye coordinate system relative to the head coordinate system is: Wherein, c represents the cos() function, and s represents the sin() function.
9. The teleoperation system according to claim 6, characterized in that: According to the following formula: Solve the three-dimensional spatial pose information of the target (x W ,y W ,z W ), where the coordinate point of the target in the left image is (u L ,v L ), the coordinate point of the right eye image is (u R ,v R ).
10. The teleoperation system according to claim 6, characterized in that: The image from the slave is transmitted to the master after distortion correction.
Citation Information
Patent Citations
Sight tracking method based on single-purpose fixation point estimation
CN115359093A
Systems and methods for eye-operated three-dimensional object location
US20060210111A1
Cited By
Gaze point evaluation method, guide method and device integrating head movement and eye movement data
CN121277363A
Holographic wall display method based on AI target perception and head-mounted device positioning
CN122049304A
