Method and device for determining 6DOF posture based on 3D gesture recognition

Abstract

The present application relates to the field of virtual display technology and provides a method and device for determining 6DOF posture based on 3D gesture recognition. The four fingertips except the thumb are selected as marking points for positioning the 3DOF handle. The four fingertips are fixedly pressed on the preset position points of the 3DOF handle. The thumb can move the buttons without affecting the operation of the 3DOF handle. Through 3D gesture recognition technology, the 3D coordinates of the four fingertips in the virtual display device coordinate system are directly calculated. Combined with the 3D coordinates of the four fingertips in the 3DOF handle coordinate system, the relative posture of the handle and the virtual display device is obtained through 3D-3D coordinate alignment, and unified alignment of the handle and the reference coordinate system of the virtual display device is achieved. After alignment, the handle IMU integral posture and 3D gesture estimated posture are jointly optimized and output, thereby achieving the output of high-precision 6DOF posture using a low-cost, low-power, and simple-structured 3DOF handle.

Description

Technical Field

[0001] The present application relates to the field of virtual display technology, and in particular to a method and device for determining 6DOF posture based on 3D gesture recognition. Background Art

[0002] For virtual display devices such as virtual reality (VR) and augmented reality (AR), a handle is usually used to achieve conventional interaction.

[0003] At present, commonly used controllers include 3-degree-of-freedom (DOF) controllers and 6DOF controllers. The 3DoF controller has a relatively simple structure, low production cost, and relatively mature positioning technology. It mainly uses the inertial measurement unit (IMU) on the 3DOF controller to provide rotation posture. The 3DoF controller is based on rotation posture to achieve basic functions such as clicking and dragging in the virtual world. It is often used in conjunction with 3DoF virtual display devices such as VR and AR. However, the cumulative error of translation due to IMU integration is large, so the translation position of the 3DOF controller will not be used. In this way, when using a 3DOF controller, complex game actions cannot be performed due to the lack of translation position. Therefore, various manufacturers have successively launched 6DoF controllers that include rotation posture and translation position.

[0004] For 6DoF controllers, visual positioning technology is primarily used to output 6DoF pose. Specifically, an LED light ring is placed on the controller. Based on images of the LED light ring captured by a multi-camera on these virtual display devices, a visual positioning algorithm is used to output the controller's 6DoF pose relative to the AR, VR, or other virtual display devices. However, compared to 3DoF controllers that use an IMU to locate 3DoF pose, 6DoF controllers have a more complex structure and higher hardware and power costs.

[0005] Therefore, using a simple and low-cost 3DOF controller to output 6DOF pose is an urgent problem to be solved. Summary of the Invention

[0006] The embodiments of the present application provide a method and device for determining 6DOF posture based on 3D gesture recognition, which is used to reduce the cost and power consumption of a handle outputting 6DOF posture.

[0007] On the one hand, an embodiment of the present application provides a method for determining 6DOF posture based on 3D gesture recognition, which is applied to a 3DOF controller, including:

[0008] Recognize a hand gesture in a hand image captured by a multi-camera of a virtual display device, and determine first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions of the 3DOF handle in a virtual display device coordinate system;

[0009] Obtaining second 3D coordinates of the four fingertips in the 3DOF handle coordinate system;

[0010] Determining a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF handle;

[0011] Integrate the measurement data of the IMU of the 3DOF controller from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system;

[0012] The 3DOF handle is visually positioned, and the initial 6DOF pose is updated using the visual 6DOF pose.

[0013] On the other hand, an embodiment of the present application provides a virtual display device, the virtual display device including a memory, a processor, a first communication interface and a second communication interface, wherein the first communication interface, the second communication interface, the memory and the processor are connected via a bus;

[0014] The virtual display device is connected to the 3DOF handle through the first communication interface, and the virtual display device is connected to the multi-eye camera through the second communication interface;

[0015] The memory includes a data storage unit and a program storage unit, the program storage unit stores computer program instructions, and the processor performs the following operations according to the computer program instructions:

[0016] Acquire the hand image captured by the multi-camera through the second communication interface and store it in a data storage unit;

[0017] Acquire measurement data of the IMU of the 3DOF controller through the first communication interface and store the data in a data storage unit;

[0018] Recognizing a hand gesture in a hand image captured by the multi-camera, and determining first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions on the 3DOF handle in a virtual display device coordinate system;

[0019] Obtaining second 3D coordinates of the four fingertips in the 3DOF handle coordinate system;

[0020] Determining a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF handle;

[0021] Integrate the measurement data of the IMU of the 3DOF controller from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system;

[0022] The 3DOF handle is visually positioned, and the initial 6DOF pose is updated using the visual 6DOF pose.

[0023] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the method for determining 6DOF posture based on 3D gesture recognition provided by an embodiment of the present application.

[0024] The embodiments of the present application provide a method and device for determining 6DOF posture based on 3D gesture recognition, which can output 6DOF posture using a simple and low-cost 3DOF handle. This method uses hand images captured by a multi-camera virtual display device to perform gesture recognition. It determines the first 3D coordinates of the four fingertips, excluding the thumb, fixed at preset positions on a 3DOF handle in the virtual display device coordinate system. Based on the structure of the 3DOF handle itself, it obtains the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system, determines the relative pose relationship between the virtual display device and the 3DOF handle, and thus aligns the reference coordinate systems of the virtual display device and the 3DOF handle. Starting from the time point of reference coordinate system alignment, the IMU measurement data of the 3DOF handle is integrated to determine the initial 6DOF pose of the 3DOF handle in the reference coordinate system. Considering that the cumulative error of IMU integration for translational position is large but the rotational pose is relatively accurate, this method uses hand images captured by the multi-camera to visually locate the 3DOF handle to obtain the visual 6DOF pose. Since the visual positioning error is small, it can be used to update the initial 6DOF pose to reduce the cumulative error of IMU integration for translational position, thereby improving the usability of the translational position output by the 3DOF handle, and thus obtaining a 6DOF pose including translational position and rotational pose. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 Schematic diagram of application scenarios provided by embodiments of the present application;

[0027] Figure 2A Schematic diagram of a 3DOF handle provided in an embodiment of the present application;

[0028] Figure 2B A schematic diagram of a 6DOF handle provided in an embodiment of the present application;

[0029] Figure 2C A schematic diagram of a virtual display device including multiple cameras provided in an embodiment of the present application;

[0030] Figure 2D A schematic diagram of another 6DOF handle provided in an embodiment of the present application;

[0031] Figure 3 Schematic diagram of the overall solution of using a 3DOF controller to output 6DOF posture according to an embodiment of the present application;

[0032] Figure 4 Flowchart of a method for determining 6DOF pose based on 3D gesture recognition provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the process of extracting the 3D coordinates of four fingertips in the virtual display device coordinate system through gesture recognition provided in an embodiment of the present application;

[0034] Figure 6 A flow chart of a method for determining the 3D coordinates of hand joint points provided in an embodiment of the present application;

[0035] Figure 7 Schematic diagram of 21 hand joints provided in the embodiment of this application;

[0036] Figure 8 A flow chart of a method for determining the 3D coordinates of hand joints in a virtual display device coordinate system provided in an embodiment of the present application;

[0037] Figure 9 A schematic diagram of a process for improving the 3D coordinate accuracy of hand joints through gesture correction provided in an embodiment of the present application; Figure 10 A flowchart of a method for determining the position and orientation of a 3DOF controller in a reference coordinate system after alignment, provided in an embodiment of the present application;

[0038] Figure 11 A flow chart of a method for updating IMU positioning based on visual positioning provided in an embodiment of the present application;

[0039] Figure 12A flowchart of a complete method for determining 6DOF pose based on 3D gesture recognition provided in an embodiment of the present application;

[0040] Figure 13 A hardware structure diagram of a virtual display device provided in an embodiment of the present application;

[0041] Figure 14 This is a functional structure diagram of the virtual display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] Virtual display devices such as AR and VR generally refer to head-mounted display devices (referred to as HMDs or helmets, such as VR glasses and AR glasses) equipped with independent processors, which provide independent computing, input, and output capabilities. Virtual display devices can be connected to external controllers, which users use to control the virtual images displayed by the virtual display device, enabling conventional interaction. Therefore, controllers are often sold in combination with virtual display devices such as AR and VR.

[0043] Take the game scene as an example, see Figure 1 , is a schematic diagram of the application scenario of the VR device and the handle provided in the embodiment of the present application. Figure 1 This demonstrates that VR gaming can be projected onto a TV screen, leveraging the TV's large screen for enhanced entertainment. Players control the VR headset's gaming screen with a controller and react to changes in the game scene with their body movements, creating an immersive experience that makes the game more engaging.

[0044] like Figure 1 In the game scene shown, during the interaction process, the relative position of the handle and virtual display devices such as AR and VR is calculated through the handle positioning algorithm, thereby realizing three-dimensional interaction of the virtual display device in three-dimensional space and enhancing the immersive experience.

[0045] Currently, commonly used controllers include 3DOF controllers and 6DOF controllers, depending on the output posture. Among them, 3DOF controllers output 3D rotation posture, and 6DOF controllers output 3D translation position and 3D rotation posture. Compared with 3DOF controllers, 6DOF controllers can perform more complex and interesting game actions.

[0046] like Figure 2AAs shown in the figure, it is a schematic diagram of a 3DOF handle provided in an embodiment of the present application. The 3DOF handle uses the internal IMU to provide 3DOF rotation posture for 3DOF VR, AR and other virtual display devices. Specifically, based on the measurement data collected by the IMU, algorithms such as complementary filtering or Kalman filtering are used to output a stable rotation posture, so that basic functions such as clicking and dragging in the virtual world are realized based on the rotation posture of the 3DoF handle. However, the cumulative error of the translation due to the IMU integration is large, so the translation position of the 3DOF handle will not be used. In this way, when using the 3DOF handle, due to the lack of translation position, complex game actions cannot be performed, and the user experience is poor.

[0047] like Figure 2B As shown in FIG, a schematic diagram of a 6DOF handle provided in an embodiment of the present application is provided. The 6DOF handle uses visual positioning technology to provide 6DOF posture (including dimensional rotation posture and translation position) for 6DOF AR, VR and other virtual display devices. Figure 2B As shown in the figure, the 6DOF controller has an additional LED light ring compared to the ordinary 3DOF controller, and the white dot holes are the locations of each LED light. Virtual display devices such as AR and VR contain multiple cameras, usually 2 or 4, such as Figure 2C The multi-camera system on virtual display devices such as AR and VR captures the LED light ring image and uses a visual positioning algorithm to output the 6DoF position of the 6DOF controller relative to the AR and VR virtual display device. After transferring it to the virtual world, unconstrained 6DoF interactive operations can be achieved.

[0048] like Figure 2D Figure 2 shows another schematic diagram of a 6DOF controller provided by the present invention. This 6DOF controller can emit infrared light. Since infrared light is generally difficult to observe, the multi-camera of virtual display devices such as AR and VR is usually set to a cyclic switching mode between short exposure and natural exposure to observe the position of the 6DOF controller and output the 6DOF pose.

[0049] Although the 6DOF controller has an additional translation position output compared to the 3DOF controller, the structural design and circuit control of the 6DoF controller are more complex than those of 3DoF, the cost is higher, and the power consumption is larger, which is not conducive to reducing the cost of using VR and AR devices, and limits the popularization and application of VR and AR devices.

[0050] In view of this, the embodiment of the present application provides a method and device for determining 6DOF posture based on 3D gesture recognition, using a 3DOF handle to output 6DOF posture through 3D gesture recognition technology, thereby reducing the cost and power consumption of 6DOF posture. The embodiment of the present application selects four fingertips other than the thumb as the marking points for positioning the 3DOF handle. The four fingertips are fixedly pressed at the preset position points of the 3DOF handle, and the thumb can move the buttons. In this way, while ensuring the positioning accuracy, it does not affect the operation of the 3DOF handle; and through 3D gesture recognition technology, the 3D coordinates of the four fingertips in the virtual display device coordinate system can be directly calculated. Combined with the 3D coordinates of the four fingertips in the 3DOF handle coordinate system, 3D-3D coordinate alignment is achieved, and the measurement data of the IMU of the aligned 3DOF handle is integrated to obtain the relative 6D posture of the 3DOF handle and the head display, and then converted to the aligned reference coordinate system and output, thereby achieving the output of high-precision 6DOF posture using a low-cost, low-power, and simple-structured 3DOF handle. At the same time, in order to ensure the stable output of accurate fingertip 3D coordinates, the embodiment of the present application adds a hand posture correction module to the 3D gesture recognition technology, and uses the standard hand model of holding the handle as a reference to correct and optimize the recognition results, thereby outputting accurate fingertip 3D coordinates and improving the accuracy of 6DOF posture.

[0051] See also Figure 3 , is a schematic diagram of the overall solution of using a 3DOF handle to output 6DOF posture according to an embodiment of the present application. Figure 3As shown, the virtual display device has a multi-camera system that can capture images of hands holding a 3DOF handle and send them to a processor of the virtual display device. The processor performs gesture recognition on the received hand images, extracts four fingertip points fixed at preset position points of the 3DOF handle except the thumb, and calculates the 3D coordinates of the four fingertips in the virtual display device coordinate system. The preset position points of the 3DOF handle where the four fingertips are located can be determined based on the structure of the 3DOF handle itself. In this way, with the help of the preset position points, the 3D coordinates of the four fingertips in the 3DOF handle coordinate system can be obtained. By aligning the 3D coordinates in the virtual display device coordinate system and the 3D coordinates in the 3DOF handle coordinate system, the relative posture relationship between the virtual display device and the 3DOF handle is determined, and the unification of the reference coordinate systems of the virtual display device and the 3DOF handle is completed. Furthermore, starting from the time point of alignment with the reference coordinate system, the measurement data of the IMU of the 3DOF handle is integrated to determine the initial 6DOF pose of the 3DOF handle in the reference coordinate system. Considering that the cumulative error of the IMU integration for the translation position is large, but the rotation posture is relatively accurate, this scheme uses the hand images collected by the multi-camera to perform visual positioning of the 3DOF handle to obtain the visual 6DOF pose. Since the visual positioning error is small, it can be used to update the initial 6DOF pose of the 3DOF handle to reduce the cumulative error of the IMU integration for the translation position, thereby improving the availability of the translation position output by the 3DOF handle, and then obtaining the 6DOF pose including the translation position and rotation posture.

[0052] The embodiment of the present application takes into account that the thumb needs to operate the handle buttons, which is not conducive to the calculation of 3D gesture posture. Therefore, only the other four fingertips fixed at the preset position points of the 3DOF handle are selected for 6DoF positioning. Structurally, grooves are designed for these four fingertips according to ergonomics, which makes it convenient to hold the 3DOF handle and facilitate the calculation of 3D posture. On the other hand, the embodiment of the present application uses hand images captured by a multi-camera to visually locate the 3DOF handle, and uses the visual positioning results to optimize the initial 6DOF posture of the 3DOF handle, thereby improving the accuracy of the 6DOF posture.

[0053] based on Figure 3 The overall solution shown is applied to a 3DOF handle. The embodiment of the present application provides a flow chart of a method for determining 6DOF posture based on 3D gesture recognition, see Figure 4 The process is executed by a virtual display device connected to a 3DOF controller and having an independent processor. It mainly includes the following steps:

[0054] S401: Identify gestures in a hand image captured by a multi-camera of a virtual display device, and determine first 3D coordinates of four fingertips, except the thumb, fixed at preset positions on a 3DOF handle in a virtual display device coordinate system.

[0055] Generally, to facilitate the user's grip, the handle is provided with grooves (i.e., preset positions) in the handle, where the user can place the fingertips. Taking into account the differences between the thumb and the other four fingers, and the need to reserve fingers for operating the handle buttons, the embodiment of the present application places the remaining four fingertips, excluding the thumb, in the grooves of the 3DOF handle, that is, fixes the four fingertips, excluding the thumb, at the preset positions of the 3DOF handle.

[0056] See also Figure 5 , is a schematic diagram of the process of extracting the first 3D coordinates of four fingertips in the virtual display device coordinate system through gesture recognition provided by an embodiment of the present application. Figure 5 As shown, the main contents of this process include gesture area detection, hand joint point extraction, and first 3D coordinate determination. For the specific implementation process, see Figure 6 , mainly includes the following steps:

[0057] S4011: Detect the gesture area of ​​holding the 3DOF handle from the hand image captured by the multi-camera.

[0058] The outer surface of the virtual display device contains multiple cameras, each facing a different direction, which can capture hand images from different angles. Each hand image is input into a pre-trained object detection model, which is used to detect the gesture area holding the 3DOF handle, such as Figure 5 shown.

[0059] Among them, the embodiments of the present application do not impose restrictive requirements on the target detection model. For example, traditional machine learning algorithms (such as support vector machines (SVM)) can be used, and deep learning algorithms (such as convolutional neural networks (CNN) and YOLOv3 networks) can also be used.

[0060] S4012: Perform gesture estimation on the gesture area and extract hand joint points.

[0061] When executing S4012, the pre-trained hand joint detection model is used to estimate the hand gestures in each gesture area and extract 21 hand joints. Figure 7 , is a schematic diagram of 21 hand joint points provided in the embodiment of this application, each hand node corresponds to a unique identifier. Among them, the extraction technology of hand joint points is mature, and this part is not the focus of this application, so it will not be described in detail.

[0062] S4013: Determine the first 3D coordinate of each hand joint point in the virtual display device coordinate system through a multi-object matching algorithm.

[0063] In the embodiment of the present application, the depth information of each hand joint point can be determined based on the hand joint points extracted from the gesture area corresponding to the multi-camera, and the first 3D coordinate of each hand joint point in the virtual display device coordinate system can be determined by combining the pre-calibrated intrinsic parameters of each camera. Figure 8 , mainly includes the following steps:

[0064] S4013_1: Match the hand joint points extracted from the gesture area corresponding to the main camera with the hand joint points extracted from the gesture areas corresponding to other cameras.

[0065] Since multi-cameras capture hand images from different angles, one camera is selected as the primary camera based on the richness of hand information contained in the captured hand images, and the other cameras are selected as auxiliary cameras. The hand joint points extracted from the gesture area corresponding to the primary camera are then matched with the hand joint points extracted from the gesture areas corresponding to the other cameras.

[0066] S4013_2: Determine the depth information of each hand joint point based on the matching results.

[0067] Based on the hand joint points extracted in the gesture area corresponding to the main camera and the matching results of the hand joint points extracted in the gesture areas corresponding to other cameras, the distance from each hand joint point to the corresponding camera is calculated. Since the camera is located on the virtual display device, the distance from the hand joint point to the camera can be used as the distance from the hand joint point to the virtual display device, thereby obtaining the depth information of each hand joint point.

[0068] S4013_3: Determine the first 3D coordinate of each hand joint in the virtual display device coordinate system based on the pre-calibrated intrinsic parameters of the multi-camera, the depth information of each hand joint, and the image coordinates of each hand joint in the corresponding gesture area.

[0069] In S4013_3, the image coordinates of each hand joint point in the gesture area can be read directly. With the depth information as the Z axis perpendicular to the virtual display device, combined with the pre-calibrated intrinsic parameters of the multi-eye camera, the first 3D coordinate of each hand joint point in the virtual display device coordinate system can be determined.

[0070] S4014: Obtain the first 3D coordinates of the four fingertips fixed at the preset position points of the 3DOF handle from each hand joint point.

[0071] For example, still Figure 7For example, according to the identification of the 21 hand joints, the identifications of the four fingertips except the thumb are 8, 12, 16, and 20 respectively. Therefore, from the first 3D coordinates of the 21 hand joints, according to the identification of the hand joints, the first 3D coordinates of the four fingertips fixed at the preset position points of the 3DOF handle can be obtained.

[0072] Considering that the hand joint points extracted by gesture recognition may have deviations due to hand shaking or the way of holding the handle, which may increase the error of the first 3D coordinates of the hand joint points, therefore, in some embodiments, before determining the first 3D coordinates of the four fingertips in the virtual display device coordinate system, gesture 3D correction content is also included, such as Figure 9 In specific implementation, the least squares method is used to optimize the detected gestures based on a pre-established standard gesture reference model to reduce the error in determining the first 3D coordinates of the four fingertips caused by hand shaking or incorrect gestures, thereby improving the accuracy of 6DOF pose determination.

[0073] S402: Obtain the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system.

[0074] In an embodiment of the present application, the preset position points of the 3DOF handle where the four fingertips are located can be determined based on the structure of the 3DOF handle itself. In this way, with the help of the preset position points, the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system can be obtained.

[0075] When using the virtual display device and the 3DOF controller, the virtual display device and the 3DOF controller are two independent devices that can move independently. Therefore, the virtual display device and the 3DOF controller have their own reference coordinate systems and need to be aligned. For details, see S403.

[0076] S403: Determine a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinate and the second 3D coordinate to align reference coordinate systems of the virtual display device and the 3DOF handle.

[0077] Assume that the first 3D coordinates of the four fingertips in the virtual display device coordinate system are The second 3D coordinates in the 3DOF handle coordinate system are Determine the relative position relationship between the virtual display device and the 3DOF controller by aligning the first 3D coordinate in the virtual display device coordinate system and the second 3D coordinate in the 3DOF controller coordinate system The formula is as follows:

[0078]

[0079] Through the above relative posture relationship The position of the 3DOF handle in the first reference coordinate system can be converted to the second reference coordinate system where the virtual display device is located, or the position of the virtual display device in the second reference coordinate system can be converted to the first reference coordinate system where the 3DOF handle is located, so as to achieve alignment of the reference coordinate systems.

[0080] S404: Integrate the measurement data of the IMU of the 3DOF controller from the reference coordinate system alignment time point to determine the initial 6DOF position of the 3DOF controller in the reference coordinate system.

[0081] After the reference coordinate systems of the virtual display device and the 3DOF controller are aligned, the pose of the virtual display device and the 3DOF controller can be determined in the same reference coordinate system (e.g., the second reference coordinate system in which the virtual display device resides). The 6DOF pose of the virtual display device in the reference coordinate system can be directly read out by the positioning device within the virtual display device. The pose of the 3DOF controller is predicted using a Kalman filter starting from the time of reference coordinate system alignment. This involves integrating the measurement data of the 3DOF controller's IMU in the aligned reference coordinate system to complete initial positioning.

[0082] The process of determining the position of the 3DOF controller in the reference coordinate system after alignment is described in Figure 10 , mainly includes the following steps:

[0083] S4041: Obtain the acceleration measurement value of the accelerometer in the IMU at the time point of reference coordinate system alignment, and perform a quadratic integration of the acceleration measurement value in the time dimension to obtain the translation position of the 3DOF controller in the reference coordinate system.

[0084] According to the mathematical relationship between acceleration, velocity, and displacement, starting from the alignment time point of the reference coordinate system, the acceleration measurement value collected by the accelerometer in the IMU is integrated once in the time dimension to obtain the velocity information of the 3DOF handle. After integrating the velocity once (that is, integrating the acceleration measurement value twice), the translation position of the 3DOF handle in the reference coordinate system (that is, displacement information) can be obtained.

[0085] S4042: Obtain the angular velocity measurement value of the gyroscope in the IMU at the time point of alignment with the reference coordinate system, and integrate the angular velocity measurement value once in the time dimension to obtain the rotation posture of the 3DOF controller in the reference coordinate system.

[0086] According to the mathematical relationship between rotation angle and angular velocity, the rotational posture of the 3DOF controller in the reference coordinate system (i.e., the three-axis rotation angle) can be obtained by integrating the angular velocity measurements collected by the gyroscope in the IMU in the time dimension starting from the alignment time of the reference coordinate system.

[0087] S4043 determines the initial 6DOF position of the 3DOF handle in the reference coordinate system based on the translation position and rotation posture.

[0088] Among them, the first three dimensions of the 6DOF posture are translation positions, and the last three dimensions are rotation postures.

[0089] However, since IMU integration will drift, the longer the accumulation time, the greater the offset. The offset has a smaller impact on the rotational posture and a greater impact on the translation position, which will lead to inaccurate translation position. Therefore, the initial 6DOF posture needs to be corrected.

[0090] S405: Perform visual positioning on the 3DOF controller and use the visual 6DOF pose to update the initial 6DOF pose.

[0091] In the embodiment of the present application, in order to offset the cumulative error of IMU integral positioning, the 3DOF handle can be visually positioned and the initial 6DOF pose can be updated with the visual 6DOF pose. Figure 11 , mainly includes the following steps:

[0092] S4051: Starting from the reference coordinate system alignment time point, the 3DOF handle is tracked using the hand images collected by the multi-camera to redetermine the relative position relationship between the virtual display device and the 3DOF handle.

[0093] In an embodiment of the present application, after the gesture area is detected in the hand image captured by the multi-camera, the 3DOF handle is tracked using the hand image from the time point of alignment with the reference coordinate system, the gesture area at the current moment is re-determined, and the first 3D coordinates of the hand joint points extracted within the gesture area at the current moment are determined. Combined with the second 3D coordinates of each hand joint point at the current moment, Formula 1 is used to re-determine the relative posture relationship between the virtual display device and the 3DOF handle at the current moment.

[0094] In S4051, compared with performing gesture area detection on each frame of hand image, the tracking method can save calculation amount and improve positioning speed.

[0095] S4052: Determine the visual 6DOF pose of the 3DOF controller in the reference coordinate system based on the 6DOF pose of the virtual display device in the reference coordinate system and the new relative pose relationship.

[0096] In S4052, the 6DOF pose of the virtual display device in the reference coordinate system can be directly read out by the positioning device in the virtual display device. Combined with the new relative pose relationship at the current moment, the visual 6DOF pose of the 3DOF handle in the reference coordinate system can be determined.

[0097] S4053: Update the initial 6DOF pose using the visual 6DOF pose.

[0098] Generally, the IMU's measurement frequency is higher than the camera's acquisition frame rate. This means that within the time it takes for the camera to capture two consecutive frames of hand images, the IMU has already completed an integration process. Therefore, the visual positioning results from each frame of the multi-camera can be used to update the initial 6DOF pose of the 3DOF controller. Due to the high accuracy of the 6DOF pose obtained through visual positioning, the accumulated error caused by the IMU's integration of translational position can be accurately corrected, improving positioning accuracy.

[0099] In an optional embodiment, the method of updating IMU positioning results with visual positioning results can be implemented by introducing a Kalman filter. A Kalman filter is an error estimation algorithm that uses state equations to minimize the system variance based on predicted and observed data input into a linear system. Its core concept is to first select a random dynamic variable in the system, establish a prediction model, and then calculate the optimal estimate from the state equation based on the system's real-time observation data. This is a continuous "prediction-update" computational process with the advantages of low data processing volume and strong real-time performance.

[0100] In the prediction part of the embodiment of the present application, the angular velocity and acceleration measurements of the IMU in the 3DOF controller are integrated separately to roughly calculate the initial 6DOF pose of the 3DOF controller in the reference coordinate system. In addition, during the integration process, the covariance matrix of the prediction error is iteratively calculated: k+1 =FP k F T +Q. Among them, F represents the predicted matrix of posture, Q is the Gaussian white noise matrix, P k is the covariance matrix of the previous moment. Due to Gaussian white noise and random walk, the longer the prediction time, the greater the error caused by drift.

[0101] In the update section of this embodiment, after successful visual positioning, the relative position relationship between the virtual display device and the 3DOF controller is obtained. Combined with the 6DOF position of the virtual display device in the reference coordinate system, the 6DOF position of the 3DOF controller in the reference coordinate system can be inferred. The 6DOF position obtained by visual positioning is highly accurate and can be used to update the controller IMU integral position. During the update process, the Kalman filter calculates the Kalman gain, updates the measured value based on the predicted value, and suppresses noise interference.

[0102] It should be noted that the embodiment of the present application uses a linear Kalman filter to complete the fusion positioning of vision and IMU. This is only an example and is not a restrictive requirement of the embodiment of the present application. For example, a nonlinear algorithm can also be used to optimize the IMU positioning results using the visual positioning results. In specific implementation, the IMU pre-integration theory is adopted, and the reprojection error of the 6DOF pose obtained by N visual positioning is used together with the IMU pre-integration residual to jointly optimize the 6DOF pose of the 3DOF handle.

[0103] In some embodiments, since the IMU integration will continuously predict the 6DOF pose of the handle, low-probability handle tracking anomalies will not affect the output handle positioning results, but only affect the positioning accuracy. However, if the tracking fails for a long time, it indicates that the handle has moved out of the camera's field of view. At this time, it will seriously affect the translation positioning accuracy, and it is possible to fall back to the optimization of the rotation posture. Specifically, when the 3DOF handle fails to be tracked using the hand image collected by the multi-camera, and the tracking failure duration is greater than the set time threshold, the rotation information in the initial 6DOF pose is optimized to obtain the 3DOF pose of the 3DOF handle in the reference coordinate system. However, when the 3DOF handle moves into the camera's field of view again, it will be tracked again, and the Kalman filter process will be restarted to continue outputting the 6DoF pose of the 3DOF handle.

[0104] See also Figure 12 , is a flowchart of a complete method for outputting 6DOF poses using a 3DOF controller according to an embodiment of the present application, which mainly includes the following steps:

[0105] S1201: Acquire a hand image captured by a multi-camera.

[0106] S1202: Recognize the hand gesture in the hand image, and determine the first 3D coordinates of four fingertips, except the thumb, fixed at preset positions of the 3DOF handle in the virtual display device coordinate system.

[0107] S1203: Determine whether the visual initialization positioning is successful. If not, execute S1204; if so, execute S1207.

[0108] If the first reference coordinate system of the 3DOF controller is aligned with the second reference coordinate system of the virtual display device, the visual initialization positioning is successful; otherwise, it indicates that the visual initialization positioning has failed.

[0109] S1204: Obtain the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system.

[0110] S1205: Determine a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinate and the second 3D coordinate to align reference coordinate systems of the virtual display device and the 3DOF handle.

[0111] S1206: Starting from the reference coordinate system alignment time point, start the Kalman filter to integrate the measurement data of the IMU of the 3DOF controller to determine the initial 6DOF position of the 3DOF controller in the reference coordinate system.

[0112] S1207: Determine whether the visual tracking positioning is successful. If so, execute S1208; otherwise, execute S1211.

[0113] If the hand image captured by the multi-camera is tracked to the 3DOF handle, it indicates that the visual tracking positioning is successful, otherwise it fails.

[0114] S1208: Re-determine the relative position relationship between the virtual display device and the 3DOF handle.

[0115] S1209: Determine the visual 6DOF pose of the 3DOF controller in the reference coordinate system according to the 6DOF pose of the virtual display device in the reference coordinate system and the new relative pose relationship.

[0116] S1210: Use Kalman filtering to update the initial 6DOF pose using the visual 6DOF pose.

[0117] S1211: When the visual positioning tracking failure duration is greater than the set time threshold, the rotation information in the initial 6DOF pose is optimized to obtain the 3DOF pose of the 3DOF handle in the reference coordinate system.

[0118] The embodiment of the present application utilizes a 3DOF handle with a simple structure, low cost, and low power consumption to provide a method for determining 6DOF posture based on 3D gesture recognition. In this method, gesture recognition is performed with the help of hand images captured by the multi-camera of the virtual display device to determine the first 3D coordinates of the four fingertips fixed at the preset position points of the 3DOF handle except the thumb in the virtual display device coordinate system. The thumb can move the buttons. In this way, while ensuring the positioning accuracy, it does not affect the operation of the 3DOF handle; and based on the structure of the 3DOF handle itself, the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system are obtained to determine the relative posture relationship between the virtual display device and the 3DOF handle, thereby aligning the virtual display device and the reference coordinate system of the 3DOF handle, and integrate the measurement data of the IMU of the 3DOF handle from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF handle in the reference coordinate system. Considering that the cumulative error of IMU integration for translation position is large, but it is more accurate for rotation posture, this method uses the hand images collected by multiple cameras to perform visual positioning of the 3DOF handle to obtain the visual 6DOF pose. Since the visual positioning error is small, it can be used to update the initial 6DOF pose to reduce the cumulative error of IMU integration for translation position, thereby improving the availability of the translation position output by the 3DOF handle, and then obtaining the 6DOF pose including translation position and rotation posture.

[0119] Based on the same technical concept, an embodiment of the present application provides a virtual display device, which can be a VR device or an AR device, and can implement the method steps of determining 6DOF posture based on 3D gesture recognition in the above embodiment, and can achieve the same technical effect.

[0120] See also Figure 13 The virtual display device includes a processor 1301, a memory 1302, a multi-eye camera 1303 and a communication interface 1304, wherein the communication interface 1304, the multi-eye camera 1303, the memory 1302 and the processor 1301 are connected via a bus 1305;

[0121] The memory 1302 includes a data storage unit and a program storage unit. The program storage unit stores computer program instructions. The processor 1301 performs the following operations according to the computer program instructions:

[0122] The virtual display device is connected to the 3DOF handle via the communication interface 1304, obtains the measurement data of the IMU of the 3DOF handle, and stores the data in the data storage unit;

[0123] Acquire the hand image captured by the multi-camera 1303 and store it in a data storage unit;

[0124] Recognizing a hand gesture in a hand image captured by the multi-camera, and determining first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions on the 3DOF handle in a virtual display device coordinate system;

[0125] Obtaining second 3D coordinates of the four fingertips in the 3DOF handle coordinate system;

[0126] Determining a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF handle;

[0127] Integrate the measurement data of the IMU of the 3DOF controller from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system;

[0128] The 3DOF handle is visually positioned, and the initial 6DOF pose is updated using the visual 6DOF pose.

[0129] Optionally, the processor 1301 performs visual positioning on the 3DOF handle and updates the initial 6DOF pose using a visual 6DOF pose, specifically by:

[0130] Starting from the reference coordinate system alignment time point, the 3DOF handle is tracked using the hand image captured by the multi-camera to re-determine the relative position relationship between the virtual display device and the 3DOF handle;

[0131] Determining the visual 6DOF pose of the 3DOF controller in the reference coordinate system according to the 6DOF pose of the virtual display device in the reference coordinate system and the new relative pose relationship;

[0132] The initial 6DOF pose is updated with the visual 6DOF pose.

[0133] Optionally, when tracking the 3DOF handle using the hand image captured by the multi-camera fails, and the tracking failure duration is greater than a set time threshold, the processor 1301 further executes:

[0134] The rotation information in the initial 6DOF pose is optimized to obtain the 3DOF pose of the 3DOF handle in the reference coordinate system.

[0135] Optionally, before determining the first 3D coordinates of the four fingertips in the virtual display device coordinate system, the processor 1301 further executes:

[0136] According to a pre-established standard gesture reference model, the detected gesture is optimized using the least square method to reduce the determination error of the first 3D coordinates of the four fingertips caused by hand shaking or erroneous gestures.

[0137] Optionally, the processor 1301 recognizes a gesture in a hand image captured by a multi-camera of a virtual display device, and determines first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions of the 3DOF handle in a virtual display device coordinate system, specifically by:

[0138] Detecting a gesture area of ​​holding the 3DOF handle from the hand image captured by the multi-camera;

[0139] Performing gesture estimation on the gesture area and extracting hand joint points;

[0140] The first 3D coordinates of each hand joint point in the virtual display device coordinate system are determined through a multi-objective matching algorithm; the first 3D coordinates of the four fingertips fixed at the preset position points of the 3DOF handle are obtained from each hand joint point.

[0141] Optionally, the processor 1301 determines the first 3D coordinate of each hand joint point in the virtual display device coordinate system through a multi-objective matching algorithm, specifically by:

[0142] Match the hand joint points extracted from the gesture area corresponding to the main camera with the hand joint points extracted from the gesture areas corresponding to other cameras;

[0143] According to each matching result, the depth information of each hand joint point is determined;

[0144] According to the pre-calibrated intrinsic parameters of the multi-camera, the depth information of each hand joint point, and the image coordinates of each hand joint point in the corresponding gesture area, the first 3D coordinates of each hand joint point in the virtual display device coordinate system are determined.

[0145] Optionally, the processor 1301 integrates the measurement data of the IMU of the 3DOF controller from the reference coordinate system alignment time point to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system, specifically by:

[0146] Obtaining an acceleration measurement value of an accelerometer in the IMU at a time point aligned with the reference coordinate system, and performing a quadratic integration of the acceleration measurement value in the time dimension to obtain a translation position of the 3DOF controller in the reference coordinate system;

[0147] Obtaining the angular velocity measurement value of the gyroscope in the IMU at the time point of alignment with the reference coordinate system, and integrating the acceleration measurement value once in the time dimension to obtain the rotation posture of the 3DOF controller in the reference coordinate system;

[0148] An initial 6DOF position of the 3DOF handle in the reference coordinate system is determined according to the translation position and the rotation posture.

[0149] Optionally, a formula for determining the relative posture relationship between the virtual display device and the 3DOF handle is:

[0150]

[0151] in, respectively represent the first 3D coordinates of the four fingertips in the virtual display device coordinate system, Respectively represent the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system.

[0152] It should be noted that Figure 13 This is only an example, and provides the hardware necessary for the virtual display device to execute the steps of the method for determining 6DOF posture based on 3D gesture recognition provided in the embodiments of the present application. If not shown, the virtual display device may also include conventional hardware such as left and right eyeglasses, speakers, and microphones.

[0153] Embodiments of the present application Figure 13 The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0154] See also Figure 14 , which is a functional structure diagram of a virtual display device capable of implementing a method for determining 6DOF pose based on 3D gesture recognition, provided in an embodiment of the present application. The virtual display device includes a visual positioning module 1401, an acquisition module 1402, a coordinate system alignment module 1403, an IMU positioning module 1404, and a pose update module 1405, wherein:

[0155] A visual positioning module 1401 is configured to recognize hand gestures in a hand image captured by a multi-camera of a virtual display device, and determine first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions on the 3DOF handle in a virtual display device coordinate system;

[0156] An acquisition module 1402 is configured to acquire second 3D coordinates of the four fingertips in a 3DOF handle coordinate system;

[0157] A coordinate system alignment module 1403 is configured to determine a relative position relationship between the virtual display device and the 3DOF controller based on the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF controller;

[0158] An IMU positioning module 1404 is configured to integrate the IMU measurement data of the 3DOF controller from the reference coordinate system alignment time point to determine the initial 6DOF position of the 3DOF controller in the reference coordinate system;

[0159] The pose updating module 1405 is used to perform visual positioning on the 3DOF handle and update the initial 6DOF pose with a visual 6DOF pose.

[0160] The above functional modules cooperate with each other to realize the method steps of determining 6DOF posture based on 3D gesture recognition, and can achieve the same technical effect. The specific implementation of each functional module can be found in the above embodiment and will not be repeated here.

[0161] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0162] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0163] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for determining 6DOF pose based on 3D gesture recognition, characterized in that: Applicable to 3DOF controllers, including: Recognize a hand gesture in a hand image captured by a multi-camera of a virtual display device, and determine first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions of the 3DOF handle in a virtual display device coordinate system; Obtaining second 3D coordinates of the four fingertips in the 3DOF handle coordinate system; Determining a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF handle; Integrate the measurement data of the IMU of the 3DOF controller from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system; The 3DOF handle is visually positioned, and the initial 6DOF pose is updated using the visual 6DOF pose.

2. The method according to claim 1, wherein The visual positioning of the 3DOF handle and updating the initial 6DOF pose using the visual 6DOF pose include: Starting from the reference coordinate system alignment time point, the 3DOF handle is tracked using the hand image captured by the multi-camera to re-determine the relative position relationship between the virtual display device and the 3DOF handle; Determining the visual 6DOF pose of the 3DOF controller in the reference coordinate system according to the 6DOF pose of the virtual display device in the reference coordinate system and the new relative pose relationship; The initial 6DOF pose is updated with the visual 6DOF pose.

3. The method according to claim 2, wherein When tracking of the 3DOF handle using the hand image captured by the multi-camera fails, and the tracking failure duration is greater than a set time threshold, the method further includes: The rotation information in the initial 6DOF pose is optimized to obtain the 3DOF pose of the 3DOF handle in the reference coordinate system.

4. The method according to claim 1, wherein Before determining the first 3D coordinates of the four fingertips in the virtual display device coordinate system, the method further includes: According to a pre-established standard gesture reference model, the detected gesture is optimized using the least square method to reduce the determination error of the first 3D coordinates of the four fingertips caused by hand shaking or erroneous gestures.

5. The method according to claim 1, wherein The step of identifying a gesture in a hand image captured by a multi-camera of a virtual display device and determining first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions of the 3DOF handle in a virtual display device coordinate system includes: Detecting a gesture area of ​​holding the 3DOF handle from the hand image captured by the multi-camera; Performing gesture estimation on the gesture area and extracting hand joint points; Determine the first 3D coordinate of each hand joint point in the virtual display device coordinate system through a multi-objective matching algorithm; The first 3D coordinates of four fingertips fixed at preset position points of the 3DOF handle are obtained from each hand joint point.

6. The method according to claim 5, wherein Determining the first 3D coordinate of each hand joint point in the virtual display device coordinate system by a multi-objective matching algorithm includes: Match the hand joint points extracted from the gesture area corresponding to the main camera with the hand joint points extracted from the gesture areas corresponding to other cameras; According to each matching result, the depth information of each hand joint point is determined; According to the pre-calibrated intrinsic parameters of the multi-camera, the depth information of each hand joint point, and the image coordinates of each hand joint point in the corresponding gesture area, the first 3D coordinates of each hand joint point in the virtual display device coordinate system are determined.

7. The method according to claim 1, wherein The step of integrating the IMU measurement data of the 3DOF controller from the reference coordinate system alignment time point to determine the initial 6DOF position of the 3DOF controller in the reference coordinate system includes: Obtaining an acceleration measurement value of an accelerometer in the IMU at a time point aligned with the reference coordinate system, and performing a quadratic integration of the acceleration measurement value in the time dimension to obtain a translation position of the 3DOF controller in the reference coordinate system; Obtaining the angular velocity measurement value of the gyroscope in the IMU at the time point of alignment with the reference coordinate system, and integrating the acceleration measurement value once in the time dimension to obtain the rotation posture of the 3DOF controller in the reference coordinate system; An initial 6DOF position of the 3DOF handle in the reference coordinate system is determined according to the translation position and the rotation posture.

8. The method according to any one of claims 1 to 7, wherein The formula for determining the relative posture relationship between the virtual display device and the 3DOF handle is: in, respectively represent the first 3D coordinates of the four fingertips in the virtual display device coordinate system, Respectively represent the second 3D coordinates of the four fingertips in the 3DOF handle coordinate system.

9. A virtual display device, characterized in that: The virtual display device includes a memory, a processor, a multi-eye camera and a communication interface, wherein the communication interface, the multi-eye camera, the memory and the processor are connected via a bus; The memory includes a data storage unit and a program storage unit, the program storage unit stores computer program instructions, and the processor performs the following operations according to the computer program instructions: The virtual display device is connected to the 3DOF handle through the communication interface, obtains the measurement data of the IMU of the 3DOF handle, and stores the data in the data storage unit; Acquire the hand image captured by the multi-camera and store it in a data storage unit; Recognizing a hand gesture in a hand image captured by the multi-camera, and determining first 3D coordinates of four fingertips, excluding the thumb, fixed at preset positions on the 3DOF handle in a virtual display device coordinate system; Obtaining second 3D coordinates of the four fingertips in the 3DOF handle coordinate system; Determining a relative position relationship between the virtual display device and the 3DOF handle according to the first 3D coordinates and the second 3D coordinates to align reference coordinate systems of the virtual display device and the 3DOF handle; Integrate the measurement data of the IMU of the 3DOF controller from the time point of alignment of the reference coordinate system to determine the initial 6DOF pose of the 3DOF controller in the reference coordinate system; The 3DOF handle is visually positioned, and the initial 6DOF pose is updated using the visual 6DOF pose.

10. The virtual display device according to claim 9, wherein: The processor performs visual positioning on the 3DOF handle and updates the initial 6DOF pose using the visual 6DOF pose, specifically by: Starting from the reference coordinate system alignment time point, the 3DOF handle is tracked using the hand image captured by the multi-camera to re-determine the relative position relationship between the virtual display device and the 3DOF handle; Determining the visual 6DOF pose of the 3DOF controller in the reference coordinate system according to the 6DOF pose of the virtual display device in the reference coordinate system and the new relative pose relationship; The initial 6DOF pose is updated with the visual 6DOF pose.

Images