Handle, head-mounted display, positioning method, apparatus, and readable storage medium

Abstract

The disclosure provides a handle, a head-mounted display, a positioning method, a device and a readable storage medium, and relates to the technical field of virtual reality. The handle comprises: a first main chip; a first image acquisition module connected with the first main chip, used for acquiring at least two handle perspective images of the virtual reality handle at any moment during movement; and the first main chip is used for: performing self pose calculation based on the at least two handle perspective images to generate pose information of the virtual reality handle. Through the technical scheme of the disclosure, the detection of the self pose can be realized without relying on the head-mounted display. For the handle itself, since an LED light emitter does not need to be arranged, the volume of the handle can be reduced, and then the shape of the handle can be improved to improve the holding experience of the user.

Description

Technical Field

[0001] This disclosure relates to the field of virtual reality technology, and in particular to virtual reality controllers, head-mounted displays, virtual reality controller positioning methods, pose information processing methods, virtual reality controller positioning devices, pose information processing devices, and computer-readable storage media. Background Technology

[0002] In related technologies, for VR (Virtual Reality) controllers, the LED emitters on the controllers are tracked using a camera on the VR HMD (Head-Mounted Display) to obtain LED tracking data. This data is then combined with the 6DoF (Degree of Freedom) data reported by the controller's IMU (Inertial Measurement Unit) to the HMD to determine the controller's instantaneous position, attitude, and motion trajectory. However, this approach currently has the following drawbacks:

[0003] On the one hand, the controllers are bulky due to the need to install and hide the LED emitters, which affects the user's grip experience. On the other hand, although four cameras specifically designed to track the LED emitters are installed on the VR HMD, there are still blind spots in the tracking of the controllers, resulting in a poor user experience.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a virtual reality controller, a head-mounted display, a virtual reality controller positioning method, a pose information processing method, a virtual reality controller positioning device, a pose information processing device, and a computer-readable storage medium, which at least to some extent overcome the problem of poor user experience when using a head-mounted display to track the controller in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a virtual reality controller is provided, comprising: a first main chip; a first image acquisition module connected to the first main chip, configured to acquire at least two controller view images at any time during the movement of the virtual reality controller; the first main chip is configured to: perform its own pose calculation based on the at least two controller view images to generate pose information of the virtual reality controller.

[0008] In one embodiment of this disclosure, the first main chip performs its own pose calculation based on the at least two controller view images, specifically including: performing visual real-time localization and motion trajectory vslam processing based on the at least two controller view images, so as to generate the pose information of the virtual reality controller based on the processing results.

[0009] In one embodiment of this disclosure, the first main chip performs real-time visual localization and motion trajectory VSLAM processing based on the at least two controller viewpoint images, specifically including: performing motion estimation on adjacent frame images in the at least two controller viewpoint images based on visual odometry (VO); generating the motion trajectory of the virtual reality controller based on the motion estimation result; generating pose prediction information of the virtual reality controller based on the motion trajectory; optimizing the pose prediction information for trajectory drift; and obtaining the pose information based on the optimization result.

[0010] In one embodiment of this disclosure, it further includes: an inertial measurement unit (IMU) module connected to the first main chip, used to collect IMU data from the virtual reality controller; the first main chip performs its own pose calculation based on the at least two controller view images, specifically including: performing visual inertial odometry (VIO) pose estimation based on the at least two controller view images and the IMU data to generate pose information of the virtual reality controller.

[0011] In one embodiment of this disclosure, the first main chip performs visual inertial odometry (VIO) pose estimation based on the at least two controller view images and the IMU data, specifically including: performing visual-inertial joint initialization on the at least two controller view images and the IMU data; synchronizing the visual measurement trajectory obtained based on the two view images and the inertial measurement trajectory obtained based on the IMU data to obtain synchronization data; performing visual motion tracking based on the synchronization data during the controller movement to perform feature point alignment and 3D structure optimization of multiple frames of controller view images; detecting keyframes based on the alignment and optimization results; and performing local co-view keyframe optimization on the keyframes to determine the pose information based on the optimization results.

[0012] In one embodiment of this disclosure, the first main chip performs visual inertial odometry (VIO) pose estimation based on the at least two controller viewpoint images and the IMU data, specifically including: pre-integrating the IMU data to obtain initial pose prediction information; performing feature point matching and triangulation on the at least two controller viewpoint images to obtain three-dimensional data, and creating a sparse depth map based on the three-dimensional data; tracking the virtual reality controller based on the initial pose prediction information, the sparse depth map, and pre-stored spatial data; acquiring keyframes based on the tracking operation; updating feature points based on the keyframes; performing beam adjustment optimization based on the update results to optimize the pose of the virtual reality controller and its spatial coordinates; performing position recognition on the virtual reality controller based on the optimization results to obtain pose recognition information; and performing loop closure fusion optimization on the pose recognition information to obtain the pose information.

[0013] In one embodiment of this disclosure, the first image acquisition module includes at least two first camera modules.

[0014] In one embodiment of this disclosure, it further includes: a first communication module, connected to the first main chip and communicating with the head-mounted display, for sending the pose information to the head-mounted display, and for the head-mounted display to perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0015] In one embodiment of this disclosure, it further includes: an infrared emitting module for transmitting infrared signals and configured to cooperate with the first image acquisition module; the first image acquisition module is further configured to: track the infrared signals; the first main chip is further configured to: identify infrared signal points in the view image to assist in the positioning of the virtual reality controller based on the infrared signal points.

[0016] According to another aspect of this disclosure, a head-mounted display is provided, comprising: a second main chip; a second communication module connected to the second main chip, configured to receive pose information sent by a virtual reality controller; the second main chip being configured to: perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0017] In one embodiment of this disclosure, it further includes: a second image acquisition module electrically connected to the second main chip, comprising two second camera modules for acquiring head-view images; the second main chip is also used for: visually positioning the head-mounted display based on the head-view images.

[0018] According to another aspect of this disclosure, a method for positioning a virtual reality controller is provided, comprising: acquiring at least two controller view images at any time during the movement of the virtual reality controller; and performing pose calculation based on the at least two controller view images to generate pose information of the virtual reality controller.

[0019] In one embodiment of this disclosure, the step of calculating the pose of the virtual reality controller based on the at least two controller view images to generate the pose information of the virtual reality controller includes: performing visual real-time localization and motion trajectory vslam processing based on the at least two controller view images to generate the pose information of the virtual reality controller.

[0020] In one embodiment of this disclosure, the step of performing visual real-time localization and motion trajectory VSLAM processing based on the at least two controller viewpoint images includes: performing motion estimation on adjacent frame images in the at least two controller viewpoint images based on visual odometry (VO); generating the motion trajectory of the virtual reality controller based on the motion estimation result; generating pose prediction information of the virtual reality controller based on the motion trajectory; performing trajectory drift optimization on the pose prediction information; and obtaining the pose information based on the optimization result.

[0021] In one embodiment of this disclosure, the step of calculating the pose of the virtual reality controller based on the at least two controller view images to generate the pose information of the virtual reality controller includes: acquiring IMU data of the virtual reality controller; and performing visual inertial odometry (VIO) pose estimation based on the at least two controller view images and the IMU data to generate the pose information of the virtual reality controller.

[0022] In one embodiment of this disclosure, the step of performing visual-inertial odometry (VIO) pose estimation based on the at least two controller viewpoint images and the IMU data to generate pose information for the virtual reality controller includes: performing visual-inertial joint initialization on the at least two controller viewpoint images and the IMU data; synchronizing the visual measurement trajectory obtained based on the two viewpoint images and the inertial measurement trajectory obtained based on the IMU data to obtain synchronization data; performing visual motion tracking based on the synchronization data during controller movement to perform feature point alignment and 3D structure optimization of multiple frames of controller viewpoint images; detecting keyframes based on the alignment and optimization results; and performing local co-view keyframe optimization on the keyframes to determine the pose information based on the optimization results.

[0023] In one embodiment of this disclosure, the step of performing visual inertial odometry (VIO) pose estimation based on the at least two controller viewpoint images and the IMU data to generate pose information of the virtual reality controller includes: performing feature point matching and triangulation on the at least two controller viewpoint images to obtain three-dimensional data, and creating a sparse depth map based on the three-dimensional data; tracking the virtual reality controller based on the initial pose prediction information, the sparse depth map, and pre-stored map data; acquiring keyframes based on the tracking operation; updating feature points based on the keyframes; performing bundle adjustment optimization based on the update results to optimize the pose of the virtual reality controller and its spatial coordinates; performing position recognition on the virtual reality controller based on the optimization results to obtain pose recognition information; and performing loop closure fusion optimization on the pose recognition information to obtain the pose information.

[0024] In one embodiment of this disclosure, the method further includes: identifying infrared signal points in the viewpoint image to assist in the positioning of the virtual reality controller based on the infrared signal points.

[0025] According to another aspect of this disclosure, a pose information processing method is provided, comprising: receiving pose information sent by a virtual reality controller; and performing coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0026] In one embodiment of this disclosure, the method further includes: acquiring a head-view image; and performing visual positioning of the head-mounted display based on the head-view image.

[0027] According to another aspect of this disclosure, a virtual reality controller positioning device is provided, comprising: a data acquisition module for acquiring at least two controller view images at any time during the movement of the virtual reality controller; and a pose calculation module for performing pose calculation on itself based on the at least two controller view images to generate pose information of the virtual reality controller.

[0028] According to another aspect of this disclosure, a pose information processing method is provided, comprising: a receiving module for receiving pose information sent by a virtual reality controller; and a fusion module for performing coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0029] According to another aspect of this disclosure, a virtual reality controller is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the virtual reality controller positioning method of any of the above-mentioned methods by executing the executable instructions.

[0030] According to another aspect of this disclosure, a head-mounted display is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the virtual reality controller positioning method of any of the above-described embodiments by executing the executable instructions.

[0031] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the virtual reality controller positioning method described above.

[0032] The virtual reality controller positioning scheme provided in this disclosure involves setting a first image acquisition module on the virtual reality controller to acquire images from the controller's perspective. The controller can acquire at least two viewpoint images during movement. A first main chip on the controller performs pose calculations based on these viewpoint images to obtain the controller's own pose information. Compared to schemes that use cameras on a VR headset to visually track LED emitters on the controller, this scheme does not rely on a headset to detect the controller's pose. Furthermore, the controller itself does not require LED emitters, thus reducing its size and allowing for improved shape design to enhance the user's grip experience.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0035] Figure 1 This diagram shows a schematic block diagram of a virtual reality controller structure according to an embodiment of the present disclosure;

[0036] Figure 2 A schematic block diagram of another virtual reality controller structure in an embodiment of this disclosure is shown;

[0037] Figure 3 This diagram shows a schematic block diagram of a head-mounted display structure according to an embodiment of the present disclosure;

[0038] Figure 4 This diagram illustrates an interaction between a virtual reality controller and a head-mounted display according to an embodiment of the present disclosure.

[0039] Figure 5 This diagram illustrates the architecture of the main chip in a virtual reality controller according to an embodiment of the present disclosure.

[0040] Figure 6 A flowchart illustrating a virtual reality controller positioning method according to an embodiment of this disclosure is shown;

[0041] Figure 7 A schematic diagram of another virtual reality controller positioning method in an embodiment of this disclosure is shown;

[0042] Figure 8 A flowchart illustrating another virtual reality controller positioning method in an embodiment of this disclosure is shown;

[0043] Figure 9 This diagram illustrates a pose information processing method according to an embodiment of the present disclosure.

[0044] Figure 10 This diagram illustrates a virtual reality controller positioning device according to an embodiment of the present disclosure;

[0045] Figure 11 This diagram illustrates a pose information processing device according to an embodiment of the present disclosure.

[0046] Figure 12 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0047] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0049] To facilitate understanding, the following is an explanation of several terms used in this application.

[0050] VR (Virtual Reality) encompasses technologies such as computers, electronic information, and simulation. Its basic implementation method is to simulate a virtual environment through computers to give people a sense of immersion.

[0051] MR (Mixed Reality) is a further development of virtual reality technology. It enhances the realism of the user experience by presenting virtual scene information in real scenes and building an interactive feedback loop between the real world, the virtual world and the user.

[0052] VR devices: Devices that utilize VR technology for entertainment or commercial purposes, including VR head-mounted displays (HMDs) and hand controllers.

[0053] Hand controller / controller: This is a device held by the user in a VR device, responsible for controlling and interacting with the software system.

[0054] VR Head-Mounted Display (HMD): This is a user-worn device in VR equipment, primarily used for content display.

[0055] 6DoF (6 Degrees of Freedom): Degrees of freedom (DoF) relate to the motion of a rigid body in space and can be interpreted as "different fundamental ways an object moves." There are a total of 6 degrees of freedom, which can be divided into two different types: translation and rotation. A rigid body can translate in 3 degrees of freedom: forward / backward, up / down, and left / right. A rigid body can also rotate in 3 degrees of freedom: pitch, roll, and yaw. Therefore, 3 types of translational degrees of freedom + 3 types of rotational degrees of freedom equal 6 degrees of freedom. Position tracking is a combination of hardware and software that can monitor the absolute position of an object. This is very important for VR because, combined with position tracking, the system can measure and report the true 6 degrees of freedom. Since virtual reality simulates (modifies) reality, it is necessary to accurately track how objects (such as heads or hands) move in the real world so that the system can achieve accurate mapping in the VR world.

[0056] In one embodiment, the VR device uses a camera mounted on the VR headset to visually track the LED emitters on the controllers to track their position. This implementation has the following drawbacks:

[0057] (1) This makes the handheld controller bulky because the LED emitter must be installed and hidden.

[0058] (2) The presence of LED light source makes the handle design unfriendly to ergonomics, and some postures, such as the pen grip posture, are very difficult for users.

[0059] (3) Current mobile processing chips also have limitations on the number of controller tracks due to their limited computing power.

[0060] (4) In order to track the LED emitters on the controllers over the maximum range, four dedicated cameras need to be installed on the VR headset to perform the tracking task.

[0061] (5) VR headsets have blind spots in tracking VR controllers, such as behind the user's head or behind them. Once the user leaves the area covered by the four cameras on the headset, the headset cannot visually determine the position of the controllers.

[0062] To overcome the shortcomings of the above embodiments, such as Figure 1 As shown, a virtual reality controller according to an embodiment of the present disclosure includes: a first main chip 102 and a first image acquisition module 104, wherein,

[0063] The first image acquisition module 104 is electrically connected to the first main chip 102 and is used to acquire at least two controller view images at any time during the movement of the virtual reality controller.

[0064] Specifically, in one embodiment of this disclosure, the first image acquisition module includes at least two first camera modules, and further, the two first camera modules are binocular cameras.

[0065] In addition, as a preferred embodiment, the first camera module is specifically a fisheye camera.

[0066] Specifically, at least two controller view images are spatial images of the space in which the first image acquisition module on the virtual reality controller is located, acquired during the movement.

[0067] The first main chip 102 is used to: perform its own pose calculation based on at least two controller view images to generate pose information of the virtual reality controller.

[0068] The pose information includes the position information, posture information, and motion trajectory of the virtual reality controller.

[0069] Specifically, by using the first main chip to perform pose information calculation on at least two controller view images acquired by the first camera module, the virtual reality controller can detect its own position information.

[0070] In this embodiment, a first image acquisition module is set on the virtual reality controller to acquire images from the controller's perspective. The controller can acquire at least two viewpoint images during movement. A first main chip on the controller performs pose calculations based on these viewpoint images to obtain the controller's pose information. Compared to using a camera on a VR headset to visually track LED emitters on the controller, this solution does not rely on a headset to detect the controller's pose. Furthermore, the controller itself does not require LED emitters, thus reducing its size and allowing for improved shape design to enhance the user's grip experience.

[0071] In addition, the way the handle positions itself in this embodiment, compared with the method of tracking using a head-mounted display, can reduce the probability of not being able to track the handle due to the existence of tracking blind spots, thereby improving the tracking effect of the handle.

[0072] Furthermore, since the controller only needs to track its own pose information, the performance requirements for the first main chip are not too high.

[0073] In one embodiment of this disclosure, the first main chip 102 performs its own pose calculation based on at least two controller view images, specifically including: performing visual real-time localization and motion trajectory VSLAM processing based on at least two controller view images, so as to generate virtual reality controller pose information based on the processing results.

[0074] In this embodiment, by identifying at least two controller view images, real-time visual localization can be achieved based on reference objects in the images. Furthermore, by combining motion trajectory VSLAM processing, trajectory-related features are extracted from the view images, and the features are correlated, estimated, and updated to obtain the position information of the virtual reality controller, thus realizing the controller's self-localization and detection of its own movement trajectory.

[0075] Here, vslam refers to Visual SLAM, and SLAM (Simultaneous Localization And Mapping) specifically refers to simultaneous localization and mapping.

[0076] In one embodiment of this disclosure, the first main chip 102 performs real-time visual localization and motion trajectory VSLAM processing based on at least two controller viewpoint images, specifically including: performing motion estimation on adjacent frame images in at least two controller viewpoint images based on visual odometry (VO); generating motion trajectory of the virtual reality controller based on the motion estimation result; generating pose prediction information of the virtual reality controller based on the motion trajectory; optimizing the pose prediction information for trajectory drift; and obtaining pose information based on the optimization result.

[0077] VO (visual odometry) refers to the process of estimating the pose of an object by detecting changes in the image caused by the motion of a camera-equipped object. The input is an image or video sequence, and the output is the camera motion trajectory.

[0078] In this embodiment, visual odometry (OA) is used to detect image changes between different viewpoint images, and then the self-pose is estimated based on the image changes to obtain the estimated motion trajectory. Furthermore, the pose prediction information of the handle is obtained based on the estimated motion trajectory, and the drift points on the trajectory are optimized based on the obtained first estimated position to obtain a more accurate motion trajectory. The pose information of the handle is determined based on the more accurate motion trajectory. This scheme can detect the pose of the handle itself when only viewpoint images are acquired.

[0079] like Figure 1 As shown, in one embodiment of this disclosure, it further includes: an inertial measurement unit (IMU) module 106, connected to the first main chip 102, for acquiring IMU data from the virtual reality controller.

[0080] The first main chip 102 performs its own pose calculation based on at least two controller view images, specifically including: performing visual inertial odometry (VIO) pose estimation based on at least two controller view images and IMU data to generate virtual reality controller pose information.

[0081] Among them, IMU (Inertial Measurement Unit) is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

[0082] VIO (visual-inertial odometry): also known as visual-inertial system (VINS), is an algorithm that fuses camera and IMU data to achieve SLAM.

[0083] In this embodiment, combining the controller view image and IMU data to perform visual inertial odometry (VIO) pose estimation is beneficial for obtaining more accurate virtual reality controller pose information.

[0084] In one embodiment of this disclosure, the first main chip performs visual inertial odometry (VIO) pose estimation based on at least two handheld view images and IMU data, specifically including: performing visual-inertial joint initialization on at least two handheld view images and IMU data, synchronizing the visual measurement trajectory obtained based on the two view images and the inertial measurement trajectory obtained based on the IMU data to obtain synchronization data.

[0085] During the movement of the controller, visual motion tracking is performed based on synchronous data to align feature points and optimize the 3D structure of multi-frame controller view images.

[0086] Keyframes are detected based on alignment and optimization results. Local co-view keyframe optimization is performed on the keyframes to determine pose information based on the optimization results.

[0087] Among them, keyframes refer to image frames that can represent the pose of the handle.

[0088] For two keyframes that can observe the same map points, if the two keyframes share more than 15 map points, they are connected by an edge. A weight value is used to represent the number of point clouds that the two keyframes can jointly observe, thus enabling keyframe optimization.

[0089] In this embodiment, as the first method of VIO pose estimation, the viewpoint image and IMU data are jointly initialized to synchronously associate the viewpoint image and IMU data at the same moment, so as to realize the synchronous association of the visual measurement trajectory and the inertial measurement trajectory and obtain synchronous data. During the movement of the handle, the spatial structure is optimized based on the synchronous data to realize the detection of key frames. Based on the detected key frames, the pose information of the handle is obtained by optimizing the local co-view key frames, thus ensuring the reliability of the handle's own pose detection.

[0090] In one embodiment of this disclosure, the first main chip 102 performs visual inertial odometry (VIO) pose estimation based on at least two handheld view images and IMU data, specifically including:

[0091] Pre-integrate the IMU data to obtain initial attitude prediction information.

[0092] Feature point matching and triangulation are performed on at least two handle viewpoint images to obtain three-dimensional data, and a sparse depth map is created based on the three-dimensional data.

[0093] Among them, the sparse depth map is an image in which the pixel values ​​represent the distance between a point in the scene and the first image acquisition module 104. Based on the sparse depth map, the current spatial scene information of the handle can be described.

[0094] The virtual reality controller is tracked based on initial pose prediction information, sparse depth maps, and pre-stored spatial data.

[0095] Specifically, by pre-storing spatial data, the sparse depth map combined with the pre-storing spatial data can accurately describe the spatial scene information where the controller is located. The combination of initial attitude prediction information and sparse depth map can also reflect the real-time position of the controller. The combination of space and position can realize the tracking of the controller.

[0096] Keyframes are obtained based on the tracking operation.

[0097] Feature points are updated based on keyframes, and bundle adjustment optimization is performed based on the update results. The pose of the virtual reality controller and its spatial coordinates are optimized. The position of the virtual reality controller is identified based on the optimization results to obtain pose recognition information. Loop fusion optimization is performed on the pose recognition information to obtain pose information.

[0098] Among them, loop fusion refers to fusing the map points associated with the feature points matched by loop fusion detection, and removing duplicate map points.

[0099] In this embodiment, as a second method of VIO pose estimation, the IMU data and viewpoint image are processed separately, and combined with the spatial data of the space in which they are located, to obtain the initial pose prediction information and the three-dimensional data of the space in which the controller is located. By further processing the three-dimensional data, the controller trajectory is tracked, and the controller pose is further identified by the keyframes obtained, which also ensures the reliability of the controller's own pose detection.

[0100] like Figure 1 As shown, in one embodiment of this disclosure, it further includes: a first communication module 108, which is connected to the first main chip 102 and communicates with the head-mounted display, for sending pose information to the head-mounted display, and for the head-mounted display to perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0101] Specifically, the first communication module 108 can be any one of a Wi-Fi module, a Bluetooth module, a ZigBee module, or an ultra-wideband (UWB) module.

[0102] In this embodiment, by setting a first communication module on the handle, the head-mounted display can receive the pose information acquired by the handle itself, and perform coordinate fusion between the head-mounted display and the handle through the pose information, thereby enabling the display of the handle's movement trajectory in the VR image, AR image or MR image seen by the user.

[0103] In one embodiment of this disclosure, it further includes: an infrared emitting module 110 for transmitting infrared signals and configured in conjunction with a first image acquisition module; the first image acquisition module is further configured to: track infrared signals; and the first main chip is further configured to: identify infrared signal points in the view image to assist in the positioning of the virtual reality controller based on the infrared signal points.

[0104] In this embodiment, by setting an infrared emitting module on the controller in cooperation with the first image acquisition module, the first image acquisition module can acquire the infrared signal emitted by the infrared emitting module. When performing visual real-time positioning and motion trajectory VSLAM processing based on at least two controller viewpoint images, the infrared signal can be used as a reference point in the image, which helps to improve the reliability of the pose information of the generated virtual reality controller.

[0105] like Figure 2 As shown, a virtual reality controller according to another embodiment of the present disclosure includes a first image acquisition module 104 comprising at least two first camera modules, a first main chip 102, an IMU module 106, a first Wi-Fi module 108, and an infrared emitting module 110.

[0106] Specifically, the first camera module is a fisheye camera, which is connected to the first main chip via a MIPI interface. The IMU module is connected to the first main chip via an SPI interface. The infrared emitting module is controlled by the first main chip. The first main chip is connected to the WI-FI module via an SPI interface or an SDIO interface to achieve communication with the head-mounted display.

[0107] Specifically, the first camera module collects visual information and transmits it to the first main chip. The frame rate can be adjusted according to the algorithm and positioning requirements. Typical frame rates are 15fps / 30fps / 60fps / 120fps or higher.

[0108] The IMU module transmits its own acceleration and angular velocity measurements to the first main chip. The transmission sampling rate can be adjusted according to the algorithm and positioning requirements.

[0109] Typical sampling rates are 100Hz / 200Hz / 400Hz / 800Hz or higher.

[0110] The first main chip processes the sensor data acquired by the IMU module and the images acquired by the first camera module, and performs VIO calculations. A specific processing procedure includes:

[0111] The image undergoes preprocessing such as distortion correction, and feature point extraction is performed on the processed image. By comparing feature points in multiple frames of images, and combining SLAM algorithm for calculation and judgment, loop closure and bundle adjustment operations, the position and pose information of the handle are output.

[0112] Furthermore, the first main chip transmits its own position and pose information to the head-mounted display via the WI-FI module, so that the head-mounted display can perform further coordinate fusion calculations.

[0113] Additionally, DDR and Nand Flash can be configured in the controller.

[0114] like Figure 3 As shown, a head-mounted display according to an embodiment of the present disclosure includes: a second main chip 302 and a second communication module 304, wherein,

[0115] The second communication module is connected to the second main chip and is used to receive pose information sent by the virtual reality controller. The second main chip is used to perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0116] In one embodiment of this disclosure, it further includes: a second image acquisition module 306, electrically connected to the second main chip, including two second camera modules for acquiring head-view images; the second main chip is also used for: visual positioning of the head-mounted display based on the head-view images.

[0117] In this embodiment, the head-mounted display is used in conjunction with the virtual reality controllers described in the above embodiments. Since it no longer undertakes the task of tracking the pose of the controllers, the number of second camera modules set on the head-mounted display can be reduced. Furthermore, since each controller can track its own pose information, the head-mounted display can receive the pose information of at least one controller based on the number of controllers. This helps to overcome the limitation of the number of controllers that the mobile processing chip of the head-mounted display can track due to its limited computing power.

[0118] In addition, since VR headsets have blind spots in tracking VR controllers, if a device leaves the area covered by the four cameras on the headset, the headset cannot track the controller's position visually. However, by receiving the controller's position information, the movement area of ​​the controller can be expanded, which in turn helps to expand the application scenarios of the controller and the headset.

[0119] like Figure 4 As shown, a virtual reality controller 402, a head-mounted display 404, and a host 406 are included.

[0120] A first image acquisition module 104 is provided on the virtual reality controller 402, which enables the virtual reality controller 402 to detect its own pose information based on the acquired viewpoint image and send the detected result to the head-mounted display 404. In addition, the virtual reality controller 402 can also send the acquired pose information to the host 406.

[0121] like Figure 5 As shown, in one embodiment of this disclosure, the first main chip includes an engine 1022 and a processor 1024. Data transmission between the engine 1022 and the processor 1024 is performed via an AXI BUS. An external IMU module is connected to the AXI BUS via an APO BUS.

[0122] Specifically, Engine 1022 is used for data preprocessing, image feature point extraction, and feature point caching and compression.

[0123] The data preprocessing process includes synchronizing and aligning the visual and inertial measurement trajectories of the data, adjusting relevant data, and correcting shadows in the image.

[0124] The image feature point extraction process includes DoG feature detection and image integration.

[0125] The processor 1024 is specifically used to implement the main processes of SLAM, including 6DoF calculation, IMU filtering and fusion, key frame sparse 3D map, local bundle adjustment, relocalization and loop closure.

[0126] Figure 6 A flowchart of a virtual reality controller positioning method according to an embodiment of this disclosure is shown.

[0127] like Figure 6 As shown, the virtual reality controller performs a positioning method, including the following steps:

[0128] Step S602: Acquire at least two controller view images at any time during the movement of the virtual reality controller.

[0129] Step S604: Perform pose calculation based on at least two controller view images to generate pose information of the virtual reality controller.

[0130] In this embodiment, a first image acquisition module is set on the virtual reality controller to acquire images from the controller's perspective. The controller can acquire at least two viewpoint images during movement. A first main chip on the controller performs pose calculations based on these viewpoint images to obtain the controller's pose information. Compared to using a camera on a VR headset to visually track LED emitters on the controller, this solution does not rely on a headset to detect the controller's pose. Furthermore, the controller itself does not require LED emitters, thus reducing its size and allowing for improved shape design to enhance the user's grip experience.

[0131] In addition, the way the handle positions itself in this embodiment, compared with the method of tracking using a head-mounted display, can reduce the probability of not being able to track the handle due to the existence of tracking blind spots, thereby improving the tracking effect of the handle.

[0132] In one embodiment of this disclosure, calculating the pose of the virtual reality controller based on at least two controller viewpoint images to generate pose information includes:

[0133] Visual real-time localization and motion trajectory VSLAM processing are performed based on at least two controller viewpoint images to generate pose information for the virtual reality controllers.

[0134] In this embodiment, by identifying at least two controller view images, real-time visual localization can be achieved based on reference objects in the images. Furthermore, by combining motion trajectory VSLAM processing, trajectory-related features are extracted from the view images, and the features are correlated, estimated, and updated to obtain the position information of the virtual reality controller, thus realizing the controller's self-localization and detection of its own movement trajectory.

[0135] In one embodiment of this disclosure, performing visual real-time localization and motion trajectory VSLAM processing based on at least two controller viewpoint images includes: performing motion estimation on adjacent frame images in at least two controller viewpoint images based on visual odometry (VO); generating motion trajectory of the virtual reality controller based on the motion estimation result; generating pose prediction information of the virtual reality controller based on the motion trajectory; optimizing the pose prediction information for trajectory drift; and obtaining pose information based on the optimization result.

[0136] In this embodiment, visual odometry (OA) is used to detect image changes between different viewpoint images, and then the self-pose is estimated based on the image changes to obtain the estimated motion trajectory. Furthermore, the pose prediction information of the handle is obtained based on the estimated motion trajectory, and the drift points on the trajectory are optimized based on the obtained first estimated position to obtain a more accurate motion trajectory. The pose information of the handle is determined based on the more accurate motion trajectory. This scheme can detect the pose of the handle itself when only viewpoint images are acquired.

[0137] In one embodiment of this disclosure, the process of calculating the pose of a virtual reality controller based on at least two controller viewpoint images to generate pose information includes: acquiring IMU data of the virtual reality controller; and performing visual inertial odometry (VIO) pose estimation based on at least two controller viewpoint images and IMU data to generate pose information of the virtual reality controller.

[0138] like Figure 7 As shown, in one embodiment of this disclosure, performing visual inertial odometry (VIO) pose estimation based on at least two controller viewpoint images and IMU data to generate virtual reality controller pose information includes:

[0139] Step S702: Perform visual-inertial joint initialization on at least two handheld view images and IMU data, and synchronize the visual measurement trajectory obtained based on the two view images and the inertial measurement trajectory obtained based on the IMU data to obtain synchronization data.

[0140] Step S704: During the movement of the handle, visual motion tracking is performed based on the synchronization data to perform feature point alignment and 3D structure optimization of multi-frame handle view images.

[0141] Step S706: Keyframes are detected based on alignment and optimization results.

[0142] Step S708: Perform local co-view keyframe optimization on the keyframes to determine pose information based on the optimization results.

[0143] In this embodiment, as the first method of VIO pose estimation, the viewpoint image and IMU data are jointly initialized to synchronously associate the viewpoint image and IMU data at the same moment, so as to realize the synchronous association of the visual measurement trajectory and the inertial measurement trajectory and obtain synchronous data. During the movement of the handle, the spatial structure is optimized based on the synchronous data to realize the detection of key frames. Based on the detected key frames, the pose information of the handle is obtained by optimizing the local co-view key frames, thus ensuring the reliability of the handle's own pose detection.

[0144] like Figure 8As shown, in one embodiment of this disclosure, performing visual inertial odometry (VIO) pose estimation based on at least two controller viewpoint images and IMU data to generate virtual reality controller pose information includes:

[0145] Step S802: Perform feature point matching and triangulation on at least two handle view images to obtain three-dimensional data, and create a sparse depth map based on the three-dimensional data.

[0146] Step S804: Track the virtual reality controller based on the initial pose prediction information, sparse depth map and pre-stored map data.

[0147] Step S806: Obtain keyframes based on the tracking operation.

[0148] Step S808: Update the feature points based on the keyframes.

[0149] Step S810: Based on the updated results, perform beam adjustment optimization to optimize the pose of the virtual reality controller and its spatial coordinates.

[0150] Step S812: Based on the optimization results, the virtual reality controller is used for position recognition to obtain pose recognition information.

[0151] Step S814: Perform loop fusion optimization on the pose recognition information to obtain pose information.

[0152] In this embodiment, as a second method of VIO pose estimation, the initial pose prediction information and the three-dimensional data of the space where the controller is located are obtained by processing the IMU data and the view image respectively and combining the spatial data of the space. By further processing the three-dimensional data, the controller trajectory is tracked. Then, the controller pose is identified by the key frames obtained, which also ensures the reliability of the controller's own pose detection.

[0153] In one embodiment of this disclosure, the method further includes: identifying infrared signal points in a viewpoint image to assist in the positioning of the virtual reality controller based on the infrared signal points.

[0154] In this embodiment, by collecting infrared signals emitted by the infrared emitting module, when performing real-time visual positioning and motion trajectory VSLAM processing based on at least two controller viewpoint images, the infrared signals can be used as reference points in the images, thereby improving the reliability of the pose information of the generated virtual reality controllers.

[0155] like Figure 9 As shown, a pose information processing method according to an embodiment of the present disclosure includes:

[0156] Step S902: Receive pose information sent by the virtual reality controller.

[0157] Step S904: Perform coordinate fusion between the virtual reality controller and the head-mounted display based on pose information.

[0158] In one embodiment of this disclosure, the method further includes: acquiring head-view images; and performing visual positioning of the head-mounted display based on the head-view images.

[0159] In this embodiment, the head-mounted display is used in conjunction with the virtual reality controllers described in the above embodiments. Since it no longer undertakes the task of tracking the pose of the controllers, the number of second camera modules set on the head-mounted display can be reduced. Furthermore, since each controller can track its own pose information, the head-mounted display can receive the pose information of at least one controller based on the number of controllers. This helps to overcome the limitation of the number of controllers that the mobile processing chip of the head-mounted display can track due to its limited computing power.

[0160] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0161] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0162] The following reference Figure 10 To describe a virtual reality controller positioning device 1000 according to this embodiment of the present invention. Figure 10 The virtual reality controller positioning device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0163] The virtual reality controller positioning device 1000 is manifested in the form of a hardware module. The components of the virtual reality controller positioning device 1000 may include, but are not limited to: a data acquisition module 1002, used to acquire at least two controller view images at any time during the movement of the virtual reality controller; and a pose calculation module 1004, used to perform pose calculation based on at least two controller view images to generate pose information of the virtual reality controller.

[0164] The following reference Figure 11The pose information processing apparatus 1100 according to this embodiment of the present invention will be described. Figure 11 The pose information processing device 1100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0165] The pose information processing device 1100 is manifested in the form of a hardware module. The components of the pose information processing device 1100 may include, but are not limited to: a receiving module 1102 for receiving pose information sent by the virtual reality controller; and a fusion module 1104 for performing coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

[0166] The following reference Figure 12 To describe an electronic device 1200 according to this embodiment of the present invention, the electronic device 1200 may be a virtual reality controller or a head-mounted display as described in the above embodiments. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0167] like Figure 12 As shown, the electronic device 1200 is manifested in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, and a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1210).

[0168] The storage unit stores program code, which can be executed by the processing unit 1210 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1210 can perform, as follows: Figures 5 to 8 The steps shown, and other steps defined in the virtual reality controller positioning method of this disclosure.

[0169] Storage unit 1220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 12201 and / or cache memory 12202, and may further include read-only memory (ROM) 12203.

[0170] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0171] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0172] Electronic device 1200 can also communicate with one or more external devices 1300 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0173] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0174] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0175] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0176] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0178] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0179] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0180] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0181] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A virtual reality controller, characterized in that, include: First main chip; The first image acquisition module includes at least two first camera modules and is connected to the first main chip, for acquiring at least two controller view images at any time during the movement of the virtual reality controller; The first main chip is used to: perform its own pose calculation based on the at least two controller view images to generate pose information of the virtual reality controller, including: performing visual real-time localization and motion trajectory VSLAM processing based on the at least two controller view images to extract trajectory-related features from the at least two controller view images, and associating and estimating the state of the features to obtain the pose information; or the virtual reality controller is equipped with an inertial measurement unit (IMU) module that collects IMU data of the virtual reality controller, performs joint initialization of the view images and the IMU data, synchronously associates the view images and IMU data at the same moment to obtain synchronization data, optimizes the spatial structure based on the synchronization data to detect keyframes during controller movement, optimizes the keyframes for local co-view keyframes after detecting the keyframes, and obtains the pose information; or processes the IMU data and the view images respectively, combines the spatial data of the space in which they are located to obtain initial pose prediction information and three-dimensional data of the space in which the controller is located, further processes the three-dimensional data to track the controller trajectory to obtain keyframes, and identifies the pose information based on the keyframes.

2. The virtual reality controller according to claim 1, characterized in that, The first main chip performs real-time visual localization and motion trajectory VSLAM processing based on the at least two controller view images to extract trajectory-related features from the at least two controller view images, and correlates and estimates the state of the features to obtain the pose information, specifically including: Motion estimation is performed on adjacent frames in the at least two handle viewpoint images based on visual odometry (VO). The motion trajectory of the virtual reality controller is generated based on the motion estimation results; The pose prediction information of the virtual reality controller is generated based on the motion trajectory; The pose prediction information is optimized by trajectory drift, and the pose information is obtained based on the optimization result.

3. The virtual reality controller according to claim 1, characterized in that, The first main chip performs joint initialization on the viewpoint image and the IMU data, and synchronously associates the viewpoint image and IMU data at the same moment to obtain synchronization data. During the movement of the handle, the spatial structure is optimized based on the synchronization data to detect keyframes. When the keyframe is detected, local co-view keyframe optimization is performed on the keyframe to obtain the pose information, specifically including: Visual-inertial joint initialization is performed on the at least two handheld view images and the IMU data, and the visual measurement trajectory obtained based on the two view images and the inertial measurement trajectory obtained based on the IMU data are synchronized to obtain synchronization data; During the movement of the handle, visual motion tracking is performed based on the synchronization data to perform feature point alignment and 3D structure optimization of multiple frames of the handle's viewpoint images; Keyframes were detected based on alignment and optimization results; Local co-view keyframe optimization is performed on the keyframes to determine the pose information based on the optimization results.

4. The virtual reality controller according to claim 1, characterized in that, The first main chip processes the IMU data and the viewpoint image respectively, and combines the spatial data of the surrounding space to obtain initial pose prediction information and three-dimensional data of the space where the controller is located. Further processing of the three-dimensional data enables tracking of the controller's trajectory to obtain keyframes. Recognition of the pose information based on the keyframes specifically includes: The IMU data is pre-integrated to obtain initial attitude prediction information; Feature point matching and triangulation are performed on the at least two handle view images to obtain three-dimensional data, and a sparse depth map is created based on the three-dimensional data; The virtual reality controller is tracked based on the initial pose prediction information, the sparse depth map, and the pre-stored spatial data. Keyframes are obtained based on tracking operations; Update the feature points based on the keyframes; Based on the updated results, beam adjustment optimization is performed to optimize the pose of the virtual reality controller and the spatial coordinates under the pose. Based on the optimization results, the virtual reality controller is position-identified to obtain pose recognition information; The pose recognition information is optimized by loop fusion to obtain the pose information.

5. The virtual reality controller according to claim 1, characterized in that, Also includes: The first communication module is connected to the first main chip and communicates with the head-mounted display. It is used to send the pose information to the head-mounted display, and the head-mounted display performs coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

6. The virtual reality controller according to claim 1 or 2, characterized in that, Also includes: An infrared emitting module is used to transmit infrared signals and is configured to work in conjunction with the first image acquisition module. The first image acquisition module is also used to: track the infrared signal; The first main chip is also used to: identify infrared signal points in the view image, so as to assist in the positioning of the virtual reality controller based on the infrared signal points.

7. A head-mounted display, characterized in that, include: Second main chip; The second communication module, connected to the second main chip, is used to receive pose information sent by the virtual reality controller, the pose information being obtained by the virtual reality controller based on any of the following methods: Visual real-time localization and motion trajectory VSLAM processing are performed based on at least two controller viewpoint images, wherein the at least two controller viewpoint images are viewpoint images of the virtual reality controller at any time during the movement, so as to extract trajectory-related features from the at least two controller viewpoint images, and associate and estimate the state of the features to obtain the pose information; The virtual reality controller is equipped with an inertial measurement unit (IMU) module that collects IMU data from the virtual reality controller. The viewpoint image and the IMU data are jointly initialized, and the viewpoint image and IMU data at the same moment are synchronously associated to obtain synchronization data. During the movement of the controller, the spatial structure is optimized based on the synchronization data to detect keyframes. When the keyframe is detected, the local co-view keyframe is optimized to obtain the pose information. The IMU data and the viewpoint image are processed separately, and combined with the spatial data of the space, initial pose prediction information and three-dimensional data of the space where the controller is located are obtained. The three-dimensional data is further processed to track the controller trajectory to obtain key frames. The pose information is identified based on the key frames. The second main chip is used to: perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

8. The head-mounted display according to claim 7, characterized in that, Also includes: The second image acquisition module is electrically connected to the second main chip and includes two second camera modules for acquiring head-view images. The second main chip is also used for: visually positioning the head-mounted display based on the head-view image.

9. A virtual reality controller positioning method, characterized in that, Applications to virtual reality controllers include: Acquire at least two controller view images at any given moment during the movement of the virtual reality controller; The virtual reality controller's pose information is generated by calculating its own pose based on the at least two controller viewpoint images. This includes: performing visual real-time localization and motion trajectory VSLAM processing based on the at least two controller viewpoint images to extract trajectory-related features from the at least two controller viewpoint images, and associating and estimating the state of the features to obtain the pose information; or having an inertial measurement unit (IMU) module on the virtual reality controller to collect IMU data, jointly initializing the viewpoint images and the IMU data, synchronously associating the viewpoint images and IMU data at the same moment to obtain synchronization data, optimizing the spatial structure based on the synchronization data to detect keyframes during controller movement, optimizing the keyframes for local co-view keyframes upon detection to obtain the pose information; or processing the IMU data and the viewpoint images separately, combining the spatial data of the space to obtain initial pose prediction information and three-dimensional data of the space where the controller is located, further processing the three-dimensional data to track the controller trajectory to obtain keyframes, and identifying the pose information based on the keyframes.

10. The virtual reality controller positioning method according to claim 9, characterized in that, The step of performing visual real-time localization and motion trajectory VSLAM processing based on the at least two controller view images to extract trajectory-related features from the at least two controller view images, and then associating and estimating the state of the features to obtain the pose information includes: Motion estimation is performed on adjacent frames in the at least two handle viewpoint images based on visual odometry (VO). The motion trajectory of the virtual reality controller is generated based on the motion estimation results; The pose prediction information of the virtual reality controller is generated based on the motion trajectory; The pose prediction information is optimized by trajectory drift, and the pose information is obtained based on the optimization result.

11. The virtual reality controller positioning method according to claim 9, characterized in that, The process involves jointly initializing the viewpoint image and the IMU data, synchronizing and associating the viewpoint image and IMU data at the same moment to obtain synchronization data. During handle movement, the spatial structure is optimized based on the synchronization data to detect keyframes. Once a keyframe is detected, local co-view keyframe optimization is performed on the keyframe to obtain the pose information, including: Visual-inertial joint initialization is performed on the at least two handheld view images and the IMU data, and the visual measurement trajectory obtained based on the two view images and the inertial measurement trajectory obtained based on the IMU data are synchronized to obtain synchronization data; During the movement of the handle, visual motion tracking is performed based on the synchronization data to perform feature point alignment and 3D structure optimization of multiple frames of the handle's viewpoint images; Keyframes were detected based on alignment and optimization results; Local co-view keyframe optimization is performed on the keyframes to determine the pose information based on the optimization results.

12. The virtual reality controller positioning method according to claim 9, characterized in that, The process involves processing the IMU data and the viewpoint image separately, combining them with spatial data of the surrounding space to obtain initial pose prediction information and three-dimensional data of the controller's location. Further processing of the three-dimensional data enables tracking of the controller's trajectory to acquire keyframes. Recognition of the pose information based on these keyframes includes: Feature point matching and triangulation are performed on the at least two handle view images to obtain three-dimensional data, and a sparse depth map is created based on the three-dimensional data; The virtual reality controller is tracked based on the initial pose prediction information, the sparse depth map, and the pre-stored spatial data. Keyframes are obtained based on tracking operations; Update the feature points based on the keyframes; Based on the updated results, beam adjustment optimization is performed to optimize the pose of the virtual reality controller and the spatial coordinates under the pose. Based on the optimization results, the virtual reality controller is position-identified to obtain pose recognition information; The pose recognition information is optimized by loop fusion to obtain the pose information.

13. The virtual reality controller positioning method according to any one of claims 9 to 12, characterized in that, Also includes: Identify infrared signal points in the viewpoint image to assist in the positioning of the virtual reality controller based on the infrared signal points.

14. A method for processing pose information, characterized in that, Applications in head-mounted displays include: Receive pose information sent by the virtual reality controller, wherein the pose information is obtained by the virtual reality controller based on any of the following methods: Visual real-time localization and motion trajectory VSLAM processing are performed based on at least two controller viewpoint images, wherein the at least two controller viewpoint images are viewpoint images of the virtual reality controller at any time during the movement, so as to extract trajectory-related features from the at least two controller viewpoint images, and associate and estimate the state of the features to obtain the pose information; The virtual reality controller is equipped with an inertial measurement unit (IMU) module that collects IMU data from the virtual reality controller. The viewpoint image and the IMU data are jointly initialized, and the viewpoint image and IMU data at the same moment are synchronously associated to obtain synchronization data. During the movement of the controller, the spatial structure is optimized based on the synchronization data to detect keyframes. When the keyframe is detected, the local co-view keyframe is optimized to obtain the pose information. The IMU data and the viewpoint image are processed separately, and combined with the spatial data of the space, initial pose prediction information and three-dimensional data of the space where the controller is located are obtained. The three-dimensional data is further processed to track the controller trajectory to obtain key frames. The pose information is identified based on the key frames. Coordinate fusion is performed between the virtual reality controller and the head-mounted display based on the pose information.

15. The pose information processing method according to claim 14, characterized in that, Also includes: Acquire images from the head's perspective; Visual positioning of the head-mounted display is performed based on the head-view image.

16. A virtual reality controller positioning device, characterized in that, Applications to virtual reality controllers include: The acquisition module is used to acquire at least two controller view images at any time during the movement of the virtual reality controller; The pose calculation module is used to perform its own pose calculation based on the at least two controller viewpoint images to generate the pose information of the virtual reality controller. This includes: performing visual real-time localization and motion trajectory (VSLAM) processing based on the at least two controller viewpoint images to extract trajectory-related features from the at least two controller viewpoint images, and associating and estimating the state of the features to obtain the pose information; or having an inertial measurement unit (IMU) module on the virtual reality controller to collect IMU data, jointly initializing the viewpoint images and the IMU data, synchronously associating the viewpoint images and IMU data at the same moment to obtain synchronization data, optimizing the spatial structure based on the synchronization data during controller movement to detect keyframes, optimizing the keyframes for local co-view keyframes upon detection, and obtaining the pose information; or processing the IMU data and the viewpoint images separately, combining the spatial data of the space to obtain initial pose prediction information and three-dimensional data of the space where the controller is located, further processing the three-dimensional data to track the controller trajectory to obtain keyframes, and identifying the pose information based on the keyframes.

17. A pose information processing device, characterized in that, Applications in head-mounted displays include: A receiving module is configured to receive pose information sent by a virtual reality controller, wherein the pose information is obtained by the virtual reality controller using any of the following methods: Visual real-time localization and motion trajectory VSLAM processing are performed based on at least two controller viewpoint images, wherein the at least two controller viewpoint images are viewpoint images of the virtual reality controller at any time during the movement, so as to extract trajectory-related features from the at least two controller viewpoint images, and associate and estimate the state of the features to obtain the pose information; The virtual reality controller is equipped with an inertial measurement unit (IMU) module that collects IMU data from the virtual reality controller. The viewpoint image and the IMU data are jointly initialized, and the viewpoint image and IMU data at the same moment are synchronously associated to obtain synchronization data. During the movement of the controller, the spatial structure is optimized based on the synchronization data to detect keyframes. When the keyframe is detected, the local co-view keyframe is optimized to obtain the pose information. The IMU data and the viewpoint image are processed separately, and combined with the spatial data of the space, initial pose prediction information and three-dimensional data of the space where the controller is located are obtained. The three-dimensional data is further processed to track the controller trajectory to obtain key frames. The pose information is identified based on the key frames. The fusion module is used to perform coordinate fusion between the virtual reality controller and the head-mounted display based on the pose information.

18. A virtual reality controller, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the virtual reality controller positioning method according to any one of claims 9 to 13 by executing the executable instructions.

19. A head-mounted display, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the pose information processing method of claim 14 or 15 by executing the executable instructions.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the virtual reality controller positioning method according to any one of claims 9 to 13 and / or the pose information processing method according to claim 14 or 15.

Images