Handheld controller, positioning and tracking system and method

Abstract

The invention discloses a handheld controller and a positioning tracking system and method.The handheld controller comprises a shell cover plate and at least one control piece arranged on the shell cover plate, and the at least one control piece is used for controlling equipment to be controlled; one end of the handle main body is connected to the shell cover plate, and the side wall of the handle main body is provided with a holding surface; the at least one light reflecting part is arranged on the handle main body, and the reflectivity of the light reflecting part is different from that of the handle main body; or at least one light reflecting part is arranged on the shell cover plate, and the reflectivity of the light reflecting part is different from that of the shell cover plate. The light emitting part does not consume energy, so that the cruising ability of the handheld controller can be improved compared with positioning depending on active light emitting light in the prior art.

Description

Technical Field

[0001] The present application relates to the field of handheld controller positioning technology, and in particular to a handheld controller, a positioning tracking system and a method. Background Art

[0002] Virtual Reality (VR) games allow users to wear virtual reality devices to enter an interactive virtual live game scene, which can not only simulate the current scene, but also allow virtual interaction, giving users an immersive gaming experience. Existing virtual reality devices mainly include head-mounted displays and handheld controllers. The interaction of handheld controllers is generally carried out through the handheld controller itself and the cooperation of cameras. In the prior art, a light that can emit light is installed on the handheld controller, and the handheld controller lights up the light on it, and then the camera captures the light to locate the handheld controller. This positioning method will cause the handheld controller to quickly consume the power, resulting in poor battery life of the handheld controller.

[0003] That is, the battery life of the handheld controller in the prior art is relatively poor. Summary of the invention

[0004] The embodiments of the present application provide a handheld controller, a positioning tracking system and a method, which can improve the battery life of the handheld controller.

[0005] In a first aspect, the handheld controller provided by the present application comprises:

[0006] A housing cover and at least one control element arranged on the housing cover, wherein the at least one control element is used to control a device to be controlled;

[0007] A handle body, one end of which is connected to the housing cover, and a side wall of the handle body has a gripping surface;

[0008] At least one reflective portion, wherein the at least one reflective portion is disposed on the handle body, wherein the reflectivity of the reflective portion is different from the reflectivity of the handle body; or, the at least one reflective portion is disposed on the outer shell cover, wherein the reflectivity of the reflective portion is different from the reflectivity of the outer shell cover.

[0009] In a second aspect, the positioning and tracking system provided in the present application includes: a device to be controlled and a handheld controller as described in any one of the first aspects, the device to be controlled includes an image acquisition device, the image acquisition device is used to take a picture of the handheld controller, and the reflective part on the handheld controller is used to reflect light so as to be photographed by the image acquisition device of the device to be controlled and perform positioning tracking.

[0010] In a third aspect, the positioning and tracking method provided by the present application includes:

[0011] In the case of acquiring a target image sent by the device to be controlled, detecting whether there is at least one reflective portion on the target image;

[0012] In the case where at least one of the reflective portions exists on the target image, acquiring position information of at least one of the reflective portions;

[0013] The handheld controller is located and tracked based on the position information of at least one reflective portion.

[0014] In the present application, compared with the related art, the handheld controller includes: a shell cover and at least one control member arranged on the shell cover, the at least one control member is used to control the device to be controlled; a handle body, one end of the handle body is connected to the shell cover, and the side wall of the handle body has a gripping surface; at least one reflective portion, at least one reflective portion is arranged on the handle body, wherein the reflectivity of the reflective portion is different from the reflectivity of the handle body; or at least one reflective portion is arranged on the shell cover, wherein the reflectivity of the reflective portion is different from the reflectivity of the shell cover. In the present application, at least one reflective portion is arranged on the shell cover or the handle body of the handheld controller, which is used to reflect light to the image acquisition device on the device to be controlled, so that the image acquisition device on the device to be controlled can collect the target image containing the reflective portion, the reflectivity of the reflective portion is different from the reflectivity of the handle body or the shell cover where the reflective portion is located, the reflective portion can be accurately identified and used to locate and track the handheld controller, and since the light-emitting portion does not consume energy, compared with the prior art that relies on active light-emitting light positioning, the endurance of the handheld controller can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a structural diagram of an embodiment of a positioning and tracking system provided in an embodiment of the present application;

[0017] Figure 2 It is a structural schematic diagram of an embodiment of a device to be controlled provided in an embodiment of the present application;

[0018] Figure 3 is a structural diagram of a handheld controller provided in an embodiment of the present application from a perspective;

[0019] Figure 4 is a structural schematic diagram of another perspective of an embodiment of a handheld controller provided in an embodiment of the present application;

[0020] Figure 5 is a schematic structural diagram of a housing cover plate in one embodiment of a handheld controller provided in an embodiment of the present application;

[0021] Figure 6 is a structural schematic diagram of a housing cover plate in another embodiment of a handheld controller provided in an embodiment of the present application;

[0022] Figure 7 It is a flowchart of an embodiment of the positioning tracking method provided by the embodiment of the present application;

[0023] Figure 8 is a schematic diagram of the camera exposure time and the infrared light lighting time in one embodiment of the positioning tracking method provided by the embodiment of the present application;

[0024] Fig. 9 It is a schematic diagram of a flow chart of positioning and tracking a handheld controller based on position information of at least one reflective portion in an embodiment of a positioning and tracking method provided by an embodiment of the present application;

[0025] Fig.10 This is another flow chart of the positioning and tracking method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0027] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] In order to improve the battery life of a handheld controller, an embodiment of the present application provides a handheld controller, comprising: an outer shell cover and at least one control component arranged on the outer shell cover, the at least one control component is used to control a device to be controlled; a handle body, one end of the handle body is connected to the outer shell cover, and the side wall of the handle body has a gripping surface; at least one reflecting portion, at least one reflecting portion is arranged on the handle body, wherein the reflectivity of the reflecting portion is different from the reflectivity of the handle body; or, at least one reflecting portion is arranged on the outer shell cover, wherein the reflectivity of the reflecting portion is different from the reflectivity of the outer shell cover.

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0032] See also Figure 1 , Figure 1 Schematic diagram of the structure of a positioning tracking system according to an embodiment of the present application. Figure 1 As shown, in the embodiment of the present application, the positioning tracking system includes a handheld controller 20 and a device to be controlled 30. Further, the positioning tracking system also includes a positioning tracking device 10, and the positioning tracking device 10 is connected to the handheld controller 20 and the device to be controlled 30. Of course, the positioning tracking device 10 can be integrated on the device to be controlled 30, and can also be integrated on the handheld controller 20.

[0033] Among them, the device to be controlled 30 can be a virtual reality (VR) device, or an augmented reality (AR) device or other devices that need to be controlled. Augmented reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. After simulating computer-generated text, images, three-dimensional models, music, videos and other virtual information, they are applied to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world. Virtual reality technology, also known as virtual reality or spiritual technology, is a brand-new practical technology developed in the 20th century. Virtual reality technology encompasses computers, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments of various high technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. With the help of computers and other equipment, a realistic three-dimensional visual, tactile, olfactory and other sensory experiences are generated in the virtual world, so that people in the virtual world have an immersive feeling.

[0034] Among them, the device to be controlled 30 can be a mobile VR head-mounted device with a head-mounted display. The device to be controlled 30 includes an image acquisition device 32, which is used to take pictures of the handheld controller 20. Among them, the image acquisition device 32 can be a monocular camera or a binocular camera. A monocular camera refers to a camera system with only one camera. It uses a single lens to capture a scene and converts it into a two-dimensional image. Monocular cameras have the advantages of low cost and easy installation, and are widely used in consumer electronic products such as mobile phones, driving recorders, and surveillance cameras. In the field of computer vision, a monocular camera can realize functions such as object detection, posture estimation, and target tracking by processing and analyzing images. A binocular camera refers to a camera system composed of two parallel cameras. It simultaneously captures and analyzes images from multiple perspectives to obtain richer information. The advantage of a binocular camera is reflected in stereoscopic vision. By calculating the parallax between the two cameras, the depth information of the objects in the scene can be obtained, thereby realizing three-dimensional reconstruction and depth perception. Binocular cameras are widely used in robot navigation, augmented reality, virtual reality and other fields. It can provide more accurate object positioning, obstacle detection, and distance measurement.

[0035] See also Figure 2 , Figure 2 Schematic diagram of the structure of an embodiment of the device to be controlled provided by the embodiment of the present application. Figure 2As shown, in the embodiment of the present application, the device to be controlled 30 includes an image acquisition device 32, and the image acquisition device 32 is used to take a picture of the handheld controller 20. The image acquisition device 32 can be a monocular camera or a binocular camera, which can be set according to the specific situation.

[0036] In the embodiment of the present application, the device to be controlled 30 includes a device body 31, an image acquisition device 32, and a fill light device 33. The fill light device 33 and the image acquisition device 32 are arranged in front of the device body 31. The fill light device 33 is used to emit light for fill light when the image acquisition device 32 is exposed. The fill light device 33 can be a light source that emits infrared light. Of course, in other embodiments, the fill light device 33 can be a light source that emits other colors of light, and the fill light device 33 can also be arranged below the device body 31.

[0037] See also Figures 3 to 5 , Figure 3 is a structural diagram of a handheld controller provided in an embodiment of the present application from a perspective. Figure 4 is a structural diagram of another perspective of an embodiment of a handheld controller provided in an embodiment of the present application, Figure 5 It is a schematic diagram of the structure of a shell cover in one embodiment of a handheld controller provided in an embodiment of the present application.

[0038] like Figures 3 to 5 As shown, in the embodiment of the present application, the handheld controller 20 includes a housing cover 22 , a handle body 21 and at least one reflective portion 223 .

[0039] Among them, at least one control element 224 is provided on one side surface of the outer shell cover 22, and the at least one control element 224 is used to control the device to be controlled 30, and the device to be controlled 30 includes an image acquisition device 32, and the image acquisition device 32 is used to take a photo of the handheld controller 20. At least one control element 224 belongs to the handheld controller 20. One end of the handle body 21 is connected to the outer shell cover 22, and the side wall of the handle body 21 is provided with a gripping surface for a person to hold. At least one reflective portion 223 is provided on the handle body 21 or the outer shell cover 22, and the reflective portion 223 is used to reflect light, and the reflectivity of the reflective portion 223 is different from the reflectivity of the handle body 21 or the outer shell cover 22 where the reflective portion 223 is located. When the gripping surface of the handle body 21 is held by a person, the image acquisition device 32 of the device to be controlled 30 captures a target image containing at least one reflective portion 223, and locates and tracks the handheld controller 20 based on the target image.

[0040] The handle body 21 is hollow, and the housing cover 22 forms a hollow cavity with the handle body 21. A circuit board is provided in the handle body 21, and at least one control member 224 is connected to the circuit board in the handle body 21. The at least one control member 224 may include a button, a rocker, a touch pad, etc., which can be set according to specific circumstances. The button may include a forward button, a backward button, etc. The device 30 to be controlled is controlled by pressing a button, shaking a rocker, touching a touch pad, etc.

[0041] Among them, at least one reflective portion 223 can be entirely arranged on the handle body 21, or entirely arranged on the shell cover 22, or partially arranged on the handle body 21 and the other part arranged on the shell cover 22, which can be set according to the situation. The reflective portion 223 is a passive component, and the reflective portion 223 does not consume the energy of the handheld controller. If at least one reflective portion 223 is entirely arranged on the handle body 21, the reflectivity of at least one reflective portion 223 is different from the reflectivity of the surface of the handle body 21. If at least one reflective portion 223 is entirely arranged on the shell cover 22, the reflectivity of at least one reflective portion 223 is different from the reflectivity of the surface of the shell cover 22. If at least one reflective portion 223 is partially arranged on the handle body 21 and the other part is arranged on the shell cover 22, the reflectivity of the reflective portion 223 on the handle body 21 is different from the reflectivity of the handle body 21, and the reflectivity of the reflective portion 223 on the shell cover 22 is different from the reflectivity of the shell cover 22.

[0042] In the present application, at least one reflective portion is provided on the outer shell cover or handle body of the handheld controller, which is used to reflect light to the image acquisition device on the device to be controlled, so that the image acquisition device on the device to be controlled can acquire the target image containing the reflective portion. The reflectivity of the reflective portion is different from the reflectivity of the handle body or the outer shell cover where the reflective portion is located. The reflective portion can be accurately identified and used to locate and track the handheld controller. Since the light-emitting portion does not consume energy, compared with the prior art that relies on active light-emitting light for positioning, the battery life of the handheld controller can be improved.

[0043] The side wall of the handle body 21 is provided with a gripping surface for a person to hold. Specifically, the gripping surface is a bionic gripping surface, which can perfectly fit a person's hand, thereby improving the accuracy of a person's gripping.

[0044] In a specific embodiment, the image acquisition device 32 is a binocular camera, and the number of the reflective parts 223 is 1 or 2. Two target images taken by the binocular camera are acquired, and three-dimensional reconstruction is performed based on the position information of at least one reflective part 223 on the two target images to obtain the position information of the handheld controller 20. The position information of the handheld controller 20 includes position and posture.

[0045] In another specific embodiment, the image acquisition device 32 is a monocular camera. Since the monocular camera can only capture two-dimensional images, it is generally necessary to use a pose estimation algorithm based on two-dimensional images to solve the camera pose. For example, the PnP algorithm is used to solve the camera pose. The PnP (Perspective-n-Point) algorithm is a method for solving the correspondence between 3D and 2D points. It describes how to estimate the pose of the camera when n 3D space points and their positions are known. If the 3D position of a feature point in two images is known, at least 3 point pairs are required to calculate the movement of the camera. PnP can be a direct linear transformation algorithm, a P3P algorithm, an EPnP (Efficient Pnp) algorithm, and a BA (Bundle Adjustment) algorithm, which can be set according to the specific situation.

[0046] Since the PNP algorithm based on two-dimensional images requires three point pairs. Therefore, at least three points are required on the target image, so it is necessary to ensure that there are at least three reflective parts 223 on the target image. Specifically, the handheld controller 20 includes at least three reflective parts 223, and at least three reflective parts 223 are arranged at intervals on the handle body 21 or the shell cover 22. When the grip surface of the handle body 21 is held by a person, the image acquisition device 32 of the device to be controlled 30 captures at least three reflective parts 223 in the target image of the handheld controller 20. At least three reflective parts 223 are arranged at intervals on the handle body 21 or the shell cover 22, and can be set according to the specific situation.

[0047] In the embodiment of the present application, the reflective portion 223 is circular. In other embodiments, the reflective portion 223 may be other shapes such as rectangular or elliptical.

[0048] In the embodiment of the present application, at least three reflective portions 223 are disposed on the outer surface of one side of the housing cover 22. At least three reflective portions 223 are disposed on the outer surface of one side of the housing cover 22, so that the image acquisition device 32 of the device to be controlled 30 can quickly capture the three reflective portions 223 for posture calculation. Of course, in other embodiments, at least three reflective portions 223 can also be disposed on the handle body 21, and can be set according to specific circumstances. When light is irradiated on the reflective portion 223, the reflective portion 223 reflects the light, and the image acquisition device 32 of the device to be controlled 30 can capture the reflective portion 223.

[0049] In the embodiment of the present application, the reflectivity of the reflective portion 223 is different from the reflectivity of the outer shell cover 22. The percentage of radiation energy reflected by an object to the total radiation energy is called reflectivity. The reflectivity of different objects is also different, which mainly depends on the properties of the object itself, as well as the wavelength and angle of incidence of the incident electromagnetic wave. The reflectivity range is always less than or equal to 1, and the properties of the object can be judged by the reflectivity. The reflectivity of the reflective portion 223 is different from the reflectivity of the outer shell cover 22. The brightness of the reflective portion 223 on the outer shell cover 22 will be significantly lower than the outer shell cover 22 next to it, which is convenient for identification.

[0050] In a specific embodiment, the reflectivity of the reflective portion 223 is greater than the reflectivity of the outer shell cover 22. Since the reflectivity of the reflective portion 223 is greater than the reflectivity of the outer shell cover 22, the light reflected by the reflective portion 223 is stronger than the light reflected by the outer shell cover 22, and the brightness of the reflective portion 223 on the outer shell cover 22 will be significantly higher than that of the outer shell cover 22 next to it, so that the reflective portion 223 can be accurately identified.

[0051] In another specific embodiment, the reflectivity of the reflective portion 223 is lower than the reflectivity of the outer shell cover 22. Since the reflectivity of the reflective portion 223 is lower than the reflectivity of the outer shell cover 22, the light reflected by the reflective portion 223 is weaker than the light reflected by the outer shell cover 22, and the brightness of the reflective portion 223 on the outer shell cover 22 will be significantly lower than that of the outer shell cover 22 next to it, so that the reflective portion 223 can be accurately identified.

[0052] In the embodiment of the present application, the reflection mode of the reflective portion 223 is diffuse reflection. Diffuse reflection is the phenomenon that light projected on a rough surface is reflected in all directions. When a beam of parallel incident light hits a rough surface, the surface will reflect the light in all directions. Therefore, although the incident rays are parallel to each other, due to the inconsistency of the normal directions of each point, the reflected light is irregularly reflected in different directions. This kind of reflection is called diffuse reflection. This reflected light is called diffuse light. The reflective mode of the reflective portion 223 is diffuse reflection, which can reflect light to more viewing angles, so that it is easy to be captured by the image acquisition device 32 of the device to be controlled 30.

[0053] In the embodiment of the present application, the reflective portion 223 is detachably connected to the handle body 21 or the outer shell cover 22. The reflective portion 223 can be connected to the handle body 21 or the outer shell cover 22 by gluing or threading, so as to facilitate replacement or repair when the reflective portion 223 is damaged or contaminated. Of course, in other embodiments, the reflective portion 223 can also be integrally formed with the handle body 21 or the outer shell cover 22, which is not limited here.

[0054] In the embodiment of the present application, the handheld controller 20 includes an inertial sensor (InerTIal measurement unit, referred to as IMU), which is arranged in the handle body 21, and the inertial sensor is used to collect the three-axis angular velocity information and displacement information of the handheld controller 20. The inertial sensor is a device for measuring the three-axis attitude angle and acceleration of an object. Gyroscopes and accelerometers are the core devices of the inertial navigation system. With the help of built-in acceleration sensors and gyroscopes, IMU can measure linear acceleration and rotational angular rate from three directions, and can obtain information such as the attitude, velocity and displacement of the carrier through solution. In sports, inertial sensors are composed of accelerometers for measuring force and acceleration, gyroscopes for indicating rotation, and magnetometers for measuring human posture. These sensors collect data across three axes and capture the subtle movements of athletes. The accelerometer detects the acceleration signal of the object in the independent three axes of the carrier coordinate system, which is very important for the whole-body motion analysis of the human body. The gyroscope can be used to determine the direction of an object in three-dimensional space, mainly measuring the angular velocity around three axes, namely pitch, roll and yaw. Combining multiple sensors connected together can simulate the movement of the entire body. Magnetometers help with orientation and can be used to calibrate and direct data to the correct location.

[0055] In the embodiment of the present application, a second trigger 211 and a first trigger 212 are respectively provided on the side walls of the handle body 21. When the grip surface of the handle body 21 is held by a person, the person's middle finger is placed on the second trigger 211, the person's index finger is placed on the first trigger 212, and the person's thumb and the second trigger 211 are located on opposite sides of the handle body 21. The second trigger 211 is used to be triggered by the person's index finger to issue a control command, and the first trigger 212 is used to be triggered by the person's middle finger to issue a control command.

[0056] In the embodiment of the present application, the outer shell cover 22 includes a first cover portion 221 close to the first trigger 212 and a second cover portion 222 far from the first trigger 212, and the number of the reflective portions 223 on the first cover portion 221 is greater than the number of the reflective portions 223 on the second cover portion 222. Since the second cover portion 222 is close to and far from the first trigger 212, when a person holds the handle body 21, the thumb of the person and the second trigger 211 are located on opposite sides of the handle body 21, and the person mainly relies on the thumb to manipulate the control member 224 on the outer shell cover 22. When the thumb manipulates the control member 224 on the outer shell cover 22, it will inevitably pass through the second cover portion 222. By providing fewer reflective portions 223 on the second cover portion 222, the risk of the reflective portion 223 being blocked can be reduced, thereby facilitating the image acquisition device 32 of the device 30 to be controlled to capture the reflective portion 223.

[0057] In the embodiment of the present application, the number of control elements 224 on the first cover plate portion 221 is less than the number of control elements 224 on the second cover plate portion 222. When the thumb manipulates the control element 224 on the housing cover plate 22, it is easier to touch the control element 224 on the second cover plate portion 222. More control elements 224 are provided on the second cover plate portion 222, which can facilitate control by the thumb and improve the control experience.

[0058] In the embodiment of the present application, at least three reflective portions 223 are arranged at equal intervals on the outer surface of one side of the housing cover 22. When the reflective portions 223 are close to each other, if the shooting angle of the image acquisition device 32 of the device to be controlled 30 is small with the surface of the housing cover 22, the reflective portions 223 may overlap on the target image, resulting in inability to be recognized. Therefore, the reflective portions 223 are arranged at equal intervals so that the area of ​​the housing cover 22 and the number of the reflective portions 223 are minimized.

[0059] In the embodiment of the present application, both the handheld controller 20 and the device to be controlled 30 are provided with a Bluetooth module. The handheld controller 20 and the device to be controlled 30 communicate with each other via the Bluetooth module. The Bluetooth module of the handheld controller 20 is connected to the control element 224, the motor and the inertial sensor of the handheld controller 20 to obtain data from the control element 224, the motor and the inertial sensor.

[0060] like Figure 6 As shown, in another embodiment, at least three reflective portions 223 are arranged on the inner surface of the outer shell cover 22, the outer shell cover 22 transmits light, and the reflective portions 223 reflect the light that passes through the outer shell cover 22. The outer shell cover 22 may be made of a transparent material or may be able to transmit the light emitted by the fill light device 33. When the light passes through the outer shell cover 22 and is irradiated on the reflective portion 223, the reflective portion 223 reflects the light, and the image acquisition device 32 of the device to be controlled 30 can capture the reflective portion 223. Since at least three reflective portions 223 are arranged on the inner surface of the outer shell cover 22, the reflective portion 223 can be prevented from falling off or being contaminated.

[0061] See also Figure 7 , Figure 7 1 is a flow chart of an embodiment of the positioning tracking method provided in the embodiment of the present application. The positioning tracking method is applied to any of the above-mentioned handheld controllers, and the positioning tracking method includes:

[0062] 201. When a target image sent by a device to be controlled is acquired, detect whether there is at least one reflective portion on the target image.

[0063] In the embodiment of the present application, when it is detected that the image acquisition device sends a strobe signal (Strobe), the fill light device of the device to be controlled is driven to start to emit a preset light; when it is detected that the strobe signal of the image acquisition device ends, the fill light device of the device to be controlled is driven to turn off. The strobe signal is a pulse signal. For a pulse strobe signal, when the image acquisition device sends a strobe signal, the image acquisition device starts to expose, and when the strobe signal of the image acquisition device ends, the image acquisition device has finished exposing.

[0064] The trigger mode of the camera is divided into internal trigger mode and external trigger mode. Internal trigger mode: The camera acquires images through the signal given inside the device. External trigger mode: The camera acquires images through the signal given outside. Soft trigger and hardware trigger are the two main forms of external signals. The external signal can be either a software signal or a hardware signal. The trigger output signal of the camera is a switch signal, which can be used to control external devices such as alarm lights and light sources. The trigger output signal can be realized through the Strobe signal. When the camera is exposed, a Strobe signal with a high effective level is immediately generated. In the following text, we mainly use this signal to hard trigger other sensors. There is a concept of Strobe pre-output here. The Strobe signal takes effect earlier than the exposure. Its working principle is to delay the exposure and output the Strobe first. This function can be applied to external devices with slower response.

[0065] like Figure 8 The camera exposure signal is sent in the form of pulses, and the fill light device is an infrared lamp. The infrared lamp is turned on before the camera is exposed and turned off after the camera is exposed. The infrared lamp is turned on for a longer time than the camera exposure time to ensure the relative consistency of the brightness of each frame.

[0066] In a specific embodiment, the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover, the light reflected by the reflective portion is stronger than the light reflected by the outer shell cover. The pixel points on the target image whose brightness value is higher than the first preset brightness value are determined as the pixel points of the reflective portion, the pixel points of the reflective portion are clustered to obtain a plurality of pixel clusters, and the number of pixel clusters is determined as the number of the reflective portion.

[0067] In another specific embodiment, the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover, the light reflected by the reflective portion is weaker than the light reflected by the outer shell cover, and the brightness of the reflective portion on the outer shell cover will be significantly lower than that of the adjacent outer shell cover. The pixel points on the target image whose brightness value is lower than the second preset brightness value are determined as the pixel points of the reflective portion, and the pixel points of the reflective portion are clustered to obtain a plurality of pixel clusters, and the number of pixel clusters is determined as the number of the reflective portion.

[0068] In another specific embodiment, the target image is input into a preset target detection model to obtain the number and position information of the reflective parts. The preset target detection model may be a Fast R-CNN, YOLOV5 or other model, which is not limited here.

[0069] 202. When there is at least one reflective portion on the target image, obtain position information of the at least one reflective portion.

[0070] In the embodiment of the present application, when there is at least one reflective portion on the target image, the position information of the at least one reflective portion is obtained. The position information is the two-dimensional coordinates of the reflective portion.

[0071] In a specific embodiment, the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover, the light reflected by the reflective portion is stronger than the light reflected by the outer shell cover. Pixel points on the target image whose brightness values ​​are higher than the first preset brightness value are determined as pixel points of the reflective portion, and pixel points on the target image whose brightness values ​​are lower than the second preset brightness value are determined as pixel points of the reflective portion. The pixel points of the reflective portion are clustered to obtain multiple pixel clusters, and the central coordinates of the pixel cluster are determined as the position information of the reflective portion, that is, the central coordinates of the pixel cluster are determined as the two-dimensional coordinates of the reflective portion.

[0072] In another specific embodiment, the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover, the light reflected by the reflective portion is weaker than the light reflected by the outer shell cover, and the brightness of the reflective portion on the outer shell cover will be significantly lower than that of the outer shell cover next to it. The pixel points on the target image whose brightness values ​​are lower than the second preset brightness value are determined as the pixel points of the reflective portion, the pixel points of the reflective portion are clustered to obtain a plurality of pixel clusters, and the central coordinates of the pixel cluster are determined as the position information of the reflective portion, that is, the central coordinates of the pixel cluster are determined as the two-dimensional coordinates of the reflective portion.

[0073] In another specific embodiment, the target image is input into a preset target detection model to obtain the number and position information of the reflective parts. The preset target detection model may be a Fast R-CNN, YOLOV5 or other model, which is not limited here.

[0074] 203. Position and track the handheld controller based on the position information of at least one reflective portion.

[0075] In a specific embodiment, the position information of the reflective portion is the two-dimensional plane coordinates of the reflective portion on the target image. The image acquisition device is a binocular camera, and the number of reflective portions is 1 or 2. Three-dimensional reconstruction is performed based on the position information of at least one reflective portion on the two target images taken by the binocular camera to obtain the position information of the handheld controller. The position information of the handheld controller includes position and posture. The handheld controller is positioned and tracked according to the position information of the handheld controller.

[0076] In another specific embodiment, the image acquisition device is a monocular camera. Since the monocular camera can only acquire two-dimensional images, it is generally necessary to use a pose estimation algorithm based on two-dimensional images to solve the camera pose. For example, the PnP algorithm is used to solve the camera pose. Since the PNP algorithm based on two-dimensional images requires three point pairs. Therefore, at least three points are required on the target image, so it is necessary to ensure that there are at least three reflective parts on the target image. Specifically, the handheld controller includes at least three reflective parts, and the at least three reflective parts are spaced apart on the handle body or the outer shell cover. When the grip surface of the handle body is held by a person, there are at least three reflective parts in the target image of the handheld controller taken by the image acquisition device of the device to be controlled.

[0077] See also Fig. 9 , Fig. 9 The present invention is a flowchart of positioning and tracking a handheld controller based on the position information of at least one reflective portion in an embodiment of a positioning and tracking method provided in an embodiment of the present application. In an embodiment of the present application, positioning and tracking a handheld controller based on the position information of at least one reflective portion includes:

[0078] 301. When there are at least three reflective parts on the target image, pose estimation is performed based on at least three two-dimensional plane coordinates of the at least three reflective parts and at least three pre-stored three-dimensional space coordinates to obtain pose information of the image acquisition device.

[0079] The position information of the reflective portion is a two-dimensional plane coordinate.

[0080] In the embodiment of the present application, at least three two-dimensional plane coordinates are obtained by detecting the target image, and at least three three-dimensional space coordinates are three-dimensional coordinates in a preset space coordinate system. For example, the preset space coordinate system is a space coordinate system established with the surface of the shell cover as the plane where the X-axis and the Y-axis are located, and the axis perpendicular to the surface of the shell cover as the Z-axis. At least three three-dimensional space coordinates of at least three reflective parts can be obtained by pre-measurement.

[0081] In an embodiment of the present application, after obtaining at least three two-dimensional plane coordinates and corresponding at least three three-dimensional space coordinates, the PnP algorithm can be used to perform pose estimation based on the at least three two-dimensional plane coordinates and at least three three-dimensional space coordinates to obtain the pose information of the image acquisition device.

[0082] 302. Determine first position information of the handheld controller based on the position information of the image acquisition device.

[0083] The first position information of the handheld controller is the position information of the handheld controller in the world coordinate system. The position information of the image acquisition device is the position information of the image acquisition device in the preset space coordinate system. The first position information of the handheld controller is determined based on the position information of the image acquisition device through coordinate system conversion.

[0084] The world coordinate system is the reference system for the position of the target object. The dot position can be freely set according to the convenience of operation. It can be located on the base of the robot or the front actuator of the robot. Since the camera can be placed anywhere in the environment, a reference coordinate system is selected in the environment to describe the position of the camera, and it is used to describe the position of any object in the environment. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix and the translation vector. The world coordinate system is the absolute coordinate system of the system. Before the user coordinate system is established, the coordinates of all points on the screen are determined by the origin of this coordinate system.

[0085] 303. Determine current posture information of the handheld controller based on the first posture information.

[0086] In a specific embodiment, the first pose information is determined as the current pose information of the handheld controller.

[0087] In another specific embodiment, the second posture information of the handheld controller determined based on the inertial sensor is obtained; the first posture information and the second posture information are fused to obtain the current posture information. Specifically, the three-axis angular velocity information and displacement information collected by the inertial sensor are obtained; the second posture information of the handheld controller is determined based on the three-axis angular velocity information and displacement information collected by the inertial sensor; the first posture information and the second posture information are fused to obtain the current posture information. Specifically, the first posture information and the second posture information are weighted and summed to obtain the current posture information. IMU can output the acceleration and angular velocity of the carrier, and obtain the posture trajectory of the carrier movement through the inertial navigation solution principle. When solving, it is also necessary to eliminate the errors in the IMU sampling data and the errors generated in the process. IMU outputs three-axis acceleration and three-axis angular velocity data based on its own coordinate system, which need to be converted into parameter information in the robot world coordinate system. The coordinate system conversion parameters here are represented by the posture matrix, regarded as posture information, and are derived using the initial posture in the posture update link. When the carrier's posture changes, the initial posture can be used to complete the conversion of IMU sampling data between navigation coordinate systems, and the calculation and feedback of the carrier's posture, speed and other information can be realized. The fusion of the second posture information calculated by the IMU and the first posture information determined based on the target image can improve the accuracy of posture information calculation.

[0088] 304. Position and track the handheld controller based on the current posture information.

[0089] In the embodiment of the present application, after obtaining the current posture information of the handheld controller, the posture information of the handheld controller is updated in real time to achieve positioning tracking.

[0090] See also Fig.10 , Fig.10 1 is another embodiment flow chart of the positioning tracking method provided in the embodiment of the present application. The positioning tracking method includes:

[0091] 401. When it is detected that an image acquisition device sends out a stroboscopic signal, a fill light device of a device to be controlled is driven to start up to send out a preset light.

[0092] In the embodiment of the present application, when it is detected that the image acquisition device sends a strobe signal (Strobe), the fill light device of the device to be controlled is driven to start to emit a preset light; when it is detected that the strobe signal of the image acquisition device ends, the fill light device of the device to be controlled is driven to turn off. The strobe signal is a pulse signal. For a pulse strobe signal, when the image acquisition device sends a strobe signal, the image acquisition device starts to expose, and when the strobe signal of the image acquisition device ends, the image acquisition device has finished exposing.

[0093] 402. Detect whether a target image sent by the device to be controlled is obtained.

[0094] Specifically, it is detected whether the target image sent by the device to be controlled is obtained according to the preset frequency. The preset frequency can be 1000 Hz or other values, depending on the specific setting.

[0095] In an embodiment of the present application, when the target image sent by the device to be controlled is detected and obtained, 403 is executed: the number of reflective parts on the target image is detected. When the target image sent by the device to be controlled is not detected, 404 is executed: the second posture information of the handheld controller determined based on the inertial sensor is determined as the current position information. In some cases, the IMU data frame rate can reach 1000hz, while the camera exposure image is usually 30-60hz. If only the image is used, the posture data is less, and it is more likely to have abnormalities such as jitter in scenes such as fast movement and waving. The IMU can obtain the real-time motion state of the handle body, and by combining the image with the IMU posture, the output posture is smoother.

[0096] 403. Detect whether there is at least one reflective portion on the target image.

[0097] In the embodiment of the present application, when there is at least one reflective portion on the target image, execute 405: determine whether there are at least three reflective portions on the target image. When there is no at least one reflective portion on the target image, execute 404: determine the second position information of the handheld controller determined based on the inertial sensor as the current position information.

[0098] In a specific embodiment, the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is greater than the reflectivity of the outer shell cover, the light reflected by the reflective portion is stronger than the light reflected by the outer shell cover. Pixel points on the target image whose brightness values ​​are higher than the first preset brightness value are determined as pixel points of the reflective portion, and pixel points on the target image whose brightness values ​​are lower than the second preset brightness value are determined as pixel points of the reflective portion. The pixel points of the reflective portion are clustered to obtain multiple pixel clusters, and the central coordinates of the pixel clusters are determined as the position information of the reflective portion, that is, the central coordinates of the pixel clusters are determined as the two-dimensional coordinates of the reflective portion; the number of pixel clusters is determined as the number of reflective portions.

[0099] In another specific embodiment, the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover. Since the reflectivity of the reflective portion is less than the reflectivity of the outer shell cover, the light reflected by the reflective portion is weaker than the light reflected by the outer shell cover, and the brightness of the reflective portion on the outer shell cover will be significantly lower than that of the outer shell cover next to it. The pixel points on the target image whose brightness values ​​are lower than the second preset brightness value are determined as the pixel points of the reflective portion, the pixel points of the reflective portion are clustered to obtain a plurality of pixel clusters, and the central coordinates of the pixel clusters are determined as the position information of the reflective portion, that is, the central coordinates of the pixel cluster are determined as the two-dimensional coordinates of the reflective portion; the number of pixel clusters is determined as the number of reflective portions.

[0100] In another specific embodiment, the target image is input into a preset target detection model to obtain the number and position information of the reflective parts. The preset target detection model may be a Fast R-CNN, YOLOV5 or other model, which is not limited here.

[0101] 404. Determine the second position information of the handheld controller determined based on the inertial sensor as the current position information.

[0102] Specifically, three-axis angular velocity information and displacement information collected by the inertial sensor are obtained; and second position information of the handheld controller is determined based on the three-axis angular velocity information and displacement information collected by the inertial sensor.

[0103] 405. Determine whether there are at least three reflective parts on the target image.

[0104] In the embodiment of the present application, when there are at least three reflective parts on the target image, execute 406: perform pose estimation based on at least three two-dimensional plane coordinates and at least three three-dimensional space coordinates to obtain pose information of the image acquisition device. When there are no at least three reflective parts on the target image, perform three-dimensional reconstruction based on the position information of at least one reflective part on the two target images taken by the binocular camera to obtain pose information of the handheld controller, and then perform positioning tracking.

[0105] 406. Perform pose estimation based on at least three two-dimensional plane coordinates and at least three three-dimensional space coordinates to obtain pose information of the image acquisition device.

[0106] In an embodiment of the present application, after obtaining at least three two-dimensional plane coordinates and corresponding at least three three-dimensional space coordinates, the PnP algorithm can be used to perform pose estimation based on the at least three two-dimensional plane coordinates and at least three three-dimensional space coordinates to obtain the pose information of the image acquisition device.

[0107] 407. Determine the first position information of the handheld controller based on the position information of the image acquisition device.

[0108] The first position information of the handheld controller is the position information of the handheld controller in the world coordinate system. The position information of the image acquisition device is the position information of the image acquisition device in the preset space coordinate system. The first position information of the handheld controller is determined based on the position information of the image acquisition device through coordinate system conversion.

[0109] 408. Fusing the first posture information and the second posture information of the handheld controller determined based on the inertial sensor to obtain current posture information.

[0110] Specifically, the first pose information and the second pose information are weightedly summed to obtain the current pose information.

[0111] 409. Position and track the handheld controller based on the current posture information.

[0112] In the embodiment of the present application, after obtaining the current posture information of the handheld controller, the posture information of the handheld controller is updated in real time to achieve positioning tracking.

[0113] In the present application, compared with the related art, the handheld controller includes: a shell cover and at least one control member arranged on the shell cover, the at least one control member is used to control the device to be controlled; a handle body, one end of the handle body is connected to the shell cover, and the side wall of the handle body has a gripping surface; at least one reflective portion, at least one reflective portion is arranged on the handle body, wherein the reflectivity of the reflective portion is different from the reflectivity of the handle body; or at least one reflective portion is arranged on the shell cover, wherein the reflectivity of the reflective portion is different from the reflectivity of the shell cover. In the present application, at least one reflective portion is arranged on the shell cover or the handle body of the handheld controller, which is used to reflect light to the image acquisition device on the device to be controlled, so that the image acquisition device on the device to be controlled can acquire the target image containing the reflective portion, the reflectivity of the reflective portion is different from the reflectivity of the handle body or the shell cover where the reflective portion is located, and the reflective portion can be accurately identified and used to locate and track the handheld controller, because the light-emitting portion does not consume energy.

[0114] The handheld controller, positioning tracking system and method provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0115] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A handheld controller, characterized in that: include: A housing cover and at least one control element arranged on the housing cover, wherein the at least one control element is used to control a device to be controlled; A handle body, one end of which is connected to the housing cover, and a side wall of the handle body has a gripping surface; At least one reflective portion, wherein the at least one reflective portion is disposed on the handle body, wherein the reflectivity of the reflective portion is different from the reflectivity of the handle body; or, the at least one reflective portion is disposed on the outer shell cover, wherein the reflectivity of the reflective portion is different from the reflectivity of the outer shell cover.

2. The handheld controller according to claim 1, characterized in that: The handheld controller comprises at least three reflective parts, and the at least three reflective parts are arranged at intervals on the handle body or the shell cover.

3. The handheld controller according to claim 2, characterized in that: At least three of the reflective portions are arranged on an outer surface of one side of the housing cover.

4. The handheld controller according to claim 2, characterized in that: At least three of the reflective portions are disposed on the inner surface of the outer shell cover, the outer shell cover transmits light, and the reflective portions reflect light that passes through the outer shell cover.

5. The handheld controller according to claim 1, characterized in that: The reflection mode of the reflective part is diffuse reflection mode.

6. The handheld controller according to claim 1, characterized in that: The handheld controller comprises an inertial sensor, which is arranged in the handle body and is used to collect three-axis angular velocity information and displacement information of the handheld controller.

7. The handheld controller according to claim 1, characterized in that: A first trigger is provided on the side wall of the handle body, and the outer shell cover includes a first cover portion close to the first trigger and a second cover portion away from the first trigger, and the number of the reflective portions on the first cover portion is greater than the number of the reflective portions on the second cover portion.

8. A positioning tracking system, characterized in that: The positioning and tracking system includes a device to be controlled and a handheld controller according to any one of claims 1 to 7, wherein the device to be controlled includes an image acquisition device, the image acquisition device is used to take a picture of the handheld controller, and the reflective portion on the handheld controller is used to reflect light so as to be photographed by the image acquisition device of the device to be controlled and perform positioning tracking.

9. The positioning tracking system according to claim 8, characterized in that: The device to be controlled is provided with a fill light device, and the fill light device is used to emit light for fill light when the image acquisition device is exposed.

10. A positioning tracking method, characterized in that: The positioning tracking system according to claim 8, wherein the positioning tracking system comprises a positioning tracking device, wherein the positioning tracking device is used to execute the positioning tracking method, and the positioning tracking method comprises: In the case of acquiring a target image sent by the device to be controlled, detecting whether there is at least one reflective portion on the target image; In the case where at least one of the reflective portions exists on the target image, acquiring position information of at least one of the reflective portions; The handheld controller is located and tracked based on the position information of at least one of the reflective portions.

11. The positioning tracking method according to claim 10, characterized in that: The handheld controller comprises at least three reflective parts, which are arranged at intervals on the handle body or the shell cover. The target image captured by the image acquisition device of the device to be controlled for the handheld controller contains three reflective parts. The position information is a two-dimensional plane coordinate. The method for locating and tracking the handheld controller based on the position information of at least one reflective part comprises: In the case where there are at least three reflective parts on the target image, pose estimation is performed based on at least three two-dimensional plane coordinates of the at least three reflective parts and at least three pre-stored three-dimensional space coordinates to obtain pose information of the image acquisition device; Determining first posture information of the handheld controller based on the posture information of the image acquisition device; Determining current posture information of the handheld controller based on the first posture information; The handheld controller is positioned and tracked based on the current posture information.

12. The positioning tracking method according to claim 11, characterized in that: The determining the current posture information of the handheld controller based on the first posture information includes: Acquire second position information of the handheld controller determined based on the inertial sensor; The first posture information and the second posture information are fused to obtain the current posture information.

13. The positioning tracking method according to claim 10, characterized in that: The positioning tracking method comprises: When it is detected that the image acquisition device emits a stroboscopic signal, a fill light device of the device to be controlled is driven to start emitting a preset light; When it is detected that the stroboscopic signal of the image acquisition device ends, the fill light device of the device to be controlled is driven to be turned off.

Images