Handle tracking method, smart wearable device and readable storage medium

Abstract

The application discloses a handle tracking method, a smart wearable device and a readable storage medium. The handle tracking method comprises the following steps: dynamically detecting whether first feature identification points and second feature identification points exist in image information collected by a head-mounted device; if the first feature identification points exist in the image information and the second feature identification points are missing, tracking and positioning a first handle according to the first feature identification points in the image information, and tracking and positioning a second handle through a first depth-sensing camera. The application can expand the motion range of the handle, improve the positioning accuracy of the handle, and solve the technical problem that the handle position information of the smart wearable device is easily lost when tracking and positioning the handle.

Description

Technical Field

[0001] This application relates to the technical field of wearable devices, and in particular to a handle tracking method, a smart wearable device, and a readable storage medium. Background Art

[0002] VR (Virtual Reality) devices or AR (Augmented Reality) devices are smart wearable devices that are currently developing and popularizing rapidly. Many smart wearable devices on the market, such as AR / VR glasses or AR / VR helmets, are generally equipped with handles. The handle is mainly an auxiliary device for operating and controlling the virtual environment created by the smart wearable device or for operating and controlling characters / roles in the virtual environment. Therefore, in order to enable the handle to accurately operate and control the virtual environment and / or characters / roles, it is necessary to track and position the handle.

[0003] Currently, the widely adopted solution is to track and position the handle through the camera on the head-mounted display device. However, the camera has many viewing blind spots and cannot capture areas outside the viewing range, which easily leads to the loss of the position information of the handle and poor positioning accuracy. That is, the handle may move to an area that is not covered by the FOV (Field of View) of the camera on the head-mounted display device, and at this time, the position information of the handle will be lost. At the same time, if the positions of the two handles overlap in the viewing direction of the camera (that is, the two handles are relatively blocked), it will also cause the loss of the position information of one of the handles. Summary of the Invention

[0004] The main purpose of this application is to provide a handle tracking method, a smart wearable device, and a readable storage medium, aiming to solve the technical problem of easily losing the position information of the handle when tracking and positioning the handle of a smart wearable device.

[0005] To achieve the above object, this application provides a handle tracking method. The handle tracking method is applied to a smart wearable device. The smart wearable device includes a head-mounted display device, a first handle and a second handle that are配套 with the head-mounted display device. The first handle is provided with a first feature identification point and a first depth camera, and the second handle is provided with a second feature identification point and a second depth camera. The handle tracking method includes:

[0006] Dynamically detecting whether the first feature identification point and the second feature identification point are present in the image information collected by the head-mounted display device;

[0007] If the image information contains the first feature marker but lacks the second feature marker, then the first handle is tracked and located based on the first feature marker in the image information, and the second handle is tracked and located using the first depth-sensing camera.

[0008] Optionally, the step of tracking and locating the first handle based on the first feature marker in the image information includes:

[0009] Detect the spatial position information of the head-mounted display device;

[0010] Based on the first feature marker point in the image information, the first relative position information between the head-mounted display device and the first handle is determined;

[0011] Based on the spatial position information of the head-mounted display and the first relative position information, the first handle is tracked and positioned to obtain the spatial position information of the first handle.

[0012] Optionally, the head-mounted display device includes a head-mounted display body and a position sensor, and the step of detecting the head-mounted display spatial position information of the head-mounted display device includes:

[0013] The position sensor detects the coordinate position vector of the head-mounted display body in the world coordinate system.

[0014] The coordinate position vector of the head-mounted display body in the world coordinate system is used as the head-mounted display spatial position information of the head-mounted display device;

[0015] The step of determining the first relative position information between the head-mounted display device and the first handle based on the first feature marker point in the image information includes:

[0016] Based on the first feature marker point in the image information, determine the first relative spatial coordinate vector between the first handle and the head-mounted display device in the head-mounted display coordinate system;

[0017] The first relative spatial coordinate vector is used as the first relative position information between the head-mounted display device and the first handle.

[0018] Optionally, the step of tracking and locating the first handle based on the head-mounted display spatial position information and the first relative position information to obtain the spatial position information of the first handle includes:

[0019] By using a preset first matrix transformation rule, the first relative spatial coordinate vector is converted into a second relative spatial coordinate vector between the first handle and the head-mounted display device in the world coordinate system;

[0020] The second relative spatial coordinate vector and the coordinate position vector of the head-mounted display device in the world coordinate system are summed to obtain the coordinate position vector of the first handle in the world coordinate system.

[0021] The coordinate position vector of the first handle in the world coordinate system is used as the spatial position information of the first handle.

[0022] Optionally, the step of tracking and locating the second handle using the first depth-sensing camera includes:

[0023] The depth information of the second handle is acquired through the first depth-sensing camera;

[0024] Based on the depth information of the second handle, determine the relative position information between the first handle and the second handle;

[0025] Based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the controllers, the second controller is tracked and positioned to obtain the spatial position information of the second controller.

[0026] Optionally, the step of determining the relative position information between the first handle and the second handle based on the depth information of the second handle includes:

[0027] Based on the depth information of the second handle, determine the third relative spatial coordinate vector between the first handle and the second handle in the handle coordinate system;

[0028] The third relative spatial coordinate vector is used as the relative position information between the handles.

[0029] Optionally, the step of tracking and locating the second controller based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the controllers to obtain the spatial position information of the second controller includes:

[0030] By using a preset second matrix transformation rule, the third relative spatial coordinate vector is converted into a fourth relative spatial coordinate vector between the first handle and the second handle in the world coordinate system;

[0031] Based on the spatial position information of the head-mounted display and the first relative position information, determine the coordinate position vector of the first handle in the world coordinate system;

[0032] The fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system are summed to obtain the coordinate position vector of the second handle in the world coordinate system.

[0033] The coordinate position vector of the second handle in the world coordinate system is used as the spatial position information of the second handle.

[0034] Optionally, after the step of dynamically detecting whether the image information acquired by the head-mounted display device contains the first feature marker and the second feature marker, the method further includes:

[0035] If it is detected that both the first feature marker and the second feature marker are missing in the image information, then the acquisition period corresponding to the most recently acquired image information where both the first feature marker and the second feature marker are not missing will be the target acquisition period.

[0036] The spatial position information of the first handle and the second handle during the target acquisition cycle is used as the initial spatial position information of the first handle and the second handle.

[0037] Acquire the spatial pose change information of the first and second handles from the target acquisition cycle to the current acquisition cycle;

[0038] Based on the initial spatial position information and the spatial pose change information, the first handle and the second handle are tracked and positioned.

[0039] This application also provides a smart wearable device, which is a physical device. The smart wearable device includes: a memory, a processor, and a program of the handle tracking method stored in the memory and executable on the processor. When the program of the handle tracking method is executed by the processor, it can implement the steps of the handle tracking method as described above.

[0040] This application also provides a readable storage medium, which is a computer-readable storage medium, on which a program implementing a controller tracking method is stored, and the program implementing the controller tracking method is executed by a processor to implement the steps of the controller tracking method as described above.

[0041] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the handle tracking method described above.

[0042] This application sets a first feature marker and a first depth-sensing camera on a first handle, and a second feature marker and a second depth-sensing camera on a second handle. It then dynamically detects whether the image information captured by the head-mounted display (HMD) contains the first and second feature markers. If the first feature marker is detected but the second feature marker is missing, the first handle is tracked and located based on the first feature marker in the image information to determine its real-time position. When the camera on the HMD does not capture the second handle, and the second handle is outside the camera's field of view, this application activates the first depth-sensing camera on the first handle to continue capturing the position of the handle within the HMD's field of view. A second handle can be provided outside the field of view of the camera, or the first depth-sensing camera can be used to continue capturing the second handle that is covered by the first handle in the field of view of the head-mounted display. Since the first depth-sensing camera can acquire the surrounding depth information, when the first depth-sensing camera is turned on to further capture the second handle, the second handle can be identified through feature matching, and the relative position relationship between the second handle and the first handle can be obtained. Based on the relative position relationship, the current position information of the second handle can be determined, and the second handle can be tracked and located. This expands the range of motion of the second handle, improves the positioning accuracy of the handle, and thus solves the technical problem that the handle position information is easily lost when tracking and locating the handle of the smart wearable device. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the first embodiment of the handle tracking method of this application;

[0046] Figure 2 This is a flowchart illustrating the second embodiment of the handle tracking method of this application;

[0047] Figure 3 This is a flowchart illustrating the third embodiment of the handle tracking method of this application;

[0048] Figure 4 This is a flowchart illustrating the fourth embodiment of the handle tracking method of this application;

[0049] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the smart wearable device in the embodiment of the present application.

[0050] The realization of the purpose, functional features and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. Specific Embodiments

[0051] To make the above objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] In this embodiment, the smart wearable device of the present application may be, for example, a Mixed Reality (MR) device (such as an MR helmet or MR glasses), an Augmented Reality (AR) device (such as an AR helmet or AR glasses), a Virtual Reality (VR) device (such as a VR helmet or VR glasses), an Extended Reality (XR) device (such as an XR helmet or XR glasses), or a combination thereof.

[0053] Currently, the widely adopted solution is to track and locate the handle through the camera on the head-mounted display device. However, there are many blind spots in the camera's perspective, and it is impossible to capture the area outside the perspective range, which is likely to result in the loss of the handle's position information and poor positioning accuracy. That is, the handle may move to an area that is not covered by the Field of View (FOV) of the camera on the head-mounted display device, and at this time, the position information of the handle will be lost. At the same time, if the positions of the two handles overlap in the perspective direction of the camera (i.e., the two handles are relatively blocked), it will also lead to the loss of the position information of one of the handles.

[0054] Embodiment 1

[0055] Based on this, please refer to Figure 1 , this embodiment provides a handle tracking method. The handle tracking method is applied to a smart wearable device, which includes a head-mounted display device, a first handle and a second handle that are配套 with the head-mounted display device. The first handle is provided with a first feature identification point and a first depth camera, and the second handle is provided with a second feature identification point and a second depth camera. The handle tracking method includes:

[0056] Step S10: Dynamically detect whether the image information acquired by the head-mounted display device contains the first feature identifier point and the second feature identifier point;

[0057] In this embodiment, the first and second handles can be tracked and located using the camera of the head-mounted display device. Specifically, the camera of the head-mounted display device captures current image information. If the image information contains the first feature marker, it indicates that the camera on the head-mounted display device has captured the first handle, meaning the first handle is within the field of view of the camera on the head-mounted display device. If the image information contains the second feature marker, it indicates that the camera on the head-mounted display device has captured the second handle, meaning the second handle is within the field of view of the camera on the head-mounted display device.

[0058] Correspondingly, if the first feature marker is missing from the image information, it indicates that the camera on the head-mounted display device has not captured the first handle, meaning the first handle is outside the field of view of the camera on the head-mounted display device. If the second feature marker is missing from the image information, it indicates that the camera on the head-mounted display device has not captured the second handle, meaning the second handle is outside the field of view of the camera on the head-mounted display device.

[0059] The camera on the head-mounted display can be a binocular camera or a quad-camera, etc. Furthermore, in this embodiment, the camera can perform 6DoF (degree of freedom) tracking on the head-mounted display based on SLAM (simultaneous localization and mapping) technology. In this embodiment, the camera can be a regular camera, an RGBD (Red, Green, Blue Depth) camera, or a fisheye camera.

[0060] It should be noted that the feature marker refers to a feature mark set on the outer shell of the handle to assist in the positioning and identification of the handle. For example, the feature mark can be manifested as a specific pattern, an object of a specific shape, or a light ring that emits infrared light, etc. This embodiment does not impose specific limitations and can be determined according to the specific implementation. The processor of the smart wearable device can identify the handle by recognizing the feature marker in the image information.

[0061] In this embodiment, the first handle and the second handle can refer to the left handle and the right handle. The first and second feature markers can be differentiated through different designs. For example, the feature markers are the light-emitting points of an infrared light source. The number and / or arrangement of the light-emitting points of the infrared light source on the left handle and the right handle can be designed differently. For instance, the light-emitting points on the left handle can be arranged in a semi-circular shape, while the light-emitting points on the right handle can be arranged in a circular shape.

[0062] As will be understood by those skilled in the art, a depth-sensing camera is a TOF (Time of Flight) camera. TOF is simply an abbreviation for a technology that involves emitting a set of infrared light (laser pulses) invisible to the human eye. Upon encountering an object, the light reflects off and reaches the camera. The time difference or phase difference between the emission and reflection is calculated, and the data is collected to form a set of distance and depth data, thereby obtaining a stereoscopic 3D model.

[0063] Step S20: If the image information contains the first feature marker but lacks the second feature marker, then the first handle is tracked and located based on the first feature marker in the image information, and the second handle is tracked and located using the first depth-sensing camera.

[0064] In this embodiment, it is easy to understand that if the image information contains both a first feature marker and a second feature marker, the first handle can be tracked and located directly based on the first feature marker in the image information, and the second handle can be tracked and located based on the second feature marker in the image information.

[0065] It should be noted that if the image information captured by the head-mounted display device contains feature marker 'a' on the left handle but lacks feature marker 'b' on the right handle, then the first handle is the left handle, the first feature marker is feature marker 'a', the second handle is the right handle, and the second feature marker is feature marker 'b'. Conversely, if the image information captured by the head-mounted display device contains feature marker 'b' on the right handle but lacks feature marker 'a' on the left handle, then the first handle is updated to the right handle, the first feature marker is updated to feature marker 'b', the second handle is updated to the left handle, and the second feature marker is updated to feature marker 'a'.

[0066] In this embodiment, if a first feature marker is detected in the image information, it indicates that the camera on the head-mounted display device has captured the first handle. Therefore, the first handle can be tracked and located by analyzing the environmental information where the first feature marker is located in the image information. Specifically, those skilled in the art will understand that the captured image information can be processed based on a preset SLAM (simultaneous localization and mapping) technology to study the transformation relationship between frames, complete the real-time pose tracking of the first handle, calculate the pose change, obtain the camera pose data of the first handle, and determine the current position information of the first handle based on the camera pose data of the first handle. Of course, the camera pose data can also be fused with the inertial navigation pose data collected by the IMU (Inertial Measurement Unit) sensor on the first handle based on a preset VIO (Visual Inertial Odometry) algorithm to determine the current position information of the first handle and achieve 6DoF positioning and tracking of the first handle. The SLAM technology and VIO algorithm have been studied in depth by those skilled in the art and will not be elaborated here. For example, current mainstream visual SLAM technologies include feature-based SLAM and direct-method SLAM. The VIO algorithm, for instance, can be divided into a front-end and a back-end. The front-end completes data association, while the back-end primarily optimizes the output of the front-end, using filtering or optimization theory to optimize trees or graphs, obtaining optimal pose estimation and maps, and performing real-time pose localization and map construction, thereby creating a more immersive and realistic experience in virtual reality content.

[0067] Of course, when capturing image information of each charging base station, the processor of the smart wearable device can also track and locate the first handle by analyzing the orientation angle of the camera of the head-mounted display device (including the rotation angle of the horizontal plane and the pitch angle of the vertical plane) to determine the current position information of the first handle.

[0068] In this embodiment, if the second feature marker is missing from the image information captured by the head-mounted display device, it indicates that the camera on the head-mounted display device has not captured the second handle, and the second handle is located outside the field of view of the camera on the head-mounted display device. Therefore, this embodiment activates the first depth-sensing camera of the first handle to continue capturing the second handle located outside the field of view of the camera on the head-mounted display device, or to continue capturing the second handle that is obscured by the first handle in the camera's field of view direction of the head-mounted display device, thus determining the current position information of the second handle. Specifically, the first depth-sensing camera of the first handle emits infrared light (laser pulse) towards the second handle, which is reflected upon encountering the second handle. The first depth-sensing camera receives the reflected infrared light, calculates the time difference or phase difference from emission to reflection back to the camera, and collects the data to form a set of distance and depth data for analysis, thereby obtaining the three-dimensional imaging information of the second handle in the environment. Based on the three-dimensional imaging information of the second handle in the environment, the environmental information of the second handle is analyzed, and the second handle is tracked and located. More specifically, based on a preset SLAM (simultaneous localization and mapping) technology, the environmental information of the second handle can be processed to study the transformation relationship between frames, complete the real-time pose tracking of the second handle, calculate the pose change, obtain the camera pose data of the second handle, and determine the current position information of the second handle based on the camera pose data. Alternatively, based on a preset VIO (Visual Inertial Odometry) algorithm, the camera pose data can be fused with the inertial navigation pose data collected by the IMU (Inertial Measurement Unit) sensor on the second handle to determine the current position information of the second handle, achieving 6DoF positioning and tracking of the second handle.

[0069] This embodiment sets a first feature marker and a first depth-sensing camera on the first handle, and a second feature marker and a second depth-sensing camera on the second handle. Then, it dynamically detects whether the image information captured by the head-mounted display (HMD) contains the first and second feature markers. If the first feature marker is detected but the second feature marker is missing, the first handle is tracked and located based on the first feature marker in the image information to determine its real-time position. When the camera on the HMD does not capture the second handle, and the second handle is outside the camera's field of view, this embodiment activates the first depth-sensing camera on the first handle to continue capturing the position of the handle within the HMD's field of view. The second handle is located outside the field of view of the camera on the device, or the first depth-sensing camera continues to capture the second handle that is covered by the first handle in the field of view of the head-mounted display device. Since the first depth-sensing camera can acquire the surrounding depth information, when the first depth-sensing camera is turned on to further capture the second handle, the second handle can be identified by feature matching, and the relative position relationship between the second handle and the first handle can be obtained. Based on the relative position relationship, the current position information of the second handle can be determined, and the second handle can be tracked and located. This expands the range of motion of the second handle, improves the positioning accuracy of the handle, and thus solves the technical problem that the handle position information is easily lost when tracking and locating the handle of the smart wearable device.

[0070] Specifically, the head-mounted display device's camera can capture a first feature marker point, collect environmental information around the first feature marker point, identify the relative positional relationship between the head-mounted display device and the first handle, and then combine the head-mounted display device's position sensors (such as cameras and inertial sensors) to locate itself in space, obtaining the head-mounted display device's spatial position information. Then, based on this head-mounted display spatial position information and the relative positional relationship between the head-mounted display device and the first handle, the spatial position information of the first handle can be calculated and deduced. Then, based on the spatial position information of the first handle and the relative positional relationship between the first handle and the second handle, the spatial position information of the second handle can be calculated and deduced. This enables the tracking and positioning of the other missing handle within the field of view of the two handles when one handle moves into an area not covered by the camera's FOV (Field of View) on the head-mounted display device, or when the two handles overlap in the camera's field of view. This expands the range of motion of the handles and solves the technical problem of easily losing handle position information when tracking and positioning the handles of smart wearable devices.

[0071] In one possible implementation, please refer to Figure 2 The step of tracking and locating the first handle based on the first feature marker in the image information includes:

[0072] Step S31: Detect the spatial position information of the head-mounted display device;

[0073] Step S32: Determine the first relative position information between the head-mounted display device and the first handle based on the first feature marker point in the image information;

[0074] Step S33: Based on the spatial position information of the head-mounted display and the first relative position information, track and locate the first handle to obtain the spatial position information of the first handle.

[0075] In this embodiment, the spatial position information of the head-mounted display (HMD) of the smart wearable device can be detected using a position sensor mounted on the device. This position sensor can be a camera and an IMU (Inertial Measurement Unit) sensor located on the HMD. The camera on the HMD can perform 6DoF (degree of freedom) tracking on the HMD based on SLAM (simultaneous localization and mapping) technology.

[0076] As will be known to those skilled in the art, the IMU sensor includes a three-axis gyroscope for obtaining acceleration and a three-axis accelerometer for obtaining angular velocity. The IMU sensor can detect the rotational and translational degrees of freedom (e.g., yaw, pitch, and roll angles) of the head-mounted display device, and determine the 6DOF degrees of freedom of the head-mounted display device based on the rotational and translational degrees of freedom, thereby obtaining the inertial navigation pose data of the head-mounted display device, and then determining the head-mounted display spatial position information of the head-mounted display device based on the inertial navigation pose data.

[0077] Furthermore, as will be readily understood by those skilled in the art, the captured image can be processed using images of the current environment captured by the camera on a smart wearable device. Based on a pre-defined SLAM (simultaneous localization and mapping) technique, the transformation relationships between frames can be studied to achieve real-time pose tracking, calculate pose changes, and obtain the camera pose data of the smart wearable device. Then, based on a pre-defined VIO (Visual Inertial Odometry) algorithm, the camera pose data and inertial navigation pose data can be fused to determine the spatial position information of the head-mounted display (HMD) of the smart wearable device, achieving 6DoF positioning and tracking of the HMD. The SLAM technology and VIO algorithm have been extensively studied by those skilled in the art and will not be elaborated upon here.

[0078] In this embodiment, it is known that the camera mounted on the head-mounted display body must not only achieve its own visual positioning, but also achieve visual positioning of the first handle that is matched with the head-mounted display body.

[0079] This embodiment detects the spatial position information of the head-mounted display device and determines the first relative position information between the head-mounted display device and the first handle based on the first feature marker point in the image information. Then, based on the spatial position information of the head-mounted display device and the first relative position information, the spatial position information of the first handle is calculated, thereby achieving accurate positioning of the first handle.

[0080] Furthermore, in one possible implementation, the head-mounted display device includes a head-mounted display body and a position sensor, and the step of detecting the head-mounted display spatial position information of the head-mounted display device includes:

[0081] Step A10: Detect the coordinate position vector of the head-mounted display body in the world coordinate system using the position sensor;

[0082] In this embodiment, those skilled in the art will understand that the world coordinate system refers to the coordinate system established by representing the spatial coordinates of the real physical world.

[0083] Step A20: Use the coordinate position vector of the head-mounted display body in the world coordinate system as the head-mounted display spatial position information of the head-mounted display device;

[0084] The step of determining the first relative position information between the head-mounted display device and the first handle based on the first feature marker point in the image information includes:

[0085] Step A30: Based on the first feature marker point in the image information, determine the first relative spatial coordinate vector between the first handle and the head-mounted display device in the head-mounted display coordinate system;

[0086] As will be understood by those skilled in the art, a head-mounted display coordinate system refers to a coordinate system established with the head-mounted display device as the reference.

[0087] Step A40: Use the first relative spatial coordinate vector as the first relative position information between the head-mounted display device and the first handle.

[0088] In this embodiment, a position sensor installed on the head-mounted display device detects the coordinate position vector of the head-mounted display body in the world coordinate system, and determines the first relative spatial coordinate vector of the first handle and the head-mounted display device in the head-mounted display coordinate system based on the first feature marker point in the image information, thereby accurately obtaining the first relative position information of the head-mounted display device and the first handle.

[0089] In one possible implementation, the step of tracking and locating the first handle based on the head-mounted display spatial position information and the first relative position information to obtain the spatial position information of the first handle includes:

[0090] Step B10: Using a preset first matrix transformation rule, the first relative spatial coordinate vector is converted into a second relative spatial coordinate vector between the first handle and the head-mounted display device in the world coordinate system.

[0091] In this embodiment, the first matrix transformation rule refers to the matrix transformation rule that transforms the coordinate vector in the head-mounted display coordinate system to the coordinate vector in the world coordinate system.

[0092] Step B20: Sum the second relative spatial coordinate vector and the coordinate position vector of the head-mounted display device in the world coordinate system to obtain the coordinate position vector of the first handle in the world coordinate system.

[0093] Step B30: Use the coordinate position vector of the first handle in the world coordinate system as the spatial position information of the first handle.

[0094] This embodiment uses a preset first matrix transformation rule to convert the first relative spatial coordinate vector into a second relative spatial coordinate vector between the first handle and the head-mounted display device in the world coordinate system. Then, the second relative spatial coordinate vector and the coordinate position vector of the head-mounted display device in the world coordinate system are summed to obtain the coordinate position vector of the first handle in the world coordinate system, thereby realizing the spatial positioning of the first handle and accurately calculating the spatial position information of the first handle.

[0095] In one possible implementation, please refer to Figure 3 The step of tracking and locating the second handle using the first depth-sensing camera includes:

[0096] Step S41: Collect depth information of the second handle using the first depth-sensing camera;

[0097] Step S42: Determine the relative position information between the first handle and the second handle based on the depth information of the second handle;

[0098] In this embodiment, the first depth-sensing camera of the first handle emits infrared light (laser pulse) towards the second handle. After encountering the second handle, the light is reflected. The first depth-sensing camera receives the reflected infrared light and calculates the time difference or phase difference from emission to reflection back to the camera. The data is collected to form a set of distance and depth data for analysis, thereby obtaining the depth information of the second handle in the environment. Based on the depth information of the second handle in the environment, the relative position between the first and second handles is analyzed.

[0099] For example, the step of determining the relative position information between the first handle and the second handle based on the depth information of the second handle includes:

[0100] Step C10: Based on the depth information of the second handle, determine the third relative spatial coordinate vector between the first handle and the second handle in the handle coordinate system;

[0101] In this embodiment, those skilled in the art will understand that the handle coordinate system refers to a coordinate system established with the first handle as a reference.

[0102] Step C20: Use the third relative spatial coordinate vector as the relative position information between the handles.

[0103] After step S42, step S43 is executed, in which the second handle is tracked and located according to the spatial position information of the head-mounted display, the first relative position information and the relative position information between the handles, to obtain the spatial position information of the second handle.

[0104] In this embodiment, the depth information of the second handle is collected by the first depth-sensing camera, and the relative position information between the first handle and the second handle is determined based on the depth information of the second handle. Then, based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the handles, the spatial position information of the second handle is calculated, thereby achieving accurate positioning of the second handle.

[0105] Further, in one possible implementation, the step of tracking and locating the second controller based on the head-mounted display spatial position information, the first relative position information, and the relative position information between the controllers to obtain the spatial position information of the second controller includes:

[0106] Step D10: Using a preset second matrix transformation rule, the third relative spatial coordinate vector is converted into a fourth relative spatial coordinate vector between the first handle and the second handle in the world coordinate system.

[0107] In this embodiment, the second matrix transformation rule refers to the matrix transformation rule that transforms the coordinate vector in the handle coordinate system to the coordinate vector in the world coordinate system.

[0108] Step D20: Determine the coordinate position vector of the first handle in the world coordinate system based on the spatial position information of the head-mounted display and the first relative position information;

[0109] Step D30: Sum the fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system to obtain the coordinate position vector of the second handle in the world coordinate system.

[0110] Step D40: Use the coordinate position vector of the second handle in the world coordinate system as the spatial position information of the second handle.

[0111] This embodiment uses a preset second matrix transformation rule to convert the third relative spatial coordinate vector into a fourth relative spatial coordinate vector between the first handle and the second handle in the world coordinate system. Based on the head-mounted display spatial position information and the first relative position information, the coordinate position vector of the first handle in the world coordinate system is determined. Then, the fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system are summed to obtain the coordinate position vector of the second handle in the world coordinate system, thereby realizing the spatial positioning of the second handle and accurately calculating the spatial position information of the second handle.

[0112] Example 2

[0113] Based on the above embodiments of this application, please refer to Figure 4 In another embodiment of this application, the same or similar content as in Embodiment 1 above can be referred to the above description and will not be repeated hereafter. Based on this, after the step of dynamically detecting whether the image information acquired by the head-mounted display device contains the first feature marker and the second feature marker, the method further includes:

[0114] Step S51: If it is detected that both the first feature marker and the second feature marker are missing in the image information, then the acquisition period corresponding to the last acquired image information where both the first feature marker and the second feature marker are not missing is taken as the target acquisition period.

[0115] To aid in understanding the embodiments of this application, an example is provided, comprising consecutive acquisition cycles: acquisition cycle a, acquisition cycle b, acquisition cycle c, acquisition cycle d, and acquisition cycle e. The image information acquired in acquisition cycles a and d contains a first feature marker but lacks a second feature marker, while the image information acquired in acquisition cycles b, c, and e completely lacks both the first and second feature markers. Therefore, if the first and second feature markers are completely missing in the image information during acquisition cycle c, the target acquisition cycle is cycle a. Similarly, if the first and second feature markers are completely missing in the image information during acquisition cycle e, the target acquisition cycle is cycle d.

[0116] Step S52: Use the spatial position information of the first handle and the second handle during the target acquisition cycle as the initial spatial position information of the first handle and the second handle.

[0117] Step S53: Obtain the spatial pose change information of the first handle and the second handle from the target acquisition period to the current acquisition period;

[0118] In this embodiment, the spatial pose change information of the first handle can be detected by an IMU (Inertial Measurement Unit) sensor provided on the first handle. The spatial pose change information of the second handle can be detected by an IMU (Inertial Measurement Unit) sensor provided on the second handle.

[0119] Step S54: Perform tracking and positioning on the first handle and the second handle according to the initial spatial position information and the spatial pose change information.

[0120] In this embodiment, if it is detected that both the first feature identification point and the second feature identification point are missing in the image information, then the acquisition period corresponding to the situation where the first feature identification point and the second feature identification point are not all missing in the most recently acquired image information is used as the target acquisition period, and the spatial position information of the first handle and the second handle in the target acquisition period is used as the initial spatial position information of the first handle and the second handle. Then, the spatial pose change information of the first handle and the second handle from the target acquisition period to the current acquisition period is obtained, and based on the initial spatial position information and the spatial pose change information, tracking and positioning are performed on the first handle and the second handle, so as to accurately obtain the spatial position information of the first handle and the spatial position information of the second handle.

[0121] Embodiment Three

[0122] The embodiment of the present invention further provides a handle tracking device. The handle tracking device is applied to a smart wearable device. The smart wearable device includes a head-mounted display device, and a first handle and a second handle that are paired with the head-mounted display device. The first handle is provided with a first feature identification point and a first depth camera, and the second handle is provided with a second feature identification point and a second depth camera. The handle tracking device includes:

[0123] A detection module, configured to dynamically detect whether the first feature identification point and the second feature identification point are present in the image information collected by the head-mounted display device;

[0124] A tracking module, configured to, if it is detected that the first feature identification point is present in the image information and the second feature identification point is missing, perform tracking and positioning on the first handle according to the first feature identification point in the image information, and perform tracking and positioning on the second handle through the first depth camera.

[0125] Optionally, the tracking module is further configured to:

[0126] Detect the spatial position information of the head-mounted display device;

[0127] Based on the first feature marker point in the image information, the first relative position information between the head-mounted display device and the first handle is determined;

[0128] Based on the spatial position information of the head-mounted display and the first relative position information, the first handle is tracked and positioned to obtain the spatial position information of the first handle.

[0129] Optionally, the head-mounted display device includes a head-mounted display body and a position sensor, and the tracking module is further configured to:

[0130] The position sensor detects the coordinate position vector of the head-mounted display body in the world coordinate system.

[0131] The coordinate position vector of the head-mounted display body in the world coordinate system is used as the head-mounted display spatial position information of the head-mounted display device;

[0132] Optionally, the tracking module is further configured to:

[0133] Based on the first feature marker point in the image information, determine the first relative spatial coordinate vector between the first handle and the head-mounted display device in the head-mounted display coordinate system;

[0134] The first relative spatial coordinate vector is used as the first relative position information between the head-mounted display device and the first handle.

[0135] Optionally, the tracking module is further configured to:

[0136] By using a preset first matrix transformation rule, the first relative spatial coordinate vector is converted into a second relative spatial coordinate vector between the first handle and the head-mounted display device in the world coordinate system;

[0137] The second relative spatial coordinate vector and the coordinate position vector of the head-mounted display device in the world coordinate system are summed to obtain the coordinate position vector of the first handle in the world coordinate system.

[0138] The coordinate position vector of the first handle in the world coordinate system is used as the spatial position information of the first handle.

[0139] Optionally, the tracking module is further configured to:

[0140] The depth information of the second handle is acquired through the first depth-sensing camera;

[0141] Based on the depth information of the second handle, determine the relative position information between the first handle and the second handle;

[0142] Based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the controllers, the second controller is tracked and positioned to obtain the spatial position information of the second controller.

[0143] Optionally, the tracking module is further configured to:

[0144] Based on the depth information of the second handle, determine the third relative spatial coordinate vector between the first handle and the second handle in the handle coordinate system;

[0145] The third relative spatial coordinate vector is used as the relative position information between the handles.

[0146] Optionally, the tracking module is further configured to:

[0147] By using a preset second matrix transformation rule, the third relative spatial coordinate vector is converted into a fourth relative spatial coordinate vector between the first handle and the second handle in the world coordinate system;

[0148] Based on the spatial position information of the head-mounted display and the first relative position information, determine the coordinate position vector of the first handle in the world coordinate system;

[0149] The fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system are summed to obtain the coordinate position vector of the second handle in the world coordinate system.

[0150] The coordinate position vector of the second handle in the world coordinate system is used as the spatial position information of the second handle.

[0151] Optionally, the tracking module is further configured to:

[0152] If it is detected that both the first feature marker and the second feature marker are missing in the image information, then the acquisition period corresponding to the most recently acquired image information where both the first feature marker and the second feature marker are not missing will be the target acquisition period.

[0153] The spatial position information of the first handle and the second handle during the target acquisition cycle is used as the initial spatial position information of the first handle and the second handle.

[0154] Acquire the spatial pose change information of the first and second handles from the target acquisition cycle to the current acquisition cycle;

[0155] Based on the initial spatial position information and the spatial pose change information, the first handle and the second handle are tracked and positioned.

[0156] The handle tracking device provided in this invention, employing the handle tracking method described in Embodiment 1 or Embodiment 2, solves the technical problem of easily losing handle position information when tracking and locating the handle of a smart wearable device. Compared with the prior art, the beneficial effects of the handle tracking device provided in this invention are the same as those of the handle tracking method provided in the above embodiments, and other technical features of the handle tracking device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0157] Example 4

[0158] This invention provides a smart wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the handle tracking method in Embodiment 1 above.

[0159] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the embodiments of the present disclosure of a smart wearable device. The smart wearable device in the embodiments of the present disclosure may include, but is not limited to, Mixed Reality (MR) devices, Augmented Reality (AR) devices, Virtual Reality (VR) devices, Extended Reality (XR) devices, or some combination thereof. Figure 5 The smart wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0160] like Figure 5 As shown, the smart wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). The RAM 1004 also stores various programs and data required for the operation of the AR glasses. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. Input / output (I / O) interfaces are also connected to the bus 1005.

[0161] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the AR glasses to communicate wirelessly or wiredly with other devices to exchange data. Although AR glasses with various systems are shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0162] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0163] The smart wearable device provided by this invention, employing the handle tracking method in Embodiment 1 or Embodiment 2 described above, can solve the technical problem of easily losing handle position information when tracking and locating the handle of a smart wearable device. Compared with the prior art, the beneficial effects of the smart wearable device provided by this invention are the same as those of the handle tracking method provided in Embodiment 1 described above, and other technical features of this smart wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0164] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0165] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0166] Example 5

[0167] This invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the handle tracking method in Embodiment 1 above.

[0168] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0169] The aforementioned computer-readable storage medium may be included in the smart wearable device; or it may exist independently and not assembled into the smart wearable device.

[0170] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the smart wearable device, the smart wearable device: dynamically detects whether the image information collected by the head-mounted display device contains the first feature identifier and the second feature identifier; if the first feature identifier is detected in the image information but the second feature identifier is missing, then the first handle is tracked and located based on the first feature identifier in the image information, and the second handle is tracked and located through the first depth-sensing camera.

[0171] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0174] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the above-described handle tracking method, which solves the technical problem that handle position information is easily lost when tracking and locating the handle of a smart wearable device. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as those of the handle tracking method provided in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0175] Example 6

[0176] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the handle tracking method described above.

[0177] The computer program product provided in this application can solve the technical problem that the handle position information is easily lost when tracking and locating the handle of a smart wearable device. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present invention are the same as the beneficial effects of the handle tracking method provided in Embodiment 1 or Embodiment 2 above, and will not be repeated here.

[0178] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A handle tracking method, characterized by, The described handle tracking method is applied to a smart wearable device, which includes a head-mounted display device, a first handle and a second handle that are paired with the head-mounted display device. The first handle is provided with a first feature identification point and a first depth camera, and the second handle is provided with a second feature identification point and a second depth camera. The handle tracking method includes: Dynamically detecting whether the first feature identification point and the second feature identification point are present in the image information collected by the head-mounted display device; If it is detected that the first feature identification point is present in the image information and the second feature identification point is missing, the first handle is tracked and positioned according to the first feature identification point in the image information, and the second handle is tracked and positioned through the first depth camera; If it is detected that both the first feature identification point and the second feature identification point are missing in the image information, the acquisition period corresponding to the situation where the first feature identification point and the second feature identification point were not both missing in the most recently collected image information is taken as the target acquisition period; the spatial position information of the first handle and the second handle in the target acquisition period is used as the initial spatial position information of the first handle and the second handle; the spatial pose change information of the first handle and the second handle from the target acquisition period to the current acquisition period is obtained, and the first handle and the second handle are tracked and positioned according to the initial spatial position information and the spatial pose change information; When tracking and positioning the second handle through the first depth camera, the handle tracking method includes: Determining a third relative spatial coordinate vector of the first handle and the second handle in the handle coordinate system according to the depth information of the second handle collected by the first depth camera; converting the third relative spatial coordinate vector into a fourth relative spatial coordinate vector of the first handle and the second handle in the world coordinate system through a preset second matrix conversion rule; calculating the spatial position information of the second handle based on the fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system.

2. The handle tracking method of claim 1, wherein, The step of tracking and positioning the first handle according to the first feature identification point in the image information includes: Detecting the head-mounted display spatial position information of the head-mounted display device; Determining a first relative position information between the head-mounted display device and the first handle according to the first feature identification point in the image information; Tracking and positioning the first handle according to the head-mounted display spatial position information and the first relative position information to obtain the spatial position information of the first handle.

3. The handle tracking method of claim 2, wherein, The head-mounted display device includes a head-mounted display body and a position sensor. The step of detecting the head-mounted display spatial position information of the head-mounted display device includes: Detecting the coordinate position vector of the head-mounted display body in the world coordinate system through the position sensor; Taking the coordinate position vector of the head-mounted display body in the world coordinate system as the head-mounted display spatial position information of the head-mounted display device; The step of determining the first relative position information between the head-mounted display device and the first handle according to the first feature identification point in the image information includes: Based on the first feature marker point in the image information, determine the first relative spatial coordinate vector between the first handle and the head-mounted display device in the head-mounted display coordinate system; The first relative spatial coordinate vector is used as the first relative position information between the head-mounted display device and the first handle.

4. The handle tracking method of claim 3, wherein, The step of tracking and locating the first controller based on the spatial position information of the head-mounted display and the first relative position information to obtain the spatial position information of the first controller includes: By using a preset first matrix transformation rule, the first relative spatial coordinate vector is converted into a second relative spatial coordinate vector between the first handle and the head-mounted display device in the world coordinate system; The second relative spatial coordinate vector and the coordinate position vector of the head-mounted display device in the world coordinate system are summed to obtain the coordinate position vector of the first handle in the world coordinate system. The coordinate position vector of the first handle in the world coordinate system is used as the spatial position information of the first handle.

5. The handle tracking method of claim 2, wherein, The step of tracking and locating the second handle using the first depth-sensing camera includes: The depth information of the second handle is acquired through the first depth-sensing camera; Based on the depth information of the second handle, determine the relative position information between the first handle and the second handle; Based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the controllers, the second controller is tracked and positioned to obtain the spatial position information of the second controller.

6. The handle tracking method of claim 5, wherein, The step of determining the relative position information between the first handle and the second handle based on the depth information of the second handle includes: Based on the depth information of the second handle, determine the third relative spatial coordinate vector between the first handle and the second handle in the handle coordinate system; The third relative spatial coordinate vector is used as the relative position information between the handles.

7. The handle tracking method of claim 6, wherein, The step of tracking and locating the second controller based on the spatial position information of the head-mounted display, the first relative position information, and the relative position information between the controllers to obtain the spatial position information of the second controller includes: By using a preset second matrix transformation rule, the third relative spatial coordinate vector is converted into a fourth relative spatial coordinate vector between the first handle and the second handle in the world coordinate system; Based on the spatial position information of the head-mounted display and the first relative position information, determine the coordinate position vector of the first handle in the world coordinate system; The fourth relative spatial coordinate vector and the coordinate position vector of the first handle in the world coordinate system are summed to obtain the coordinate position vector of the second handle in the world coordinate system. The coordinate position vector of the second handle in the world coordinate system is used as the spatial position information of the second handle.

8. An intelligent wearable device, characterized by, The smart wearable device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the handle tracking method according to any one of claims 1 to 7.

9. A readable storage medium, characterized by, The readable storage medium is a computer readable storage medium, and the computer readable storage medium stores a program for realizing the handle tracking method. The program for realizing the handle tracking method is executed by the processor to realize the steps of the handle tracking method in any one of claims 1 to 7.

Images