Method of gesture tracking and apparatus using same

Abstract

The invention provides a method of gesture tracking, a method of generating a stable keyframe for gesture tracking, a portable device and a head mounted display. A method of gesture tracking includes: receiving a stable keyframe from a head mounted display; receiving map data from the head mounted display; determining whether the stable key frame is available according to the information, wherein the information is obtained from the head-mounted display; and updating a map for gesture tracking according to the stable key frame and the map data in response to the stable key frame being available.

Description

Technical Field

[0001] The present disclosure relates to extended reality (XR) technology, and more particularly to a method of gesture tracking, a method of generating a stable key frame for gesture tracking, a portable device, and a head-mounted display (HMD). Background Art

[0002] In order to interact with the XR scene provided by the HMD, the user may operate a portable device (also referred to as a "tracker") that is capable of synchronizing a map with the HMD. According to current methods, the tracker may receive a map from the HMD and then reuse the map for tracking. However, current methods have some disadvantages, such as: the tracker must wait until the transmission is complete before reusing the map for relocalization or tracking; if the environment rarely changes, it is not efficient to wait for the transmission every time the tracker is connected to the HMD; the HMD may have created a new map, and therefore the map saved by the tracker is out of date; if a large amount of outdated map data is removed quickly, map data synchronization may hinder the tracker's tracking; if the difference between the HMD's map and the map saved by the tracker is large, there may be large pose drift; and the tracker may take a long time to load a large map. Summary of the invention

[0003] The present invention relates to a method for gesture tracking, a method for generating a stable key frame for gesture tracking, a portable device and an HMD.

[0004] The present invention relates to a method for gesture tracking, which is suitable for a portable device communicatively connected to a head-mounted display, and the method includes: receiving a stable key frame from the head-mounted display; receiving map data from the head-mounted display; determining whether the stable key frame is available based on information, wherein the information is obtained from the head-mounted display; and updating a map used for gesture tracking based on the stable key frame and the map data in response to the stable key frame being available.

[0005] In one embodiment of the present invention, the information includes a duration since the last connection between the portable device and the head-mounted display was established, and the step of determining whether a stable key frame is available based on the information includes: determining whether the duration is less than a threshold; and determining that the stable key frame is available in response to the duration being less than the threshold.

[0006] In one embodiment of the present invention, the information further includes a power off record of a head mounted display, and the step of determining whether a stable key frame is available based on the information further includes: determining whether the head mounted display is powered off during the duration based on the power off record; and determining that the stable key frame is available in response to the head mounted display not being powered off during the duration.

[0007] In one embodiment of the present invention, the information further includes a first map identification, and the step of determining whether a stable key frame is available based on the information further includes: determining whether the first map identification matches a second map identification of a map; and determining that the stable key frame is available in response to the first map identification matching the second map identification.

[0008] In one embodiment of the present invention, the method further comprises: in response to the stable keyframe being unavailable, updating the map without using the stable keyframe.

[0009] In one embodiment of the present invention, the step of updating a map for gesture tracking based on stable keyframes and map data includes: repositioning based on the stable keyframes and map data, and determining whether the repositioning is successful; and updating the map based on the stable keyframes and map data in response to successful repositioning.

[0010] In one embodiment of the present invention, the step of determining whether the relocalization is successful includes: obtaining a posture drift between a stable key frame and a current key frame included in the map data, wherein the current key frame corresponds to the stable key frame; determining whether the posture drift is less than a threshold; and determining that the relocalization is successful in response to the posture drift being less than the threshold.

[0011] In one embodiment of the present invention, the step of updating the map for gesture tracking according to the stabilized keyframes and the map data includes: updating the map based on a simultaneous localization and mapping (SLAM) algorithm.

[0012] The present invention relates to a method for generating stable key frames for gesture tracking, which is suitable for head-mounted displays. The method includes: capturing multiple key frames during a period of time, wherein the multiple key frames are spatially related to each other, wherein the multiple key frames include a first key frame; determining whether the first key frame is stable based on the multiple key frames; determining the first key frame as a first stable key frame in response to the first key frame being stable; and outputting the first stable key frame.

[0013] The present invention relates to a portable device, comprising a transceiver and a processor. The transceiver is communicatively connected to a head-mounted display. The processor is coupled to the transceiver, wherein the processor is configured to: receive a stable keyframe from the head-mounted display; receive map data from the head-mounted display; determine whether the stable keyframe is available based on information, wherein the information is obtained from the head-mounted display; and update a map for gesture tracking based on the stable keyframe and the map data in response to the stable keyframe being available.

[0014] The present invention relates to a head mounted display, comprising a transceiver and a processor. The processor is coupled to the transceiver, wherein the processor is configured to: capture a plurality of key frames during a period, wherein the plurality of key frames are spatially related to each other, wherein the plurality of key frames include a first key frame; determine whether the first key frame is stable according to the plurality of key frames; determine the first key frame as a first stable key frame in response to the first key frame being stable; and output the first stable key frame via the transceiver.

[0015] Based on the above description, the portable device of the present invention can reuse the saved map to enter six degrees of freedom (6-DOF) earlier when the environment changes little. If the map of the HMD changes too much, the portable device can no longer use the saved map. The portable device can keep tracking during map data synchronization. Therefore, the posture drift between the HMD and the portable device can be reduced, and the map loading time for the portable device can be reduced.

[0016] In order to make the foregoing content more understandable, several embodiments with accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and together with the embodiments serve to explain the principles of the present disclosure.

[0018] Figure 1 A schematic diagram showing an HMD system for gesture tracking according to an embodiment of the present invention.

[0019] Figure 2 A schematic diagram showing a tree structure of key frames according to an embodiment of the present invention.

[0020] Figure 3 A flow chart showing a method for gesture tracking according to an embodiment of the present invention is shown.

[0021] Figure 4 A flowchart of a method for generating stable key frames for gesture tracking according to an embodiment of the present invention is shown.

[0022] Figure 5 A flow chart showing a method for gesture tracking according to an embodiment of the present invention is shown.

[0023] Description of Reference Numerals

[0024] 10: Head mounted display system;

[0025] 100: Head-mounted display;

[0026] 110, 210: processor;

[0027] 120, 220: storage medium;

[0028] 130, 230: transceiver;

[0029] 140: image capturing device;

[0030] 200: Portable device;

[0031] 240: sensor;

[0032] F0, F1, F2, F3, F4, F5, F6, F7, F8, F9: key frames;

[0033] S201, S202, S301, S302, S303, S304, S305, S306, S307, S308, S309, S401, S402, S403, S404, S501, S502, S503, S504: steps. DETAILED DESCRIPTION

[0034] Figure 1 A schematic diagram of an HMD system 10 for gesture tracking according to an embodiment of the present invention is shown, wherein the HMD system 10 may be implemented in an XR system (e.g., a virtual reality (VR) system, an augmented reality (AR) system, or a mixed reality (MR) system). The HMD system 10 may include an HMD 100 and a portable device 200, wherein the portable device 200 may be, for example, a remote controller or a wireless controller of the HMD 100. The portable device 200 may be communicatively connected to the HMD 100.

[0035] The HMD 100 may include a processor 110, a storage medium 120, a transceiver 130, and an image capture device 140. The processor 110 may be, for example, a central processing unit (CPU) or other general or special programmable micro control unit (MCU), a microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), an arithmetic logic unit (ALU), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or other similar devices or combinations thereof. The processor 110 may be coupled to the storage medium 120, the transceiver 130, and the image capture device 140.

[0036] The storage medium 120 can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), or similar element, or a combination thereof, and the storage medium is configured to record multiple modules or various applications that can be executed by the processor 110.

[0037] The transceiver 130 may be configured to transmit or receive wired or wireless signals. The transceiver 130 may also perform operations such as low noise amplification, impedance matching, frequency mixing, up-conversion or down-conversion, filtering, amplification, etc. The HMD 100 may communicate with the portable device 200 via the transceiver 130 .

[0038] The image capture device 140 may be a camera or a photographic device for capturing images. The image capture device 140 may include an image sensor, such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor.

[0039] The portable device 200 may include a processor 210, a storage medium 220, a transceiver 230, and a sensor 240. The processor 210 may be, for example, a CPU or other programmable MCU, microprocessor, DSP, programmable controller, ASIC, GPU, ALU, CPLD, FPGA, or other similar device or combination of the above devices for general or special use. The processor 210 may be coupled to the storage medium 220, the transceiver 230, and the sensor 240.

[0040] The storage medium 220 may be, for example, any type of fixed or removable RAM, ROM, flash memory, HDD, SSD, or similar element or a combination thereof, configured to record a plurality of modules or various applications executable by the processor 210 .

[0041] The transceiver 230 may be configured to transmit or receive wired or wireless signals. The transceiver 230 may also perform operations such as low noise amplification, impedance matching, frequency mixing, up-conversion or down-conversion, filtering, amplification, etc. The portable device 200 may communicate with the HMD 100 via the transceiver 230 .

[0042] Sensor 240 may include, but is not limited to, an accelerometer or an image capture device. Sensor 240 may generate sensed data (eg, acceleration or image) for gesture tracking.

[0043] The HMD 100 may capture a plurality of images via the image capture device 140, and may generate map data based on the plurality of images, wherein the map data may be used to build a map. The map may be updated over time based on new map data generated by the HMD 100. In one embodiment, the HMD 100 may build or update a map based on a simultaneous localization and mapping (SLAM) algorithm.

[0044] The HMD 100 may generate one or more stable key frames for gesture tracking. The HMD 100 may generate the stable key frames according to the tree structure of the key frames obtained by the HMD 100. Figure 2 A schematic diagram of a tree structure of key frames according to an embodiment of the present invention is shown, wherein the tree structure may be composed of key frame F0, key frame F1, key frame F2, key frame F3, key frame F4, key frame F5, key frame F6, key frame F7, key frame F8, and key frame F9, but the present invention is not limited thereto. A key frame may be defined as a relationship between nodes of a tree structure and key frames. Two key frames connected by an arrow may be considered as adjacent key frames, wherein the arrow may contain information such as a weight between the two key frames (e.g., the higher the weight, the higher the similarity between the two key frames). For example, key frame F1 may be considered as an adjacent key frame of key frame F0. In one embodiment, Figure 2 The tree structure in may be generated by the HMD 100 based on a depth-first search algorithm.

[0045] In one embodiment, the HMD 100 may define the first key frame obtained by the HMD 100 as a reference key frame, wherein the reference key frame may be automatically defined as a stable key frame. For example, the HMD 100 may define the key frame F0 as a reference key frame. The HMD 100 may obtain one or more neighboring key frames (e.g., key frame F1 or key frame F2) of the reference key frame F0. Then, the HMD 100 may determine whether the neighboring key frames of the reference key frame F0 are stable key frames. If the neighboring key frames of the reference key frame F0 are greater than one, the HMD 100 may select one of the neighboring key frames of the reference key frame F0 based on, for example, a depth first search (DFS) algorithm. The HMD 100 may determine whether the selected neighboring key frame is a stable key frame at an earlier time point, and may determine whether the unselected neighboring key frame is a stable key frame at a later time point. For example, the HMD 100 may select the key frame F1 from the neighboring key frames of the reference key frame F0 based on the DFS algorithm. Therefore, the HMD 100 may determine whether the key frame F1 is a stable key frame at an earlier time point, and may determine whether the key frame F2 is a stable key frame at a later time point.

[0046] The HMD 100 may determine whether the key frame F1 is a stable key frame according to the stability and lifespan of the key frame F1. If the key frame F1 is stable and / or has a long lifespan, the HMD 100 may determine that the key frame F1 may be a stable key frame. Otherwise, the HMD 100 may regard the key frame F1 as a normal key frame. The HMD 100 may output one or more stable key frames to the portable device 200 via the transceiver 130.

[0047] Specifically, the HMD 100 may capture a plurality of key frames (including the key frame F1) during a period, wherein the plurality of key frames may be spatially related to each other (i.e., the key frames are key frames at different times in the same space). In one embodiment, the plurality of key frames may include one or more historical stable key frames (i.e., key frames used to update past maps). The HMD 100 may determine whether the key frame F1 among the plurality of key frames is stable based on the plurality of key frames.

[0048] In one embodiment, the HMD 100 may capture a plurality of features from the key frame F1 and a plurality of features from a specific key frame among the plurality of key frames, wherein the plurality of key frames (including the key frame F1) may be spatially related to each other. The HMD 100 may detect a plurality of matching features among the plurality of features of the key frame F1 and the plurality of features of the specific key frame. If the number of the plurality of matching features is greater than a threshold value, the HMD 100 may determine that the specific key frame is similar to the key frame F1. If the number of key frames similar to the key frame F1 among the plurality of key frames (or the number of historically stable key frames) is greater than a threshold period, the HMD 100 may determine that the key frame F1 is stable.

[0049] In one embodiment, the HMD 100 may determine that the key frame F1 is stable according to the number of bundle adjustments (BA) of the key frame F1. Specifically, the HMD 100 may perform a sufficient number of BAs on the key frame F1 (i.e., the number of BAs is greater than a threshold) to obtain the posture drift (or re-projection error) of the key frame F1. If the posture drift is less than the threshold, the HMD 100 may determine that the key frame F1 is stable.

[0050] In one embodiment, if the age of key frame F1 (ie, the period of time since key frame F1 was created or captured by HMD 100) is greater than a threshold, HMD 100 may determine that key frame F1 is a stable key frame.

[0051] If the HMD 100 determines that the key frame F1 is not a stable key frame, the HMD 100 may stop searching for adjacent key frames of the key frame F1, and may determine whether other adjacent key frames (e.g., key frame F2) that reference the key frame F0 are stable key frames. On the other hand, if the HMD 100 determines that the key frame F1 is a stable key frame, the HMD 100 may select one of the adjacent key frames of the key frame F1 (e.g., key frame F3 or key frame F4) based on, for example, the DFS algorithm. Thereafter, the HMD 100 may determine whether the selected key frame is a stable key frame. If the selected key frame is stable and / or has a long lifespan, the HMD 100 may determine that the selected key frame is a stable key frame. Otherwise, the HMD 100 may regard the selected key frame as a normal key frame.

[0052] For example, in Figure 2, the HMD 100 may determine that key frames F0, F1, F2, F3, F4, and F7 are stable key frames, and may determine that key frames F5, F6, F8, and F9 are normal key frames. Since key frames F5, F6, F8, and F9 are not stable key frames, the HMD 100 may stop searching for adjacent key frames of key frames F5, F6, F8, or F9. After determining the stable key frame, the HMD 100 may transmit the stable key frame to the portable device 200 via the transceiver 130.

[0053] The portable device 200 may receive one or more stabilization key frames from the HMD 100 via the transceiver 230. The portable device 200 may store the stabilization key frames in the storage medium 200, and may perform gesture tracking according to the stabilization key frames. Figure 3 A flowchart of a posture tracking method according to an embodiment of the present invention is shown, wherein Figure 3 The method can be obtained by Figure 1 The portable device 200 shown in FIG.

[0054] In step S201, the portable device 200 may receive information from the HMD 100 via the transceiver 230, wherein the information may indicate whether a stable key frame stored in the storage medium 220 is available. In step S202, the portable device 200 may keep receiving map data from the HMD 100 via the transceiver 230. In one embodiment, the portable device 200 may receive a completed map created by the HMD 100 from the HMD 100. The data received from the HMD 100 may be stored in the storage medium 220 by the portable device 200.

[0055] In step S301, the portable device 200 may load the stored stable key frame from the storage medium 120. In step S302, the portable device 200 may determine whether the stable key frame is available. If the stable key frame is available, proceed to step S306. If the stable key frame is not available, proceed to step S303.

[0056] The portable device 200 may determine whether a stable key frame is available based on information received from the HMD 100. In one embodiment, the information may include a duration since the last connection between the portable device 200 and the HMD 100 was established. If the duration is less than a threshold value, the portable device 200 may determine that a stable key frame is available. If the duration is greater than or equal to the threshold value, the portable device 200 may determine that a stable key frame is not available. That is, if the portable device 200 is not communicatively connected to the HMD 100 for a long time, the portable device 200 may determine that a stable key frame is not available.

[0057] In one embodiment, the information may include a power-off record of the HMD 100. The portable device 200 may determine whether the HMD 100 has been powered off during the duration (e.g., the duration since the last connection between the portable device 200 and the HMD 100 was established) from the power-off record. If the HMD 100 has been powered off during the duration, the portable device 200 may determine that the stable key frame is not available. If the HMD 100 has never been powered off during the duration, the portable device 200 may determine that the stable key frame is available.

[0058] In one embodiment, the information may include a map recognition, wherein the map recognition belongs to a map corresponding to a specific space. Since the HMD 100 can create different maps for different spaces, different map recognitions can correspond to different spaces, respectively. The portable device 200 can determine whether the map recognition in the information matches the map recognition corresponding to the map currently used by the portable device 200 (e.g., a map created and updated by the portable device 200). If the map recognition in the information matches the map recognition corresponding to the map currently used by the portable device 200, the portable device 200 can determine that the HMD 100 remains in the same space. Therefore, the portable device 200 can determine that a stable key frame is available. If the map recognition in the information does not match the map recognition corresponding to the map currently used by the portable device 200, the portable device 200 can determine that the HMD 100 moves to another space. Therefore, the portable device 200 can determine that a stable key frame is not available.

[0059] In step S303, the portable device 200 may clear the stable key frame stored in the storage medium 220. In step S304, the portable device 200 may load the received map. In step S305, the portable device 200 may reposition according to the received map.

[0060] In step S306, the portable device 200 may be repositioned according to the stable key frame and the map data. Thereafter, in step S307, the portable device 200 may determine whether the repositioning is successful. If the repositioning is successful, then proceed to step S308. If the repositioning is unsuccessful, then proceed to step S303.

[0061] In one embodiment, the portable device 200 may determine whether the relocation is successful based on an available stable key frame and a current key frame included in the map data, wherein the current key frame corresponds to the available stable key frame (e.g., the current key frame and the available stable key frame are spatially related to each other). Specifically, the portable device 200 may obtain a posture drift between the stable key frame and the current key frame. If the posture drift between the stable key frame and the current key frame is less than a threshold value, the portable device 200 may determine that the relocation is successful. If the posture drift between the stable key frame and the current key frame is greater than or equal to the threshold value, the portable device 200 may determine that the relocation is unsuccessful.

[0062] In step S308, the portable device 200 may update the map for gesture tracking according to the map data. The portable device 200 may update the map based on the SLAM algorithm. In one embodiment, if a stable keyframe is available, the portable device 200 may update the map for gesture tracking according to the map data and the stable keyframe. In one embodiment, if a stable keyframe is not available, the portable device 200 may update the map for gesture tracking according to the map data without a stable keyframe.

[0063] In step S309 , the portable device 200 may save the updated map in the storage medium 220 .

[0064] Figure 4 A flowchart of a method for generating a stable key frame for gesture tracking according to an embodiment of the present invention is shown, wherein Figure 4 The method in can be obtained by Figure 1 The method is implemented in the HMD 100 shown in FIG. In step S401, a plurality of key frames are captured during a period of time, wherein the plurality of key frames are spatially related to each other, wherein the plurality of key frames include a first key frame. In step S402, whether the first key frame is stable is determined according to the plurality of key frames. In step S403, in response to the first key frame being stable, the first key frame is determined as a first stable key frame. In step S404, the first stable key frame is output.

[0065] Figure 5 A flowchart showing a method for gesture tracking according to an embodiment of the present invention is shown, wherein Figure 5 The method in can be obtained by Figure 1 The method is implemented by the portable device 200 shown in FIG. In step S501, a stable keyframe is received from a head mounted display. In step S502, map data is received from the head mounted display. In step S503, it is determined whether a stable keyframe is available according to information, wherein the information is obtained from the head mounted display. In step S504, in response to the stable keyframe being available, a map for gesture tracking is updated according to the stable keyframe and the map data.

[0066] In summary, the HMD of the present invention can determine a stable key frame from the obtained key frames according to the stability and lifespan of the key frame. A stable key frame with a long lifespan can be regarded as an important key frame and thus determined as a stable key frame. The HMD can transmit the stable key frame to the portable device while transmitting the map data to the portable device. The portable device can determine whether the stable key frame is available according to a predetermined rule. If the stable key frame is available, the portable device can update the map for posture tracking according to the stable key frame. Therefore, the posture drift between the HMD and the portable device can be reduced, and the map loading time for the portable device can be reduced.

[0067] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, it is intended that the present disclosure encompasses modifications and variations as long as the modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A method of gesture tracking, adapted for use with a portable device communicatively connected to a head mounted display, characterized in that: The method comprises: receiving a stabilization keyframe from the head mounted display; receiving map data from the head mounted display; determining whether the stable keyframe is available based on information, wherein the information is obtained from the head mounted display; and In response to the stable keyframe being available, a map for the gesture tracking is updated based on the stable keyframe and the map data.

2. The method of claim 1 , wherein the information includes a duration since a last connection between the portable device and the head mounted display was established, and the step of determining whether the stable key frame is available based on the information includes: determining whether the duration is less than a threshold; as well as The stable key frame is determined to be available in response to the duration being less than the threshold.

3. The method according to claim 2, wherein the information further comprises a power-off record of the head mounted display, and the step of determining whether the stable key frame is available according to the information further comprises: determining, according to the power-off record, whether the head mounted display is powered off during the duration; as well as The stabilization keyframe is determined to be available in response to the head mounted display not being powered off during the duration.

4. The method of claim 3, wherein the information further comprises a first map identification, and the step of determining whether the stable key frame is available based on the information further comprises: determining whether the first map identification matches a second map identification of the map; as well as The stable keyframe is determined to be available in response to the first map recognition matching the second map recognition.

5. The method according to claim 1, further comprising: In response to the stable keyframe being unavailable, the map is updated without using the stable keyframe.

6. The method of claim 1 , wherein the step of updating the map for the gesture tracking according to the stabilized keyframe and the map data comprises: Repositioning according to the stable key frame and the map data, and determining whether the repositioning is successful; as well as In response to the relocalization being successful, the map is updated based on the stabilized keyframe and the map data.

7. The method of claim 6, wherein the step of determining whether the repositioning is successful comprises: obtaining a pose drift between the stable key frame and a current key frame included in the map data, wherein the current key frame corresponds to the stable key frame; determining whether the posture drift is less than a threshold; as well as The repositioning is determined to be successful in response to the posture drift being less than the threshold.

8. The method of claim 1 , wherein the step of updating the map for the gesture tracking according to the stabilized keyframe and the map data comprises: The map is updated based on a Simultaneous Localization and Mapping (SLAM) algorithm.

9. A method for generating stable keyframes for posture tracking, suitable for head mounted display, characterized in that: The method comprises: capturing a plurality of key frames during a time period, wherein the plurality of key frames are spatially related to one another, wherein the plurality of key frames includes a first key frame; Determining whether the first key frame is stable according to the multiple key frames; determining the first key frame as a first stable key frame in response to the first key frame being stable; and The first stable key frame is output.

10. A portable device, characterized in that: include: a transceiver communicatively coupled to the head mounted display; as well as a processor coupled to the transceiver, wherein the processor is configured to: receiving a stabilization keyframe from the head mounted display; receiving map data from the head mounted display; determining whether the stable keyframe is available based on information, wherein the information is obtained from the head mounted display; as well as In response to the stable keyframe being available, a map for the gesture tracking is updated based on the stable keyframe and the map data.

11. A head mounted display, characterized in that: include: Transceiver; as well as a processor coupled to the transceiver, wherein the processor is configured to: capturing a plurality of key frames during a time period, wherein the plurality of key frames are spatially related to one another, wherein the plurality of key frames includes a first key frame; Determining whether the first key frame is stable according to the multiple key frames; determining the first key frame as a first stable key frame in response to the first key frame being stable; as well as The first stable key frame is output via the transceiver.

Images