Hand tracking device, system and method

Abstract

The invention provides a hand tracking device. A hand tracking device includes a memory circuit and a processor. The storage circuit stores program code. The processor is coupled to the memory circuit and accesses the program code. The processor is configured to obtain a hand motion profile of a hand over a period of time. The processor is configured to obtain an object motion curve of the object over the time period. The processor is configured to determine whether the hand grips the object based on the hand motion curve and the object motion curve.

Description

Technical Field

[0001] The present disclosure relates to a hand tracking device; in particular, the present disclosure relates to a hand tracking device, a hand tracking system and a hand tracking method. Background Art

[0002] In order to bring an immersive experience to users, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), are constantly being developed. AR technology allows users to bring virtual elements into the real world. VR technology allows users to enter a whole new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to bring a completely immersive experience to users, visual content, audio content, or other sensory content can be provided through one or more devices. Summary of the invention

[0003] The present disclosure relates to a hand tracking device, a hand tracking system and a hand tracking method, so as to improve the accuracy of hand tracking when a hand holds an object.

[0004] In the present disclosure, a hand tracking device is provided. The hand tracking device includes a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code. The processor is configured to obtain a hand motion curve of a hand within a time period. The processor is configured to obtain an object motion curve of an object within the time period. The processor is configured to determine whether the hand is holding an object based on the hand motion curve and the object motion curve.

[0005] In the present disclosure, a hand tracking system is provided. The hand tracking system includes a camera, a tracker, a storage circuit, and a processor. The camera is configured to provide image data. The tracker is configured to provide tracker data. The tracker is suitable for being placed on an object. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to perform the following operations: receiving image data from the camera; determining a hand motion curve of a hand within a time period based on the image data; receiving tracker data from the tracker; determining an object motion curve of an object within the time period based on the tracker data; and determining whether the hand is holding the object based on the hand motion curve and the object motion curve.

[0006] In the present disclosure, a hand tracking method is provided, which comprises: obtaining a hand motion curve of a hand within a time period; obtaining an object motion curve of an object within the time period; and determining whether the hand is holding the object based on the hand motion curve and the object motion curve.

[0007] Based on the above, according to a hand tracking device, a hand tracking system and a hand tracking method, the accuracy of hand tracking when a hand holds an object is improved.

[0008] In order to make the foregoing content more understandable, several embodiments accompanying the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are included as follows to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0010] Figure 1A Schematic diagram of a hand tracking device according to an embodiment of the present disclosure.

[0011] Figure 1B is a schematic diagram of a hand tracking system according to an embodiment of the present disclosure.

[0012] Figure 2 is a schematic flowchart of a hand tracking method according to an embodiment of the present disclosure.

[0013] Figure 3A Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0014] Figure 3B Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0015] Figure 4A Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0016] Figure 4B Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0017] Figure 5 Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0018] Figure 6 Schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure.

[0019] Description of Figure Numbers

[0020] 100A: Hand tracking device;

[0021] 100B: Hand tracking system;

[0022] 200: Hand tracking method;

[0023] 300A, 300B, 400A, 400B, 500, 600: hand tracking scenarios;

[0024] 301, 302, 401, 402, 601, 602: situation;

[0025] BLT: belt;

[0026] CMR: Camera;

[0027] H: hand;

[0028] HDD: HMD device;

[0029] MSG: prompt message;

[0030] O1, O2, O3, OBJ: objects;

[0031] PCS: processor;

[0032] S210, S220, S230: steps;

[0033] SC: storage circuit;

[0034] SNR: sensor;

[0035] T_H, T_O, T_O1, T_O2, T_O3, T_T, T_U: trajectory;

[0036] t1, t2: time;

[0037] TRK: Tracker;

[0038] U: user. DETAILED DESCRIPTION

[0039] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like components.

[0040] Throughout the specification and claims of this disclosure, specific terms are used to refer to specific components. It will be understood by those skilled in the art that electronic device manufacturers may refer to the same components by different names. It is not intended herein to distinguish between components that have the same function but have different names. In the following description and claims, words such as "including" and "comprising" are open-ended terms and should be interpreted as "including but not limited to...".

[0041] The term "coupled (or connected)" used in the entire specification of the present application (including the attached claims) may refer to any direct or indirect connection member. For example, if the text describes that the first device is coupled (or connected) to the second device, it should be interpreted that the first device can be directly connected to the second device, or the first device can be indirectly connected to the second device through other devices or some connecting members. The terms "first", "second" and similar terms mentioned in the entire specification of the present application (including the attached claims) are only used to name discrete elements or to distinguish between different embodiments or scopes. Therefore, the terms should not be regarded as limiting the upper or lower limit of the number of elements and should not be used to limit the arrangement order of elements. In addition, where possible, the elements / components / steps with the same figure numbers in the drawings and embodiments represent the same or similar parts. Using the same figure numbers or using the same terms in different embodiments, the relevant descriptions of the elements / components / steps can be referenced to each other.

[0042] It should be noted that in the following embodiments, without departing from the spirit of the present disclosure, the technical features of several different embodiments may be replaced, reorganized, and mixed to complete other embodiments. As long as the features of each embodiment do not violate the spirit of the present disclosure or conflict with each other, the features may be mixed and used together at will.

[0043] In order to bring an immersive experience to users, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), are constantly being developed. AR technology allows users to bring virtual elements into the real world. VR technology allows users to enter a whole new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to bring a completely immersive experience to users, visual content, audio content, or other sensory content can be provided through one or more devices.

[0044] Hand tracking is often used to increase the immersive experience. For example, hand tracking may not require a controller, allowing users to interact with the virtual world in a more natural and intuitive way, just as if they were in the real world. However, when hand tracking is involved when grabbing objects with hands, some challenges may arise. For example, the camera may misinterpret shadows, reflections, or even parts of an object as the user's body, triggering unexpected interactions or causing the virtual environment to malfunction. This may cause confusion and disruption to the user flow. In addition, the camera field of view (FOV) of most VR headsets is limited, making certain areas around the user invisible to the system. This can lead to inconvenient situations, such as the user's hand disappearing in a blind spot or objects suddenly appearing out of thin air. In addition, there are a large number of different types of objects in the world, and their appearance varies greatly. This makes it difficult to accurately identify the type of object using images. Therefore, it is a pursuit of technicians in this field to perform hand tracking in a more natural and intuitive way when objects are involved.

[0045] Figure 1A FIG. 1 is a schematic diagram of a hand tracking device according to an embodiment of the present disclosure. Figure 1A , a hand tracking device 100A is shown. In different embodiments, the hand tracking device 100A can be implemented as various smart devices and / or computer devices, but is not limited thereto.

[0046] exist Figure 1A In the hand tracking device 100A, a storage circuit SC and a processor PCS are included. The storage circuit SC is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, other similar devices, or a combination of these devices, and can be used to record multiple program codes or modules.

[0047] The processor PCS is coupled to the storage circuit SC and can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), any other kind of integrated circuit, a state machine, a processor based on an advanced RISC machine (ARM), etc.

[0048] In an embodiment of the present invention, the processor PCS can access the modules and program codes recorded in the storage circuit SC to implement the hand tracking method proposed in the present disclosure, and the details thereof are described below.

[0049] Figure 1B Schematic diagram of a hand tracking system according to an embodiment of the present disclosure. Figure 1B , the hand tracking system 100B may include a camera CMR, a tracker TRK, a storage circuit SC, and a processor PCS. In one embodiment, the camera CMR, the storage circuit SC, and / or the processor PCS may be integrated into a single electronic device (e.g., the hand tracking device 100A) or separated from each other, but are not limited to this. In one embodiment, the camera CMR may be included in a head-mounted display (HMD) device, and the tracker TRK may be attached to an object or included in an object. In another embodiment, the camera CMR may be placed somewhere other than the user's head to capture an image of the user. However, the present disclosure is not limited to this. In addition, for ease of explanation, in the following embodiments, although the HMD device is illustrated as being worn by the user, it is worth noting that the user does not necessarily have to wear the HMD device.

[0050] In one embodiment, the camera CMR may include, for example, a complementary metal oxide semiconductor (CMOS) camera or a charge coupled device (CCD) camera. In one embodiment, the camera 110 may be mounted on an HMD device, wearable glasses (e.g., AR / VR goggles), an electronic device, other similar devices, or a combination of these devices. However, the present disclosure is not limited thereto.

[0051] In one embodiment, the HMD device may be configured to display AR, VR, MR, or XR content. The HMD device may include, for example, a head-mounted device, wearable glasses (e.g., AR / VR goggles), other similar devices suitable for AR, VR, MR, XR, or other reality-related technologies, or a combination of these devices. However, the present disclosure is not limited thereto.

[0052] In one embodiment, the tracker TRK may include, for example, a gyroscope, an accelerometer, an inertial measurement unit (IMU) sensor, other similar devices, or a combination of these devices. In addition, the tracker TRK may be associated with the object OBJ to provide information of the object OBJ. For example, the tracker TRK may be configured to provide three linear acceleration values ​​(e.g., position) and / or three angular velocities (e.g., rotation) associated with three axes. However, the present disclosure is not limited thereto.

[0053] In one embodiment, each component in the hand tracking system 100B may include a communication circuit, and the communication circuit may include, for example, a wired network module, a wireless network module, a Bluetooth module, an infrared module, a radio frequency identification (RFID) module, a Zigbee network module, or a near field communication (NFC) network module, but the present disclosure is not limited thereto. That is, each component in the hand tracking system 100B may communicate with each other through wired communication or wireless communication.

[0054] In one embodiment, the processor PCS may transmit data to and receive data from the camera CMR, the tracker TRK, and the storage circuit SC. In addition, the processor PCS may access the program code recorded in the storage circuit SC to implement the hand tracking method proposed in the present disclosure, and the details thereof are described below.

[0055] Figure 2 FIG. 1 is a schematic flow chart of a hand tracking method according to an embodiment of the present disclosure. Figure 1A , Figure 1B as well as Figure 2 , the hand tracking method 200 may be performed by Figure 1A The hand tracking device 100A or Figure 1B The hand tracking system 100B in the embodiment of the present invention is implemented and the following is combined with Figure 1A or Figure 1B The component description shown in Figure 2 Details of each step in the.

[0056] First, in step S210, the processor PCS may be configured to obtain a hand motion curve of the hand within a time period. Then, in step S220, the processor PCS may be configured to obtain an object motion curve of the object within the time period. Then, in step S230, the processor PCS may be configured to determine whether the hand is holding the object based on the hand motion curve and the object motion curve.

[0057] In this way, based on the hand motion curve and the object motion curve, it can be determined whether the hand is holding the object. In other words, the hand holding algorithm can be performed based on the hand motion curve and the object motion curve. Therefore, when objects are involved, hand tracking can still be performed in a natural and intuitive way, thereby giving users a fully immersive experience.

[0058] In addition, the implementation details of the hand tracking method 200 may refer to the following description to obtain sufficient teachings, suggestions and implementation examples, and the details are not redundantly described herein.

[0059] Figure 3A FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 3A , the hand tracking scenario 300A may include a hand H, an object OBJ, and a tracker TRK. In the hand tracking scenario 300A, the hand H may hold the object OBJ, and the tracker TRK may be placed on (eg, included in or attached to) the object OBJ.

[0060] In one embodiment, during a period of time (from time t1 to time t2), the user may move the hand H. Figure 3A As shown in , the trajectory T_H may represent the movement trajectory of the hand H during the time period. In addition, since the object OBJ is held by the hand H, the trajectory T_O may represent the movement trajectory of the object OBJ during the time period, and the trajectory T_O may be similar to the trajectory T_H. In addition, since the tracker TRK is placed on the object OBJ, the trajectory T_T may represent the movement trajectory of the tracker TRK during the time period, and the trajectory T_T may be similar to the trajectory T_O. In one embodiment, the trajectory T_H may also be referred to as a hand motion curve, and T_O may also be referred to as an object motion curve.

[0061] It is worth mentioning that when the hand H holds the object OBJ, the object OBJ may also move similarly regardless of how the hand H moves. In addition, since the tracker TRK is placed on the object OBJ, the tracker data of the tracker TRK can be used not only to represent the movement of the tracker TRK, but also to represent the movement of OBJ. Therefore, by obtaining the tracker data from the tracker TRK, the trajectory T_T can be determined. At the same time, based on the position where the tracker TRK is placed on the object OBJ, the trajectory T_T can be converted into the trajectory T_O. In addition, by comparing the trajectory T_O with the trajectory T_H, the similarity between the two trajectories can be obtained. In one embodiment, the similarity can be determined based on a curve fitting algorithm, but is not limited to this. That is, the processor PCS can be configured to determine the similarity between the trajectory T_H and the trajectory T_O. Moreover, in response to the similarity being higher than a predetermined threshold, the processor PCS can be configured to determine that the hand H holds the object OBJ. On the other hand, in response to the similarity being not higher than a predetermined threshold, the processor PCS can be configured to determine that the hand H does not hold the object OBJ. Therefore, when objects OBJ are involved, hand tracking of the hand H can still be performed in a natural and intuitive manner, thereby bringing a fully immersive experience to the user.

[0062] In addition, while tracking the user's hand H (e.g., left hand), the user's other hand (e.g., right hand) may be tracked at the same time. In addition, the user may hold the object OBJ with one of the hand H and the other hand. However, since the hand H and the other hand may look similar, the hand tracking algorithm may not be confused or distinguish between the hand H and the other hand. That is, based only on the hand tracking algorithm itself, the hand tracking algorithm may sometimes confuse which hand holds the object OBJ. It is worth noting that, regardless of whether the object OBJ is held by the hand H or the other hand, the hand holding the object OBJ may have a trajectory similar to the trajectory T_O. Therefore, by comparing the trajectory T_O, the trajectory T_H, and the trajectory of the other hand, it can be determined which hand holds the object OBJ. In other words, the processor PCS may be configured to obtain the other hand motion curve of the other hand within the time period. In addition, the processor PCS may be configured to determine whether the hand or the other hand holds the object OBJ based on the hand motion curve, the other hand motion curve, and the object motion curve. Therefore, it is easy to distinguish between the hand H and the other hand, thereby increasing the user experience.

[0063] In one embodiment, the trajectory T_H can be generated based on the image data provided from the camera CMR, and the trajectory T_O can be generated based on the tracker data provided from the tracker TRK. For example, the image data may include an image of the user's hand H and / or another hand. In addition, the tracker data may include acceleration values ​​and / or angular velocities associated with three axes (e.g., X-axis, Y-axis, and Z-axis) of the object OBJ. However, the present disclosure is not limited to this. That is, the processor PCS may be configured to receive image data from the camera CMR. In addition, the processor PCS may be configured to determine a hand motion curve based on the image data. In addition, the processor PCS may be configured to receive tracker data from the tracker TRK. The tracker TRK may be suitable for being placed on or attached to the object OBJ. For example, based on the size or weight of the tracker TRK, the tracker TRK may be mounted on the object OBJ using bolts, screws, strips, tapes, etc. Moreover, the processor PCS may be configured to determine the object motion curve based on the tracker data. Therefore, when objects OBJ are involved, hand tracking of the hand H can still be performed in a natural and intuitive manner, thereby bringing a fully immersive experience to the user.

[0064] Figure 3B FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 3B , the hand tracking scenario 300B may include scenario 301 and scenario 302. It is worth noting that for the sake of clarity, the Figure 3B For a better understanding of the present disclosure. For example, although not shown, the tracker TRK may be placed on the object OBJ. However, the present disclosure is not limited thereto.

[0065] First, refer to scenario 301. In one embodiment, the hand H may not hold the object OBJ. In other words, the holding state of the hand H may be not holding anything. Based on the holding state, the virtual hand and virtual object of the user's corresponding avatar may be displayed in the virtual world. That is, the processor PCS may be configured to: display the virtual hand and virtual object in the virtual world. In addition, the processor PCS may be configured to: in response to determining that the hand H does not hold the object OBJ, display the virtual hand not holding the virtual object.

[0066] Now refer to scenario 302. In one embodiment, the hand H may hold the object OBJ. In other words, the holding state of the hand H may be holding the object OBJ. Based on the holding state, the virtual hand and virtual object of the user's corresponding avatar may be displayed in the virtual world. That is, the processor PCS may be configured to: display the virtual hand and virtual object in the virtual world. In addition, the processor PCS may be configured to: in response to determining that the hand H is holding the object OBJ, display the virtual hand holding the virtual object.

[0067] It is worth mentioning that based on Figure 3A The track T_H and track T_O described in the figure are used to determine the holding state of the hand H. Therefore, the hand H and the object OBJ can be correctly tracked, and the virtual hand and the virtual object can be realistically displayed in the virtual world, thereby bringing a fully immersive experience to the user.

[0068] In one embodiment, hand tracking of hand H can be performed based on trajectory T_H, and object tracking of object OBJ can be performed based on trajectory T_O. In addition, based on the hand tracking results of hand tracking and the object tracking results of object tracking, the virtual hand and virtual object of the user's corresponding avatar can be displayed in the virtual world. That is, the processor PCS can be configured to display the virtual hand and virtual object in the virtual world. In addition, the processor PCS can be configured to move the virtual hand based on the hand motion curve (using the hand tracking algorithm). In addition, the processor PCS can be configured to move the virtual object based on the object motion curve (using the object tracking algorithm).

[0069] Figure 4A FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 4A, the hand tracking scenario 400A may include a user U, an object O1, an object O2, and an object O3. The user U may wear an HMD device HDD on the head and a belt BLT around the waist. The object O1 may be held by the hand H of the user U, and the objects O2 and O3 may be suspended from the belt BLT. In one embodiment, the object O1 may also be referred to as an object OBJ, and the object O2 or the object O3 may be referred to as an additional object. However, the present disclosure is not limited thereto. In one embodiment, the object O1, the object O2, or the object O3 may also be referred to as an object OBJ. It is worth noting that for the sake of clarity, the Figure 4A For a better understanding of the present disclosure. For example, although not shown, an additional tracker may be placed on each of the object O2 and the object O3. However, the present disclosure is not limited thereto.

[0070] In one embodiment, within a time period (from time t1 to time t2), the user U may move from one position to another, and the trajectory T_U may represent the movement trajectory of the user within the time period. In addition, the trajectory T_O2 and the trajectory T_O3 may represent the movement trajectory of the object O2 and the object O3 within the time period, respectively. The trajectory T_O2 or the trajectory T_O3 may also be referred to as an additional object motion curve. Since the object O2 and the object O3 are suspended on the belt BLT, the trajectory T_O2 and the trajectory T_O3 of the object O2 may be similar to the trajectory T_U. It is worth mentioning that within the time period, the user U may move the object O1 (for example, raise the object O1), and the trajectory T_O1 may represent the movement trajectory of the object O1 within the time period. That is, the trajectory T_O1 may not be similar to the trajectory T_U, the trajectory O2 and / or the trajectory O3. In other words, the processor PCS may be configured to obtain the user motion curve of the user U within the time period. In addition, the processor PCS may be configured to obtain the additional object motion curve of the additional object within the time period. The additional object is placed on the user. Furthermore, the processor PCS may be configured to: in response to the user being walking, the user motion curve being similar to the additional object motion curve, and the user motion curve being not similar to the object motion curve, determine that the object O1 is held by the hand H and the additional object is not held by the hand H. Therefore, it is possible to clearly identify which object is held by the user U, thereby increasing the user experience.

[0071] In one embodiment, when the tracker TRK is placed on the object O1, object O2, or object O3, a pairing operation between the tracker TRK and the object O1, object O2, or object O3 may be performed. By performing the pairing operation, the details of the object O1, object O2, or object O3 may be obtained and stored in the tracker TRK or the storage circuit SC. The information may be stored as pairing information, which may include the object type, object 3D model, shape, size, color, function, etc. of the object O1, object O2, or object O3 (also referred to as object OBJ). That is, the processor PCS may be configured to pair the tracker TRK with the object OBJ. In addition, the processor PCS may be configured to generate pairing information in response to the pairing of the tracker TRK with the object OBJ. The pairing information may include the object type of the object OBJ. In addition, the processor PCS may be configured to determine the object type of the object OBJ based on the pairing information. Therefore, based on the pairing information, the object type of the object OBJ may be obtained, and a virtual object corresponding to the object type may be correctly displayed in the virtual world. For example, when the object type is determined to be a baseball, a virtual baseball bat may be displayed in the virtual world. In addition, the object 3D model of the object OBJ may be used to allow the virtual baseball bat to faithfully present the shape and size of the object OBJ. However, the present disclosure is not limited thereto. In this way, the user experience may be enhanced.

[0072] Figure 4B FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 4B , the hand tracking scenario 400B may include scenario 401 and scenario 402. It is worth noting that for the sake of clarity, the Figure 4B For a better understanding of the present disclosure. However, the present disclosure is not limited thereto.

[0073] First, refer to scenario 401. In one embodiment, hand H may hold object O2 instead of object O1. Similar to object O1, a tracker TRK may be placed on object O2. The tracker TRK placed on object O2 may also be referred to as an additional tracker. In some cases, user U may need to grab one of object O1, object O2, and object O3. For example, object O1 may be a stick (e.g., a baseball bat), object O2 may be a flashlight, and object O3 may be a pistol. During a police training course, user U may need to grab the correct object according to instructions. For example, user U may need to grab object O1. However, user U may grab object O2 instead. That is, user U may perform an erroneous action that is different from the instructions.

[0074] Now refer to scenario 402. When the user U fails to follow the instructions correctly, a prompt message MSG may be generated and provided to the user U. In one embodiment, the prompt message MSG may be a visual prompt or an audio prompt. However, the present disclosure is not limited to this. In one embodiment, the prompt message MSG may be: "You are holding the wrong object. Please put the object O2 back and hold the object O1." In another embodiment, the prompt message MSG may be: "You put the flashlight in the wrong position. Please move the flashlight to the side of your head and aim the light forward." That is, the prompt message MSG may inform the user U that something is wrong and guide the user how to correct it. In other words, the processor PCS may be configured to: in response to determining that the hand H holds the object OBJ, provide the prompt message MSG to the user U. Therefore, the user U can follow the prompt message MSG and perform better during training or competition, thereby increasing the user experience.

[0075] Figure 5 FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 5 , the hand tracking scenario 500 may include a user U and an object OBJ. The user U may wear an HMD device HDD on the head. The object OBJ may be held by the hand H of the user U, and the tracker TRK may be placed on the object OBJ.

[0076] In one embodiment, the camera CMR may be included in the HMD device HDD. When the user U wears the HMD device HDD on the head, the FOV of the camera CMR may be directly in front of the user U. On the other hand, the blind spot of the camera CMR may be behind the user U. It is worth noting that when the hand H is visible in the FOV, hand tracking of the hand H can be performed based on the image data of the camera CMR. For example, the trajectory T_H of the hand H can be determined based on the image data, and hand tracking can be performed based on the trajectory T_H. However, when the hand H disappears from the FOV, hand tracking may not be performed based on the image data.

[0077] It is worth mentioning that since the object OBJ is placed on the hand H, the trajectory T_O of the object OBJ may be similar to the trajectory T_H of the hand H. That is, based on the relationship between the object OBJ and the hand H, the trajectory T_O of the object OBJ can be directly used as the trajectory T_H of the hand H, or the trajectory T_O of the object OBJ can be converted into the trajectory T_H of the hand H. In other words, the processor PCS can be configured to: in response to the hand H being in the FOV of the camera CMR, perform hand tracking of the hand H based on the hand motion curve. In addition, the processor PCS can be configured to: in response to the hand H not being in the FOV of the camera CMR, perform hand tracking of the hand H based on the object motion curve. Therefore, even if the hand H is not in the FOV of the camera CMR, hand tracking can still be performed correctly, thereby increasing the user experience.

[0078] Figure 6 FIG. 1 is a schematic diagram of a hand tracking scenario according to an embodiment of the present disclosure. Figure 6 , the hand tracking scenario 600 may include a scenario 601 and a scenario 602. The hand tracking scenario 601 may include a hand H and an object OBJ. The user U may wear a belt BLT around the waist. The object OBJ may be placed on the BLT, and the tracker TRK may be placed on the object OBJ.

[0079] First, refer to scenario 601. When the user U moves the hand H close to the waist to hold the object OBJ, the hand holding algorithm may be triggered (enabled) to determine whether the hand U holds the object OBJ based on the trajectory T_O and the trajectory T_H. That is, the hand holding algorithm is enabled only within a certain time period (e.g., time t1 to time t2). For example, the processor PCS may be configured to: in response to the hand H approaching the object OBJ, determine whether the hand H holds the object based on the hand motion curve and the object motion curve. In one embodiment, the term "close" may be defined as the distance between the hand H and the object OBJ being less than a threshold, but is not limited thereto. In other words, the hand holding algorithm may be enabled only within a certain time period. Therefore, the computing power of the processor PCS can be saved, and the overall power consumption can be reduced. In addition, if the hand H is moving away from the object OBJ, the hand H may not be detected as holding the object OBJ, thereby reducing the error of the hand holding algorithm and increasing the user experience.

[0080] Now refer to scenario 602. After the user U puts the object OBJ back to the waist, the distance between the hand H and the object OBJ may gradually increase as the hand H moves away. That is, the hand no longer holds the object OBJ. In other words, the processor PCS may be configured to: in response to the hand H not being close to the object OBJ, determine that the hand H does not hold the object OBJ. Therefore, the real-time state of the hand H can be accurately and continuously tracked, which can bring a better user experience.

[0081] It is worth mentioning that in scenarios 601 and 602, the sensor SNR may be placed close to the object OBJ. For example, the sensor SNR may be placed on the belt BLT, but is not limited thereto. In one embodiment, the object OBJ may be placed in the object fixing area of ​​the belt BLT, and the sensor SNR may be placed in the object fixing area. For example, the object OBJ may be a pistol, and the object fixing area may be a holster. In addition, the sensor SNR may be placed in the holster to detect whether the pistol is in the holster. In one embodiment, the sensor SNR may include, for example, a proximity sensor, a touch sensor, a latch, other similar devices including a detection mechanism, or a combination of these devices. However, the present disclosure is not limited thereto.

[0082] That is, the sensor SNR may provide sensor data, and the sensor data may indicate whether the object OBJ stays in the object fixed area. In other words, the sensor data may also indicate whether the object OBJ is approaching or moving away from the sensor SNR. It is worth noting that the object OBJ can only be held by the hand H when the object OBJ is not in the object fixed area. In other words, the processor PCS may be configured to receive sensor data from the sensor SNR. The sensor SNR may be placed close to the object OBJ. In addition, the processor PCS may be configured to determine whether the object OBJ is moving away from the sensor SNR based on the sensor data. In addition, the processor PCS may be configured to: in response to the object OBJ moving away from the sensor SNR, determine whether the hand H is holding the object OBJ based on the hand motion curve and the object motion curve. Simply put, the hand holding algorithm can be enabled only within a period of time (i.e., when the object OBJ is detected to move out of the object fixed area). Therefore, the computing power of the processor PCS can be saved, and the overall power consumption can be reduced. In addition, if the object OBJ still stays in the object fixed area, the hand H may not be detected holding the object OBJ, thereby reducing the error of the hand holding algorithm and increasing the user experience.

[0083] In summary, according to the hand tracking device 100A, the hand tracking system 100B, and the hand tracking method 200, it is possible to determine whether the hand is holding an object based on the hand motion curve and the object motion curve. In other words, a hand holding algorithm can be performed based on the hand motion curve and the object motion curve. Therefore, when the object OBJ is involved, the hand tracking of the hand H can still be performed in a natural and intuitive manner, thereby providing a fully immersive experience to the user.

[0084] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, it is intended that the present disclosure covers modifications and variations, provided that the modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A hand tracking device, characterized in that: include: A storage circuit for storing program codes; as well as A processor is coupled to the storage circuit and accesses the program code to perform the following operations: Obtaining a hand movement curve of a hand within a period of time; Obtaining an object motion curve of the object within the time period; and Based on the hand motion curve and the object motion curve, it is determined whether the hand is holding the object.

2. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: determining a similarity between the hand motion curve and the object motion curve; and Based on the similarity, it is determined whether the hand is holding the object.

3. The hand tracking device according to claim 2, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the similarity being higher than a predetermined threshold, it is determined that the hand is holding the object.

4. The hand tracking device according to claim 2, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the similarity being not higher than a predetermined threshold, it is determined that the hand is not holding the object.

5. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: Obtaining another hand movement curve of the other hand within the time period; Based on the hand motion curve, the other hand motion curve, and the object motion curve, it is determined whether the hand or the other hand is holding the object.

6. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: Receive image data from the camera; determining the hand motion curve based on the image data; receiving tracker data from a tracker, wherein the tracker is adapted to be placed on the object; as well as The object motion profile is determined based on the tracker data.

7. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: Display virtual hands and virtual objects in the virtual world In response to determining that the hand is holding the object, in the virtual world, the virtual hand is displayed holding the virtual object.

8. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: In the virtual world, display virtual hands and virtual objects; moving the virtual hand based on the hand motion curve; and The virtual object is moved based on the object motion curve.

9. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: pairing a tracker with the object, wherein the tracker is adapted to be placed on the object; generating pairing information in response to pairing the tracker with the object, wherein the pairing information includes an object type of the object; as well as The object type of the object is determined based on the pairing information.

10. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: Obtaining a user motion curve of the user within the time period; obtaining an additional object motion curve of an additional object within the time period, wherein the additional object is placed on the user; as well as In response to the user being walking, the user motion profile is similar to the additional object motion profile, and the user motion profile is not similar to the object motion profile, It is determined that the object is held by the hand and the additional object is not held by the hand.

11. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to determining that the hand is holding the object, a prompt message is provided to the user.

12. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the hand being in the field of view of the camera, performing hand tracking of the hand based on the hand motion curve; as well as In response to the hand not being in the field of view of the camera, the hand tracking of the hand is performed based on the object motion profile.

13. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the hand approaching the object, it is determined whether the hand is holding the object based on the hand motion profile and the object motion profile.

14. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the hand not being close to the object, it is determined that the hand is not holding the object.

15. The hand tracking device according to claim 1, characterized in that: The processor is further configured to access the program code to perform the following operations: receiving sensor data from a sensor, wherein the sensor is positioned proximate to the object; determining whether the object is moving away from the sensor based on the sensor data; as well as In response to the object moving away from the sensor, it is determined whether the hand is holding the object based on the hand motion profile and the object motion profile.

16. A hand tracking system, characterized in that: include: a camera configured to provide image data; a tracker configured to provide tracker data, wherein the tracker is adapted to be placed on an object; A storage circuit for storing program codes; as well as A processor is coupled to the storage circuit and accesses the program code to perform the following operations: receiving the image data from the camera; Determining a hand movement curve of the hand within a time period based on the image data; receiving the tracker data from the tracker; determining an object motion profile of the object within the time period based on the tracker data; as well as Based on the hand motion curve and the object motion curve, it is determined whether the hand is holding the object.

17. The hand tracking system according to claim 16, characterized in that: The processor is further configured to access the program code to perform the following operations: determining a similarity between the hand motion curve and the object motion curve; and Based on the similarity, it is determined whether the hand is holding the object.

18. The hand tracking system according to claim 17, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the similarity being higher than a predetermined threshold, it is determined that the hand is holding the object.

19. The hand tracking system according to claim 17, characterized in that: The processor is further configured to access the program code to perform the following operations: In response to the similarity being not higher than a predetermined threshold, it is determined that the hand is not holding the object.

20. A hand tracking method, characterized in that: include: Obtaining a hand movement curve of a hand within a period of time; obtaining an object motion curve of the object within the time period; as well as Based on the hand motion curve and the object motion curve, it is determined whether the hand is holding the object.

Images