Gesture tracking device and method and non-transitory computer readable storage medium thereof

Abstract

The invention discloses a gesture tracking device, a gesture tracking method and a non-transient computer readable storage medium thereof. The apparatus performs a gesture tracking corresponding to a user based on a plurality of real-time images and an inertial sensing data. The device determines whether a virtual object associated with a haptic feedback interaction is contacted based on the gesture tracking. In response to determining that the virtual object associated with the haptic feedback interaction is touched, the device triggers a haptic feedback device of a wearable device to perform the haptic feedback interaction corresponding to a time interval. The device disables the inertial sensing data corresponding to the wearable device to perform the gesture tracking corresponding to the time interval. The gesture tracking technology provided by the invention can improve the accuracy of gesture tracking.

Description

Technical Field

[0001] The present invention relates to a gesture tracking device, method, and non-transitory computer-readable storage medium thereof. Specifically, the present invention relates to a gesture tracking device, method, and non-transitory computer-readable storage medium thereof that can dynamically adjust the timing of inertial sensing data usage. Background Art

[0002] In recent years, various technologies related to virtual reality have developed rapidly, and various related technologies and applications have been proposed one after another.

[0003] In the prior art, a head-mounted display can generate auxiliary data through an inertial sensing unit installed in a wearable device (e.g., a smart ring) to improve the accuracy of gesture tracking.

[0004] Furthermore, in the prior art, wearable devices may be provided with a tactile feedback interactive element (eg, a vibrator) to provide the user with tactile feedback (eg, vibration feedback) to enhance the user's operating experience.

[0005] However, when the wearable device has a tactile feedback interactive element, when the inertial sensing unit and the tactile feedback interactive element operate simultaneously, the tactile feedback generated by the tactile feedback interactive element will cause the data generated by the inertial sensing unit to lose accuracy, thereby reducing the accuracy of gesture tracking.

[0006] Therefore, the industry urgently needs to work on providing a gesture tracking technology that can dynamically adjust the timing of using inertial sensing data. Summary of the Invention

[0007] One object of the present invention is to provide a gesture tracking device. The gesture tracking device includes a transceiver interface and a processor, the processor being electrically connected to the transceiver interface. The transceiver interface is communicatively connected to a wearable device, the wearable device being configured to generate inertial sensing data corresponding to a user, and the wearable device including a tactile feedback device. The processor performs gesture tracking corresponding to the user based on multiple real-time images and the inertial sensing data, wherein the multiple real-time images include a user's hand or the wearable device used by the user. Based on the gesture tracking, the processor determines whether a virtual object associated with a tactile feedback interaction is touched. In response to determining that the virtual object associated with the tactile feedback interaction is touched, the processor triggers the tactile feedback device of the wearable device to perform the tactile feedback interaction for a corresponding time interval. The processor deactivates the inertial sensing data corresponding to the wearable device to perform gesture tracking corresponding to the time interval.

[0008] In one embodiment of the present invention, the processor further performs the following operations: in response to determining that the tactile feedback device ends the tactile feedback interaction corresponding to the time interval, enabling the inertial sensing data corresponding to the wearable device to perform the gesture tracking corresponding to the user.

[0009] In one embodiment of the present invention, determining whether the virtual object associated with the tactile feedback interaction is touched includes the following operations: based on the gesture tracking, determining whether an interactive behavior of the user meets at least one deactivation condition; and in response to the interactive behavior meeting the at least one deactivation condition, determining whether the virtual object associated with the tactile feedback interaction is touched.

[0010] In one embodiment of the present invention, the at least one deactivation condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a line of sight determination condition, and a target path determination condition, or a combination thereof.

[0011] In one embodiment of the present invention, when the at least one deactivation condition includes the gesture judgment condition, the processor further performs the following operations: based on the gesture tracking, determining whether the interactive behavior conforms to one of a plurality of target gestures; and in response to determining that the interactive behavior conforms to one of the plurality of target gestures, determining that the virtual object associated with the tactile feedback interaction is touched.

[0012] In one embodiment of the present invention, when the at least one deactivation condition includes the action judgment condition, the processor further performs the following operations: based on the gesture tracking, determining whether the interactive behavior conforms to an arm extending action; and in response to determining that the interactive behavior conforms to the arm extending action, determining that the virtual object associated with the tactile feedback interaction is touched.

[0013] In one embodiment of the present invention, when the at least one deactivation condition includes the target area judgment condition, the processor further performs the following operations: based on the gesture tracking, determining whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region space extending outward from a center of the virtual object; and in response to determining that the interactive behavior is located in the target area of ​​the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0014] In one embodiment of the present invention, when the at least one deactivation condition includes the line of sight judgment condition, the processor further performs the following operations: generating a line of sight position corresponding to the user based on multiple eye trajectories; determining whether the line of sight position is located on the virtual object; and in response to determining that the line of sight position is located on the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0015] In one embodiment of the present invention, when the at least one deactivation condition includes the target path judgment condition, the processor further performs the following operations: calculating a hand movement path corresponding to the user; based on the gesture tracking, determining whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user and a spatial position corresponding to the virtual object; and in response to determining that the hand movement path is located on the target path, determining that the virtual object associated with the tactile feedback interaction is touched.

[0016] In one embodiment of the present invention, the tactile feedback interaction is a vibration feedback interaction.

[0017] Another object of the present invention is to provide a gesture tracking method, which is used in an electronic device, wherein the electronic device is communicatively connected to a wearable device, the wearable device is used to generate inertial sensing data corresponding to a user, and the wearable device includes a tactile feedback device. The gesture tracking method includes the following steps: performing gesture tracking corresponding to the user based on multiple real-time images and the inertial sensing data, wherein the multiple real-time images include a user's hand or the wearable device used by the user; based on the gesture tracking, determining whether a virtual object associated with a tactile feedback interaction is touched; in response to determining that the virtual object associated with the tactile feedback interaction is touched, triggering the tactile feedback device of the wearable device to perform the tactile feedback interaction corresponding to a time interval; and deactivating the inertial sensing data corresponding to the wearable device to perform the gesture tracking corresponding to the time interval.

[0018] In one embodiment of the present invention, the following steps are further included: in response to determining that the tactile feedback device ends executing the tactile feedback interaction corresponding to the time interval, enabling the inertial sensing data corresponding to the wearable device to execute the gesture tracking corresponding to the user.

[0019] In one embodiment of the present invention, determining whether the virtual object associated with the tactile feedback interaction is touched includes the following steps: based on the gesture tracking, determining whether an interactive behavior of the user meets at least one deactivation condition; and in response to the interactive behavior meeting the at least one deactivation condition, determining whether the virtual object associated with the tactile feedback interaction is touched.

[0020] In one embodiment of the present invention, the at least one deactivation condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a line of sight determination condition, and a target path determination condition, or a combination thereof.

[0021] In one embodiment of the present invention, when the at least one deactivation condition includes the gesture judgment condition, the gesture tracking method further includes the following steps: based on the gesture tracking, determining whether the interactive behavior conforms to one of a plurality of target gestures; and in response to determining that the interactive behavior conforms to one of the plurality of target gestures, determining that the virtual object associated with the tactile feedback interaction is touched.

[0022] In one embodiment of the present invention, when the at least one deactivation condition includes the action judgment condition, the gesture tracking method further includes the following steps: based on the gesture tracking, determining whether the interactive behavior conforms to an arm extending action; and in response to determining that the interactive behavior conforms to the arm extending action, determining that the virtual object associated with the tactile feedback interaction is touched.

[0023] In one embodiment of the present invention, when the at least one deactivation condition includes the target area judgment condition, the gesture tracking method further includes the following steps: based on the gesture tracking, determining whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region space extending outward from a center of the virtual object; and in response to determining that the interactive behavior is located in the target area of ​​the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0024] In one embodiment of the present invention, when the at least one deactivation condition includes the line of sight judgment condition, the gesture tracking method further includes the following steps: generating a line of sight position corresponding to the user based on multiple eye trajectories; determining whether the line of sight position is located on the virtual object; and in response to determining that the line of sight position is located on the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0025] In one embodiment of the present invention, when the at least one deactivation condition includes the target path judgment condition, the gesture tracking method further includes the following steps: calculating a hand movement path corresponding to the user; based on the gesture tracking, determining whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user and a spatial position corresponding to the virtual object; and in response to determining that the hand movement path is located on the target path, determining that the virtual object associated with the tactile feedback interaction is touched.

[0026] Another object of the present invention is to provide a non-transitory computer-readable storage medium storing a computer program comprising a plurality of program instructions. When loaded into an electronic device, the computer program executes a gesture tracking method. The electronic device is communicatively connected to a wearable device, the wearable device is configured to generate inertial sensing data corresponding to a user, and the wearable device includes a haptic feedback device. The gesture tracking method comprises the following steps: performing gesture tracking corresponding to the user based on a plurality of real-time images and the inertial sensing data, wherein the plurality of real-time images include a user's hand or the wearable device used by the user; determining whether a virtual object associated with a haptic feedback interaction is touched based on the gesture tracking; in response to determining that the virtual object associated with the haptic feedback interaction is touched, triggering the haptic feedback device of the wearable device to perform the haptic feedback interaction for a corresponding time interval; and disabling the inertial sensing data corresponding to the wearable device to perform gesture tracking for the corresponding time interval.

[0027] The gesture tracking technology provided by this disclosure (including at least an apparatus, method, and non-transitory computer-readable storage medium thereof) can pre-determine whether a virtual object has been touched based on various conditions, thereby dynamically adjusting the timing of inertial sensing data usage. Therefore, the gesture tracking technology provided by this disclosure can dynamically adjust the manner in which gesture tracking is performed, avoiding the use of inertial sensing data during the time interval specified when the tactile feedback device is triggered. This improves the accuracy of gesture tracking, overcomes the shortcomings of existing technologies, and enhances the user experience.

[0028] The following describes the detailed technology and implementation methods of the present invention in conjunction with the accompanying drawings so that a person having ordinary knowledge in the technical field to which the present invention belongs can understand the technical features of the invention for which protection is sought. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram showing an applicable scenario of the gesture tracking system of the first embodiment;

[0030] Figure 2AA schematic diagram illustrating the architecture of a gesture tracking device according to certain embodiments;

[0031] Figure 2B A schematic diagram illustrating the architecture of a gesture tracking device according to certain embodiments;

[0032] Figure 3 A schematic diagram illustrating the architecture of a wearable device according to certain embodiments;

[0033] Figure 4 Schematic diagram illustrating target gestures in certain embodiments;

[0034] Figure 5A A schematic diagram showing an arm extending motion according to certain embodiments;

[0035] Figure 5B A schematic diagram illustrating a target area according to certain embodiments;

[0036] Figure 6 A schematic diagram illustrating a target path for certain embodiments;

[0037] Figure 7 A timing diagram illustrating the operation of certain embodiments; and

[0038] Figure 8 A partial flow chart of the gesture tracking method according to the second embodiment is shown.

[0039] Explanation of symbols:

[0040] 1: Gesture tracking system

[0041] 2: Gesture tracking device

[0042] 3: Wearable devices

[0043] C: User

[0044] 21: Transceiver interface

[0045] 23: Processor

[0046] 25: Storage module

[0047] 27: Image Capture

[0048] 29: Eye Tracker

[0049] 31: Transceiver interface

[0050] 33: Processor

[0051] 35: Inertial Measurement Unit

[0052] 37: Tactile feedback device

[0053] 400: Target gesture diagram

[0054] G1, G2, G3: Target gestures

[0055] 500: Diagram of arm extension action

[0056] RD, LD, FD: Arm extension movements

[0057] OA1, OA2: target areas

[0058] VO1, VO2: Virtual objects

[0059] 600: Target Path Diagram

[0060] TP1, TP2: target path

[0061] VO: Virtual Object

[0062] 700: Runtime Sequence Diagram

[0063] T: time value

[0064] P1, P2: time points

[0065] T1: Time interval

[0066] GDC: Gesture judgment conditions

[0067] HFI: Haptic Feedback Interaction

[0068] HGT, HGT1, HGT2, HGT3: Hand tracking

[0069] 800: Gesture Tracking Method

[0070] S801, S803, S805, S807: Steps DETAILED DESCRIPTION

[0071] The following will explain a gesture tracking device, method, and non-transitory computer-readable storage medium provided by the present invention through embodiments. However, these embodiments are not intended to limit the present invention to any environment, application, or method as described in these embodiments. Therefore, the description of the embodiments is only for the purpose of illustrating the present invention and is not intended to limit the scope of the present invention. It should be understood that in the following embodiments and drawings, elements that are not directly related to the present invention have been omitted and are not shown, and the sizes of the elements and the size ratios between the elements are only examples and are not intended to limit the scope of the present invention.

[0072] First, the applicable scenario of this embodiment is described, and its schematic diagram is shown in Figure 1 .like Figure 1As shown, in the application environment of the present disclosure, a gesture tracking system 1 includes a gesture tracking device 2 and a wearable device 3 , and the gesture tracking device 2 is communicatively connected to the wearable device 3 .

[0073] like Figure 1 As shown, user C can use a gesture tracking device 2 (e.g., a head-mounted display), and user C wears a wearable device 3 on a body part (e.g., user C wears a smart ring on the index finger of his left hand) to perform interactive operations corresponding to the display screen of the gesture tracking device 2.

[0074] In this embodiment, the structure diagram of the gesture tracking device 2 is shown in FIG. Figure 2A The gesture tracking device 2 includes a transceiver interface 21 , a processor 23 , and a storage module 25 . The processor 23 is electrically connected to the transceiver interface 21 and the storage module 25 . The transceiver interface 21 is communicatively connected to the wearable device 3 .

[0075] In certain embodiments, such as Figure 2A As shown, the gesture tracking device 2 further includes an image capturer 27 coupled to the processor 23. The image capturer 27 may include multiple image capture units (e.g., multiple depth camera lenses) for capturing multiple real-time images of a body part of the wearable device 3 worn by the user C.

[0076] In some embodiments, the image capturer 27 is used to capture a plurality of real-time images corresponding to a field of view (FOV), wherein the body part of the user C wearing the wearable device 3 is included in the FOV.

[0077] In addition, in this embodiment, the structure diagram of the wearable device 3 is shown in FIG. Figure 3 The wearable device 3 includes a transceiver interface 31, a processor 33, an inertial measurement unit 35, and a tactile feedback device 37. The processor 33 is coupled to the transceiver interface 31, the inertial measurement unit 35, and the tactile feedback device 37. The inertial measurement unit 35 can be used to detect a plurality of inertial sensing data corresponding to the body parts of the user C wearing the wearable device 3.

[0078] Specifically, the inertial measurement unit 35 continuously generates a series of inertial sensing data (e.g., a stream of inertial sensing data generated at a frequency of 90 times per second). Each of the inertial sensing data may include an acceleration, a rotation, and an angular acceleration. During operation, the gesture tracking device 2 may periodically receive the inertial sensing data from the wearable device 3.

[0079] It should be noted that the inertial sensing data generated by the wearable device 3 can correspond to body parts (e.g., fingers) of the user C. For example, the user C can wear the wearable device 3 on any finger to collect data. For ease of explanation, this embodiment will illustrate the user C wearing the wearable device 3 on their index finger.

[0080] It should be noted that the haptic feedback device 37 is used to generate a haptic feedback interaction. In some embodiments, the haptic feedback interaction is a vibration feedback interaction. For example, when the wearable device 3 is a smart ring, the haptic feedback device 37 can be a vibrator that can be used to generate vibration feedback interaction on the smart ring worn by user C.

[0081] It should be noted that transceiver interface 21 and transceiver interface 31 are interfaces capable of receiving and transmitting data, or other interfaces capable of receiving and transmitting data known to those skilled in the art. The transceiver interfaces can receive data from sources such as external devices, external web pages, and external applications. Processors 23 and 33 can be various processing units, central processing units (CPUs), microprocessors, or other computing devices known to those skilled in the art. Storage module 25 can be a memory, a Universal Serial Bus (USB) disk, a hard drive, an optical disk, a portable drive, or any other storage medium or circuit known to those skilled in the art that has the same functionality.

[0082] It should be noted that Figure 1 This is for illustrative purposes only and does not include the details of the gesture tracking system 1 . For example, the present invention does not limit the number of wearable devices 3 connected to the gesture tracking device 2 . The gesture tracking device 2 can simultaneously connect to multiple wearable devices via a network, depending on the scale and actual needs of the gesture tracking system 1 .

[0083] In this embodiment, the processor 23 performs gesture tracking corresponding to the user C based on multiple real-time images and the inertial sensing data, and the multiple real-time images include one or two hands of the user C and / or the wearable device 3 used by the user C.

[0084] Next, in this embodiment, the processor 23 determines whether a virtual object associated with a haptic feedback interaction is touched based on the gesture tracking.

[0085] Subsequently, in response to determining that the virtual object associated with the tactile feedback interaction is touched, the processor 23 triggers the tactile feedback device 37 of the wearable device 3 to perform the tactile feedback interaction for a corresponding time interval.

[0086] It should be noted that when designing a virtual reality system, multiple virtual objects can be associated with the same or different tactile feedback interactions. For example, the virtual object can be a virtual button. When the processor 23 determines that the user C touches the virtual button, the wearable device 3 activates the vibration feedback interaction.

[0087] For another example, the virtual objects may be multiple three-dimensional spherical objects, and can be selected by the user C. When the processor 23 determines that the user C touches one of the three-dimensional spherical objects, the wearable device 3 activates vibration feedback interaction.

[0088] For another example, the virtual object may be a window object, which is resized or moved by the user C. When the processor 23 determines that the user C touches the window object, the wearable device 3 activates vibration feedback interaction.

[0089] Finally, in this embodiment, the processor 23 deactivates the inertial sensing data corresponding to the wearable device 3 to perform the gesture tracking corresponding to the time interval.

[0090] In some embodiments, when tactile feedback interaction is no longer active, the processor 23 can proactively activate the inertial sensing data to improve the accuracy of gesture tracking. Specifically, in response to determining that the tactile feedback device 37 has terminated the tactile feedback interaction corresponding to the time interval, the processor 23 activates the inertial sensing data corresponding to the wearable device 3 to perform the gesture tracking corresponding to the user C.

[0091] In some embodiments, the processor 23 may transmit a control signal to the wearable device 3 to stop generating the inertial sensing data, thereby disabling the inertial sensing data of the corresponding wearable device 3 .

[0092] In some embodiments, the operation of disabling the inertial sensing data corresponding to the wearable device 3 can be implemented by the processor 23 discarding or ignoring the inertial sensing data.

[0093] In some embodiments, the processor 23 may actively suspend processing the inertial sensing data received from the wearable device 3 to disable the inertial sensing data of the corresponding wearable device 3 .

[0094] In some embodiments, the processor 23 can determine that the virtual object is about to be touched by comparing user C's interaction behavior with certain conditions, thereby preemptively deactivating the inertial sensing data. Specifically, based on the gesture tracking, the processor 23 determines whether an interaction behavior of user C meets at least one deactivation condition. Then, in response to the interaction behavior meeting the at least one deactivation condition, the processor 23 determines that the virtual object associated with the haptic feedback interaction is touched.

[0095] It should be noted that the term "contact" as defined in this disclosure includes user C having contacted the virtual object or user C about to contact the virtual object (i.e., user C has not actually contacted the virtual object at the current time point). In some embodiments, when the processor 23 determines that the virtual object associated with the haptic feedback interaction has been contacted, it indicates that the virtual object has been contacted by user C. In some embodiments, when the processor 23 determines that the virtual object associated with the haptic feedback interaction has been contacted, it indicates that the virtual object is about to be contacted by user C.

[0096] In some embodiments, the at least one disabling condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a line of sight determination condition, and a target path determination condition, or a combination thereof.

[0097] The specific implementation details of each deactivation condition will be described in detail below. It should be understood that in the present disclosure, each deactivation condition can be implemented separately or in combination. For example, when the deactivation condition has two judgment conditions, the processor 23 can judge whether the interactive behavior of user C meets one of the two judgment conditions to determine whether the virtual object associated with the tactile feedback interaction is touched. In addition, the processor 23 can also judge whether the interactive behavior of user C meets all two judgment conditions to determine whether the virtual object associated with the tactile feedback interaction is touched.

[0098] In certain embodiments, the gesture tracking device 2 stores multiple target gestures. When the at least one deactivation condition includes the gesture determination condition, the processor 23 determines whether the gesture of the user C matches one of the multiple target gestures to determine whether the virtual object has been touched. The processor 23 deactivates the inertial sensing data when the virtual object is determined to have been touched. Specifically, based on the gesture tracking, the processor 23 determines whether the interaction behavior matches one of the multiple target gestures. Then, in response to determining that the interaction behavior matches one of the multiple target gestures, the processor 23 determines that the virtual object associated with the haptic feedback interaction has been touched.

[0099] In some embodiments, the processor 23 may input the gesture tracking into a convolutional neural network (e.g., a trained convolutional neural network) to determine the gesture corresponding to user C.

[0100] For easier understanding, please refer to Figure 4 Target gesture diagram 400 is shown. In this example, target gestures include target gesture G1 of extending the index finger, target gesture G2 of pinching with the index finger and thumb spread apart, and target gesture G3 of grasping with the palm open. In this example, when the processor 23 determines that the gesture of user C matches target gesture G1, G2, or G3, it determines that the virtual object has been touched.

[0101] In certain embodiments, when the at least one deactivation condition includes the action determination condition, the processor 23 determines whether the user C's hand motion (e.g., arm motion) matches the target motion to determine whether the virtual object has been touched. The processor 23 deactivates the inertial sensing data upon determining that the virtual object has been touched. Specifically, based on the gesture tracking, the processor 23 determines whether the interactive behavior matches an extended arm motion. Then, in response to determining that the interactive behavior matches the extended arm motion, the processor 23 determines that the virtual object associated with the haptic feedback interaction has been touched.

[0102] For easier understanding, please refer to Figure 5A Schematic diagram 500 of an arm extension action is shown. In this example, an arm extension action (e.g., an arm stretching action) includes an arm extension action RD, which moves the arm away from the body to the right; an arm extension action LD, which moves the arm away from the body to the left; and an arm extension action FD, which moves the arm away from the body to the front. In this example, when the processor 23 determines that the user C's action matches the arm extension action RD, LD, or FD, it determines that the virtual object has been touched.

[0103] In certain embodiments, when the at least one deactivation condition includes the target area determination condition, the processor 23 further extends the virtual object outward to expand the sensing area, and determines whether the interactive behavior of the user C contacts the expanded sensing area, and the processor 23 deactivates the inertial sensing data when determining that the sensing area is touched. Specifically, based on the gesture tracking, the processor 23 determines whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region of space extending outward from a center or a surface of the virtual object. Then, in response to determining that the interactive behavior is located in the target area of ​​the virtual object, the processor 23 determines that the virtual object associated with the tactile feedback interaction is touched.

[0104] For easier understanding, please refer to Figure 5BSchematic diagram of two target areas. In the example on the left, the virtual object VO1 is a three-dimensional spherical object. The processor 23 extends a distance outward from the center or surface of the sphere of the virtual object VO1 to generate the target area OA1. In the example on the right, the virtual object VO2 is a two-dimensional planar object. The processor 23 extends a distance outward from the center or surface of the plane of the virtual object VO2 to generate the target area OA2. In this example, when the processor 23 determines that the interactive behavior of the user C is located in the target area OA1 or OA2 of the virtual object (for example, the position of the hand), the virtual object is determined to be touched.

[0105] In some embodiments, in order to make the judgment of the deactivation condition more accurate, the gesture tracking device 2 can further add the eye movement of the user C as one of the judgment conditions. Specifically, Figure 2B As shown, the gesture tracking device 2 further includes an eye tracker 29 , which is used to generate a plurality of eye trajectories corresponding to the user C.

[0106] In some embodiments, the processor 23 further determines whether the user C's gaze position is over the virtual object to determine that the virtual object has been touched, and the processor 23 deactivates the inertial sensing data when the virtual object is determined to have been touched. Specifically, the processor 23 generates a gaze position corresponding to the user C based on multiple eye trajectories. The processor 23 then determines whether the gaze position is over the virtual object. Finally, in response to determining that the gaze position is over the virtual object, the processor 23 determines that the virtual object associated with the haptic feedback interaction has been touched.

[0107] In certain embodiments, when the at least one deactivation condition includes the target path judgment condition, the processor 23 determines whether the hand movement path of the user C conforms to the target path to determine whether the virtual object is touched, and the processor 23 deactivates the inertial sensing data when determining that the virtual object is touched. Specifically, the processor 23 calculates a hand movement path corresponding to the user C. Then, based on the gesture tracking, the processor 23 determines whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user C and a spatial position corresponding to the virtual object. Then, in response to determining that the hand movement path is located on the target path, the processor 23 determines that the virtual object associated with the tactile feedback interaction is touched.

[0108] For easier understanding, please refer to Figure 6Target path diagram 600 is shown. In this example, the processor 23 generates target path TP1 based on the current position of user C's right hand and the virtual object VO. The processor 23 generates target path TP2 based on the current position of user C's left hand and the virtual object VO. In this example, when the processor 23 determines that the movement path of user C's hand is within target path TP1 or TP2, it determines that the virtual object has been contacted.

[0109] For ease of understanding, a practical implementation example is provided. Please refer to Figure 7 The horizontal axis of the operation timing diagram 700 is a time value T. In this example, the processor 23 executes the gesture tracking HGT in each time interval, and the deactivation condition includes the gesture determination condition GDC.

[0110] First, before the time point P1 , the processor 23 determines that the virtual object is not touched, and thus the processor 23 executes the gesture tracking HGT1 corresponding to the user C based on the multiple real-time images and the inertial sensing data.

[0111] Next, the processor 23 determines that the interaction behavior of the user C matches the target gesture G3 between time points P1 and P2 (i.e., time interval T1). Therefore, the processor 23 triggers the tactile feedback device 37 of the wearable device 3 to execute the tactile feedback interaction HFI during time interval T1, and disables the inertial sensing data of the corresponding wearable device 3 during time interval T1 to execute the gesture tracking HGT2 corresponding to time interval T1.

[0112] In addition, after time point P2, the processor 23 determines that the virtual object is not touched, and thus actively activates the inertial sensing data corresponding to the wearable device 3. Then, the processor 23 performs gesture tracking HGT3 corresponding to the user C based on the multiple real-time images and the inertial sensing data.

[0113] In some embodiments, the processor 23 only performs the tactile feedback interaction HFI at the time when the tactile feedback device 37 performs the tactile feedback interaction HFI (for example: Figure 7 The inertial sensing data of the wearable device 3 is disabled during the time period between time point P1 and time point P2 in the haptic feedback device 37. Furthermore, when the haptic feedback device 37 is no longer actuated, the processor 23 enables the inertial sensing data of the wearable device 3 to perform gesture tracking of the user C.

[0114] As can be seen from the above description, the gesture tracking device 2 provided by the present disclosure can pre-determine whether a virtual object has been touched based on various conditions, thereby dynamically adjusting the timing of inertial sensing data usage. Therefore, the gesture tracking device 2 provided by the present disclosure can dynamically adjust the gesture tracking method to avoid using inertial sensing data during the specific time period when the tactile feedback device is triggered. This improves the accuracy of gesture tracking, overcomes the shortcomings of existing technologies, and enhances the user experience.

[0115] The second embodiment of the present invention is a gesture tracking method, the flow chart of which is shown in FIG. Figure 8 The gesture tracking method 800 is applicable to an electronic device, such as the gesture tracking device 2 described in the first embodiment. The electronic device is communicatively connected to a wearable device, such as the wearable device 3 described in the first embodiment. The wearable device is configured to generate inertial sensing data corresponding to a user, and the wearable device includes a tactile feedback device. The gesture tracking method 800 performs gesture tracking through steps S801 to S807.

[0116] In step S801 , the electronic device performs gesture tracking corresponding to the user based on a plurality of real-time images and the inertial sensing data, wherein the plurality of real-time images include a user's hand or the wearable device used by the user.

[0117] Next, in step S803 , the electronic device determines whether a virtual object associated with a haptic feedback interaction is touched based on the gesture tracking.

[0118] Next, in step S805 , in response to determining that the virtual object associated with the tactile feedback interaction is touched, the electronic device triggers the tactile feedback device of the wearable device to perform the tactile feedback interaction for a corresponding time interval.

[0119] Finally, in step S807 , the electronic device disables the inertial sensing data corresponding to the wearable device to perform the gesture tracking corresponding to the time interval.

[0120] In some embodiments, the gesture tracking method 800 further includes the following steps: in response to determining that the tactile feedback device ends executing the tactile feedback interaction corresponding to the time interval, enabling the inertial sensing data corresponding to the wearable device to execute the gesture tracking corresponding to the user.

[0121] In some embodiments, determining whether the virtual object associated with the tactile feedback interaction is touched includes the following steps: based on the gesture tracking, determining whether an interactive behavior of the user meets at least one deactivation condition; and in response to the interactive behavior meeting the at least one deactivation condition, determining whether the virtual object associated with the tactile feedback interaction is touched.

[0122] In some embodiments, the at least one disabling condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a line of sight determination condition, and a target path determination condition, or a combination thereof.

[0123] In some embodiments, when the at least one deactivation condition includes the gesture judgment condition, the gesture tracking method 800 further includes the following steps: based on the gesture tracking, determining whether the interactive behavior conforms to one of multiple target gestures; and in response to determining that the interactive behavior conforms to one of the multiple target gestures, determining that the virtual object associated with the tactile feedback interaction is touched.

[0124] In some embodiments, when the at least one deactivation condition includes the action judgment condition, the gesture tracking method 800 further includes the following steps: based on the gesture tracking, determining whether the interactive behavior conforms to an arm extending action; and in response to determining that the interactive behavior conforms to the arm extending action, determining that the virtual object associated with the tactile feedback interaction is touched.

[0125] In some embodiments, when the at least one deactivation condition includes the target area judgment condition, the gesture tracking method 800 further includes the following steps: based on the gesture tracking, determining whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region space extending outward from a center of the virtual object; in response to determining that the interactive behavior is located in the target area of ​​the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0126] In some embodiments, when the at least one deactivation condition includes the gaze judgment condition, the gesture tracking method 800 further includes the following steps: generating a gaze position corresponding to the user based on multiple eye trajectories; determining whether the gaze position is located on the virtual object; and in response to determining that the gaze position is located on the virtual object, determining that the virtual object associated with the tactile feedback interaction is touched.

[0127] In some embodiments, when the at least one deactivation condition includes the target path judgment condition, the gesture tracking method 800 further includes the following steps: calculating a hand movement path corresponding to the user; based on the gesture tracking, determining whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user and a spatial position corresponding to the virtual object; and in response to determining that the hand movement path is located on the target path, determining that the virtual object associated with the tactile feedback interaction is touched.

[0128] In addition to the aforementioned steps, the second embodiment also performs all operations and steps of the gesture tracking device 2 described in the first embodiment, having the same functions and achieving the same technical effects. A person skilled in the art will readily understand how the second embodiment performs these operations and steps based on the first embodiment, achieving the same functions and technical effects, and therefore, a detailed description thereof will not be provided.

[0129] The calculation method described in the second embodiment can be implemented by a computer program having multiple instructions. Each computer program can be a file that can be transmitted over a network, or can be stored in a non-transitory computer-readable storage medium. For each computer program, after the multiple instructions contained therein are loaded into an electronic device (e.g., the gesture tracking device 2), the computer program executes the gesture tracking method described in the second embodiment. The non-transitory computer-readable storage medium can be an electronic product, such as a read-only memory (ROM), a flash memory, a floppy disk, a hard disk, a compact disk (CD), a portable disk, a database accessible by a network, or any other storage medium known to a person of ordinary skill in the art to which the present invention belongs and having the same function.

[0130] In summary, the gesture tracking technology provided by the present disclosure (including at least the device, method, and non-transitory computer-readable storage medium thereof) can pre-determine whether a virtual object has been touched based on various conditions, thereby dynamically adjusting the timing of inertial sensing data usage. Therefore, the gesture tracking technology provided by the present disclosure can dynamically adjust the manner in which gesture tracking is performed, avoiding the use of inertial sensing data during the specified time period when the tactile feedback device is triggered. This improves the accuracy of gesture tracking, overcomes the shortcomings of existing technologies, and enhances the user experience.

[0131] The above embodiments are intended only to illustrate some of the embodiments of the present invention and to illustrate the technical features of the present invention, and are not intended to limit the scope and extent of protection of the present invention. Any modifications or equivalent arrangements that can be easily accomplished by a person having ordinary skill in the art to which the present invention belongs fall within the scope claimed by the present invention, and the scope of protection of the present invention is subject to the claims.

Claims

1. A gesture tracking device, characterized in that: Include: a transceiver interface communicatively connected to a wearable device, wherein the wearable device is used to generate inertial sensing data corresponding to a user, and the wearable device includes a tactile feedback device; as well as a processor electrically connected to the transceiver interface and configured to perform the following operations: Performing gesture tracking corresponding to the user based on a plurality of real-time images and the inertial sensing data, wherein the plurality of real-time images include a user's hand or the wearable device used by the user; Based on the gesture tracking, determining whether a virtual object associated with a tactile feedback interaction is touched; In response to determining that the virtual object associated with the tactile feedback interaction is touched, triggering the tactile feedback device of the wearable device to perform the tactile feedback interaction corresponding to a time interval; as well as The inertial sensing data corresponding to the wearable device is disabled to perform the gesture tracking corresponding to the time interval.

2. The gesture tracking device according to claim 1, wherein: The processor further performs the following operations: In response to determining that the tactile feedback device ends executing the tactile feedback interaction corresponding to the time interval, the inertial sensing data corresponding to the wearable device is enabled to execute the gesture tracking corresponding to the user.

3. The gesture tracking device according to claim 1, wherein: Determining whether the virtual object associated with the tactile feedback interaction is touched includes the following operations: Based on the gesture tracking, determining whether an interactive behavior of the user meets at least one deactivation condition; as well as In response to the interactive behavior meeting the at least one deactivation condition, it is determined that the virtual object associated with the haptic feedback interaction is touched.

4. The gesture tracking device according to claim 3, wherein: The at least one disabling condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a sight determination condition, and a target path determination condition, or a combination thereof.

5. The gesture tracking device according to claim 4, wherein: When the at least one deactivation condition includes the gesture determination condition, the processor further performs the following operations: Based on the gesture tracking, determining whether the interactive behavior matches one of a plurality of target gestures; as well as In response to determining that the interactive behavior conforms to one of the target gestures, determining that the virtual object associated with the haptic feedback interaction is touched.

6. The gesture tracking device according to claim 4, wherein: When the at least one deactivation condition includes the action determination condition, the processor further performs the following operations: Based on the gesture tracking, determining whether the interactive behavior conforms to an arm extending motion; as well as In response to determining that the interactive behavior is consistent with the arm extending motion, it is determined that the virtual object associated with the tactile feedback interaction is touched.

7. The gesture tracking device according to claim 4, wherein: When the at least one deactivation condition includes the target area determination condition, the processor further performs the following operations: Based on the gesture tracking, determining whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region extending outward from a center of the virtual object; as well as In response to determining that the interactive behavior is located in the target area of ​​the virtual object, it is determined that the virtual object associated with the haptic feedback interaction is touched.

8. The gesture tracking device according to claim 4, wherein: When the at least one deactivation condition includes the sight line determination condition, the processor further performs the following operations: generating a sight line position corresponding to the user based on a plurality of eyeball trajectories; Determining whether the sight line position is located on the virtual object; as well as In response to determining that the sight line is located on the virtual object, it is determined that the virtual object associated with the haptic feedback interaction is touched.

9. The gesture tracking device according to claim 4, wherein: When the at least one disabling condition includes the target path determination condition, the processor further performs the following operations: Calculating a hand movement path corresponding to the user; Based on the gesture tracking, determining whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user and a spatial position of the corresponding virtual object; and In response to determining that the hand movement path is located on the target path, determining that the virtual object associated with the tactile feedback interaction is touched.

10. The gesture tracking device according to claim 1, wherein: The tactile feedback interaction is a vibration feedback interaction.

11. A gesture tracking method, characterized in that: In an electronic device, wherein the electronic device is communicatively connected to a wearable device, the wearable device is configured to generate inertial sensing data corresponding to a user, the wearable device includes a tactile feedback device, and the gesture tracking method includes the following steps: Performing gesture tracking corresponding to the user based on a plurality of real-time images and the inertial sensing data, wherein the plurality of real-time images include a user's hand or the wearable device used by the user; Based on the gesture tracking, determining whether a virtual object associated with a tactile feedback interaction is touched; In response to determining that the virtual object associated with the tactile feedback interaction is touched, triggering the tactile feedback device of the wearable device to perform the tactile feedback interaction corresponding to a time interval; as well as The inertial sensing data corresponding to the wearable device is disabled to perform the gesture tracking corresponding to the time interval.

12. The gesture tracking method according to claim 11, wherein: It also includes the following steps: In response to determining that the tactile feedback device ends executing the tactile feedback interaction corresponding to the time interval, the inertial sensing data corresponding to the wearable device is enabled to execute the gesture tracking corresponding to the user.

13. The gesture tracking method according to claim 11, wherein: Determining whether the virtual object associated with the tactile feedback interaction is touched comprises the following steps: Based on the gesture tracking, determining whether an interactive behavior of the user meets at least one deactivation condition; as well as In response to the interactive behavior meeting the at least one deactivation condition, it is determined that the virtual object associated with the haptic feedback interaction is touched.

14. The gesture tracking method according to claim 13, wherein: The at least one disabling condition includes at least one of a gesture determination condition, an action determination condition, a target area determination condition, a sight determination condition, and a target path determination condition, or a combination thereof.

15. The gesture tracking method according to claim 14, wherein: When the at least one disabling condition includes the gesture determination condition, the gesture tracking method further includes the following steps: Based on the gesture tracking, determining whether the interactive behavior matches one of a plurality of target gestures; as well as In response to determining that the interactive behavior conforms to one of the target gestures, determining that the virtual object associated with the haptic feedback interaction is touched.

16. The gesture tracking method according to claim 14, wherein: When the at least one disabling condition includes the action determination condition, the gesture tracking method further includes the following steps: Based on the gesture tracking, determining whether the interactive behavior conforms to an arm extending motion; as well as In response to determining that the interactive behavior is consistent with the arm extending motion, it is determined that the virtual object associated with the tactile feedback interaction is touched.

17. The gesture tracking method according to claim 14, wherein: When the at least one disabling condition includes the target area determination condition, the gesture tracking method further includes the following steps: Based on the gesture tracking, determining whether the interactive behavior is located in a target area of ​​the virtual object, wherein the target area is generated based on a region extending outward from a center of the virtual object; as well as In response to determining that the interactive behavior is located in the target area of ​​the virtual object, it is determined that the virtual object associated with the haptic feedback interaction is touched.

18. The gesture tracking method according to claim 14, wherein: When the at least one disabling condition includes the gaze determination condition, the gesture tracking method further includes the following steps: generating a sight line position corresponding to the user based on a plurality of eyeball trajectories; Determining whether the sight line position is located on the virtual object; as well as In response to determining that the sight line is located on the virtual object, it is determined that the virtual object associated with the haptic feedback interaction is touched.

19. The gesture tracking method according to claim 14, wherein: When the at least one disabling condition includes the target path determination condition, the gesture tracking method further includes the following steps: Calculating a hand movement path corresponding to the user; Based on the gesture tracking, determining whether the hand movement path is located on a target path, wherein the target path is composed of a current hand position of the user and a spatial position of the corresponding virtual object; and In response to determining that the hand movement path is located on the target path, determining that the virtual object associated with the tactile feedback interaction is touched.

20. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a computer program comprising a plurality of program instructions. When loaded into an electronic device, the computer program executes a gesture tracking method. The electronic device is communicatively connected to a wearable device, the wearable device is configured to generate inertial sensing data corresponding to a user, and the wearable device includes a tactile feedback device. The gesture tracking method comprises the following steps: Performing gesture tracking corresponding to the user based on a plurality of real-time images and the inertial sensing data, wherein the plurality of real-time images include a user's hand or the wearable device used by the user; Based on the gesture tracking, determining whether a virtual object associated with a tactile feedback interaction is touched; In response to determining that the virtual object associated with the tactile feedback interaction is touched, triggering the tactile feedback device of the wearable device to perform the tactile feedback interaction corresponding to a time interval; as well as The inertial sensing data corresponding to the wearable device is disabled to perform the gesture tracking corresponding to the time interval.

Images