Hand pose estimation method, mobile device, head-mounted display, and system

Abstract

A method includes: when a mobile device is bound to a head-mounted display (HMD) and when the mobile device is held by a first hand, the mobile device senses first data; the mobile device detects, from the first data, information indicating which hand is used to perform a gesture; the mobile device transmits the information to the HMD; the HMD receives the information, and the HMD does not detect the information by an image-based detection method, wherein the gesture is performed by a second hand, and the information is an update indication about the gesture, which takes into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of first image data; and the HMD performs, using the information, side-adapted hand pose estimation on the first image data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of estimating hand pose through head-mounted displays. BACKGROUND

[0002] Gestures can be used for a user to interact with a virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) environment presented in a head-mounted display (HMD). A gesture can be estimated by performing hand pose estimation on image data of the gesture. Performing hand pose estimation on image data is the task of finding a set of joints (e.g., 2D or 3D keypoint locations) of a hand used to perform a gesture from image data. For hand pose estimation using side adaptation, information of which of the two hands (i.e., left hand and right hand) is used to perform a gesture is needed. For one type of side-adapted hand pose estimation, information of which of the two hands is used to perform a gesture is needed to determine whether to use right hand pose estimation or left hand pose estimation to perform side-dependent hand pose estimation on side-dependent image data. For another type of side-adapted hand pose estimation, information of which of the two hands is used to perform a gesture is needed to determine whether to flip image data to perform side-dependent hand pose estimation on side-agnostic image data.

[0003] One method for obtaining information of which of the two hands is used to perform a gesture is to require a user to explicitly specify the user’s dominant hand (i.e., preferred hand), and the dominant hand is always used as the information of which of the two hands is used to perform each gesture. However, this method is not feasible for all cases. For example, for some cases, a user can use his / her right hand to perform one gesture, and then the user can switch to using his / her left hand to perform another gesture. Another method for obtaining information of which of the two hands is used to perform a gesture is to detect the information from image data through imaging-based detection methods. For this method, it is advantageous that the information of which of the two hands is used to perform a gesture is updated according to the gesture. SUMMARY

[0004] For existing hand pose estimation, information indicating which hand is used to perform a gesture is detected from image data of the gesture using imaging-based detection methods. The overall speed of this hand pose estimation is slowed down since the step of detecting information indicating which hand is used to perform a gesture is at least one of a slow step and a sequential processing step. When a head-mounted display (HMD) is bound to a mobile device, and the mobile device is also used as an input device for a user to interact with a virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) environment presented in the HMD, the existing hand pose estimation using imaging-based detection methods cannot achieve the speed improvement of using the mobile device to detect information indicating which hand is used to perform a gesture as proposed in this disclosure.

[0005] According to a first aspect of the present disclosure, a method performed by a mobile device comprises: when the mobile device is bound to a HMD, and when the mobile device is held by a first hand, a sensing device of the mobile device senses first data; at least one processor of the mobile device detects information indicating which hand is used to perform a gesture, the gesture to be estimated by at least one native or proxy processor of the HMD from first image data, wherein the information indicating which hand is used to perform the gesture is not detected from the first image data by the HMD with imaging-based detection methods, but from the first data; the at least one processor of the mobile device sends the information indicating which hand is used to perform the gesture to the at least one native or proxy processor of the HMD, the execution time point of the sending step being such that the at least one native or proxy processor of the HMD uses the information indicating which hand is used to perform the gesture to perform a side-adapted hand pose estimation on the first image data; wherein the gesture is performed by a second hand, and the information indicating which hand is used to perform the gesture is an updated indication about the gesture, the updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data.

[0006] According to a second aspect of the invention, a method performed by a HMD comprises: at least one native or proxy processor of the HMD receiving information indicative of which hand is used to perform a gesture, said to be estimated from first image data, and said at least one native or proxy processor of the HMD not detecting said information indicative of which hand is used to perform said gesture by an image-based detection method; wherein said information indicative of which hand is used to perform said gesture is detected from first data sensed by a sensing device of a mobile device while the HMD is bound to the mobile device and when the mobile device is held by a first hand, and wherein a gesture is performed by a second hand, and said information indicative of which hand is used to perform said gesture is an update indication about said gesture, said update indication about said gesture taking into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data; and at least one native or proxy processor of the HMD performing a side-adapted hand pose estimation on the first image data using said information indicative of which hand is used to perform said gesture.

[0007] According to a third aspect of the invention, a method performed by a mobile device and a HMD comprises: a sensing device of the mobile device sensing first data while the mobile device is bound to the HMD and when the mobile device is held by a first hand; at least one processor of the mobile device detecting information indicative of which hand is used to perform a gesture, said gesture to be estimated by at least one native or proxy processor of the HMD from first image data, wherein said information indicative of which hand is used to perform said gesture is detected from said first data; said at least one processor of the mobile device sending said information indicative of which hand is used to perform said gesture to said at least one native or proxy processor of the HMD; said at least one native or proxy processor of the HMD receiving said information indicative of which hand is used to perform said gesture, and said at least one native or proxy processor of the HMD not detecting said information indicative of which hand is used to perform said gesture by an image-based detection method; wherein said gesture is performed by a second hand, and said information indicative of which hand is used to perform said gesture is an update indication about said gesture, said update indication about said gesture taking into account a hand switching state during a first elapsed time period between a sensing time of said first data and a sensing time of said first image data; and said at least one native or proxy processor of the HMD performing a side-adapted hand pose estimation on said first image data using said information indicative of which hand is used to perform said gesture.

[0008] According to a fourth aspect of the present disclosure, a mobile device comprises: a sensing device configured to perform a step of sensing first data when the mobile device is bound to a head-mounted display (HMD) and when the mobile device is held by a first hand; a memory; and at least one processor coupled to the memory and configured to perform steps comprising: detecting information indicative of which hand is used to perform a gesture to be estimated from first image data by at least one native or proxy processor of the HMD, wherein the information indicative of which hand is used to perform the gesture is not detected from the first image data by the HMD by means of an imaging-based detection method, but from the first data; sending the information indicative of which hand is used to perform the gesture to the at least one native or proxy processor of the HMD, the point in time of performance of the sending step being such that the at least one native or proxy processor of the HMD uses the information indicative of which hand is used to perform the gesture for performing a side-adapted hand pose estimation on the first image data; wherein the gesture is performed by a second hand and the information indicative of which hand is used to perform the gesture is an updated indication about the gesture, the updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a time of sensing of the first data and a time of sensing of the first image data.

[0009] According to a fifth aspect of the present disclosure, a HMD comprises: a memory; and at least one native processor coupled to the memory and configured to perform steps comprising: receiving information indicative of which hand is used to perform a gesture to be estimated from first image data and not detected by means of an image-based detection method; wherein the mobile device detects the information indicative of which hand is used to perform the gesture from first data sensed by a sensing device of the mobile device when the HMD is bound to the mobile device and when the mobile device is held by a first hand; and wherein the gesture is performed by a second hand and the information indicative of which hand is used to perform the gesture is an updated indication about the gesture, the updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a time of sensing of the first data and a time of sensing of the first image data; and performing a side-adapted hand pose estimation on first image data using the information indicative of which hand is used to perform the gesture.

[0010] According to a sixth aspect of the present disclosure, a system comprising a mobile device and a head-mounted display (HMD). The mobile device comprises: a sensing device configured to perform a step of sensing first data when the mobile device is bound to a head-mounted display (HMD) and when the mobile device is held by a first hand; a first memory; and at least one processor coupled to the first memory and configured to perform steps comprising: detecting information indicative of which hand is used to perform a gesture to be estimated from first image data by at least one native or proxy processor of the HMD, wherein the information indicative of which hand is used to perform the gesture is detected from the first data; the at least one processor of the mobile device sending the information indicative of which hand is used to perform the gesture to the at least one native or proxy processor of the HMD. The HMD comprises: a second memory; and at least one native or proxy processor coupled to the second memory and configured to perform steps comprising: receiving the information indicative of which hand is used to perform the gesture, and not detecting the information indicative of which hand is used to perform the gesture by an image-based detection method; wherein the gesture is performed by a second hand, and the information indicative of which hand is used to perform the gesture is an updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data; and wherein the steps performed by the at least one native or proxy processor of the HMD further comprise: performing a side-adapted hand pose estimation on the first image data using the information indicative of which hand is used to perform the gesture. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate embodiments of the present disclosure or related art, the following drawings used in the description of embodiments will be briefly described. It is obvious that the drawings are only some embodiments of the present disclosure, and a person having ordinary skill in the art can obtain other drawings from these drawings without making creative efforts.

[0012] Figure 1 is a block diagram illustrating an existing hand pose estimation using an imaging-based detection method to detect information of which hand is used to perform a gesture from image data of the gesture.

[0013] Figure 2 is a flowchart of a method performed by a mobile device for estimating a gesture according to some embodiments of the present disclosure.

[0014] Figure 3is a flowchart of a method for estimating a gesture performed by a head-mounted display (HMD) according to some embodiments of the present disclosure.

[0015] Figure 4 and Figure 5 includes a flowchart of steps to cooperatively implement information indicating which hand is used to perform a gesture in connection with an update indication of the gesture according to some embodiments of the present disclosure, taking into account that no hand switch is possible during a first elapsed time period when the first elapsed time period is shorter than a longest time period in which a hand switch is not possible, and when the first elapsed time period is equal to the longest time period in which a hand switch is not possible.

[0016] Figure 6 and Figure 7 includes a flowchart of steps to cooperatively implement information indicating which hand is used to perform a gesture in connection with an update indication of the gesture according to some embodiments of the present disclosure, taking into account that each of a first occurrence of a hand switch is detected within a first elapsed time period when the first elapsed time period is shorter than a longest time period in which a hand switch is not possible, when the first elapsed time period is equal to the longest time period in which a hand switch is not possible, and when the first elapsed time period is longer than the longest time period in which a hand switch is not possible.

[0017] Figure 7 and Figure 8 includes a flowchart of steps to cooperatively implement information indicating which hand is used to perform a gesture in connection with an update indication of the gesture according to some embodiments of the present disclosure, taking into account that no hand switch is possible during a first elapsed time period when the first elapsed time period is shorter than a longest time period in which a hand switch is not possible, and when the first elapsed time period is equal to the longest time period in which a hand switch is not possible, and each of a second occurrence of a hand switch is detected within the first elapsed time period when the first elapsed time period is longer than the longest time period in which a hand switch is not possible.

[0018] Figure 9 includes a flowchart of steps related to use mode data and validation data for detecting information indicating which hand is used to perform a gesture according to some embodiments of the present disclosure.

[0019] Figure 10 includes a flowchart of steps performed by a mobile device and corresponding steps performed by an HMD in Figure 9

[0020] Figure 11 and Figure 12 shows illustrative examples for some embodiments in which preferred features of validation data based on idle hand pose image data and preferred features of touch input mode data are used. ​

[0021] Figure 13 is a flowchart of steps related to using data reflecting orientation of a mobile device to detect information indicating which hand is used to perform a gesture, in accordance with some embodiments of the present disclosure.

[0022] Figure 14 illustrates an illustrative example for some embodiments of features in which data reflecting orientation of a mobile device is used.

[0023] Figure 15 is a flowchart of steps related to using image data sensed by an imaging device of a mobile device to detect information indicating which hand is used to perform a gesture, in accordance with some embodiments of the present disclosure.

[0024] Figure 16 illustrates an illustrative example for some embodiments of features in which image data sensed by an imaging device of a mobile device is used.

[0025] Figure 17 illustrates a timeline diagram of parallel processing of a portion of a method performed by at least one native or proxy processor of a HMD and a portion of a method performed by at least one processor of a mobile device, based on some embodiments of the present disclosure.

[0026] Figure 18 illustrates a timeline diagram of parallel processing of a portion of a method performed by at least one native or proxy processor of a HMD and a portion of a method performed by at least one processor of a mobile device, based on some embodiments of the present disclosure.

[0027] Figure 19 illustrates a diagram of a system in which the methods described herein can be implemented, in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] As used herein, when at least one operation is referred to as being “used,” “from,” “on,” or “based on” at least one object, the at least one operation is performed “directly using,” “directly from,” “directly on,” or “directly based on” the at least one object, or there can be at least one intervening operation. In contrast, when at least one operation is referred to as being “directly used,” “directly from,” “directly on,” or “directly based on” at least one object, there are no intervening operations.

[0029] As used herein, when at least one operation is referred to as being “in response to” at least one other operation, the at least one operation is performed “directly in response to” the at least one other operation, or there can be at least one intervening operation. In contrast, when at least one operation is referred to as being performed “directly in response to” at least one other operation, there are no intervening operations.

[0030] Referring to Figure 1 , Figure 1is a block diagram illustrating an existing hand pose estimation that uses an imaging-based detection method to detect from image data of a hand gesture information which hand was used to perform the hand gesture. An example of the existing hand pose estimation 100 can be found in “Learning to Estimate 3D hand pose from Single RGB Images” by C. Zimmermann and T. Brox in ICCV 2017. The existing hand pose estimation 100 receives image data 112 of the hand gesture and is configured to find from the image data a set of joints of a hand 122 that was used to perform the hand gesture. The existing hand pose estimation 100 includes an object detection network 102 and a side-adapted hand pose estimation module 110. The object detection network 102 includes a hand detection module 104, a hand image cropping module 106, and a hand gesture performing hand information detection module 108. The object detection network 102 receives the image data 112 and is configured to detect from the image data 112 information which hand was used to perform the hand gesture. As such, the hand gesture performing hand information detection module 108 is implemented by a deep learning model of object detection (i.e., an imaging-based detection method). More specifically, the object detection network 102 is first trained offline and then used for online inference. The hand detection module 104 receives the image data 112 and is configured to output during online inference a hand presence probability 114 (ranging from 0 to 1) and a hand location 116 (e.g., a bounding box around the hand in the image data 112). The hand presence probability 114 determines whether the existing hand pose estimation 100 will perform subsequent parts of the existing hand pose estimation 100. For example, if the hand presence probability 114 is below a certain threshold (e.g., 0.5), then the subsequent parts of the existing hand pose estimation 100 will not be performed. If the hand presence probability 114 is above the certain threshold, then the hand image cropping module 106 receives the image data 112 and the hand location 116 and is configured to crop the image data 112 using the hand location 116 to generate cropped image data 118. Thus, the subsequent parts of the existing hand pose estimation 100 focus only on the hand. Because a smaller number of background pixels are processed, better hand pose estimation accuracy results. The hand gesture performing hand information detection module 108 receives the cropped image data 118 and is configured to determine from the cropped image data 118 information which hand was used to perform the hand gesture 120 (i.e., a probability range that the hand is a left hand or a right hand ranging from 0 to 1). For a type of side-adapted hand pose estimation module 110, the information which hand was used to perform the hand gesture 120 is used to determine whether to use a right hand pose estimation or a left hand pose estimation to perform a side-dependent hand pose estimation on the side-dependent cropped image data 118.For another type of side-adapted hand pose estimation module 110, the information of which hand is used to perform the gesture 120 is used to determine whether to flip the cropped image data 118 along the y-axis to perform side-dependent hand pose estimation on the cropped image data that is independent of the side. The side-adapted hand pose estimation module 110 outputs a set of joints of the hand 122 used to perform the gesture.

[0031] For the existing hand pose estimation 100 that uses imaging-based detection methods to detect the information of which hand is used to perform the gesture from the image data of the gesture, the overall speed of the hand pose estimation method is slowed down since the step of detecting the information of which hand is used to perform the gesture performed by the gesture-performing hand information detection module 108 is at least one of a sequential processing step and a slower step. When a head-mounted display (HMD) is tethered to a mobile device, and the mobile device is additionally used as an input device for a user to interact with a virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) environment presented in the HMD, using the existing hand pose estimation 100 cannot recognize that using the mobile device to detect the information indicating which hand is used to perform the gesture improves the speed of the existing hand pose estimation 100 as proposed by the present disclosure.

[0032] According to the first embodiment of the present disclosure, the mobile device can be used to detect information of which hand holds the mobile device from the data sensed by the mobile device. Since the data sensed by the mobile device is separate from the image data of the gesture, in order to make the information of which hand holds the mobile device provide an indication of which hand is used to perform the gesture updated with respect to the hand switch, the hand switch state during the elapsed time period from the sensing time of the data sensed by the mobile device to the sensing time of the image data of the gesture needs to be impossible for the hand switch or needs to be possible for the hand switch, and the occurrence of each hand switch during the elapsed time period is detected. In this way, since when the hand switch is impossible, the hand that performs the gesture must be opposite to the hand that holds the mobile device, and when the hand switch is possible and the occurrence of each hand switch within the elapsed time period is detected, the hand that performs the gesture must be opposite to the last hand that holds the mobile device inferred from the information of which hand holds the mobile device and the number of occurrences of the hand switch, the information of which hand is used to perform the gesture can be inferred from the information of which hand holds the mobile device. By parallel processing the HMD-performed part of the hand pose estimation and the mobile device-performed part of the hand pose estimation, the deficiency that the step of detecting the information of which hand performs the gesture of the existing hand pose estimation 100 is a sequential processing step can be improved. In addition, for the existing hand pose estimation 100, the step of detecting the information of which hand performs the gesture is a slower step, which deficiency can be improved by using the type of data sensed by the mobile device that leads to detecting the information of which hand holds the mobile device faster compared to the imaging-based detection method. Any solution to improve this deficiency can be used in combination with sequentially processing the step of detecting the information of which hand performs the gesture in the hand pose estimation, or parallel processing the HMD-performed part of the hand pose estimation and the mobile device-performed part of the hand pose estimation.

[0033] Referring to Figure 2 and Figure 3 , Figure 2 is a flowchart of a method 200 performed by the mobile device 1904 for estimating a gesture according to the first embodiment of the present disclosure, Figure 3 is a flowchart of a method 300 performed by the HMD 1902 for estimating a gesture.

[0034] According to the first embodiment of the present disclosure, the method 200 performed by the mobile device 1904 comprises the following steps.

[0035] In step 202, when the mobile device 1904 is bound to the HMD 1902 and when the mobile device 1904 is held by a first hand, a sensing device of the mobile device 1904 senses first data. In the embodiment of the sensing device to be described with reference to Figures 9 to 12 , the sensing device is a touch screen 1918. In the embodiments to be described with reference to Figure 13 andFigure 14 In another embodiment of the described sensing device, the sensing device is at least one first inertial sensor 1920. In the following, reference will be made to Figure 15 and Figure 16 In yet another embodiment of the described sensing device, the sensing device is an imaging device 1922.

[0036] In step 212, the at least one processor 1926 of the mobile device 1904 detects information indicative of which hand is used to perform the gesture estimated from the first image data. The information indicative of which hand is used to perform the gesture is detected from the first data.

[0037] In step 222, the at least one processor 1926 of the mobile device 1904 transmits the information indicative of which hand is used to perform the gesture to the at least one native or proxy processor 1912 or 1926 of the HMD 1902.

[0038] The gesture performed by the second hand, the information indicative of which hand is used to perform the gesture is an updated indication about the gesture that takes into account the hand switching state during a first elapsed time period between the sensing time of the first data and the sensing time of the first image data.

[0039] According to a first embodiment of the present disclosure, the method 300 performed by the HMD 1902 comprises the following steps.

[0040] In step 332, the imaging device 1908 of the HMD 1902 senses first image data. Preferably, the imaging device 1908 is an outward-facing on-board imaging device of the HMD 1902. Optionally, the first image data is sensed by a plurality of imaging devices arranged in the space in which the user is located. Illustratively, the first image data is a single image. Optionally, the first image data is a plurality of images or a plurality of video frames.

[0041] In step 342, in response to step 222, the information indicative of which hand is used to perform the gesture to be estimated from the first image data is received by the at least one native or proxy processor 1912 or 1926 of the HMD 1902 and the information is not detected by the at least one native or proxy processor 1912 or 1926 of the HMD 1902 from the first image data by an image-based detection method.

[0042] As used herein, the term “image-based detection method” refers to a method of performing object detection on image data. The image data can be produced by the imaging device 1908 in Figure 19 In the existing hand pose estimation 100, the deep learning-based object detection is an example of an image-based detection method. Other types of object detection are within the intended scope of the present disclosure.

[0043] In step 352, the at least one native or proxy processor 1912 or 1926 of the HMD 1902 uses the information indicative of which hand was used to perform the gesture to perform side- adapted hand pose estimation on the first image data.

[0044] The gesture is performed by the second hand, the information indicative of which hand was used to perform the gesture is an updated indication about the gesture that takes into account the hand switching state during the first elapsed time period between the sensing time of the first data and the sensing time of the first image data of the gesture.

[0045] Similarly to the existing hand pose estimation 100, the method 300 performed by the HMD 1902 exemplarily further comprises a step of hand detection similar to the step performed by the hand detection module 104, and a step of cropping the first image data similar to the step performed by the hand image cropping module 106 after the step 332 in Figure 3 and before the step 352 in Figure 3 .

[0046] The at least one step further comprised by the method 200 and / or each of the at least one step in the method 200 further comprised at least a subset of steps cooperates with each of the at least one step further comprised by the method 300 and / or the at least one step comprised in the method 300 comprises at least one subset of steps to achieve that the information indicative of which hand was used to perform the gesture is an updated indication about the gesture that takes into account the hand switching state during the first elapsed time period.

[0047] Illustratively, the hand switching state during the first elapsed time period is that a hand switch is impossible to occur when the first elapsed time period is shorter than and when the first elapsed time period is equal to a longest time period during which a hand switch is impossible to occur. This longest time period during which a hand switch is impossible to occur is a longest time period that is not enough to occur that the second hand becomes this possible to the first hand after the mobile device 1904 is switched to be held by the relative hand of the first hand. The detailed implementation will be described with reference to Figure 4 and Figure 5 .

[0048] For the longest time period during which a hand switch is impossible to occur, when the sensing time of the first data and the sensing time of the first image data of the gesture sensed by the mobile device 1904 overlap with each other, it is determined that a hand switch is impossible to occur during the first elapsed time period. When the sensing time of the first data and the sensing time of the first image data of the gesture sensed by the mobile device 1904 do not overlap with each other, this longest time period during which a hand switch is impossible to occur (i.e. a threshold) is determined by empirical study. An example of how to use the threshold will be described in step 420 in Figure 4 .

[0049] Figure 4 and Figure 5 The flowchart includes a step of collaboratively implementing information indicating which hand is used to perform a gesture, which is an update instruction on the gesture, taking into account that a hand switch cannot occur during the first elapsed time period when the first elapsed time period is shorter than the longest time period during which a hand switch is impossible and when the first elapsed time period is equal to the longest time period during which a hand switch is impossible. Figure 4 The flowchart includes at least one step further included in method 200 and / or at least one subset of steps included in each step of at least one step in method 200. Figure 5 The flowchart includes at least one step further included in method 300 and / or each subset of at least one step included in at least one step in method 300.

[0050] Reference Figure 4 , Figure 2 Step 212 includes steps 414 and 416 executed by at least one processor 1926 of the mobile device 1904. Method 200 also includes steps 418, 420, and 424 executed by at least one processor 1926 of the mobile device 1904. Step 422 is... Figure 2 An illustrative implementation of step 222. See reference. Figure 5 Method 300 further includes steps 540, 550, and 554 performed by at least one of the proprietary or proxy processors 1912 or 1926 of HMD 1902. The "Yes" branch of step 550 is... Figure 3 An illustrative implementation of step 342. Step 552 is... Figure 3 An illustrative implementation of step 352.

[0051] In step 414, information about which hand is holding the mobile device is detected from the first data. Illustratively, the detection can be performed at a fixed predetermined frequency, such as sixty times per second. Optionally, method 300 further performs hand tracking, which continuously associates the hand in the current frame with the hand in the previous frame. When it is determined that the hand in the current frame is the same as the hand in the previous frame, the detection is skipped.

[0052] In step 416, information about which hand is used to perform the gesture is obtained based on which hand is holding the mobile device.

[0053] Step 540 Figure 3 The process proceeds after step 332. In step 540, at least one of the proprietary or proxy processors 1912 or 1926 of the HMD 1902 requests the latest detection results and sends at least one system timestamp of the first image data.

[0054] In step 418, responsive to step 540, the latest detection result is obtained by the at least one processor 1926 of the mobile device 1904, where the latest detection result is information of which hand is used to perform the gesture. From the description of steps 414 and 416, many detection results are obtained over time. Any one of these detection results is obtained in the same manner as step 202 and step 212 in FIG. 2A. To condition that the first elapsed time period is shorter than or equal to the longest time period in which it is not possible to satisfy hand switching, the latest detection result is used in step 418. Step 202 and step 212 produce the latest detection result. Figure 2

[0055] In step 420, a determination is made by the at least one processor 1926 of the mobile device 1904 whether a first elapsed time period from at least one system timestamp of first data to at least one system timestamp of first image data satisfies a threshold. The at least one system timestamp of the first data is recorded by a sensing device of the mobile device 1904 that senses the first data. The at least one system timestamp of the first image data is recorded by the imaging device 1908 of the HMD 1902 that senses the first image data. Illustratively, the threshold is an absolute upper limit on the first elapsed time period, so the threshold is a real number. Alternatively, the threshold is a relative upper limit on the first elapsed time period, so the threshold is expressed in a ratio or percentage.

[0056] If the condition in step 420 is satisfied, then in step 422, the latest detection result is sent by the at least one processor 1926 of the mobile device 1904 to the at least one native or proxy processor 1912 or 1926 of the HMD 1902. Step 422 is an illustrative implementation of step 222 in FIG. 2A. Figure 2

[0057] If the condition in step 420 is not satisfied, then in step 424, the latest detection result is not sent by the at least one processor 1926 of the mobile device 1904 to the at least one native or proxy processor 1912 or 1926 of the HMD 1902.

[0058] In step 550, responsive to step 422 or 424, a determination is made whether the latest detection result is received. The "yes" branch of step 550 is an illustrative implementation of step 342 in FIG. 3A. Figure 3

[0059] ​​​In step 552, a side-adapted hand pose estimation is performed on the first image data using the latest detection results by at least one of the own or proxy processors 1912 or 1926 of the HMD 1902. The side-adapted hand pose estimation is similar to the steps performed by any type of side-adapted hand pose estimation module 110 of the existing hand pose estimation 100. Thus, the latest detection results are updated with respect to the gesture. Step 552 is Figure 3 the illustrative implementation of step 352.

[0060] If the condition in step 550 is satisfied, then in step 554, a side-adapted hand pose estimation is performed on the first image data using the user's dominant hand by at least one of the own or proxy processors 1912 or 1926 of the HMD 1902. The user's dominant hand (i.e., the hand that the user uses most often) can be explicitly specified by the user.

[0061] Optionally, when the first elapsed time period is shorter than the longest time period in which a hand switch is not possible, when the first elapsed time period is equal to the longest time period in which a hand switch is not possible, and when the first elapsed time period is longer than the longest time period in which a hand switch is not possible, the hand switch status during the first elapsed time period is that each of the first occurrences of a hand switch is detected during the first elapsed time period. The detailed implementation will be described with reference to Figure 6 and Figure 7 .

[0062] Figure 6 and Figure 7 The flowchart of the steps of updating the indication with respect to the gesture in coordination with implementing the information indicating which hand is used to perform the gesture includes each of the subsets of at least one of the steps that the method 200 further includes and / or the subset of at least one of the steps in the method 200 includes. Figure 6 The flowchart of the steps of updating the indication with respect to the gesture in coordination with implementing the information indicating which hand is used to perform the gesture includes each of the subsets of at least one of the steps that the method 200 further includes and / or the subset of at least one of the steps in the method 200 includes. Figure 7 The flowchart of the steps of updating the indication with respect to the gesture in coordination with implementing the information indicating which hand is used to perform the gesture includes each of the subsets of at least one of the steps that the method 300 further includes and / or the subset of at least one of the steps in the method 300 includes.

[0063] Reference is made to Figure 6 , Figure 2Step 212 in includes steps 614, 6204, 6206, 6208, and 616 performed by the at least one processor 1926 of the mobile device 1904. The method 200 also includes step 6202 performed by the at least one second inertial sensor 1924 of the mobile device 1904, the second data. Step 622 includes portions of the method 200 and as Figure 2 part of the illustrative implementation of step 222 in Figure 7 , the method 300 also includes step 740 performed by the at least one native or proxy processor 1912 or 1926 of the HMD 1902. Step 742 is Figure 3 part of the illustrative implementation of step 342 in Figure 3 part of the illustrative implementation of step 352 in

[0064] Step 614 is performed after step 202. In step 614, the at least one processor 1926 of the mobile device 1904 detects from the first data information which hand holds the mobile device. Step 614 is similar to Figure 4 step 414 in

[0065] The loop formed by steps 6202, 6204, and 6206 starts at the time of sensing of the first data in step 202. In step 6202, the at least one second inertial sensor 1924 of the mobile device 1904 senses the second data.

[0066] In step 6204, the at least one processor 1926 of the mobile device 1904 detects from the second data occurrence of a hand switch.

[0067] In step 6206, the at least one processor 1926 of the mobile device 1904 determines whether a request for the latest detection result and at least one system time stamp of the first image data is received. If the condition in step 6206 is not met, the method 200 loops back to step 6202.

[0068] Step 740 is performed after step 332 in Figure 3 . In step 740, the at least one native or proxy processor 1912 or 1926 of the HMD 1902 requests the latest detection result and sends at least one system time stamp of the first image data. Then, the condition in step 6206 is met and the method 200 proceeds to step 6208.

[0069] In step 6208, the at least one processor 1926 of the mobile device 1904 obtains the detected occurrence of each hand switch during the first elapsed time period.

[0070] In step 616, the at least one processor 1926 of the mobile device 1904 obtains information of which hand is used to perform the gesture from the information of which hand holds the mobile device and the detected occurrences of each hand switch during the first elapsed time period.

[0071] In step 622, the at least one processor 1926 of the mobile device 1904 obtains the latest detection result and sends the latest detection result to the at least one native or proxy processor 1912 or 1926 of the HMD 1902. The latest detection result is the information of which hand is used to perform the gesture. The obtaining part of step 622 is similar to step 418. The sending part of step 622 is Figure 2 the illustrative embodiment of step 222. Using the latest detection result is less important than the embodiments described with reference to Figure 4 and Figure 5 each occurrence of hand switch during the first elapsed time period is detected.

[0072] In step 742, the at least one native or proxy processor 1912 or 1926 of the HMD 1902 receives the latest detection result in response to step 622.

[0073] In step 752, the at least one native or proxy processor 1912 or 1926 of the HMD 1902 performs side- adapted hand pose estimation on the first image data using the latest detection result. Step 752 is similar to step 552.

[0074] Optionally, when the first elapsed time period is shorter than the longest time period during which no hand switch is possible, and when the first elapsed time period is equal to the longest time period during which no hand switch is possible, the hand switch status during the first elapsed time period is that no hand switch is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which no hand switch is possible, the hand switch status during the first elapsed time period is that each second occurrence of hand switch is detected during the first elapsed time period. Detailed embodiments will be described with reference to Figure 7 and Figure 8

[0075] Figure 7 and Figure 8 ​The flowchart including the step of cooperatively implementing information indicative of which hand to use to perform the gesture is an update indication regarding the gesture that takes into account that no hand switch is possible during the first elapsed time period when the first elapsed time period is shorter than a longest time period in which a hand switch is not possible, and when the first elapsed time period is equal to the longest time period in which a hand switch is not possible, and that each second hand switch is detected during the first elapsed time period when the first elapsed time period is longer than the longest time period in which a hand switch is not possible. Figure 8 The flowchart of

[0076] Referring to Figure 8 , Figure 2 Step 212 in includes steps 614, 6204, 6206, 820, 8162, 8208, and 8164 performed by the at least one processor 1926 of the mobile device 1904. The method 200 further includes step 6202 performed by the at least one second inertial sensor 1924 of the mobile device 1904, the second data. Each of steps 8222 and 8224 includes a portion of the method 200 further comprising and as a portion of the illustrative implementation of step 222 in Figure 2 The illustrative implementation of step 614, 6202, 6204, and 6206 is described above in reference to Figure 6 Step 742 is described below. Other steps of Figure 7 have been described above. The description of other steps of Figure 7 is omitted here.

[0077] In step 820, a determination is made by the at least one processor 1926 of the mobile device 1904 whether a first elapsed time period from the at least one system timestamp of the first data to the at least one system timestamp of the first image data satisfies a threshold. Step 820 is similar to step 420.

[0078] If the condition in step 820 is satisfied, steps 8162 and 8222 are performed. Step 8162 is similar to step 416. Step 8222 includes a portion similar to step 418 and a portion similar to step 422.

[0079] If the condition in step 820 is not satisfied, steps 8208, 8164, and 8224 are performed. Steps 8208, 8164, and 8224 are similar to steps 6208, 616, and 622, respectively.

[0080] In step 742, the latest detection result is received by at least one of the own or proxy processors 1912 or 1926 of the HMD 1902 in response to step 8222 or 8224.

[0081] The detection methods and alternative detection methods described above for step 414 (and similarly for step 614) are illustrative. Optionally, the detection can be triggered after the hand detection step in the method 300 described above. For example, in the flowcharts of Figure 4 and Figure 5 , the request step 540 is replaced by the trigger step to which the detection step 414 in Figure 4 responds. For another example, in the flowcharts of Figure 6 and Figure 7 , the request step 740 is replaced by the trigger step of the detection step 614 and the step of detecting, by at least one processor of the mobile device, the occurrence of each hand switch during the first elapsed time period from the second data in response thereto, similar to the flowcharts of Figure 7 and Figure 8 . The step of detecting the occurrence of each hand switch during the first elapsed time period replaces steps 6204, 6206, and 6208.

[0082] Illustratively, the latest detection result is selected from the detection results retained over time. Optionally, only the latest detection result is retained. For example, in the flowcharts of Figure 4 and Figure 5 . In step 418, the latest detection result is obtained in either of the two ways described above. For another example, in the flowcharts of Figure 6 and Figure 7 , the latest detection result is obtained in either of the two ways in step 622. For yet another example, in the flowcharts of Figure 7 and Figure 8 , the latest detection result is obtained in either of the two ways in step 8222 and step 8224.

[0083] The use of the latest detection result is illustrative. Optionally, an earlier detection result that satisfies at least one condition set by considering the hand switch status during the first elapsed time period described with reference to Figures 4 to 5 , Figures 6 to 7 or Figures 7 to 8 may also be used.

[0084] The determination by the mobile device of whether at least one condition set by considering the hand switch status during the first elapsed time period is illustrative. Optionally, the determination of whether at least one condition set by considering the hand switch status during the first elapsed time period can be performed by the HMD 1902. For example, in the flowcharts of Figure 4 and Figure 5In the flowchart, step 422 is executed as soon as the latest detection result is obtained in step 418. Step 420 is executed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902, replacing step 550. As another example, in Figure 4 and Figure 5 In the flowchart, the request for the latest detection result by HMD 1902 in step 418 can also be replaced by sending the detection result to HMD 1902 when the mobile device 1904 has access to a new detection result. Thus, Figure 3 The execution time of step 342 depends on the execution time of step 222, and can occur before or after step 332. For example, in... Figure 6 and Figure 7 In the flowchart, any one of steps 6204, 6206, and 6208 can be adapted and executed by HMD 1902. For example, in... Figure 7 and Figure 8 In the flowchart, any one of steps 6204, 6206, and 8208 can be adapted and executed by HMD 1902.

[0085] exist Figure 2 The information indicating which hand is used to perform the gesture in steps 222 and 342 is illustrative, derived from information about which hand is holding the mobile device, taking into account the hand switching states during the first elapsed time period. Optionally, in Figure 2 The information indicating which hand is used to perform the gesture in steps 222 and 342 is information about which hand is holding the mobile device. For example, in Figure 4 and Figure 5 In the flowchart, the latest detection result sent in step 422 is information about which hand is holding the mobile device in step 414. Step 416 is executed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902. For example, in... Figure 6 and Figure 7 In the flowchart, the latest detection result sent in step 622 is information about which hand is holding the mobile device in step 614. Step 616 is executed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902. For example, in... Figure 7 and Figure 8 In the flowchart, the latest detection result sent in step 8222 or 8224 is information about which hand is holding the mobile device in step 614. Steps 8162 and 8164 are executed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902.

[0086] To improve the step of detecting information which hand is used to perform a gesture, any type of data sensed by the mobile device 1904 or a combination of any type of data sensed by the mobile device 1904 and corroborating data reflecting which hand holds the mobile device 1904 can be used alone. To improve the step of detecting information which hand is used to perform a gesture, any type of data sensed by the mobile device 1904 or a combination of any type of data sensed by the mobile device 1904 and corroborating data reflecting which hand holds the mobile device 1904 can be used alone, which can achieve faster detection of information which hand holds the mobile device compared to imaging-based detection methods. The second embodiment, the third embodiment, and the fourth embodiment described below provide examples of different types of data sensed by the mobile device 1904 suitable for the purpose of reflecting which hand holds the mobile device 1904. Further advantages of some of the different types of data sensed by the mobile device 1904 are described in connection with the corresponding embodiments.

[0087] A first type of data sensed by the mobile device 1904 is pattern data caused by a hand. The hand belongs to the hand holding the mobile device 1904 and does not participate in holding the mobile device 1904. The hand is movable when the hand holding the mobile device 1904 has the hand. However, since the hand is also movable when the opposite hand of the hand holding the mobile device 1904 holds the mobile device 1904, it cannot be determined from the pattern data alone that the pattern data is caused by the hand when the hand holding the mobile device 1904 has the hand. Therefore, for this type of data, corroborating data based on image data of the other hand not holding the mobile device 1904 is needed to achieve higher confidence in the information which hand holds the mobile device.

[0088] Referring to Figure 9 , Figure 9 A flowchart including steps related to using pattern data and corroborating data for detecting information indicating which hand is used to perform a gesture according to the second embodiment of the present disclosure. The second embodiment is based on the first embodiment, so the same content as the first embodiment is omitted here. Step 904 is a step included in the method 300 in Figure 3 The steps related to using pattern data and corroborating data for detecting information indicating which hand is used to perform a gesture include the following steps.

[0089] Step 902 is a step included in the method 300 in Figure 2An embodiment of step 202. In step 902, first data is sensed by a sensing device of the mobile device 1904 when the mobile device 1904 is bound to the HMD 1902 and when the mobile device 1904 is held by a first hand. The first data pattern data is caused by the hand. The hand belongs to the first hand holding the mobile device 1904 and does not participate in holding the mobile device 1904.

[0090] In step 904, second image data of a third hand not holding the mobile device 1904 is sensed by an imaging device 1908 of the HMD 1902. Exemplarily, step 904 can be performed similar to step 332 in Figure 3

[0091] Step 912 is an embodiment of step 212 in Figure 2 Step 912 is performed after step 902. In step 912, in response to step 904, the at least one processor 1926 of the mobile device 1904 detects information indicative of which hand is used to perform the gesture to be estimated from the first image data. The information indicative of which hand is used to perform the gesture is detected from the first data and further based on the second image data. A second elapsed time period between the sensing time of the first data and the sensing time of the second image data is insufficient for a third hand to become a first hand after the mobile device 1904 is switched to be held by a relative hand of the first hand to occur. The second elapsed time period between the sensing time of the first data and the sensing time of the second image data is insufficient for a condition for a hand switch to be possible to occur similar to the first elapsed time period described in Figure 4 and Figure 5 The second elapsed time period is similarly checked.

[0092] Referring to Figure 10 , Figure 10 A flowchart comprising step 912 performed by the mobile device 1904 and the corresponding step performed by the HMD 1902. Steps 1006 and 1008 are steps performed by the mobile device 1904 and the corresponding steps performed by the HMD 1902 are steps performed by the HMD 1902. Figure 9 Steps 1006 and 1008 are steps performed by the mobile device 1904 and the corresponding steps performed by the HMD 1902 are steps performed by the HMD 1902. Figure 3 The method 300 in

[0093] Step 1006 is performed after step 904. In step 1006, hand detection is performed on the second image data by the at least one own or proxy processor 1912 or 1926 of the HMD 1902. Step 1006 is similar to step performed by the hand detection module 104 of the existing hand pose estimation 100.

[0094] Step 912 comprises steps 1014, 1015 and 1016.

[0095] ​Step 1014 is similar to step 414 in Figure 4

[0096] In step 1015, attestation data of the first data is requested by the at least one processor 1926 of the mobile device 1904, and at least one system timestamp of the first data is sent.

[0097] In step 1008, in response to step 1015, the hand detection result of the second image data as the attestation data is sent by the at least one native or proxy processor 1912 or 1926 of the HMD 1902, wherein a second elapsed time period from the at least one system timestamp of the first data to the at least one system timestamp of the second image data satisfies the threshold. Details about the at least one system timestamp of the image data and the threshold have been provided above for step 420, and are omitted here. Thus, the second elapsed time period between the sensing time of the first data to the sensing time of the second image data is not enough for the third hand to become the first hand after the mobile device 1904 is switched to be held by the relative hand of the first hand.

[0098] In step 1016, in response to step 1008, the at least one processor 1926 of the mobile device 1904 obtains the information of which hand is used to perform the gesture according to the information of which hand holds the mobile device 1904 and the attestation data based on the second image data.

[0099] Preferably, the sensing time of the mode data and the sensing time of the attestation data based on the image data of the hand not holding the mobile device 1904 overlap with each other (i.e., at least partially synchronized), because the confidence of the information of which hand holds the mobile device is the highest. This is possible because when the hand holding the mobile device 1904 uses the mobile device 1904 to perform an operation resulting in the generation of the mode data, the other hand not holding the mobile device 1904 can be waiting in the air to perform the gesture (i.e., perform the idle hand gesture), as explained below using illustrative examples.

[0100] ​In addition, the mode data is preferably touch input mode data. When the HMD 1902 is tethered to the mobile device 1904, using the mobile device 1904 as an input device can work in concert with performing gestures by the other hand that is not holding the mobile device 1904 to provide user interaction with a VR, AR, MR, or XR environment rendered in the HMD 1902. This type of user interaction is referred to herein as a "multimodal interaction paradigm." Because of the multimodal interaction paradigm, the touch input mode data can be used to detect which hand is holding the mobile device using the touch input mode data. In addition, because of the multimodal interaction paradigm, if the elapsed time period from the sensing time of the touch input mode data to the sensing time of the image data of the hand gesture is frequently less than or equal to the longest time period in which hand switching is not possible, as explained below using illustrative examples. In addition, using the touch input mode data to detect which hand is holding the mobile device is faster, more accurate, and consumes less power compared to imaging-based detection methods.

[0101] The preferred features of the attestation data based on the above idle hand pose image data can be implemented independently of the preferred features of the touch input mode data. However, when they are implemented together, because of the multimodal interaction paradigm, it is likely that when the hand holding the mobile device 1904 uses the mobile device 1904 to perform an operation that results in the generation of touch input mode data, the other hand that is not holding the mobile device 1904 is waiting in the air to perform a hand gesture, as explained below using illustrative examples.

[0102] Reference is made to Figure 11 and Figure 12 , Figure 11 and Figure 12 An illustrative example of a second embodiment is shown in which the preferred features of the attestation data based on the idle hand pose image data and the preferred features of the touch input mode data are used. Figure 11 An illustrative example of a second embodiment is shown in which the preferred features of the attestation data based on the idle hand pose image data and the preferred features of the touch input mode data are used. Figure 12 The next in-air tab gesture 12020 shown. The HMD 11002, the mobile device 11004, and the touch screen 11018 correspond to the HMD 1902, the mobile device 1904, and the touch screen 1918, respectively, in Figure 19 The next in-air tab gesture 12020 shown. The HMD 11002, the mobile device 11004, and the touch screen 11018 correspond to the HMD 1902, the mobile device 1904, and the touch screen 1918, respectively, in

[0103] Reference is made toFigure 9 , Figure 10 and Figure 11 In step 902, the touch screen 11018 of the mobile device 11004 senses first data when the mobile device 11004 is bound to the HMD 11002 and when the mobile device 11004 is held by a first hand (e.g., the left hand 11008), the first data being pattern data caused by a hand part (e.g., the thumb 11006). The hand part belongs to the left hand 11008 that holds the mobile device 11004 and does not participate in the part that holds the mobile device 11004. Because the user 11016 uses the touch screen 11018 of the mobile device 11004 to perform the swipe input 11022, the sensing device is the touch screen 11018, and the pattern data is touch input pattern data. Moreover, in the example in Figure 11 , the hand part is the thumb 11006 of the left hand 11008, and the mobile device 11004 is held from below by the left hand 11008. It is more ergonomic to use the thumb 11006 of the left hand 11008 to perform the swipe input 11022 while the mobile device 11004 is held from below by the left hand 11008.

[0104] In step 904, the imaging device 1908 of the HMD 11002 senses second image data of a third hand (e.g., the right hand 11010) that does not hold the mobile device 11004. The right hand 11010 performs the idle hand pose 11020.

[0105] In step 912, the at least one processor 1926 of the mobile device 11004 detects information indicating which hand is used to perform a gesture to be estimated from the first image data, wherein the information indicating which hand is used to perform the gesture is detected from the touch input pattern data and further based on the second image data. Details of step 912 are provided below.

[0106] In step 1014, the at least one processor 1926 of the mobile device 11004 detects information of which hand holds the mobile device from the touch input pattern data. For example, the touch input pattern data is a trajectory of a swipe input 11022 using the touchscreen 11018. For details of information of using the trajectory of the swipe input 11022 to detect which hand holds the mobile device, see “Detecting Handedness from Device Use”, Collarbone, stack overflow, 2015, https: / / stackoverflow.com / questions / 27720226 / detecting-handedness-from-device-use. For another example, the touch input pattern data is raw capacitance data of the swipe input 11022 using the touchscreen 11018. For details of information of using the raw capacitance data of the swipe input 11022 to detect which hand holds the mobile device, see “Investigating the feasibility of finger identification on capacitive touchscreens using deep learning”, Huy Viet Le, Sven Mayer, and Niels Henze, UUI’19: Proceedings of the 24th International Conference on Intelligent User Interfaces.

[0107] In step 1015, the at least one processor 1926 of the mobile device 11004 requests attestation data of the touch input pattern data, and sends at least one system timestamp of the touch input pattern data.

[0108] In step 1006 following step 904, the at least one native or proxy processor 1912 or 1926 of the HMD 11002 performs hand detection on the second image data. Because the second image data is of the idle hand pose 11020 of the right hand 11010, the hand detection result indicates that a hand is present.

[0109] In step 1008, at least one of the HMD 11002's own or proxy processors 1912 or 1926 sends the hand detection result for the second image data as the validation data, where a second elapsed time period from the at least one system timestamp of the touch input mode data to the at least one system timestamp of the second image data satisfies the threshold. Because the right hand 11010, which is not holding the mobile device 11004, performs the idle hand pose 11020 in the air while the left hand 11008, which is holding the mobile device 11004, performs the swipe input 11022 using the mobile device 11004, the sensing time of the touch input mode data and the sensing time of the second image data overlap each other. In other words, in the illustrative implementation of step 1008, the second elapsed time period from the at least one system timestamp of the touch input mode data to the at least one system timestamp of the second image data satisfies the threshold.

[0110] In step 1016, the mobile device 11004's at least one processor 1926 obtains the information of which hand is used to perform the gesture based on the information of which hand is holding the mobile device and the validation data based on the second image data. The gesture is the air tag gesture 12020 described below. Figure 12 The air tag gesture 12020 described below.

[0111] Figure 12 The user 11016 is shown to perform the air tag gesture 12020 with the right hand 11010 to confirm Figure 11 The list item 11014 highlighted in the middle is the selected list item 12014. The left hand 11008 is at this time in the state 12022 of not performing any touch input.

[0112] Reference is made to Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 12 In step 332, the HMD 11002's imaging device 1908 senses the first image data from which the air tag gesture 12020 is to be estimated.

[0113] In step 540, at least one of the HMD 11002's own or proxy processors 1912 or 1926 requests the latest detection result and sends the at least one system timestamp of the first image data.

[0114] In step 418, the mobile device 11004's at least one processor 1926 obtains the latest detection result. Because the left hand 11008 is in the state 12022 of not performing any touch input while the right hand 11010 is used to perform the air tag gesture 12020, the latest detection result is the information of which hand is used to perform the air tag gesture 12020 obtained in step 1016 described below. Figure 11 In step 418, the mobile device 11004's at least one processor 1926 obtains the latest detection result. Because the left hand 11008 is in the state 12022 of not performing any touch input while the right hand 11010 is used to perform the air tag gesture 12020, the latest detection result is the information of which hand is used to perform the air tag gesture 12020 obtained in step 1016 described below.

[0115] In step 420, the at least one processor 1926 of the mobile device 11004 determines whether a first elapsed time period from the at least one system timestamp of the touch input mode data to the at least one system timestamp of the first image data satisfies a threshold. Because the reference Figure 11 The swipe input 11022 and the air tag gesture 12020 are described to be used for interacting with the virtual list 11012, so Figure 11 and Figure 12 the scenario in is one example of the above-mentioned multi-modal interaction paradigm. The condition in step 420 is satisfied.

[0116] In step 422, the at least one processor 1926 of the mobile device 11004 sends information to the HMD 11002 which hand was used to perform the air tag gesture 12020.

[0117] In step 342 (i.e., the “yes” branch of step 550), the at least one native or proxy processor 1912 or 1926 of the HMD 11002 receives the information which hand was used to perform the air tag gesture 12020 and does not detect the information indicating which hand was used to perform the gesture by an image-based detection method.

[0118] In step 552, the at least one native or proxy processor 1912 or 1926 of the HMD 11002 performs side-adapted hand pose estimation on the first image data using the information which hand was used to perform the air tag gesture 12020.

[0119] Reference is made to Figure 11 A second embodiment is illustratively described in which preferred features of the validation data based on idle hand pose image data are used. Optionally, the validation data can be based on image data of the air tag gesture 12020 in Figure 12

[0120] In the illustrative example in Figure 11 the thumb 11006 of the left hand 11008 is used to perform the swipe input 11022 while the mobile device 11004 is held by the left hand 11008. Optionally, the index finger of the left hand 11008 is used to perform the swipe input while the mobile device 11004 is held by the left hand 11008 from above.

[0121] Illustratively, in the second embodiment, the sensing device is preferably a touch screen 11018 and the mode data is touch input mode data. Optionally, the sensing device is a fingerprint sensing device and the mode data is fingerprint mode data.

[0122] ​The second type of data sensed by the mobile device 1904 is data reflecting the orientation of the mobile device 1904 caused by the hand holding the mobile device 1904. For this type of data, no attestation data similar to the attestation data of the second embodiment is required. In addition to the touch input mode data available due to the multi-modal interaction paradigm, data reflecting the orientation of the mobile device 1904 is also available due to the multi-modal interaction paradigm. The mobile device 1904 part of the multi-modal interaction paradigm can be provided by using the mobile device 1904 as a pointing device pointing to elements in a VR, AR, MR, or XR environment rendered in the HMD 1902. Because the pointing direction of a virtual pointer beam generated by the pointing device is controlled by the data reflecting the orientation of the mobile device 1904, the data reflecting the orientation of the mobile device 1904 can be used to detect information which hand holds the mobile device using the data reflecting the orientation of the mobile device 1904. Furthermore, due to the multi-modal interaction paradigm, the condition that the elapsed time period from the sensing time of the data reflecting the orientation of the mobile device 1904 to the sensing time of the image data of the hand gesture is shorter than or equal to the longest time period in which it is not possible to perform hand switching will be more often satisfied than with imaging-based detection methods. Moreover, detecting information which hand holds the mobile device using the data reflecting the orientation of the mobile device 1904 is faster and consumes less power than imaging-based detection methods. Preferably, detecting information which hand holds the mobile device using the second type of data is implemented together with detecting information which hand holds the mobile device using the first type of data. In this way, because the time during which the mobile device 1904 is used as a touch input device supplements the time during which the mobile device 1904 is used as a pointing device, the condition that the elapsed time period from the sensing time of the data sensed by the mobile device 1904 to the sensing time of the image data of the hand gesture is shorter than or equal to the longest time period in which it is not possible to perform hand switching will be more often satisfied.

[0123] Referring to Figure 13 , Figure 13 is a flowchart of steps related to detecting information indicating which hand is used to perform a hand gesture using data reflecting the orientation of the mobile device 1904 according to the third embodiment of the present disclosure. The third embodiment is based on the first embodiment, so the same content as the first embodiment is omitted here.

[0124] Step 1302 is Figure 2 an embodiment of step 202 in FIG. 2. In step 1302, when the mobile device 1904 is bound to the HMD 1902 and when the mobile device 1904 is held by the first hand, at least one first inertial sensor 1920 of the mobile device 1904 senses data reflecting the orientation of the mobile device 1904 caused by the first hand holding the mobile device 1904. In step 1302, the data reflecting the orientation of the mobile device 1904 is used to detect information which hand holds the mobile device 1904. Figure 19The first data is data that reflects an orientation of the mobile device 1904 caused by the first hand holding the mobile device 1904, where the data that reflects the orientation of the mobile device 1904 is used to control a pointing direction of a virtual pointer beam when the mobile device 1904 is used as a pointing device.

[0125] In embodiments of the orientation of the mobile device 1904, the orientation of the mobile device 1904 can be represented by three degrees of freedom (3DOF) comprising three rotational components, e.g., pitch (i.e., rotation about the y-axis), yaw (i.e., rotation about the z-axis), and roll (i.e., rotation about the x-axis). Other degrees of freedom known in the art that are suitable for orientation detection of the mobile device 1904 are within the intended scope of the present disclosure.

[0126] Referring to Figure 14 , Figure 14 An illustrative example of a third embodiment that uses a feature of the data that reflects the orientation of the mobile device 1904 is shown. In the example of Figure 14 In the example of Figure 14 , the user 14016 holds the mobile device 14004 in the left hand and uses the mobile device 14004 as a pointing device that generates a virtual pointer beam 14022 that points to one of the list items (e.g., list item 14014) of the virtual list 14012 displayed by the HMD 14002. The virtual pointer beam 14022 has been used to point up and down to highlight different ones of the list items. In

[0127] Referring to Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 13 and Figure 14In step 1302, at least one first inertial sensor 1920 of the mobile device 14004 senses first data that is data reflecting an orientation of the mobile device 14004 caused by the left hand 14008 holding the mobile device 14004, when the mobile device 14004 is bound to the HMD 14002 and when the mobile device 14004 is held by the first hand (e.g., the left hand 14008), wherein the data reflecting the orientation of the mobile device 14004 is used to control a pointing direction of the virtual pointer beam 14022 when the mobile device 14004 is used as a pointing device. The sensing device is the at least one first inertial sensor 1920. In an embodiment of the pointing direction of the virtual pointer beam 14022, the pointing direction is parallel to the screen 14018 of the mobile device 14004. Other relationships between the pointing direction and the body of the mobile device 14004 are within the intended scope of the present disclosure. The virtual pointer beam 14022 is, for example, a virtual laser beam.

[0128] In step 212, the at least one processor 1926 of the mobile device 14004 detects information indicating which hand is used to perform a gesture to be estimated from the first image data, wherein the information indicating which hand is used to perform the gesture is detected from the data reflecting the orientation of the mobile device 14004. Details of step 212 are provided below.

[0129] In step 414, the at least one processor 1926 of the mobile device 14004 detects information of which hand holds the mobile device 14004 from the data reflecting the orientation of the mobile device 14004. The information of which hand holds the mobile device 14004 from the data reflecting the orientation of the mobile device 14004 can be detected by methods known in the art.

[0130] In step 416, the at least one processor 1926 of the mobile device 14004 obtains information of which hand is used to perform a gesture from the information of which hand holds the mobile device 14004. The gesture is the air tag gesture 14020.

[0131] In step 332, the imaging device 1908 of the HMD 14002 senses first image data from which the air tag gesture 14020 is to be estimated.

[0132] In step 540, the at least one native or proxy processor 1912 or 1926 of the HMD 14002 requests the latest detection result and sends at least one system timestamp of the first image data.

[0133] In step 418, the at least one processor 1926 of the mobile device 14004 obtains the latest detection result. Because the left hand 14008 uses the mobile device 14004 as a pointing device that generates the virtual pointer beam 14022 pointing at the list item 14014, and the right hand 14010 is used to perform the air-tap gesture 14020, the latest detection result is the information obtained in step 416 above that which hand is used to perform the air-tap gesture 14020.

[0134] In step 420, the at least one processor 1926 of the mobile device 14004 determines whether a first elapsed time period from at least one system timestamp of the data reflecting the orientation of the mobile device 14004 to at least one system timestamp of the first image data satisfies a threshold. Because the mobile device 14004 is used as a pointing device and performs the air-tap gesture 14020 in coordination to interact with the virtual list

[0135] 14012, the scenario in FIG. 14 is an example of the above-described multi-modal interaction paradigm. Figure 14

[0136] In the example in FIG. 14, the sensing time of the data reflecting the orientation of the mobile device 14004 and the sensing time of the first image data overlap with each other. The condition in step 420 is satisfied.

[0137] In step 422, the at least one processor 1926 of the mobile device 14004 sends to the HMD 14002 the information of which hand is used to perform the air-tap gesture 14020.

[0138] In step 342 (i.e., the “yes” branch of step 550), the at least one native or proxy processor 1912 or 1926 of the HMD 14002 receives the information of which hand is used to perform the air-tap gesture 14020, and does not detect the information indicating which hand is used to perform the gesture by the image-based detection method.

[0139] In step 552, the at least one native or proxy processor 1912 or 1926 of the HMD 14002 performs a side-adapted hand pose estimation on the first image data using the information of which hand is used to perform the air-tap gesture 14020.

[0140] ​The third type of data sensed by the mobile device 1904 is image data sensed by an imaging device 1922 (e.g., at least one front imaging device) of the mobile device 1904. For this type of data, no corroboration data similar to the corroboration data of the second embodiment is needed. When the hand holding the mobile device 1904 is biased from the torso of the user to a first side (e.g., the left side), the imaging device 1922 senses the head of the user from the first side, such that the head of the user sensed in the image data sensed by the imaging device 1922 is biased to a second side (e.g., the right side) of the image data sensed by the imaging device 1922. The first side is the same side as the user through which the arm connected to the same hand as the hand holding the mobile device 1904 is dangling. Thus, the imaging data of the head of the user sensed biased to the second side can be used to detect the information of which hand is holding the mobile device, as explained below using illustrative examples.

[0141] The third type of data sensed by the mobile device 1904 is not limited to any type of use of the mobile device 1904 as an input device. The third type of data sensed by the mobile device 1904 can be used when the mobile device 1904 is used as a touch input device or a pointing device, or when the mobile device 1904 is not used as an input device. Furthermore, the sensing time of the image data sensed by the imaging device 1922 of the mobile device 1904 does not have to correspond to the time when the mobile device 1904 is used as an input device, and can correspond to the time when the image data of the gesture is sensed. Due to at least one of the higher confidence, the faster speed, the higher accuracy, and the lower power consumption of using at least one of the first type of data and the second type of data for detecting the information of which hand is holding the mobile device, it is preferable that the use of the third type of data for detecting the information of which hand is holding the mobile device is implemented together with the use of at least one of the first type of data and the second type of data for detecting the information of which hand is holding the mobile device. It is more preferable that the use of the third type of data for detecting the information of which hand is holding the mobile device is implemented together with the use of the first type of data and the second type of data for detecting the information of which hand is holding the mobile device. In this way, since the time when the mobile device 1904 is not used as an input device supplements the time when the mobile device 1904 is used as a touch input device and the time when the mobile device 1904 is used as a pointing device, the condition that the elapsed time period from the sensing time of the data sensed by the mobile device 1904 to the sensing time of the image data of the gesture is shorter than or equal to the longest time period in which hand switching is not possible will be met more frequently.

[0142] Referring to Figure 15 , Figure 15is a flowchart of steps related to using image data sensed by an imaging device 1922 of the mobile device 1904 for detecting information indicative of which hand is used to perform a gesture according to a fourth embodiment of the present disclosure. The fourth embodiment is based on the first embodiment, and thus the same content as the first embodiment is omitted here.

[0143] Step 1502 is Figure 2 an embodiment of step 202. In step 1502, first data is sensed when the mobile device 1904 is bound to the HMD 1902 and when the mobile device 1904 is held by a first hand. The first data is third image data sensed by an imaging device 1922 of the mobile device 1904 (corresponding to the imaging device 1922 of the mobile device 1904 shown in Figure 19 ) with information of a head of the user in the third image data being offset to one side of the third image data being used for detecting information indicative of which hand is used to perform a gesture.

[0144] Illustratively, the imaging device 1922 is a front imaging device.

[0145] Referring to Figure 16 , Figure 16 illustrative examples of the fourth embodiment using features of image data sensed by an imaging device 1922 of the mobile device 1904 are shown. In the examples in Figure 16 , the user 16016 holds the mobile device 16004 with the left hand 16008 and does not use the mobile device 16004 as an input device. A virtual list 16012 is displayed by the HMD 16002 and has a plurality of list items (e.g., list item 16014). The user 16016 has used an air swipe gesture of the right hand 16010 to swipe up and down to highlight a different one of the list items. As an example, the list item 16014 is highlighted. In Figure 16 , when the user 16016 does not use the mobile device 16004 as an input device, the user 16016 performs an air tap gesture 16020 with the right hand 16010 to confirm that the highlighted list item 16014 is selected. The HMD 16002 and the mobile device 16004 correspond to the HMD 1902 and the mobile device 1904, respectively, in Figure 19 .

[0146] Referring to Figure 2 , Figure 3 , Figure 4 , Figure 15 and Figure 16 , in step 1502, a front imaging device 16024 of the mobile device 16004 (corresponding to the imaging device 1922 of the mobile device 1904 shown in Figure 19the imaging device 1922 in the mobile device 16004 senses when the mobile device 16004 is bound to the HMD 16002 and when the mobile device 16004 is held by the first hand (e.g., the left hand 16008), the first data is third image data 16026, where information in the third image data 16026 that the head 16028 of the user (corresponding to the user 16016) is offset to one side of the third image data 16026 is used to detect information indicating which hand is used to perform the gesture. The sensing device is the front imaging device 16024.

[0147] In step 212, the at least one processor 1926 of the mobile device 16004 detects information indicating which hand is used to perform the gesture to be estimated from the first image data, where the information indicating which hand is used to perform the gesture is detected from third image data 16026 sensed by the front imaging device 16024 of the mobile device 16004. Details of step 212 are provided below.

[0148] In step 414, the at least one processor 1926 of the mobile device 16004 detects information which hand holds the mobile device 16004 from third image data 16026 sensed by the front imaging device 16024 of the mobile device 16004.

[0149] In step 416, the at least one processor 1926 of the mobile device 16004 obtains information which hand is used to perform the gesture from the information which hand holds the mobile device 16004. The gesture is the air tab gesture 16020.

[0150] In step 332, the imaging device 1908 of the HMD 16002 senses first image data from which the air tab gesture 16020 is to be estimated.

[0151] In step 540, the at least one native or proxy processor 1912 or 1926 of the HMD 16002 requests the latest detection result and sends at least one system time stamp of the first image data.

[0152] In step 418, the at least one processor 1926 of the mobile device 16004 obtains the latest detection result. Because the left hand 16008 uses the mobile device 16004 as a pointing device that produces the virtual pointer beam 16022 that points to the list item 16014, and the right hand 16010 is used to perform the air tab gesture 16020, the latest detection result is the information obtained in step 416 above that uses which hand to perform the air tab gesture 16020.

[0153] In step 420, the at least one processor 1926 of the mobile device 16004 determines whether a first elapsed time period from at least one system timestamp of the third image data 16026 sensed by the front imaging device 16024 of the mobile device 16004 to at least one system timestamp of the first image data satisfies a threshold. Because, as described above, the sensing time of the image data sensed by the imaging device 1922 of the mobile device 1904 does not necessarily correspond to the time when the mobile device 1904 is used as an input device, and can correspond to the time when the image data of the gesture is sensed. Therefore, the sensing time of the third image data 16026 sensed by the front imaging device 16024 of the mobile device 16004 and the sensing time of the first image data overlap with each other. The condition in step 420 is satisfied.

[0154] In step 422, the at least one processor 1926 of the mobile device 16004 sends the information of which hand is used to perform the air-tag gesture 16020 to the HMD 16002.

[0155] In step 342 (i.e., the “Yes” branch of step 550), the at least one native or proxy processor 1912 or 1926 of the HMD 16002 receives the information of which hand is used to perform the air-tag gesture 16020, and does not detect the information indicating which hand is used to perform the gesture by the image-based detection method.

[0156] In step 552, the at least one native or proxy processor 1912 or 1926 of the HMD 16002 performs the side-adapted hand pose estimation on the first image data using the information of which hand is used to perform the air-tag gesture 16020.

[0157] With reference to Figure 11 , Figure 12 , Figure 14 and Figure 16 The illustrative examples described herein are based on the embodiments described with reference to Figure 4 and Figure 5 , and apply mutatis mutandis to the embodiments described with reference to Figure 6 , Figure 7 and Figure 8 .

[0158] To improve the step of detecting the information of which hand performs the gesture, a part of the method performed by the at least one native or proxy processor 1912 or 1926 of the HMD 1902 (i.e., the method 300) is performed by at least one first process, and a part of the method performed by the at least one processor 1926 of the mobile device 1904 (i.e., the method 200) is performed by at least one second process, wherein the at least one first process and the at least one second process are parallel processes.

[0159] Referring to Figure 17 , Figure 17 is a timeline diagram illustrating parallel processing of a portion of the method performed by at least one native or proxy processor 1912 or 1926 of the HMD 1902 and a portion of the method performed by at least one processor 1926 of the mobile device 1904 based on the second embodiment of the disclosure. To illustrate how the steps of the method 300 performed by at least one processor 1926 of the mobile device 1904 and the method 200 performed by at least one processor 1926 of the mobile device 1904 improve upon the deficiencies of sequentially processing the steps, the steps performed by the method 300 and the method 200 are grouped into tasks similar to how the steps performed by the existing hand pose estimation 100 are grouped into modules in Figure 1 .

[0160] Exemplarily, the steps performed by the method 300 are grouped into the idle hand pose performing hand detection task 1702, the gesture performing hand detection task 1704, the gesture performing hand image cropping task 1706, and the side-adapted hand pose estimation task 1708 performed by a first process of at least one native or proxy processor 1912 or 1926 of the HMD 1902. The steps performed by the method 200 are grouped into the gesture performing hand information detection task 1710 performed by a second process of at least one processor 1926 of the mobile device 1904.

[0161] The following steps are performed in the method 200. The steps 912 and 222 are implemented by (1) the steps in Figure 4 , (2) the steps performed by at least one processor 1926 of the mobile device 1904, or (3) the steps in Figure 6 performed by at least one processor 1926 of the mobile device 1904, where the portion corresponding to the step 212 is replaced by the steps included in the step 912 in Figure 8 . The steps of each of the three implementations are grouped into the gesture performing hand information detection task 1710. Figure 10 The following steps are performed in the method 300. The steps 1006 and 1008 are grouped into the idle hand pose performing hand detection task 1702. Reference is made to

[0162] for a description but the hand detection steps and the steps of cropping the first image data are not shown in Figure 2 . The steps 342 are implemented by (1) the steps 540 and 550 in Figure 2 , or (2) the step 742 in Figure 5 . Since the steps 342 are implemented by the steps 540 and 550 in Figure 7 , the steps 342 are grouped into the gesture performing hand detection task 1704. The steps 344 are implemented by (1) the steps 550 and 560 in Figure 17Steps in the time are trivial compared to other tasks in the process, and for the purpose of parallel processing, steps in each of the two implementations are omitted to group them as tasks. Step 352 is implemented by (1) Figure 5 step 552 in the process, or (2) Figure 7 step 752 in the process. Steps in each of the two implementations are grouped as a side-adapted hand pose estimation task 1708.

[0163] Exemplarily, a part of the gesture-performing hand information detection task 1710 relies on the idle hand pose performing hand detection task 1702. Therefore, a part of the gesture-performing hand information detection task 1710 that does not rely on the idle hand pose performing hand detection task 1702 can be executed in parallel with the idle hand pose performing hand detection task 1702. Although the gesture-performing hand detection task 1704 does not rely on the idle hand pose performing hand detection task 1702, the idle hand pose performing hand detection task 1702 and the gesture-performing hand detection task 1704 are sequentially executed due to the sequence of the idle hand pose and the gesture. The gesture-performing hand image cropping task 1706 depends on the gesture-performing hand detection task 1704. Therefore, the gesture-performing hand detection task 1704 and the gesture-performing hand image cropping task 1706 are sequentially executed. The gesture-performing hand detection task 1704 and the gesture-performing hand image cropping task 1706 do not rely on the gesture-performing hand information detection task 1710, and vice versa. Therefore, the gesture-performing hand detection task 1704 and the gesture-performing hand image cropping task 1706 can be executed in parallel with the gesture-performing hand information detection task 1710. The side-adapted hand pose estimation task 1708 depends on the gesture-performing hand image cropping task 1706 and the gesture-performing hand information detection task 1710. Therefore, the gesture-performing hand image cropping task 1706 and the side-adapted hand pose estimation task 1708 are sequentially executed, and the gesture-performing hand information detection task 1710 and the side-adapted hand pose estimation task 1708 are sequentially executed.

[0164] Referring to Figure 18 , Figure 18This is a timeline diagram illustrating the parallel processing of a portion of a method executed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902 and a portion of a method executed by at least one processor 1926 of mobile device 1904, based on the third and fourth embodiments of this disclosure. To illustrate how parallel processing of method 300 and method 200 executed by at least one processor 1926 of mobile device 1904 improves upon the shortcomings of sequential processing steps in detecting which hand is used to perform a gesture, the steps performed by method 300 and method 200 are grouped into tasks, similar to how steps performed by existing hand pose estimation 100 are grouped into tasks. Figure 1 The module in.

[0165] For example, the steps performed by method 300 are grouped into a first process performed by at least one proprietary or proxy processor 1912 or 1926 of HMD 1902, comprising a hand detection task 1804, a hand image cropping task 1806, and a hand pose estimation task 1808 for side adaptation. The steps performed by method 200 are grouped into a hand information detection task 1810 performed by a second process of at least one processor 1926 of mobile device 1904.

[0166] Perform the following steps in method 200. Steps 212 and 222 are performed via (1) Figure 4 Step (2) is performed by at least one processor 1926 of the mobile device 1904. Figure 6 The steps in (3) are performed by at least one processor 1926 of the mobile device 1904. Figure 8 The steps in the three embodiments are grouped into a gesture execution hand information detection task 1810.

[0167] Perform the following steps in method 300. (See reference) Figure 2 Description but Figure 2 The hand detection step (not shown) and the step of cropping the first image data correspond to the hand detection task 1804 and the hand image cropping task 1806, respectively. Step 342 is performed via (1) Figure 5 Steps 540 and 550 or (2) Figure 7 This is achieved through step 742. Due to... Figure 18 Compared to other tasks in the process, its time-consuming nature necessitates the omission of steps in each of the two implementations to group it into tasks for the purpose of parallel processing. Step 352 is performed via (1). Figure 5 Step 552 or (2) in the middle. Figure 7Steps 752 in each of the two embodiments are grouped as the side-adapted hand pose estimation task 1808.

[0168] By way of example, the gesture-performing hand image cropping task 1806 depends on the gesture-performing hand detection task 1804. Therefore, the gesture-performing hand detection task 1804 and the gesture-performing hand image cropping task 1806 are sequentially executed. The gesture-performing hand detection task 1804 and the gesture-performing hand image cropping task 1806 do not depend on the gesture-performing hand information detection task 1810, and vice versa. Therefore, the gesture-performing hand detection task 1804 and the gesture-performing hand image cropping task 1806 can be executed in parallel with the gesture-performing hand information detection task 1810. The side-adapted hand pose estimation task 1808 depends on the gesture-performing hand image cropping task 1806 and the gesture-performing hand information detection task 1810. Therefore, the gesture-performing hand image cropping task 1806 and the side-adapted hand pose estimation task 1808 are sequentially executed, and the gesture-performing hand information detection task 1810 and the side-adapted hand pose estimation task 1808 are sequentially executed.

[0169] Referring to Figure 19 , Figure 19 is a diagram illustrating a system 1900 in which the methods described herein can be implemented. The system 1900 includes an HMD 1902 and a mobile device 1904. The HMD 1902 can be an AR glasses, a VR headset, or a smart glasses without a full 3D display screen (e.g., Google glasses). The mobile device 1904 can be a smartphone or a computing box, etc. The HMD 1902 is wirelessly bound to the mobile device 1904 via Wi-Fi 1906 or Bluetooth, etc., or is bound to the mobile device 1904 via a USB cable.

[0170] The HMD 1902 includes an imaging device 1908, at least one processor 1912 (i.e., at least one self-owned processor), a memory 1914, and a bus 1910.

[0171] The imaging device 1908 includes at least one vision sensor (e.g., at least one RGB camera). Optionally, the imaging device 1908 includes at least one ultrasonic sensor. Still optionally, the imaging device 1908 includes at least one millimeter wave sensor.

[0172] The at least one processor 1912 can be implemented as a "processing system." Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLD), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. At least one processor in the processing system can execute software. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0173] The at least one agent processor of the HMD 1902 can be the at least one processor 1926 of the mobile device.

[0174] The functions implemented in the software can be stored in or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Computer storage media can also be referred to as a non-transitory computer-readable medium. The terms "non-transitory computer-readable medium" is intended to include a computer storage medium excluding a transitory, propagating signal. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer. The memory 1914 can be referred to as a computer-readable medium.

[0175] The bus 1910 couples the imaging device 1908 and the memory 1914 to the at least one processor 1912.

[0176] The mobile device 1904 includes a touch screen 1918, at least one first inertial sensor 1920, an imaging device 1922, at least one second inertial sensor 1924, at least one processor 1926, a memory 1928, and a bus 1928.

[0177] Figure 2 The sensing device in step 202 includes the touch screen 1918. Optionally, the sensing device includes the at least one first inertial sensor 1920. Still optionally, the sensing device includes the imaging device 1922.

[0178] The touch screen 1918 can be a capacitive touch screen, a resistive touch screen, an infrared touch screen, or an ultrasonic touch screen, etc.

[0179] The at least one first inertial sensor 1920 and the at least one second inertial sensor 1924 are included in an inertial measurement unit (IMU) of the mobile device 1904. Examples of inertial sensors include accelerometers, gyroscopes, and magnetometers. The at least one first inertial sensor 1920 and the at least one second inertial sensor 1924 can be at least partially identical.

[0180] The imaging device 1922 includes at least one vision sensor (e.g., at least one RGB camera). Optionally, the imaging device 1922 includes at least one ultrasonic sensor. Still optionally, the imaging device 1922 includes at least one millimeter wave sensor.

[0181] The at least one processor 1926 can be implemented as a “processing system.” Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLD), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. At least one processor in the processing system can execute software. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0182] Accordingly, in one or more embodiments, the functions implemented by the software can be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Computer-storage media can also be referred to as a non-transitory computer-readable medium. The term "non-transitory" does not modify the duration or presence of signals, and thus is not to be construed as related to signal duration or presence. Computer-readable media does not include a transitory signal per se. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer. Memory 1928 can be referred to as a computer-readable medium.

[0183] Bus 1928 couples the touch screen 1918, the at least one first inertial sensor 1920, the imaging device 1922, the at least one second inertial sensor 1924, and the memory 1928 to the at least one processor 1926.

Claims

1. A method performed by a mobile device, comprising: sensing, by a sensing device of the mobile device, first data when the mobile device is bound to a head-mounted display (HMD) and when the mobile device is held by a first hand; detecting, by at least one processor of the mobile device, information indicative of which hand is used to perform a gesture, the gesture being estimated from first image data by at least one native or proxy processor of the HMD, wherein the information indicative of which hand is used to perform the gesture is not detected from the first image data by the HMD by means of an imaging-based detection method, but is detected from the first data; sending, by the at least one processor of the mobile device, the information indicative of which hand is used to perform the gesture to the at least one native or proxy processor of the HMD, the execution time point of the sending step being such that the at least one native or proxy processor of the HMD uses the information indicative of which hand is used to perform the gesture for performing a side-adapted hand pose estimation on the first image data; wherein the gesture is performed by a second hand and the information indicative of which hand is used to perform the gesture is an updated indication about the gesture, the updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data.

2. The method according to claim 1, wherein a longest time period during which a hand switch is not possible is a longest time period during which the mobile device is not held by a relative hand of the first hand sufficient for the second hand to become such a possibility of the first hand; wherein the information indicative of which hand is used to perform the gesture is the updated indication about the gesture taking into account: when the first elapsed time period is shorter than the longest time period during which a hand switch is not possible, and when the first elapsed time period is equal to the longest time period during which a hand switch is not possible, no hand switch is possible during the first elapsed time period; when the first elapsed time period is shorter than the longest time period during which a hand switch is not possible, when the first elapsed time period is equal to the longest time period during which a hand switch is not possible, and when the first elapsed time period is longer than the longest time period during which a hand switch is not possible, each first occurring hand switch is detected during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period during which a hand switch is not possible, and when the first elapsed time period is equal to the longest time period during which a hand switch is not possible, no hand switch is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which a hand switch is not possible, each second occurring hand switch is detected during the first elapsed time period.

3. The method according to claim 1, wherein the method further comprising: sensing, by at least one inertial sensor of the mobile device, second data; and the at least one processor of the mobile device detects from the second data an occurrence of each hand switch during the first elapsed time period; wherein the information indicative of which hand is used to perform the gesture is the updated indication regarding the gesture that takes into account at least the occurrence of each hand switch during the first elapsed time period.

4. The method of claim 1, wherein the first data is pattern data caused by a hand that belongs to the first hand holding the mobile device and does not participate in holding the mobile device; wherein the information indicative of which hand is used to perform the gesture is further detected based on second image data of a third hand not holding the mobile device, wherein the second image data is sensed by an imaging device of the HMD; and wherein a second elapsed time period between the sensing time of the first data and a sensing time of the second image data is insufficient for the third hand to become a likely candidate for the first hand after the mobile device is switched to be held by the opposite hand of the first hand.

5. The method of claim 4, wherein in the second image data of the third hand, the third hand performs an idle hand pose; and wherein the sensing time of the first data and the sensing time of the second image data overlap each other.

6. The method of claim 4, wherein, the pattern data is touch input pattern data.

7. The method of claim 6, wherein the longest time period in which hand switching is impossible is the longest time period in which the second hand is insufficient to become a likely candidate for the first hand after the mobile device is switched to be held by the opposite hand of the first hand; wherein the information indicative of which hand is used to perform the gesture is the updated indication regarding the gesture that takes into account: when the first elapsed time period is shorter than the longest time period in which hand switching is impossible, and when the first elapsed time period is equal to the longest time period in which hand switching is impossible, no hand switch is possible during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period in which hand switching is impossible, and when the first elapsed time period is equal to the longest time period in which hand switching is impossible, no hand switch is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period in which hand switching is impossible, each second occurrence of hand switching is detected during the first elapsed time period.

8. The method of claim 6, wherein the hand is a thumb; and wherein the mobile device is held from below by the first hand.

9. The method of claim 1, wherein the first data is data reflecting an orientation of the mobile device caused by the first hand holding the mobile device, wherein the data reflecting the orientation of the mobile device is used to control a pointing direction of a virtual pointer beam when the mobile device is used as a pointing device.

10. The method of claim 9, wherein a longest time period during which hand switching is not possible is a time period after the mobile device is switched to be held by a relative hand of the first hand, which is insufficient for the second hand to become the first hand such possible longest time period; wherein the information indicating which hand is used to perform the gesture is an updated indication about the gesture taking into account: when the first elapsed time period is shorter than the longest time period during which hand switching is not possible, and when the first elapsed time period is equal to the longest time period during which hand switching is not possible, hand switching is not possible during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period during which hand switching is not possible, and when the first elapsed time period is equal to the longest time period during which hand switching is not possible, hand switching is not possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which hand switching is not possible, each detected second hand switching during the first elapsed time period.

11. The method of claim 1, wherein the first data is third image data sensed by an imaging device of the mobile device, wherein the information indicating which hand is used to perform the gesture is detected using information of a user head bias to a side of the third image data in the third image data.

12. The method of claim 1, wherein a portion of the method performed by the at least one processor of the mobile device is performed by at least one second process in parallel with at least one first process of the at least one native or proxy processor of the HMD in which the gesture is estimated from the first image data.

13. A method performed by a head-mounted display (HMD), comprising: receiving, by at least one native or proxy processor of the HMD, information indicating which hand is used to perform a gesture, the gesture to be estimated from first image data, and the at least one native or proxy processor of the HMD not detecting the information indicating which hand is used to perform the gesture by an image-based detection method; wherein the information indicating which hand is used to perform the gesture is detected from first data sensed by a sensing device of a mobile device while the HMD is bound to the mobile device and while the mobile device is held by a first hand; and wherein the gesture is performed by a second hand and the information indicating which hand is used to perform the gesture is an updated indication about the gesture taking into account a hand switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data; and performing, by the at least one native or proxy processor of the HMD, side-adapted hand pose estimation on the first image data using the information indicating which hand is used to perform the gesture.

14. The method of claim 13, wherein, a longest time period during which hand switching is not possible is a time period after the mobile device is switched to being held by an opposite hand of the first hand, insufficient for the second hand to become such a possible longest time period of the first hand; wherein the information indicative of which hand is used to perform the gesture is the updated information about the gesture taking into account that: when the first elapsed time period is shorter than the longest time period during which hand switching is not possible, and when the first elapsed time period is equal to the longest time period during which hand switching is not possible, no hand switching is possible during the first elapsed time period; when the first elapsed time period is shorter than the longest time period during which hand switching is not possible, when the first elapsed time period is equal to the longest time period during which hand switching is not possible, and when the first elapsed time period is longer than the longest time period during which hand switching is not possible, each first occurring hand switching is detected during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period during which hand switching is not possible and when the first elapsed time period is equal to the longest time period during which hand switching is not possible, no hand switching is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which hand switching is not possible, each second occurring hand switching is detected during the first elapsed time period.

15. The method of claim 13, wherein, the first data is pattern data caused by a hand that belongs to the first hand holding the mobile device and does not participate in holding the mobile device; wherein the method further comprises sensing, by an imaging device of the HMD, second image data of a third hand not holding the mobile device, wherein the information indicative of which hand is used to perform the gesture is further detected based on the second image data; and wherein a second elapsed time period between the sensing time of the first data and a sensing time of the second image data is insufficient for the third hand to become such a possible longest time period of the first hand after the mobile device is switched to being held by an opposite hand of the first hand.

16. The method of claim 15, wherein in the second image data of the third hand, the third hand performs an idle hand gesture; and wherein the sensing time of the first data and the sensing time of the second image data overlap each other.

17. The method of claim 13, wherein the portion of the method performed by the at least one native or proxy processor of the HMD is performed by at least one first process that is parallel to at least one second process of the mobile device in which the information indicative of which hand is used to perform the gesture is detected.

18. A method performed by a mobile device and a head-mounted display (HMD), comprising: when the mobile device is bound to the HMD and when the mobile device is held by a first hand, a sensing device of the mobile device senses first data; at least one processor of the mobile device detects information indicative of which hand is used to perform a gesture, the gesture to be estimated from first image data by at least one native or proxy processor of the HMD, wherein the information indicative of which hand is used to perform the gesture is detected from the first data; the at least one processor of the mobile device sends the information indicative of which hand is used to perform the gesture to the at least one native or proxy processor of the HMD; the at least one native or proxy processor of the HMD receives the information indicative of which hand is used to perform the gesture, and the at least one native or proxy processor of the HMD does not detect the information indicative of which hand is used to perform the gesture by an image-based detection method; wherein the gesture is performed by a second hand, and the information indicative of which hand is used to perform the gesture is an updated indication about the gesture that takes into account a hand-switching state during a first elapsed time period between a sensing time of the first data and a sensing time of the first image data; and the at least one native or proxy processor of the HMD performs a side-adapted hand pose estimation on the first image data using the information indicative of which hand is used to perform the gesture.

19. The method of claim 18, wherein, a longest time period during which hand-switching is not possible is a longest time period during which the mobile device is held by a second hand after being switched to be held by the first hand insufficient for the second hand to become such a possible first hand; wherein the information indicative of which hand is used to perform the gesture is the updated indication about the gesture that takes into account: when the first elapsed time period is shorter than the longest time period during which hand-switching is not possible and when the first elapsed time period is equal to the longest time period during which hand-switching is not possible, no hand-switching is possible during the first elapsed time period; when the first elapsed time period is shorter than the longest time period during which hand-switching is not possible, when the first elapsed time period is equal to the longest time period during which hand-switching is not possible, and when the first elapsed time period is longer than the longest time period during which hand-switching is not possible, each first occurring hand-switching is detected during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period during which hand-switching is not possible and when the first elapsed time period is equal to the longest time period during which hand-switching is not possible, no hand-switching is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which hand-switching is not possible, each second occurring hand-switching is detected during the first elapsed time period.

20. The method of claim 18, wherein the method further comprising: sensing, by at least one inertial sensor of the mobile device, second data; and detecting, by at least one processor of the mobile device, from the second data, an occurrence of each hand switch during the first elapsed time period; wherein the information indicative of which hand is used to perform the gesture is the updated indication regarding the gesture that takes into account at least the occurrence of each hand switch during the first elapsed time period.

21. The method of claim 18, wherein the first data is pattern data caused by a hand that belongs to the first hand holding the mobile device and does not participate in holding the mobile device; wherein the method further comprises sensing, by an imaging device of the HMD, second image data of a third hand not holding the mobile device; wherein the information indicative of which hand is used to perform the gesture is further detected based on the second image data of a third hand not holding the mobile device; and wherein a second elapsed time period between the sensing time of the first data and a sensing time of the second image data is insufficient for the third hand to become a possibility for the first hand after the mobile device is switched to be held by the opposite hand of the first hand.

22. The method of claim 21, wherein in the second image data of the third hand, the third hand performs an idle hand pose; and wherein the sensing time of the first data and the sensing time of the second image data overlap each other.

23. The method of claim 21, wherein the pattern data is touch input pattern data.

24. The method of claim 23, wherein the longest time period during which hand switching is impossible is the longest time period insufficient for the second hand to become a possibility for the first hand after the mobile device is switched to be held by the opposite hand of the first hand; wherein the information indicative of which hand is used to perform the gesture is the updated indication regarding the gesture that takes into account: when the first elapsed time period is shorter than the longest time period during which hand switching is impossible and when the first elapsed time period is equal to the longest time period during which hand switching is impossible, no hand switch is possible during the first elapsed time period; or when the first elapsed time period is shorter than the longest time period during which hand switching is impossible and when the first elapsed time period is equal to the longest time period during which hand switching is impossible, no hand switch is possible during the first elapsed time period; and when the first elapsed time period is longer than the longest time period during which hand switching is impossible, each second occurrence of hand switching is detected during the first elapsed time period.

25. The method of claim 23, wherein, the hand is a thumb; and wherein the mobile device is held from below by the first hand.

26. The method of claim 18, wherein, the first data is data reflecting an orientation of the mobile device caused by the first hand holding the mobile device, wherein, when the mobile device is used as a pointing device, the data reflecting the orientation of the mobile device is used to control the pointing direction of a virtual pointer beam.

27. The method of claim 26, wherein, the longest period of time during which hand switching is not possible is a longest period of time, after the mobile device is switched to be held by the opposite hand of the first hand, during which the second hand does not become available to the first hand; wherein the information indicating which hand is used to perform the gesture is the updated indication of the gesture taking into account: when the first elapsed time period is shorter than the longest period of time during which hand switching is not possible and when the first elapsed time period is equal to the longest period of time during which hand switching is not possible, no hand switching is detected during the first elapsed time period; or when the first elapsed time period is shorter than the longest period of time during which hand switching is not possible and when the first elapsed time period is equal to the longest period of time during which hand switching is not possible, no hand switching is detected during the first elapsed time period; and when the first elapsed time period is longer than the longest period of time during which hand switching is not possible, every second hand switching is detected during the first elapsed time period.

28. The method of claim 18, wherein the first data is third image data sensed by an imaging device of the mobile device, wherein the information indicating which hand is used to perform the gesture is detected using information of a user head bias to one side of the third image data.

29. The method of claim 18, wherein a portion of the method performed by the at least one native or proxy processor of the HMD is performed by at least one first process, a portion of the method performed by at least one processor of the mobile device is performed by at least one second process, wherein the at least one first process and the at least one second process are parallel to each other.

30. A mobile device comprising: a sensing device configured to sense first data when the mobile device is bound to a head mounted display (HMD) and when the mobile device is held by a first hand; a memory; and at least one processor coupled to the memory and configured to perform the method of any one of claims 1-12.

31. A head mounted display (HMD) comprising: a memory; and at least one native processor coupled to the memory and configured to perform the method of any one of claims 13-17.

Images