Method for tracking hand gesture and electronic device thereof

Abstract

A method for tracking hand gestures in a virtual reality (VR) environment through a wearable device is provided. The method includes calibrating a plurality of sensors configured to detect positions of fingers relative to a user's hand, identifying sensor data obtained by the plurality of sensors, and tracking hand gestures based on the sensor data.

Description

Technical Field

[0001] The present disclosure relates to wearable devices. More specifically, the present disclosure relates to wearable devices and methods for tracking hand gestures in a virtual reality (VR) environment. Background Art

[0002] Typically, a VR environment is a simulated environment created and presented to a user through a VR device. When a user wants to perform some interactions with their hands in a VR environment, such as pulling, pushing, or taking things, the VR environment provides a virtual hand. In addition, a handheld controller such as a button, joystick, or touchpad is used to operate the virtual hand to perform the interaction, which does not provide an intuitive and realistic user experience. In addition, the use of a handheld controller lacks the intuitiveness and seamlessness expected of a VR device.

[0003] In some VR systems of the related art, virtual hands are used in conjunction with hand tracking systems in a VR environment to track the posture, movement, and position of a user's hands. However, hand tracking systems use sensors that are generally not able to accurately track the position of a user's fingers, resulting in poor performance of the user's hand movements in a VR environment. In addition, due to the number of sensors and the placement of the sensors, the functionality of wearable devices for hand tracking is limited. Furthermore, a wearable device with sensors designed for a single user may not be able to track the hand posture of another user. And, due to the variability in fit, there is no well-defined calibration technique for wearable devices.

[0004] The above information is provided as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be used as prior art with respect to the present disclosure. Summary of the invention

[0005] Technical Solution

[0006] Aspects of the present disclosure are intended to solve at least the above problems and / or disadvantages and provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a wearable device and method for hand gesture tracking in a virtual reality (VR) environment.

[0007] Another aspect of the present disclosure is to receive position information of multiple sensors in a wearable device for multiple hand gestures of a user.

[0008] Another aspect of the present disclosure is to map the received position information of multiple sensors with a predetermined position of a user's hand gesture.

[0009] Another aspect of the present disclosure is to dynamically track hand gestures in a VR environment based on position information received from multiple sensors in a wearable device and one or more position tracking sensors in a VR device.

[0010] Another aspect of the present disclosure is to track a user's hand gestures when the user's hands are outside the field of view of a position tracking sensor of a VR device.

[0011] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0012] According to one aspect of the present disclosure, a method for tracking hand gestures in a VR environment through a wearable device is provided. The method includes receiving orientation information of multiple sensors in a wearable device for multiple hand gestures of a user, and mapping the received orientation information of the multiple sensors with a predetermined orientation of the user's hand gestures. In addition, the method also includes determining an absolute initial orientation of the wearable device relative to the VR device. In addition, the method includes dynamically tracking hand gestures in a VR environment based on orientation information received from multiple sensors in the wearable device and one or more position tracking sensors in the VR device.

[0013] According to another aspect of the present disclosure, a wearable device for tracking hand postures in a VR environment is provided. The wearable device includes a glove covering at least three fingers of a user's hand and a plurality of sensors embedded in the glove. In addition, the wearable device is powered by a battery and further includes one or more position tracking sensors. The wearable device is configured to receive orientation information of a plurality of sensors in the wearable device for a plurality of hand postures of the user, and map the received orientation information of the plurality of sensors with a predetermined orientation of the user's hand posture. In addition, the wearable device is further configured to determine an absolute initial orientation of the wearable device relative to the VR device, and dynamically track the hand posture in the VR environment based on the orientation information received from the plurality of sensors in the wearable device and the one or more position tracking sensors in the VR device.

[0014] According to another aspect of the present disclosure, a method for tracking hand gestures in an interactive environment through a wearable device is provided. The method includes receiving position information of multiple sensors in a wearable device for multiple hand gestures of a user, and mapping the received position information of the multiple sensors with a predetermined position of the user's hand gesture. The method also includes determining an absolute initial position of the wearable device relative to the interactive electronic device, and dynamically tracking the hand gestures in the interactive environment based on the position information received from the multiple sensors in the wearable device and the information received from one or more position tracking sensors in the interactive electronic device.

[0015] According to another aspect of the present disclosure, a method for operating an electronic device is provided, which includes calibrating a plurality of sensors configured to detect the orientation of fingers relative to a user's hand, identifying sensor data obtained by the plurality of sensors, and tracking a hand gesture based on the sensor data.

[0016] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes a display, a transceiver, and at least one processor, the at least one processor being electrically coupled to the transceiver and the display and configured to calibrate a plurality of sensors configured to detect the orientation of fingers relative to a user's hand, recognize sensor data obtained by the plurality of sensors, and track a hand gesture based on the sensor data.

[0017] According to another aspect of the present disclosure, a wearable device for tracking hand gestures in a VR environment is provided. The wearable device includes: a glove covering at least a portion of a thumb, at least a portion of an index finger, and at least a portion of a middle finger; a fixed member coupled to the glove; a plurality of sensors arranged on the glove and configured to detect the positions of the thumb, the index finger, and the middle finger; and a processor located on the fixed member and configured to send sensor data obtained by the plurality of sensors.

[0018] Other aspects, advantages, and salient features of the invention will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0020] Figure 1A An example of hand pose tracking relative to a virtual reality (VR) device before calibrating a wearable device relative to the VR device according to an embodiment of the present disclosure is shown;

[0021] Figure 1B An example of hand pose tracking relative to a VR device after calibrating a wearable device relative to the VR device according to an embodiment of the present disclosure is shown;

[0022] Figure 2 A wearable device for hand gesture tracking in a VR environment according to an embodiment of the present disclosure is shown;

[0023] Figure 3 is a flowchart illustrating a method for performing hand gesture tracking in a VR environment by a wearable device according to an embodiment of the present disclosure;

[0024] Figure 4A method for determining the position of a user's hand based on sensor data fusion technology according to an embodiment of the present disclosure is shown;

[0025] Figure 5 shows a specific hand gesture for performing user-specific calibration of a wearable device according to an embodiment of the present disclosure;

[0026] Fig. 6A A method for determining the approximate finger positions of the ring finger and the little finger of a user's hand that are not covered by a wearable device according to an embodiment of the present disclosure is shown;

[0027] Figure 6B A method for determining the approximate finger positions of the ring finger and the little finger of a user's hand that are not covered by a wearable device according to an embodiment of the present disclosure is shown;

[0028] Figure 6C A method for determining the approximate finger positions of the ring finger and the little finger of a user's hand that are not covered by a wearable device according to an embodiment of the present disclosure is shown;

[0029] Figure 7 An example of calibration of a wearable device relative to a VR device according to an embodiment of the present disclosure is shown;

[0030] Figure 8 shows a comparison of sensor orientations calculated from the Davenport q method and a sensor fusion method for orientation estimation according to an embodiment of the present disclosure;

[0031] Fig. 9 An apparatus for determining the accuracy of a sensor fusion method for orientation estimation using a protractor as a reference according to an embodiment of the present disclosure is shown;

[0032] Fig. 10A Hand gesture tracking in a VR environment using a camera and an IMU sensor when a user's hand is within the field of view of an IMU sensor according to an embodiment of the present disclosure is shown;

[0033] Fig. 10B Hand gesture tracking in a VR environment using a camera and an IMU sensor when a user's hand is outside the field of view of the IMU sensor according to an embodiment of the present disclosure is shown;

[0034] Fig.11 The figure shows a comparison between the expected rotation angle and the predicted angle obtained by the sensor fusion method for orientation estimation according to an embodiment of the present disclosure;

[0035] Fig.12An example scenario for determining the absolute position of a user's hand according to an embodiment of the present disclosure is shown; and

[0036] Fig.13 Detailed description is a flow chart illustrating a method for performing hand gesture tracking in an interactive environment by a wearable device according to an embodiment of the present disclosure.

[0037] Throughout the drawings, like reference numerals will be understood to refer to like parts, components, and structures. DETAILED DESCRIPTION

[0038] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details that aid in understanding, but these are merely exemplary. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0039] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the present disclosure as defined by the appended claims and their equivalents.

[0040] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes one or more of such surfaces.

[0041] Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0042] Here, unless otherwise indicated, the term "or" used herein refers to a non-exclusive "or". For example, the paragraph "at least one of a, b, or c" can be understood as "only a", "only b", "only c", "both a and b", "both a and c", "both b and c", "all of a, b, and c" or variations thereof. The examples used herein are merely to facilitate understanding of the ways in which the embodiments of the present invention may be practiced, and to further enable those skilled in the art to practice the embodiments of the present invention. Therefore, these examples should not be interpreted as limiting the scope of the embodiments of the present invention.

[0043] As is conventional in the art, embodiments may be described and illustrated in terms of blocks that perform one or more functions described. These blocks may be referred to herein as units, engines, managers, modules, etc., which are physically implemented by analog and / or digital circuits, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may be optionally driven by firmware and / or software. The circuit may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the blocks and processors that perform other functions of the blocks. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more discrete blocks that interact with each other. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0044] Therefore, embodiments of the present invention provide a method for tracking hand gestures in a virtual reality (VR) environment through a wearable device. The method includes receiving position information of multiple sensors in a wearable device for multiple hand gestures of a user, and mapping the received position information of the multiple sensors with a predetermined position of the user's hand gestures. In addition, the method also includes determining an absolute initial position of the wearable device relative to the VR device. In addition, the method includes dynamically tracking hand gestures in a VR environment based on the position information received from multiple sensors in the wearable device and one or more position tracking sensors in the VR device.

[0045] In one embodiment, determining the absolute initial orientation of the wearable device relative to the VR device includes receiving a relative orientation of the hand relative to the VR device.

[0046] In one embodiment, the relative position of the user's hand with respect to the VR device is determined using at least one hand pose from a predetermined set of hand poses on the VR device.

[0047] In one embodiment, multiple sensors are placed at predetermined locations on the wearable device to track the movement of the user's hands relative to the VR device.

[0048] In one embodiment, multiple sensors are calibrated relative to the VR device based on a single predefined hand gesture or a predefined sequence of hand gestures of a user.

[0049] In one embodiment, mapping the received orientation information of the plurality of sensors to the predetermined orientation of the user's hand posture includes determining a rotation axis from the first hand posture to the second hand posture using the orientation information received from the plurality of sensors. In addition, the method also includes determining a quaternion that minimizes the difference between the rotation axis and the associated virtual axis, and mapping the quaternion for each finger for all degrees of motion of the finger.

[0050] In one embodiment, a wearable device includes a transmitter, one or more processors, and a plurality of sensors located at predetermined locations for dynamically tracking hand gestures in a VR environment.

[0051] In the methods and systems of the related art, a vision / ultrasound-based method using a camera / transmitter mounted on a VR device is used to track the user's hand gestures. However, the vision-based method is limited by the field of view, the accuracy of detecting the hand position, and the computational complexity. Unlike the methods and systems of the related art, the proposed method uses a combination of a vision system and an inertial sensor on a wearable device to track the hand position when it is outside the field of view of the VR device.

[0052] In the wearable device of the related art, a resistive bend sensor is used to track the bend of each finger of the user to determine the movement of the user's hand. However, the bend sensor can detect the movement around one axis. Therefore, the wearable device has two or more bend sensors at each finger joint to capture two degrees of freedom, which makes the wearable device bulky and reduces the ease of use.

[0053] In the game environment of the related art, game consoles and wireless game controllers are used to interact with the game environment, which may be troublesome for users. In addition, the target interaction that the user wants to perform in the game environment may not be accurate. Unlike the methods and systems in the related art, the proposed method uses various sensors on the wearable device to track the position of the hand to perform various interactions in the interactive environment.

[0054] Referring now to the drawings and more particularly to Figure 1A , Figure 1B , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Fig. 6A , Figure 6B , Figure 6C , Figure 7 , Figure 8 , Fig. 9 , Fig. 10A , Fig. 10B , Fig.11 , Fig.12 and Fig.13, wherein like reference numerals indicate corresponding features consistently throughout the drawings, there is shown a preferred embodiment.

[0055] Figure 1A An example of hand gesture tracking relative to a VR device before calibrating a wearable device relative to the VR device according to an embodiment of the present disclosure is shown, and Figure 1B An example of hand pose tracking relative to a VR device after calibrating a wearable device relative to the VR device according to an embodiment of the present disclosure is shown.

[0056] refer to Figure 1A and Figure 1B , the wearable device 110 and the VR device 120 interact with each other. In other words, according to an embodiment of the present disclosure, the wearable device 110 and the VR device 120 send and receive data for providing VR services. In one embodiment, the wearable device 110 can obtain sensor data and send the sensor data to the VR device 120, and the VR device 120 can analyze the sensor data. In another embodiment, the wearable device 110 can obtain and analyze the sensor data and send the analysis result to the VR device 120. The VR device 120 can display virtual reality content based on the received data or results. In order to perform these operations, the wearable device 110 or the VR device 120 may include at least one of a transceiver, a processor, or a memory. In addition, the wearable device 110 may include at least one sensor, and the VR device 120 may include a display. In the present disclosure, the wearable device 110 or the VR device 120 may be referred to as an "electronic device".

[0057] Considering that the wearable device 110 is not calibrated with respect to the user's fitment. When the user holds a hand pose, the visualization of the hand pose in the VR environment will be different from the visualization of the hand pose held by the user. For example, when the user holds the hand pose by holding the hand parallel to the VR device 120, the visualization of the hand pose in the VR environment may not be parallel to the VR device 120, as in the embodiment of the present disclosure. Figure 1A shown.

[0058] The hand orientation in VR is a function of sensors on the wearable device 110 and the VR device 120. Therefore, when the wearable device 110 is calibrated relative to the user, the visualization of the hand posture held by the user will be correctly provided in the VR environment. When the user spreads all the fingers of the hand, the same visualization is provided in the VR environment, such as Figure 1B According to the embodiments of the present disclosure.

[0059] Figure 2 A wearable device for hand gesture tracking in a VR environment according to an embodiment of the present disclosure is shown.

[0060] refer to Figure 2 The wearable device 110 includes a glove 202 covering at least a portion of a thumb, at least a portion of an index finger, and at least a portion of a middle finger, a fixed member 204 coupled to the glove 202, a plurality of sensors 206 arranged on the glove and configured to detect movements of the thumb, the index finger, and the middle finger, and a processor 208 located on the fixed member 204 and configured to transmit sensor data obtained by the plurality of sensors. In addition, the wearable device 110 includes a battery 210 for powering the processor, at least one linear vibrator 212, and at least one ultrasonic transmitter 214.

[0061] Usually, the thumb, index finger and middle finger of the hand have independent movement and also have flexibility.In addition, when holding various hand postures, the ring finger and the little finger tend to move with the effect of the index finger and the middle finger.In addition, the proximal joint and the middle joint of the hand can represent most hand postures.The distal joint is quite dependent on the rotation of the middle joint.In addition, the position of different joints is different for different hand sizes of different users.Therefore, the preferred position of the sensor is different for different individual users.For example, a plurality of sensors 206 can include a first sensor arranged to detect the position of the middle phalanx of the thumb, a second sensor arranged to detect the position of the proximal phalanx of the thumb, a third sensor arranged to detect the position of the middle phalanx of the index finger, a fourth sensor arranged to detect the position of the proximal phalanx of the index finger, a fifth sensor arranged to detect the position of the middle phalanx of the middle finger, and a sixth sensor arranged to detect the position of the proximal phalanx of the middle finger.

[0062] The wearable device is designed to track the position of the proximal and middle joints of the user's index and middle fingers, and the metacarpal and proximal joints of the thumb. Figure 2 As shown, the wearable device may cover three fingers of the user's hand. However, the positions of the ring finger and the pinky finger are approximately estimated and used to track the positions of the user's fingers. Although the wearable device may be extended to cover all fingers of the user's hand, the proposed method includes covering three fingers of the user's hand with sensors to provide ease of use.

[0063] The wearable device 110 for hand gesture tracking in a VR environment includes two main functions, for example, tracking the position of the user's hand relative to the VR device and tracking the orientation of the user's hand. A set of inertial sensors 206 (e.g., inertial measurement unit (IMU) sensors) embedded in the wearable device 110 is used to track the orientation of the user's hand and the orientation of each finger. The IMU sensor includes at least one of an accelerometer, a gyroscope, or a magnetometer. In addition, the position of the user's hand relative to the VR device 120 is tracked by using an ultrasonic sensing system. The wearable device 110 is equipped with an ultrasonic transmitter 214, and the VR device is equipped with an ultrasonic receiver. Therefore, based on the ultrasonic waves received from the wearable device 110, the VR device 120 can track the position of the user's hand. In addition, the wearable device 110 is equipped with a 16MHz processor 208, which can provide a connection using Bluetooth, Bluetooth low energy (BLE), etc. The wearable device 110 also includes a battery 210 to power the sensor 206 for tracking various hand gestures of the user. The wearable device 110 also includes a linear vibrator 212 that generates vibrations. For example, the linear vibrator 212 may generate vibrations according to actions in a VR environment.

[0064] Figure 3 3 is a flowchart 300 illustrating a method for performing hand gesture tracking in a VR environment by a wearable device according to an embodiment of the present disclosure. Figure 3 The operation of the wearable device 110 according to an embodiment of the present disclosure is illustrated.

[0065] refer to Figure 3 In operation 302, the wearable device receives position information of multiple sensors for multiple hand postures of a user. The positions of the joints in the hand are different for different users. Therefore, the positions of the sensors in the wearable device are different for different users who provide different positions to the multiple sensors. Here, the position information refers to the position of the sensor relative to a reference frame. For example, the earth coordinates are used as the reference frame.

[0066] In operation 304, the wearable device maps the received orientation information of the multiple sensors to the predetermined orientation of the user's hand posture. In operation 306, the wearable device determines an absolute initial orientation relative to the VR device. The absolute initial orientation can be used to calculate a relative orientation relative to the VR device. In operation 308, the wearable device tracks the hand posture in the VR environment based on the orientation information received from the multiple sensors and one or more position tracking sensors in the VR device. For example, the one or more position tracking sensors include at least one ultrasonic sensor.

[0067] The various actions, behaviors, blocks, operations, etc. in the method may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some actions, behaviors, blocks, operations, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0068] exist Figure 3 In the illustrated embodiment, the operation is performed by a wearable device. In another embodiment, Figure 3 At least one operation shown can be performed by a VR device. For example, the wearable device obtains sensor data by using a sensor (e.g., an IMU sensor) and sends the sensor data to the VR device. Then, the VR device can perform operations 304, 306, and 308.

[0069] Figure 4 A method for determining the position of a user's hand based on sensor data fusion technology according to an embodiment of the present disclosure is shown.

[0070] The wearable device includes a set of IMU sensors to track various movements of the user's hands and individual fingers. The IMU sensors include at least one of an accelerometer, a gyroscope, or a magnetometer. Figure 4 In the proposed method, according to an embodiment of the present disclosure, a quaternion-based complementary filter is used to determine the orientation of the user's hand based on the results obtained from the accelerometer, magnetometer, and gyroscope. A quaternion is a four-element vector that represents a rotation in a three-dimensional (3D) coordinate system. The quaternion is uniquely determined for each hand posture. Therefore, a quaternion set including quaternions of multiple sensors can be used to detect hand postures. Accelerometer data and magnetometer data A combination of Davenport's q method and the Factored Quaternion method is used to fuse to obtain the absolute orientation of the sensor on the wearable device. Traditionally, the orientation calculated by the Davenport q method is inconsistent due to the influence of the earth's magnetic field inclination. The inconsistency is resolved by splitting the quaternion calculation into two operations. a ) is first calculated from the accelerometer measurements using Davenport's q method:

[0071] Accelerometer measurements,

[0072] Magnetometer measurements,

[0073] where q a It is through Derived using Davenport's q method.

[0074] The quaternion is used to tilt-compensate the magnetometer readings, and then Gives the heading quaternion.

[0075]

[0076] where × is quaternion multiplication.

[0077] if

[0078] but

[0079] Among them, h x represents the magnetic field along the x-axis, and is the in-plane angle of the Earth's magnetic north pole.

[0080]

[0081] q tc is a quaternion representing the orientation in the plane of the Earth’s magnetic north pole. These two quaternions are combined to give the final orientation of the sensor (q a_m ).

[0082] q a_m =q tc ×q a ...Equation 7

[0083] Quaternions are unstable due to noise in sensor measurements. In addition, a complementary filter based on the quaternion is used to compare it with the gyroscope data. Fusion, as shown below:

[0084] Gyroscope measurements,

[0085] q w = {cosθ / 2, (sinθ / 2*θ x / θ), (sinθ / 2*θ y / θ), (sinθ / 2*θ z / θ)}…Equation 9

[0086] Among them, θ x =w x dt, θ is the total amplitude of the rotation, and q w is the quaternion calculated from the gyroscope. Let q f Indicates q a_m and q w The fused quaternion between . The initial value of the fused quaternion is calculated using the accelerometer and magnetometer (q f，0 ),

[0087] q f，0 =q a_m，0...Equation 10

[0088] At a given time n, use q w，n The new orientation of the sensor is

[0089] q f_w，n =q f，n-1 ×q w，n ...Equation 11

[0090] The corresponding sensor orientation calculated from the accelerometer and magnetometer is q a_m，n In addition, calculate q f_dev，n , n, which is the quaternion q f_w，n and q a_m , n deviation.

[0091]

[0092] This is a measure of the drift of the quaternion calculated using the gyroscope only. The fused quaternion is calculated by taking a fraction of the above deviation and adding it to the gyroscope only quaternion. f_dev The scaling is for the scalar component (q f_dev，n ) s The scalar component is the absolute rotation angle.

[0093] q f_dev_s，n =0.02*(q f_dev，n ) s +(q f_dev，n ) v …Equation 13

[0094] q f，n =q f_w，n ×q f_dev_s，n ...Equation 14

[0095] Fused quaternion q f is the final orientation of the sensor at time n. The importance of the gyroscope must be increased by reducing the value of the scaling parameter. As shown in equations 10 and 11, the quaternion calculated from the gyroscope is relative to the initial orientation q a_m，0 The accuracy of the initial orientation is critical to the accurate prediction of any subsequent orientation. The initial orientation is derived from the accelerometer and magnetometer readings, which are known to be noisy. In addition, a number of accelerometer and magnetometer measurements are averaged while the wearable is stationary, and then a stable q is calculated. a_m，0 value.

[0096] A hierarchical representation of the user's hand is used, where relative to the proximal joint q int_prox Calculate the orientation of the middle finger joints relative to the orientation of the palm qprox_palm Calculate the orientation of the proximal joint. This can be expressed as follows:

[0097] q prox_palm|s =q palm|s -1 ×q palm|s ...Equation 15

[0098] q int_prox|s =q prox|s -1 ×q int|s v...Equation 16

[0099] The subscript S refers to the global coordinate system of the sensor. When the wearable device moves as a rigid body, the compensation quaternion ensures that the finger orientation relative to the palm always remains unchanged. The accuracy of the sensor fusion algorithm can be inferred from the deviation of the compensation quaternion from the rigid body motion. When the VR application is viewed on a VR device, the orientation of the wearable device is calculated in the VR device reference coordinate system. The orientation of the palm sensor is compensated relative to the orientation of the VR device. This provides the user with a more realistic experience of the VR content as the orientation of the hand model changes as the VR device moves.

[0100] Another aspect of the representation of the user's hands in a VR environment is tracking the position of the hands relative to an external reference or to the VR device. Wearable devices use an inside out tracking approach, where sensors mounted on the VR device track the relative position of the hands. In order to track the user's hand posture outside the field of view of the VR device, a fusion of an ultrasonic tracking system and inertial sensors is used. An ultrasonic receiver is mounted on the VR device and an ultrasonic transmitter is mounted on the wearable device. An inertial sensor is placed near the transmitter to calculate the position outside the field of view (FoV). The FoV and accuracy of the position estimate are a function of the orientation of the ultrasonic transmitter relative to the ultrasonic receiver. The ultrasonic transmitter is placed on the back of the palm, such as Figure 2 shown.

[0101] The orientation and position of the hand are determined by the wearable device in the coordinate system of the sensor and must be mapped to the reference frame of the virtual hand model. The coordinate axes of the position measurements from the ultrasonic receiver are aligned with the VR device and thus with the VR application. Therefore, this does not require axis mapping. The orientation of the sensors on the wearable device is determined in the reference frame of the VR device. In order to determine the corresponding orientation in the virtual model, the mapping quaternion (qmap) is calculated. The mapping quaternion is calculated for each sensor and its corresponding representation in the model. The hand model in the VR application is built in unity (Unity) and has a hierarchical structure for each joint. The orientation of the intermediate joints is relative to the proximal joints (q int_prox ), and then relative to the position of the palm (qprox_palm ).

[0102] q palm|M =q -1 palm_map ×q palm|s ×q palm_map ...Equation 17

[0103] q palm|M =q -1 prox_map ×q prox_palm|s ×q prox_map ...Equation 18

[0104] Here, q palm_map and q prox_map represents the mapping quaternion of the palm and proximal bones, and the subscript M represents the coordinate system of the VR hand model. In addition,

[0105] q meta|M =q palm|M ×q meta_palm|M ...Equation 19

[0106] q prox|M =q meta|M ×q prox_meta|M ...Equation 20

[0107] q int|M =q prox|M ×q int_prox|M ...Equation 21

[0108] Figure 5 Specific hand gestures are shown for performing user-specific calibration of a wearable device according to embodiments of the present disclosure (although not limited to only these gestures).

[0109] Wearable devices fit the hands of different users. Therefore, before using wearable devices in a VR environment, the orientation of each sensor on the wearable device must be captured for each user.

[0110] To determine a change in a hand pose of a user's hand, sensor data that undergoes a change due to the performance of the hand pose is selected. In various embodiments, at least one predefined hand pose is used. The hand pose is defined based on at least one of a gap or angle between fingers, an angle of each finger, a bend of the fingers, contact between fingers, or an angle of the hand. Figure 5 Examples of hand gestures 501 - 505 are shown.

[0111] Consider hand pose 501 and hand pose 502; and hand pose 502 and hand pose 503, such as Figure 5As shown. In operation 510, the relative change from hand posture 501 to hand posture 502 is determined based on the change of the rotation axis of the user's hand. In addition, in operation 512, the relative change from hand posture 502 to hand posture 503 is determined based on the change of the rotation axis of the user's hand. In addition, in operation 514, a quaternion that minimizes the difference between the rotation axes and their corresponding virtual world axes is determined. Then, in operation 516, the quaternion is mapped to the user's index finger. That is, through calibration, the mapping relationship between the quaternion and the hand posture is redefined to optimize the user's hand.

[0112] Based on equations 17 and 18, the mapping uses known quaternions in the sensor frame and the corresponding model frame. Since there are 3 unknowns in the qmap, three hand poses are used for each joint. The hand poses will be performed by a new user wearing the wearable device during the calibration operation, and the wearable device will be available thereafter. Figure 5 As shown, a set of 6 hand poses 501-506 are used to map all 7 sensors of the wearable device. The hand poses are selected so that all sensor quaternion measurements are captured with minimal user input. A single hand pose can be used to map multiple joints if there is orientation variation in the corresponding sensors.

[0113] refer to Figure 5 According to an embodiment of the present disclosure, hand pose 501 is used as a reference orientation for all sensors, and quaternions corresponding to each subsequent hand pose are calculated relative to hand pose 501. Hand poses 503, 504, and 506 are used to map the palm sensor, hand poses 502 and hand pose 506 map the index and middle fingers, and hand poses 502, 503, and 505 map the thumb. The index and middle fingers can be calculated using only two hand poses because these fingers have two degrees of freedom.

[0114] Alternatively, we can also use only a single posture to map the orientation of all sensors on the device to the corresponding virtual frame. We find that the mapping quaternion is determined using the reference posture and the corresponding sensor orientation and the orientation in the virtual model. In one embodiment, the VR device can display a guide for requesting to take a single posture and place the hand wearing the wearable device in a predetermined space. The VR device can then obtain at least one image of the hand and calibrate the sensor. Here, at least one image can be obtained by using a camera or an infrared (IR) sensor. For example, the VR device calibrates the sensor by analyzing at least one image, identifying the position and orientation of the sensor, and defining a rule (e.g., function) for converting from the sensor's measurement value to the hand posture based on the sensor's position and orientation. In another embodiment, another wearable device (e.g., a ring device or a bracelet device) can be used with the wearable device.

[0115] For a given set of sensor and model quaternions, the qmap is the effective rotation that, when applied to the sensor orientation, minimizes the error between the predicted and known model orientations. The vector components of the quaternion used in the qmap calculation are used when the same absolute rotation is used in computing the model's quaternion. Davenport's q method is used, where for all hand poses i, the following function is minimized over q.

[0116] L(q)=∑ i＝2，3 (q -1 ×((q palm|s ) v ×q)-q palm|M ...Equation 22

[0117] (q palm|s ) v are the vector components of the palm orientation in the sensor coordinate system for each hand pose. The above example is for the palm sensor, but is repeated independently for each sensor. The best value of q will be the mapped quaternion for that sensor. Once qmap is determined, any new sensor orientation can be mapped to the representation in the model (Equations 17 and 18). The representation of the intermediate skeleton is much simpler because it has one degree of freedom. Therefore, the bend angle (θ) calculated from the sensor is directly mapped to the bend axis in the virtual model.

[0118] θ=cos -1 ((q int_prox|S ) s )...Equation 23

[0119] q int_prox|M ={cosθ, -sinθ, 0, 0} ... Equation 24

[0120] During the mapping of the index and middle fingers, the effective rotation angle for hand pose 2 (abduction) is much smaller than that for hand pose 6 (flexion). The vector components of the quaternion corresponding to each hand pose are inverse functions of the sine of the rotation angle. The smaller the angle, the higher the variability of the calculated vector, which further affects the qmap calculation. In Davenport's q-method, the hand pose is weighted so that the greater the movement of the hand pose, the more the hand pose is weighted. For the proximal joints of the index and middle fingers, the weights for flexion and abduction were 0.9 and 0.1, respectively, which were determined empirically. When the various sensors on the wearable device are calibrated relative to the user's hand, the hand poses performed by the user can be accurately represented in the VR environment.

[0121] Fig. 6A , Figure 6B and Figure 6CA method for determining the approximate finger positions of the ring finger and the little finger of the user's hand that are not covered by the wearable device according to an embodiment of the present disclosure is shown.

[0122] The position of the ring finger and little finger is similar to that of the index finger and middle finger, such as Fig. 6A and Figure 6B As shown in FIG. 1 , the wearable device tracks the positions of three fingers, namely the thumb, index finger, and middle finger of the user's hand. However, the positions of the ring finger and pinky finger are determined based on the middle finger.

[0123] Θ 2 =0.75*Θ 1

[0124] Θ 3 =1.5*Θ 1

[0125] Among them, Θ 1 are the Euler angles around the abduction axis of rotation. Quaternion calculations are used to determine the orientation (abduction) of the ring and pinky fingers. The curvature of the proximal bone of the middle finger is used to update the quaternions corresponding to the curvature of the ring and pinky fingers. The curvature angles are scaled by a factor of 1.1 to give a realistic representation.

[0126] In addition, if Figure 6C As shown, the distal joint on each finger is approximated by the medial joint orientation. The curvature of the distal bone is approximated by the medial bone (a portion of the medial bone curvature). The mapping of the thumb is a bit more complicated because it has 3 degrees of freedom at the metacarpal joint. The orientation changes measured by the sensor placed on the metacarpal of the thumb are very small, even in the limbs. Due to the small changes in the measurements, its mapping to the virtual model is prone to high errors. Alternatively, the relative orientation of the proximal bones with respect to the metacarpals and the proximal bones with respect to the palm is quite large. These measurements are used to determine the orientation of the metacarpal bones in the model coordinate system.

[0127] q meta_palm|M =q prox_palm|M ×(q prox_palm|M ×q meta_prox|M )×q -1 prox_palm|M ...Equation 25

[0128] The wearable device has no sensors on the ring finger and pinky finger, and their orientations are approximated to those of the middle and index fingers. The flexion angles of the ring and pinky fingers are obtained by scaling the flexion angle of the middle finger. The Euler angles are rotations around a single axis ∈ rinh and (∈ ring ) x Used, such as Figure 2 shown.

[0129] (∈ pinky|M )Z =0.75*(∈ middle_index|M ) Z ...Equation 26

[0130] (∈ pinky|M ) x =1.1*(∈ middle|M ) x ...Equation 27

[0131] Similarly, for the little finger,

[0132] (∈ pinky|M ) Z =1.5*(∈ middle_index|M ) Z ...Equation 28

[0133] (∈ pinky|M ) x =1.21*(∈ middle|M ) x ...Equation 29

[0134] Furthermore, according to empirical data, the maximum observable rotation of the ring finger and pinky finger relative to the initial reference orientation is limited. The limitation of this system is in the case where the ring finger and pinky finger move independently. After all sensors are mapped to the corresponding joints in the hand model, any new hand pose of the hand can be directly reconstructed from the sensor data.

[0135] Figure 7 An example of calibrating a wearable device relative to a VR device according to an embodiment of the present disclosure is shown.

[0136] The user places both hands on a flat surface of the VR device 120, such as the front of the VR device 120 and one of the sides of the VR device 120. Figure 7 When the hand is placed on the front of the VR device 120 and one of the sides of the VR device 120, the relative orientation between the user's hand and the VR device 120 is determined. In addition, the wearable device stores the relative orientation information, which is used for all subsequent predictions of the orientation of the user's hand.

[0137] Typically, the fit of the wearable for each user is different. Therefore, the position of the sensor on the wearable device and the orientation of the wearable device relative to the VR device also vary from individual user to individual user. The proposed method includes performing a calibration to determine the orientation of the sensor for each user to obtain an accurate VR representation of the hand. In addition, the rotation axis of the sensor should be mapped to the rotation axis of the VR device. The user is usually asked to perform a specific hand gesture. In addition, the method includes detecting the user's hand gesture and mapping accordingly. Initially, the user should perform a specific hand gesture. In this example, the user is asked to perform abduction. The orientation of the sensor on each finger is determined relative to the sensor on the palm of the user. The angle between each two fingers is determined, and the relative quaternion is mapped to the virtual environment, that is, for two hand gestures, the angle between the index finger and the thumb is mapped differently in the virtual environment. Therefore, the proposed method ensures that the sensor orientation is accurately mapped.

[0138] Figure 8 A comparison of sensor orientation calculated from Davenport's q method and a sensor fusion method for estimating orientation according to an embodiment of the present disclosure is shown.

[0139] The stability and accuracy of the proposed fusion method are determined based on two experiments. The sensor is kept stable and the fused quaternion is calculated relative to the initial quaternion to determine the stability of the proposed method. Figure 8 The predicted quaternions are compared with those from Davenport's Q-method. Moreover, a significant improvement in the stability of the quaternions is observed in the proposed method. The deviation in the proposed method is less than 3°, which is mainly due to the fluctuations in the magnetometer measurements.

[0140] Fig. 9 A device for determining the accuracy of a sensor fusion method for orientation estimation using a protractor as a reference according to an embodiment of the present disclosure is shown.

[0141] According to an embodiment of the present disclosure, the accuracy of the absolute orientation predicted by the fusion method by rotating the sensor around a known reference can be calculated using Fig. 9 to determine the settings shown.

[0142]

[0143] Among them, q initial is the initial orientation, and q current is the current position. Therefore, q delta A quaternion representing the current orientation of the sensor rotated into the initial reference coordinate frame.

[0144] Fig. 10ATracking the position of a user's hand relative to a VR device when the user's hand is within the field of view of an IMU sensor according to an embodiment of the present disclosure is shown.

[0145] In one embodiment, the camera / ultrasonic sensor of the VR device 120 (i.e., an embedded camera or a related camera accessory) can be used to track the position of the user's hand relative to the VR device 120. In addition to the inertial sensors on the wearable device, the camera is used to determine the position of the user's hand relative to the VR device 120. Fig. 10A , the user's hand is within the field of view of the camera, so the position of the user's hand can be tracked relative to the VR device 120. Therefore, when the user's hand is within the field of view of the VR device 120, the hand gestures performed by the user are accurately tracked and mapped in the VR environment.

[0146] Fig. 10B Hand gesture tracking in a VR environment using a camera and an IMU sensor when the user's hand is outside the field of view of the IMU sensor according to an embodiment of the present disclosure is shown.

[0147] refer to Fig. 10B , when the user's hand is outside the field of view of the VR device, the inertial sensor embedded in the wearable device 110 is used to track the position of the user's hand relative to the VR device 120. Therefore, according to an embodiment of the present disclosure, based on the fusion of the camera / ultrasonic sensor and the inertial sensor, the hand gestures performed by the user when the user's hand is outside the field of view of the VR device 120 are accurately tracked and mapped in the VR environment.

[0148] Fig.11 A comparison of an expected rotation angle and a predicted rotation angle obtained by a sensor fusion method for orientation estimation according to an embodiment of the present disclosure is shown.

[0149] refer to Fig.11 According to an embodiment of the present disclosure, a graph of the rotation angle is plotted with respect to time. In addition, a graph of the expected rotation angle of the user's hand and the predicted rotation angle is obtained. The graph represents the absolute orientation obtained from the sensor on the wearable device, 2*cos -1 ((q delta )) s1 are the five rotation angles expected from the protractor readings. The maximum deviation between the expected and predicted rotation angles of the user's hand is less than 3°.

[0150] A wearable device for tracking a user's hand pose in a VR environment uses a compensated quaternion to determine the orientation of each finger joint. In addition, the accuracy of the proposed compensated quaternion is quantified by mounting the sensor on a rigid body and moving the sensor and the rigid body together. At some positions, the maximum observed deviation is 10°. However, the average deviation is 2.9°. A mapping quaternion is determined for each finger joint using a set of predetermined hand poses. Table 1 shows the error in the predicted orientation of the hand model of the palm and index finger using the mapping quaternion and the corresponding target orientation. According to an embodiment of the present disclosure, the coordinate system of the Euler angle is related to Figure 2 The motion is made independently along the two principal axes, so the expected Euler angles can be measured using a protractor. The error in the predicted angles varies from less than 1 degree to as much as 19.5°. In most axes, the error is in the 5-10° range. The error increases with the bending angle, and the maximum percentage error is about 15-20% for small or large motions.

[0151] Table 1

[0152]

[0153] The hardware design plays a key role in the corresponding accuracy of the sensor measurements and the corresponding hand pose representation. It is important that the inertial sensors mounted on the wearable device move with the fingers and do not slip.

[0154] Fig.12 An example scenario for determining the absolute position of a user's hand according to an embodiment of the present disclosure is shown.

[0155] In order to obtain a complete virtual representation of the hand, the absolute initial orientation information of the user's hand relative to the VR device should be determined. Therefore, when the VR device is started to determine the absolute orientation of the user's hand, the user must place the hand in a stable position for a fixed duration. In addition, the absolute initial orientation information must be determined every time a new user uses the VR device.

[0156] The proposed method provides a user interface (UI) that ensures that the user's hands are stable when the VR device is turned on. In addition, the UI can provide virtual buttons that can be clicked by keeping the hands stable.

[0157] In another embodiment, other UI elements may be used to provide a side push / pull button or the like.

[0158] Fig.13 13 is a flowchart 1300 showing a method for performing hand gesture tracking in an interactive environment through a wearable device according to an embodiment of the present disclosure.

[0159] The interactive environment may be, for example, a VR environment, an augmented reality (AR) environment, a mixed reality (MR) environment, and a gaming environment, where the user's hand gestures may be used to perform various interactions in the interactive environment.

[0160] refer to Fig.13 In operation 1302, the wearable device receives position information of multiple sensors for multiple hand gestures of the user. The positions of the joints in the hand are different for different users. Therefore, according to an embodiment of the present disclosure, the positions of the sensors in the wearable device are different for different users who provide different positions to the multiple sensors.

[0161] In operation 1304, the wearable device maps the received position information of the plurality of sensors with a predetermined position of the user's hand gesture.

[0162] At operation 1306, the wearable device determines an absolute initial orientation relative to the interactive electronic device.

[0163] At operation 1308, the wearable device dynamically tracks the hand gesture in the interactive environment based on the position information received from the plurality of sensors and the one or more position tracking sensors in the interactive electronic device.

[0164] The various actions, behaviors, blocks, operations, etc. in the method may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some actions, behaviors, blocks, operations, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0165] The embodiments disclosed herein may be implemented using at least one software program running on at least one hardware device and performing network management functions to control the elements.

[0166] Various aspects of the present disclosure may also be embodied as computer-readable code on a non-transitory computer-readable recording medium. A non-transitory computer-readable recording medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of non-transitory computer-readable recording media include read-only memory (ROM), random access memory (RAM), compact disk (CD)-ROM, magnetic tape, floppy disk, and optical data storage device. Non-transitory computer-readable recording media may also be distributed on network-coupled computer systems to store and run computer-readable code in a distributed manner. Furthermore, the functional programs, codes, and code segments used to implement the present disclosure may be easily interpreted by programmers in the field to which the present disclosure belongs.

[0167] At this point, it should be noted that the various embodiments of the present disclosure as described above generally involve the processing of input data and the generation of output data to some extent. The input data processing and output data generation can be implemented with hardware or software combined with hardware. For example, specific electronic components can be used in mobile devices or similar or related circuits to implement the functions associated with the various embodiments of the present disclosure as described above. Alternatively, one or more processors operating according to stored instructions can implement the functions associated with the various embodiments of the present disclosure as described above. If this is the case, it is also within the scope of the present disclosure that such instructions can be stored on one or more non-transitory processor-readable media. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Processor-readable media can also be distributed on network-coupled computer systems, so as to store and execute instructions in a distributed manner. And, the functional computer programs, instructions, and instruction segments used to implement the present disclosure can be easily interpreted by programmers in the field to which the present disclosure belongs.

[0168] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for operating an electronic device, the method comprising: displaying a guide for requesting the user to position the hand in a posture of the user's hand in a predetermined space; identifying a position and orientation of a plurality of sensors configured to detect an orientation of a finger relative to a hand of a user; obtaining at least one image of a user's hand; calibrating the plurality of sensors based on at least one image, the positions and orientations of the plurality of sensors, and rules defined based on the positions and orientations, wherein the rules for calibration are defined for conversion from measurements of the sensors to hand poses; identifying sensor data obtained by the plurality of sensors; and A hand gesture is tracked based on the sensor data.

2. The method according to claim 1, wherein: Tracking hand gestures includes: identifying position information of the plurality of sensors based on the sensor data; mapping the position information to a predetermined position of a user's hand gesture; Determining an initial orientation of the wearable device relative to the virtual reality (VR) device; and The hand posture is tracked in the VR environment based on the position information.

3. The method according to claim 1, wherein: Display guide includes: displaying guide information for requesting to maintain at least one placement of the user's hand; and Among other things, calibrating multiple sensors includes: The plurality of sensors are calibrated based on at least one of a hand gesture or a hand placement of a user.

4. An electronic device comprising: monitor; Transceiver; and at least one processor electrically coupled to the transceiver and the display and configured to: displaying a guide for requesting the user to position the hand in a posture of the user's hand in a predetermined space; identifying a position and orientation of a plurality of sensors configured to detect an orientation of a finger relative to a hand of a user; obtaining at least one image of a user's hand; calibrating the plurality of sensors based on at least one image, the positions and orientations of the plurality of sensors, and rules defined based on the positions and orientations, wherein the rules for calibration are defined for conversion from measurements of the sensors to hand poses; identifying sensor data obtained by the plurality of sensors; and A hand gesture is tracked based on the sensor data.

5. The electronic device according to claim 4, wherein: To track hand gestures, the at least one processor is further configured to: identifying position information of the plurality of sensors based on the sensor data; mapping the position information to a predetermined position of a user's hand gesture; Determining an initial position of the wearable device relative to the electronic device; as well as A hand gesture is tracked in a virtual reality (VR) environment based on the position information.

6. The electronic device according to claim 5, wherein: The at least one processor is also configured to receive a relative position of a user's hand relative to the electronic device.

7. The electronic device according to claim 5, wherein: The plurality of sensors are placed at predetermined locations on the wearable device to track the movement of a user's hands relative to the electronic device.

8. The electronic device according to claim 5, wherein: The at least one processor is further configured to: determining an axis of rotation from a first hand pose to a second hand pose using orientation information received from a plurality of sensors; determining a quaternion that minimizes a difference between the rotation axis and an associated virtual axis; and For all degrees of motion of the fingers, a quaternion is mapped for each finger.

9. The electronic device according to claim 5, wherein: The wearable device includes one or more position tracking sensors, one or more processors, and a plurality of sensors located at predetermined locations for tracking hand gestures in a VR environment.

10. The electronic device according to claim 9, wherein: When the hand is out of the field of view relative to the electronic device, the at least one processor is further configured to: receiving position information of the plurality of sensors; as well as Based on the orientation information and information from one or more position tracking sensors in the electronic device, a hand gesture is tracked in a VR environment.

11. The electronic device according to claim 4, wherein: To display the guide, the at least one processor is further configured to: displaying guide information for requesting to maintain at least one placement of the user's hand; and In order to calibrate the multiple sensors, the at least one processor is further configured to: calibrate the multiple sensors based on a hand posture or at least one hand placement of the user.

Images