Motion computing system and motion computing method applied to mixed reality
By combining wearable devices and head-mounted displays, and utilizing inertial measurement units and hand tracking algorithms, the problems of inertial measurement unit drift error and head-mounted display blind spot error have been solved, enabling more accurate interactive input and free operation with both hands.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HTC CORP
- Filing Date
- 2023-03-03
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, inertial measurement units have drift errors, controllers cannot free the user's hands, and head-mounted displays have blind spot errors, resulting in deviations in interactive input data.
By combining wearable devices and head-mounted displays, and using inertial measurement units and hand tracking algorithms, the device's position is set, its position and rotation are identified, the indicator direction is calculated, and a beam of light is generated to achieve precise interaction.
It eliminates inertial measurement unit drift error and head-mounted display blind spot error, improves the accuracy of interactive input, and allows users to interact with virtual reality scenes with their hands free.
Smart Images
Figure CN116700475B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a mixed reality technology, and more particularly to a motion computing system and motion computing method applied to mixed reality. Background Technology
[0002] To provide intuitive operation of electronic devices (e.g., game consoles, computers, smartphones, smart home appliances, etc.), these devices can detect user movements to perform actions directly based on those movements. Traditionally, technologies such as Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR) are widely used to simulate sensations, perception, and / or environment. Controllers, inertial measurement units (IMUs), and head-mounted displays are often used to detect interactive input data (e.g., input via pressing controller buttons), inertial data of the user's hands, and the user's hand skeleton, respectively, to interact with the virtual reality scene. However, commonly used IMUs suffer from drift errors, commonly used controllers cannot free the user's hands, and commonly used head-mounted displays have dead zone errors. Therefore, these factors introduce bias into the interactive input data. Summary of the Invention
[0003] This disclosure provides a motion computing system for mixed reality, including a wearable device and a head-mounted display. The wearable device includes an inertial measurement unit (IMU) for detecting inertial data. The head-mounted display is coupled to the wearable device and is used to perform the following steps: setting a wearing position of the wearable device, wherein the wearing position represents the position where the wearable device is worn on a user's hand; determining whether a hand tracking algorithm has detected a hand model with finger skeleton data of the user's hand in a monitored scene; in response to detecting a hand model in the monitored scene, identifying the device position of the wearable device based on the wearing position and finger skeleton data, and identifying the device rotation amount of the wearable device in the monitored scene based on the inertial data; calculating an indicator direction in the monitored scene based on the device position and device rotation amount; and generating a light beam in the virtual reality scene based on the indicator direction and device position.
[0004] In some embodiments of motion computing systems applied to mixed reality, the head-mounted display is further used to perform the following steps: in response to the absence of a hand model in the monitored scene, identifying the device position of the wearable device based on inertial data and the latest available device position of the wearable device, and identifying the amount of device rotation of the wearable device in the monitored scene based on the inertial data; calculating the indicator direction in the monitored scene based on the device rotation and corresponding to the inertial data and the latest available device position; and generating a beam in the virtual reality scene based on the indicator direction and corresponding to the inertial data and the latest available device position.
[0005] In some embodiments of motion computing systems applied to mixed reality, the head-mounted display is further used to perform the following steps: after generating a beam of light in the virtual reality scene, for the hand, performing a hand tracking algorithm on the monitoring scene to determine whether a new hand model is found in the monitoring scene; in response to the discovery of a new hand model of the user's hand in the monitoring scene, updating the device position in the monitoring scene based on inertial data and the new hand model; updating the device rotation amount of the wearable device in the monitoring scene based on inertial data; and updating the indicator direction in the monitoring scene based on the device position and the updated device rotation amount, and generating a beam of light in the virtual reality scene based on the indicator direction and the device position.
[0006] In some embodiments of motion computing systems applied to mixed reality, the step of performing a hand tracking algorithm on a monitored scene to determine whether a new hand model is found in the monitored scene includes: capturing the monitored scene to generate a recent scene image, and determining whether multiple joint data representing multiple joints of the hand are found; and in response to finding multiple joint data, generating a new hand model based on the multiple joint data.
[0007] In some embodiments of motion computing systems applied to mixed reality, the step of performing a hand tracking algorithm on a monitored scene to determine whether a new hand model is found in the monitored scene further includes: performing an image recognition algorithm on recent scene images to determine whether a hand exists in the monitored scene; and in response to the presence of a hand in the monitored scene, identifying multiple joint data representing multiple joints of the hand, and generating a new hand model based on the multiple joint data.
[0008] In some embodiments of motion computing systems applied to mixed reality, the detection frequency of inertial data is higher than the detection frequency of the hand tracking algorithm, wherein the head-mounted display is used to perform the following steps: between the time of generating a hand model and the time of generating a new hand model, the device position of the wearable device in the monitoring scene is updated by integrating the inertial data.
[0009] In some embodiments of motion computing systems applied to mixed reality, the step of updating the device position in the monitoring scene based on inertial data and a novice hand model includes: updating the device position of the wearable device in the monitoring scene by integrating the inertial data, and calibrating the device position based on the novice hand model.
[0010] In some embodiments of motion computing systems applied to mixed reality, the head-mounted display is further used to perform the following steps: in response to the absence of a new hand model in the monitored scene, identifying the device position of the wearable device based on inertial data and the latest available device position of the wearable device, and identifying the amount of device rotation of the wearable device in the monitored scene based on the inertial data; calculating the indicator direction in the monitored scene based on the device rotation and corresponding to the inertial data and the latest available device position; and generating a beam in the virtual reality scene based on the indicator direction and corresponding to the inertial data and the latest available device position.
[0011] In some embodiments of motion computing systems applied to mixed reality, the hand model and the new hand model each contain joint data representing multiple joints of the hand.
[0012] In some embodiments of motion computing systems applied to mixed reality, the step of setting the wearing position of a wearable device includes setting the wearing position of the wearable device to one of a plurality of phalanges of the hand.
[0013] This disclosure provides a motion calculation method for mixed reality, comprising: detecting inertial data and setting a wearing position of a wearable device, wherein the wearing position represents the position where the wearable device is worn on a user's hand; determining whether a hand tracking algorithm detects a hand model with finger skeleton data of the user's hand in a monitoring scene; in response to detecting a hand model in the monitoring scene, identifying the device position of the wearable device based on the wearing position and finger skeleton data, and identifying the device rotation amount of the wearable device in the monitoring scene based on the inertial data; calculating the indicator direction in the monitoring scene based on the device position and device rotation amount; and generating a light beam in the virtual reality scene based on the indicator direction and device position.
[0014] The foregoing and other features, aspects and advantages of this disclosure will be better understood by referring to the following description and the scope of the patent application.
[0015] It should be understood that the foregoing general description and the following detailed description are examples intended to provide further explanation of this disclosure. Attached Figure Description
[0016] To make the above and other objects, features, advantages and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described below:
[0017] Figure 1 This is a functional block diagram of a motion calculation system applied to mixed reality according to some embodiments;
[0018] Figure 2A This is a flowchart of a motion calculation method applied to mixed reality according to some embodiments;
[0019] Figure 2B A flowchart showing further steps of a motion calculation method applied to mixed reality according to some embodiments;
[0020] Figure 2C A flowchart showing further steps of a motion calculation method applied to mixed reality according to some embodiments;
[0021] Figure 2D A flowchart showing further steps of a motion calculation method applied to mixed reality according to some embodiments;
[0022] Figure 3 This is a schematic diagram of a surveillance scene according to some embodiments;
[0023] Figure 4 This is a schematic diagram illustrating the generation of a hand model according to some embodiments;
[0024] Figure 5 This is a schematic diagram illustrating the setting of the indicator direction according to some embodiments;
[0025] Figure 6 A schematic diagram of the position values of an axis according to some embodiments; and
[0026] Figure 7 This is a schematic diagram of generating a light beam in a virtual reality scene according to some embodiments.
[0027] Symbol explanation:
[0028] 100: Action Calculation System
[0029] 110: Wearable devices
[0030] 111: Inertial Measurement Unit
[0031] 120: Head-mounted display
[0032] 121: Camera
[0033] 122: Hand tracking module
[0034] 123: Device Tracking Module
[0035] S210~S290, S230'~S250', S270'~S290': Steps
[0036] MF: Monitoring Scene
[0037] HND: Hand
[0038] TIP: Fingertips
[0039] Joint0~Joint18: Joints
[0040] Palm: hand
[0041] Wrist: wrist
[0042] WP: Wearing Location
[0043] PT: Hand area
[0044] VD: Virtual Device
[0045] PD: Indicator Direction
[0046] TS1~TS3: Time Series
[0047] t1~t5: Detection time points
[0048] IP, IMUP, PT1, PT2, OP1, OP2: Device location
[0049] VRF: Virtual Reality Scene
[0050] RY: Beam
[0051] VH: Virtual Hand Detailed Implementation
[0052] The embodiments of this disclosure will be described below with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similar elements or method flows.
[0053] Please refer to Figure 1 , Figure 1 This is a functional block diagram of a motion computing system 100 applied to mixed reality according to some embodiments. The motion computing system 100 applied to mixed reality includes a wearable device 110 and a head-mounted display 120. The head-mounted display 120 is coupled to the wearable device 110.
[0054] In some embodiments, the wearable device 110 may be a handheld device or a ring-shaped device. In some embodiments, the wearable device 110 may include one or more buttons for interactive input. In some embodiments, the wearable device 110 does not have any markings (e.g., light-emitting diodes or specific patterns) that can be recognized by any computer vision algorithm (e.g., convolutional neural networks or YOLO (You Only Look Once)).
[0055] Furthermore, the wearable device 110 includes an inertial measurement unit 111 for detecting inertial data. In some embodiments, the inertial measurement unit 111 may be an electronic device using a combination of an accelerometer and a gyroscope (sometimes further including a magnetometer) for measuring and reporting the specific force, angular velocity, and acceleration (sometimes further including the orientation of the wearable device 110) of the wearable device 110. In some embodiments, at multiple detection time points, the inertial measurement unit 111 senses the movement of corresponding body parts (e.g., fingers) of the user wearing the wearable device 110 to generate a sequence of inertial data including the sensed data based on the sensing results (e.g., acceleration, angular velocity, magnetic force, etc.) of the inertial measurement unit 111.
[0056] In some embodiments, the inertial measurement unit 111 may consist of a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer and the three-axis gyroscope can detect acceleration and angular velocity, and the inertial measurement unit 111 can generate inertial data (i.e., six-degrees-of-freedom (6DoF) sensing data) based on the acceleration and angular velocity.
[0057] In some embodiments, the head-mounted display 120 may be a virtual reality headset or other head-mounted device used in mixed reality. In some embodiments, the head-mounted display 120 may display a virtual reality scene and virtual objects located within the virtual reality scene. For example, the virtual reality scene may be a virtual classroom, and the virtual objects may be virtual tables.
[0058] In some embodiments, the head-mounted display 120 may include a camera 121, a hand tracking module 122, and a device tracking module 123. In some embodiments, the camera 121 may be a camera for capturing images or a camera with continuous shooting capabilities to capture a monitored scene (e.g., the space where the user is located). In some embodiments, the hand tracking module 122 and the device tracking module 123 may be implemented by any software (e.g., Python), hardware (e.g., a processor, memory, or processing circuitry), or any combination thereof. For example, the hand tracking module 122 and the device tracking module 123 may be program modules stored in the memory of the head-mounted display 120, and the processor of the device tracking module 123 may execute the hand tracking module 122 and the device tracking module 123.
[0059] In some embodiments, at multiple detection time points, the hand tracking module 122 can perform a hand tracking algorithm on the images captured by the camera 121. In some embodiments, the device tracking module 123 can capture the device position and rotation amount of the wearable device 110 in the monitored scene. Detailed steps for operating the hand tracking module 122 and the device tracking module 123 will be described in subsequent paragraphs. In some embodiments, the detection frequency of inertial data is higher than the detection frequency of performing the hand tracking algorithm (i.e., the number of detection time points of the inertial measurement unit 111 is greater than the number of detection time points of the hand tracking module 122).
[0060] In some embodiments, the device position can be a coordinate in a three-axis space, and the device rotation can be three rotations in three rotation directions along the three axes (e.g., rotation in the rotation direction along the X-axis in the three-axis space).
[0061] Please refer to further details. Figure 2A , Figure 2A This is a flowchart of a motion calculation method applied to mixed reality according to some embodiments. Figure 1 The action calculation system 100 shown can be used to execute Figure 2A The action calculation method is shown.
[0062] like Figure 2A As shown, firstly, in step S210, inertial data is detected, and the wearing position of the wearable device is set. In this embodiment, the wearable device 110 detects inertial data, and the head-mounted display 120 sets the wearing position of the wearable device, wherein the wearing position indicates the position where the wearable device 110 is worn on the user's hand.
[0063] In some embodiments, the wearing position can be set by the user via the head-mounted display 120, depending on the wearing scenario. In other embodiments, the wearing position can be identified and set by executing computer vision algorithms. In some embodiments, the head-mounted display 120 can set the wearing position to one of the knuckles of the hand. For example, when a user wants to select a virtual object in a virtual reality scene using their index finger, the head-mounted display 120 will set the wearing position to the second knuckle of the user's index finger.
[0064] In step S220, it is determined whether the hand tracking algorithm has detected a hand model with finger skeleton data of the user's hand in the monitored scene. In some embodiments, the hand tracking algorithm can be any computer vision algorithm used to construct a hand model in a virtual reality scene. In some embodiments, the hand model can be a virtual hand skeleton having finger skeleton data of the user's hand corresponding to the hand in the virtual reality scene.
[0065] In some embodiments, the monitored scene is captured to generate recent scene images, and it is determined whether multiple joint data representing multiple hand joints are found. Then, in response to the discovery of multiple joint data, a hand model (i.e., a hand skeleton model with finger skeleton data of the user's hand) is generated based on the multiple joint data.
[0066] The following examples illustrate the capture of surveillance footage. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of a video surveillance scene (MF) according to some embodiments. For example... Figure 3 As shown, the head-mounted display 120 can capture the monitored scene MF, i.e., the space in front of the user, and the user can wear a wearable device 110. If the user's hand is within the field of view of the head-mounted display 120, the head-mounted display 120 can capture the hand to identify multiple joint data representing multiple hand joints. Conversely, if the user's hand is not within the field of view of the head-mounted display 120 (i.e., the user's hand is in a blind spot), the head-mounted display 120 cannot identify multiple joint data representing multiple hand joints.
[0067] In some embodiments, for recent scene images, an image recognition algorithm is executed to determine whether a hand is present in the monitored scene. Then, in response to the presence of a hand in the monitored scene, multiple joint data representing multiple hand joints are identified, and a hand model is generated based on the multiple joint data.
[0068] The following example illustrates the creation of a hand model. Please refer to further details. Figure 4 , Figure 4 This is a schematic diagram illustrating the generation of a hand model (HM) according to some embodiments. For example... Figure 4 As shown, the hand model HM is generated based on the hand HND. The hand model HM is a hand skeleton model with finger skeleton data of the user's hand, and includes multiple fingertips, a palm, a wrist, and joints Joint1 to Joint18.
[0069] In some embodiments, the above steps S210 to S220 are performed by the hand tracking module 122.
[0070] like Figure 2AAs shown, in step S230, in response to the detection of a hand model in the monitoring scene, the device position of the wearable device in the monitoring scene is identified based on the wearing position and finger skeleton data, and the device rotation amount of the wearable device in the monitoring scene is identified based on inertial data. In some embodiments, the coordinates of adjacent joints in three-axis space can be added and divided by 2 to generate the coordinates of the device position. In some embodiments, the device rotation amount can be identified based on the angular velocity at multiple detection time points of the inertial measurement unit 111.
[0071] In step S240, the direction of the indicator in the monitoring scene is calculated based on the device position and the amount of device rotation.
[0072] The following example illustrates how to set the starting indicator direction. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a schematic diagram illustrating the setting of the indicator direction PD according to some embodiments. For example... Figure 5 As shown, assuming the wearing position WP is set to the second joint of the user's index finger, then the wearing position WP in the hand model HM will be located between joints Joint3 and Joint4. In this case, the device position can be identified in the monitoring scene as the position between joints Joint3 and Joint4. For example, the coordinates of joints Joint3 and Joint4 in three-axis space can be added together and divided by 2 to obtain the coordinates of the device position.
[0073] The hand portion PT can be magnified and viewed from the side of the hand model HM. The head-mounted display 120 can generate a virtual device VD at the device location and set the initial indicator direction PD as the radial direction of the virtual device VD, wherein the virtual device VD has a ring-shaped shape at the wearing position WP, and the radial direction of the virtual device VD points behind the knuckle between joints Joint3 and Joint4.
[0074] It should be noted that, in addition to the above-mentioned method for setting the initial indicator direction PD, the initial indicator direction PD can also be set to a direction that is perpendicular to the radius of the virtual device VD and passes through the center point of the virtual device VD.
[0075] Furthermore, the starting indicator direction can be changed as the device position and the amount of device rotation change, and the changed starting indicator direction can be set as the indicator direction.
[0076] like Figure 2AAs shown, in step S250, a light beam is generated in the virtual reality scene according to the indicator direction and the device position. In this way, the starting point of the light beam is located at the device position in the virtual reality scene, and the direction of the light beam is the indicator direction. Therefore, the user can point to or select virtual objects in the virtual reality scene using the light beam. In some embodiments, the above steps S230 to S250 are executed by the device tracking module 123.
[0077] The following explains situations where no hand model was detected in the monitored scene. Please refer to further details. Figure 2B , Figure 2B This is a flowchart of further steps in a motion calculation method applied to mixed reality according to some embodiments. Figure 1 The action calculation system 100 shown can be used to execute Figure 2B Further steps of the action calculation method shown.
[0078] like Figure 2B As shown, firstly, in step S230', in response to the absence of a hand model in the monitored scene, the device position of the wearable device is identified based on inertial data and the latest available device position of the wearable device, and the amount of device rotation of the wearable device in the monitored scene is identified based on the inertial data. Furthermore, the latest available device position can be set by the user or identified based on the previously detected hand model.
[0079] In step S240', the indicator direction in the monitoring scene is calculated based on the device rotation amount and the corresponding inertial data and the latest available device position. In step S250', a light beam is generated in the virtual reality scene based on the indicator direction and the corresponding inertial data and the latest available device position.
[0080] The following section will explain how to update the hand model. Please refer to [link / reference needed]. Figure 2C , Figure 2C This is a flowchart illustrating further steps of a motion calculation method applied to mixed reality according to some embodiments. Figure 1 The action calculation system 100 shown can be used to execute Figure 2C Further steps of the action calculation method shown.
[0081] like Figure 2CAs shown, firstly, in step S260, after generating a light beam in the virtual reality scene, a hand tracking algorithm is performed on the monitoring scene for the hand to determine whether a new hand model is found in the monitoring scene. In some embodiments, the monitoring scene is captured to generate a recent scene image, and it is determined whether multiple joint data representing multiple hand joints are found. Then, in response to the discovery of multiple joint data, a new hand model is generated based on the multiple joint data.
[0082] In some embodiments, for recent scene images, an image recognition algorithm is executed to determine whether a hand exists in the monitored scene. Then, in response to the presence of a hand in the monitored scene, multiple joint data representing multiple hand joints are identified, and a new hand model is generated based on the multiple joint data.
[0083] In step S270, in response to the discovery of a new hand model of the user's hand in the monitoring scene, the device position in the monitoring scene is updated based on inertial data and the new hand model. In some embodiments, the detection frequency of the inertial data is higher than the detection frequency of the hand tracking algorithm. In some embodiments, the device position of the wearable device in the monitoring scene is updated by integrating inertial data between the time of generating the hand model and the time of generating the new hand model. In some embodiments, each hand model and each new hand model includes joint data representing multiple hand joints.
[0084] In some embodiments, the device position of the wearable device in the monitoring scenario is updated by integrating inertial data from the first hand model.
[0085] In step S280, the rotation amount of the wearable device in the monitoring scene is updated based on inertial data. In some embodiments, the device position of the wearable device in the monitoring scene is updated using integrated inertial data, and the device position is calibrated based on a new hand model.
[0086] In some embodiments, when hand tracking is successful, a new hand model can be used to calibrate multiple permutations generated from integrated inertial data to produce the device position of the wearable device in the monitoring scene. In some embodiments, when hand tracking is successful, a new hand model can be used and an Error-State Kalman Filter (ESKF) can be used to calibrate multiple permutations generated from integrated inertial data (as shown in the examples below). In some embodiments, at one of the detection times during hand tracking, if the positional difference between the device position generated from the inertial data and the device position generated by performing hand tracking is greater than a difference threshold, the ESKF can be used to combine the device position generated from the inertial data and the device position generated by performing hand tracking.
[0087] In step S290, the indicator direction in the monitoring scene is updated according to the device position and the updated device rotation amount, and a beam is generated in the virtual reality scene according to the indicator direction and the device position.
[0088] The following explains the situation where no new hand models were detected in the monitored scene. Please refer to further details. Figure 2D , Figure 2D This is a flowchart illustrating further steps of a motion calculation method applied to mixed reality according to some embodiments. Figure 1 The action calculation system 100 shown can be used to execute Figure 2D Further steps of the action calculation method shown.
[0089] like Figure 2D As shown, firstly, in step S270', in response to the absence of a new hand model in the monitoring scene, the device position of the wearable device is identified based on inertial data and the latest available device position of the wearable device, and the device rotation amount of the wearable device in the monitoring scene is identified based on inertial data.
[0090] In some embodiments, multiple permutations can also be generated using integrated inertial data to determine the device position of the wearable device in the monitoring scenario. In some embodiments, the device position of the wearable device in the monitoring scenario can be calculated by combining multiple permutations with the latest available device position.
[0091] In step S280', the indicator direction in the monitoring scene is calculated based on the device rotation amount and the corresponding inertial data and the latest available device position. In step S290', a light beam is generated in the virtual reality scene based on the indicator direction and the corresponding inertial data and the latest available device position.
[0092] The following example illustrates how the device position is generated. Please refer to further details. Figure 6 , Figure 6 This is a schematic diagram of the position values of an axis according to some embodiments. For example... Figure 6 As shown, the device position is indicated as a position value on an axis (e.g., the X-axis).
[0093] At the first detection time point t1 in time series TS1, assuming the hand model is detected and generated, the detection frequency of the inertial measurement unit 111 exceeds 200 Hz, and the detection frequency of the hand tracking algorithm is 30 Hz, the device position is initialized based on the hand model. Between the first detection time point t1 and the second detection time point t2, based on the device position IP at the first detection time point t1, multiple permutations can be generated by integrating the inertial data into a normal state, and multiple device positions IMUP can be generated based on these permutations. The normal state and the error state can be represented by the following Equations 1 and 2, respectively:
[0094]
[0095]
[0096] As shown in Equation 1 above, X represents the standard state, p represents the position of the wearable device, v represents the velocity of the wearable device, q represents the rotation of the wearable device (i.e., the quaternion), ab represents the accelerometer deviation of the wearable device, wb represents the gyroscope deviation of the wearable device, and g represents the gravity vector of the wearable device. In other words, the standard state consists of the position, velocity, rotation, accelerometer deviation, gyroscope deviation, and gravity vector of the wearable device. These parameters are estimated from the measurement data of the inertial sensor.
[0097] It should be noted that the standard state does not account for noise and other model defects. Therefore, errors will accumulate, and these errors will be accumulated in the error state through ESKF and Gaussian estimation.
[0098] As shown in Equation 2 above, δX represents the error state, δp represents the error in the wearable device's position, δv represents the error in the wearable device's velocity, δθ represents the error in the wearable device's angle (i.e., angular velocity), δab represents the error in the wearable device's accelerometer deviation, δwb represents the error in the wearable device's gyroscope deviation, and δg represents the error in the wearable device's gravity vector. In other words, the error state is composed of the errors in the wearable device's position, velocity, rotation, accelerometer deviation, gyroscope deviation, and gravity vector.
[0099] At the second detection time point t2 in the time series TS1, a permutation can be generated by integrating the inertial data. At this time, the device position IMUP can be generated based on this permutation. Furthermore, another device position OP1 can be generated based on the new hand model. Information from other sensors besides the inertial measurement unit 111 (i.e., the new hand model) constitutes the true state. The position error (i.e., position difference) between the device position IMUP and the other device position OP1 can be used to update and calibrate the ESKF and helps reset the error state. The true state can be represented by the following Equation 3:
[0100]
[0101] As shown in Equation 3 above, Xt represents the real state, while Represents operations of a generic composition.
[0102] At the third detection time point t3 in time series TS1, the positional difference between device position PT1 and another device position PT2 is input into ESKF to re-estimate the optimized device position (i.e., combining device position PT1 and another device position PT2) as the device position at the third detection time point t3. In another example, device position PT1 can also be directly calibrated to another device position PT2.
[0103] At the fourth detection time point t4, if the hand model is not detected and generated, the device position IMUP can be updated in time series TS2 using integrated inertial data and standard conditions. At the fifth detection time point t5 in time series TS3, if the hand model is again not detected and generated, a displacement can be generated using integrated inertial data. At this time, the device position IMUP can be calculated based on this displacement. In addition, another device position OP1 can also be calculated based on the new hand model. The ESKF can be updated again based on the position difference between the device position of the inertial measurement unit 111 and the other device position OP1 of the hand model.
[0104] The following example illustrates the generation of light beams in a virtual reality scene. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of generating a beam RY in a virtual reality scene (VRF) according to some embodiments. For example... Figure 7As shown, the head-mounted display 120 displays a virtual reality scene VRF and a virtual hand VH (also generated by a hand tracking algorithm), and the virtual device VD is located at the device position. The starting point of the beam RY in the virtual reality scene VRF is located at the device position, and the direction of the beam RY is the direction of the pointer. The user can point to virtual objects (e.g., a shortcut key in the virtual reality scene VRF) in the virtual reality scene VRF using the beam RY. In addition, the user can interact with virtual objects in the virtual reality scene VRF using other fingers not wearing the wearable device 110.
[0105] In summary, the motion computing system for mixed reality disclosed in this document combines wearable devices and head-mounted displays to determine the device position and rotation amount of the wearable device using hand models and inertial data. This method significantly improves the accuracy of interactive input by eliminating drift errors of the inertial measurement unit or blind spot errors of the head-mounted display. Furthermore, this method frees up the user's hands to interact with the virtual reality scene using their hands.
[0106] While this disclosure has been described in considerable detail with reference to embodiments, other embodiments may also be practiced. Therefore, the spirit and scope of the claims of this disclosure should not be limited to the embodiments described herein.
[0107] Those skilled in the art will understand that various modifications and equivalent changes can be made to the structure of this disclosure without departing from its scope or spirit. In summary, all modifications and equivalent changes made to this disclosure within the scope of the following claims are within the scope of this disclosure.
Claims
1. A motion computing system for mixed reality, comprising: Include: A wearable device includes an inertial measurement unit (IMU) for detecting inertial data; and A head-mounted display is coupled to the wearable device, wherein the head-mounted display is used to perform the following steps: The head-mounted display is used to set a wearing position of the wearable device, wherein the wearing position represents a position on a user's hand where the wearable device is worn. Determine whether a hand tracking algorithm has discovered a hand model containing finger skeleton data of the user's hand in a surveillance scene; In response to the detection of the hand model in the monitoring scene, the wearable device's position in the monitoring scene is identified based on the wearing position set by the head-mounted display and the finger skeleton data, and the amount of rotation of the wearable device in the monitoring scene is identified based on the inertial data. Calculate the direction of an indicator in the monitored scene based on the device's location and rotation amount; and A beam of light is generated in a virtual reality scene based on the direction of the indicator and the position of the device.
2. The motion calculation system applied to mixed reality as described in claim 1, characterized in that, This head-mounted display is further used to perform the following steps: In response to the absence of the hand model in the monitored scene, the device position of the wearable device is identified based on the inertial data and the latest device position of the wearable device, and the amount of rotation of the wearable device in the monitored scene is identified based on the inertial data. Based on the device's rotation and the corresponding inertial data and the latest available device position, the orientation of the indicator in the monitoring scene is calculated; and The beam is generated in the virtual reality scene based on the direction of the indicator and the corresponding inertial data and the location of the latest available device.
3. The motion calculation system applied to mixed reality as described in claim 1, characterized in that, This head-mounted display is further used to perform the following steps: After the beam of light is generated in the virtual reality scene, the hand tracking algorithm is executed on the monitoring scene for the hand to determine whether a new hand model is found in the monitoring scene. In response to the discovery of a new hand model of the user's hand in the monitoring scene, the position of the device in the monitoring scene is updated based on the inertial data and the new hand model; The amount of rotation of the wearable device in the monitored scene is updated based on the inertial data. as well as The direction of the indicator in the monitoring scene is updated based on the position of the device and the rotation amount of an updating device, and the beam is generated in the virtual reality scene based on the direction of the indicator and the position of the device.
4. The motion calculation system applied to mixed reality as described in claim 3, characterized in that, For this hand, the steps of executing the hand tracking algorithm on the monitored scene to determine whether the new hand model is detected in the monitored scene include: Capture the surveillance scene to generate a recent image and determine if data representing multiple joints of the hand is found; and In response to the discovery of the multiple joint data, the new hand model is generated based on the multiple joint data.
5. The motion calculation system applied to mixed reality as described in claim 4, characterized in that, For this hand, the steps of executing the hand tracking algorithm on the monitored scene to determine whether the new hand model is detected in the monitored scene further include: An image recognition algorithm is applied to the recent scene image to determine whether the hand exists in the monitored scene; as well as In response to the presence of the hand in the monitoring scene, the joint data representing the multiple joints of the hand are identified, and the new hand model is generated based on the joint data.
6. The motion calculation system applied to mixed reality as described in claim 3, characterized in that, The detection frequency of the inertial data is higher than the detection frequency of the hand tracking algorithm, wherein the head-mounted display is further used to perform the following steps: Between the time the hand model is generated and the time the new hand model is generated, the position of the wearable device in the monitoring scene is updated by integrating the inertial data.
7. The motion calculation system applied to mixed reality as described in claim 3, characterized in that, The steps for updating the device's position in the monitoring scene based on the inertial data and the new hand model include: By integrating the inertial data, the position of the wearable device in the monitoring scene is updated, and the position of the device is calibrated according to the new hand model.
8. The motion calculation system applied to mixed reality as described in claim 3, characterized in that, This head-mounted display is further used to perform the following steps: In response to the absence of the new hand model in the monitoring scene, the device position of the wearable device is identified based on the inertial data and the latest device position of the wearable device, and the amount of rotation of the wearable device in the monitoring scene is identified based on the inertial data. Based on the device's rotation and the corresponding inertial data and the latest available device position, the orientation of the indicator in the monitoring scene is calculated; and The beam is generated in the virtual reality scene based on the direction of the indicator and the corresponding inertial data and the location of the latest available device.
9. The motion calculation system applied to mixed reality as described in claim 3, characterized in that, The hand model and the new hand model each contain joint data representing multiple joints of the hand.
10. The motion calculation system applied to mixed reality as described in claim 1, characterized in that, The steps for setting the wearing position of the wearable device include: The wearing position of the wearable device is set to one of the multiple phalanges of the hand.
11. A motion calculation method applied to mixed reality, characterized in that, Include: Detect inertial data and set a wearing position of a wearable device via a head-mounted display, wherein the wearing position represents a position where the wearable device is worn on a user's hand. Determine whether a hand tracking algorithm has discovered a hand model containing finger skeleton data of the user's hand in a surveillance scene; In response to the detection of the hand model in the monitoring scene, the wearable device's position in the monitoring scene is identified based on the wearing position set by the head-mounted display and the finger skeleton data, and the amount of rotation of the wearable device in the monitoring scene is identified based on the inertial data. Calculate the direction of an indicator in the monitored scene based on the device's location and rotation amount; and A beam of light is generated in a virtual reality scene based on the direction of the indicator and the position of the device.