A tracking system for simulating physical movements

Abstract

According to a first aspect of the present invention, there is provided a tracking system for simulating physical motion in a computing environment, the system comprising: an optical sensor configured to signal that a movement of the optical sensor has occurred via one or more processors detecting that successive captured frames are different, the one or more processors measuring the movement by referencing a set point across successive frames; a plurality of inertial measurement units each controlled by the one or more processors measuring rotation data; and a hub in communication with the inertial measurement units and the optical sensor, the hub receiving the rotation data from the plurality of inertial measurement units via one or more wireless communication channels, the hub being controlled by the one or more processors to output a data stream that combines the rotation data obtained while tracking the physical motion and the measured movement data obtained while tracking the physical motion to enable simulation of physical motion in the computing environment, wherein movement of a body part in the computing environment is inferred from the measured rotation data and the measured rotation data of other body parts concatenated therewith, and a position of the body in the computing environment is inferred from the measured movement data.

Description

[Technical field]

[0001] The present disclosure relates to a tracking system for simulating body motion within a computing environment. [Background technology]

[0002] Virtual reality (VR) relates to applications involving immersive, highly visual, computer-simulated environments that typically simulate a user's physical presence in locations in the real or imaginary world.

[0003] In VR systems, the problem is to track the user's movement and map it to the computing environment. Commercial and industrial full-body tracking suits based on "outside-in" tracking use base station sensors. Entities in the VR system rely on these base station sensors to estimate their position and / or orientation. They suffer from occlusion and require a lot of space.

[0004] Alternatively, in "inside-out" tracking, a camera is placed on the tracked device to look outward and determine its position in the environment. Known headsets that operate without markers have multiple cameras facing in different directions to get a view of their surroundings. These headsets require the controllers to be visible in the headset's cameras in order to track hand movements. Therefore, they also suffer from occlusion. Summary of the Invention

[0005] The object of the present invention is to provide a solution that addresses the above mentioned drawbacks.

[0006] According to a first aspect of the present invention, there is provided a tracking system for simulating a body motion in a computing environment, the system comprising: one or more processors; an optical sensor configured to signal that a movement of the optical sensor has occurred via the one or more processors detecting that successive captured frames are different, the one or more processors measuring the movement by referencing a set point across the successive frames; a plurality of inertial measurement units each controlled by the one or more processors to measure rotation data; and a hub in communication with the inertial measurement units and the optical sensor, the hub receiving the rotation data from the plurality of inertial measurement units via one or more wireless communication channels, the hub controlled by the one or more processors to output a data stream that combines the rotation data acquired while tracking the body motion and the measured movement data acquired while tracking the body motion to enable simulation of the body motion in the computing environment; and a hub configured to combine the rotation data acquired while tracking the body motion with the measured movement data acquired during tracking the body motion to output a data stream that enables simulation of the body motion in the computing environment, wherein a movement of a body part in the computing environment is inferred from the measured rotation data thereof and the measured rotation data of other body parts combined, and a position of the body in the computing environment is inferred from the measured movement data.

[0007] A quaternion representation of the rotational data may be used to estimate the movement of the body part in the computing environment.

[0008] The estimation of the movement of the body part in the computing environment may be based on one or more forward kinematics algorithms.

[0009] The optical sensor is integral with the hub, and movement of the optical sensor can result from movement of the body to which the hub is attached.

[0010] Measurement of movement begins with the optical sensor capturing an initial frame, which can provide a starting point for tracking the body motion.

[0011] The hub may be configured to pair the inertial measurement units by proximity detection of emitted radio signals of the inertial measurement units.

[0012] The hub may be further configured to determine a body part assignment to which each of the plurality of inertial measurement units is attached by analysis of the output rotational data and strength of radio signals emitted to the hub.

[0013] The hub may be further configured to store, during pairing, a unique identifier of each of the plurality of inertial measurement units for each of the assigned body parts.

[0014] The one or more processors may be configured to perform a calibration using data regarding dimensions of body parts to which the multiple inertial measurement units are attached following pairing and before commencing tracking of the body movements.

[0015] The one or more processors may analyze images containing the body parts to derive their dimensions.

[0016] The dimensions may be derived using one or more of a machine learning algorithm and a skeletal structure model.

[0017] The image may be captured by the optical sensor.

[0018] The derivation of the dimensions is performed in conjunction with the multiple inertial measurement units attached to each body part, and the strength of the radio signals emitted by the multiple inertial measurement units can be compared with measurement data based on images of the corresponding body parts.

[0019] A base pose to apply before starting body motion capture may be pre-determined in the computing environment.

[0020] The measured rotational data is used to derive an offset from the base pose, which can be used to construct a current pose.

[0021] The hub may be configured to enable extraction of the measured rotational data from one or more of the multiple inertial measurement units and / or the measured movement data from the optical sensor acquired while tracking the body motion for recording as a macro.

[0022] The body position obtained from the measured movement data may be based on visual localization and mapping.

[0023] The optical sensor may be any one or more of a stereo camera, a LIDAR, and an optical sonar sensor.

[0024] At least one of the one or more processors may be hosted within a computer platform.

[0025] The estimation of the movement of the body parts in the computing environment and the estimation of the position of the body in the computing environment may be performed in the computer platform to generate the data stream in the computer platform.

[0026] According to a second aspect of the present invention, there is provided a method for simulating a physical motion in a computing environment, comprising: measuring a movement of the optical sensor obtained during tracking of the physical motion by referencing different set points across successive frames captured by the optical sensor, combining in a hub measured rotation data obtained during tracking of the physical motion with measured movement data, the measured rotation data being received at the hub from a plurality of inertial measurement units via one or more wireless communication channels, and outputting a data stream enabling the simulation of the physical motion in the computing environment, wherein a movement of a body part in the computing environment is estimated from its measured rotation data and the measured rotation data of other body parts in conjunction therewith, and a position of the body in the computing environment is estimated from the measured movement data. [Brief description of the drawings]

[0027] Exemplary embodiments of the present invention are herein described, by way of example only, with reference to the accompanying drawings.

[0028] [Figure 1] FIG. 1 is a diagram illustrating the deployment of a tracking system on the body according to one embodiment of the present invention. [Diagram 2] 2 is a flow chart for pairing an inertial measurement unit with a hub of the tracking system of FIG. [Diagram 3] FIG. 3 is a diagram illustrating an internal kinematic humanoid structure used to drive the animation of an avatar representation of the body of FIG. 1 in a computing environment. [Figure 4] FIG. 4 is a flow chart of data acquisition by the tracking system of FIG. 1 during body motion capture and transmission to a receiving computer platform. [Diagram 5]FIG. 5 is a flow chart for sending data from the application layer to the tracking system of FIG. [Figure 6] FIG. 6 shows a kinematic model of the arm. [Figure 7] FIG. 7 is a flow chart for simulating physical motion in a computing environment used by the tracking system of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] In the following description, various embodiments are described with reference to the drawings, in which like reference numerals generally refer to the same parts throughout the various views.

[0030] This application is in the field of virtual reality, augmented reality, or other forms of visual and immersive computer-simulated environments provided to a user. Virtual reality, or VR, refers to a computer-simulated environment in which a user can interact using hardware that can map or simulate the user's actions onto the computer-simulated environment. Augmented reality, or AR, refers to the overlaying of virtual images onto a real-world environment, i.e., combining the real-world environment with virtual images using a head-mounted device (HMD) that allows the user to see the real world as it is.

[0031] To map a user's movements to a computer-simulated environment, there are known approaches that use one or more sensors, such as lighthouses, configured to monitor photosensors present in an HMD worn by a user to determine the user's position relative to the lighthouse. The HMD also has sensors, such as cameras, that require a body part (such as a hand) to be in frame in order to determine the hand's position relative to the HMD. These known approaches can suffer from occlusions that occur in the sensors used to determine the user's position and the position of the user's limbs.

[0032] The optical sensor, inertial measurement unit, and hub hardware components used herein to capture the user's motion and map it in the computer-simulated environment attempt to address such occlusion issues by avoiding the need to calibrate against pre-determined landmarks when initializing the user's avatar representation in the computing environment. To achieve this, the first frame captured by the optical sensor already provides a starting point for beginning the user's motion capture in the real-world environment. That is, the starting position is automatically obtained when the optical sensor is initialized and detects that the background in the frame is changing. Thus, no pre-determined landmarks are required.

[0033] A frame is one of many still images that constitute a full body motion capture. As a user moves in a real-world environment, the position of the user's avatar in the computing environment depends on the successive frames captured by the optical sensor. When successive frames are detected to be different, it indicates that a user movement has occurred. The amount of movement is then measured by referencing or comparing set points between successive frames. Set points are landmarks that are automatically determined in each frame and are not predefined. Thus, the set points may be different in each successive frame. The difference in spatial distance between such set points in the subsequent and preceding frames is used to calculate the shift of the user's position during the tracked movement. In one approach, successive frames are considered to be different if they contain enough distinct features, resulting in different set points in each of these successive frames. That is, in addition to being used to measure the change in the user's position, the set points are used as a measure of tolerance to determine whether successive frames are different. On the other hand, if the set points in successive frames are the same, it is determined that the user's position has not changed, and such successive frames are not considered to be different frames. When determining whether successive frames are different, a filter is used to ignore phenomena such as noise artifacts (such as a person walking past the frame).

[0034] A mapping of body part motion, such as limb movement and rotation, is then obtained by processing rotation data measured by multiple inertial measurement units attached to the body parts, respectively. For example, each limb may have two inertial measurement units, one attached to the upper limb and one attached to the lower limb. The inertial measurement units may measure acceleration and / or angular velocity and / or magnetic fields along x, y, z coordinates.

[0035] The hub acts as a central module that consolidates the rotational data from each of the multiple inertial measurement units. Performing data combining in the hub is advantageous because it allows for use across different operating systems, such as those used in smartphones, game consoles, Linux, and Macs.

[0036] The hub receives this data from the multiple inertial measurement units over one or more wireless communication channels, i.e., each of the multiple inertial measurement units can wirelessly communicate with the hub over a dedicated frequency bandwidth. Such communication over wireless channels is distinct from determining the position of the hand controller by the headset camera being visible within its field of view, which does not use a communication channel. Thus, the present approach does not suffer from occlusion, which occurs when the line of sight between the headset camera and the hand controller is broken.

[0037] The hub also combines the movement data measured from successive frames captured by the optical sensors with the concatenated rotation data measured by the multiple inertial measurement units. This data combination facilitates the output of a data stream that enables the simulation of user motion in the computing environment. User motion refers to all the poses the body performs when moving from one position to another, including the rotations and translations of all body parts and the changes in the coordinate position of the body. The combination of the rotation data, acceleration data, and measured movement data can use a fusion algorithm that combines position or motion vectors from different coordinate systems to give a global vector.

[0038] In one implementation, the inertial measurement units can measure both linear and rotational acceleration data of the body parts to which they are attached, but the hub uses only the measured rotation data obtained during body motion capture to estimate the movement of the body part. The hub does this by evaluating the measured rotation data of the body part against the measured rotation data of other connected body parts. The use of rotation data to estimate the movement of the body part is based on the principle that when a body part moves, the joints of that part and the joints of other connected body parts rotate with strain and boundaries of joint rotation. For example, referring to FIG. 6, when the elbow joint 604 is moved, this causes a movement in the connecting shoulder joint 602 and a movement in the connecting wrist joint 606. Inertial measurement units located proximate each of these joints 602, 604, 606 measure the rotation data (θ1, θ2, θ3), (θ4, θ5), and (θ6, θ7), respectively. The motion of the elbow joint 604 can then be simulated by the measured rotation data output by these inertial measurement units. For example, forward kinematic algorithms can be used to estimate the motion of the body parts in the computing environment, where the limb positions are obtained after the limb rotations are measured. For example, there may be an algorithm for each body part that establishes a set of transformations from one joint frame to the next. By combining all these transformations from frame 0 to frame n and defining the dimensions of each link between adjacent joint frames, an entire transformation matrix can be obtained, characterizing the relative movement allowed at each joint.

[0039] The simulation of the user's movements is then completed by fixing the body position in the computing environment, i.e., by determining the coordinates of the user's avatar in the computing environment, which is determined by the movement data measured from successive frames captured by the optical sensor.

[0040] One approach uses quaternion representations of rotation data to estimate the motion of body parts in a computing environment. Quaternions are rotation data derived from complex numbers and are an alternative way to describe orientation or rotation in three-dimensional space. Quaternion matrices are represented by 4x1 data values. They uniquely describe any 3D rotation around any axis and do not suffer from gimbal lock associated with Euler rotation matrices. Quaternions provide the information needed to rotate a vector with only four numbers, instead of the 3x3 or 4x4 matrices required for Euler rotations.

[0041] The operation of the optical sensor, the inertial measurement unit, and the hub are described in more detail below in conjunction with FIGS.

[0042] 1 shows a body 108 on which a tracking system 100 is deployed, the tracking system 100 comprising a hub 102 with multiple inertial measurement units 104a, 104b, 104c, and 104d and an integrated optical sensor 106. In one approach, the tracking system 100 may be populated with between 9 and 17 inertial measurement units, although for simplicity only four are shown.

[0043] The tracking system 100 further includes one or more processors, not shown. The term "processor" may refer to one or more units for processing, including an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device (PLD), a microcontroller, a field programmable gate array (FPGA), a microprocessor, a digital signal processor (DSP), or other suitable components. The processor may be configured using machine-readable instructions stored on a memory. The processor may be centralized or distributed, such as distributed on various components forming part of the tracking system 100 or in communication with the tracking system 100. The processor may be located in one or more of the peripheral devices, which may include a user interface device, an HMD, a personal computer, and the like. Thus, the one or more processors may be distributed across any of the inertial measurement units 104a, 104b, 104c, and 104d, the hub 102, the optical sensor 106, and a computing platform that receives the output data stream from the hub 102 and maps the body motion to the computing environment. In implementations where one of the processors is hosted on such a computing platform, the computing platform is one of the components of the tracking system 100. The estimation of the movement of the body parts in the computing environment and the estimation of the position of the body in the computing environment may be performed on the computing platform to generate a data stream on the computing platform.

[0044] Each of the multiple inertial measurement units (IMUs) 104a, 104b, 104c, and 104d is attached to a body part to measure data collected along each of the roll, yaw, and pitch axes. Each of the inertial measurement units 104a, 104b, 104c, and 104d uses one or more accelerometers to measure linear acceleration along one or more directions, one or more gyroscopes to measure angular motion about one or more axes, and utilizes a magnetometer to provide an orientation reference. Since acceleration is proportional to an external force, the accelerometer readings can reflect both the intensity and frequency of the body part's movement. By integrating the accelerometer readings with respect to time, velocity and displacement information of the body part can be derived. Each of the multiple inertial measurement units 104a, 104b, 104c, and 104d may also have a battery power source, a status LED, and a vibration motor to provide tactile feedback.

[0045] As described above, the inertial measurement units 104a, 104b, 104c, and 104d can measure both linear and rotational acceleration data of the body parts to which they are attached, but the rotational data measured by each of the inertial measurement units 104a, 104b, 104c, and 104d can simply be used to obtain an estimate of the position and orientation of each body part during motion capture using a forward kinematics algorithm. However, errors in the measured data will result in drift in these estimates, which can be corrected by fusing IMU rates with other data measurements. In this application, one possible implementation corrects the positions of the inertial measurement units derived from their measured rotational data against the visual data when the inertial measurement units are captured by the optical sensor 106. It should be noted that the visual capture here is not a determining factor of the positions of the inertial measurement units. Other data measurements include corrections based on including linear acceleration data measured by multiple inertial measurement units 104a, 104b, 104c, and 104d. It will also be appreciated that calibration to body part dimensions improves the accuracy of the body part simulation, as described in more detail below with respect to FIG.

[0046] In addition to acting as a central module that integrates the output data from its optical sensor 106 and the multiple inertial measurement units 104a, 104b, 104c, and 104d, the hub 102 also serves as an access point for the multiple inertial measurement units 104a, 104b, 104c, and 104d. This allows wireless communication between the multiple inertial measurement units 104a, 104b, 104c, and 104d and the hub 102, expanding the number of inertial measurement units that can be used for motion capture tracking. The more inertial measurement units used, the more detailed the animation will be.

[0047] Types of sensors that may be used for the optical sensor 106 include stereo cameras, LIDAR, and optical sonar sensors, and the optical sensor 106 may be configured to use one or more such sensors. The optical sensor 106 may transmit frames captured by the optical sensor 106 over a wired connection to the processor of the hub 102, or over a wireless communication channel in implementations in which the optical sensor 106 communicates wirelessly with the processor of the hub 102.

[0048] FIG. 2 illustrates a flow chart 200 for pairing multiple inertial measurement units 104a, 104b, 104c, and 104d with the hub 102.

[0049] Prior to deployment, the tracking system 100 is operated in a pairing mode in step 202 to pair multiple inertial measurement units 104a, 104b, 104c, and 104d with the hub 102 and enable them to function as a single entity.

[0050] The hub 102 can be implemented using a microcontroller based on the ESP32 chipset, which supports wireless communication (e.g., wireless communication via WiFi and Bluetooth) and can be configured to pair with the inertial measurement units 104a, 104b, 104c, and 104d through proximity detection of wireless signals 110 emitted by the inertial measurement units 104a, 104b, 104c, and 104d in step 204. Wireless Received Signal Strength Indicators (RSSIs) are accurate to 1m, which allows the hub 102 to pair with the nearest inertial measurement units 104a, 104b, 104c, and 104d as they have the strongest RSSI. Thus, if there are other hubs in the vicinity that are paired with the respective inertial measurement units, the hub 102 will ignore the inertial measurement units paired with the other hubs even if they are detected. In one approach, the pairing is performed without the inertial measurement units being attached to the body 108, to allow the hub 102 to recognize the inertial measurement units 104a, 104b, 104c, and 104d.

[0051] Following this pairing, the hub 102 can be used to estimate the body parts to which each of the multiple inertial measurement units 104a, 104b, 104c, and 104d is attached. This estimation is facilitated by the actuations performed by these body parts (such as, but not limited to, limbs). For example, the user may be asked to assume a given starting position (e.g., arms folded) and then be asked to assume a second position (e.g., arms raised), whereby it may be specified how the limbs should be actuated. A range of rotational data for each limb in the process of moving to the second position is predicted, whereby the hub 102 can then perform limb assignment from detecting which of the multiple inertial measurement units measured the corresponding rotational data.

[0052] In step 206, the body part to which each of the multiple inertial measurement units 104a, 104b, 104c, and 104d is attached is assigned through analysis of the rotation data output by each of the multiple inertial measurement units 104a, 104b, 104c, and 104d during this operation and the strength of the wireless signal emitted to the hub 102. The assignment can be performed simultaneously by estimating the concatenation of the measured rotation data resulting from the shift of the end points of the limb, for example, the shift of the wrist or foot. Also, by attaching the hub 102 near the center of the body 108, the accuracy of the body part assignment by this estimation is improved. The hub 102 then stores a unique identifier (such as a MAC address) of each of the multiple inertial measurement units 104a, 104b, 104c, and 104d with respect to the respective assigned body part. Each MAC address facilitates the exchange of data and commands over the wireless channel that each of the multiple inertial measurement units uses to communicate with the hub 102. Tracking of each particular body part is then obtained by referencing the inertial measurement units with their corresponding unique identifiers, thereby enabling the hub 102, in step 208, to send and receive commands and data to the multiple inertial measurement units 104a, 104b, 104c and 104d.

[0053] Such inferential body part assignment makes pairing seamless. This allows for replacement of any of the multiple inertial measurement units 104a, 104b, 104c, and 104d without the need to tie any of the multiple inertial measurement units to a specific body part. The hub 102 recognizes that a change has occurred from the original assignment configuration, which is missing the MAC address of the removed inertial measurement unit and the MAC address of the new inertial measurement unit. The hub 102 then stores the MAC address of the replacement inertial measurement unit, and the assignment to each body part is automatic, since the MAC addresses of the other inertial measurement units are unchanged. Also, the user does not need to identify which limb each of the multiple inertial measurement units 104a, 104b, 104c, and 104d is attached to. The hub 102 may also have a battery power source, a status LED, and a vibration motor to provide haptic feedback.

[0054] 3, the assignment of body parts to inertial measurement units populates nodes 302 of an internal kinematic humanoid structure 300 used by the hub 102 to drive animation of an avatar representation of the body 108 in a computing environment. The internal kinematic humanoid structure 300 that models the body 108 is further refined by using data on the real-world dimensions of the body parts to which the multiple inertial measurement units 104a, 104b, 104c, and 104d are attached. This calibration of the internal kinematic humanoid structure 300 is performed before tracking the motion of the body 108 begins to calculate dimensions, as described in more detail below. Such calibration factors the influence of the data on the dimensions of the body parts when combining output data from the multiple inertial measurement units 104a, 104b, 104c, and 104d with measured movement data obtained from the output of the optical sensor 106 when tracking the motion of the body 108.

[0055] An aggregate distance 304 between adjacent nodes 302 can be derived from measurements of body parts wearing multiple inertial measurement units (see 104a, 104b, 104c and 104d in FIG. 1). The example shown in FIG. 3 shows an aggregate length 304 of the left upper arm between a left shoulder node 302 and a left elbow node 302. Images of the various body parts can be used to derive those measurements, for example, using one or more machine learning algorithms or through reference to a skeletal structure model retrieved from a library.

[0056] In one approach, the optical sensor 106 of the hub 102 can be used to capture images of the body part, and the hub 102 can run machine learning algorithms or perform matching against a skeletal structure model. In this approach, the hub 102 is not attached to the body 108, but can be turned to face the body 108 to capture images required for skeletal tracking of the body part, providing, for example, different limb lengths. Also, multiple inertial measurement units 104a, 104b, 104c, and 104d can be attached during image capture by the optical sensor 106, and measurements of RSSI data can be cross-referenced with visual measurement data for accuracy. This allows the length derivation algorithm to also derive the location of the multiple inertial measurement units 104a, 104b, 104c, and 104d on the body 108 from measurements of RSSI data. That is, the dimensions are derived in conjunction with multiple inertial measurement units 104a, 104b, 104c, and 104d attached to respective body parts, and the strength of the wireless signals 110 emitted by the inertial measurement units is matched with measurement data based on images of the corresponding body parts. In another approach, the hub 102 can derive the body part dimensions from images taken by a separate camera, or receive these dimensions from another source using a different skeletal tracking algorithm.

[0057] Once the hub 102 has been paired with multiple inertial measurement units 104a, 104b, 104c and 104d, the assignment of inertial measurement units to body parts has been stored, and the internal kinematic humanoid structure 300 has been calibrated, the tracking system 100 can be used to track the motion of the body 108.

[0058] FIG. 4 shows a flow chart 400 of data acquisition by the tracking system 100 during motion capture of the body 108 and transmission of the data to a receiving computer platform.

[0059] In step 402, a home pose for application (typically a T-pose) is predefined in the computing environment. The body 108 is requested to copy this home pose before motion capture of the body 108 can begin. The home pose serves to zero the internal kinematic humanoid structure 300, thereby preparing it to be driven by the motion of the body 108. The zero pose is the pose when the quaternion matrix of each node 302 of the internal kinematic humanoid structure 300 is identical. The sampling rate of each of the multiple inertial measurement units 104a, 104b, 104c, and 104d is then specified, for example between 90 and 200 Hz. This sampled data can be used to derive an offset from the home pose, which can be used to construct the current pose. The sampled data includes rotation data and acceleration data measured by each of the multiple inertial measurement units 104a, 104b, 104c, and 104d.

[0060] In step 404, the multiple inertial measurement units 104a, 104b, 104c, and 104d transmit sampled data to the hub 102 using a wireless data communication protocol such as WiFi or Bluetooth. In steps 406 and 408, the hub 102 integrates the rotational and acceleration data from the multiple inertial measurement units 104a, 104b, 104c, and 104d with the measured movement obtained from the optical sensor 106. When the hub 102 is attached to the body 108, detection of differences in successive frames captured by the optical sensor 106 indicates that the optical sensor 106 has moved to a new position from the body 108. Data measuring the degree of movement resulting in this change is obtained from referencing a set point over successive frames according to visual simultaneous localisation and mapping techniques. Referring to step 412, this movement data is converted into a corresponding shift in the position of the internal kinematic humanoid structure 300. Meanwhile, the measured rotation and acceleration data of the body parts are converted into rotations and translations of the corresponding segments of the internal kinematic humanoid structure 300. As mentioned above, the first frame captured by the optical sensor 106 provides a starting point for starting the motion capture of the body 108 in the real-world environment, i.e., the starting position is automatically obtained upon initialization of the optical sensor 106.

[0061] Returning to step 410, the hub 102 transmits rotational and acceleration data from the multiple inertial measurement units 104a, 104b, 104c and 104d, along with measured movement data from the optical sensor 106, to a computing platform hosting the computing environment of the internal kinematic humanoid structure 300.

[0062] The integrated data in the hub 102 will be combined before transmission as a data stream that drives the internal kinematic humanoid structure 300 to simulate the motion of the body 108 in the computing environment when the hub 102 is operated in the "integrated" mode. This allows the output of the multiple inertial measurement units 104a, 104b, 104c, and 104d and the output from the optical sensor 106 to be fused, thereby allowing the output data from one of the multiple inertial measurement units to affect the output data from another inertial measurement unit. Next, step 412 is performed, in which the platform derives the internal kinematic humanoid structure 300 from the combined rotational and acceleration data and the measured translation data, both obtained while tracking the motion of the body 108. In step 414, the internal kinematic humanoid structure 300 is then transmitted to the application layer for use in the virtual reality application.

[0063] Alternatively, when operated in "developer" mode, the hub 102 is configured to enable extraction of measured rotation and acceleration data from one or more of the multiple inertial measurement units 104a, 104b, 104c, and 104d, and / or movement data from the optical sensor 106, obtained while tracking the motion of the body 108, for recording as a macro. Step 416 is then performed, where one or more of the outputs of the multiple inertial measurement units 104a, 104b, 104c, and 104d can be individually extracted and sent to an application layer for use in a virtual reality application. A recorded macro can, for example, describe controlling a volume knob or describe a vertical hand raise.

[0064] FIG. 5 shows a flow chart for sending data from the application layer to the tracking system 100.

[0065] In step 502, the application sends commands to the hub 102. Example commands include putting the tracking system 100 into pairing mode (see FIG. 2) or having the tracking system 100 track body movements after the hub 102 has been calibrated.

[0066] In step 504, the hub 102 receives the command and relays the command to one or more of the plurality of inertial measurement units 104a, 104b, 104c, and 104d using a wireless data communication protocol such as WiFi or Bluetooth. Each of the plurality of inertial measurement units 104a, 104b, 104c, and 104d receives the command in step 506 and acts accordingly. Exemplary commands are briefly described in steps 508, 510, 512, 514, 516, 518, 520, and 522.

[0067] Step 508 involves commands to operate vibration motors in the hub 102 and the multiple inertial measurement units 104a, 104b, 104c, and 104d. These commands enable haptic feedback in response to scenarios occurring in the computing environment.

[0068] Step 510 occurs when the hub 102 is paired with multiple inertial measurement units 104a, 104b, 104c, and 104d, as described with respect to FIG.

[0069] Step 512 is to restart, shut down, or put into a shutdown mode the multiple inertial measurement units 104a, 104b, 104c, and 104d.

[0070] Step 514 allows the user to define the sampling rate, as described with respect to FIG.

[0071] Step 516 allows for configuring the power supply.

[0072] Steps 518 and 520 allow for the calibration of the multiple inertial measurement units 104a, 104b, 104c, and 104d to the dimensions of the body part, as described with respect to FIG.

[0073] Step 522 enables the setting of status LEDs on the hub 102 and the multiple inertial measurement units 104a, 104b, 104c, and 104d.

[0074] FIG. 7 shows a flow chart used by the tracking system 100 to simulate body movements in a computing environment.

[0075] In step 702, the movement of the optical sensor obtained while tracking the body motion is measured by referencing different set points across successive frames captured by the optical sensor.

[0076] In step 704, the hub combines the measured rotational data obtained while tracking the body motion with the measured movement data from step 702, the measured rotational data being received within the hub from multiple inertial measurement units via one or more wireless communication channels.

[0077] In step 706, a data stream enabling a simulation of a body movement in the computing environment is output, and a movement of a body part in the computing environment is estimated from the measured rotation data of the body part and the measured rotation data of other connected body parts. A position of the body in the computing environment is estimated from the measured movement data.

[0078] In this application, unless otherwise specified, the terms "comprising," "comprise," and grammatical variations thereof are intended to express "open" or "inclusive" expression that includes the recited elements, but permits the inclusion of additional, not expressly recited elements.

[0079] Although the present invention has been described with reference to exemplary embodiments, those skilled in the art will recognize that various modifications can be made and equivalents can be substituted for the elements without departing from the spirit and scope of the invention. In addition, modifications can be made to adapt the teachings of the invention to a particular situation without departing from the essential scope of the invention. Therefore, the present invention is not limited to the specific examples disclosed herein, but includes all embodiments falling within the scope of the appended claims.

Claims

1. 1. A tracking system for simulating body movements in a computing environment, comprising: one or more processors; an optical sensor configured to signal that optical sensor movement has occurred via the one or more processors detecting differences between successive captured frames, wherein the one or more processors measure the movement by referencing an automatically determined, undefined set point across the successive frames; a plurality of inertial measurement units, each controlled by the one or more processors, for measuring rotational data; a hub in communication with the inertial measurement units and the optical sensors, the hub receiving the rotation data from the plurality of inertial measurement units via one or more wireless communication channels, the hub controlled by the one or more processors to combine the rotation data acquired while tracking the physical motion with the measured movement data acquired while tracking the physical motion to output a data stream that enables simulation of the physical motion in the computing environment; Equipped with a movement of a body part in the computing environment is estimated from the measured rotation data of the body part and the measured rotation data of other body parts concatenated therewith, and a position of the body in the computing environment is estimated from the measured movement data. Tracking system.

2. The tracking system of claim 1 , wherein a quaternion representation of the rotational data is used to estimate movement of the body part in the computing environment.

3. The tracking system of claim 1 , wherein the estimation of the movement of the body part in the computing environment is based on one or more forward kinematics algorithms.

4. The tracking system of claim 1 , wherein the optical sensor is integral with the hub, and movement of the optical sensor results from movement of the body to which the hub is attached.

5. The tracking system of claim 1 , wherein movement measurement begins with the optical sensor capturing an initial frame, the initial frame providing a starting point for tracking the body motion.

6. The tracking system of claim 1 , wherein the hub is configured to pair the inertial measurement units by proximity detection of radio signals emitted from the inertial measurement units.

7. 7. The tracking system of claim 6, wherein the hub is further configured to determine a body part assignment to which each of the plurality of inertial measurement units is attached by analysis of the output rotational data and strength of radio signals emitted to the hub.

8. The tracking system of claim 7 , wherein the hub is further configured to store a unique identifier for each of the plurality of inertial measurement units relative to an assigned body part during pairing.

9. 7. The tracking system of claim 6, wherein the one or more processors are configured to perform a calibration using data related to dimensions of body parts to which the plurality of inertial measurement units are attached following pairing and before starting tracking of the body motion.

10. The tracking system of claim 9 , wherein the one or more processors analyze an image including the body part to derive dimensions of the body part.

11. The tracking system of claim 10 , wherein the dimensions are derived using one or more of a machine learning algorithm and a skeletal structure model.

12. The tracking system of claim 10 , wherein the image is captured by the optical sensor.

13. 11. The tracking system of claim 10, wherein the derivation of the dimensions is performed in conjunction with the plurality of inertial measurement units attached to each body part, and by comparing the strength of radio signals emitted by the plurality of inertial measurement units with measurement data based on images of the corresponding body part.

14. The tracking system of claim 1 , wherein a base pose to apply before starting body motion capture is predetermined in the computing environment.

15. The tracking system of claim 14 , wherein the measured rotational data is used to derive an offset from the base pose, the offset being usable to construct a current pose.

16. 2. The tracking system of claim 1, wherein the hub is configured to enable extraction of the measured rotation data from one or more of the plurality of inertial measurement units and / or the measured movement data from the optical sensor obtained while tracking the body motion for recording as a macro.

17. The tracking system of claim 1 , wherein the body position obtained from the measured movement data is based on visual localization and mapping.

18. The tracking system of claim 1 , wherein the optical sensor is one or more of a stereo camera, a LIDAR, and an optical sonar sensor.

19. The tracking system of claim 1 , wherein at least one of the one or more processors is hosted within a computer platform.

20. 20. The tracking system of claim 19, wherein the estimation of the movement of the body parts in the computing environment and the estimation of the position of the body in the computing environment are performed on the computer platform to generate the data stream on the computer platform.

21. 1. A method for simulating physical movements in a computing environment, comprising: measuring the movement of the optical sensor obtained while tracking the body motion by referencing different automatically determined undefined set points across successive frames captured by the optical sensor; combining, at a hub, measured rotation data with measured movement data obtained during tracking of the body motion, the measured rotation data being received at the hub from a plurality of inertial measurement units via one or more wireless communication channels; outputting a data stream enabling a simulation of said physical movement in said computing environment; Including, a movement of a body part in the computing environment is estimated from the measured rotation data of the body part and the measured rotation data of other body parts concatenated therewith, and a position of the body in the computing environment is estimated from the measured movement data. method.

Images