Motion sensor calibration for whole-body or partial body tracking

Abstract

This technology relates to full-body or partial body tracking in multi-user augmented reality (XR) applications. A client device running in a multi-user XR application may be configured to determine a plurality of zones associated with the wearer, map a first motion sensor of at least one motion sensor to a corresponding zone where the first motion sensor corresponds to the first zone, determine a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone, receive first motion data of the wearer from the first motion sensor, map the first motion data to an avatar in a virtual world based on the first measurement offset, and draw the avatar in the virtual world based on the first motion data associated with the first zone.

Description

Background Art

[0001] (Cross-reference to related applications) This application claims the benefit of priority of U.S. Patent Application No. 18 / 190,505, filed on March 27, 2023, entitled "MOTION SENSOR CALIBRATION FOR FULL BODY OR PARTIAL BODY TRACKING", and incorporates the same in its entirety herein for all purposes.

[0002] Users of computing systems utilize avatars as stand-ins for their physical presence in a variety of applications, from simple chat applications to sophisticated three-dimensional (3D) environments used in video game applications and virtual reality applications. A simple version of an avatar can be a formless shoulder and head shape with no distinguishing features. Some avatars can be complex, associated with detailed graphics and textures, and capable of various animations. For example, some avatars are animated separately for realistic or unrealistic movements and include some parts such as hair, tails, ears, clothing, etc. In some cases, multi-user virtual reality applications can have hundreds of avatars and corresponding client devices interacting within that environment.

Summary of the Invention

[0003] In some examples, systems and techniques for tracking motion and mapping the motion to a virtual world are described. The systems and techniques can improve the tracking of motion in the physical world and map that motion to a virtual world to improve visual fidelity and immersive experience.

[0004] According to at least one example, the method includes: receiving data from at least one motion sensor, each of the at least one motion sensor being attached to a different body part of the wearer to track the wearer's movements; determining a plurality of zones associated with the wearer, each zone corresponding to the motion of a trackable body part or joint of the wearer; mapping a first motion sensor of the at least one motion sensor to a corresponding zone, the first motion sensor corresponding to a first zone; determining a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone; receiving first motion data of the wearer from the first motion sensor; mapping the first motion data to an avatar in a virtual world based on the first measurement offset; and drawing the avatar in the virtual world based on the first motion data associated with the first zone. For example, the device receives data from at least one motion sensor, where each of the at least one motion sensors is attached to a different body part of the wearer to track the wearer's movements, determines a plurality of zones associated with the wearer, where each zone corresponds to the motion of a trackable body part or joint of the wearer, maps a first motion sensor of the at least one motion sensor to the corresponding zone, where the first motion sensor corresponds to a first zone, determines a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone, receives first motion data of the wearer from the first motion sensor, maps the first motion data to an avatar in a virtual world based on the first measurement offset, and draws the avatar in the virtual world based on the first motion data associated with the first zone.

[0005] In another example, a device is provided that includes storage (for example, memory configured to store data such as virtual content data, one or more images, etc.) and one or more processors coupled to the memory and configured to execute instructions, along with various components (for example, a network interface, a display, an output device, etc.), which cause the device to receive data from at least one motion sensor, where each of the at least one motion sensor is attached to a different body part of the wearer to track the wearer's movements and to determine a plurality of zones associated with the wearer, where each zone corresponds to the motion of a trackable body part or joint of the wearer, and to map a first motion sensor of the at least one motion sensor to the corresponding zone, where the first motion sensor corresponds to the first zone, and to determine a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone, receive first motion data of the wearer from the first motion sensor, map the first motion data to an avatar in a virtual world based on the first measurement offset, and draw the avatar in the virtual world based on the first motion data associated with the first zone.

[0006] Aspects of this disclosure include two offset systems that enable the mapping of motion from the physical world to a virtual world. The disclosed avatar offsets can map motion from the physical world to the virtual world with high fidelity to generate an immersive and engaging experience. When a user changes avatars at different ratios, the disclosed aspects enable seamless changes between avatars without requiring separate calibration. For example, an XR multi-user application can map the user's position at the point when the user switches from the first avatar to the second avatar without requiring any calibration. [Brief explanation of the drawing]

[0007] To facilitate the identification of any particular element or behavior description, the most significant digit or number of digits in the reference number indicate the figure number in which that element is first introduced. Details of one or more aspects of the subject matter described herein are illustrated in the accompanying drawings and the following description. However, the accompanying drawings only illustrate some typical aspects of the disclosure and should not be considered limiting. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. [Figure 1] Figure 1 shows an exemplary virtual world platform for playing and hosting multiplayer virtual reality (VR) experiences according to several aspects of this technology. [Figure 2] Figure 2 shows an exemplary rendering of an avatar animation that can be performed using full-body tracking, according to some aspects of this disclosure. [Figure 3] Figure 3 shows a front view of a human being, which may be calibrated for partial or whole-body tracking using a motion sensor according to an aspect of the present disclosure. [Figure 4A] Figure 4A is a conceptual diagram of the calibration of a sensor attached to a human knee, according to some aspects of the present disclosure. [Figure 4B] Figure 4B shows a 3D conceptual diagram of a sensor attached to a human knee according to some aspects of the present disclosure. [Figure 4C] Figure 4C is a conceptual diagram illustrating how to determine the avatar offset based on the calibration offset, according to some aspects of this disclosure. [Figure 5] Figure 5 shows a conceptual diagram illustrating the determination of an avatar offset applied to an avatar in an extended reality (XR) multi-user application according to an aspect of this disclosure. [Figure 6A] , [Figure 6B] Figures 6A and 6B show conceptual diagrams of sensors attached to a human knee and the mapping of sensor motion to an avatar's knee in a virtual world, according to some aspects of the present disclosure. [Figure 7] Figure 7 shows an exemplary method 700 for tracking and calibrating motion sensors attached to a wearer and mapping the wearer's motion to a virtual world, according to some aspects of the present disclosure. [Figure 8] Figure 8 shows an example of a system for implementing a specific aspect of this technology. [Modes for carrying out the invention]

[0008] User interaction in virtual worlds, such as those hosted by several virtual reality (VR) platforms, continues to evolve. Initially, interaction was limited to coexisting in the same world and playing games together. Interaction has progressed to live communication and participation in virtual events. In recent years, users have begun to increase the range of motion of their avatars in order to interact with these virtual worlds and with objects within them. For example, characters in virtual worlds can now exercise and dance in virtual nightclubs, various contact-based interactions between characters can occur, and events occurring within the virtual world can be triggered.

[0009] A virtual world is a three-dimensional (3D) space that can have a set of interactions, including interactions within the environment and interactions with other avatars. In some cases, interactions within the environment can be performed using an input controller that provides an interface for interacting with the environment. A game controller can be a type of input controller, and an input controller may include a VR controller that can provide data for identifying a point in space where the VR controller is physically positioned. For example, a VR controller can be used to identify a point in space and the orientation of the VR controller.

[0010] The virtual world may implement inverse kinematics (IK), which is a mathematical process that calculates the variable joint parameters necessary to position the ends of a kinematic chain, such as a robot manipulator or the skeleton of an animated character, at a given position and orientation relative to the viewpoint of the chain. IK can be used to determine the position of various aspects of the virtual world, such as the location of a player's arm within the virtual world.

[0011] Using a VR controller or another conventional input controller can only provide a limited range of input within the virtual world, and IK alone cannot accurately determine the position of corresponding body parts of an avatar within the virtual world. For example, IK can accurately determine the position of the wrist corresponding to the location of the VR controller in space, but IK may not be able to accurately determine the position of the elbow due to the multiple types of movements possible in the physical space of a human shoulder.

[0012] Some virtual worlds can support more complex interactions based on the use of full-body tracking, which can be achieved using various technologies. One technology is called inside-out tracking, which uses several motion sensors, such as inertia measurement units (IMUs), placed at various locations on the body. Other examples of motion sensors include cameras, photodiodes, gyrometers, accelerometers, and time-of-flight (ToF) sensors. Inside-out tracking refers to tracking the movement and position of a person using sensors attached to them. Another technology for full-body tracking is called outside-in tracking, which uses a full-body tracking suit with an object that is tracked using an external system. Outside-in tracking refers to using an external sensor system to identify the movement and position of the tracked subject. Another technology for full-body tracking is attaching an object to the body that is tracked using an external system, such as a suit, straps, or other attachment methods.

[0013] Complex movements in a virtual world can be achieved through full-body tracking, which requires several motion sensors to track the positions of the shoulders, elbows, hips, feet, and knees. However, motion in the physical world is difficult to track because it is tracked using external devices that do not directly distinguish between different types of movement (e.g., orthogonal motion, rotational motion, etc.). Furthermore, motion is not always directly convertible to the virtual world due to differences in proportions and sizes between virtual avatars and humans. For example, a virtual avatar could be a robot with different proportions than a human. In another example, a virtual avatar could be a dinosaur with fundamentally different anatomical proportions than a human.

[0014] One aspect of this technology relates to calibrating motion sensors based on provided information. For example, a user's height may be provided to a system performing whole-body or partial body tracking, and the technology may determine zones based on that height to calibrate each corresponding sensor. In some cases, the fixed point of interest may be a joint such as the knee or ankle, but it may also be related to a bone such as the upper arm. The zones identify the area in which the sensor is expected to be within its range, based on the user's height. The technology may also determine calibration associated with motion sensors based on fixed points of interest within zones associated with body parts, which are used to identify appropriate movements. For example, complex movements of the shoulder and arm may be determined by a combination of a held controller and a fixed sensor attached to the user's upper arm. An offset may be used to identify motion in the physical world that can be accurately mapped to a virtual world.

[0015] One aspect of this technology also relates to mapping motion detected in the physical world to a corresponding avatar in the virtual world. In some aspects, fixed points associated with avatar features can be aligned with estimated positions based on the user. For example, the estimated positions of the user's eyes (based on the user's input height) and the positions of the avatar's eyes can be aligned in space, and a calibration offset can be determined from a fixed point on the user's body (e.g., ankle, center of upper arm) to a fixed point on the avatar's geometry. This technology uses avatar offsets and calibration offsets to map motion in the physical world to the virtual world. For example, based on the calibration offsets and avatar offsets, movement in the physical world can be accurately mapped to the virtual world regardless of ratio differences, differences in avatar anatomical configuration, etc. According to aspects of this disclosure, the disclosed avatar offsets can map motion from the physical world to the virtual world with high fidelity to generate an immersive and engaging experience. When the user changes avatars by different ratios, the disclosed aspects enable seamless changes between avatars without requiring separate calibration. For example, an XR multi-user application can map the user's position when switching from a first avatar to a second avatar without requiring any calibration, and the second avatar can be aligned, scaled, and positioned within the virtual world without generating any visual artifacts.

[0016] Figure 1 shows an exemplary virtual world platform 102 for playing and hosting a multiplayer VR experience suitable for implementing this technology. The virtual world platform 102 connects clients 104 through web services 110 and networking services 112, enabling them to socially interact with each other in a virtual world hosted by the virtual world platform 102.

[0017] The virtual world platform 102 primarily includes clients 104, which are instances of applications running on client devices 106. Clients 104 interact with web services 110 via network connectivity, which support them by providing various services through one or more application programming interfaces (APIs). Some of the main services provided by web services 110 relate to supporting virtual worlds through the world API 128, user profiles through the user API 132, trust and security through the trust API 144, and complex avatars through the avatar API 136. Among other functions, web services 110 generally store and provide long-term state information.

[0018] Client 104 also interacts with the networking service 112, which provides communication services between Client 104, the networking service 112, and remote instances of Client 104 (not shown), in order to share state information between each instance of Client 104. In particular, state information is received by the networking service 112 from multiple instances of Client 104 when each instance of Client 104 controls its local player 116. The networking service 112 can forward state information about each player to other instances of Client 104 when all local players 116 of each client instance are participating in gameplay in the same virtual world. The networking service 112 provides optimized packet routing through the optimized packet routing service 140 and moderation between one or more clients through the moderation service 142.

[0019] Client 104 is a runtime environment that runs on a specific client device 106. In this specification, it may be referred to as Client 104, local client, and remote client, but these are all instances of Client 104 running on their respective client devices 106. One specific user account logs in to a specific instance of Client 104. The local client and the remote client are distinguished to explain how Client 104 processes first-person input from the user of the client device 106 on which Client 104 runs, and how it processes third-party input received from the person who operates the client device of another user on which the remote client runs.

[0020] The client device 106 can be any computing device. While Client 104 is particularly adapted to provide an immersive virtual reality experience through interactions that require a VR headset, Client 104 can also be run by computers and mobile devices. Some virtual worlds or complex avatars may not be configured to operate well on a specific device type. Therefore, while Client 104 can operate on many platforms and devices, not all virtual worlds or complex avatars are available or fully functional on all client devices 106.

[0021] The user interface service 108 is one service that is part of the client 104. The user interface service 108 is configured to provide various user interface elements such as menus that display various user settings, available worlds, saved composite avatars, friend lists, etc. The user interface service 108 can add its menu through interaction with one or more APIs provided by the web service 110, while other parts of the menu are read directly from the user interface service 108.

[0022] The user interface service 108 can provide a menu of available worlds by calling the World API 128 to obtain a list of worlds that the user account logged in to the client 104 is permitted to access. The World API 128 can obtain all public worlds from the world asset database 130 and send a list of them to the client 104. Further, the World API 128 can request the world IDs of any private worlds associated with the user account logged in to the client 104 and obtain the private worlds from the world asset database 130 for sending to the client 104. The user interface service 108 can navigate through the world menu and receive user input through the hardware interface to receive a selection of the world to visit.

[0023] Another user interface provided by the user interface service 108 relates to various user settings. Such settings can relate to whether a human player is sitting or standing, settings for minimizing motion sickness for players who are prone to motion sickness when playing in VR, settings for selecting complex avatars, how a player is viewed, and by whom a player is viewed in the virtual world.

[0024] One notable user interface provided by the user interface service 108 is the Trust and Safety menu. The user interface service 108 can contact the user API 132 to retrieve current trust and safety settings from the user profile database 134 and display these settings within the Trust and Safety menu. The Trust and Safety menu provides the user account with the ability to determine which remote players 124 can see the user's avatar (local player 116), or which remote players the user's avatar can see, when both are in the same world. For example, it may be desirable to avoid interaction with new users on the virtual world platform 102 because they have not yet established trust within the virtual world platform 102. It may also be desirable to limit the characteristics of remote player avatars handled by the instance of client 104 to which the local user is logged in, because some avatars may have malicious data embedded in them, or the avatars may be too complex to render without degrading the performance of the client device 106. For example, a user account might decide to turn off the lights on a remote avatar to avoid shaders, or not allow custom animations. In some embodiments, each of these options might be set based on how much trust the remote player is given. For example, a user account might allow their friends' avatars to have full features, while others only display basic avatar features.

[0025] The user interface service 108 can also provide an option to mute or block specific remote players. Furthermore, the user interface service 108 can provide a panic mode that audibly and visually mutes anyone who is not a friend.

[0026] After the user selects a virtual world from a menu provided by the user interface service 108, the client 104 can download an instance of the virtual world by calling the world API 128, which can retrieve the virtual world from the world asset database 130, and send it to the client 104 for execution.

[0027] A world asset is a large binary file built like a game engine such as Unity, using an editor with a software development kit (SDK) provided for use with the virtual world platform 102. When a user travels to a world, they need to download that world asset from the world asset database 130. If there are already people in that instance of the world, the client 104 also needs a list of those people's avatars so that the avatars can be rendered in the instance of the virtual world.

[0028] In some embodiments, the functionality of the World API 128 can verify that a user account has access to the requested world. While a user account should only have the ability to browse public worlds within the user interface menu, or only knowledge of links to worlds shared with the user account, the World API 128 can, as a redundancy measure, verify whether the user account is permitted to access the virtual world in question.

[0029] In addition to downloading an instance of the virtual world, client 104 can also establish a session with networking service 112 for a specific instance of the world. Networking service 112 can provide information about the current state of the virtual world instance. For example, networking service 112 can provide client 104 with a list of remote avatars 126 that exist in the virtual world instance. Client 104 can then contact avatar API 136 to download complex avatar assets for a list of remote complex avatars from avatar asset database 138.

[0030] If client 104 does not have an asset for local avatar 118, client 104 can contact avatar API 136 to request and receive local avatar assets. An avatar asset is a single binary file containing all the textures and models necessary to render the avatar, as well as animation data. In some examples, it may include more complex features, such as a particle system or light source, or data on whether the avatar follows or does not follow the established physical laws in the virtual world, or whether the avatar has non-standard motion dynamics. In some embodiments, an avatar asset may include colliders and receivers defined on parts of the avatar, or a tree of transformations that cause parts of the avatar to exhibit secondary motion behavior (for example, dynamic or physical bones (aka phys. bones) are an example of a system that can configure parts of an avatar to exhibit secondary motion behavior).

[0031] The downloaded instance of the virtual world can be run by client 104 as the current world 120. The current world 120 can include the coordinates within the current world 120 in which the local player 116 and each remote player 124 are located. The local player 116 and remote players 124 are each collision volume of space occupied by the local player 116 or the remote player 124, respectively.

[0032] Local avatars 118 can be mapped to local players 116, and each remote avatar 126 can be mapped to each remote player 124, thereby allowing each player to appear as their avatar in the current world 120. The movement of the remote avatars 126 is handled by receiving state data about each remote avatar / player and rendering the movement or sound by the client 104.

[0033] The VR tracking service 114 relates to a client 104 that operates on a client device 106 having access to VR tracking peripherals. For example, some VR headsets have (integrated or external) cameras for tracking the player's hands and feet. Many VR headsets can be paired with controllers that can report the location of the user's hands in space. Some client devices 106 include other peripherals configured to perform full skeleton tracking. The VR tracking service 114 can merge all VR inputs connected to the client.

[0034] The VR tracking service 114 can map fused VR input to the local player 116 in order to enable the local player 116 to interact with the current world 120. Meanwhile, the local player 116 can interact with the local avatar 118, map the local avatar 118 to the local player, and make the local player 116 appear as that person's avatar.

[0035] In some embodiments, there is variation in which parts of the user's body are tracked by the VR tracking service 114. Some users may have full skeleton tracking, while many may only have the ability to perform hand tracking. To address this imbalance in the hardware capabilities of possible client devices 106, the local player 116 can derive parts of the skeleton that are not tracked by the VR tracking service 114. For example, if the VR tracking service 114 only provides the user with information about hand tracking, the local player can still derive a full skeleton for the user and move parts of the skeleton in response to hand movements. In this way, the avatar's hands do not move as if they were detached from the rest of the avatar.

[0036] Local Player 116 is an entity that moves around the environment within the current World 120. It can pick up and put down objects. It has no animations and is a collision volume. It can do anything in the world, but has no appearance and does not need to be animated.

[0037] The local player is further connected to the networking layer, which is referred to as Runtime Networking Service 122, and broadcasts status information about the local player 116 to other users in the current instance of World 120 over the network.

[0038] Local player 116 and remote player 124 are similar in that they are collision volumes moving around in the current world environment 120. The main difference is that local player 116 is controlled by client 104, and the user of client 104 is creating the experience. In contrast, remote player 124 is a playback mechanism that represents actions broadcast to client 104, representing other players present in the current world 120.

[0039] As described above, the local avatar 118 is overlaid on the local player 116 to give the user a visual appearance. The actions of the local player 116 are animated as the local player interacts with the current world. For example, the local player 116 can interact to pick up an object in the current world 120, but without the local avatar 118, the object would appear to be floating in mid-air. When the local avatar 118 is overlaid on the local player 116, the object immediately appears to be held in the avatar's hand.

[0040] The remote player 124 and remote avatar 126 function similarly to their local counterparts, except for the source of the input that controls the remote player 124. The remote player 124 and remote avatar 126 are playback devices for state information received from the networking service 112 by the runtime networking service 122. Figure 1 shows only one remote player 124 and remote avatar 126, but many may exist.

[0041] Client 104 can also support contact interactions between avatars, between a part of an avatar and another part of the same avatar, or between a part of an avatar and an object in the virtual world. To detect these interactions, client 104 may be configured to detect collisions between objects using a collision detection system 148. In some embodiments, the collision detection system 148 may be a broad-phase collision detection system.

[0042] The current world 120 also has features that require networking. The current world 120 may have objects that users can interact with and that can dynamically change the state of the current world 120, and such objects need to broadcast their state across the network so that other users in the current world 120 can see the current state of those objects. In one exemplary example, the current world 120 may include a light switch that dynamically affects light sources within the current world 120, such as turning lights on or off.

[0043] Local player 116, current world 120, and remote player 124 are each connected to runtime networking service 122. Local player 116 primarily sends updated state information for local player 116 to a remote instance of client 104, which is similarly running the same virtual world. Current world 120 can send and receive state information about its virtual world instance. Current world, running on client 104, sends state information when the state change is owned by local player 116 and receives state information when the state change is owned by remote player 124.

[0044] The networking service 112 is the network-side portion of the networking layer of the virtual world platform 102. In some embodiments, a portion of the networking service 112 is provided by a networking service, such as the PHOTON networking engine, which broadcasts state information to all users within the virtual world instance.

[0045] In addition to broadcasting general state information to all users interacting with virtual world instances, the optimized packet routing service 140 provides more advanced features that deliver an enhanced user experience and enhances other virtual world platform 102 characteristics such as trust and security settings.

[0046] For example, to provide an enhanced user experience, an optimized packet routing service 140 can filter out voice packets from remote players 124 that may be far away from the local player 116 in the current world instance 120. Without such optimization, a remote player 124 that does not interact with the local player, or is not even visible to the local player, could receive voice packets from tens or hundreds of other remote players 124, making it difficult for any subset of remote players 124 to communicate.

[0047] In another example, the optimized packet routing service 140 can enhance trust and security settings. As mentioned above, trust and security settings can specify that certain user accounts or groups of user accounts should be filtered so that they cannot interact with the local player 116, or their interaction with the local player 116 is restricted. The optimized packet routing service 140 can call the trust API 144 to know, for a local player 116 with trust and security settings, a list of remote players 124 whose network traffic going to or from client 104 may need to be filtered or blocked to some extent.

[0048] The Trust API 144 can determine which remote players 124 should be blocked from the local player 116, or which remote players 124 should have their complex avatar configurations restricted. Some of these decisions are based on logic and rules that classify remote players 124 based on the amount and type of their past interactions with the virtual world platform 102. The Trust API 144 may make these decisions by using settings stored in the local player 116's user profile and comparing these settings with data stored in the remote player 124's user profile.

[0049] Another aspect of the networking service 112 is the moderation service 142, which can provide conflict resolution and access control. For example, before a user can access a world, especially a private world, the moderation service 142 can call the world API 128 to check if the user is allowed to enter the world. In another example, two different users may attempt to claim control of an object in a virtual world at almost the same time. The moderation service 142 can handle these types of conflicts by selecting a specific user and controlling the object until that user relinquishes control of it, thereby allowing another user to claim control of the object. The user who has control of the object can broadcast a packet to the remote player 124 informing it of the object's state.

[0050] In some embodiments, the client 104, virtual world, and complex avatar may be configured to operate in a specific game engine, particularly one that supports a three-dimensional (3D) environment. Two common game engines include Unity and Unreal Engine.

[0051] In some embodiments, the virtual world and complex avatars must be developed in accordance with the Software Development Kit (SDK) in order to be supported by the virtual world platform 102. For example, a complex avatar requires specific scripts that are available on the virtual world platform 102. In another example, there may be several requirements that must be followed to obtain the animation of the avatar to be played. In some embodiments, the SDK can define other necessary details to support a particular client device. For example, the SDK can define specific shaders to be used when the avatar is used on an OCULUS QUEST VR headset.

[0052] In some embodiments, the SDK requires a virtual world to utilize a specific coding language to ensure that the world has compliant behavior. For example, the SDK may require that behavior within the world be defined using UDON, a programming language specific to a particular virtual world platform 102, VRCHAT. In some embodiments, the programming language facilitates the creation of worlds using that language to comply with file access protections provided by the virtual world platform 102. For example, the world may not be able to read or write anything to the hard drive, and only authorized web pages may be rendered on the virtual world platform 102.

[0053] In some embodiments, the virtual world platform 102 may also include a simplified avatar service 146. As described herein, the simplified avatar service 146 can create simplified versions of complex avatars and store the avatar assets of these simplified versions of complex avatars in an avatar asset database 138.

[0054] While the virtual world platform 102 is suitable for implementing this technology, those skilled in the art will understand that this technology can be used in other environments.

[0055] Figure 2 shows an exemplary rendering of an avatar animation that can be performed using full-body tracking, according to some aspects of the present disclosure. As will be further explained with reference to Figure 3, a user can wear motion sensors on various body parts, enabling the wearer to perform motions that are tracked from the physical world and mapped to the virtual world. Conventionally, a person may hold a VR controller for hand-associated input and may have a VR headset that tracks the orientation of the person's head. However, the animation shown in Figure 2 cannot be performed using conventional methods because different regions cannot be directly mapped from the physical world to the virtual world. Conventionally, multi-user applications can trigger programmed effects, but dynamic positioning, such as that shown in Figure 2, applied in the physical world, cannot be mapped without additional motion controllers.

[0056] In one exemplary embodiment, a user (e.g., a wearer) may have multiple motion sensors that can be attached to the user's body to track motion in order to achieve the motion and nonverbal communication shown in Figure 2. The number of motion sensors may vary based on the desired amount of motion tracking. For example, a user may have three motion sensors, which may be attached to the right arm region 202, the left arm region 204, and the chest region 206. For clarity, the regions are identified based on their orientation as perceived by the user.

[0057] In this example, the user may use motion sensors for upper body tracking, and additional controllers, such as a VR controller (not shown) held by the user, may be used to identify additional movements. The VR controller is physically held and provides a reference for hand position, but may also include additional controls such as joysticks and buttons. In some embodiments, the VR controller and arm motion sensors may be used to determine shoulder position. Based on the three motion sensors, the VR headset, and the VR controller, the user may be able to obtain upper body tracking that can be used for various interactions in the virtual world. For example, the user may high-five another player, thereby generating interactive effects in the virtual world (e.g., fireworks, a halo, etc.). Interactive effects may be designed by the user, subject to several restrictions and approval processes to ensure that offensive content is restricted.

[0058] In another example, the user can wear motion sensors in the waist region 208, right knee region 210, left knee region 212, right foot region 214, and left foot region 216. Wearing motion sensors in the waist region 208, right knee region 210, left knee region 212, right foot region 214, and left foot region 216 would enable lower body tracking and movement. In some embodiments, the lower body sensors can be used to play virtual games or virtual sports, such as soccer matches, dancing, or other activities that can be participated in within a virtual world.

[0059] To animate various motions, motions can be applied to an IK solution, which can then identify the motions and animate the avatar in the world.

[0060] In some aspects, motion sensors should be calibrated to align the user's motion with that of an avatar. For example, the height of a virtual avatar may differ from the user's height. Calibrating motion sensors can be difficult because their physical placement varies based on various external factors such as the user, clothing, and environment, the type of sensor, and other factors. Calibration errors propagate when applied to a virtual world, potentially generating inaccurate motion, impossible motion, and other effects that negatively impact the user experience. Furthermore, calibration errors can be amplified when a user switches between their representative avatars, further negatively impacting the user experience.

[0061] A high-quality experience within a virtual world enhances immersion, increases engagement with other avatars, and provides meaningful engagement. Virtual worlds can be used for a variety of functions, including business meetings, creative planning, games, sports, and creative features (e.g., music), and providing a high-quality experience increases opportunities to grow the virtual world and enhance connectivity.

[0062] Figure 3 shows a front view of a human being that can be calibrated for partial or full-body tracking using motion sensors according to embodiments of this disclosure. In some embodiments, the calibration method disclosed herein aligns motion sensors identified in specific zones to determine a calibration offset, which is mapped to a virtual world to improve motion tracking and apply motion from the physical world to the virtual world. User avatars in the virtual world do not have to have the same physical characteristics (e.g., height, torso length, etc.), and motion in the physical world may not be accurately reflected in the virtual world.

[0063] In some embodiments, an Extended Reality (XR) multi-user application is configured to receive or detect physical information, such as the user's height. For example, an XR headset may include multiple sensors and be configured to detect the user's height. In another embodiment, the user may input their height into the XR multi-user application or a service associated with the XR multi-user application. In some embodiments, the XR application may be a full VR experience or an augmented reality (AR) experience that combines physical world content with virtual content. The XR multi-user application may also receive other physical information, such as information related to physical disabilities (e.g., color blindness, disability information, etc.), gender, etc. Based on the physical information, the XR multi-user application can identify multiple calibration zones in the physical world and map detected sensors to their corresponding calibration zones. As described below, each calibration zone is used to identify sensors associated with a particular body part and a calibration offset relative to a fixed point in the corresponding calibration zone.

[0064] In one exemplary example, an XR multi-user application may use median characteristics based on input information to identify different regions. For example, a taller person will have different zones in different physical locations than a shorter person. In some embodiments, multiple zones include a right arm zone 302, a left arm zone 304, a chest zone 306, a waist zone 308, a right knee zone 310, a left knee zone 312, a right foot zone 314, and a left foot zone 316. Although each zone is shown as two-dimensional (2D) in Figure 3, the zones exist in 3D space because motion sensors are generally not aligned to a single plane. For example, a foot sensor may be located at the end of the foot, and an arm tracker may be attached to the arm with an armband; these sensors are not aligned.

[0065] An XR multi-user application is configured to identify various sensors attached to the wearer when the application is launched, before the user enters the virtual world, or during a calibration operation that can be triggered by the user (e.g., recalibration). For example, an XR multi-user application may be configured with information related to various sensors, allowing it to identify those sensors and map them to specific joints. In some cases, the sensors may be attached to multiple locations on the user (e.g., a motion sensor may be attached to both the arm and the knee). The calibration process may require the user to stand to perform an initial calibration or to input a pose that can be used to identify movement. An example of a calibration pose is the T-pose shown in Figure 3, where the XR multi-user application identifies the corresponding sensor in the calibration zone (e.g., the right arm sensor into the right arm zone 302) and maps the sensor to the user's body part associated with the calibration zone. Thus, the XR multi-user application maps each sensor to the location worn by the user, regardless of where the sensor was previously attached to the user. For example, if a user previously wore a motion sensor on their right knee, and then wears the same motion sensor on their left arm, the XR multi-user application will detect the sensor in the left arm zone 304. The XR multi-user application will dynamically detect which motion sensor belongs to which body part, regardless of previous manual mapping.

[0066] In a further embodiment, the XR multi-user application may also use the calibration pose shown in Figure 3 to identify one or more calibration offsets associated with the sensor based on the fixed points associated with the zones. In some cases, the fixed points can be center points (e.g., the central chest region), but some joints may have neutral positions that result in the fixed points being offset. For example, the ankle fixed point is biased towards the wearer's vertical centerline, and the center point of the right foot zone 314 is further to the left.

[0067] In some embodiments, an XR multi-user application may be configured to identify a calibration offset from a sensor to a fixed point based on identifying a corresponding sensor within a calibration zone. The fixed point may be associated with a joint corresponding to the user and can be mapped to a virtual reality world. For example, the right knee zone 310 and the left knee zone 312 may be associated with the knee joint, and the right foot zone 314 and the left foot zone 316 may be associated with the ankle. In some cases, the sensors do not necessarily have to be associated with a joint. For example, an arm motion sensor may be attached to the wearer's upper arm. In some embodiments, the positions of the elbow and shoulder joints can be identified based on a single motion sensor.

[0068] An XR multi-user application can be configured to identify zones based on the wearer's height. For example, if the user is in the calibrated upright position shown in Figure 3, the XR multi-user application can identify the position of the corresponding motion sensor relative to a fixed point and determine a first calibration offset. The first calibration offset indicates the position of the motion sensor relative to the fixed point. For example, Figure 4A shows the right knee calibration offset based on the position of a sensor attached to the knee.

[0069] Although not illustrated, the wearer may also hold a controller containing motion sensors that enable various interactions within the virtual reality world. In one exemplary example, the controller is a separate handheld device that detects motion and inputs it to various input components (e.g., joystick, buttons, etc.). The controller can be used to identify wrist position in relation to arm motion sensors to enable complex arm tracking, for example, non-verbal input as shown in Figure 2.

[0070] In some embodiments, a user of an XR multi-user application may configure partial motion tracking by omitting one or more sensors for corresponding areas. For example, a user may omit knee and foot trackers instead of using alternative inputs for movement in a virtual reality world. The zones shown in Figure 3 are illustrative and illustrate one mechanism for calibrating the positions of various motion sensors. Additional zones may be configured, zones may be removed, and zones may be in different positions. For example, an arm zone may be associated with a neutral position where the user's hand is located on the corresponding side of their body. The zones shown in Figure 3 are clearly distinct and do not overlap, but zones can overlap. For example, the right arm zone 302 and the left arm zone 304 may at least partially overlap the chest zone 306. In some embodiments, the calibration zone size may be user-configurable and can be adjusted by the user. For example, a user may find it difficult to calibrate a particular zone for various reasons and may choose to enlarge that particular zone to obtain calibration. In other embodiments, all calibration zones can be scaled using a single setting to allow for trackers that may be active in the VR system but are not worn on the body.

[0071] Figure 4A is a conceptual diagram of a calibration pose for identifying a knee calibration offset, relating to some aspects of this disclosure. As described above, the human 400 may be required to perform an initial calibration based on a calibration pose. An example of a calibration pose is the T-pose shown in Figure 4A. Other examples of calibration poses include a neutral pose, a seated pose, a kneeling pose, and so on. In some cases, the calibration pose may vary based on the number of sensors, other physical impairments, and other factors.

[0072] In the illustrated embodiment, a human 400 wears a sensor 402 on their knee within a calibration zone 404 associated with their right knee, which has a fixed point 406. In this embodiment, the fixed point is aligned with the joint of the human 400 in 3D space, and the motion sensor is shown as being detected at point 408. The XR multi-user application is configured to determine the sensor 402 based on its placement within the calibration zone and to map the sensor 402 to the right knee of the avatar. The XR multi-user application may also determine a calibration offset from the fixed point 406 to point 408. This vector is determined for each sensor associated with the human 400 based on the various calibration zones shown in Figure 3. In this case, the calibration offset maps the position of the sensor 402 in the real world to the corresponding point in the virtual world. In this case, the fixed point corresponds to the center point of the knee joint.

[0073] The XR multi-user application is configured to determine the avatar offset based on the calibration offset. For example, regardless of the avatar used by the user, the XR multi-user application uses the calibration offset 410 and the avatar offset 426 to map motion from the physical world to the virtual world with high fidelity to create an immersive and engaging experience. Figure 4C below visually illustrates an example of mapping the motion of a human 400 to the virtual world based on the calibration offset and the avatar offset.

[0074] Figure 4B shows a 3D conceptual diagram of a sensor mounted on a human knee according to several embodiments of the present disclosure. Figure 4B shows a 2D diagram of the sensor, but the sensor 402 exists in 3D space in the physical world. In one exemplary embodiment, the human knee is shown as a fixed point 406, and data from the motion sensor is shown as being detected at point 408. In this case, an XR multi-user application can identify the position of the motion sensor being calibrated based on its placement within a calibration zone 404, identify the fixed point based on the identified body part (e.g., the knee), and determine a calibration offset 410 associated with the motion sensor. The calibration offset identifies the 3D position from the human joint to the position of the motion sensor.

[0075] Figure 4C is a conceptual diagram illustrating how an avatar offset is determined based on a calibration offset, according to some aspects of the present disclosure. In some aspects, the avatar offset is specific to each avatar based on the calibration offset. For example, the avatar offset is calculated each time the user deploys an avatar in the virtual world, since the calibration offset changes based on the position of the sensor and each sensor has some flexibility to allow the user to position the sensor for their comfort.

[0076] In some embodiments, Figure 4C shows that a human and their corresponding calibration zones are aligned in 3D space by an XR multi-user application. In one embodiment, a 3D model of a human 415 and an avatar 420 are aligned based on eye position 422. Eye position 422 is used as a calibration zone based on the XR multi-user application (which has accurate data related to the user's eye position, as the user is wearing an HMD to consume the XR multi-user application). In this case, the human 3D model is representative and generated based on the user's input height, and only the head of human 415 is illustrated so as not to obscure more relevant details. Human 415 is also shown in Figure 3, and the corresponding calibration zones 302-316 are mapped in 3D space based on eye position 422.

[0077] In this exemplary embodiment, each calibration zone includes a fixed point, a calibration offset, and an avatar offset. As described above, the fixed point is mapped to a position associated with a human in the physical world, and the calibration offset includes a vector from the fixed point to the position of the corresponding sensor. For example, the calibration offset 410 associated with the right knee of human 415 indicates that the sensor 402 is mounted on the outer edge of the right knee.

[0078] In some embodiments, the avatar offset includes a vector from the motion sensor to the corresponding fixed point on the avatar 420. For example, when the avatar is loaded into an XR multi-user application for use in a virtual world, the avatar offset 426 is determined for each motion sensor based on the calibration offset. For example, when a user uses the avatar 420, the XR multi-user application determines the avatar offset from the calibration offset 410 to the corresponding point on the avatar 420, such as the right knee shown in Figure 4C.

[0079] XR multi-user applications use avatar offsets to map motion from sensors to the virtual world. For example, since a human in the physical world is taller than avatar 420, motion detected in the physical world is calibrated to the virtual world based on avatar offset 426. This ensures that motion detected by various motion sensors of a human in the physical world is accurately mapped to the virtual world. Because the avatar's height is dynamically scaled to best match the user to the position of the avatar's body parts, the difference in proportion between the avatar and the human is more important than the difference in absolute height. In some embodiments, the position of the avatar's body parts can differ significantly from the position of the human's body parts, with a degree of positional dispersion that can cause motion sensors to align with different joints, even when aligned to the same eye level. For example, a physical motion sensor worn on a human's waist may be positioned closest to the avatar's knee. A simple approach is to map the nearest motion sensor to the nearest avatar bone / joint, but in this example where the proportion mismatch between the user and the avatar is significant, this simple approach results in inaccurate mapping of motion sensors to joints. In various configurations, the calibration zone prevents the motion sensor from inaccurately mapping to joints.

[0080] In some embodiments, the XR multi-user application stores the calibration offset during calibration or recalibration, eliminating the need for the application to recalibrate the sensors when the user switches to a different avatar. Rather, the XR multi-user application determines the avatar offset based on the stored calibration offset when the avatar is loaded and deployed into the virtual world. For example, the alignment in the virtual world replaces the user by aligning the calibration zone with the loaded avatar, by aligning the eye position associated with the calibration zone with the eye position defined within the avatar. As a result, the user does not need to physically assume a calibration pose (e.g., a T-pose) each time a different avatar is loaded.

[0081] A single avatar offset is described, but as shown in Figure 4C, an avatar offset is determined for each of the calibration zones 302–316 that have corresponding motion sensors. For example, the user may only have an upper body tracker, and the XR multi-user application may assign motion sensors to the corresponding zones during calibration and disable motion sensing for calibration zones without sensors. Furthermore, some calibration zones may have multiple associated avatar offsets. As an example, the upper arm bone of the avatar may include offsets for the elbow and shoulder joints.

[0082] Figure 5 shows a conceptual diagram illustrating the determination of an avatar offset to be applied to an avatar in an XR multi-user application according to an aspect of this disclosure. In particular, Figure 5 shows the profiles of a human 510 with different heights and the profiles of an avatar 520. Figure 5 has been distorted for ease of explanation, and various properties may not be depicted to scale.

[0083] In some embodiments, motion applied to a motion sensor is not mapped to an XR multi-user application based on an offset in the physical world. For example, avatar 520 is approximately half the height of human 510. Human 510 includes eye fixation point 512, waist fixation point 514, knee fixation point 516, and foot fixation point 518, while avatar 520 includes eye fixation point 522, waist fixation point 524, knee fixation point 526, and foot fixation point 528. As described below, various fixation points can be used to identify various avatar offsets, which can be used by an XR multi-user application to map motion in the physical world to the virtual world with high fidelity.

[0084] In one embodiment, at least one fixed point may be used to align the human 510 and the avatar 520 to a common reference point. For example, the eye fixed point 512 of the human 510 and the eye fixed point 522 of the avatar 520 are aligned, and the positions from various fixed points are calculated. For example, the waist avatar offset 534 corresponds to the difference between the waist fixed point 514 of the human 510 and the waist fixed point 524 of the avatar 520, the knee avatar offset 536 corresponds to the difference between the knee fixed point 516 of the human 510 and the knee fixed point 526 of the avatar 520, and the ankle joint avatar offset 538 corresponds to the difference between the foot fixed point 518 of the human 510 and the foot fixed point 528 of the avatar 520.

[0085] In some embodiments, the avatar offsets in Figure 5 (e.g., waist avatar offset 534, knee avatar offset 536, foot avatar offset 538) may be used in conjunction with a calibration offset (e.g., calibration offset 410) to map motion from the human 510 to the avatar 520. As described above, the calibration offset 410 is associated with measurements from fixed points on the wearer's body to the sensors. Based on the various avatar offsets in Figure 5, an XR multi-user application may determine mapping information that maps the motion of the sensors in the physical world to the motion of the avatar in the virtual world. In some cases, an XR multi-user application may also determine the mapping information during an instance of the XR multi-user application. For example, in some cases, a user may switch to wearing different avatars within the XR multi-user application, and the XR multi-user application may recalculate the mapping information based on the selected avatar, as described above.

[0086] Figure 5 illustrates the transformation of human proportions in a virtual world based on motion sensors, using a single dimension (e.g., height). While the avatar offset is illustrated as a one-dimensional difference based on height, the avatar offset can be two-dimensional or three-dimensional. For example, the avatar may be significantly wider or narrower, and may have joints positioned in anatomically inaccurate locations. Techniques for mapping motion in the physical world can be applied regardless of the anatomical differences of the avatar in the virtual world.

[0087] Figure 6A shows a conceptual diagram of a sensor mounted on a human knee and the mapping of the sensor's motion to the avatar's knee in a virtual world, according to some aspects of the present disclosure. In some aspects, a fixed point 602 may correspond to the center point of the human knee, determined based on the human's input height in an XR multi-user application. The motion sensor is mounted on the human knee, and the XR multi-user application determines the position 604 of the motion sensor mounted on the knee, and then determines a calibration offset 606 from the fixed point 602 to position 604. As described above, the fixed point 602 may be associated with a corresponding zone 608, and the fixed point 602 corresponds to the knee joint. In some cases, the XR multi-user application may be able to calibrate the motion sensor mounted on the human knee based on the identification that the human is in a seated position. For example, the human torso would not be aligned in the vertical plane with the motion sensor positioned at position 604.

[0088] An XR multi-user application may determine the knee fixation point 610 based on the difference between the estimated physical anatomical structure of a human and the anatomical structure of the user avatar. In some cases, the user avatar may include information that identifies various points of interest, such as the knee fixation point 610 (e.g., during design, via standard avatar armature rigging practices). Based on the knee fixation point 610, the XR multi-user application may be configured to calculate at least one avatar offset 614. Based on the calibration offset 606 and the avatar offset 614, the XR multi-user application may be configured to translate the motion relative to the fixation point 602 to the avatar at the knee fixation point 610. For example, the motion may relate to a leg bone (e.g., fibula) attached to the avatar's knee.

[0089] In some cases, avatar offsets can be applied to other aspects of the virtual world. For example, a user may hold a controller that is not fixed to the user's body itself, and the user may input motions that identify hand and wrist motions. For example, a controller may provide six degrees of freedom (6DoF) that enable motion input in Cartesian and rotational coordinates (e.g., roll, pitch, yaw). In some cases, avatar offsets and / or calibration offsets can be used to identify hand and wrist motions to enable complex nonverbal communication. An example of nonverbal communication is described above with reference to Figure 2.

[0090] In other embodiments, a VR headset may be used in conjunction with motion sensors to identify calibration offsets, motion sensors, etc. For example, a VR headset may include a dot projection device and at least one image sensor to measure distances to various objects in the environment. In one exemplary embodiment, the dot projection may identify the position of at least one sensor attached to the user's body and identify the sensor and its calibration based on its position relative to the VR headset. In another example, the VR headset may present a visual see-through (VST) based user interface (UI) that uses an image sensor to capture an image in the physical world and projects the image onto the wearer using the VR headset, and the UI may facilitate the identification of various sensors.

[0091] In another embodiment, motion in the physical world may be scaled due to differences in the anatomical structure of humans and avatars in the virtual world. In one exemplary example, an avatar may include metadata that identifies motion zones that can be used to calibrate the avatar offset.

[0092] Figure 6A is a 3D view of the user's knees in the physical world while the user is in a seated pose. The user may switch avatars while running an XR multi-user application in a seated pose. In this case, since the new avatar may have different physical properties in the virtual world, the XR multi-user application may be configured to determine the avatar offset based on a stored calibration offset. In one embodiment, the XR multi-user application may use the stored offset to determine the updated knee fixation point of the avatar in a fixed pose (e.g., a T-pose) and calculate a second avatar offset 620. In the example shown in Figure 6A, the user is in a seated position with the knees bent at a 90-degree angle, and the XR multi-user application uses the motion sensor position 604 and the second avatar offset 620 to identify the second knee fixation point 622 based on different avatar dimensions. In effect, the XR multi-user application can map the user's position at the point when the user switches from the first avatar to the second avatar without requiring any calibration.

[0093] Figure 6B is a 3D conceptual diagram of the second avatar 650 in a seated position, representing the offset. The fixed point 630 corresponds to the right knee of the first avatar (not shown) in the calibration pose, and the avatar offset 632 represents the distance from a point associated with the motion sensor (e.g., the calibration offset). The user is seated, and the avatar offset 632 is rotated in 3D space to point 634. When the user chooses to switch to the second avatar 650, the XR multi-user application determines an updated fixed point 640 associated with the right knee of the second avatar 650, and then determines the second avatar offset 642 relative to the knee. Based on the current position of the motion sensor, the XR multi-user application rotates the second avatar offset 642 in 3D space to point 644. Next, the XR multi-user application can draw and display the avatar in the virtual world based on the second avatar offset 642 applied to the current position and the rotation of the motion sensor resulting in point 644.

[0094] Figure 7 shows an exemplary method 700 relating to several aspects of the present disclosure for tracking and calibrating motion sensors attached to a wearer and mapping the wearer's motion to a virtual world. While exemplary method 700 describes a particular sequence of actions, the sequence can be modified without departing from the scope of the present disclosure. For example, some of the actions described may be performed in parallel or in different sequences that do not substantially affect the functionality of method 700. In other examples, different components of an exemplary device or system implementing method 700 may perform functions substantially simultaneously or in a particular sequence.

[0095] Method 700 may be carried out by a software application running on a hardware device, which may also be called an XR device, that provides VR or AR content. For example, the software application may be an XR multi-user application running on a head-mounted device (HMD), such as a VR headset or AR glasses. In other embodiments, the software application may run on a host computer and be provided to the XR device. The software application may also run on a mobile device or on another device, including a computing system (e.g., computing system 1200), which is capable of drawing graphics, displaying graphics, or providing graphics for display on another device. Based on input motion using a controller such as an XR controller, the HMD device may be configured to draw the movements of an avatar in the XR multi-user application and display the motion to the wearer.

[0096] According to some examples, the method is performed by an XR device, which receives data from at least one motion sensor in block 705. As mentioned above, the user of the XR device, which may also be referred to as a person wearing the XR device, may wish to interact in the virtual world using multiple motion sensors, each motion sensor may be attached to a different body part. For example, the user may have motion sensors to track arm motion, chest motion, waist motion, knee motion, and foot motion. In some cases, arm motion may be mapped to shoulder motion, and the user may also have controllers (e.g., VR or XR controllers) held in the user's hands.

[0097] In some examples, an XR device may determine multiple zones associated with the wearer in block 710. In this embodiment, each zone corresponds to the motion of a trackable body part or joint of the wearer. For example, the multiple zones include at least one of a chest zone, a waist zone, a left knee zone, a right knee zone, a left foot zone, a right foot zone, a left elbow zone, and a right elbow zone. The properties of the zones (e.g., the 3D shape of the zones) and the fixed points associated with the zones may be based on the average characteristics of at least one property associated with the human. For example, the multiple zones may be determined by dividing a region in space into multiple zones based on the wearer's height.

[0098] In some embodiments, multiple zones may be determined before motion is received. For example, when an XR device starts running an XR multi-user application, the XR multi-user may identify zones based on previously entered user heights, determined heights of the XR device, etc.

[0099] In block 715, the XR device may map a first motion sensor of at least one motion sensor to a corresponding zone. Mapping can occur at different stages; for example, when an instance of an XR multi-user application is started, the XR multi-user application may need to map sensors attached to the user's body. As mentioned above, some sensors may be attachable to multiple locations on the body. For example, a motion sensor may be attachable to different body parts (e.g., either the knee or the arm) and calibration can be performed to identify the corresponding body part associated with each user. In an exemplary example, the XR multi-user application may require the user to perform a specific pose (e.g., the pose shown in Figure 3) in order to identify and calibrate the sensor.

[0100] When a user inputs a specific pose, the XR device may determine a measured offset in block 720 based on the distance from the position of the corresponding motion sensor in the corresponding zone to a fixed point within that zone. As mentioned above, the fixed point may be the center point, or it may be biased toward the neutral position. For example, referring to Figure 3, the right foot zone 314 has a fixed point biased toward the human midline. When the XR device wearer enters a pose, the XR device can track the motion and identify each sensor within each zone. After the user holds the position for a short time, the XR device can identify a calibration offset from the sensor to the fixed point that can be used for motion tracking. The fixed point may be a joint, or it may be a bone, such as in the upper arm. In some cases, the pose may also be used to calibrate an external controller held by the XR device wearer.

[0101] In some examples, the XR device receives first motion data of the wearer from a first motion sensor in block 725. For example, the wearer may input arm motions or take a step to navigate the avatar in the virtual world. In one embodiment, the motion mapping is based on a second offset associated with the avatar (e.g., avatar offset).

[0102] In some embodiments, the second offset may be determined based on mapping human information to an avatar. For example, an XR device may determine a first calibration offset for a first body part or joint corresponding to a first zone, based on the eye height of the avatar and a first measurement offset. An example of determining the first calibration offset includes aligning the eye height of the avatar to the estimated eye height of the wearer and determining a second offset from a fixed point on the corresponding zone to a center point within the first zone. For example, the second offset may be from a fixed point associated with the human right foot (e.g., ankle) to a fixed point associated with the avatar's right foot.

[0103] Since not all avatar properties correspond to movement in the world, the alignment of assets such as eyes can be important. For example, a significant amount of hair may affect height but not joint mobility in the virtual world. After obtaining the second offset, the XR device may apply the second offset to the first measurement offset to generate the first calibration offset. The XR device can then map the second offset to the corresponding object associated with the avatar. Human zones are determined based on median properties, and the size of the avatar zones may be determined by the designer at design time or by the application at runtime. For example, the avatar designer may determine joint mobility, while in other cases, an XR multi-user application may limit mobility based on joints, collisions, etc.

[0104] After receiving and mapping the motion, in block 735, the XR device may draw an avatar in the virtual world based on the first motion data associated with the first zone.

[0105] In a further embodiment of the method in Figure 7, the XR device may also allow the user to modify their avatar. For example, the XR device may receive input from the wearer of the XR device and modify their avatar. In this case, the XR device may determine a second calibration offset (e.g., an avatar offset) for a first body part or joint corresponding to a first zone based on the eye height and first measurement offset of the second avatar, and then draw the second avatar in the virtual world based on second motion data associated with the first zone. An example of alignment and avatar offset determination based on eye height is shown in Figure 4C, and an example of recalculating the avatar offset based on a changing avatar is shown in Figures 6A and 6B.

[0106] Figure 8 shows an example of a computing system 800, which may be any computing device or any component of an XR system, where the components of the system communicate with each other using connections 805. Connections 805 can be a physical connection via a bus, or a direct connection to a processor 810, such as a chipset architecture. Connections 805 can also be a virtual connection, a network connection, or a logical connection.

[0107] In some embodiments, the computing system 800 is a distributed system in which the functions described herein may be distributed across a data center, a plurality of data centers, a peer network, etc. In some embodiments, one or more of the described system components represent many such components, each performing some or all of the functions described for that component. In some embodiments, the components may be physical or virtual devices.

[0108] An exemplary system 800 includes at least one processing unit (CPU or processor) 810 and connections 805 that connect various system components to the processor 810, including system memory 815 such as read-only memory (ROM) 820 and random access memory (RAM) 825. The computing system 800 may include a cache of high-speed memory 812 that is directly connected to, in close proximity to, or integrated as part of the processor 810.

[0109] The processor 810 may include an arbitrary general-purpose processor, hardware or software services such as services 832, 834, and 836 stored in a storage device 830 and configured to control the processor 810, and an application-specific processor in which software instructions are incorporated into the actual processor design. The processor 810 may be a fully self-contained computing system including a multicore or processor, bus, memory controller, cache, etc. The multicore processor may be symmetric or asymmetric.

[0110] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gestures and graphical input, a keyboard, a mouse, motion input, and voice. The computing system 800 may also include an output device 835 that can be one or more of a number of output mechanisms known to those skilled in the art. In some examples, a multimodal system can be made to allow a user to provide multiple types of inputs / outputs for communication with the computing system 800. The computing system 800 may generally include a communication interface 840 that can control and manage user inputs and system outputs. There are no restrictions on operating on any particular hardware configuration, and therefore the basic features of this specification can be easily replaced by improved hardware or firmware configurations as they are developed.

[0111] The storage device 830 can be a non-volatile memory device, such as a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital general-purpose disk, cartridge, random-access memory (RAM), read-only memory (ROM), and / or some combination of these devices.

[0112] The storage device 830 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by the processor 810. In some embodiments, a hardware service that performs a particular function may include software components stored on a computer-readable medium in relation to necessary hardware components such as the processor 810, connection 805, and output device 835 in order to perform that function.

[0113] To clarify the explanation, in some examples, the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0114] Any step, operation, function, or process described herein may be performed or implemented by hardware and software services, or a combination of services, either alone or in combination with other devices. In some embodiments, a service may be software residing in the memory of a client device and / or one or more servers of a content management system, and may perform one or more functions when the processor executes the software associated with the service. In some embodiments, a service may be a program or a set of programs that perform a particular function. In some embodiments, a service may be considered a server. Memory may be a non-temporary computer-readable medium.

[0115] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0116] The methods described in the examples above may be implemented using computer-executable instructions stored on or available from computer-readable media. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or set of functions, or that constitute such functions. Some of the computer resources used may be accessible via a network. Executable computer instructions may be, for example, binaries, intermediate form instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods described in the examples include magnetic or optical disks, solid-state memory devices, flash memory, USB devices with non-volatile memory, and networked storage devices.

[0117] Devices implementing the methods described herein may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and the like. The functions described herein may also be embodied in peripherals or expansion cards. As a further example, such functionality may also be implemented on a circuit board across different chips or across different processes performed in a single device.

[0118] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0119] While various examples and other information have been used to illustrate aspects within the scope of the accompanying claims, a person skilled in the art will be able to derive a wide variety of implementations from these examples, and therefore, the claims should not be limited based on any particular features or configurations in such examples. Furthermore, while some subject matter may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the accompanying claims is not necessarily limited to these described features or actions. For example, such functions may be distributed in different ways and may be performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the accompanying claims.

[0120] To clarify the explanation, in some examples, the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0121] Any step, operation, function, or process described herein may be performed or implemented by hardware and software services, or a combination of services, either alone or in combination with other devices. In some embodiments, a service may be software residing in the memory of a client device and / or one or more servers of a content management system, and may perform one or more functions when the processor executes the software associated with the service. In some embodiments, a service may be a program or a set of programs that perform a particular function. In some embodiments, a service may be considered a server. Memory may be a non-temporary computer-readable medium.

[0122] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0123] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0124] While various examples and other information have been used to illustrate aspects within the scope of the accompanying claims, those skilled in the art will be able to derive a wide variety of implementations from these examples, and therefore, the claims should not be limited based on any particular features or configurations in such examples. Furthermore, while some subject matter may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the accompanying claims is not necessarily limited to these described features or actions. For example, such functions may be distributed in different ways and may be performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the accompanying claims.

[0125] Exemplary aspects of this disclosure include:

[0126] Embodiment 1. The method includes receiving data from at least one motion sensor, each of which motion sensors is attached to a different body part of the wearer to track the wearer's movements; determining a plurality of zones associated with the wearer, each of which zones corresponds to the motion of a trackable body part or joint of the wearer; mapping a first motion sensor of the at least one motion sensor to a corresponding zone, the first motion sensor corresponding to a first zone; determining a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone; receiving first motion data of the wearer from the first motion sensor; mapping the first motion data to an avatar in a virtual world based on the first measurement offset; and drawing the avatar in the virtual world based on the first motion data associated with the first zone.

[0127] Embodiment 2. The method of Embodiment 1, wherein determining the plurality of zones comprises dividing a region in space into the plurality of zones based on the wearer's height, the plurality of zones being determined based on median characteristics associated with the person.

[0128] Embodiment 3. A method in any of Embodiments 1 to 2, wherein each of the at least one motion sensors is attached to the wearer.

[0129] Embodiment 4. A method of any of Embodiments 1 to 3, wherein the plurality of zones include at least one of a chest zone, a waist zone, a left knee zone, a right knee zone, a left foot zone, a right foot zone, a left elbow zone, and a right elbow zone.

[0130] Embodiment 5. A method according to any of Embodiments 1 to 4, further comprising determining a first calibration offset for a first body part or joint corresponding to the first zone, based on the eye height of the avatar and the first measurement offset.

[0131] Embodiment 6. A method according to any of Embodiments 1 to 5, wherein determining the first calibration offset includes aligning the eye height of the avatar to the estimated or measured eye height of the wearer, and determining a second offset from a fixed point in a corresponding zone to the fixed point in the first zone, wherein the second offset is applied to the first measured offset to obtain the first calibration offset.

[0132] Embodiment 7. A method according to any of Embodiments 1 to 6, wherein mapping the first motion data corresponding to the first zone to the avatar in the virtual world based on the first measurement offset includes mapping the second offset to a corresponding object associated with the avatar.

[0133] Embodiment 8. A method according to any of Embodiments 1 to 7, wherein mapping the first motion data to the avatar in the virtual world includes scaling the motion applied to the avatar based on metadata associated with the avatar.

[0134] Embodiment 9. A method according to any of Embodiments 1 to 8, comprising: receiving an input for changing from the avatar to a second avatar; determining a second calibration offset for the first body part or joint corresponding to the first zone based on the eye height of the second avatar and the first measurement offset; and rendering the second avatar in the virtual world based on second motion data associated with the first zone. It also includes.

[0135] Embodiment 10. The device includes a storage (implemented in the circuit) and a processor configured to store instructions. The processor is configured to execute the instructions and cause the processor to receive data from at least one motion sensor, where each of the at least one motion sensor is attached to a different body part of the wearer to track the wearer's movements and to determine a plurality of zones associated with the wearer, where each zone corresponds to the motion of a trackable body part or joint of the wearer, and to map a first motion sensor of the at least one motion sensor to the corresponding zone, where the first motion sensor corresponds to the first zone, and to determine a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone, and to receive first motion data of the wearer from the first motion sensor, and to map the first motion data to an avatar in a virtual world based on the first measurement offset, and to draw the avatar in the virtual world based on the first motion data associated with the first zone.

[0136] Embodiment 11. The apparatus of Embodiment 10, wherein determining the plurality of zones comprises dividing a region in space into the plurality of zones based on the wearer's height, the plurality of zones being determined based on median characteristics associated with the person.

[0137] Embodiment 12. The apparatus according to any of Embodiments 10 to 11, wherein each of the at least one motion sensors is attached to the wearer.

[0138] Embodiment 13. An apparatus according to any of Embodiments 10 to 12, wherein the plurality of zones include at least one of a chest zone, a waist zone, a left knee zone, a right knee zone, a left foot zone, a right foot zone, a left elbow zone, and a right elbow zone.

[0139] Embodiment 14. An apparatus according to any of Embodiments 10 to 13, wherein the processor is configured to execute the instruction, and causes the processor to determine a first calibration offset for a first body part or joint corresponding to the first zone, based on the height of the avatar's eyes and the first measurement offset.

[0140] Embodiment 15. An apparatus according to any of Embodiments 10 to 14, wherein the processor is configured to execute the instructions, and the processor determines the first calibration offset, which causes the eye height of the avatar to be aligned with the wearer's estimated or measured eye height, and determines a second offset from a fixed point in a corresponding zone to the fixed point in the first zone, the second offset being applied to the first measured offset to obtain the first calibration offset.

[0141] Embodiment 16. An apparatus according to any of Embodiments 10 to 15, wherein mapping the first motion data corresponding to a first zone to the avatar in the virtual world based on the first measurement offset includes mapping the second offset to a corresponding object associated with the avatar.

[0142] Embodiment 17. An apparatus according to any of Embodiments 10 to 16, wherein motion scaling is applied to the avatar based on metadata associated with the avatar.

[0143] Embodiment 18. An apparatus according to any of Embodiments 10 to 17, wherein the processor is configured to execute the instructions, and the processor receives an input to change from the avatar to a second avatar, determines a second calibration offset for the first body part or joint corresponding to the first zone based on the eye height of the second avatar and the first measurement offset, and draws the second avatar in the virtual world based on second motion data associated with the first zone.

[0144] Embodiment 19. A computer-readable medium containing instructions for using a computer system. The computer includes memory (for example, implemented by circuitry) and a processor (or a number of processors) coupled to the memory. The processor (or multiple processors) is configured to execute the computer-readable medium and cause the processor to receive data from at least one motion sensor, where each of the at least one motion sensors is attached to a different body part of the wearer to track the wearer's movements and to determine a plurality of zones associated with the wearer, where each zone corresponds to the motion of a trackable body part or joint of the wearer, and to map the first motion sensor of the at least one motion sensor to the corresponding zone, where the first motion sensor corresponds to the first zone, and to determine a first measurement offset based on the distance from the position of the first motion sensor in the first zone to a fixed point in the first zone, receive first motion data of the wearer from the first motion sensor, map the first motion data to an avatar in a virtual world based on the first measurement offset, and draw the avatar in the virtual world based on the first motion data associated with the first zone.

[0145] Embodiment 20. A computer-readable medium according to Embodiment 19, wherein determining the plurality of zones comprises dividing a region in space into the plurality of zones based on the wearer's height, the plurality of zones being determined based on a median characteristic associated with the person.

[0146] Embodiment 21. A computer-readable medium according to any of Embodiments 19 to 20, wherein each of the at least one motion sensors is attached to the wearer.

[0147] Embodiment 22. A computer-readable medium according to any of Embodiments 19 to 21, wherein the plurality of zones include at least one of a chest zone, a waist zone, a left knee zone, a right knee zone, a left foot zone, a right foot zone, a left elbow zone, and a right elbow zone.

[0148] Embodiment 23. A computer-readable medium according to any of Embodiments 19 to 22, wherein the processor is configured to execute the computer-readable medium, and the processor causes the processor to determine a first calibration offset for a first body part or joint corresponding to the first zone, based on the eye height of the avatar and the first measurement offset.

[0149] Embodiment 24. A computer-readable medium according to any of Embodiments 19 to 23, wherein the processor is configured to execute the computer-readable medium, and the processor causes the eye height of the avatar to be aligned with the estimated or measured eye height of the wearer, and to determine a second offset from a fixed point in a corresponding zone to a fixed point in a first zone, wherein the second offset is applied to the first measurement offset to obtain the first calibration offset.

[0150] Embodiment 25. A computer-readable medium according to any of Embodiments 19 to 24, wherein mapping the first motion data corresponding to the first zone to the avatar in the virtual world based on the first measurement offset includes mapping the second offset to a corresponding object associated with the avatar.

[0151] Embodiment 26. A computer-readable medium according to any of Embodiments 19 to 25, wherein motion scaling is applied to the avatar based on metadata associated with the avatar.

[0152] Embodiment 27. A computer-readable medium according to any of Embodiments 19 to 26, wherein the processor is configured to execute the computer-readable medium, and the processor receives input to change from the avatar to a second avatar, determines a second calibration offset for the first body part or joint corresponding to the first zone based on the eye height of the second avatar and the first measurement offset, and draws the second avatar in the virtual world based on second motion data associated with the first zone.

Claims

1. Receiving data from at least one motion sensor, wherein each of the at least one motion sensors is attached to a different body part of the wearer to track the wearer's movements, Determining a plurality of zones associated with the wearer, wherein each zone corresponds to the motion of a trackable body part or joint of the wearer, Mapping the first motion sensor of the at least one motion sensor to a corresponding zone, wherein the first motion sensor corresponds to the first zone, The first measurement offset is determined based on the distance from the position of the first motion sensor within the first zone to the fixed point within the first zone, Receiving the wearer's first motion data from the first motion sensor, Based on the first measurement offset, the first motion data is mapped to an avatar in the virtual world, Drawing the avatar in the virtual world based on the first motion data associated with the first zone, including, A method characterized by the following:

2. Determining the plurality of zones includes dividing the area within the space into the plurality of zones based on the wearer's height. The aforementioned multiple zones are determined based on the median characteristics associated with the human being. The method according to feature 1.

3. Each of the at least one motion sensors is attached to the wearer. The method according to feature 1.

4. The aforementioned plurality of zones include at least one of the following: chest zone, waist zone, left knee zone, right knee zone, left foot zone, right foot zone, left elbow zone, and right elbow zone. The method according to feature 1.

5. The process further includes determining a first calibration offset for a first body part or joint corresponding to the first zone, based on the eye height of the avatar and the first measurement offset. The method according to feature 1.

6. Determining the first calibration offset means To align the eye height of the aforementioned pockmark with the estimated or measured eye height of the wearer, Determining a second offset from a fixed point in the corresponding zone to the fixed point in the first zone, wherein the second offset is applied to the first measurement offset to obtain the first calibration offset, including, The method according to specification 5.

7. Mapping the first motion data corresponding to the first zone to the avatar in the virtual world based on the first measurement offset includes mapping the second offset to the corresponding object associated with the avatar. The method according to feature 6.

8. Receiving input to change from the aforementioned avatar to a second avatar, Based on the eye height of the second avatar and the first measurement offset, a second calibration offset is determined for the first body part or joint corresponding to the first zone, Based on the second motion data associated with the first zone, the second avatar in the virtual world is rendered, Further including, The method according to specification 5.

9. Mapping the first motion data to the avatar in the virtual world is This includes scaling the motion applied to the avatar based on the metadata associated with the avatar. The method according to feature 6.

10. Storage configured to store instructions, A processor configured to execute the aforementioned instructions and cause the processor to perform the following: Receiving data from at least one motion sensor, wherein each of the at least one motion sensors is attached to a different body part of the wearer to track the wearer's movements, Determining a plurality of zones associated with the wearer, wherein each zone corresponds to the motion of a trackable body part or joint of the wearer, Mapping the first motion sensor of the at least one motion sensor to a corresponding zone, wherein the first motion sensor corresponds to the first zone, The first measurement offset is determined based on the distance from the position of the first motion sensor within the first zone to the fixed point within the first zone, Receiving the wearer's first motion data from the first motion sensor, Based on the first measurement offset, the first motion data is mapped to an avatar in the virtual world, Drawing the avatar in the virtual world based on the first motion data associated with the first zone, Having, A system characterized by the following features.

11. Determining the plurality of zones includes dividing the area within the space into the plurality of zones based on the wearer's height. The aforementioned multiple zones are determined based on the median characteristics associated with the human being. The system according to feature 10.

12. Each of the at least one motion sensors is attached to the wearer. The system according to feature 10.

13. The aforementioned plurality of zones include at least one of the following: chest zone, waist zone, left knee zone, right knee zone, left foot zone, right foot zone, left elbow zone, and right elbow zone. The system according to feature 10.

14. The aforementioned processor is configured to execute the aforementioned instructions and cause the processor to do the following: Based on the eye height of the avatar and the first measurement offset, a first calibration offset is determined for a first body part or joint corresponding to the first zone. The system according to feature 10.

15. The aforementioned processor is configured to execute the aforementioned instructions and cause the processor to do the following: The eye height of the aforementioned pockmark is aligned to the estimated or measured eye height of the wearer, Determining a second offset from a fixed point in the corresponding zone to the fixed point in the first zone, wherein the second offset is applied to the first measurement offset to obtain the first calibration offset. The system according to feature 14.

16. Mapping the first motion data corresponding to the first zone to the avatar in the virtual world based on the first measurement offset includes mapping the second offset to the corresponding object associated with the avatar. The system according to claim 15, characterized in that it is the same as described above.

17. The aforementioned processor is configured to execute the aforementioned instructions and cause the processor to do the following: Based on the metadata associated with the avatar, the motion applied to the avatar is scaled. The system according to claim 15, characterized in that it is the same as described above.

18. The aforementioned processor is configured to execute the aforementioned instructions and cause the processor to do the following: The system receives an input to change from the first avatar to the second avatar, Based on the eye height of the second avatar and the first measurement offset, a second calibration offset is determined for the first body part or joint corresponding to the first zone. Based on the second motion data associated with the first zone, the second avatar in the virtual world is rendered. The system according to feature 14.

19. A non-temporary computer-readable medium containing instructions, When the aforementioned instruction is executed by a computing system, it causes the computing system to do the following: Receiving data from at least one motion sensor, wherein each of the at least one motion sensors is attached to a different body part of the wearer to track the wearer's movements, Determining a plurality of zones associated with the wearer, wherein each zone corresponds to the motion of a trackable body part or joint of the wearer, Mapping the first motion sensor of the at least one motion sensor to a corresponding zone, wherein the first motion sensor corresponds to the first zone, The first measurement offset is determined based on the distance from the position of the first motion sensor within the first zone to the fixed point within the first zone, Receiving the wearer's first motion data from the first motion sensor, Based on the first measurement offset, the first motion data is mapped to an avatar in the virtual world, Drawing the avatar in the virtual world based on the first motion data associated with the first zone, Having, A computer-readable medium characterized by the following:

20. Determining the plurality of zones includes dividing the area within the space into the plurality of zones based on the wearer's height. The aforementioned multiple zones are determined based on the median characteristics associated with the human being. The computer-readable medium according to feature 19.

Images