Locomotion using finger tracking

Abstract

The technology provides a mechanism by which a player can make gestures with their hands and invoke virtual controllers, and the tracking of hand movements and fingers can effect movement of an avatar through a virtual world without the need for a hardware controller. Thus, a player can have natural translations or otherwise control the avatar's hands to interact with other objects in the virtual world, and can then use the avatar's hands to invoke virtual controllers to control the movement of the avatar without the player having a physical controller.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Patent Application No. 18 / 167,486, entitled "LOCOMOTION USING FINGER TRACKING," filed February 10, 2023, which is expressly incorporated by reference herein in its entirety for all purposes. [Background technology]

[0002] Virtual reality (VR) provides an immersive virtual world environment in which a player often experiences the virtual world from a first-person perspective. In many ways, a player experiencing a virtual world from a first-person perspective feels as if they are actually present in the virtual world, except of course for the equipment they use to access the virtual world and control their avatar. Players often access the virtual world through a VR headset combined with handheld controllers. Sometimes, players also utilize additional skeletal tracking and haptic feedback devices. While equipment can provide additional inputs to support more realistic movement by an avatar, the equipment can also limit how immersed a player can be in the virtual world. [Brief explanation of the drawings]

[0003] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which that element is first introduced.

[0004] [Figure 1] FIG. 1 illustrates an exemplary virtual world platform for playing and hosting multiplayer virtual reality (VR) experiences in accordance with some aspects of the present technology.

[0005] [Figure 2]FIG. 2 illustrates an exemplary quick menu in accordance with some aspects of the present technology.

[0006] [Figure 3] FIG. 3 illustrates an example relationship between gestures and virtual controller instantiations and manipulations in accordance with some aspects of the present technology.

[0007] [Figure 4A] FIG. 4A shows an example routine for controlling avatar movement through finger tracking in accordance with some aspects of the present technology.

[0008] [Figure 4B] FIG. 4B illustrates an exemplary routine for controlling avatar movement through finger tracking in accordance with some aspects of the present technology.

[0009] [Figure 4C] FIG. 4C illustrates an exemplary routine for controlling avatar movement through finger tracking in accordance with some aspects of the present technology.

[0010] [Figure 5] FIG. 5 is a diagram illustrating the relationship between a player in a room space and an avatar in a world space, in accordance with some aspects of the present technology.

[0011] [Figure 6] FIG. 6 illustrates an example of an avatar performing a pinching gesture with its left hand to instantiate a virtual controller, in accordance with some aspects of the present technology.

[0012] [Figure 7] FIG. 7 illustrates an example of using a virtual controller to make an avatar walk forward, in accordance with some aspects of the present technology.

[0013] [Figure 8]FIG. 8 illustrates an example of using a virtual controller to rotate an avatar left, in accordance with some aspects of the present technology.

[0014] [Figure 9] FIG. 9 illustrates an example of using a virtual controller to rotate an avatar to the right, in accordance with some aspects of the present technology.

[0015] [Figure 10] FIG. 10 illustrates an avatar performing a pinching gesture with its left and right hands in accordance with some aspects of the present technology.

[0016] [Figure 11] FIG. 11 illustrates an example of using a left-hand virtual controller and a right-hand virtual controller to make an avatar walk forward and turn right, in accordance with some aspects of the present technology.

[0017] [Figure 12] FIG. 12 illustrates an exemplary routine for controlling avatar movement through finger tracking in accordance with some aspects of the present technology.

[0018] [Figure 13] FIG. 13 illustrates an example of a system for implementing certain aspects of the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0019] Some client devices for interacting in virtual reality include hand tracking capabilities. Hand tracking has been used to control the limb movements of avatars in virtual reality environments. For example, in virtual worlds where players, represented by their avatars, are conversing, it can be useful to show the avatar's hands moving, since many people "talk with their hands" by making various gestures in sync with their speech. Of course, participants in a conversation in which at least one member is deaf may speak using sign language.

[0020] While hand tracking creates a more realistic conversational experience in virtual reality, it has the drawback that most players must let go of any controller that may normally be used for movement (the player provides input to the controller to navigate their avatar through the virtual world). This leads to the problem that a player who lets go of the controller must find the controller before they can navigate their avatar throughout the virtual world, which is a cumbersome user experience.

[0021] The experience can even become more complicated if the client (VR headset) needs to detect the controllers again and remap the controllers in physical space (also called room space) to the virtual world, which can take some time.

[0022] While virtual reality offers a fully immersive experience, the need to interact with a controller can sometimes detract from the virtual reality experience. Therefore, it is necessary to be able to navigate a virtual reality world without a controller, and to allow players to seamlessly transition from conversation to movement.

[0023] The technology provides a mechanism by which a player can make gestures with their hands and invoke virtual controllers, and the tracking of hand movements and fingers can effect movement of an avatar through a virtual world without the need for a hardware controller. Thus, a player can have natural transformations or otherwise control the hands of their avatar to interact with other objects in the virtual world, and can then use their hands to invoke virtual controllers to control the movement behavior of their avatar, without the player having a physical controller.

[0024] The technology also addresses the complexity of controlling the movement of an avatar in a virtual world through hand and finger tracking of the player's hands in room space. The complexity is that while the avatar changes position in the virtual world, the player does not change his or her position in room space; i.e., the player only moves his or her hands. Therefore, if the avatar changes position but the player does not, the player's hands may be mapped to a position other than the avatar's. This may give the impression that the avatar has left the virtual controller behind.

[0025] This phenomenon may be a result of the fact that the client device can report the position of the fingers making the gesture, which fingers are involved in the gesture, a confidence value that the detected gesture is correctly identified, the hand position, the hand rotation, etc. A virtual controller is instantiated in the virtual world at the corresponding position, but the avatar quickly moves from that position.

[0026] To overcome this complexity, the present technology can change the position of the joystick to match the movement of the avatar. This can become complicated when the avatar is rotating and moving in multiple directions at the same time, such as when the avatar is moving sideways and rotating at the same time. An additional level of complexity arises not only from changing the position of the virtual controller, but also from maintaining the position of the virtual controller relative to the avatar in a position that provides a good player experience.

[0027] 1 illustrates an exemplary virtual world platform 102 for playing and hosting multiplayer virtual reality (VR) experiences suitable for implementing the present technology. The virtual world platform 102 connects clients 104 via web services 110 and networking services 112, allowing them to interact socially together in virtual worlds hosted by the virtual world platform 102.

[0028] The virtual world platform 102 primarily includes clients 104, which are instances of applications running on client devices 106. The clients 104 interact via a network connection with web services 110, which support the clients 104 by providing various services through one or more application programming interfaces (APIs). Some of the primary services provided by web services 110 relate to virtual world support through world API 128, user profiles through user API 132, trust and security through trust API 144, and complex avatars through avatar API 136. Web services 110 generally store and provide long-term state information, among other functions.

[0029] The client 104 also interacts with a networking service 112, which provides communication services between the client 104, the networking service 112, and remote instances of the client 104 (not shown) and shares state information between the respective instances of the client 104. In particular, state information is received by the networking service 112 from multiple instances of the client 104 when each instance of the client 104 controls its local player 116. The networking service 112 can forward state information about each player to other instances of the client 104 when the local players 116 of each client instance are all engaged in gameplay in the same virtual world. The networking service 112 provides optimized packet routing through an optimized packet routing service 140 and provides moderation between one or more clients through a moderation service 142.

[0030] A client 104 is a runtime environment that executes on a particular client device 106. This specification may refer to a client 104, a local client, and a remote client, all of which are instances of a client 104 executing on a respective client device 106. One particular user account is logged into a particular instance of a client 104. The local client and remote client are distinguished to illustrate how a client 104 handles first-person input from a player on the client device 106 on which the client 104 is executing and how a remote client handles third-person input received from another player operating a client device on which the remote client is executing.

[0031] The client device 106 can be any computing device. The client 104 is particularly adapted to provide an immersive virtual reality experience through interactions that require a VR headset to be experienced, but the client 104 can also be executed by computers and mobile devices. Some virtual worlds or complex avatars may not be configured to work well with particular device types, and therefore, although the client 104 is operable on many platforms and devices, not all virtual worlds or complex avatars may be available or have full functionality on all client devices 106.

[0032] The user interface service 108 is one service that is part of the client 104. The user interface service 108 is configured to provide various user interface elements, such as menus that display various player settings, available worlds, saved complex avatars, friends lists, etc. The user interface service 108 can populate its menus through interactions with one or more APIs provided by the web services 110, while other portions of the menus are loaded directly from the user interface service 108.

[0033] The user interface service 108 can provide a menu of available worlds by calling the world API 128 to retrieve a list of worlds that the user account logged into the client 104 is authorized to enter. The world API 128 can retrieve all public worlds from the world asset database 130 and send the list to the client 104. Additionally, the world API 128 can request the world IDs of any private worlds associated with the user account logged into the client 104, retrieve the private worlds from the world asset database 130, and send them to the client 104. The user interface service 108 can receive player input through the hardware interface to navigate through the world menu and receive a selection of worlds to visit.

[0034] Another user interface provided by user interface services 108 relates to various player settings, such as whether a human player is sitting or standing, settings for minimizing motion sickness for players prone to motion sickness when playing in VR, settings for selecting complex avatars, settings related to how the player can be seen in the virtual world and by whom the player can be seen, etc.

[0035] One notable user interface provided by the user interface services 108 is the Trust and Safety menu. The user interface services 108 can contact the user API 132 to retrieve current trust and safety settings from the user profile database 134 and display these settings in the Trust and Safety menu. The Trust and Safety menu provides a user account with the ability to determine which remote players 124 can see the player's avatar (the local player 116) or can be seen by the player's avatar when both are in the same world. For example, it may be desirable to avoid interacting with newer players in the virtual world platform 102 because the newer players have not yet established a trust relationship within the virtual world platform 102. It may also be desirable to limit the functionality of remote players' avatars handled by the instance of the client 104 to which the local player is logged in. This is because some avatars may have malicious data embedded in them or may be too complex to render without degrading the performance of the client device 106. For example, a user account may decide to turn off lights on a remote avatar to avoid shaders, not allow custom animations, etc. In some embodiments, each of these options may be set based on how much the remote player is trusted. For example, a user account may allow a friend's avatar to have a full appearance, while other user accounts may only display a basic avatar appearance.

[0036] The user interface service 108 may also provide options to mute or block specific remote players. Additionally, the user interface service 108 may provide a panic mode that audio and visually mutes everyone except friends.

[0037] After a player selects a virtual world from a menu provided by user interface services 108, client 104 can download an instance of the virtual world by calling world API 128, which can retrieve the virtual world from world asset database 130 and send it to client 104 for execution.

[0038] World assets are large binary files built for game engines such as UNITY using an editor with a software development kit (SDK) provided for use with the virtual world platform 102. When a player moves into a world, the player needs to download the world assets from the world asset database 130. If there are already people in that instance of that world, the client 104 also needs a list of those people's avatars so that it can render them in the instance of the virtual world.

[0039] In some embodiments, functionality in the worlds API 128 can verify that a user account has access to a requested world. While a user account should only have the ability to see public worlds in a user interface menu or only have knowledge of links to worlds shared with the user account, the worlds API 128 can verify that a user account is authorized to access a virtual world as a redundancy measure.

[0040] In addition to downloading an instance of a virtual world, the client 104 can also establish a session with the networking service 112 for a particular instance of the world. The networking service 112 can provide information about the current state of the instance of the virtual world. For example, the networking service 112 can provide the client 104 with a list of remote avatars 126 that exist within the virtual world instance. The client 104 can then contact the avatar API 136 to download complex avatar assets for the list of remote complex avatars from the avatar asset database 138.

[0041] If the client 104 does not have assets for the local avatar 118, the client 104 can also contact the avatar API 136 to request and receive local avatar assets. An avatar asset is a single binary file that contains all of the texture, model, and animation data needed to render the avatar. In some instances, more complex features can be included, such as data about particle systems and lighting, whether the avatar should obey or defy the laws of physics established in the virtual world, and whether the avatar has non-standard movement dynamics.

[0042] The downloaded instance of the virtual world may be executed by the client 104 as the current world 120. The current world 120 may include the coordinates within the current world 120 where the local player 116 and each remote player 124 are located, as well as the respective collision volumes of the space occupied by the local player 116 or remote player 124, respectively.

[0043] A local avatar 118 can be mapped to a local player 116, and each remote avatar 126 can be mapped to a respective remote player 124, thereby allowing each player to appear as their own avatar in the current world 120. The movements of the remote avatars 126 are manipulated by the client 104 by receiving state data for each remote avatar / player and rendering the movements or sounds.

[0044] The VR tracking service 114 relates to a client 104 running on a client device 106 that has access to VR tracking peripherals. For example, some VR headsets have cameras (integrated or external) for tracking a player's limbs. Many VR headsets can be paired with controllers that can report the position of the player's hands in space. Some client devices 106 include other peripherals configured to perform full skeletal tracking. The VR tracking service 114 can fuse all VR inputs connected to the client.

[0045] The VR tracking service 114 can map the blended VR input to the local player 116, allowing the local player 116 to interact in and with the current world 120. Meanwhile, the local player 116 can interact with a local avatar 118, mapping the local avatar 118 to the local player and causing the local player 116 to appear as their avatar.

[0046] In some embodiments, which parts of a player's body are tracked by the VR tracking service 114 vary. Some players may have full skeleton tracking, while many players may only have the capability to perform hand tracking. To accommodate these differences in the hardware capabilities of possible client devices 106, the local player 116 can derive parts of the skeleton that are not tracked by the VR tracking service 114. For example, even if the VR tracking service 114 only provides information about the tracking of a player's hands, the local player can still derive the player's complete skeleton and move parts of the skeleton to accommodate hand movements. In this way, an avatar's hands do not move in isolation from the rest of the avatar.

[0047] The local player 116 is an entity that moves around the environment in the current world 120. It can pick up and place things. It does not have any animations and is a collision volume. It can do anything in the world, but it has no appearance and does not need to animate.

[0048] The local player is further connected to a networking layer, shown as runtime networking services 122, which broadcasts state information about the local player 116 to other players within the current instance of world 120 over the network.

[0049] The local player 116 and the remote player 124 are similar in that they are collision volumes that move around within the environment of the current world 120. The main difference is that the local player 116 is controlled by the client 104, and the player at the client 104 is authoring the experience. In contrast, the remote player 124 is a playback mechanism that represents the actions broadcast to the client 104, which represents other players present in the current world 120.

[0050] As described above, the local avatar 118 is overlaid with the local player 116 to give the player a visual appearance. Actions by the local player 116 are animated as the local player interacts with the current world. For example, the local player 116 can interact with the current world 120 to pick up an object, but without the local avatar 118, the object would appear to be floating in the air. When the local avatar 118 is overlaid on the local player 116, the object appears to be held by the avatar's hand.

[0051] Remote player 124 and remote avatar 126 function similarly to their local counterparts, except for where the input that controls remote player 124 comes from. Remote player 124 and remote avatar 126 are playback devices for state information received by runtime networking services 122 from networking services 112. While Figure 1 shows only one remote player 124 and remote avatar 126, there may be many.

[0052] The current world 120 also has functionality that requires networking: the current world 120 may have objects, such as scissors or light switches, that the player can pick up, and the objects need to broadcast their state over the network so that other players in the current world 120 can see the object's current state.

[0053] The local player 116, the current world 120, and the remote player 124 are each connected to the runtime networking services 122. The local player 116 primarily sends updated state information for the local player 116 to remote instances of clients 104 that are also running the same virtual world. The current world 120 can send and receive state information about instances of the virtual world. The current world running on the client 104 sends state information when a state change is owned by the local player 116 and receives state information when a state change is owned by the remote player 124.

[0054] Networking services 112 are the network-side portion of the network layer of virtual world platform 102. In some embodiments, portions of networking services 112 are provided by a networking plug-in, such as the PHOTON networking engine, that broadcasts state information to all players within an instance of a virtual world.

[0055] In addition to general broadcasting of state information to all players interacting with an instance of a virtual world, the optimized packet routing service 140 provides more advanced functionality to provide an enhanced player experience and enforce other virtual world platform 102 characteristics such as trust and safety settings.

[0056] For example, to provide an enhanced player experience, the optimized packet routing service 140 can filter audio packets coming from remote players 124 that may be far away from the local player 116 in the current instance of the world 120. Without such optimization, a remote player 124 that is not interacting with or even visible to the local player may receive audio packets from dozens or even hundreds of remote players 124, making it difficult to communicate with any subset of the remote players 124.

[0057] In another example, the optimized packet routing service 140 can implement trust and safety settings. As described above, the trust and safety settings can specify specific user accounts or groups of user accounts that cannot interact with the local player 116 or that have limited interaction with the local player 116. The optimized packet routing service 140 can call the trust API 144 to learn a list of remote players 124 that may need to be subject to filtering or blocking of some range of network traffic to or from the client 104 for a local player 116 with trust and safety settings.

[0058] The trust API 144 can determine which remote players 124 should be blocked from the local player 116 or which remote players 124 should have aspects of their complex avatars limited. Some of these decisions are based on logic and rules that categorize remote players 124 based on the amount and type of past interactions with the virtual world platform 102. The trust API 144 can make these decisions by using settings stored in the user profile of the local player 116 and comparing these settings to data stored in the user profile of the remote player 124.

[0059] Another networking service 112 is a moderation service 142 that can provide conflict resolution and access control. For example, before a player can access a world, particularly a private world, the moderation service 142 can call the world API 128 to ensure that the player can enter the world. In another example, two different players may attempt to claim control of an object in a virtual world at approximately the same time. The moderation service 142 can handle these types of conflicts by selecting a specific player to control the object until the player relinquishes control of the object, allowing another player to claim control of the object. A player with control of an object can broadcast a packet informing remote players 124 of the object's state.

[0060] In some embodiments, the client 104, virtual world, and complex avatar may be configured to run in a particular game engine, particularly one that supports three-dimensional (3D) environments. Two common game engines include UNITY and UNREAL ENGINE.

[0061] In some embodiments, to be supported by the virtual world platform 102, virtual worlds and complex avatars must be developed in accordance with a software development kit (SDK). For example, a complex avatar requires specific scripts to be usable by the virtual world platform 102. In another example, there may be numerous requirements that must be met to play the avatar's animations. In some embodiments, the SDK may define other necessary details to support a particular client device. For example, the SDK may define specific shaders to be used when the avatar is used with an OCULUS QUEST VR headset.

[0062] In some embodiments, the SDK requires that virtual worlds utilize a specific coding language to ensure that the worlds have compliant behavior. For example, the SDK may require that behavior within a world be defined using UDON, a programming language specific to a particular virtual world platform 102, VRCHAT. In some embodiments, the programming language facilitates worlds built using the programming language to comply with file access protections provided by the virtual world platform 102. For example, a world cannot read or write anything to a hard drive, and only approved web pages can be rendered in a world on the virtual world platform 102.

[0063] In some embodiments, virtual world platform 102 may also include simplified avatar service 146. As described herein, simplified avatar service 146 may create simplified versions of complex avatars and store avatar assets for the simplified versions of the complex avatars in avatar asset database 138.

[0064] Although the virtual world platform 102 is suitable for implementing the present techniques, those skilled in the art will appreciate that the present techniques can be used in other environments.

[0065] 2 illustrates an exemplary quick menu 202 in accordance with some aspects of the present technology. In particular, the quick menu 202 may be surfaced at any time or place in the virtual world platform 102 by the user interface service 108 on the client 104.

[0066] The quick menu 202 includes a quick links 204 section that contains many commonly used menu options, such as menus for browsing worlds, avatars, and friends, and a safety menu 208 for setting safety settings for the user's profile.

[0067] The Trust and Security menu 208 provides a user account with the ability to determine which remote players 124 can see the user's avatar (local player 116) or be seen by the player's avatar when both are in the same world. For example, it may be desirable to avoid interacting with newer players in the virtual world platform 102 because the newer players have not yet established a trust relationship within the virtual world platform 102. It may also be desirable to limit the functionality of remote players' avatars handled by the instance of the client 104 to which the local player is logged in. This is because some avatars may have malicious data embedded in them or may be too complex to render without slowing down the client device 106. For example, a user account may decide to turn off lights on remote avatars to avoid shaders, not allow custom animations, etc. In some embodiments, each of these options may be set based on how trusted the remote player is. For example, a user account may allow a friend's avatar to have full functionality, while other user accounts may only display basic avatar functionality.

[0068] The user interface service 108 may also provide options to mute or block specific remote players. Additionally, the user interface service 108 may provide a panic or safe mode 210 that audio and visually mutes everyone except friends.

[0069] The quick menu 202 may also include a quick actions 206 section to provide frequently used actions in a convenient location. Some example quick actions include an action to go to your home world, an action to respawn in the last world you were in, an action to select another player's avatar (to communicate privately, to prevent the player from seeing or talking to the local player 116, to copy an avatar or other functionality), and an action to select an emoji.

[0070] The quick menu 202 also includes a docking 212 that provides access to several common features such as a virtual camera, volume settings, and a settings menu, among other features.

[0071] FIG. 3 shows an example of the relationship between gestures and the instantiation and manipulation of virtual controllers.

[0072] In the top image of FIG. 3, the client device 106 recognizes a pinch gesture made by the player's hand and sends a description of the player's hand position in room space to the client 104, along with information about the player's finger positions and the recognized gesture with a confidence score associated with the gesture. The VR tracking service 114 can map the player's hand position to an offset 302. The offset 302 is approximated as the location near the avatar's palm relative to the character root. Therefore, when the player controls the avatar to reach for or point at something, the offset is a good place to place the hand near the avatar's hand. The offset is set to work well with multiple avatars, regardless of the avatar's scale (size). The offset 302 can be used to map a hand of any size on a player of any size to an avatar's hand of any size on an avatar of any size. The offset 302 serves as a point of reference from which the avatar's hand 304 can be drawn.

[0073] In response to receiving information from the client device 106 that the hands are posed for a pinch gesture, the VR tracking service 114 causes the client 104 to instantiate a virtual controller at the location where the pinch gesture was made. The virtual controller may include a larger control surface, as indicated by the virtual controller's perimeter 306, although the control surface may be transparent. Additionally, the virtual controller includes a joystick instantiation point at the initial pinch point 308. This joystick instantiation point may be a visible dot or other indicator that informs the player that the joystick is active.

[0074] In the bottom diagram of FIG. 3 , the player's hand has moved from an original position in the room space to a second position in the room space. The client device 106 can recognize that the player's hand has moved while maintaining the pinch gesture pose. The client device 106 can report data regarding the hand position, finger position, recognition of the gesture, and confidence that the client device 106 correctly recognized the gesture back to the VR tracking service 114. The VR tracking service 114 can move the avatar's hand 304 with the offset 302, which also relocates the pinch point to the current pinch location 310. In response, the client 104 can illustrate a joystick 312 originating from the joystick's instantiation point at the initial pinch point 308 of the virtual controller and ending at the current pinch location 310 of the avatar's hand 304, indicating that movement input has been received.

[0075] In some embodiments, the joystick 312 has a slope or wedge shape that is used to visually indicate the speed of the avatar's movement. The further the current pinch location 310 is from the joystick's instantiation point of the initial pinch point 308, the faster the avatar will move. This is indicated by a larger wedge shape.

[0076] The movement of the virtual controller may be constrained to a plane formed by the x and y axes (ie, forward, backward, and sideways, but not up and down).

[0077] 4A illustrates an example routine for controlling avatar movement through finger tracking. While the example routine illustrates a particular order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the illustrated operations may be performed in parallel or in a different order without substantially affecting the functionality of the routine. In other examples, different components of an example device or system implementing the routine may perform functions substantially simultaneously or in a particular order.

[0078] At block 402, the player may perform a pinch gesture to spawn a virtual controller.

[0079] The client device 106 may be equipped with finger tracking technology. Typically, the client device 106 may be a VR headset that includes one or more cameras capable of capturing images of the player's fingers. The cameras may be physically integrated with the virtual reality headset or may be external to and communicatively coupled to the VR headset. The VR headset may also include software that can map the position of the player's hands and fingers in room space (real world space) to positions in world space (virtual world space). The VR headset may report information regarding hand and finger positions, finger positions, recognized gestures, confidence that the gesture was correctly identified, etc. to the client 104, which receives this information through the VR tracking service 114.

[0080] Using information received by the VR tracking service 114 from the client device 106 (e.g., a VR headset), the client 104 can generate 404 a virtual controller made up of a circular area whose center is visible to the player. In some examples, only the center of the circular area is visible to the player. In some examples, the center of the circular area is a joystick for controlling the movement of the avatar.

[0081] The client 104 can place the virtual controller at a pinch point in world space in block 406. The pinch point is a location in world space to which the player's hands and fingers in room space are mapped. In some examples, the VR tracking service 114 maps the hand positions in room space to positions in world space, and the avatar's finger positions are approximated from the hand positions in world space.

[0082] The client 104 may store, in block 408, an offset in world space for the hand position relative to the character root in the virtual world.

[0083] A complication of controlling the movement of an avatar in a virtual world through hand and finger tracking of a player's hands in room space is that while the avatar changes position in the virtual world, the player does not change position in room space, i.e., the player only moves their hands. Thus, if the avatar moves position but the player does not, the player's hands may map to positions in world space other than the avatar's position. This can give the impression that the avatar has left the virtual controller behind.

[0084] This phenomenon may be a result of the fact that the client device can report the location of the fingers making the gesture in room space, which fingers are involved in the gesture, a confidence value that the detected gesture is correctly identified, the position of the hand in room space, the rotation of the hand, etc. The virtual controller is instantiated in the virtual world at a corresponding position in world space, but the avatar quickly moves from that position.

[0085] To overcome this complexity, the present technology can change the position of the joystick to match the movement of the avatar. This can become complicated when the avatar is moving and rotating in multiple directions at the same time, such as when the avatar is moving sideways and rotating at the same time. An additional level of complexity arises not only from changing the position of the virtual controller, but also from maintaining the position of the virtual controller relative to the avatar in a position that provides a good player experience.

[0086] This technology can receive data on hand and finger position, gesture, confidence value, and finger rotation from a VR client device with hand and finger tracking capabilities, and map the data recorded in room space to world space.

[0087] An avatar can have a root and a base, which collectively constitute a character root. The character root can be an invisible structure in world space to which a visible avatar is mapped. The technology can determine an offset transformation from the avatar's root position in world space and initially place a virtual controller at a position offset from the avatar's root position. The offset is approximated as a position near the avatar's palm. Therefore, when a player controls the avatar to reach for something or point at something, the offset is a good location near the avatar's hand. The offset is set to work well with multiple avatars, regardless of the avatar's scale (size).

[0088] As mentioned above, the space in which hand and finger tracking occurs is relative to the player in room space, while the avatar moves through the virtual world (also known as world space). To compensate for this, our technique takes a snapshot of the position of the player's hands relative to the avatar's position in world space. The snapshot is recorded as a transformation from the avatar's root position to the initial position of the virtual controller.

[0089] The client 104 may store transformations for converting between world space and room space, at block 410 .

[0090] The transformation between room space and the avatar in world space takes into account the point the player picks when pinching in room space, which is mapped to an offset from the character root. As the avatar's orientation and position change, the transformation is used to move the entire relative position as well. The transformation uses matrix math to convert between room space and relative position based on the avatar's root.

[0091] The client 104 transforms and stores the world position in world space using the transformation into room space in block 412. The client 104 also stores the player's current forward vector in world space as "Look Rotation," transforms it into room space, and stores it as ORIG_ROTATION in block 414.

[0092] In summary, the client 104 encodes the character root position in world space and the player position in room space, snapshots the positions for a transformation, and saves the transformation. As the avatar moves, the transformation is also moved so that it moves with the player.

[0093] This transformation allows the placement of the controller in the virtual reality environment relative to the position of the character root to which the avatar is mapped, while tracking the player's hand and finger movements occurring relative to the player in the real-world environment. Thus, when the avatar moves or rotates, the snapshotted transformation moves with the avatar. As the player moves their fingers in room space relative to the initial position where the virtual controller was instantiated in world space (the position where the gesture was made and the virtual controller was instantiated), the client 104 measures the distance of the movement based on the real-world movement. Simultaneously, the client 104 moves the center position of the virtual controller relative to the base of the character root and transforms the hand movements in room space into world space. In this way, the position of the virtual controller relative to the character root is maintained even if the player moves the avatar's head relative to the character root (the avatar's base). Thus, movement operations are relative to the avatar's character root, even if the avatar's head is rotated relative to the avatar's base to look in a direction different from the direction the avatar is moving. The same behavior applies whether the movement is rotational, forward, lateral, or any combination thereof. The same behavior applies in other dimensions, which may allow for roll, pitch, and yaw to facilitate lift, decent, turns, etc.

[0094] FIG. 4B illustrates an exemplary routine for controlling avatar movement, particularly directional movement using a directional virtual controller, through finger tracking. While FIG. 4B is illustrated as following from the routine shown in FIG. 4A, it should be understood that the routine of FIG. 4B can be executed independently of the routine of FIG. 4A. While FIG. 4B may still rely on a transformation from room space to world space, the transformation need not be created as illustrated in FIG. 4A. While the exemplary routine illustrates a specific order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the illustrated operations may be performed in parallel or in a different order without substantially affecting the functionality of the routine. In other examples, different components of an exemplary device or system implementing the routine may perform functions substantially simultaneously or in a specific order.

[0095] A virtual controller may be presented in a virtual world associated with a player's avatar. As described above, only the center of the virtual controller may be visible. The center of the virtual controller may appear like a joystick, allowing the player to move their hand to control the avatar's hand by pushing the joystick in a certain direction. While maintaining a pinch or other gesture used to invoke the virtual controller, the player can move their hand to control the movement of their avatar.

[0096] As mentioned above, one type of virtual controller is a directional controller that can be used to provide inputs for moving the avatar forward, backward, and sideways to the right or left, and another type of virtual controller is a rotational controller that can be used to provide movement for rotating the avatar to the right or left. In some embodiments, the directional controller is associated with the left hand and the rotational controller is associated with the right hand. In some embodiments, the controllers can be in opposite hands (e.g., a directional controller in the right hand) or can be in the same hand, with different gestures used to distinguish between types of movement. In some embodiments, the controllers can be controlled by other parts of the player's body (such as the feet or hips) by tracking these body parts in the room space.

[0097] As described above, the routine includes detecting the movement of the player's hand associated with the directional controller while the hand maintains the gesture at block 416. The client device 106 can detect the player's hand movement and that the hand continues to maintain the gesture and can report the position of the player's hand over time in room space to the VR tracking service 114 of the client 104. The client 104 can position the avatar's hand in world space using the previously determined transformation.

[0098] The routine further includes interpreting the position of the player's hand to determine the speed and direction of movement of the avatar at block 418. For example, the client 104 can determine the offset in room space of the player's hand from the initial pinch point in world space and interpret the offset as a movement speed. The farther the player's hand is from the initial pinch point in world space, the greater the avatar's movement speed. In some embodiments, the client 104 can utilize a function to determine the movement speed using distance as a variable.

[0099] The speed of movement controlled by the directional virtual controller can be in any direction. The client 104 can determine the direction of movement by determining a vector from the initial pinch point in world space to the player's hand position in room space and resolving the movement direction. In the case of a directional controller, the avatar's movement can be in the direction of the vector.

[0100] Thus, client 104 can determine the direction and speed of the avatar's movement, where the direction of movement is the direction of the vector from the initial pinch point in the room space to the position of the player's hand in the room space, and the speed is a function of the length of the vector (the distance from the initial pinch point in the room space to the position of the player's hand in the room space).

[0101] As the avatar moves through world space, the tracking transform is utilized to update the orig_position (the center of the virtual controller) to maintain the relative position of the avatar to the character root to which the avatar is mapped. The routine includes moving the character root based on input to the virtual controller, at block 418, and updating the position of the center of the virtual controller using the tracking transform to maintain the relative position of the avatar to the character root to which the avatar is mapped, at block 422. For example, the client 104 can move the character root based on input to the virtual controller and update the position of the center of the virtual controller using the tracking transform to maintain its relative position to the character root to which the avatar is mapped.

[0102] Inputs to continue moving the avatar are applied until the player returns their hands to their original position in the room space where the virtual controller was instantiated (the initial pinch point), or until the player discontinues the gesture used to instantiate the virtual controller.

[0103] As described above, inputs for moving the avatar are obtained from the player's movements in room space. Virtual controllers are displayed in the avatar's hands in world space to provide an intuitive feedback mechanism for demonstrating that the client 104 has properly interpreted the inputs (in addition to the avatar's movements in world space). The fact that inputs for moving the avatar are obtained from the player's movements in room space provides the advantage that inputs given to the avatar are provided at human scale. An avatar can be small or huge in world space. Thus, the same amount of movement in room space results in the same amount of movement for an avatar of any size, but the display of the virtual joystick can be adjusted to the scale of the avatar.

[0104] FIG. 4C illustrates an example routine for controlling avatar movement through finger tracking, particularly directional movement using a rotational virtual controller. While FIG. 4C is illustrated as following from the routine illustrated in FIG. 4A, it should be understood that the routine of FIG. 4C can be executed independently of the routine of FIG. 4A. While FIG. 4C may still rely on a transformation from room space to world space, the transformation need not be created as illustrated in FIG. 4A. Furthermore, although FIGS. 4B and 4C are illustrated separately, it should be understood that both routines can be executed simultaneously, and in some embodiments, they can be executed by the same hardware or software object. While the example routine illustrates a specific order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the illustrated operations may be executed in parallel or in a different order without substantially affecting the functionality of the routine. In other examples, different components of an example device or system implementing the routine may perform functions substantially simultaneously or in a specific order.

[0105] A virtual controller may be presented in the virtual world associated with the player's avatar. As described above, only the center of the virtual controller may be visible. The center of the virtual controller may appear like a joystick, allowing the player to move their hand to control the avatar's hand by pushing the joystick in a certain direction. While maintaining a pinch gesture, or other gesture used to invoke the virtual controller, the player may move their hand to control the movement of their avatar.

[0106] As mentioned above, one type of virtual controller is a directional controller that can be used to provide inputs for moving the avatar forward, backward, and sideways to the right or left, and another type of virtual controller is a rotational controller that can be used to provide movement for rotating the avatar to the right or left. In some embodiments, the directional controller is associated with the left hand and the rotational controller is associated with the right hand. In some embodiments, the controllers can be in opposite hands (e.g., a directional controller in the right hand) or can be in the same hand, with different gestures used to distinguish between types of movement. In some embodiments, the controllers can be controlled by other parts of the player's body (such as the feet or hips) by tracking these body parts in the room space.

[0107] As described above, the routine includes detecting the movement of the player's hand associated with the rotation controller while the hand maintains the gesture at block 424. The client device 106 can detect the player's hand movement and that the hand continues to maintain the gesture and can report the position of the player's hand over time in room space to the VR tracking service 114 of the client 104. The client 104 can position the avatar's hand in world space using the previously determined transformation.

[0108] The routine further includes constraining the joystick movement to the x-axis at block 426. For example, the client 104 can determine that the player's hand movement (while maintaining the gesture) is to the positive or negative side of the virtual controller's origin (the initial pinch point in room space). While the player may move their hand and, accordingly, the avatar's hand in both the x- and y-directions, the client 104 can constrain the joystick movement to only the x-axis. In some embodiments, the rotation controller can be constrained to other axes (e.g., rotation can also be pitched up or down based on the z-axis). In some embodiments, the rotation virtual controller can be unconstrained to any axis, allowing for six degrees of freedom of rotation.

[0109] The routine further includes interpreting the position of the player's hand to determine a rotation speed and rotation of the movement at block 428. For example, the client 104 may determine an offset in room space of the player's hand from an initial pinch point in room space and interpret that offset in the x-axis as a rotation speed. The farther the player's hand is along the x-axis from the initial pinch point in room space, the greater the rotation speed of the avatar. In some embodiments, the client 104 may utilize a function to determine the rotation speed using distance as a variable.

[0110] The rotation speed controlled by the rotation virtual controller can be in any direction. The example given here is left / right rotation based on the x-axis, but the client 104 can determine the movement direction by determining a vector from the initial pinch point in world space to the player's hand position in room space to resolve the movement direction. However, the rotation controller described above is limited to the x-axis component of the vector. With the rotation controller, the avatar's movement can be in the direction of the vector along the x-axis (i.e., right or left rotation).

[0111] Thus, client 104 can determine the direction and speed of the avatar's movement, where the direction of the movement is the direction of the vector along the X axis from the initial pinch point in room space to the position of the player's hand in room space, and the speed is a function of the length of the vector along the X axis (the distance from the initial pinch point in world space to the position of the player's hand in room space).

[0112] As the avatar rotates in world space, the tracking transform is utilized to update the orig_position (center of the virtual controller) to maintain the relative position of the avatar to the character root to which the avatar is mapped. The routine includes rotating the character root based on the virtual controller input at block 430 and updating the position of the center of the virtual controller using the tracking transform to maintain the relative position of the avatar to the character root to which the avatar is mapped at block 432. For example, the client 104 can rotate the character root based on input to the virtual controller provided by moving the player's hands in room space and can update the center position of the virtual controller using the tracking transform to maintain the relative position of the avatar to the character root to which the avatar is mapped.

[0113] The input to continue moving the avatar is applied until the player returns their hands to their original positions within the room space where the virtual controller was instantiated, or until the player discontinues the gesture used to instantiate the virtual controller.

[0114] As described above, inputs for moving the avatar are obtained from the player's movements in room space. Virtual controllers are displayed in the avatar's hands in world space to provide an intuitive feedback mechanism for demonstrating that the client 104 has properly interpreted the inputs (in addition to the avatar's movements in world space). The fact that inputs for moving the avatar are obtained from the player's movements in room space provides the advantage that inputs given to the avatar are provided at human scale. An avatar can be small or huge in world space. Thus, the same amount of movement in room space results in the same amount of movement for an avatar of any size, but the display of the virtual joystick can be adjusted to the scale of the avatar.

[0115] 5 shows the relationship between a player in a room space 506 and an avatar in a world space 508. The area around the player in the room space 506 is the area in which the player's movements are tracked. As the player moves their hands throughout the room space, the hand and finger tracking client device 106 can record the positions and orientations of the player's hands and fingers and provide input indicating recognized gestures and the locations of those gestures to the client 104, which renders the virtual world and avatars.

[0116] The client 104 stores the world position in world space 508 transformed using the transformations described above in room space 506. The client 104 also stores the player's current forward vector 502 in world space 508 and transforms it into a "Look Rotation" based on ORIG_ROTATION 504. As described herein, one or more transformations can be used to map the player's hand positions in world space 508 to the relative position of the character root in room space 506.

[0117] FIG. 6 shows an example of an avatar performing a pinching gesture with its left hand to instantiate a virtual controller.

[0118] FIG. 6 illustrates an exemplary world space 602 as seen from a first-person perspective by a player controlling an avatar. The avatar's left hand 604 is making a pinch pose with its ring finger and thumb, mirroring the same pose made by the player's left hand in room space. As a result of the detected gesture, the client 104 instantiates a virtual controller represented by a joystick 606, indicated by a dot near the intersection of the tip of the avatar's thumb and the tip of the avatar's ring finger. Additionally, the client 104 renders a directional arrow 608 to instruct the player to move the avatar's left hand 604 forward or backward, which controls the avatar to move forward or backward. Although the directional arrow 608 indicates only forward or backward movement, the hand can move in any direction, and left hand movements can be used to provide input to move the avatar forward, backward, and sideways to the right or left.

[0119] Figure 7 shows an example of using a virtual controller to make an avatar walk forward.

[0120] For example, the player provides an input that moves the avatar's left hand 604 slightly forward, as indicated by the enlarged joystick 606, from the controller's initial position to the controller's current position. The avatar can walk in the direction of a vector that begins at the controller's initial position and points in the direction of the controller's current position. The avatar can walk at a speed relative to or proportional to the distance between the controller's initial position and the controller's current position. As the avatar moves through the virtual world, the controller's initial position remains in the same relative position to the character's root.

[0121] FIG. 7 shows world space 602 after the avatar has moved forward in world space 602 in response to inputs provided to the virtual controller.

[0122] FIG. 8 shows an example of using the virtual controller to rotate the avatar to the left.

[0123] For example, after a player generates a pinch gesture to instantiate a virtual controller, the player provides input to the avatar to move its hand leftward, as indicated by the enlarged joystick 804, from the controller's initial position to the controller's current position. In some embodiments, if the virtual controller is configured to provide commands to rotate the avatar left and right, the virtual controller's joystick 804 may be constrained to the left and right directions. The avatar can rotate in the direction of a vector that originates from the controller's initial position and points in the direction of the controller's current position. The avatar can rotate at a speed relative to or proportional to the distance between the controller's initial position and the controller's current position. While the avatar rotates through the virtual world, the controller's initial position remains in the same relative position to the character root.

[0124] Figure 9 shows an example of using the virtual controller to rotate the avatar to the right, a movement opposite to that shown in Figure 8. Figure 9 shows world space 602 after the avatar has moved to the right in world space 602 in response to inputs provided to the virtual controller.

[0125] 10 shows an avatar simultaneously performing a pinching gesture with left hand 604 and right hand 802. A virtual controller is instantiated on each hand, as indicated by joystick 606 and joystick 804 shown on left hand 604 and right hand 802, respectively. In some embodiments, movement of left hand 604 can be used to provide input for moving the avatar forward, backward, and sideways to the right or left, and movement of right hand 802 can be used to provide movement for rotating the avatar right or left. Input can be provided to both virtual controllers simultaneously.

[0126] FIG. 11 shows an example of using the virtual controller in the left hand 604 and the virtual controller in the right hand 802 simultaneously to make the avatar walk forward and turn right as described above.

[0127] The technology can be used for many different movements and can achieve more complex commands such as jumping, or combinations of movements such as running while jumping, turning while moving sideways, etc. The technology can also be used to create ports, such as teleportation or holographic portation (holoports), to move an avatar from one location to another, or from one world to another.

[0128] This technology can also be used to control or in conjunction with adaptive technology. For example, some virtual reality players may experience motion sickness, and adaptive technology can change the animation or perception of movement to mitigate the motion sickness. In another example, this technology can be used to change the horizon of the field of view to enable a virtual reality experience for people lying in bed. In another example, this technology can be used to maintain hand positions used in sign language.

[0129] The technology can also be used to create avatar effects, such as allowing an avatar to change shape or allow an avatar to grow or shrink. Other effects include muting and unmuting, activating facial expressions and emotes, etc.

[0130] In some embodiments, each image in Figures 6-11 can be thought of as a frame in a sequence in an animation.

[0131] 12 illustrates an exemplary routine for controlling avatar movement through finger tracking. While the exemplary routine illustrates a particular order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the illustrated operations may be performed in parallel or in a different order without substantially affecting the functionality of the routine. In other examples, different components of an exemplary device or system implementing the routine may perform functions substantially simultaneously or in a particular order.

[0132] According to some examples, the method includes receiving a description of a first gesture performed by one or more fingers of a player and an initial position in room space of the one or more fingers performing the first gesture at block 1202. For example, the client 104 shown in FIG. 1 may receive the description of a first gesture performed by one or more fingers of a player and the initial position in room space of the one or more fingers performing the first gesture from a tracking service of the client device 106.

[0133] According to some examples, the method includes determining a relative position of the one or more fingers performing the first gesture relative to the pinch initiation point detected in the room space at block 1204. For example, the client 104 shown in FIG. 1 may determine a relative position of the one or more fingers performing the first gesture relative to the pinch initiation point detected in the room space.

[0134] According to some examples, the method includes instantiating a virtual controller at an offset position relative to the character root at block 1206. For example, the client 104 shown in FIG. 1 may instantiate a virtual controller at an offset position relative to the character root. Instantiating the virtual controller at the offset position further includes positioning a center of the virtual controller at the offset position and displaying a virtual controller user interface at the offset position.

[0135] According to some examples, the method includes displaying a rendering of the avatar's hand with a joystick of a virtual controller at an offset position within the rendering of the hand at block 1208. For example, the client 104 shown in FIG. 1 may display a rendering of the avatar's hand with a joystick of a virtual controller at an offset position within the rendering of the hand.

[0136] According to some examples, the method includes receiving, from a tracking service, updated positions in room space of one or more fingers performing the first gesture, at block 1210. For example, the client 104 shown in FIG. 1 may receive, from the tracking service, updated positions in room space of one or more fingers performing the first gesture.

[0137] According to some examples, the method includes determining a distance and a direction of an updated position in the room space of the one or more fingers performing the first gesture compared to an initial position in the room space of the one or more fingers performing the first gesture at block 1212. For example, client 104 shown in FIG. 1 may determine a distance and a direction of an updated position in the room space of the one or more fingers performing the first gesture compared to an initial position in the room space of the one or more fingers performing the first gesture.

[0138] According to some examples, the method includes displaying an animation of a hand moving from an initial position (initial offset position) toward an updated position (updated offset position) at block 1214. For example, the client 104 shown in FIG. 1 may display an animation of a hand moving from the initial position toward the updated position.

[0139] According to some examples, the method includes displaying a movement input indicator in conjunction with an animation of a hand moving from the initial offset position toward the updated position at block 1216. For example, the client 104 shown in FIG. 1 may display a movement input indicator in conjunction with an animation of a hand moving from the initial position toward the updated position. The movement input indicator may be a magnified joystick, as described above.

[0140] According to some examples, the method includes, at block 1218, controlling the movement of the avatar based on the distance and direction of the current position of the player's hand in the room space compared to the initial position of the player's hand in the room space. For example, the client 104 illustrated in FIG. 1 may control the movement of the avatar based on the distance and direction of the current position of the player's hand in the room space compared to the initial position of the player's hand. In this example, the virtual controller instance is a linear motion controller. The movement of the avatar is the linear direction of a vector that originates from the initial position of the player's hand and points in the direction of the updated position of the player's hand. The movement of the avatar has a velocity that corresponds to the distance between the initial position of the player's hand and the updated position of the player's hand, and a larger distance between the initial position and the updated position corresponds to a larger velocity than a smaller distance between the initial position and the updated position.

[0141] When the virtual controller instance is a rotational motion controller, the avatar's movement is in the rotation direction of a vector that starts from the initial position of the player's hand in the room space and points in the direction of the updated position of the player's hand, and the avatar's movement has a speed that corresponds to the distance between the initial position of the player's hand and the updated position of the player's hand, and a larger distance between the initial position and the updated position corresponds to a larger speed than a smaller distance between the initial position and the updated position.

[0142] Although the virtual controller is illustrated and described throughout this description as a virtual joystick-type controller, one skilled in the art will understand that the virtual controller can be of other types. The virtual controller can be represented by a movement indicator, a virtual button, or other UI object. A joystick is just one example of a virtual controller. Furthermore, the present technology can be utilized without displaying a virtual controller. Input from gestures can be provided to affect gameplay without displaying a virtual controller.

[0143] While this specification has focused on two types of movement controllers (i.e., directional controllers and rotational controllers), it will be understood by those skilled in the relevant arts that the use of virtual controllers and / or gestures to provide input for controlling an avatar or gameplay is not limited to movement. For example, other inputs that may be provided by a player performing a gesture include providing a gesture to grab an object, interact with an object, open or close a menu, mute or unmute the microphone, cancel input or drop an object, jump, or lock out other gestures from being interpreted as a command to do something other than the avatar's hand and finger positions.

[0144] Although this specification defines the pinch gesture as a gesture that is valid for instantiating a virtual controller, it should be understood that other gestures can be defined to instantiate this or other functionality. That is, other gestures can be used to instantiate a movement controller, and certain pinch gestures can be remapped to different functions.

[0145] As an example, some example function-gesture mappings are shown. 1. Interact / Grab - Pinch with left / right thumb and index finger, palm facing away from user 2. Open / close the quick menu (e.g., Quick Menu 202) - Pinch with left thumb and index finger, palm facing the user 3. Movement - Pinch and hold with left thumb and middle finger 4. Mute / Unmute Microphone - Pinch and hold left thumb and ring finger 5. Cancel / Drop - Pinch with left / right thumb and pinky 6. Open / Close Oculus Menu - Hold your right thumb and index finger together with your palm facing the user 7. Rotational Movement - Pinch and hold with your right thumb and middle finger 8. Jump - Pinch with your right thumb and ring finger 9. Gesture Lock - Touch with your left thumb and pinky to lock the gesture, disabling all movement until you perform Gesture Lock (touch with your left thumb and pinky together) again.

[0146] As mentioned above, these gestures may be rebindable to other controls.

[0147] Many of the gestures described above can be performed simultaneously, achieving multiple functions simultaneously. For example, directional movement, rotational movement, and jumping can all be performed simultaneously.

[0148] The gesture lock gesture can be particularly useful in gameplay environments where a player controls an avatar in a conversational environment and directs the avatar to take other actions. Because many people "talk with their hands," and indeed, some deaf players may literally communicate using sign language, it may be desirable to prevent gestures from being mapped to other functions. This allows a player, when gesture lock is enabled, to control a conversational avatar and perform various hand gestures without having to move the avatar, mute it, open menus, and so on. When the player completes the conversation or wants to re-enable the mapping of gestures to functions, the player can disable gesture lock.

[0149] 13 illustrates an example of a computing system 1300, which may be, for example, a client device 106, a web service 110, or any computing device comprising a networking service 112, or any of its components in which components of the system communicate with each other using a connection 1302. The connection 1302 may be a physical connection via a bus or a direct connection to a processor 1304, such as in a chipset architecture. The connection 1302 may also be a virtual connection, a network connection, or a logical connection.

[0150] In some embodiments, computing system 1300 is a distributed system in which the functions described in this disclosure may be distributed within a data center, within multiple data centers, within a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functions for which that component is described. In some embodiments, a component may be a physical or virtual device.

[0151] The exemplary computing system 1300 includes at least one processing unit (CPU or processor) 1304 and connections 1302 coupling various system components to the processor 1304, including system memory 1308 such as read-only memory (ROM) 1310 and random access memory (RAM) 1312. The computing system 1300 may include a cache of high-speed memory 1306 directly connected to the processor 1304, closely connected to the processor 1304, or integrated as part of the processor 1304.

[0152] Processor 1304 can include any general-purpose processor, hardware services configured to control processor 1304, or software services, such as services 1316, 1318, 1320 stored on storage device 1314, as well as special-purpose processors where software instructions are embedded in the actual processor design. Processor 1304 may essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0153] To enable player interaction, computing system 1300 includes input devices 1326, which can represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1300 also includes output devices 1322, which can be one or more of several output mechanisms known to those skilled in the art. In some examples, a multimodal system may allow a player to provide multiple types of input / output for communicating with computing system 1300. Computing system 1300 may generally include a communications interface 1324 that can control and manage player input and system output. There are no constraints on operation with a particular hardware configuration, so the basic functionality herein can be easily replaced with improved hardware or firmware configurations as they are developed.

[0154] The storage device 1314 may be a non-volatile memory device, such as a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM), a read-only memory (ROM), and / or some combination of these devices.

[0155] The storage devices 1314 may include software services, servers, services, etc., where code defining such software, when executed by the processor 1304, causes the system to perform functions. In some embodiments, a hardware service that performs a particular function may include software components stored on a computer-readable medium in connection with the hardware components necessary to perform that function, such as the processor 1304, the connections 1302, the output devices 1322, etc.

[0156] For clarity of explanation, in some examples, the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a software-implemented method, or functional blocks comprising a combination of hardware and software.

[0157] Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software service(s), alone or in combination with other devices. In some embodiments, a service may be software that resides in memory of a client device and / or one or more servers of a content management system and performs one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or collection of programs that perform a particular function. In some embodiments, a service may be considered a server. Memory may be a non-transitory computer-readable medium.

[0158] In some embodiments, computer-readable storage devices, media, and memories can include cables or wireless signals containing bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0159] Methods according to the above-described embodiments can be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. Some of the computer resources used can be accessible over a network. The executable computer instructions can be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store instructions used, information used, and / or information generated during the course of methods according to the described embodiments include magnetic or optical disks, solid-state memory devices, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.

[0160] Devices implementing methods according to these disclosures may comprise hardware, firmware, and / or software and may take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, etc. The functionality described herein may also be implemented in peripheral devices or expansion cards. Such functionality may also be implemented on a circuit board, across different chips or different processes running on a single device, by way of further example.

[0161] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functionality described in these disclosures.

[0162] Embodiments of the present technology can be further understood from the following sections.

[0163] Item 1. A method for controlling the movement of an avatar via finger tracking, comprising: receiving, from a tracking service, a description of a first gesture performed by one or more fingers of a player and an initial position in space of the one or more fingers performing the first gesture; determining a relative position of the one or more fingers performing the first gesture with respect to a character root, wherein the character root is a volume in a virtual world to which the avatar representing the player is mapped; and generating an instance of a virtual controller at the relative position.

[0164] Item 2. The method according to Item 1, wherein generating the instance of the virtual controller at the relative position further comprises: placing a center of the virtual controller at the relative position; and displaying a virtual controller user interface at the relative position.

[0165] Item 3. The method according to any one of Items 1 to 2, comprising: displaying a rendering of the avatar's hand at the relative position, together with the virtual controller within the rendering of the hand; determining a distance and a direction of an updated position in space of the one or more fingers performing the first gesture, compared to the initial position in space of the one or more fingers performing the first gesture; displaying an animation of the hand moving in the direction from the relative position to the updated position; and displaying an indicator of movement input in conjunction with the animation of the hand moving in the direction from the relative position to the updated position.

[0166] Clause 4. The method of any of clauses 1 to 3, further comprising: receiving, from the tracking service, an updated position in space of the one or more fingers performing the first gesture; determining a distance and a direction of the updated position in space of the one or more fingers performing the first gesture compared to an initial position in space of the one or more fingers performing the first gesture; and controlling a movement behavior of the avatar based on the distance and the direction of the updated position compared to the initial position.

[0167] Clause 5. The method of any of clauses 1 to 4, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that starts at the initial position and points in the direction of the updated position.

[0168] Clause 6. The method of any of clauses 1 to 5, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar has a velocity corresponding to a distance between the initial position and the updated position, and a larger distance between the initial position and the updated position corresponds to a larger velocity than a smaller distance between the initial position and the updated position.

[0169] Clause 7. The method of any of clauses 1 to 6, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that starts at the initial position and points in the direction of the updated position.

[0170] Clause 8. The method of any of clauses 1 to 7, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar has a velocity corresponding to a distance between the initial position and the updated position, and a larger distance between the initial position and the updated position corresponds to a larger velocity than a smaller distance between the initial position and the updated position.

[0171] Clause 9. A computing system comprising a processor and memory storing instructions that, when executed by said processor, configure said system to perform the method of any of clauses 1-8.

[0172] Clause 10. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any of clauses 1-8.

Claims

1. 1. A method for controlling avatar movement through finger tracking, comprising: receiving, from a tracking service, a description of a first gesture performed by one or more fingers of a player in a room space and initial positions in the room space of the one or more fingers performing the first gesture; determining a relative position of the one or more fingers in the room space performing the first gesture with respect to a character root in world space, the character root being a volume in a virtual world onto which the avatar representing the player is mapped; generating an instance of a virtual controller at the relative position; A method comprising:

2. Generating the instance of the virtual controller at the relative position further comprises: locating a center of the virtual controller at the relative position; displaying a virtual controller user interface at the relative position; 2. The method of claim 1, comprising:

3. displaying a rendering of the avatar's hand at the relative position with the virtual controller within the rendering of the hand; determining a distance and a direction of an updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; displaying an animation of the avatar's hand moving from the relative position in the direction of the updated position; displaying a movement input indicator in conjunction with the animation of the hand moving from the relative position in the direction of the updated position; The method of claim 2 , comprising:

4. receiving, from the tracking service, updated positions in space of the one or more fingers performing the first gesture; determining a distance and a direction of the updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; controlling a movement movement of the avatar based on the distance and the direction of the updated position compared to the initial position; The method of claim 1 further comprising:

5. The method of claim 4 , wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that originates at the initial position and points in the direction of the updated position.

6. 5. The method of claim 4, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar has a velocity corresponding to the distance between the initial position and the updated position, with a greater distance between the initial position and the updated position corresponding to a greater velocity than a smaller distance between the initial position and the updated position.

7. The method of claim 4 , wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that originates at the initial position and points in the direction of the updated position.

8. 5. The method of claim 4, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar has a velocity corresponding to the distance between the initial position and the updated position, with a greater distance between the initial position and the updated position corresponding to a greater velocity than a smaller distance between the initial position and the updated position.

9. 1. A computing system comprising: a processor; When executed by the processor, the system is receiving, from a tracking service, a description of a first gesture performed by one or more fingers of a player in a room space and initial positions in the room space of the one or more fingers performing the first gesture; determining a relative position of the one or more fingers performing the first gesture in the room space with respect to a character root in world space, the character root being a volume in a virtual world onto which an avatar representing the player is mapped; generating an instance of a virtual controller at the relative position; a memory storing instructions for configuring the A computing system comprising:

10. Generating the instance of the virtual controller at the relative position further comprises: placing a center of the virtual controller at the relative position; displaying a virtual controller user interface at the relative position; The computing system of claim 9.

11. The instructions cause the system to: displaying a rendering of the avatar's hand at the relative position with a virtual controller within the rendering of the hand; determining a distance and a direction of an updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; displaying an animation of the hand moving from the relative position in the world space toward the updated position; displaying an indicator of a movement input in conjunction with the animation of the hand moving from the relative position in the direction of the updated position; 11. The computing system of claim 10 configured to:

12. The instructions further cause the system to: receiving, from the tracking service, updated positions in the room space of the one or more fingers performing the first gesture; determining a distance and a direction of the updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; controlling a movement of the avatar in the world space based on the distance and the direction of the updated position compared to the initial position; 10. The computing system of claim 9, configured to:

13. 13. The computing system of claim 12, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that originates at the initial position and points in the direction of the updated position.

14. 13. The computing system of claim 12, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is at a velocity corresponding to the distance between the initial position and the updated position, with a greater distance between the initial position and the updated position corresponding to a greater velocity than a smaller distance between the initial position and the updated position.

15. 13. The computing system of claim 12, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that originates at the initial position and points in the direction of the updated position.

16. 13. The computing system of claim 12, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar has a velocity corresponding to the distance between the initial position and the updated position, with a greater distance between the initial position and the updated position corresponding to a greater velocity than a smaller distance between the initial position and the updated position.

17. When executed by at least one processor, the at least one processor: receiving, from a tracking service, a description of a first gesture performed by one or more fingers of a player in a room space and initial positions in the room space of the one or more fingers performing the first gesture; determining a relative position of the one or more fingers performing the first gesture in the room space with respect to a character root in world space, the character root being a volume in a virtual world onto which an avatar representing the player is mapped; generating an instance of a virtual controller at the relative position; A non-transitory computer-readable storage medium containing instructions.

18. Generating the instance of the virtual controller at the relative position further comprises: placing the center of the virtual controller at the relative position; displaying a virtual controller user interface at the relative position; 20. The computer-readable storage medium of claim 17.

19. The at least one processor further comprises: displaying a rendering of the avatar's hand at the relative position with a virtual controller within the rendering of the hand; determining a distance and a direction of an updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; displaying an animation of the hand moving from the relative position in the world space in the direction of the updated position; displaying a movement input indicator in conjunction with the animation of the hand moving from the relative position in the direction of the updated position; 20. The computer-readable storage medium of claim 18 containing instructions.

20. The at least one processor further comprises: receiving, from the tracking service, updated positions in the room space of the one or more fingers performing the first gesture; determining a distance and a direction of the updated position in the room space of the one or more fingers performing the first gesture compared to the initial position in the room space of the one or more fingers performing the first gesture; controlling a movement action of the avatar based on the distance and the direction of the updated position compared to the initial position; 20. The computer-readable storage medium of claim 17, comprising instructions.

Images