Movement using finger tracking

Abstract

The technology provides a mechanism by which a player can make gestures with their hands and invoke virtual controllers, and the tracking of hand movements and fingers can effect movement of an avatar through a virtual world without the need for a hardware controller. Thus, a player can have natural translations or otherwise control the avatar's hands to interact with other objects in the virtual world, and can then use the avatar's hands to invoke virtual controllers to control the movement of the avatar without the player having a physical controller.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority of U.S. Patent Application No. 18 / 167,486, entitled "LOCOMOTION USING FINGER TRACKING", filed on February 10, 2023, which is hereby expressly incorporated by reference in its entirety for all purposes.

Background Art

[0002] Virtual Reality (VR) provides an immersive virtual world environment, and players often experience the virtual world from a first - person perspective. In many respects, a player experiencing a virtual world from a first - person perspective feels as if they are actually in the virtual world, except for the equipment they use to access and control the avatar. Players often access the virtual world via a VR headset, which is often combined with a hand - held controller. Sometimes, players may also utilize additional skeleton tracking devices and tactile feedback devices. While the equipment can provide additional inputs to support more realistic movements by the avatar, the equipment can also place limitations on how immersed a player can be in the virtual world.

Brief Description of the Drawings

[0003] To easily identify the discussion of any particular element or act, the top one or more digits in the reference number refer to the drawing number in which the element was first introduced.

[0004] [Figure 1] FIG. 1 shows an exemplary virtual world platform for playing and hosting a multi - player virtual reality (VR) experience, according to some aspects of the present technology.

[0005] [Figure 2]Figure 2 shows an exemplary quick menu according to several embodiments of this technology.

[0006] [Figure 3] Figure 3 shows an example of the relationship between gestures and virtual controller instantiation, and the operation of the virtual controller according to several embodiments of this technology.

[0007] [Figure 4A] Figure 4A shows an exemplary routine for controlling the movement of an avatar through finger tracking, according to several embodiments of this technology.

[0008] [Figure 4B] Figure 4B shows an exemplary routine for controlling the movement of an avatar through finger tracking, according to several embodiments of the present technology.

[0009] [Figure 4C] Figure 4C shows an exemplary routine for controlling the movement of an avatar through finger tracking, according to several embodiments of this technology.

[0010] [Figure 5] Figure 5 shows the relationship between the player in the room space and the avatar in the world space according to several embodiments of this technology.

[0011] [Figure 6] Figure 6 shows an example of an avatar that performs a pinching gesture with the left hand to instantiate a virtual controller, according to several aspects of this technology.

[0012] [Figure 7] Figure 7 shows an example of using a virtual controller to make an avatar walk forward, according to several embodiments of this technology.

[0013] [Figure 8]FIG. 8 illustrates an example of using a virtual controller to rotate an avatar to the left according to some aspects of the present technology.

[0014] [Figure 9] FIG. 9 illustrates an example of using a virtual controller to rotate an avatar to the right according to some aspects of the present technology.

[0015] [Figure 10] FIG. 10 shows an avatar performing a pinching gesture with the left and right hands according to some aspects of the present technology.

[0016] [Figure 11] FIG. 11 illustrates an example of using a virtual controller for the left hand and a virtual controller for the right hand to make an avatar walk forward and rotate to the right according to some aspects of the present technology.

[0017] [Figure 12] FIG. 12 shows an exemplary routine for controlling the movement of an avatar through finger tracking according to some aspects of the present technology.

[0018] [Figure 13] FIG. 13 shows an example of a system for implementing a particular aspect of the present technology.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Some client devices for interacting in virtual reality include hand tracking capabilities. Hand tracking has been used to control the limb movements of an avatar in a virtual reality environment. For example, in a virtual world where a player represented by their avatar is conversing, many people "talk with their hands" by making various gestures in accordance with their utterances, so it can be useful to show the hands of the avatar moving. Of course, participants in a conversation where at least one member of the conversation has a hearing impairment may use sign language to speak.

[0020] Hand tracking creates a more realistic conversation experience in virtual reality, but most players have the drawback of having to let go of any controller that can normally be used for movement (players input to the controller to navigate their avatar through the virtual world). This leads to the problem that the player who has let go of the controller needs to find the controller before they can navigate their avatar across the entire virtual world, which results in a cumbersome user experience.

[0021] This experience can even become more cumbersome when the client (VR headset) needs to redetect the controller and remap the controller in the physical space (also called the room space) to the virtual world, which takes a little time.

[0022] Virtual reality provides a fully immersive experience, but the need to interact with a controller can sometimes detract from the virtual reality experience. Therefore, it is necessary to be able to navigate a virtual reality world without a controller. And it is necessary to enable the player to seamlessly transition from conversation to movement.

[0023] This technology provides a mechanism that allows players to make hand gestures and invoke a virtual controller, and hand movement and finger tracking can bring about avatar movement through the virtual world without the need for a hardware controller. Thus, players can have a natural transformation, or otherwise, control their avatar's hands to interact with other objects in the virtual world, and then use their hands to invoke a virtual controller and control the movement of their avatar without the player having a physical controller.

[0024] Furthermore, this technology addresses the complexity of controlling the movement of the avatar in the virtual world through hand and finger tracking of the player's hands in the room space. This complexity stems from the fact that while the avatar changes position in the virtual world, the player does not change their own position in the room space; in other words, the player only moves their hands. Therefore, if the avatar moves but the player does not, the player's hands may be mapped to a position other than the avatar's. This can give the impression that the avatar has left the virtual controller behind.

[0025] This phenomenon may be a result of the fact that the client device can report the position of the fingers performing the gesture, which fingers are involved in the gesture, a confidence value indicating that the detected gesture is correctly identified, the position of the hand, and the rotation of the hand. The virtual controller is instantiated in the virtual world at the corresponding position, but the avatar immediately moves from that position.

[0026] To overcome this complexity, the technology allows the joystick's position to change in accordance with the avatar's movement. This can become complicated when the avatar is rotating in multiple directions simultaneously, such as when it is moving sideways while simultaneously rotating. A further level of complexity arises not only from changing the virtual controller's position, but also from maintaining the virtual controller's position relative to the avatar in a way that provides a good player experience.

[0027] Figure 1 shows an exemplary virtual world platform 102 suitable for playing and hosting multiplayer virtual reality (VR) experiences to implement this technology. The virtual world platform 102 connects clients 104 via web services 110 and networking services 112, allowing them to socially interact together in the virtual world hosted by the virtual world platform 102.

[0028] The virtual world platform 102 primarily includes clients 104, which are instances of applications running on client devices 106. Clients 104 interact via network connections with web services 110, which support them by providing various services through one or more application programming interfaces (APIs). Some of the main services provided by web services 110 include support for the virtual world through the world API 128, user profiles through the user API 132, trust and security through the trust API 144, and complex avatars through the avatar API 136. Among other functions, web services 110 generally store and provide long-term state information.

[0029] Client 104 also interacts with Networking Service 112, which provides communication services between Client 104, Networking Service 112, and remote instances of Client 104 (not shown), and shares state information between each instance of Client 104. In particular, state information is received by Networking Service 112 from multiple instances of Client 104 when each instance of Client 104 controls its local player 116. Networking Service 112 can forward state information about each player to other instances of Client 104 when all local players 116 of each client instance are engaged in gameplay in the same virtual world. Networking Service 112 provides optimized packet routing through Optimized Packet Routing Service 140 and moderation between one or more clients through Moderation Service 142.

[0030] Client 104 is a runtime environment that runs on a specific client device 106. In this specification, client 104, local client, and remote client may be referred to as all instances of client 104 running on their respective client devices 106. A specific user account is logged into a specific instance of client 104. Local and remote clients are distinguished to indicate how client 104 handles first-person input from a player on the client device 106 on which client 104 is running, and how the remote client handles third-party input received from another player operating the client device on which the remote client is running.

[0031] The client device 106 can be any computing device. While the client 104 is specifically adapted to provide an immersive virtual reality experience through interactions that a VR headset needs to experience, the client 104 can also run on computers and mobile devices. Some virtual worlds or complex avatars may not be configured to work well on certain device types, and therefore, while the client 104 can run on many platforms and devices, not all virtual worlds or complex avatars are available, nor are they fully functional on all client devices 106.

[0032] User Interface Service 108 is a service that is part of Client 104. User Interface Service 108 is configured to provide various user interface elements, such as menus that display various player settings, available worlds, saved complex avatars, and friend lists. User Interface Service 108 can position its menus through interaction with one or more APIs provided by Web Service 110, while other parts of the menus are loaded directly from User Interface Service 108.

[0033] The user interface service 108 can provide a menu of available worlds by calling the world API 128 to obtain a list of worlds that the user account logged into client 104 is permitted to enter. The world API 128 can retrieve all public worlds from the world asset database 130 and send a list of them to client 104. Furthermore, the world API 128 can request the world ID of any private worlds associated with the user account logged into client 104, retrieve the private worlds from the world asset database 130, and send them to client 104. The user interface service 108 can receive player input through the hardware interface to navigate through the world menu and receive the selection of worlds to visit.

[0034] Another user interface provided by User Interface Service 108 relates to various player settings. Such settings may relate to whether the human player is sitting or standing, settings to minimize motion sickness for players prone to motion sickness when playing in VR, settings for selecting complex avatars, and settings regarding how the player may appear in the virtual world and who may see the player.

[0035] One notable user interface feature provided by the user interface service 108 is the Trust and Safety menu. The user interface service 108 can contact the user API 132 to retrieve the current trust and safety settings from the user profile database 134 and display these settings in the Trust and Safety menu. The Trust and Safety menu provides a user account with the ability to determine which remote players 124 can see the player's avatar (local player 116), or which player's avatar can be seen by the player's avatar when both players are in the same world. For example, it may be desirable to avoid interaction with newer players on the virtual world platform 102 because they have not yet established a trust relationship within the virtual world platform 102. It may also be desirable to restrict the functionality of remote players' avatars handled by the instance of client 104 to which the local player is logged in. This is because some avatars may have malicious data embedded in them, or the avatars may be too complex to render without degrading the performance of the client device 106. For example, a user account might decide to turn off the lights on a remote avatar to avoid shaders, or not allow custom animations. In some embodiments, each of these options may be set based on how much the remote player is trusted. For example, a user account might allow a friend's avatar to have a full appearance, while other user accounts might only display a basic avatar appearance.

[0036] The user interface service 108 can also provide the option to mute or block specific remote players. Furthermore, the user interface service 108 can provide a panic mode that mutes all non-friends both audibly and visually.

[0037] After the player selects a virtual world from a menu provided by the user interface service 108, the client 104 can download an instance of the virtual world by calling the world API 128, which can retrieve the virtual world from the world asset database 130 and send it to the client 104 for execution.

[0038] A world asset is a large binary file built for game engines such as Unity, using an editor with a Software Development Kit (SDK) provided for use with the virtual world platform 102. When a player enters a world, the player needs to download that world asset from the world asset database 130. If there are already people in that instance of that world, the client 104 also needs a list of those people's avatars so that the avatars can be rendered in the instance of the virtual world.

[0039] In some embodiments, the functionality of World API 128 can verify that a user account has access to the requested world. A user account should only have the ability to view public worlds in the user interface menu, or only knowledge of the link to a world shared with the user account, but as a redundancy measure, World API 128 can verify that the user account is permitted to access the virtual world.

[0040] In addition to downloading an instance of the virtual world, client 104 can also establish a session with networking service 112 for a specific instance of the world. Networking service 112 can provide information about the current state of the virtual world instance. For example, networking service 112 can provide client 104 with a list of remote avatars 126 that exist within the virtual world instance. Client 104 can then contact avatar API 136 to download complex avatar assets for the list of remote complex avatars from avatar asset database 138.

[0041] If client 104 does not have an asset for local avatar 118, client 104 can also contact avatar API 136 to request and receive local avatar assets. An avatar asset is a single binary file containing all the textures, models, and animation data necessary to render the avatar. In some examples, it may include more complex features such as data on particle systems and light sources, whether the avatar follows or violates the established physical laws in the virtual world, or whether the avatar has non-standard motion dynamics.

[0042] An instance of the downloaded virtual world may be run by client 104 as the current world 120. The current world 120 may include the coordinates within the current world 120 in which the local player 116 and each remote player 124 are located. The local player 116 and remote players 124 are each collision volume of space occupied by the respective local player 116 or remote player 124.

[0043] Local avatars 118 can be mapped to local players 116, and each remote avatar 126 can be mapped to each remote player 124, thereby allowing each player to appear as their own avatar in the current world 120. The movement of the remote avatars 126 is controlled by receiving state data about each remote avatar / player and rendering movement or sound by the client 104.

[0044] The VR tracking service 114 relates to a client 104 that operates on a client device 106 having access to VR tracking peripherals. For example, some VR headsets have cameras (integrated or external) for tracking the player's hands and feet. Many VR headsets can be paired with controllers that can report the position of the player's hands in space. Some client devices 106 include other peripherals configured to perform full skeleton tracking. The VR tracking service 114 can fuse all VR inputs connected to the client.

[0045] The VR tracking service 114 can map the fused VR input to the local player 116, allowing the local player 116 to interact within and with the current world 120. Meanwhile, the local player 116 can interact with the local avatar 118, map the local avatar 118 to the local player, and display the local player 116 as its own avatar.

[0046] In some embodiments, the parts of the player's body tracked by the VR tracking service 114 vary. Some players may have full skeleton tracking, while many may only have the ability to perform hand tracking. To accommodate such differences in the hardware capabilities of possible client devices 106, the local player 116 can derive parts of the skeleton that are not tracked by the VR tracking service 114. For example, even if the VR tracking service 114 only provides information regarding the player's hand tracking, the local player can further derive the player's complete skeleton and move parts of the skeleton in accordance with the hand movements. In this way, the avatar's hands do not move in isolation from the rest of the avatar.

[0047] Local Player 116 is an entity that moves around the environment within the current World 120. It can pick up and place objects. It has no animations and is a collision volume. It can do anything in the world, but has no appearance and does not need to animate.

[0048] The local player is further connected to the networking layer, which is referred to as Runtime Networking Service 122, and broadcasts state information about the local player 116 to other players in the current World 120 instance via the network.

[0049] Local player 116 and remote player 124 are similar in that they are collision volumes that move around within the current world environment 120. The main difference is that local player 116 is controlled by client 104, and the player on client 104 is authoring the experience. In contrast, remote player 124 is a playback mechanism that represents actions broadcast to client 104, representing other players present in the current world 120.

[0050] As mentioned above, local avatar 118 is overlaid on local player 116 to give the player a visual appearance. Actions by local player 116 are animated as the local player interacts with the current world. For example, local player 116 can interact to pick up an object in the current world 120, but without local avatar 118, the object would appear to be floating in mid-air. When local avatar 118 is overlaid on local player 116, the object appears to be held by the avatar's hands.

[0051] The remote player 124 and remote avatar 126 function similarly to their local counterparts, except for the source of the input that controls the remote player 124. The remote player 124 and remote avatar 126 are playback devices for state information received from the networking service 112 by the runtime networking service 122. Figure 1 shows only one remote player 124 and remote avatar 126, but there may be many.

[0052] The current World 120 also has features that require networking. The current World 120 can have objects such as scissors and light switches that players can pick up, and these objects need to broadcast their state over the network so that other players in the current World 120 can see the object's current state.

[0053] Local player 116, current world 120, and remote player 124 are each connected to runtime networking service 122. Local player 116 primarily sends updated state information of local player 116 to a remote instance of client 104, which is also running the same virtual world. Current world 120 can send and receive state information about the virtual world instance. Current world, running on client 104, sends state information when the state change is owned by local player 116 and receives state information when the state change is owned by remote player 124.

[0054] The networking service 112 is the network-side portion of the network layer of the virtual world platform 102. In some embodiments, part of the networking service 112 is provided by a networking plugin, such as the PHOTON networking engine, which broadcasts state information to all players within the virtual world instance.

[0055] In addition to general broadcasting of state information to all players interacting with the virtual world instance, the optimized packet routing service 140 provides more advanced features that enhance the player experience and implement other virtual world platform 102 characteristics such as trust and security settings.

[0056] For example, to provide an enhanced player experience, the optimized packet routing service 140 can filter out voice packets coming from remote players 124 that may be far away from the local player 116 in the current instance of world 120. Without such optimization, remote players 124 that are not interacting with the local player, or are not even visible to the local player, could receive voice packets from tens or even hundreds of remote players 124, making it difficult to communicate with any subset of remote players 124.

[0057] In another example, the optimized packet routing service 140 can implement trust and safety settings. As mentioned above, trust and safety settings can specify a particular user account or group of user accounts, resulting in them being unable to interact with the local player 116 or having limited interaction with the local player 116. The optimized packet routing service 140 can call the trust API 144 to learn a list of remote players 124 that may need to be filtered or blocked by local players 116 with trust and safety settings, which may need to filter or block a portion of the network traffic going to or coming from client 104.

[0058] The Trust API 144 can determine which remote players 124 should be blocked from the local player 116, or which remote players 124 should have their complex avatar configurations restricted. Some of these decisions are based on logic and rules that classify remote players 124 based on the amount and type of their past interactions with the virtual world platform 102. The Trust API 144 can make these decisions by using the settings stored in the local player 116's user profile and comparing these settings with data stored in the remote player 124's user profile.

[0059] Another networking service 112 is a moderation service 142 that can provide conflict resolution and access control. For example, before a player can access a world, especially a private world, the moderation service 142 can call the world API 128 to ensure that the player can enter the world. In another example, two different players may attempt to claim control of an object in a virtual world at almost the same time. The moderation service 142 can handle these types of conflicts by selecting a specific player to control the object until that player relinquishes control of the object, thereby allowing the other player to claim control of the object. The player who has control of the object can broadcast a packet informing the remote player 124 of the state of that object.

[0060] In some embodiments, the client 104, virtual world, and complex avatar may be configured to operate in a specific game engine, particularly one that supports three-dimensional (3D) environments. Two common game engines include Unity and Unreal Engine.

[0061] In some embodiments, the virtual world and complex avatars must be developed in accordance with the Software Development Kit (SDK) in order to be supported by the virtual world platform 102. For example, a complex avatar may require specific scripts to be usable on the virtual world platform 102. In another example, there may be many requirements that must be followed to play the avatar's animations. In some embodiments, the SDK may define other necessary details to support specific client devices. For example, the SDK may define specific shaders to be used when the avatar is used with the OCULUS QUEST VR headset.

[0062] In some embodiments, the SDK requires that virtual worlds utilize a specific coding language to ensure that the worlds exhibit compliant behavior. For example, the SDK may require that behavior within a world be defined using UDON, a programming language specific to a particular virtual world platform 102, VRCHAT. In some embodiments, the programming language facilitates the construction of worlds using the programming language to comply with file access protections provided by the virtual world platform 102. For example, a world may not be able to read or write anything to the hard drive, and only authorized web pages may be rendered in a world on the virtual world platform 102.

[0063] In some embodiments, the virtual world platform 102 may also include a simplified avatar service 146. As described herein, the simplified avatar service 146 can create simplified versions of complex avatars and store the avatar assets of the simplified versions of complex avatars in an avatar asset database 138.

[0064] While the virtual world platform 102 is suitable for implementing this technology, those skilled in the art will understand that this technology can be used in other environments.

[0065] Figure 2 shows an exemplary quick menu 202 according to several embodiments of the present technology. In particular, the quick menu 202 can be surfaced by the user interface service 108 on the client 104 at any time or place on the virtual world platform 102.

[0066] The Quick Menu 202 includes a Quick Links 204 section containing many commonly used menu options, such as menus for browsing the world, avatars, and friends, and a Safety Menu 208 for configuring the user's profile security settings.

[0067] The Trust and Safety menu 208 provides a user account with the ability to determine which remote players 124 can see the user's avatar (local player 116) or can be seen by the player's avatar when both are in the same world. For example, it may be desirable to avoid interacting with newer players on the virtual world platform 102 because they have not yet established a trust relationship within the virtual world platform 102. It may also be desirable to restrict the functionality of a remote player's avatar as it is handled by the instance of client 104 to which the local player is logged. This is because some avatars may have malicious data embedded in them, or the avatars may be too complex to render without degrading the performance of the client device 106. For example, a user account may decide to turn off the lights on a remote avatar to avoid shaders, or not allow custom animations. In some embodiments, each of these options may be set based on how trusted the remote player is. For example, a user account may allow a friend's avatar to have full functionality, while other user accounts may only be able to see basic avatar functionality.

[0068] The user interface service 108 can also provide the option to mute or block specific remote players. Furthermore, the user interface service 108 can provide a panic or safe mode 210 that mutes all non-friends both audibly and visually.

[0069] Quick Menu 202 may also include a Quick Actions 206 section to provide frequently used actions in convenient locations. Some example Quick Actions include actions to go to your home world, actions to respawn in the world you were last in, actions to select another player's avatar (to communicate privately, to prevent players from seeing or talking to local players 116, to copy avatars or other features), and actions to select emojis.

[0070] The Quick Menu 202 also includes a docking station 212 that provides access to several common functions, including a virtual camera, volume settings, and a settings menu, among other features.

[0071] Figure 3 shows an example of the relationship between gestures and the instantiation and operation of virtual controllers.

[0072] In the upper image of Figure 3, the client device 106 recognizes a pinch gesture made by the player's hand and sends a description of the player's hand position in the room space, along with information about the player's finger positions and the recognized gesture with a confidence score associated with the gesture, to the client 104. The VR tracking service 114 can map the player's hand position to an offset 302. The offset 302 is approximated as the position of the avatar's palm relative to the character root. Therefore, when the player controls the avatar to reach for something or point at something, the offset is a suitable place to place the hand near the avatar's hand. The offset is set to work well with multiple avatars, regardless of the avatar's scale (size). The offset 302 can be used to map a hand of any size on a player of any size to a hand of any size on an avatar of any size. The offset 302 acts as a reference point from which the avatar's hand 304 can be drawn.

[0073] In response to receiving information from client device 106 that the hand is in a pinch gesture pose, the VR tracking service 114 instructs client 104 to instantiate a virtual controller at the location where the pinch gesture was created. The virtual controller may include a larger control surface, as indicated by its periphery 306, although the control surface may be transparent. Furthermore, the virtual controller includes a joystick instantiation point at the initial pinch point 308. This joystick instantiation point may be a visible dot or other indicator that informs the player that the joystick is active.

[0074] In the lower diagram of Figure 3, the player's hand has moved from its original position in the room space to a second position in the room space. The client device 106 can recognize that the player's hand has moved while maintaining the pinch gesture pose. The client device 106 can again report data regarding the hand position, finger positions, gesture recognition, and the confidence that the client device 106 correctly recognized the gesture to the VR tracking service 114. The VR tracking service 114 can move the avatar hand 304 along with the offset 302, thereby also repositioning the pinch point to the current pinch position 310. In response, the client 104 can indicate that a movement input has been received by illustrating the joystick 312, starting from the joystick instance point at the initial pinch point 308 of the virtual controller and ending at the current pinch position 310 of the avatar hand 304.

[0075] In some embodiments, the joystick 312 has a slope or wedge shape used to visually indicate the speed of the avatar's movement. The further the current pinch position 310 is from the joystick's instantiation point of the initial pinch point 308, the faster the avatar's movement speed. This is indicated by a larger wedge shape.

[0076] The movement of the virtual controller may be constrained to a plane composed of the x and y axes (i.e., forward, backward, and lateral movement, but not up and down).

[0077] Figure 4A shows an exemplary routine for controlling the movement of an avatar through finger tracking. The exemplary routine shows a specific sequence of operations, but the sequence can be changed without departing from the scope of this disclosure. For example, some of the illustrated operations can be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of an exemplary device or system implementing this routine can perform functions substantially simultaneously or in a specific order.

[0078] In block 402, the player can generate a virtual controller by performing a pinch gesture.

[0079] The client device 106 may be equipped with finger tracking technology. Generally, the client device 106 may be a VR headset that includes one or more cameras capable of capturing images of the player's fingers. The cameras may be physically integrated with the virtual reality headset or be external to the VR headset and communicatively coupled to it. The VR headset may also include software that can map the position of the player's hands and fingers in room space (real world space) to their position in world space (virtual world space). The VR headset can report information to the client 104, such as the position of the hands and fingers, the position of the fingers, recognized gestures, and the confidence that the gestures were correctly identified, and the client 104 receives this information through the VR tracking service 114.

[0080] Using information received by the VR tracking service 114 from the client device 106 (e.g., a VR headset), the client 104 can generate a virtual controller in 404, which is made up of a circular region whose center is visible to the player. In some examples, only the center of the circular region is visible to the player. In some examples, the center of the circular region is a joystick for controlling the movement of the avatar.

[0081] Client 104 can position a virtual controller at a pinch point in world space within block 406. The pinch point is the location in world space to which the player's hand and fingers in room space are mapped. In some examples, the VR tracking service 114 maps the hand position in room space to a location in world space, and the avatar's finger positions are approximated from the hand position in world space.

[0082] In block 408, client 104 can store the offset in world space for the position of the hand relative to the character root in the virtual world.

[0083] The complexity of controlling the movement of an avatar in the virtual world via hand and finger tracking of the player's hands in the room space lies in the fact that while the avatar changes its position in the virtual world, the player does not change its position in the room space; that is, the player only moves their hands. Therefore, if the avatar moves but the player does not, the player's hands can be mapped to a position in world space other than the avatar's position. This can give the impression that the avatar has left the virtual controller behind.

[0084] This phenomenon may be a result of the fact that the client device can report the position of the fingers performing the gesture in the room space, which fingers are involved in the gesture, a confidence value indicating that the detected gesture is correctly identified, the position of the hand in the room space, and the rotation of the hand. The virtual controller is instantiated in the virtual world at its corresponding position in world space, but the avatar immediately moves from that position.

[0085] To overcome this complexity, the technology allows the joystick's position to change in accordance with the avatar's movement. This can become complex when the avatar is rotating and moving in multiple directions simultaneously, such as when it is moving sideways while simultaneously rotating. A further level of complexity arises not only from changing the virtual controller's position, but also from maintaining the virtual controller's position relative to the avatar in a way that provides a good player experience.

[0086] This technology can receive data on hand and finger position, gestures, confidence levels, and finger rotation from a VR client device with hand and finger tracking capabilities, and map the data recorded in the room space to the world space.

[0087] An avatar can have a root and a base, which together constitute the character root. The character root can be an invisible structure in world space to which the visible avatar is mapped. This technology can determine an offset transformation from the avatar's root position in world space, and can initially position a virtual controller at an offset location from the avatar's root position. The offset is approximated as being near the avatar's palm. Therefore, when a player controls the avatar to reach for something or point at something, the offset will be in a good location near the avatar's hand. The offset is set to work well with multiple avatars, regardless of the avatar's scale (size).

[0088] As mentioned above, the space in which hand and finger tracking takes place is relative to the player in the room space, but the avatar moves through the virtual world (also known as world space). To compensate for this, this technology takes a snapshot of the relative position of the player's hands to the avatar's position in world space. The snapshot is recorded as a transformation from the avatar's root position to the initial position of the virtual controller.

[0089] Client 104 can store conversions in block 410 for mutually converting between world space and room space.

[0090] The transformation between the room space and the avatar in world space takes into account the point the player selected when pinching in the room space, which is mapped to an offset from the character root. As the avatar's orientation and position change, the transformation is used to move the overall relative position as well. The transformation utilizes matrix mathematics to transform between the room space and the relative position based on the avatar's root.

[0091] Client 104 stores the world position in world space in block 412, converting it to room space using a transformation. Also, in block 414, Client 104 stores the player's current forward vector in world space as "Look Rotation," converts it to room space, and stores it as ORIG_ROTATION.

[0092] In summary, client 104 encodes the character root's position in world space and the player's position in room space, takes a snapshot of the position for the transformation, and saves the transformation. As the avatar moves, the transformation moves with the player, so the transformation moves with the player as well.

[0093] This transformation allows the placement of the controller in the virtual reality environment relative to the character root position to which the avatar is mapped, while tracking the player's hand and finger movements in the real world environment. Therefore, when the avatar moves or rotates, the snapshotted transformation moves with the avatar. When the player moves their fingers in room space relative to the initial position where the virtual controller was instantiated in world space (the position where the gesture was made and the virtual controller was instantiated), client 104 measures the distance of movement based on the movement in the real world. At the same time, client 104 moves the center position of the virtual controller relative to the base of the character root, transforming the hand movement in room space into world space. In this way, even if the player moves the avatar's head relative to the character root (the avatar's base), the position of the virtual controller relative to the character root is maintained. Therefore, even if the avatar's head is rotated relative to the avatar's base and can see in a different direction than the direction the avatar is moving, the movement operation remains relative to the avatar's character root. The same behavior applies whether the movement is rotation, forward movement, lateral movement, or any combination thereof. The same behavior also applies to other dimensions that may allow for roll, pitch, and yaw, facilitating lift, decent, turn, etc.

[0094] Figure 4B shows an exemplary routine for controlling avatar movement, particularly directional movement, using a virtual controller for directionality, through finger tracking. Although Figure 4B is shown as a continuation of the routine shown in Figure 4A, it should be understood that the routine in Figure 4B can be executed independently of the routine in Figure 4A. Figure 4B may still rely on a transformation from room space to world space, but the transformation does not need to be created as illustrated in Figure 4A. The exemplary routine shows a particular sequence of operations, but the sequence can be changed without departing from the scope of this disclosure. For example, some of the illustrated operations may be executed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of an exemplary device or system implementing this routine may perform functions substantially simultaneously or in a particular order.

[0095] A virtual controller may be presented in a virtual world associated with the player's avatar. As mentioned above, only the center of the virtual controller may be visible. The center of the virtual controller may look like a joystick, and as a result, the player can move their hand to control the avatar's hands as if pushing a joystick in a certain direction. While maintaining a pinch gesture or other gesture used to invoke the virtual controller, the player can move their hand to control the movement of their avatar.

[0096] As described above, one type of virtual controller is a directional controller that can be used to provide input for moving the avatar forward, backward, and sideways to the right or left, and another type of virtual controller is a rotation controller that can be used to provide movement for rotating the avatar to the right or left. In some embodiments, the directional controller is associated with the left hand and the rotation controller is associated with the right hand. In some embodiments, the controllers may be on the opposite hand (e.g., the directional controller in the right hand) or on the same hand, and different gestures are used to distinguish the type of movement. In some embodiments, the controllers may be controlled by other parts of the player's body (such as the feet or hips) by tracking these body parts in room space.

[0097] As described above, the routine includes detecting the player's hand movements associated with the directional controller in block 416 while the hand maintains the gesture. Client device 106 can detect the player's hand movements and that the hand continues to maintain the gesture, and can report the player's hand position over time in the room space to the VR tracking service 114 of client 104. Client 104 can then use the previously determined transformation to position the avatar's hand in world space.

[0098] This routine further includes interpreting the player's hand position in block 418 to determine the avatar's movement speed and direction. For example, client 104 can determine the offset of the player's hand in room space from the initial pinch point in world space and interpret that offset as the movement speed. The further the player's hand is from the initial pinch point in world space, the greater the avatar's movement speed. In some embodiments, client 104 can utilize a function to determine the movement speed with distance as a variable.

[0099] The speed of movement controlled by the directional virtual controller can be in any direction. Client 104 can determine the direction of movement by determining the vector from the initial pinch point in world space to the position of the player's hand in room space and resolving the direction of movement. In the case of a directional controller, the movement of the avatar can be the direction of the vector.

[0100] Therefore, client 104 can determine the direction and speed of the avatar's movement. The direction of movement is the direction of the vector from the initial pinch point in the room space to the position of the player's hand in the room space, and the speed is a function of the length of the vector (the distance from the initial pinch point in the room space to the position of the player's hand in the room space).

[0101] As the avatar moves through world space, tracking transformations are used to update orig_position (the center of the virtual controller) to maintain the avatar's relative position to the character root to which it is mapped. This routine includes moving the character root based on input to the virtual controller in block 418, and updating the position of the virtual controller's center using tracking transformations in block 422 to maintain the avatar's relative position to the character root to which it is mapped. For example, client 104 can move the character root based on input to the virtual controller and update the position of the virtual controller's center using tracking transformations to maintain the avatar's relative position to the character root to which it is mapped.

[0102] The input to keep the avatar moving is applied until the player returns their hand to its original position (initial pinch point) in the room space where the virtual controller was instantiated, or until the player stops the gesture used to instantiate the virtual controller.

[0103] As described above, the input for moving the avatar is obtained from the player's movements in the room space. The virtual controller is displayed on the avatar's hand in world space to provide an intuitive feedback mechanism to demonstrate that client 104 has properly interpreted the input (in addition to the avatar's movements in world space). The fact that the input for moving the avatar is obtained from the player's movements in the room space has the advantage that the input given to the avatar is provided on a human scale. The avatar can be small or huge in world space. Therefore, the same amount of movement in room space will result in the same amount of movement for an avatar of any size, but the display of the virtual joystick can be scaled to the avatar's size.

[0104] Figure 4C shows an example routine for controlling the movement of an avatar through finger tracking, particularly for controlling directional movement using a rotational virtual controller. Although Figure 4C is shown as a continuation of the routine shown in Figure 4A, it should be understood that the routine in Figure 4C can be executed independently of the routine in Figure 4A. Figure 4C may still depend on a transformation from room space to world space, but the transformation does not need to be created as illustrated in Figure 4A. Furthermore, although Figures 4B and 4C are illustrated separately, it should be understood that both routines can be executed simultaneously, and in some embodiments they can be executed by the same hardware or software object. The exemplary routines show a specific sequence of operations, but the sequence can be changed without departing from the scope of this disclosure. For example, some of the illustrated operations can be executed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the exemplary device or system implementing this routine may perform functions substantially simultaneously or in a specific order.

[0105] A virtual controller may be presented in a virtual world associated with the player's avatar. As mentioned above, only the center of the virtual controller may be visible. The center of the virtual controller may look like a joystick, and as a result, the player can move their hand to control the avatar's hands as if pushing a joystick in a certain direction. While maintaining a pinch gesture or other gestures used to invoke the virtual controller, the player can move their hand to control the movement of their avatar.

[0106] As described above, one type of virtual controller is a directional controller that can be used to provide input for moving the avatar forward, backward, and sideways to the right or left, and another type of virtual controller is a rotation controller that can be used to provide movement for rotating the avatar to the right or left. In some embodiments, the directional controller is associated with the left hand and the rotation controller is associated with the right hand. In some embodiments, the controllers may be on the opposite hand (e.g., the directional controller in the right hand) or on the same hand, and different gestures are used to distinguish the type of movement. In some embodiments, the controllers may be controlled by other parts of the player's body (such as the feet or hips) by tracking these body parts in room space.

[0107] As described above, this routine includes detecting the player's hand movements associated with the rotation controller in block 424 while the hand maintains the gesture. The client device 106 can detect the player's hand movements and that the hand continues to maintain the gesture, and can report the player's hand position in the room space over time to the VR tracking service 114 of client 104. Client 104 can then use the previously determined transformation to position the avatar's hand in world space.

[0108] This routine further includes restricting the joystick movement to the x-axis in block 426. For example, client 104 can determine that the player's hand movement (while maintaining the gesture) is on the positive or negative side of the virtual controller's origin (the initial pinch point in room space). The player may move their hand and the avatar's hand may move accordingly in both the X and Y directions, but client 104 can restrict the joystick movement to the X-axis only. In some embodiments, the rotation controller may be restricted to other axes (for example, rotation may also be pitched up and down based on the z-axis). In some embodiments, the rotation virtual controller may not be restricted to any axis and may allow 6-degree-of-freedom rotation.

[0109] This routine further includes interpreting the player's hand position in block 428 to determine the rotation speed and the rotation of the movement. For example, client 104 can determine the offset of the player's hand in room space from the initial pinch point in room space and interpret that offset along the x-axis as the rotation speed. The further the player's hand is from the initial pinch point in room space along the x-axis, the greater the rotation speed of the avatar. In some embodiments, client 104 can utilize a function to determine the rotation speed with distance as a variable.

[0110] The rotation speed controlled by the rotation virtual controller can be in any direction. The example given here rotates left and right based on the x-axis, but client 104 can determine the direction of movement and resolve it by determining the vector from the initial pinch point in world space to the position of the player's hand in room space. However, the rotation controller described above is limited to the x-axis component of the vector. In the case of the rotation controller, the movement of the avatar can be done in the direction of the vector along the x-axis (i.e., right rotation or left rotation).

[0111] Therefore, client 104 can determine the direction and speed of the avatar's movement. The direction of movement is the direction of the vector along the X-axis from the initial pinch point in room space to the position of the player's hand in room space, and the speed is a function of the length of the vector along the X-axis (the distance from the initial pinch point in world space to the position of the player's hand in room space).

[0112] When the avatar rotates in world space, tracking transformations are used to update orig_position (the center of the virtual controller) to maintain the avatar's relative position to the character route to which it is mapped. This routine includes, in block 430, rotating the character route based on the virtual controller input, and in block 432, updating the position of the virtual controller's center using tracking transformations to maintain the avatar's relative position to the character route to which it is mapped. For example, client 104 can rotate the character route based on input to the virtual controller given by moving the player's hand in room space, and can use tracking transformations to update the virtual controller's center position to maintain the avatar's relative position to the character route to which it is mapped.

[0113] The input to keep the avatar moving is applied until the player returns their hand to its original position in the room space where the virtual controller was instantiated, or until the player stops the gesture used to instantiate the virtual controller.

[0114] As described above, the input for moving the avatar is obtained from the player's movements in the room space. The virtual controller is displayed on the avatar's hand in world space to provide an intuitive feedback mechanism to demonstrate that client 104 has properly interpreted the input (in addition to the avatar's movements in world space). The fact that the input for moving the avatar is obtained from the player's movements in the room space has the advantage that the input given to the avatar is provided on a human scale. The avatar can be small or huge in world space. Therefore, the same amount of movement in room space will result in the same amount of movement for an avatar of any size, but the display of the virtual joystick can be scaled to the avatar's size.

[0115] Figure 5 shows the relationship between the player in room space 506 and the avatar in world space 508. The area surrounding the player in room space 506 is the area where the player's movements are tracked. As the player moves their hands across the room space, the hand and finger tracking client device 106 can record the position and orientation of the player's hands and fingers and provide input indicating the recognized gestures and the positions of those gestures to the client 104 that renders the virtual world and the avatar.

[0116] Client 104 stores the world position in world space 508, converted using the transformation described above, in room space 506. Client 104 also stores the player's current forward vector 502 in world space 508 and converts it to "Look Rotation" based on ORIG_ROTATION 504. As described herein, one or more transformations can be used to map the player's hand position in world space 508 to a relative position of the character root in room space 506.

[0117] Figure 6 shows an example of an avatar that performs a pinching gesture with its left hand to instantiate a virtual controller.

[0118] Figure 6 shows an exemplary world space 602 as seen from a first-person perspective by a player controlling an avatar. The avatar's left hand 604 performs a pinch pose using its ring finger and thumb, mirroring the same pose performed by the player's left hand in room space. As a result of the detected gesture, client 104 instantiates a virtual controller indicated by a joystick 606. The joystick 606 is indicated by a dot shown near the intersection of the tip of the avatar's thumb and the tip of the avatar's ring finger. Furthermore, client 104 renders a directional arrow 608 to instruct the player to move the avatar's left hand 604 forward or backward, which controls the avatar to move forward or backward. Although the directional arrow 608 only indicates forward or backward movement, the hand can move in any direction, and the movement of the left hand can be used to provide input to move the avatar forward, backward, and sideways to the right or left.

[0119] Figure 7 shows an example of using a virtual controller to make the avatar walk forward.

[0120] For example, the player inputs by moving the avatar's left hand 604 slightly forward from the controller's initial position to the controller's current position, as indicated by the magnified joystick 606. The avatar can walk in the direction of a vector that points from the controller's initial position toward the controller's current position. The avatar can walk at a speed relative to the distance between the controller's initial position and the controller's current position, or a speed proportional to it. While the avatar moves around the virtual world, the controller's initial position remains in the same relative position with respect to the character root.

[0121] Figure 7 shows the world space 602 after the avatar has moved forward in world space 602 in response to the input given to the virtual controller.

[0122] Figure 8 shows an example of rotating an avatar to the left using a virtual controller.

[0123] For example, after a player generates a pinch gesture to instantiate a virtual controller, the player provides input to the avatar to move its hand to the left, as indicated by the magnified joystick 804, from the controller's initial position to the controller's current position. In some embodiments, if the virtual controller is configured to provide commands to rotate the avatar left or right, the virtual controller's joystick 804 may be restricted to the left or right direction. The avatar can rotate in the direction of a vector that points from the controller's initial position toward the controller's current position. The avatar can rotate at a speed relative to the distance between the controller's initial position and the controller's current position, or at a speed proportional thereto. While the avatar rotates in the virtual world, the controller's initial position remains in the same relative position with respect to the character root.

[0124] Figure 9 shows an example of using a virtual controller to rotate an avatar to the right in the opposite direction to that shown in Figure 8. Figure 9 shows world space 602 after the avatar has moved to the right in world space 602 in response to the input given to the virtual controller.

[0125] Figure 10 shows an avatar performing pinching gestures simultaneously with the left hand 604 and the right hand 802. Virtual controllers are instantiated in each hand, as indicated by joysticks 606 and 804, respectively, shown for the left hand 604 and the right hand 802. In some embodiments, the movement of the left hand 604 can be used to provide input for moving the avatar forward, backward, and laterally to the right or left, and the movement of the right hand 802 can be used to provide movement for rotating the avatar to the right or left. Input can be given to both virtual controllers simultaneously.

[0126] Figure 11 shows an example of using the virtual controllers 604 (left hand) and 802 (right hand) simultaneously to make the avatar walk forward and rotate to the right, as described above.

[0127] This technology can be used for many different movements, enabling more complex commands such as jumping, or combinations of actions like jumping while running or moving sideways while spinning. This technology can also be used to create ports, such as teleportation or holographic porting (holoport), to move avatars from one location to another, or from one world to another.

[0128] This technology can also be used to control adaptive technology, or in conjunction with adaptive technology. For example, some players of virtual reality may experience motion sickness, and adaptive technology can alter the animation and perception of movement to mitigate this motion sickness. In another example, this technology can be used to alter the horizon of the field of view to enable a virtual reality experience for people lying in bed. In yet another example, this technology can be used to maintain the hand positions used in sign language.

[0129] This technology can also be used to generate avatar effects, such as allowing avatars to change shape or to enlarge or shrink. Other effects include muting and unmuting, and enabling facial expressions and emotes.

[0130] In some embodiments, each image in Figures 6 to 11 can be considered a frame in a sequence of animations.

[0131] Figure 12 shows an exemplary routine for controlling the movement of an avatar through finger tracking. The exemplary routine shows a specific sequence of operations, but the sequence can be changed without departing from the scope of this disclosure. For example, some of the illustrated operations can be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of an exemplary device or system implementing this routine can perform functions substantially simultaneously or in a specific order.

[0132] According to some examples, the method includes receiving in block 1202 a description of a first gesture performed by one or more fingers of the player, and the initial spatial position of the one or more fingers performing the first gesture. For example, client 104 shown in Figure 1 can receive from the tracking service of client device 106 a description of a first gesture performed by one or more fingers of the player, and the initial spatial position of the one or more fingers performing the first gesture.

[0133] According to some examples, the method includes determining the relative positions of one or more fingers performing a first gesture with respect to a pinch start point detected in the room space in block 1204. For example, the client 104 shown in Figure 1 can determine the relative positions of one or more fingers performing a first gesture with respect to a pinch start point detected in the room space.

[0134] According to some examples, this method includes creating an instance of a virtual controller at an offset position relative to the character root in block 1206. For example, client 104 shown in Figure 1 can create an instance of a virtual controller at an offset position relative to the character root. Creating an instance of a virtual controller at an offset position further includes positioning the center of the virtual controller at the offset position and displaying the virtual controller user interface at the offset position.

[0135] According to some examples, this method includes displaying the rendering of the avatar's hand at an offset position within the hand rendering, along with the virtual controller's joystick, in block 1208. For example, client 104 shown in Figure 1 can display the rendering of the avatar's hand at an offset position within the hand rendering, along with the virtual controller's joystick.

[0136] According to some examples, the method includes receiving, in block 1210, the updated position in the room space of one or more fingers performing the first gesture from the tracking service. For example, the client 104 shown in Figure 1 can receive from the tracking service the updated position in the room space of one or more fingers performing the first gesture.

[0137] According to some examples, the method includes determining the distance and direction of the updated position in the room space of one or more fingers performing the first gesture, compared with the initial position in the room space of one or more fingers performing the first gesture, in block 1212. For example, the client 104 shown in Figure 1 can determine the distance and direction of the updated position in the room space of one or more fingers performing the first gesture, compared with the initial position in the room space of one or more fingers performing the first gesture.

[0138] According to some examples, this method includes displaying an animation of a hand moving from an initial position (initial offset position) to an updated position (updated offset position) in block 1214. For example, client 104 shown in Figure 1 can display an animation of a hand moving from an initial position to an updated position.

[0139] According to some examples, the method includes displaying a movement input indicator in block 1216 in conjunction with the animation of a hand moving from an initial offset position to an updated position. For example, client 104 shown in Figure 1 can display a movement input indicator in conjunction with the animation of a hand moving from an initial position to an updated position. The movement input indicator may be a magnified joystick, as described above.

[0140] In some examples, this method involves controlling the avatar's movement in block 1218 based on the distance and direction of the player's current position in room space compared to the player's initial position in room space. For example, client 104, illustrated in Figure 1, can control the avatar's movement based on the distance and direction of the player's current position in room space compared to the player's initial position. In this example, the instance of the virtual controller is a linear motion controller. The avatar's movement is a linear direction of a vector that originates from the player's initial position and points in the direction of the player's updated position. The avatar's movement is a velocity corresponding to the distance between the player's initial position and the player's updated position, with a larger distance between the initial and updated positions corresponding to a greater velocity than when the distance between the initial and updated positions is smaller.

[0141] If the virtual controller instance is a rotational motion controller, the avatar's movement is in the rotational direction of a vector that points from the initial position of the player's hand in room space to the direction of the updated position of the player's hand, and the avatar's movement is a velocity corresponding to the distance between the initial position of the player's hand and the updated position of the player's hand, with a larger distance between the initial and updated positions corresponding to a greater velocity than when the distance between the initial and updated positions is smaller.

[0142] While the virtual controller is illustrated and described throughout this explanation as a virtual joystick, those skilled in the art will understand that the virtual controller can be of other types. The virtual controller can be represented by movement indicators, virtual buttons, or other UI objects. A joystick is just one example of a virtual controller. Furthermore, this technique can be utilized without displaying a virtual controller. Input from gestures can be provided to influence gameplay without displaying a virtual controller.

[0143] While this specification has focused on two types of movement controllers (i.e., directional controllers and rotation controllers), it will be understood by those skilled in the art that the use of virtual controllers and / or gestures to provide input for controlling an avatar or gameplay is not limited to movement. For example, other inputs that may be provided by a player performing a gesture include grabbing an object, interacting with an object, opening or closing a menu, muting or unmuting a microphone, canceling an input, dropping an object, jumping, or providing a gesture to lock out other gestures so that they are not interpreted as commands to do anything other than the position of the avatar's hands and fingers.

[0144] This specification defines the pinch gesture as a valid gesture for instantiating a virtual controller, but it should be understood that other gestures can be defined to instantiate this or other functions. That is, a different gesture can be used to instantiate a move controller, and a specific pinch gesture can be remapped to a different function.

[0145] As an example, here's a mapping between some features and gestures. 1. Interact / Grab - Pinch with your left / right thumb and index finger, with your palm facing away from the user. 2. Open / close the quick menu (e.g., Quick Menu 202) - Pinch with your left thumb and index finger while facing your palm towards the user. 3. Movement - Pinch and hold with your left thumb and middle finger. 4. Mute / Unmute the microphone - Pinch and hold with your left thumb and ring finger. 5. Cancel / Drop - Pinch with your left / right thumb and pinky finger. 6. Open / Close the Oculus Menu - Hold your right thumb and index finger with your palm facing the user. 7. Rotate - Pinch and hold with your right thumb and middle finger. 8. Jump - Pinch with your right thumb and ring finger. 9. Gesture Lock - Touch with your left thumb and little finger to lock the gesture. All movement actions will be disabled until the gesture lock (touching with your left thumb and little finger together) is performed again.

[0146] As mentioned above, these gestures may be rebind to other controls.

[0147] Many of the gestures mentioned above can be performed simultaneously, allowing for multiple functions to be achieved at the same time. For example, directional movement, rotation, and jumping can all be performed simultaneously.

[0148] Gesture lock can be particularly useful in gameplay environments where players control their avatar in a conversational setting and instruct it to perform other actions. Since many people "speak with their hands," and some hearing-impaired players actually communicate using sign language, it may be desirable to prevent gestures from being mapped to other functions. This allows players to control their avatar during a conversation and perform various hand gestures without having to move, mute, or open menus. Players can deactivate gesture lock when they have completed the conversation or wish to re-enable the mapping of gestures to functions.

[0149] Figure 13 shows an example of a computing system 1300, which may be any computing device or system component comprising, for example, a client device 106, a web service 110, or a networking service 112, communicating with each other using connection 1302. Connection 1302 can be a physical connection via a bus or a direct connection to a processor 1304, such as in a chipset architecture. Connection 1302 may be a virtual connection, a network connection, or a logical connection.

[0150] In some embodiments, the computing system 1300 is a distributed system in which the functions described in this disclosure can be distributed within a data center, across multiple data centers, across a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functions described. In some embodiments, the components may be physical or virtual devices.

[0151] An exemplary computing system 1300 includes at least one processing unit (CPU or processor) 1304 and a connection 1302 that connects various system components to the processor 1304, including system memory 1308 such as read-only memory (ROM) 1310 and random access memory (RAM) 1312. The computing system 1300 may include a cache of high-speed memory 1306 that is directly connected to the processor 1304, connected in close proximity to the processor 1304, or integrated as part of the processor 1304.

[0152] The processor 1304 may include any general-purpose processor, and hardware services configured to control the processor 1304, or software services such as services 1316, 1318, 1320 stored in the memory device 1314, and may further include application-specific processors in which software instructions are incorporated into the actual processor design. The processor 1304 may essentially be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. The multicore processor may be symmetric or asymmetric.

[0153] To enable interaction with the player, the computing system 1300 includes an input device 1326 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, and voice. The computing system 1300 may also include an output device 1322 that may be one or more of several output mechanisms known to those skilled in the art. In some examples, a multimodal system may allow the player to provide multiple types of inputs / outputs for communicating with the computing system 1300. The computing system 1300 may generally include a communication interface 1324 that can control and manage the player's inputs and system outputs. There are no constraints on operation with a particular hardware configuration, and therefore the basic functions described here can be easily replaced with improved hardware or firmware configurations as they are developed.

[0154] The storage device 1314 may be a non-volatile memory device, or it may be another type of computer-readable medium capable of storing computer-accessible data, such as a hard disk, magnetic cassette, flash memory card, solid-state memory device, digital multipurpose disk, cartridge, random access memory (RAM), read-only memory (ROM), and / or some combination of these devices.

[0155] The storage device 1314 may include software services, servers, and other services, and when the code defining such software is executed by the processor 1304, it causes the system to perform functions. In some embodiments, a hardware service that performs a particular function may include software components stored in a computer-readable medium connected to the hardware components necessary to perform that function, such as the processor 1304, connection 1302, output device 1322, etc.

[0156] To clarify the explanation, in some examples, the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or functional blocks that include combinations of hardware and software.

[0157] Any of the steps, operations, functions, or processes described herein may be performed or implemented, alone or in combination with other devices, by a combination of hardware and software services(s). In some embodiments, a service may be software that resides in the memory of one or more servers of a client device and / or content management system, and performs one or more functions when a processor runs the software associated with the service. In some embodiments, a service may be a program or a collection of programs that perform a particular function. In some embodiments, a service may be considered a server. Memory may be a non-temporary computer-readable medium.

[0158] In some embodiments, computer-readable storage devices, media, and memory may include cable or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0159] The methods according to the embodiments described above can be implemented using computer-executable instructions stored from or otherwise available on a computer-readable medium. Such instructions may include instructions and data that cause a general-purpose computer, a purpose-specific computer, or a purpose-specific processing device to perform or otherwise configure a particular function or group of functions. Some of the computer resources used may be accessible over a network. The executable computer instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store the instructions used, the information used, and / or the information generated in the process of the methods according to the embodiments described include magnetic or optical disks, solid-state memory devices, flash memory, USB devices with non-volatile memory, network-connected storage devices, and the like.

[0160] Devices implementing the methods described herein may comprise hardware, firmware, and / or software and may take on any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, and personal digital assistants. The functions described herein can also be implemented in peripherals or expansion cards. Such functions may also be implemented on a circuit board, in a further exemplary manner, across different chips or different processes running in a single device.

[0161] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0162] Embodiments of this technology can be further understood from the following sections.

[0163] Paragraph 1. A method for controlling the movement of an avatar via finger tracking, comprising: receiving from a tracking service a description of a first gesture made by one or more fingers of a player, and the initial spatial positions of the one or more fingers making the first gesture; determining the relative position of the one or more fingers making the first gesture with respect to a character root, where the character root is a volume in a virtual world to which the avatar representing the player is mapped; and generating an instance of a virtual controller at the relative position.

[0164] Paragraph 2. The method of Paragraph 1, wherein generating the instance of the virtual controller at the relative position further comprises locating the center of the virtual controller at the relative position and displaying the virtual controller user interface at the relative position.

[0165] The method according to any one of the first to second paragraphs, comprising: displaying a rendering of the avatar's hand at the relative position, together with the virtual controller within the rendering of the hand; determining the distance and direction of the updated position in space of the one or more fingers performing the first gesture, compared with the initial position in space of the one or more fingers performing the first gesture; displaying an animation of the hand moving from the relative position in the direction of the updated position; and displaying an indicator of movement input in conjunction with the animation of the hand moving from the relative position in the direction of the updated position.

[0166] Paragraph 4. The method according to any one of paragraphs 1 to 3, further comprising: receiving an updated position in the space of one or more fingers performing the first gesture from the tracking service; determining the distance and direction of the updated position in the space of one or more fingers performing the first gesture by comparing it with the initial position in the space of one or more fingers performing the first gesture; and controlling the movement of the avatar based on the distance and direction of the updated position by comparing it with the initial position.

[0167] Paragraph 5. The method according to any one of Paragraphs 1 to 4, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that starts from the initial position and points in the direction of the updated position.

[0168] Paragraph 6. The method according to any one of Paragraphs 1 to 5, wherein the instance of the virtual controller is a linear motion controller, and the movement of the avatar is a velocity corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the velocity that corresponds to compared with the case where the distance between the initial position and the updated position is small.

[0169] Paragraph 7. The method according to any one of Paragraphs 1 to 6, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that starts from the initial position and points in the direction of the updated position.

[0170] The method according to any one of the first to seventh paragraphs, wherein the instance of the virtual controller is a rotational motion controller, and the movement of the avatar is a velocity corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the velocity that corresponds to compared with the case where the distance between the initial position and the updated position is small.

[0171] Paragraph 9. A computing system including a processor and a memory for storing instructions, which, when executed by the processor, configures the system to perform any of the methods described in paragraphs 1 to 8.

[0172] Clause 10. A non-temporary computer-readable storage medium, which, when executed by at least one processor, includes instructions causing the at least one processor to perform the actions described in any of Clauses 1 to 8.

Claims

1. A method for controlling the movement of an avatar through finger tracking, The tracking service receives a description of a first gesture performed by one or more fingers of the player in the room space, and the initial position of the one or more fingers performing the first gesture in the room space. Determining the relative positions of one or more fingers in the room space performing the first gesture with respect to the character root in world space, wherein the character root is a volume in the virtual world to which the avatar representing the player is mapped; The virtual controller is instantiated at the aforementioned relative position, When the virtual controller is instantiated, the spatial relationship between the character root in world space and the initial position associated with the virtual controller is recorded. To store information representing the aforementioned spatial relationship, While the first gesture is maintained, the position information of one or more fingers is received, Interpreting the changes in positional information with respect to the stored spatial relationship information, and determining the operation input to the avatar, When the avatar moves or rotates in the virtual world, the position associated with the virtual controller is updated in order to maintain its spatial relationship with the character root. Methods that include...

2. Instantiating the virtual controller at the aforementioned relative position further means, The center of the virtual controller is positioned at the aforementioned relative position, Displaying the virtual controller user interface at the aforementioned relative position, The method according to claim 1, including the method described in claim 1.

3. The rendering of the avatar's hand is displayed at the aforementioned relative position, along with the virtual controller within the rendering of the hand. Determining the distance and direction of the updated position of the one or more fingers performing the first gesture in the room space with respect to the initial position of the one or more fingers performing the first gesture in the room space, Displaying an animation of the avatar's hand moving in the direction from the relative position to the updated position, In conjunction with the animation of the hand moving in the direction from the relative position to the updated position, an indicator for motion input is displayed. The method according to claim 2, including the method described in claim 2.

4. The tracking service receives the location information of one or more fingers performing the first gesture in the room space, The distance and direction of the updated position of the one or more fingers performing the first gesture in the room space are determined by comparing it with the initial position of the one or more fingers performing the first gesture in the room space, The operation of the avatar is further controlled based on the distance and direction of the updated position relative to the initial position. The method according to claim 1.

5. The method according to claim 4, wherein the instantiated virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that starts from the initial position and points in the direction of the updated position.

6. The method according to claim 4, wherein the instantiated virtual controller is a linear motion controller, and the movement of the avatar is a velocity corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the velocity that corresponds to the case where the distance between the initial position and the updated position is smaller.

7. The method according to claim 4, wherein the instantiated virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that starts from the initial position and points in the direction of the updated position.

8. The method according to claim 4, wherein the instantiated virtual controller is a rotational motion controller, and the movement of the avatar is a speed corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the speed that corresponds to compared with the case where the distance between the initial position and the updated position is small.

9. A computing system, Processor and When executed by the aforementioned processor, the system The tracking service receives a description of a first gesture performed by one or more fingers of the player in the room space, and the initial position of the one or more fingers performing the first gesture in the room space. Determine the relative positions of the one or more fingers performing the first gesture in the room space with respect to the character root in the world space, wherein the character root is a volume in the virtual world to which the avatar representing the player is mapped. A virtual controller is instantiated at the aforementioned relative position, When the virtual controller is instantiated, the spatial relationship between the character root in world space and the initial position associated with the virtual controller is recorded. The information representing the aforementioned spatial relationship is stored, While the initial gesture is maintained, receive the position information of one or more fingers. Interpret the changes in the position information with respect to the stored spatial relationship information, and determine the action input to the avatar. When the avatar moves or rotates in the virtual world, the position associated with the virtual controller is updated in order to maintain its spatial relationship with the character root. A memory that stores instructions configured in this way, A computing system equipped with [the following features].

10. The aforementioned instruction further controls the system, The center of the virtual controller is positioned at the aforementioned relative position, The virtual controller user interface is displayed at the aforementioned relative position. The computing system according to claim 9, configured as described above.

11. The aforementioned instruction controls the system At the aforementioned relative position, the rendering of the avatar's hand is displayed within the rendering of the hand, along with a virtual controller. Determine the distance and direction of the updated position of the one or more fingers performing the first gesture in the room space relative to the initial position of the one or more fingers performing the first gesture in the room space, The animation of the hand moving in the direction from the relative position in the world space to the updated position is displayed. An indicator for motion input is displayed in conjunction with the animation of the hand moving in the direction from the relative position to the updated position. The computing system according to claim 10, configured as follows.

12. The aforementioned instruction further controls the system, The tracking service receives the position information of one or more fingers performing the first gesture in the room space. Determine the distance and direction of the updated position of the one or more fingers performing the first gesture in the room space with respect to the initial position of the one or more fingers performing the first gesture in the room space. The system is configured to control the movement of the avatar in world space based on the distance and direction of the updated position compared to the initial position. The computing system according to claim 9.

13. The computing system according to claim 12, wherein the instantiated virtual controller is a linear motion controller, and the movement of the avatar is in the linear direction of a vector that starts from the initial position and points in the direction of the updated position.

14. The computing system according to claim 12, wherein the instantiated virtual controller is a linear motion controller, and the movement of the avatar is a speed corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the speed that corresponds to compared with the case where the distance between the initial position and the updated position is small.

15. The computing system according to claim 12, wherein the instantiated virtual controller is a rotational motion controller, and the movement of the avatar is in the rotational direction of a vector that starts from the initial position and points in the direction of the updated position.

16. The computing system according to claim 12, wherein the instantiated virtual controller is a rotational motion controller, and the movement of the avatar is a speed corresponding to the distance between the initial position and the updated position, and the greater the distance between the initial position and the updated position, the greater the speed that corresponds to compared with the case where the distance between the initial position and the updated position is small.

17. When executed by at least one processor, at least one processor The tracking service receives a description of a first gesture performed by one or more fingers of the player in the room space, and the initial position of the one or more fingers performing the first gesture in the room space. The relative positions of the one or more fingers performing the first gesture in the room space with respect to the character root in the world space are determined, wherein the character root is a volume in the virtual world to which the avatar representing the player is mapped. A virtual controller is instantiated at the aforementioned relative position, When the virtual controller is instantiated, the spatial relationship between the character root in world space and the initial position associated with the virtual controller is recorded. The information representing the aforementioned spatial relationship is stored, While the initial gesture is maintained, receive the position information of one or more fingers. Interpret the changes in the position information with respect to the stored spatial relationship information, and determine the action input to the avatar. When the avatar moves or rotates in the virtual world, the position associated with the virtual controller is updated in order to maintain its spatial relationship with the character root. A computer program that includes instructions.

18. The aforementioned at least one processor is further, The center of the virtual controller is positioned at the aforementioned relative position. The virtual controller user interface is displayed at the aforementioned relative position. A computer program according to claim 17, including instructions.

19. The aforementioned at least one processor is further, At the aforementioned relative position, the rendering of the avatar's hand is displayed within the rendering of the hand, along with a virtual controller. Determine the distance and direction of the updated position of the one or more fingers performing the first gesture in the room space relative to the initial position of the one or more fingers performing the first gesture in the room space. Display the animation of the hand moving in the direction from the relative position in the world space to the updated position. An indicator for motion input is displayed in conjunction with the animation of the hand moving in the direction from the relative position to the updated position. A computer program according to claim 18, including instructions.

20. The aforementioned at least one processor is further, The tracking service receives the position information of one or more fingers performing the first gesture within the room space. Determine the distance and direction of the updated position of the one or more fingers performing the first gesture in the room space with respect to the initial position of the one or more fingers performing the first gesture in the room space. Based on the distance and direction of the updated position compared to the initial position, the movement of the avatar is controlled. Including commands, The computer program according to claim 17.

Images