Gesture-based virtual space configuration

Abstract

A virtual space configuration system of a human reality system can detect user posture and provide various corresponding customizations of the virtual space of the system. When a user is in a seated posture, the virtual space configuration system can provide seated virtual space customizations. In various implementations, these customizations can include allowing adjustment of floor height, setting a flag that can be exposed to applications to adjust mechanisms of the applications for seated users, customizing display of virtual space boundaries while in seated mode to reduce interference, providing options to detect when a user leaves seated mode and trigger corresponding actions, providing a pass-through workspace region allowing a user to naturally interact with certain real-world objects without removing a virtual reality headset, or automatically determining virtual space dimensions for seated users.

Description

TECHNICAL FIELD

[0001] The present invention relates to controlling configuration of virtual space for artificial reality environments. BACKGROUND

[0002] When a user sees and interacts with "virtual objects" (i.e. computer-generated object representations that appear in an artificial reality environment), the user's physical movements take place in the real world. In some cases, the artificial reality system can prevent the user from seeing part or all of the real world, or the user can be distracted by virtual objects, causing the user to inadvertently collide with real-world objects or leave an area designated for the user to interact in the artificial reality environment. In other cases, the user's movements can be limited by the user's physical posture in the real world, making it somewhat difficult to interact in the artificial reality environment. For example, some virtual objects can be placed out of reach, making it difficult for the user to interact with them from the user's current posture. SUMMARY

[0003] According to a first aspect of the invention, there is provided a method for self-defining a virtual space based on a user posture, the method comprising: determining that the user posture corresponds to a seated mode; in response to the determination, setting seated self-definitions for the virtual space by: providing a first seated self-definition comprising: obtaining a measure for a floor height; adjusting a system floor height based on the measure for the floor height; or providing a second seated self-definition comprising: setting a seated flag; and exposing the seated flag to one or more applications, wherein the one or more applications adjust an operating mechanism based on the seated flag; or providing a third seated self-definition comprising: receiving a boundary mode selection for the virtual space; detecting a boundary display event; and based on the selected boundary mode: displaying a boundary configured based on the selected boundary mode; or displaying a real-world object in the virtual space.

[0004] Setting seated self-definitions can preferably comprise providing the first seated self-definition.

[0005] In some embodiments, the measure for the floor height is based on one or more of: a determined standing height of the user; a determined arm span of the user; a determined size of an object on which the user is seated; or any combination thereof.

[0006] In some embodiments, the measure for the floor height is determined by a machine learning model trained based on previous user selections to receive an indication of sensor or camera measurements and produce a floor height measure.

[0007] In some embodiments, the measure for floor height is one or more user-selected values indicated by one or more of: a user hand pose, a user input to a controller, a user voice command, a user gaze direction, or any combination thereof.

[0008] In some embodiments, setting the seated customization includes providing a second seated customization.

[0009] In some embodiments, setting the seated customization includes providing a third seated customization.

[0010] The boundary mode selection can preferably be based on a mapping of poses to boundary modes provided by the current application.

[0011] In some embodiments, the selected boundary mode is a pass-through mode that causes the display to display real-world objects in the virtual space.

[0012] Detecting the boundary display event can include applying a machine learning model trained to receive one or more of inertial data, position data, camera image data, a model of a user skeletal structure, or any combination thereof and produce a prediction of whether the user will intersect the boundary.

[0013] According to a second aspect of the invention, there is provided a computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for customizing a virtual space based on a user pose, the operations comprising: determining that the user pose corresponds to a seated mode; in response to determining that the user pose corresponds to the seated mode, setting a seated customization for the virtual space by: providing a first seated customization including: while a virtual experience configured for the seated mode is ongoing: determining that a second user pose corresponds to no longer being in the seated mode; in response, triggering a response action for the virtual experience; or providing a second seated customization including: determining that the second user pose corresponds to a forward lean; determining a workspace region; and implementing a display mode that displays real-world objects in the virtual environment that are in the workspace region; or providing a third seated customization including: automatically determining a size for the virtual space based on physical characteristics of the user; and adjusting the virtual space based on the determined size.

[0014] In some embodiments, setting the seated customization includes providing a first seated customization.

[0015] In some embodiments, the response action includes one or more of: automatically stopping or pausing the virtual experience; providing a notification to resume a previous pose or to not use a current pose for the virtual experience; recording a time for which various poses are held; switching to a display mode that shows real-world objects; changing an input modality; or any combination thereof.

[0016] In some embodiments, setting the seating customization comprises providing a second seating customization.

[0017] In some embodiments, the workspace region is determined based on one or more of: a region defined based on a determined arm span of the user; an average of workspace regions previously manually set by other users; a region corresponding to a top of a planar real-world object in front of the user; a region determined to contain one or more specified real-world tools; or any combination thereof.

[0018] In some embodiments, setting the seating customization comprises providing a third seating customization.

[0019] The physical characteristics of the user can preferably include an automatically determined arm span of the user.

[0020] The arm span of the user can preferably be automatically determined by: setting an initial arm span equal to a determined height of the user; and updating the initial arm span based on identifying a hand or controller position of the user that extends beyond the determined arm span.

[0021] According to a third aspect of the present application, there is provided a computing system for customizing a virtual space based on a user posture, the computing system comprising: one or more processors; one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: determining that a user posture corresponds to a seated mode; in response to determining that the user posture corresponds to the seated mode, setting a seating customization for the virtual space by one or more of: providing a first seating customization comprising: obtaining a measure for a floor height; adjusting a system floor height based on the measure for the floor height; or providing a second seating customization comprising: setting a seated flag; and exposing the seated flag to one or more applications, wherein the one or more applications adjust an operating mechanism based on the seated flag; or providing a third seating customization comprising: receiving a boundary mode selection for the virtual space; detecting a boundary display event; and displaying a boundary type or object in the virtual space based on the selected boundary mode; or providing a fourth seating customization comprising: while a virtual experience configured for the seated mode is ongoing: determining that a second user posture corresponds to no longer being in the seated mode; triggering a responsive action for the virtual experience; or, providing a fifth seating customization comprising: determining that a second user posture corresponds to a forward lean; determining a workspace region; and implementing a display mode that displays real-world objects in the virtual environment that are in the workspace region; or providing a sixth seating customization comprising: automatically determining a size for the virtual space based on physical characteristics of the user; and adjusting the virtual space based on the determined size. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a block diagram illustrating an overview of a device on which some implementations of the technology can operate.

[0023] Figure 2A is a wiring diagram illustrating a virtual reality headset that can be used in some implementations of the technology.

[0024] Figure 2B is a wiring diagram illustrating a mixed reality headset that can be used in some implementations of the technology.

[0025] Figure 3 is a block diagram illustrating an overview of an environment in which some implementations of the technology can operate.

[0026] Figure 4 is a block diagram illustrating components that can be used in a system employing the disclosed technology in some implementations.

[0027] Figure 5 is a flow diagram illustrating a process for setting a virtual space configuration based on a user pose, used in some implementations of the technology.

[0028] Figure 6A is a flow diagram illustrating a process for implementing floor height customization when a user is seated, used in some implementations.

[0029] Figure 6B is a flow diagram illustrating a process for setting a flag to allow an application to adjust a seated configuration mechanism, used in some implementations.

[0030] Figure 6C is a flow diagram illustrating a process for customizing a virtual space boundary display in response to a user pose, used in some implementations.

[0031] Figure 6D is a flow diagram illustrating a process for implementing a seated-only virtual experience, used in some implementations.

[0032] Figure 6E is a flow diagram illustrating a process for implementing a seated workspace virtual area, used in some implementations.

[0033] Figure 6F is a flow diagram illustrating a process for automatically customizing a virtual area in seated mode, used in some implementations.

[0034] Figure 7A is a conceptual diagram illustrating an example of implementing floor height customization when a user is seated.

[0035] Figure 7BFIG. 1 is a conceptual diagram illustrating an example of a user interface for adjusting a virtual space for seated use.

[0036] Figure 7C FIG. 2 is a conceptual diagram illustrating an example of a user interface for adjusting a virtual space for seated use.

[0037] Figure 7D FIG. 3 is a conceptual diagram illustrating an example of a user interface for adjusting a virtual space for seated use.

[0038] Figure 7E FIG. 4 is a conceptual diagram illustrating an example of a user interface for adjusting a virtual space for seated use.

[0039] Figure 7F FIG. 5 is a conceptual diagram illustrating an example of a user interface for adjusting a virtual space for seated use.

[0040] The technology introduced herein can be better understood with reference to the following detailed description and accompanying drawings, in which like reference numerals refer to similar elements or functions throughout the description and drawings. DETAILED DESCRIPTION

[0041] Embodiments are described herein for customizing virtual spaces based on user posture. An artificial reality system can define a particular“virtual space” for a user experience, which can define the range of movement for the user during the experience, control how virtual objects are displayed or placed in the experience, and / or set system actions in response to posture changes. For example, if a user approaches the edge of a defined virtual space, the artificial reality system can provide a warning or implement a pass-through mode to display to the user real-world objects with which she can collide. As used herein, a“posture” is a position or configuration of one or more parts of a user’s body. For example, a posture can be seated, standing, lying down, arms extended, a particular hand position or posture, head orientation, torso rotation, etc. In some implementations, a posture can also include movement, such as a particular motion of one or more body parts and / or a particular motion relative to a point or object. For example, a first identified posture can be standing still, while a second identified posture can be standing moving (e.g., the user has made a threshold lateral movement relative to a center point).

[0042] A virtual space configuration system (which can be a subsystem of a artificial reality system) can detect user poses and provide various corresponding customizations to the virtual space of the system. In some implementations, poses that the virtual space configuration system can identify include standing (which can be divided into standing mobile or standing still), sitting, lying down, etc. In various implementations, these determinations can be automatic (based on user input), or a combination of automatic determination and user confirmation. For example, the virtual space configuration system can determine the height of a head mounted device of the artificial reality system compared to an identified floor, and using a known user height or average of heights of multiple users, can determine whether the head mounted device height corresponds to a standing or sitting pose. Further, when a position is determined to be standing, the virtual space configuration system can determine whether the lateral position of the head mounted device has moved more than a threshold amount from a center point to determine whether the standing user is stationary or mobile. The virtual space configuration system can provide an indication of the determined pose to the user for user confirmation or modification.

[0043] When in a standing mobile pose, the virtual space configuration system can set the virtual space that the user has defined for the current real world environment of the artificial reality system, and / or the virtual space configuration system can automatically detect objects around the user’s current position and set the virtual space to avoid collisions with those objects. When the user approaches this boundary, the virtual space configuration system can present a warning or display a grid indicating the boundary. When in a standing still pose, the virtual space configuration system can define the virtual space around the user, for example, as a cylindrical region or a “chalice” shape (i.e., a cylinder that is narrow at the bottom and wide at the top), to account for the user’s legs being stationary, but providing a space around the user’s upper half in which they can move their arms. In some implementations, the diameter of the upper portion of this cylinder or chalice shape can be based on a feature of the user, such as a determined arm span.

[0044] When determining that the user is in a sitting or lying down mode, the virtual space configuration system can provide various other virtual space customizations. In one instance, the virtual space configuration system can obtain metrics for different floor heights to use when the user is seated. These metrics can come from, for example, a machine learning model trained to predict a desired floor height, user input specifying a floor height change (e.g., using a controller, a pose, or a tracked user gaze), and / or past floor height settings determined by one or more users to have similar characteristics. The virtual space configuration system can then set the floor height based on the metrics. This sets a minimum height for virtual objects related to the user, improving the accessibility of the user when in the virtual space by eliminating situations where the user would otherwise have to move to the edge of a chair or couch and reach the floor.

[0045] In another example, the virtual space configuration system can facilitate adjustment of application mechanisms specific to a seated or lying user. For example, when a user is seated or lying down, a notification application can adjust placement of virtual objects to be within a typical or measured user arm span. For example, a virtual object that a user would typically step up to interact with can be automatically moved within reach. This can be in response to a flag set by the virtual space configuration system for seated and / or lying down modes, which in turn can be exposed to applications. Applications can be customized to have different mechanisms based on such flags.

[0046] In another case, the virtual space configuration system can configure a boundary mode based on a user’s posture. In one case, when a user is in a standing posture, the virtual space can set a boundary, and when the virtual space configuration system predicts that the user can connect with the boundary, the virtual space configuration system will display the boundary or a warning. For example, when a user is in a standing posture, the boundary can be a red grid that, if displayed in the virtual space, would immediately draw the user’s attention. However, because the user can move more slowly or just her arms, a collision with the boundary while seated is less likely to be a problem. Thus, the boundary while seated can be a less immersive pattern, such as a pattern of small gray cross (e.g., +) markers. Alternatively, when the virtual space configuration system predicts that the user can collide with real-world objects around the user (e.g., when they are within the user’s arm span), the system can identify the real-world objects around the user and display them in the virtual space, rather than displaying a boundary while seated.

[0047] In yet another example, the virtual space configuration system can implement an experience that is only available when a user is in a particular posture or trigger a particular action when a user transitions between postures. For example, after determining that a user is in a seated posture, the artificial reality system can initiate a “seated-only” experience. The virtual space configuration system can continuously monitor the user’s posture throughout the experience. If the user stands up, this can trigger the artificial reality system to take action, such as automatically stopping the seated-only experience, providing a notification to the user to return to a seated position, logging the time the user stood during the experience, switching to a pass-through mode that displays aspects of the real world rather than portions of the experience, and / or changing aspects of the experience, such as providing different input modalities or changing virtual objects.

[0048] In addition, the virtual space configuration system can provide a "workspace" virtual area that appears when the user is seated and also in a particular additional posture, such as leaning forward. The workspace can be an area in front of the user, e.g., based on one or more of: a determined user arm span, general user arm length statistics, previous user settings, a user-drawn area, and / or identification of an area that includes particular objects (e.g., keyboard, display, mouse, desktop area, etc.). The virtual space configuration system can also detect that the user is leaning forward at least a threshold amount while seated. In some implementations, this can also depend on identification of a flat workspace (such as a desk) in front of the user. Upon making this further posture determination, the virtual space configuration system can implement a pass-through mode, i.e., a mode that shows a representation of at least part of the real world (in this case, the determined workspace area). This allows the user to quickly and easily transition between interacting with real-world items in the workspace area and virtual objects in the virtual space.

[0049] In another case, the virtual space configuration system can automatically customize the size (e.g., size and / or shape) of the virtual area for a seated mode. The virtual space configuration system can determine the size based on context or user details, such as a determined user arm span, average or determined statistics of similar users, previous user settings, a user-drawn area, or by identifying objects in the surrounding area. The virtual space configuration system can then set the virtual area based on the determined size, e.g., as a rectangle or semi-circle in front of the user or a full circle around the user.

[0050] Embodiments of the disclosed technology can include or be implemented in conjunction with an artificial reality system. Artificial reality or a virtual reality (VR) is a form of reality that has been adjusted in some manner before presentation to a user, which can include, e.g., a VR, an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and / or derivatives thereof. Artificial reality content can include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content can include video, audio, touch feedback, or some combination thereof, any of which can be presented in a single channel or in multiple channels (such as stereo videos that produce a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality can be associated with applications, products, accessories, services, or some combination thereof, that are used to create content in an artificial reality or used in an artificial reality (e.g., enter a virtual environment). An artificial reality system that provides artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computer system, a “cave” environment, or other projection systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0051] As used herein, “virtual reality” or “VR” refers to an immersive experience in which a user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to a system in which a user views real-world images through a computing system. For example, a tablet computer with a camera on the back can capture real-world images and then display the images on a screen on the side of the tablet computer opposite the camera. The tablet computer can process and adjust or “augment” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to a system in which light entering a user’s eyes is generated in part by a computing system and in part includes light reflected from objects in the real world. For example, an MR headset can be shaped like a pair of glasses with a transmissive display that allows light from the real world to pass through a waveguide, while also emitting light from a projector in the MR headset, allowing the MR headset to present virtual objects that are mixed with real objects that the user can see. As used herein, “artificial reality,” “ultra reality,” or “XR” refers to any of VR, AR, MR, or any combination or mixture thereof.

[0052] Existing XR systems exist that provide virtual spaces. However, these XR systems can be difficult to use and provide limited functionality. Existing XR systems often do not distinguish between user postures when configuring virtual spaces, require users to manually adjust virtual spaces, or require users to operate in virtual spaces that can be difficult to use, distracting, or disallow certain options. For example, standard XR systems do not provide an option to adjust floor position when a user is seated, often requiring the user to leave their seat to reach virtual objects placed on the floor. As another example, existing XR systems often have a single warning system for when a user is about to collide with a virtual space wall. However, this can be distracting and unnecessary when the user is seated, as such collisions are less likely to cause any damage. Further, existing XR systems require a large amount of setup for virtual spaces, which can be unnecessary for seated configurations where the virtual space can be small and less likely to require a particular contour.

[0053] The virtual space configuration systems and processes described herein are expected to overcome such problems associated with traditional XR systems and are expected to provide users with greater control over virtual spaces. The disclosed virtual space configuration systems and processes are also expected to provide more functionality and a more natural and intuitive user experience compared to interactions in existing XR systems. Despite being natural and intuitive, the virtual space configuration systems and processes described herein are rooted in computerized artificial reality systems, rather than simulations of traditional interactions. For example, these virtual space configuration systems and processes can determine when a user is seated and, in response, provide virtual space customizations. One such virtual space customization can be to allow adjustment of floor height. Another virtual space customization can be to set a flag that can be exposed to applications to adjust mechanisms of the applications. Further, a virtual space customization can be to customize the display of seated mode virtual space boundaries to reduce distractions. An additional virtual space customization can be to provide an option to detect when a user leaves seated mode and trigger a corresponding action. Another virtual space customization can be to provide a pass-through workspace area that allows a user to naturally interact with certain real-world objects without removing a virtual reality headset. Another virtual space customization can be to automatically determine virtual space dimensions for seated users.

[0054] Several implementations are discussed in more detail below with reference to the figures. Figure 1is a block diagram illustrating an overview of a device on which some implementations of the disclosed technology can operate. The device can include hardware components of the computing system 100, which can determine user gestures and set corresponding virtual space customizations. In various implementations, the computing system 100 can include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over wired or wireless channels to distribute processing and share input data. In some implementations, the computing system 100 can include a standalone head-mounted device that is capable of providing a computer-created or augmented experience for a user without the need for external processing or sensors. In other implementations, the computing system 100 can include multiple computing devices, such as a head-mounted device and a core processing component (such as a console, a mobile device, or a server system), where some processing operations are performed on the head-mounted device and other processing operations are offloaded to the core processing component. The following description is made in the context of a head-mounted device, but the disclosed technology can be implemented in other computing devices as well. Figure 2A and Figure 2B An example head-mounted device is described. In some implementations, location and environmental data can be gathered only by sensors incorporated in the head-mounted device, while in other implementations, one or more of the non-head-mounted device computing devices can include sensor components that can track environmental or location data.

[0055] The computing system 100 can include one or more processors 110 (e.g., central processing units (CPUs), graphics processing units (GPUs), holographic processing units (HPUs), etc.). The processor 110 can be a single processing unit or a plurality of processing units located in one device or distributed across multiple devices (e.g., distributed across two or more computing devices 101-103).

[0056] The computing system 100 can include one or more input devices 120 that provide input to the processor 110, informing them of actions. Actions can be mediated by a hardware controller that interprets signals received from the input device and communicates information to the processor 110 using a communication protocol. Each input device 120 can include, for example, a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptic glove, bracelet, ring, earring, necklace, watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0057] The processor 110 can be coupled to other hardware devices, for example, by using internal or external buses, such as a PCI bus, SCSI bus, or wireless connections. The processor 110 can communicate with hardware controllers of devices such as the display 130. The display 130 can be used to display text and graphics. In some implementations, the display 130 includes an input device as part of the display, such as when the input device is a touchscreen or is equipped with an eye direction monitoring system. In some implementations, the display is separate from the input device. Examples of display devices are: LCD displays, LED displays, projection, holographic or augmented reality displays, such as heads-up display devices or head-mounted devices, and the like. Other I / O devices 140 can also be coupled to the processor, such as network chips or cards, video chips or cards, audio chips or cards, USB, Firewire, or other external devices, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, and the like.

[0058] The computing system 100 can include a communication device that enables wireless or wired communication between the computing system 100 and other local computing devices or network nodes. The communication device can communicate with another device or server over a network using, for example, TCP / IP protocols. The computing system 100 can utilize the communication device to distribute operations across multiple network devices.

[0059] The processor 110 can access memory 150, which can be contained on one of the computing devices of the computing system 100 or can be distributed across multiple computing devices of the computing system 100 or other external devices. The memory includes one or more hardware devices for volatile or non-volatile storage, and can include both read-only memory and writable memory. For example, the memory can include one or more of: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory (such as flash memory, hard disk drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, etc. The memory is not a propagating signal detached from the underlying hardware; thus, the memory is non-transitory. The memory 150 can include program memory 160 storing programs and software, such as an operating system 162, a virtual space configuration system 164, and other application programs 166. The memory 150 can also include data memory 170, which can include various models (e.g., pose classifiers, boundary collision predictors, user height or wingspan identifiers, etc.), floor height settings, seated flag variables, boundary mode variables and associated display configurations, pose change mappings, virtual experiences, work area settings, virtual area settings, other configuration data, settings, user options or preferences, etc., which can be provided to the program memory 160 or any element of the computing system 100.

[0060] Some implementations can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be suitable for use with this technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0061] Figure 2A is a wiring diagram of a virtual reality head-mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a localizer 225, and one or more computing units 230. The position sensors 220, the IMU 215, and the computing units 230 can be inside the HMD 200 and can not be visible to a user. In various implementations, the IMU 215, the position sensors 220, and the localizer 225 can track movements and positions of the HMD 200 in real world and virtual environments in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the localizer 225 can emit infrared light beams that produce light points on real objects around the HMD 200. One or more cameras (not shown) integrated with the HMD 200 can detect the light points. The computing units 230 in the HMD 200 can use the detected light points to infer the position and movements of the HMD 200, as well as to identify shapes and positions of real objects around the HMD 200.

[0062] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to a user as instructed by the computing units 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of the electronic display 245 include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode sub-pixels (QOLED), a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.

[0063] In some implementations, HMD 200 can be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor HMD 200 (e.g., via light emitted from HMD 200), which the PC can use, in conjunction with output from IMU 215 and position sensor 220, to determine the position and movement of HMD 200.

[0064] In some implementations, HMD 200 can communicate with one or more other external devices, such as a controller (not shown) that a user can hold in one or both hands. The controller can have its own IMU unit, position sensor, and / or can emit more light points. HMD 200 or the external sensors can track these controller light points. Computing unit 230 in HMD 200 or the core processing component can use this tracking, in conjunction with the IMU and position output, to monitor the user's hand position and motion. The controller can also include various buttons that a user can actuate to provide input and interact with virtual objects. In various implementations, HMD 200 can also include additional subsystems, such as an eye tracking unit, an audio system, various networking components, etc. In some implementations, one or more cameras included in the HMD or external thereto can monitor the user's hand position and gestures to determine gestures and other hand and body motion, instead of or in addition to a controller.

[0065] Figure 2B is a wiring diagram for a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. Mixed reality HMD 252 and core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as shown by link 256. In other implementations, mixed reality system 250 includes only a head-mounted device, without an external computing device, or includes other wired or wireless connections between mixed reality HMD 252 and core processing component 254. Mixed reality HMD 252 includes a pass-through display 258 and a frame 260. Frame 260 can house various electronic components (not shown), such as projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, MEMS components, networking components, etc.

[0066] The projector can be coupled to the pass-through display 258, e.g., via optical elements, to display media to the user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., to direct light from the projector to the user's eye. Image data can be transmitted from the core processing component 254 to the HMD 252 via the link 256. A controller in the HMD 252 can convert the image data to light pulses from the projector, which can be transmitted as output light to the user's eye via the optical elements. The output light can mix with light passing through the display 258, allowing the output light to represent virtual objects that appear to exist in the real world.

[0067] Similar to the HMD 200, the HMD system 250 can also include motion and position tracking units, cameras, light sources, etc., that allow the HMD system 250 to track itself, track parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear stationary when the HMD 252 moves, and make virtual objects react to gestures and other real-world objects, e.g., in 3DoF or 6DoF.

[0068] Figure 3 is a block diagram illustrating an overview of an environment 300 in which some implementations of the disclosed technology can operate. The environment 300 can include one or more client computing devices 305A-D, examples of which can include the computing system 100. In some implementations, some of the client computing devices (e.g., client computing device 305B) can be an HMD 200 or an HMD system 250. The client computing devices 305 can operate in a networked environment using logical connections to one or more remote computers, such as the server computing devices, through the network 330.

[0069] In some implementations, the server 310 can be an edge server that receives client requests and coordinates fulfillment of those requests through other servers, such as the servers 320A-C. The server computing devices 310 and 320 can include computing systems, such as the computing system 100. While each server computing device 310 and 320 is shown logically as a single server, each server computing device can be a distributed computing environment that includes multiple computing devices located at geographically different or the same physical locations.

[0070] Client computing device 305 and server computing devices 310 and 320 can each act as a server or client to other server / client device(s). Server 310 can be connected to database 315. Servers 320A-C can each be connected to corresponding databases 325A-C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share one database or can have their own databases. Although databases 315 and 325 are shown logically as single units, databases 315 and 325 can each be a distributed computing environment containing multiple computing devices, which can be located within their corresponding servers, or can be located at geographically different or the same physical locations.

[0071] Network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing device 305 can connect to network 330 through a network interface, such as through wired or wireless communication. Although connections between servers 310 and servers 320 are shown as separate connections, these connections can be any kind of local area network, wide area network, wired network, or wireless network, including network 330 or separate public or private networks.

[0072] Figure 4 is a block diagram illustrating components 400 that can be used in a system employing the disclosed technology, in some implementations. Components 400 can be included in one device of computing system 100 or can be distributed across multiple devices of computing system 100. Components 400 include hardware 410, broker 420, and specialized components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including processing unit 412, working memory 414, input and output devices 416 (e.g., cameras, displays, IMU units, network connections, etc.), and storage memory 418. In various implementations, storage memory 418 can be one or more of: a local device, an interface with a remote storage device, or a combination thereof. For example, storage memory 418 can be one or more hard disk drives or flash drives accessible through a system bus, or can be a cloud storage provider such as in storage 315 or 325 or other network storage accessible via one or more communication networks. In various implementations, components 400 can be implemented in a client computing device such as client computing device 305 or on a server computing device such as server computing devices 310 or 320.

[0073] Mediator 420 can include components that mediate resources between mediation hardware 410 and specialized components 430. For example, mediator 420 can include an operating system, services, drivers, basic input output system (BIOS), controller circuitry, or other hardware or software systems.

[0074] Specialized components 430 can include software or hardware configured to perform operations for customizing a virtual space based on a user's posture. For example, specialized components 430 can include a posture analysis module 434, a standing mode function 436, a seated-floor height function 438, a seated-landmark configuration function 440, a seated-boundary display function 442, a seated-seated experience function 444, a seated-workspace area function 446, a seated-automated virtual area function 448, and components such as interfaces 432 and APIs that can be used to provide user interfaces, transmit data, and control. In some implementations, components 400 can be in a computing system distributed across multiple computing devices, or can be an interface to a server-based application executing one or more specialized components 430.

[0075] Posture analysis module 434 can receive sensor inputs (e.g., images from a camera, position sensor data, controller sensor inputs, etc.) and / or determined biomechanical models (e.g., a kinematic skeletal model of a user, hand position, etc.) and use these to determine a user's posture. In various implementations, a posture can specify that a user is standing, seated, lying down, etc. In some implementations, a standing posture can be divided into standing, moving or standing, stationary or other movement-based postures. Additional details regarding determining a user's posture are described below in connection with block 502 of Figure 5

[0076] Some implementations can include a standing mode function 436. In these implementations, a virtual space configuration system can execute these functions in response to posture analysis module 434 identifying a standing posture. Executing standing mode function 436 can include receiving a user-specified boundary for a standing, moving posture or an automatically-sized cylinder or goblet-shaped boundary (based on a determined arm span) for a standing, stationary posture. If the system predicts that a user can collide with the boundary, the boundary can be displayed to the user. Additional details regarding standing mode functions are provided below in connection with blocks 506 and 510 of Figure 5

[0077] ​​Some implementations can include a seated-floor height function 438. In these implementations, the virtual space configuration system can perform these functions in response to the posture analysis module 434 identifying a seated posture. Performing the seated-floor height function 438 can include receiving a floor height metric, such as a user selection of a floor height, a floor height based on a determined user height, an average of floor heights selected by other users, etc. The seated-floor height function 438 can use this metric to set the virtual floor height. Additional details are provided below in connection with Figure 6A and Figure 7A Additional details are provided below in connection with setting the virtual floor height when the user is in a seated posture.

[0078] Some implementations can include a seated flag configuration function 440. In these implementations, the virtual space configuration system can perform these functions in response to the posture analysis module 434 identifying a seated posture. Performing the seated flag configuration function 440 can include setting a flag in response to determining that the user is in a seated posture. This flag can then be exposed to applications, allowing them to adjust positioning of objects and other mechanisms based on whether the flag is set. Additional details are provided below in connection with Figure 6B and Figure 7B Additional details are provided below in connection with setting a seated flag and exposing it to allow applications to adjust mechanisms.

[0079] Some implementations can include a seated-boundary display function 442. In these implementations, the virtual space configuration system can perform these functions in response to the posture analysis module 434 identifying a seated posture. Performing the seated-boundary display function 442 can include determining a virtual space boundary mode, such as a pattern, color, or type, where the type can be a virtual wall, showing objects in a pass-through mode, a warning message, or other alert, etc. When the virtual space configuration system detects a boundary display event, such as predicting that the user will intersect a boundary or a real-world object, or that a real-world object has entered the virtual space, the seated-boundary display function 442 can further include displaying a boundary or representation of the real-world object according to the determined virtual space boundary mode. Additional details are provided below in connection with Figure 6C and Figure 7C Additional details are provided below in connection with selecting a virtual space boundary mode and corresponding display event.

[0080] Some implementations can include a seated-seated experience function 444. In these implementations, the virtual space configuration system can perform these functions in response to the posture analysis module 434 identifying a seated posture. Performing the seated-seated experience function 444 can include detecting further changes in posture while the user is engaged in a seated-only virtual experience. Using a mapping of postures to responsive actions, for example provided by the seated-only virtual experience, a responsive action for the change in posture can be determined and performed. For example, if the user stands up, the application can pause, provide a notification, record that the user is standing, change the input modality, and / or change the virtual objects. Additional details regarding triggering responsive actions for changes in posture are provided below in connection with Figure 6D and Figure 7D Additional details regarding detecting a forward leaning posture, a seated posture, and displaying a workspace region in a pass-through mode are provided below in connection with

[0081] Some implementations can include a seated-workspace region function 446. In these implementations, the virtual space configuration system can perform these functions in response to the posture analysis module 434 identifying a seated posture. Performing the seated-workspace region function 446 can include detecting a posture that further leans forward while the user remains in a seated posture. In response, the performance of the seated-workspace region function 446 can implement a pass-through display mode for a determined workspace region, allowing the user to see a representation of real-world objects in the workspace region without having to remove a head-mounted device or other hardware of the artificial reality system. The workspace region, such as the top of a desk or a region that includes various tools such as a keyboard, mouse, and / or display, can be determined based on a region predefined by the user, a determined reach of the user, an average of workspace set by other users, and / or using one or more of computer vision and object detection to identify the region. In some implementations, a trained machine learning model can determine the workspace region based on a current context (e.g., user details and / or camera input), where the model is trained based on similar inputs that match a user-selected workspace region or an automatically identified workspace region determined based on object identification with a high confidence value. Additional details regarding detecting a forward leaning posture, a seated posture, and displaying a workspace region in a pass-through mode are provided below in connection with Figure 6E and Figure 7E Additional details regarding detecting a forward leaning posture, a seated posture, and displaying a workspace region in a pass-through mode are provided below in connection with

[0082] Some implementations can include seated-automated virtual area functionality 448. In these implementations, the virtual space configuration system can perform these functions in response to the pose analysis module 434 identifying a seated pose. Performing the seated-automated virtual area functionality 448 can include automatically determining the size of the virtual area for a seated position based on one or more of user-defined areas, user arm span, average areas set by other users, etc. In some implementations, a trained machine learning model can determine the virtual space size based on the current context (e.g., user details and / or camera input), where the model is trained based on similar inputs matching the user-selected virtual space. The shape of the virtual area can be automatically determined based on one or more of settings in the current application, user selection, a current use determined for the virtual space with a mapping of uses to virtual space shapes, etc. The following discussion of the seated-automated virtual area functionality 448 is provided in the context of a seated user, but the functionality can be applied to other user poses as well. Figure 6F and Figure 7F Automatic determination of virtual areas is provided.

[0083] Those skilled in the art will appreciate that the components illustrated in FIG. 13, and in each of the flow diagrams discussed below, can be changed in a variety of ways. For example, the order of the logic can be rearranged, sub-steps can be performed in parallel, logic illustrated can be omitted, other logic can be included, etc. In some implementations, one or more of the above components can perform one or more of the processes described below. Figures 1-4

[0084] Figure 5 is a flow diagram illustrating a process 500 for setting virtual space configuration based on user pose, used in some implementations of the present technology. In various implementations, the process 500 can be performed by an artificial reality system (e.g., by a virtual space configuration subsystem) when the artificial reality system is first turned on, when the artificial reality system detects a change in user, continuously on a periodic basis (e.g., every 1-2 seconds), or in response to a detected pose change (e.g., the pose detection of block 502 is performed periodically, or in response to input signals such as a change in height of a head-mounted device or controller or other motion).

[0085] ​At block 502, the process 500 can determine a user pose. A “pose” is a position or configuration of one or more parts of a user’s body. For example, a pose can be seated, standing, lying down, arms extended, a particular hand position or pose, a head orientation, a torso rotation or angle, etc. In some implementations, a pose can also include a movement, such as a particular motion of one or more body parts and / or a particular motion relative to a point or object. For example, a first identified pose can be stationary, while a second identified pose can be standing and have made a threshold level of lateral movement relative to a center point. In various implementations, the process 500 can automatically determine a user pose, e.g., based on determined height of a head-mounted device of the artificial reality system, particular detected movements (e.g., of the head-mounted device, a controller, a hand, a leg, or other body part), images captured by the artificial reality system, other inputs such as position data, IMU data, etc. In some implementations, various measurements and determinations from the artificial reality system can be provided to a machine learning model trained to classify a current pose of a user. Determining a user pose (or “pose”) is discussed in more detail in U.S. Patent Application No. 16 / 663,141, filed October 9, 2019, entitled “Systems and Methods for Generating Dynamic Obstacle Collision Warnings Based On Detecting Poses of Users,” the entirety of which is incorporated by reference herein. In some implementations, a user pose can be specified by user input, or a user input can verify an automatically detected pose.

[0086] At block 504, the process 500 can determine whether the pose determined at block 502 corresponds to a standing mobile pose. A standing mobile pose can indicate that the user is standing and in a situation where she can move laterally (as opposed to standing in the same place generally). This can be indicated by the artificial reality system determining that the user is standing (e.g., based on a determined headset height, a user pose selection input, etc.) and one or more of the following: the user has indicated a boundary region for the virtual space, the user has specified an intent to move, the current application is designed for movement while standing, or determining that the user has moved laterally at least a threshold amount from a center point (i.e., determining that they have moved from their standing location). When the pose corresponds to a standing mobile mode, the process 500 can continue to block 506, where it sets a standing mobile virtual space customization. For example, the virtual space can be a user-defined space and / or a space defined to avoid the user from colliding with objects detected in the real-world space around the user. The customization can also include setting display features to show the boundary, such as using a very noticeable red grid pattern to immediately draw the user’s attention, if the artificial reality system determines that the user is in danger of colliding with the boundary. If the pose is not standing mobile, the process 500 can continue to block 508.

[0087] At block 508, the process 500 can determine whether the pose determined at block 502 corresponds to a standing stationary pose. A standing stationary pose can indicate that the user is standing and is not likely to move laterally (i.e., can be standing within a range of a few feet from the same location). Similar to a standing mobile pose, a standing stationary pose can be indicated by the artificial reality system determining that the user is standing (e.g., based on a determined headset height, a user pose selection input, etc.). But in this case, the user can have specified an intent not to move, the current application can be designed for being stationary while standing, or the artificial reality system can determine that the user has not moved at least a threshold amount from a center point (i.e., determining that they have not moved a significant amount, such as 1-2 feet, laterally from their current standing location). When the pose is stationary standing, the process 500 can continue to block 510, where it sets a standing stationary virtual space customization. For example, the virtual space can be a user-defined space, or a cylinder or goblet shape defined around the user. The customization can also include setting display features to show the boundary, such as using a noticeable red grid pattern to immediately draw the user’s attention, if the artificial reality system determines that the user is in danger of colliding with the boundary, or a less immersive pattern using a gray cross (e.g., +) shape because such a collision is less likely to cause damage. If the pose is not stationary standing, the process 500 can continue to block 512.

[0088] At block 512, the process 500 can determine whether the pose determined at block 502 corresponds to a seated pose. The seated pose can be indicated by the artificial reality system determining that the head mounted device of the system is within a threshold distance of an average seated head mounted device height, can come from user input specifying the pose, can be determined based on a machine learning model that takes sensor input and classifies the current pose, can be assumed based on using a designed current application while seated or from directions provided to the user or using other metrics. If the determined pose is seated, the process 500 can continue to block 514 where any of a variety of seated virtual space customizations can be applied. In various implementations, the available seated virtual space customizations can include: allowing adjustment of floor height (see additional details below regarding Figure 6A ), setting a logo that can be revealed to the application to adjust the mechanism of the application (see additional details below regarding Figure 6B ), customizing the display of virtual space boundaries in seated mode to reduce interference (see additional details below regarding Figure 6C ), providing an option to detect when the user leaves seated mode and trigger a corresponding action (see additional details below regarding Figure 6D ), providing a pass-through workspace area allowing the user to naturally interact with certain real world objects without removing the artificial reality head mounted device (see additional details below regarding Figure 6E ), and automatically determining virtual space dimensions for seated users (see additional details below regarding Figure 6F ). If the pose is not seated, the process 500 can return to block 502 to continue monitoring the user pose for the recognized variant.

[0089] Figure 6A is a flowchart illustrating a process 600 for implementing floor height customization when a user is seated, used in some implementations. In some implementations, the process 600 can be performed as a sub-process of block 514 of the process 500. In some cases, the process 600 can be performed in response to other triggers such as an application changing a pose mode, a user selection, a user’s change, a start of a particular application, etc.

[0090] At block 602, process 600 can obtain a measure for seated floor height. The measure can be determined based on conditions such as a user's standing height, a user's arm span, dimensions of a chair, sofa, or other object on which the user is seated (e.g., determined using cameras and computer vision techniques integrated with the artificial reality system), an average or other statistical data of floor heights set by other users (or users determined to be similar to the current user), or the like. In some implementations, one or more of these features can be used as inputs to a machine learning model trained to predict a desired floor height based on previous other user selections. In other implementations, the features can be used to map the features to a floor height. In some implementations, the measure can be a user-selected value, e.g., indicated by a hand pose, input to a controller, voice command, gaze, or the like. In some implementations, the user can be identified and the obtained measure can be based on previous selections for that user.

[0091] At block 604, process 600 can set the floor height based on the measure obtained at block 602. This sets a minimum height for virtual objects relative to the user, improving accessibility for the user when in the virtual space by eliminating cases where the user would otherwise have to move to the edge of a chair or sofa and reach the floor. In some implementations, the floor height can be set for an artificial reality system across applications, or can be set for a particular application, with different floor heights set for other applications. Examples of setting floor heights are discussed below in connection with Figure 7A

[0092] Figure 6B is a flowchart illustrating a process 610 used in some implementations to set a flag to allow an application to adjust mechanisms for seated configurations. In some implementations, process 610 can be performed as a sub-process of block 514 of process 500. In some cases, process 610 can be performed in response to other triggers, such as an application changing a pose mode, a user selection, a change in the user, a start of a particular application, or the like.

[0093] ​At block 612, in response to determining that the user is in a seated posture (e.g., determined at blocks 502 and 512), process 610 can set a flag indicating the seated posture. The flag can be any type of variable, such as a binary value, a posture identifier, a posture name, etc. Setting the flag can include writing to memory of various types, such as setting a program or operating system variable, writing to a database field, writing to a file, etc. For example, an operating system for a human reality system can maintain a set of operating condition variables, one of which can be a posture indicator or “isSeated” flag. At block 614, process 610 can expose the flag set at block 612 to applications. For example, an application can send a request to the operating system to get the value of the flag, can read from a database where the flag is set, etc. Reading the flag can allow the application to modify certain mechanisms to better accommodate a seated user. For example, the application can change the positioning of virtual objects, moving them within reach, whereas if the user is standing, such adjustments can not be needed as the user can take a step to reach further objects, more easily reach objects on the floor, etc. Examples of using the flag to allow applications to adjust mechanisms for seated configurations are discussed below in connection with Figure 7B FIG. 6 is a flowchart illustrating a process 620 for customizing virtual space boundary display in response to user posture, in accordance with some implementations.

[0094] Figure 6C is a flowchart illustrating a process 620 for customizing virtual space boundary display in response to user posture, in accordance with some implementations. In some implementations, process 620 can be performed as a sub-process of block 514 of process 500. In some cases, process 620 can be performed in response to other triggers, such as an application changing posture mode, a user selection, a user’s change, launch of a particular application, etc.

[0095] At block 622, process 620 can receive a virtual space boundary display mode selection. In some implementations, this selection can be a mapping from a pose to a boundary display mode. For example, when the system detects that the user is standing (a situation in which the user can be walking around and moving more quickly than when seated), collisions of the user with the boundary are more likely to cause harm. Thus, a standing pose can be mapped to a boundary display mode that is more likely to cause the user’s attention, such as a virtual wall that appears as a particular color (e.g., bright red, green, orange, etc.) and / or has a particular pattern (e.g., a grid, tightly packed dots, flashing lines, etc.). However, when the system detects that the user is sitting or lying down (a situation in which the user can be less mobile and move more slowly than when standing), collisions of the user with the boundary are less likely to cause harm. Thus, a seated or lying down pose can be mapped to a boundary display mode that is less likely to interfere with the user’s artificial reality experience, such as a boundary that appears in a less bright color (e.g., gray, tan, brown, etc.) and / or has a particular pattern (e.g., a grid that shows only the +s at the intersection of lines, scattered dots, dim lines, etc.). In some implementations, for a seated or lying down pose, the selected boundary display mode can be a pass-through that shows real world objects into the virtual environment. In this pass-through mode, if the artificial reality system determines that the user is approaching a boundary or is about to collide with an object, the artificial reality system can show the real world object in the virtual space to allow the user to identify and avoid them. In some implementations, the real world objects shown in this pass-through mode can be dimmed or shown only as a shadow to avoid taking the user too far away from her virtual experience.

[0096] At block 624, process 620 can detect a boundary display event. This can occur by detecting that the user is within a threshold distance of a boundary, that the user’s determined trajectory is expected to intersect with a boundary, and / or that the user’s body position and configuration (e.g., arm or leg span, step length, height, motion profile, etc.) makes it likely that the user will intersect with a boundary. In some implementations, a machine learning model can be trained to make this determination. For example, inputs to the machine learning model can include inertia, position, camera and / or other sensor inputs from the artificial reality system’s head mounted device and / or controllers, a model of the user’s body (e.g., skeletal structure), past movement data for the user or an average user, boundary configuration details, etc. The machine learning model can be trained to predict whether the user is likely to intersect with a boundary (e.g., based on being trained on prior identifications of context when users intersected with boundaries). In some implementations, there can be other triggers for displaying a boundary, such as another person or object entering the virtual space, which can cause a change in the boundary or implement a pass-through mode or current application signaling to display the boundary.

[0097] At block 626, process 620 can display the boundaries or objects in the surrounding area using the selected virtual space boundary display mode selected at block 622 in response to the detected boundary display event of block 624 (i.e., implement the pass-through mode). For example, process 620 can show virtual walls or portions of virtual walls with a specified pattern and / or color, can show objects or people within a threshold distance of the user or objects that the artificial reality system determines the user is likely to collide with, can show virtual walls or objects that fade away in correspondence with their distance from the user, and so on. The following discussion illustrates examples of customizing virtual space boundary display to show boundaries when the user is in a seated posture. Figure 7C

[0098] Figure 6D is a flowchart illustrating a process 630 for implementing posture-specific virtual experiences, used in some implementations. In some implementations, process 630 can be performed as a sub-process of block 514 of process 500. In some cases, process 630 can be performed in response to other triggers, such as an application changing posture mode, a user selection, a change in the user, launch of a particular application, and so on.

[0099] At block 632, process 630 can initialize the virtual experience designed to be executed in the current posture. For example, an application can be set to be used in seated-only postures, standing-only postures, lying-down-only postures, and so on. As a more specific example, a company can specify a training program that the user should be seated for the entire duration of the training program, or else the program should be paused; a game can be configured to have a first input modality (e.g., a stationary virtual control pad) when the user is seated, and a different input modality (e.g., based on monitored user motion) if the user is standing; an application developer can wish to adjust the use of their application to the most common user postures, and thus can gather statistics about aggregate user postures; and so on.

[0100] ​At block 634, the process 630 can detect whether the user posture has changed. This can occur in a similar manner as block 502. In some implementations, only certain specific posture changes will trigger a "yes" (Y) branch from block 634. For example, while the process 630 can detect a variety of changes in posture, such as arm position or torso tilt, the current application can specify that only a change from a seated to a standing posture or a lateral movement more than a threshold distance will trigger the "yes" branch. In various implementations, the operating system for the artificial reality system or an application running by the artificial reality system can specify certain mappings between posture changes and responsive actions, and it can be the posture changes that trigger the "yes" branch from block 634 that are these mappings. If no such posture change is detected (NO branch), the process 630 can continue to block 638. If such a posture change is detected (YES branch), the process 630 can continue to block 636.

[0101] At block 636, in response to the posture change detected at block 634, the process 630 can trigger a responsive action. In some implementations, there can be responsive actions set by the artificial reality system, such as automatically stopping or pausing the virtual experience, providing a notification to resume the previous posture or a notification that the current posture is not recommended for the experience, logging the time spent in various postures, switching to another mode (e.g., a delivery mode), and so forth. In some implementations, the current application can specify one or more mappings of triggering actions to perform for specific posture changes. For example, the application can specify that if the user stands up from a sitting posture, a notification to return to the seated position should be displayed and the time spent standing should be logged, and if the time spent standing exceeds a threshold, the virtual experience should be paused. As another example, a game application can specify that when the user is in a seated posture, the virtual experience should be driving a virtual car, but if the user stands up, the virtual experience should change to show exiting the virtual car and transitioning to a virtual walking mode.

[0102] At block 638, the process 630 can determine whether the virtual experience initiated at block 632 is still in progress. The process 630 can monitor the posture and trigger responsive actions while the virtual experience continues. Once the virtual experience ends, the process 630 can also end. The following discussion implements an example of a posture-specific virtual experience. Figure 7D

[0103] Figure 6E is a flowchart illustrating a process 640 for implementing a seated workspace virtual area, used in some implementations. In some implementations, the process 640 can be performed as a sub-process of block 514 of the process 500. In some cases, the process 640 can be performed in response to other triggers, such as an application changing a posture mode, a user selection, a user's change, the launch of a particular application, and so forth. ​

[0104] At block 642, process 640 can detect a forward lean posture while the user remains in a seated posture. This can be a threshold number of degrees from vertical, such as 10, 15, or 20 degrees. In some implementations, other postures for triggering a workspace region can be detected, such as raised arms and hands or forearms resting on the desk in preparation for a virtual keyboard.

[0105] At block 644, process 640 can determine a workspace region. For example, this can be a region that is pre-established by the user; a region that has been set in size and shape (e.g., a three foot by four foot rectangle); a region defined based on characteristics of the user (e.g., a semicircle or rectangle with a size based on the user's determined arm span, such as a region that the user can reach completely); a region corresponding to an arm span of an average user (a user with similar height and other characteristics as the current user), or an average of regions manually set by other users; or a region based on computer vision / object identification (e.g., a region corresponding to the top of a desk in front of the user or a region containing the user's tools such as a keyboard, notebook, or display), among others. In various implementations, the region can be established in response to the posture detected at block 642 or can be pre-established.

[0106] At block 646, process 640 can implement a pass-through display mode for the determined workspace region. As discussed above, a pass-through mode can show portions of the real world in the artificial reality environment. For example, images taken by an external camera can be input into the artificial reality environment, portions of the display can be disabled to allow the user to see light through the display, portions of the display can be moved to allow light to enter the user's eyes, among others. By implementing a pass-through mode for the determined workspace region when a particular posture is detected, the user can easily interact with tools and other objects in the workspace region without having to remove the head mounted device of the artificial reality system or manually implement a pass-through mode. Examples of implementing pass-through for a workspace virtual region when a user is seated and leaning forward are discussed below in connection with FIGS. 7A-7C. Figure 7E

[0107] Figure 6F is a flow diagram illustrating a process 650 for automatically customizing a virtual region in a seated mode in some implementations. In some implementations, process 650 can be performed as a sub-process of block 514 of process 500. In some cases, process 650 can be performed in response to other triggers, such as an application changing posture mode, a user selection, a change in the user, launch of a particular application, among others.

[0108] ​At block 652, process 650 can automatically determine a region for the virtual space for the user in the seated posture. Process 650 can use a pre-established shape for the region, such as a cylinder centered on the user, a half-cylinder in front of the user, a cuboid, or others. The size of the region can be set based on settings specified by user input, a determined arm span of the user, statistical data about the user or users identified as similar to the current user regarding general arm span, an identification of real-world objects identified by the artificial reality system to exclude from the virtual space, or an identification of virtual objects identified by the artificial reality system to include in the virtual space. For example, process 650 can determine the virtual region to be a half-cylinder in front of the user with a radius equal to half of the user’s arm span (i.e., the length of one arm). As another example, process 650 can determine the virtual region to be a cuboid in front of the user that excludes all real-world objects in the region. In some implementations, the determined virtual region can be suggested to the user, who can manually adjust its parameters or define a different virtual region, e.g., with different size, shape, or activation features.

[0109] In some implementations, the arm span of the user can be determined by setting the arm span to equal the height of the user. In some cases, the setting can be updated based on identifying the hand or controller positions and extending the determined arm span if the distance of the identified hand or controller extends beyond the determined arm span. In other cases, the arm span can be retracted if the identified hand or controller never extends to the determined arm span for a threshold amount of time or amount of use of the artificial reality system. In some implementations, rather than determining from an initial height determination, the arm span can be determined directly by observing these hand or controller distances. In some implementations, the arm span determination can be determined or adjusted by identifying the user’s arms in an image of the user and projecting a maximum reach using a body movement model (e.g., a kinematic skeleton model).

[0110] At block 654, process 650 can set the virtual region determined at block 652. In various implementations, the virtual region can be set globally for all instances of the current user, for use cases when running a particular application, or for use cases during a particular task. The following discussion of examples of automatically setting virtual space dimensions can apply to any of these cases. Figure 7F

[0111] Figure 7A ​is a conceptual diagram illustrating an example 700 used in some implementations for implementing floor height customization when a user is seated. In example 700, the virtual space configuration system has determined that user 702 is in a seated posture. In response, the virtual space configuration system implements an option for user 702 to adjust the height of virtual floor 704. As shown by arrow 706, user 702 can activate a control (e.g., a virtual control shown in the artificial reality environment, a control on a controller, using her gaze, etc.) to adjust the height of virtual floor 704. The height of virtual floor 704 can cause applications to place objects no lower than that virtual floor 704. As shown in example 700, by raising virtual floor 704, user 702 can more easily reach any objects placed on virtual floor 704 while seated.

[0112] Figure 7B is a conceptual diagram illustrating an example 710 used in some implementations for using a flag to allow applications to adjust for seated use configurations. In example 710, user 702 is initially standing, surrounded by objects 712A-E. When the virtual space configuration system detects a change in posture of user 702 to now be seated, the virtual space configuration system sets a seated flag, which is exposed to applications in the control of objects 712. Applications can adjust the position of objects 712 (as shown by arrows 714A-E) to be within reach of seated user 702.

[0113] Figure 7C is a conceptual diagram illustrating an example 720 used in some implementations for customizing virtual space boundary display in response to user posture. Example 720 first shows user 702 in a standing posture, with virtual space 722 having boundary walls, portions of which appear when user 702 is within a threshold distance of the portions of the boundary walls. The boundary walls are configured with a virtual space boundary display mode selection, which causes a red grid pattern (e.g., lines 724) to be displayed on the walls. In the second portion of example 720, user 702 has assumed a seated posture. At this time, the virtual space boundary display mode is changed for virtual space 722 to cause real world objects within a threshold distance of user 702 to be shown in a pass-through mode. In example 720, these objects are truncated at lines 726A-D, showing where they are no longer within the threshold distance of user 702.

[0114] Figure 7Dis a conceptual diagram illustrating an example 730 used in some implementations for implementing a seated-only virtual experience. In example 730, user 702 is initially in a seated position, and the virtual space configuration system executes a seated-only virtual experience. When user 702 transitions to a standing posture, the virtual space configuration system pauses the seated-only virtual experience and displays a message 732 informing user 702 that the seated-only virtual experience will resume when user 702 returns to a seated posture.

[0115] Figure 7E is a conceptual diagram illustrating an example 740 used in some implementations for implementing a seated workspace virtual region. In example 740, user 702 starts in a seated, upright posture. As indicated by arrow 742, user 702 then leans forward while remaining seated. In response to detecting this seated, leaning-forward posture, the virtual space configuration system displays a workspace region 744, which is a region of the virtual space that the virtual space configuration system has identified as corresponding to the surface of a desk in front of the user that contains a keyboard and monitor.

[0116] Figure 7F is a conceptual diagram illustrating an example 750 used in some implementations for automatically customizing virtual regions in seated mode. Example 750 shows a first instance in which the virtual space configuration system automatically determines the size 752 of the virtual space based on a determined arm span of user 702, which is configured to be a half-cylinder in front of user 702. Example 750 also shows a second instance in which the virtual space is a cuboid in front of user 702, the length and width dimensions 754 of which are set based on a statistical average of regions selected by other users whose heights are within a threshold of the height of user 702.

[0117] References in the specification to “implementation” (e.g., “some implementations,” “various implementations,” “one implementation,” “an implementation,” etc.) indicate that the particular feature, structure, or characteristic being discussed is included in at least one implementation of the present disclosure. The appearance of the phrases “in some implementations” or “in other implementations” in various places in the specification are not necessarily all referring to the same implementation, nor are separate or alternative implementations mutually exclusive of other implementations. Moreover, various features are described which can be exhibited by some implementations and not by others. Similarly, various requirements are described which can be requirements for some implementations and are not for other implementations.

[0118] As used herein, above a threshold means that the value of the item being compared is higher than a specified other value, the item being compared is among a specified number of items having the highest value, or the item being compared has a value within a specified top percentage of values. As used herein, below a threshold means that the value of the item being compared is lower than a specified other value, the item being compared is among a specified number of items having the lowest value, or the item being compared has a value within a specified bottom percentage of values. As used herein, within a threshold means that the value of the item being compared is between two specified other values, the item being compared is among an intermediate specified number of items, or the item being compared has a value within an intermediate specified percentage range. Relative terms, such as high or insignificant, when not otherwise defined, can be understood to assign a value and determine how that value compares to an intended threshold. For example, the phrase "select a fast connection" can be understood to mean selecting a connection with an assigned value corresponding to its connection speed being above a threshold.

[0119] As used herein, the word "or" means any possible permutation of a set of items. For example, the phrase "A, B, or C" means at least one of A, B, C, or any combination thereof, such as: A; B; C; A and B; A and C; B and C; A, B, and C; or any multiple thereof, such as A and A; B, B, and C; A, A, B, B, C, and C; and so on.

[0120] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications are possible without departing from the scope of the embodiments and implementations. The specific features and acts as described above are disclosed as example forms of implementing the appended claims. Accordingly, the embodiments and implementations are not limited to the specific features and acts described above.

[0121] Any patents, patent applications, and other references noted above are incorporated herein by reference. Aspects can be modified to employ systems, functions, and concepts of the various references noted above to provide further implementations, if desired. If there is a conflict between a statement or teaching of a document incorporated by reference and a statement or teaching of the present application, the present application shall control.

Claims

1. A method for customizing a virtual space based on user gestures, the method comprising: Determine if the user's posture corresponds to the seating mode; as well as In response to the determination, seating customization for the virtual space is set in the following manner: Offers customizable first seating, including: Obtain a measurement for floor height; and Based on the metric for the floor height, the system floor height is adjusted, wherein the system floor height is set to a minimum height for virtual objects relevant to the user, and wherein the system floor height is adjusted so that the system floor height is higher than the height of the real-world floor; or Offers second seating customization, including: Set up seating signs; and The seating sign is displayed to a first application, wherein the first application adjusts a first operating mechanism based on the seating sign; and The seating sign is displayed to a second application, which then performs a second operational mechanism adjustment based on the seating sign, different from the first operational mechanism adjustment. Setting the seating customization includes providing the first seating customization.

2. The method of claim 1, wherein the measurement of the floor height is based on one or more of the following: The user's determined standing height; The user's determined arm span; The defined dimensions of the object on which the user sits; or Any combination of them.

3. The method of claim 1 or claim 2, wherein the metric for the floor height is determined by a machine learning model trained based on previous user selection to receive indications from sensor or camera measurements and generate the floor height metric.

4. The method of claim 1 or claim 2, wherein the measurement for the floor height is a value selected by one or more users, indicated by one or more of the following: User hand gestures User input to the controller, User voice commands. User's gaze direction, or Any combination of them.

5. The method according to claim 1 or claim 2, wherein setting the seating customization further includes providing the second seating customization.

6. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for customizing a virtual space based on user gestures, the operations including: Determine if the user's posture corresponds to the seating mode; as well as In response to the determination, seating customization for the virtual space is set in the following manner: Offers customizable first seating, including: Obtain a measurement for floor height; and Based on the metric for the floor height, the system floor height is adjusted, wherein the system floor height is set to a minimum height for virtual objects relevant to the user, and wherein the system floor height is adjusted so that the system floor height is higher than the height of the real-world floor; or Offers second seating customization, including: Set up seating signs; and The seating sign is displayed to a first application, wherein the first application adjusts a first operating mechanism based on the seating sign; and The seating sign is displayed to a second application, which then performs a second operational mechanism adjustment based on the seating sign, different from the first operational mechanism adjustment. Setting the seating customization includes providing the first seating customization.

7. The computer-readable storage medium of claim 6, wherein the measurement of the floor height is based on one or more of the following: The user's determined standing height; The user's determined arm span; The defined dimensions of the object on which the user sits; or Any combination of them.

8. The computer-readable storage medium of claim 6 or claim 7, wherein the measurement of the floor height is a user-selected value indicated by one or more of the following: User hand gestures User input to the controller, User voice commands. User's gaze direction, or Any combination of them.

9. The computer-readable storage medium of claim 6 or claim 7, wherein the measurement of the floor height is determined by a machine learning model trained based on prior user selection to receive indications from sensor or camera measurements and generate the floor height measurement.

10. A computing system for customizing a virtual space based on user gestures, the computing system comprising: One or more processors; as well as One or more memories, the one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform operations including: Determine if the user's posture corresponds to the seating mode; as well as In response to the determination, seating customization for the virtual space is set in the following manner: Offers customizable first seating, including: Obtain a measurement for floor height; and Based on the metric for the floor height, the system floor height is adjusted, wherein the system floor height is set to a minimum height for virtual objects relevant to the user, and wherein the system floor height is adjusted so that the system floor height is higher than the height of the real-world floor; or Offers second seating customization, including: Set up seating signs; and The seating sign is displayed to a first application, wherein the first application adjusts a first operating mechanism based on the seating sign; and The seating sign is displayed to a second application, which then performs a second operational mechanism adjustment based on the seating sign, different from the first operational mechanism adjustment. Setting the seating customization includes providing the first seating customization.

Images