Posture-based virtual space configuration

Abstract

A virtual space configuration system of an artificial reality system can detect user posture and provide various corresponding customizations of the system's virtual space. The virtual space configuration system can provide sitting virtual space customizations when the user is in a sitting posture. In various implementations, these customizations may include allowing adjustment of floor height; setting flags that can be surfaced for applications to adjust the dynamics of applications for seated users; customizing the display of virtual space boundaries to be less obtrusive when in sitting mode; providing options to detect when the user leaves sitting mode and trigger corresponding actions; providing a pass-through workspace area that allows the user to naturally interact with certain real-world objects without needing to remove the virtual reality headset; or automatically determining virtual space dimensions for seated users.

Description

Technology Field

[0001] The present invention relates to controlling the configurations of a virtual space for an artificial reality environment. Background Technology

[0002] While a user views and interacts with "virtual objects"—computer-generated object representations appearing in an artificial reality environment—the user's physical movements occur in the real world. In some cases, the artificial reality system may prevent the user from seeing part or all of the real world, or the user may be distracted by virtual objects, potentially causing the user to accidentally collide with real-world objects or exit the designated area for interaction within the artificial reality environment. In other cases, the user's movements may be restricted by their physical posture in the real world, which can cause some difficulties in interacting within the artificial reality environment. For example, some virtual objects may be located out of reach, making it difficult for the user to interact with them from their current posture.

[0003] According to a first aspect of the present invention, a method for customizing a virtual space based on a user posture is provided, said method:

[0004] A step of determining that the user posture corresponds to a sitting mode;

[0005] In response to the above decision, as a step of setting seated customization for the virtual space:

[0006] As providing first seating customization:

[0007] Acquiring metrics for floor height; and

[0008] Providing the first seating customization, including adjusting the system floor height based on a metric for the floor height; or

[0009] As providing a second sitting customization:

[0010] Setting the seated flag; and

[0011] Providing the second sitting customization, wherein the one or more applications surface the sitting flag for one or more applications, the applications adjusting operation dynamics based on the sitting flag; or

[0012] As providing third-party seating customization:

[0013] Receiving a boundary mode selection for the above virtual space;

[0014] Detecting boundary display events; and

[0015] Based on the above-mentioned selected boundary mode,

[0016] A boundary configured based on the above-mentioned selected boundary mode; or

[0017] The method includes the step of setting the seating customization by providing the third seating customization, which includes displaying real-world objects in the virtual space.

[0018] Setting the above-mentioned seating customization may preferably include providing the above-mentioned first seating customization.

[0019] In some embodiments, the metrics for the floor height are based on: the user's determined standing height; the user's determined arm span; the determined dimensions of the object on which the user is sitting; or any combination thereof.

[0020] In some embodiments, metrics for floor height are determined by a machine learning model trained based on previous user selections to receive indications of sensor or camera measurements and to generate floor height metrics.

[0021] In some embodiments, the metrics for floor height are one or more user-selected values ​​indicated by one or more of: user hand posture, user input to the controller, user voice command, user gaze direction, or any combination thereof.

[0022] In some embodiments, setting the sitting customization includes providing a second sitting customization.

[0023] In some embodiments, setting the sitting customization includes providing a third sitting customization.

[0024] The boundary mode selection can preferably be based on the mapping of attitudes versus boundary modes provided by the current application.

[0025] In some embodiments, the selected boundary mode is a passthrough mode that allows the display to become a display of real-world objects in virtual space.

[0026] Detecting boundary display events may include receiving inertial data, position data, camera image data, a model of the user's bone structure, or any combination thereof, and applying a machine learning model trained to generate a projection of whether the user will interact with the boundary.

[0027] According to a second aspect of the present invention, a computer-readable storage medium is provided that stores instructions for causing the computing system to perform actions for customizing a virtual space based on a user posture when executed by the computing system.

[0028] The above operations are:

[0029] Determining that the user posture corresponds to sitting mode;

[0030] In response to the above determination that the above user posture corresponds to the above sitting mode,

[0031] As setting seating customization for the above virtual space:

[0032] As providing first sitting customization, while the virtual experience configured for sitting mode is in progress:

[0033] Determining that the second user posture corresponds to the fact that it is no longer in the sitting mode; and

[0034] In response to this, providing the first sitting customization, including triggering a response action for the virtual space; or

[0035] As providing a second sitting customization:

[0036] Determining that the second user posture corresponds to bending forward;

[0037] Determining the workspace area; and

[0038] Providing the second sitting customization, including activating a display mode that displays real-world objects in the workspace area in a virtual environment; or

[0039] As providing third-party seating customization:

[0040] Automatically determining the dimensions of the virtual space based on the physical characteristics of the user; and

[0041] It includes setting the seating customization by providing the third seating customization, which includes adjusting the virtual space based on the determined dimensions above.

[0042] In some embodiments, setting the sitting customization includes providing a first sitting customization.

[0043] In some embodiments, the response action comprises: automatically stopping or pausing the virtual experience; providing a notice to resume a previous pose or that the current pose is not recommended for the virtual experience; logging the times during which various poses are maintained; switching to a display mode that shows real-world objects; changing input forms; or any combination thereof.

[0044] In some embodiments, setting the sitting customization includes providing a second sitting customization.

[0045] In some embodiments, the workspace area is determined based on an area defined based on the user's determined upper limb width; the average of workspace areas previously manually set by other users; an area corresponding to the top of a flat real-world object in front of the user; an area determined to surround one or more specified real-world tools; or any combination thereof.

[0046] In some embodiments, setting the sitting customization includes providing a third sitting customization.

[0047] The user's physical characteristics may preferably include an automatically determined user upper limb width.

[0048] The user's upper limb width can preferably be automatically determined by setting the initial upper limb width to be equal to the determined user height; and by updating the initial upper limb width based on identifying the user's hand or controller positions that extend beyond the determined upper limb width.

[0049] According to a third aspect of the present invention, a computing system for customizing a virtual space based on a user posture is provided, and the computing system:

[0050] One or more processors; and

[0051] It includes one or more memories that store instructions for the computing system to perform operations when executed by the one or more processors mentioned above, and

[0052] The above operations are:

[0053] Determining that the user posture corresponds to sitting mode;

[0054] In response to the determination that the above user posture corresponds to the above sitting mode, setting a sitting customization for the above virtual space:

[0055] As providing first seating customization:

[0056] Acquiring metrics for floor height; and

[0057] Providing the first seating customization, including adjusting the system floor height based on the metrics for the floor height; or

[0058] As providing a second sitting customization:

[0059] Setting the sit flag; and

[0060] Providing the second sitting customization, comprising surfacing the sitting flag for one or more applications, wherein the one or more applications adjust motion dynamics based on the sitting flag; or

[0061] As providing third-party seating customization:

[0062] Receiving a boundary mode selection for the above virtual space;

[0063] Detecting boundary display events; and

[0064] Providing the third sitting customization, including displaying boundary types or objects in the virtual space based on the selected boundary mode; or

[0065] As providing the fourth sitting customization, while the virtual experience configured for sitting mode is in progress:

[0066] Determining that the second user posture corresponds to the fact that it is no longer in sitting mode; and

[0067] Providing the fourth sitting customization, including triggering a response action to the virtual experience; or

[0068] As providing fifth sitting customization:

[0069] Determining that the second user posture corresponds to bending forward;

[0070] Determining the workspace area; and

[0071] Providing the fifth seating customization, including activating a display mode that displays real-world objects in the workspace area in a virtual environment; or

[0072] As providing the 6th seating customization:

[0073] Automatically determining the size of the virtual space based on the physical characteristics of the user; and

[0074] It includes setting the seating customization by one or more of providing the sixth seating customization, including adjusting the virtual space based on the size determined above. Brief explanation of the drawing

[0075] FIG. 1 is a block diagram illustrating an overview of devices on which some implementations of the present technology can operate. FIG. 2a is a wiring diagram illustrating a virtual reality headset that can be used in some implementations of the present technology. FIG. 2b is a wiring diagram illustrating a mixed reality headset that can be used in some implementations of the present technology. FIG. 3 is a block diagram illustrating an overview of an environment in which some implementations of the present technology can operate. FIG. 4 is a block diagram illustrating components that can be used in a system using the disclosed technology in some implementations. FIG. 5 is a flowchart illustrating a process used in some implementations of the present technology for setting up virtual space configurations based on user posture. Figure 6a is a flowchart illustrating a process used in some implementations to enable floor height customization when a user sits. FIG. 6b is a flowchart illustrating a process used in some implementations to set a flag to allow applications to adjust the dynamics for sitting configurations. Figure 6c is a flowchart illustrating a process used in some implementations to customize virtual space boundary displays in response to user posture. Fig. 6d is a flowchart illustrating a process used in some implementations to enable sit-only virtual experiences. Figure 6e is a flowchart illustrating a process used in some implementations to enable a virtual area of ​​the sitting workspace. Figure 6f is a flowchart illustrating a process used in some implementations to automatically customize the virtual area in sitting mode. Figure 7a is a conceptual diagram illustrating an example that enables floor height customization when a user sits. FIG. 7b is a conceptual diagram illustrating an example of using flags to allow applications to adjust dynamics for sitting usage configurations. FIG. 7c is a conceptual diagram illustrating an example of customizing virtual space boundary displays in response to user posture. Fig. 7d is a conceptual diagram illustrating an example that enables sitting-only virtual experiences. Figure 7e is a conceptual diagram illustrating an example that enables a virtual area of ​​a sitting workspace. Fig. 7f is a conceptual diagram illustrating an example of automatically customizing a virtual area in sitting mode. The technologies introduced herein can be better understood by referring to the following detailed description together with the accompanying drawings, and similar reference numbers indicate identical or functionally similar elements. Specific details for implementing the invention

[0076] Although embodiments for customizing a virtual space based on user posture are described herein, artificial reality systems may define a specific “virtual space” for a user experience, which may define the range of movement of the user during the experience, control how virtual objects are displayed or positioned within the experience, and / or set system actions in response to changes in posture. For example, when a user approaches the edge of a defined virtual space, the artificial reality system may provide a warning or activate a pass-through mode that shows user real-world objects to which she might collide. As used herein, “posture” is the position or configuration of one or more parts of the user’s body. For example, a posture may be sitting, standing, lying down, reaching out arms, a specific hand position or gesture, head orientation, torso rotation, etc. In some embodiments, a posture may also include movement, such as a specific motion of one or more body parts and / or a point or object. For example, the first identification posture may be standing still, but the second identification posture may be standing moving (e.g., the user makes critical lateral movements about a center point).

[0077] A virtual space configuration system, which may be a subsystem of an artificial reality system, can detect user posture and provide various corresponding customizations of the system's virtual space. In some implementations, postures that the virtual space configuration system can identify include standing (which can be divided into standing and moving or standing and stationary), sitting, lying down, etc. In various implementations, these decisions may be automatic based on user input, or may be automatically determined and user-confirmed. For example, the virtual space configuration system can determine the height of the artificial reality system's headset by comparing it to the identified floor, and can determine whether the headset height corresponds to a standing or sitting position by using a known user height or the average of the heights of multiple users. Additionally, when the position is determined to be standing, the virtual space configuration system can determine whether the lateral position of the headset has moved beyond a threshold amount from the center point to determine whether the standing user is stationary or moving. The virtual space configuration system can provide the user with an indication of the determined posture for the user to confirm or modify.

[0078] When in a standing moving position, the virtual space configuration system may set up a virtual space defined by the user relative to the current real-world environment of the artificial reality system, and / or the virtual space configuration system may automatically detect objects around the user's current location and set up the virtual space to avoid collisions with these objects. When the user approaches these boundaries, the virtual space configuration system may provide a warning or display a grid indicating the edges. When in a standing stationary position, the virtual space configuration system may define a virtual space around the user as a cylindrical area or a "wine glass" shape (i.e., a cylinder that is narrow at the bottom and wide at the top), taking into account, for example, the user's legs while stationary but providing space around the upper part of the user where her arms are moving. In some implementations, the diameter of the upper part of this cylindrical or wine glass shape may be based on user characteristics, such as a determined upper limb width.

[0079] When it is determined that the user is in a sitting or lying mode, the virtual space configuration system can provide various different virtual space customizations. In one instance, the virtual space configuration system can obtain metrics regarding different floor heights to use when the user is sitting. These metrics may come from, for example, a machine learning model trained to predict the desired floor height, user input specifying floor height changes (e.g., using controllers, gestures, or tracked user gaze), and / or past floor height settings from the user or users determined to have similar characteristics. The virtual space configuration system can then set the floor height based on these metrics. This improves user accessibility while in the virtual space by setting a minimum height for virtual objects related to the user, thereby eliminating instances where the user must move to the corners of chairs or couches and reach the floor.

[0080] In another instance, a virtual space configuration system can enable adjustments for application dynamics specific to seated or reclining users. For example, a notified application can adjust the placement of virtual objects so that they are within the normal or measured upper limb width of the user when the user is sitting or lying down. For example, virtual objects that the user typically takes action to interact with can be automatically moved within reach. This can respond to flags set by the virtual space configuration system for sitting and / or reclining modes, which can consequently surface for applications. Applications can be customized to have different dynamics based on these flags.

[0081] In another case, the virtual space configuration system can configure a boundary mode based on the user's posture. In one scenario, when the user is in a standing position, the virtual space may have established boundaries, and the virtual space configuration system will display a boundary or warning when it predicts that the user might interact with the boundary. For example, when the user is in a standing position, the boundary could be a red grid, which would immediately attract the user's attention if displayed in the virtual space. However, collisions with boundaries are less likely to be an issue when sitting, as the user is likely to move more slowly or only move their arms. Therefore, when sitting, the boundary could be a much less distracting pattern, such as a pattern of small gray crosses (e.g., +). Alternatively, instead of displaying boundaries when sitting, the system can identify real-world objects around the user and display them in the virtual space when the virtual space configuration system predicts that the user might collide with them (e.g., when they are within the user's upper-limb width).

[0082] In another instance, a virtual space configuration system can enable experiences that are available only when the user is in a specific posture or that trigger specific actions when the user transitions between postures. For example, after determining that the user is in a seated posture, the artificial reality system can initiate a "sit-only" experience. The virtual space configuration system can continuously monitor the user's posture throughout the experience. If the user is standing, this can trigger the artificial reality system to take actions such as automatically pausing the sit-only experience, notifying the user to return to a seated position, logging the time the user spent standing during the experience, switching to a pass-through mode where real-world aspects are displayed instead of parts of the experience, and / or changing aspects of the experience, such as providing different input modes or altering virtual objects.

[0083] Furthermore, the virtual space configuration system can provide a virtual "workspace" area that appears when the user is in a specific additional posture, such as sitting and leaning forward. The workspace may be an area in front of the user based on one or more of the identification of areas including, for example, determined user upper limb width, general user arm length statistics, previous user settings, the area where the user is drawn, and / or specific objects (e.g., keyboard, monitors, mouse, desk area, etc.). The virtual space configuration system can also detect that the user has leaned forward by at least a threshold amount while sitting. In some implementations, this may also depend on whether a flat workspace (such as a desk) is identified in front of the user. When making these additional posture determinations, the virtual space configuration system may activate a pass-through mode, that is, a mode that displays a representation of at least a portion of the real world, in this case, the determined workspace area. This allows the user to quickly and easily transition between interacting with real-world items in the workspace area and virtual objects in the virtual space.

[0084] In another case, the virtual space configuration system can automatically customize the dimensions (e.g., size and / or shape) of the virtual area for the sitting mode. The virtual space configuration system may determine the dimensions based on context or user details, such as determined user upper limb width, average or determined statistics of similar users, previous user settings, or the area drawn by the user, or by identifying objects in the surrounding area. Based on the determined dimensions, the virtual space configuration system can then set the virtual area, for example, as a rectangle or semicircle in front of the user or a complete circle around the user.

[0085] Embodiments of the disclosed technology may include or be implemented with an artificial reality system. Artificial reality or extra-reality (XR) is a form of reality that has been adjusted in some way prior to presentation to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivatives thereof. Artificial reality content may include entirely generated content or generated content combined with captured content (e.g., real-world photographs). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be provided in a single channel or in multiple channels (such as stereo video that creates a 3-dimensional effect for the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof used to generate content in artificial reality and / or used in artificial reality (e.g., performing activities in it). An artificial reality system that provides artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0086] As used herein, “Virtual Reality” or “VR” refers to an immersive experience in which a user’s visual input is controlled by a computing system. “Augmented Reality” or “AR” refers to systems in which a user views images of the real world after passing through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display those images from the camera onto a screen on the opposite side of the tablet. The tablet can process, adjust, or “enlarge” the images as they pass through the system, such as by adding virtual objects. “Mixed Reality” or “MR” refers to systems in which light entering the user’s eyes is partially generated by a computing system and partially composed of light reflected from objects in the real world. For example, an MR headset can be formed as glasses with a pass-through display, which allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, thereby allowing the MR headset to provide virtual objects mixed with real objects that the user can see. "Artificial Reality," "Extra Reality," or "XR" refers to any of VR, AR, MR, or any combination or hybrid thereof, as used herein.

[0087] There are existing XR systems that provide virtual spaces. However, these XR systems can be difficult to use and offer limited functionality. Existing XR systems often fail to distinguish user postures when configuring virtual spaces, requiring users to manually adjust the virtual space or to perform actions within a virtual space that is difficult to use, distracting, or may not allow for specific options. For example, when a user is sitting, standard XR systems do not provide options to adjust the floor position, often requiring the user to move out of their seat to reach virtual objects located on the floor. As another example, existing XR systems typically feature a single warning system for when a user attempts to collide with a virtual space wall. However, this is distracting, and since such collisions are unlikely to cause random damage, it may be unnecessary when the user is sitting. Furthermore, existing XR systems require extensive setup for virtual spaces, which may not be necessary for sitting configurations where the virtual space is likely smaller and less likely to require specific contours.

[0088] The virtual space configuration systems and processes described herein are expected to overcome these problems associated with conventional XR systems and provide users with greater control over virtual spaces. The disclosed virtual space configuration systems and processes are also expected to provide more functionality and a user experience that is more natural and intuitive than interactions in existing XR systems. Despite being natural and intuitive, the virtual space configuration systems and processes described herein are rooted in computerized artificial reality systems rather than being analogs of conventional interactions. For example, these virtual space configuration systems and processes determine when a user is seated and can provide virtual space customizations in response. One such virtual space customization may allow for adjustment of floor height. Another virtual space customization may set flags that can be surfaced for applications to adjust application dynamics. In addition, virtual space customizations may customize the display of seat-mode virtual space boundaries to be less intrusive. Further virtual space customizations may provide options to detect when a user leaves seat mode and trigger corresponding actions. Another virtual space customization can provide a pass-through workspace area that allows users to naturally interact with specific real-world objects without the need to remove the virtual reality headset. Another virtual space customization can automatically determine virtual space dimensions for seated users.

[0089] Several implementations are discussed in more detail below with reference to the drawings. FIG. 1 is a block diagram illustrating an overview of devices in which some implementations of the disclosed technology may operate. The devices may include hardware components of a computing system (100) capable of determining user posture and setting corresponding virtual space customizations. In various implementations, the computing system (100) may include a single computing device (130) and a plurality of computing devices (e.g., computing device (101), computing device (102), and computing device (103)) that communicate via wired or wireless channels to distribute processing and share input data. In some implementations, the computing system (100) may include a standalone headset capable of providing a computer-generated or augmented experience to a user without requiring external processing or sensors. In other implementations, the computing system (100) may include a plurality of computing devices, such as a headset and a core processing component (such as a console, mobile device, or server system), in which some processing operations are performed on the headset and others are offloaded to a core processing component. Exemplary headsets are described below with respect to FIGS. 2a and FIGS. 2b. In some implementations, location and environment data may be collected only by sensors integrated into the headset device, but in other implementations, one or more of the non-headset computing devices may include sensor components capable of tracking environment or location data.

[0090] A computing system (100) may include one or more processor(s) (110) (e.g., central processing units (CPUs), graphics processing units (GPUs), hologram processing units (HPUs), etc.). The processors (110) may be a single processing unit or multiple processing units in a device, or may be distributed across a number of devices (e.g., may be distributed across two or more of the computing devices (101 to 103)).

[0091] A computing system (100) may include one or more input devices that provide input to processors (110) and may notify actions to them. Actions may be mediated by a hardware controller that interprets signals received from the input devices and transmits information to the processors (110) using a communication protocol. Each input device (120) may include, for example, a mouse, keyboard, touchscreen, touchpad, wearable input device (e.g., haptic glove, bracelet, ring, earring, necklace, watch, etc.), camera (or other light-based input device, e.g., infrared sensor), microphone, or other user input device.

[0092] Processors (110) may be coupled to other hardware devices by using an internal or external bus, such as a PCI bus, a SCSI bus, or a wireless connection, for example. Processors (110) may communicate with a hardware controller for the devices, such as for a display (130). The display (130) may be used to display text and graphics. In some implementations, the display (130) includes an input device as part of the display, such as when the input device is a touchscreen or equipped with an eye direction monitoring system. In some implementations, the display is separated from the input device. Examples of display devices include: an LCD display screen, an LED display screen, a projection, a hologram, or an augmented reality display (such as a head-up display device or a head-mounted device), etc. Other I / O devices (140) may also be coupled to the processor, such as a network chip or card, a video chip or card, an audio chip or card, a USB, a FireWire or other external device, a camera, a printer, speakers, a CD-ROM drive, a DVD drive, a disk drive, etc.

[0093] The computing system (100) may include a communication device capable of communicating wirelessly or wire-based with other local computing devices or network nodes. The communication device may communicate with another device or server over a network using, for example, TCP / IP protocols. The computing system (100) may use the communication device to distribute operations across multiple network devices.

[0094] Processors (110) may have access to memory (150), which may be included in one of the computing devices of the computing system (100) or distributed across multiple computing devices of the computing system (100) or other external devices. Memory may include one or more hardware devices for volatile or non-volatile storage and may include both read-only and writable memory. For example, memory may include random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and one or more writable non-volatile memory such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, etc. Memory is not a propagating signal separated from the underlying hardware; memory is therefore non-transient. Memory (150) may include program memory (160) for storing programs and software, such as an operating system (162), a virtual space configuration system (164), and other application programs (166). The memory (150) may also include a data memory (170) that may include various models (e.g., posture classifiers, boundary collision predictors, user height or upper limb-width identifiers, etc.), floor height settings, sitting flag variables, boundary mode variables and associated display configurations, posture change mappings, virtual experiences, workspace area settings, virtual area settings, other configuration data, settings, user options or preferences, etc., which may be provided as any element of the program memory (160) or computing system (100).

[0095] Some implementations may be capable of operating with a number of different computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, mobile phones, wearable electronic devices, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0096] FIG. 2a is a wiring diagram of a virtual reality head-mounted display (HMD) (200) according to some embodiments. The HMD (200) includes a front rigid body (205) and a band (210). The front rigid body (205) includes one or more electronic display elements of an electronic display (245), an inertial motion unit (IMU) (215), one or more position sensors (220), locators (225), and one or more computing units (230). The position sensors (220), IMU (215), and computing units (230) may be located inside the HMD (200) and may not be visible to the user. In various implementations, the IMU (215), position sensors (220), and locators (225) may track the movement and position of the HMD (200) in the real world and in a virtual environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locators (225) may emit infrared light beams that generate light points on real objects around the HMD (200). One or more cameras (not shown) integrated with the HMD (200) may detect the light points. Computing units (230) in the HMD (200) may use the detected light points to estimate the location and movement of the HMD (200) as well as to identify the shape and location of real objects around the HMD (200).

[0097] The electronic display (245) may be integrated with the front rigid body (205) and may provide image light to the user as directed by the computing units (230). In various embodiments, the electronic display (245) may be a single electronic display or a plurality of electronic displays (e.g., displays for each user's eye). Examples of the electronic display (245) include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, laser, etc.), some other displays, or some combinations thereof.

[0098] In some implementations, the HMD (200) may be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor the HMD (200) available to the PC in combination with the output from the IMU (215) and position sensors (220) to determine the location and movement of the HMD (200) (e.g., through light emitted from the HMD (200)).

[0099] In some implementations, the HMD (200) may communicate with one or more other external devices, such as controllers (not shown) that the user may hold in one or both hands. The controllers may have their own IMU units, position sensors, and / or emit additional light points. The HMD (200) or external sensors may track these controller light points. Computing units (230) or core processing components in the HMD (200) may use this tracking in combination with the IMU and position outputs to monitor the user's hand positions and motions. The controllers may also include various buttons that the user can operate to provide input and interact with virtual objects. In various implementations, the HMD (200) may also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc. In some implementations, instead of or in addition to controllers, one or more cameras included in or outside the HMD (200) may monitor the positions and postures of the user's hands to determine gestures and other hand and body motions.

[0100] FIG. 2b is a wiring diagram of a mixed reality HMD system (250) comprising a mixed reality HMD (252) and a core processing component (254). The mixed reality HMD (252) and the core processing component (254) may communicate via a wireless connection (e.g., a 60 GHz link) as indicated by a link (256). In other implementations, the mixed reality system (250) may include only a headset without an external computing device, or may include other wired or wireless connections between the mixed reality HMD (252) and the core processing component (254). The mixed reality HMD (252) includes a pass-through display (258) and a frame (260). The frame (260) may house various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye-tracking sensors, MEMS components, networking components, etc.

[0101] Projectors may be coupled to a pass-through display (258), for example, via optical elements, to display media to a user. Optical elements may include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., to direct light from the projectors toward the user's eye. Image data may be transmitted from the core processing component (254) to the HMD (252) via a link (256). Controllers in the HMD (252) may convert the image data from the projectors into light pulses, which may be transmitted to the user's eye via optical elements as output light. The output light may be mixed with light passing through the display (258) to allow the output light to provide virtual objects that appear to exist in the real world.

[0102] Similar to the HMD (200), the HMD system (250) may also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system (250) to track itself, for example, in 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects so that they appear not to move as the HMD (252) moves, and have virtual objects for gestures and other real-world objects.

[0103] FIG. 3 is a block diagram illustrating an overview of an environment (300) in which some implementations of the disclosed technology may operate. The environment (300) may include one or more client computing devices (305A to 305D), examples of which may include a computing system (100). In some implementations, some of the client computing devices (e.g., client computing device (305B)) may be an HMD (200) or an HMD system (250). The client computing devices (305) may operate in a networking environment using logical connections to one or more remote computers, such as a server computing device, through the network (300).

[0104] In some implementations, the server (310) may be an edge server that receives client requests and coordinates the fulfillment of these requests through other servers, such as servers (320A through 320C). The server computing devices (310 and 320) may include computing systems, such as a computing system (100). Although each server computing device (310 and 320) is logically displayed as a single server, the server computing devices may be a distributed computing environment comprising multiple computing devices located in the same or geographically completely different physical locations.

[0105] Client computing devices (305) and server computing devices (310 and 320) can each act as a server or client for other server / client device(s). The server (310) can connect to the database (315). The servers (320A to 320C) can each connect to the corresponding database (325A to 325C). As discussed above, each server (310 or 320) can correspond to a group of servers, and each of these servers can share a database or have its own database. Although the databases (315 and 325) are logically displayed as single units, the databases (315 and 325) can each be a distributed computing environment including multiple computing devices, located within their corresponding servers, or located at the same or geographically completely different physical locations.

[0106] The network (330) may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless networks. The network (330) may be the Internet or some other public or private network. Client computing devices (305) may be connected to the network (330) through a network interface, such as by wired or wireless communication. Although the connections between the server (310) and the servers (320) are shown as separate connections, these connections may be any kind of local, wide area, wired, or wireless network, including the network (330) or a separate public or private network.

[0107] FIG. 4 is a block diagram illustrating components (400) that may be used in a system using the disclosed technology in some implementations. The components (400) may be included in one device of the computing system (100) or distributed across multiple devices of the computing system (100). The components (400) include hardware (410), an arbitrator (420), and specialized components (430). As discussed above, a system implementing the disclosed technology may use various hardware including processing units (412), a working memory (414), input and output devices (416) (e.g., cameras, displays, IMU units, network connections, etc.), and a storage memory (418). In various implementations, the storage memory (418) may be one or more of: local devices, interfaces to remote storage devices, or a combination thereof. For example, the storage memory (418) may be one or more hard drives or flash drives accessible via a system bus, or a cloud storage provider (such as in the storage device (315 or 325)) or other network storage device accessible via one or more communication networks. In various implementations, the components (400) may be implemented on a client computing device such as client computing devices (305) or on a server computing device such as a server computing device (310 or 320).

[0108] The mediator (420) may include components that mediate resources between the hardware (410) and specialized components (430). For example, the mediator (420) may include operating systems, services, drivers, a basic input / output system (BIOS), controller circuits, or other hardware or software systems.

[0109] The specialized components (430) may include software or hardware configured to perform actions to customize the virtual space based on the user's posture. For example, the specialized components (430) may include a posture analysis module (434), standing mode functions (436), sitting-floor height functions (438), sitting-flag configuration functions (440), sitting-edge display functions (442), sitting-sitting experience functions (444), sitting-workspace area functions (446), sitting-automatic virtual area functions (448), and components and APIs that can be used to provide user interfaces, transmit data, and control specialized components such as the interfaces (432). In some implementations, the components (400) may be in a computing system distributed across multiple computing devices or may be an interface to a server-based application running one or more of the specialized components (430).

[0110] The posture analysis module (434) receives sensor inputs (e.g., images from a camera, position sensor data, controller sensor inputs, etc.) and / or determined body mechanics models (e.g., user's kinematic skeletal model, hand positions, etc.) and can use these to determine the user's posture. In various implementations, the posture can specify whether the user is standing, sitting, lying down, etc. In some implementations, standing postures can be divided into standing, moving or standing, stationary, or other motion-based postures. Additional details regarding the determination of the user's posture are described below with respect to block (502) of FIG. 5.

[0111] Some implementations may include standing mode functions (436). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a standing posture. Executing the standing mode functions (436) may include receiving user-specific boundaries for standing, moving postures or automatically sized (based on determined upper limb width) cylindrical or wine glass-shaped boundaries for standing, fixed postures. These boundaries may be shown to the user if the system predicts that the user is likely to collide with the boundaries. Additional details regarding the standing mode functions are provided below with respect to blocks (506 and 510) of FIG. 5.

[0112] Some implementations may include sitting-floor height functions (438). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sitting posture. Executing the sitting-floor height functions (438) may include receiving floor height metrics, such as a user selection of a floor height, a floor height based on a determined user height, or the average of floor heights selected by other users. The sitting-floor height functions (438) may use these metrics to set a virtual floor height. Additional details regarding setting a virtual floor height when a user is in a sitting posture are provided below with respect to FIGS. 6a and FIGS. 7a.

[0113] Some implementations may include sit-flag configuration functions (440). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sit posture. Executing the sit-flag configuration functions (440) may include setting a flag in response to the determination that the user is in a sit posture. This flag may then be surfaced to applications, allowing them to adjust the positioning of objects and other metrics based on whether the flag is set. Additional details regarding setting the sit flag and surface it to allow applications to adjust dynamics are provided below with respect to FIGS. 6b and FIGS. 7b.

[0114] Some implementations may include sitting-edge display functions (442). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sitting posture. Executing the sitting-edge display functions (442) may include determining virtual space boundaries such as patterns, colors, or types, and the types may be virtual walls displaying objects in pass-through mode, warning messages, or other alerts. When the virtual space configuration system detects a boundary display event, such as a prediction that a user will cross a boundary or a real-world object, or that a real-world object has entered the virtual space, executing the sitting-edge display functions (442) may further include displaying the boundaries or representations of real-world objects according to the determined virtual space boundary mode. Additional details regarding the selection of virtual space boundary modes and corresponding display events are provided below with respect to FIGS. 6c and FIGS. 7c.

[0115] Some implementations may include sit-sitting experience functions (444). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sitting posture. Executing the sit-sitting experience functions (444) may include detecting additional changes in posture while the user is participating in the sit-only virtual experience. For example, using a mapping of postures versus response actions provided by the sit-only virtual experience, response actions for changes in posture may be determined and performed. For example, if the user stands, the application may pause, provide a notification, log that the user is standing, change input forms, and / or change virtual objects. Additional details regarding triggering response actions for changes in posture are provided below with respect to FIGS. 6d and FIGS. 7d.

[0116] Some implementations may include sitting-workspace area functions (446). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sitting posture. Executing the sitting-workspace area functions (446) may include detecting an additional forward bending posture while the user is still in a sitting posture. In response, the execution of the sitting-workspace area functions (446) may enable a pass-through display mode for the determined workspace area, allowing the user to view representations of real-world objects in the workspace area without needing to remove the headset or other hardware of the artificial reality system. The workspace area may be determined using computer vision and object detection to identify an area such as the top of a desk or an area including various tools such as a keyboard, mouse, and / or monitors, based on one or more of the area predefined by the user, the user's determined upper limb width, and the average of workspaces set by other users. In some implementations, a trained machine learning model may determine a workspace area based on the current context (e.g., user details and / or camera input), wherein the model is trained based on similar inputs that match user-selected workspace areas or automatically identified workspace areas determined based on object identification with high confidence values. Additional details regarding detecting forward-bent, sitting postures and displaying workspace areas in pass-through mode are provided below with respect to FIGS. 6e and FIGS. 7e.

[0117] Some implementations may include sitting-automatic virtual area functions (448). In these implementations, the virtual space configuration system may execute these functions in response to the posture analysis module (434) identifying a sitting posture. Executing the sitting-automatic virtual area functions (448) may include automatically determining the dimensions of the virtual area for the sitting position based on one or more of user-defined areas, user upper limb widths, average areas set by other users, etc. In some implementations, a trained machine learning model may determine the virtual space dimensions based on the current context (e.g., user details and / or camera input), wherein the model was trained based on similar inputs matched to user-selected virtual spaces. The shape of the virtual area may be automatically determined based on one or more of: settings in the current application, user selection, determined current use of the virtual space having mappings of uses for virtual space shapes, etc. Additional details regarding the automatic determination of the aspects of the virtual area are provided below with respect to FIGS. 6f and FIGS. 7f.

[0118] Those skilled in the art will understand that the components illustrated in FIGS. 1 through 4 described above, and each of the flowcharts discussed below, may be modified in various ways. For example, the order of logic may be rearranged, substeps may be performed simultaneously, illustrated logic may be omitted, and other logic may be included. In some implementations, one or more of the components described above may execute one or more of the processes described below.

[0119] FIG. 5 is a flowchart illustrating a process (500) used in some implementations of the present technology for setting up virtual space configurations based on user posture. In various implementations, the process (500) may be performed by the artificial reality system (e.g., by a virtual space configuration sub-system) when the artificial reality system is first turned on, when the artificial reality system detects user changes periodically (e.g., every 1-2 seconds), or in response to detected changes in posture (e.g., when posture detection of block (502) is performed periodically, or in response to input signals such as changes in height or other movements of the headset or controllers).

[0120] In block (502), the process (500) can determine the user posture. "Posture" is the position or configuration of one or more parts of the user's body. For example, the posture may be sitting, standing, lying down, extending arms, a specific hand position or gesture, head orientation, torso rotation or angle, etc. In some implementations, the posture may also include movements such as specific motions of one or more body parts and / or motions toward a point or object. For example, the first identified posture may be standing still, but the second identified posture may be standing and create a threshold level of lateral movements relative to a center point. In various implementations, the process (500) may automatically determine the user posture based on other inputs, such as the determined height of the headset of the artificial reality system, specific detected movements (e.g., of the headset, controller, hands, legs, or other body parts), images captured by the artificial reality system, position data, IMU data, etc. In some embodiments, various measurements and determinations from an artificial reality system may be fed to a machine learning model trained to classify the user's current posture. Determining the user posture (or "pose") is discussed in more detail in U.S. Patent Application No. 16 / 663,141, filed October 9, 2019, titled "Systems and methods for generating dynamic obstacle collision warnings based on detecting user poses," which is incorporated herein by reference in its entirety. In some embodiments, the user posture may be specified by user input, or the user input may automatically verify the detected posture.

[0121] In block (504), the process (500) can determine whether the posture determined in block (502) corresponds to a standing-moving posture. A standing-moving posture may indicate that the user is standing and is in a situation where she can move laterally (as opposed to standing in the same spot). This may be indicated by one or more of the following: the artificial reality system determining that the user is standing (e.g., determined headset height, user posture selection input, etc.), and the user indicating an edge area for the virtual space, the user specifying an intention to move, the current application being designed to move while standing, or the determination that the user has moved laterally by at least a threshold amount from a center point (i.e., determining that they have moved from their standing place). When the posture corresponds to a standing-moving mode, the process (500) may continue to block (506) to set up the standing-moving virtual space customizations. For example, the virtual space may be a user-defined space and / or a space defined to avoid the user colliding with objects detected in the real-world space around the user. Customizations may also include setting display features to show boundaries if the artificial reality system determines that the user is at risk of colliding with it, such as by using a very distinct red grid pattern to immediately attract the user's attention. If the posture is not standing or moving, the process (500) can continue to block (508).

[0122] In block (508), the process (500) can determine whether the posture determined in block (502) corresponds to a standing still posture. A standing still posture may indicate that the user is standing and unlikely to move laterally (i.e., likely to stand within a few feet of the same spot). Similar to a standing moving posture, a standing still posture may be indicated by the artificial reality system determining that the user is standing (e.g., based on determined headset height, user posture selection input, etc.). However, in this case, the user may specify an intention not to move, and the current application may be designed not to move while standing, or the artificial reality system may determine that the user has not moved at least a critical amount from the center point (i.e., determined that they have not moved laterally by a significant amount, such as 1-2 feet from their current standing position). When the posture reaches standing still, the process (500) may continue to block (510) where it sets up standing still virtual space customizations. For example, the virtual space may be a user-defined space, or a cylinder or wine glass shape defined around the user. Customizations may also include setting up a display to show boundaries if the artificial reality system determines that the user is at risk of colliding with it, such as using a distinct red grid pattern to immediately attract the user's attention, or using a less distracting pattern of gray crosses (e.g., +) shapes, since such collisions are less likely to cause damage. If the posture is not standing still, the process (500) may continue to block (512).

[0123] In block (512), the processor (500) can determine whether the posture determined in block (502) corresponds to a sitting posture. A sitting posture may be indicated by the artificial reality system determining that the system's headset is at a threshold distance from the average sitting headset height, may come from user input specifying the posture, may be determined based on a machine learning model that takes sensor input and classifies the current posture, may be assumed based on a current application designed while sitting, or from the direction provided to the user, or using other metrics. If the determined posture is sitting, the process (500) may continue to block (514), where any of the various sitting virtual space customizations may be applied. In various implementations, available sitting virtual space customizations may include allowing adjustment of floor height (see additional details regarding FIG. 6a below), setting flags that may be surfaced for applications to adjust the dynamics of applications (see additional details regarding FIG. 6b below), customizing the display of virtual space boundaries to be less obtrusive when in sitting mode (see additional details regarding FIG. 6c below), providing options to detect when a user leaves sitting mode and trigger corresponding actions (see additional details regarding FIG. 6d below), allowing the user to interact naturally with certain real-world objects without needing to remove the artificial reality headset (see additional details regarding FIG. 6e below), and automatically determining virtual space dimensions for seated users (see additional details regarding FIG. 6f below). If the posture is not sitting, the process (500) may return to block (502) to continue monitoring the user posture for the recognized variation.

[0124] FIG. 6a is a flowchart illustrating a process (600) used in some implementations to enable floor height customization when a user sits. In some implementations, the process (600) may be performed as a sub-process of a block (514) of the process (500). In some cases, the process (600) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0125] In block (602), the process (600) may obtain metrics regarding the seated floor height. The metrics may be determined based on circumstances such as the user's standing height, the user's upper limb width, the dimensions of the chair, couch, or other object on which the user is sitting (e.g., determined using computer vision technologies and a camera integrated with an artificial reality system), averages of floor heights set by other users (or users determined to be similar to the current user), or other statistics. In some implementations, one or more of these features may be used as input to a machine learning model trained to predict a desired floor height based on previous user choices. In other implementations, these features may be used in mapping features to floor heights. In some implementations, the metric may be a user-selected value, for example, by indicating a floor height with a hand gesture, input to a controller, voice command, gaze, etc. In some implementations, the user may be recognized and the obtained metric may be based on previous choices for said user.

[0126] In block (604), the process (600) can set the floor height based on the metric obtained in block (602). This improves user accessibility when in virtual space by setting a minimum height for virtual objects with respect to the user, thereby eliminating instances where the user must move to the corners of a chair or couch and reach the floor. In some implementations, the floor height may be set for the artificial reality system across applications, or for a specific application where different floor heights are set for other applications. An example of setting the floor height is discussed below with respect to FIG. 7a.

[0127] FIG. 6b is a flowchart illustrating a process (610) used in some implementations to set a flag to allow applications to adjust the dynamics for sitting configurations. In some implementations, the process (610) may be performed as a sub-process of a block (514) of the process (500). In some cases, the process (610) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0128] In block (612), in response to a determination that a user is in a sitting position (e.g., determined in blocks (502 and 512)), the process (610) may set a flag indicating the sitting position. The flag may be any type of variable, such as a binary value, a posture identifier, a posture name, etc. Setting the flag may involve various types of recording into memory, such as setting a program or operating system variable, writing to a data field, writing to a file, etc. For example, an operating system for an artificial reality system may maintain a set of motion condition variables, one of which may be a posture identifier or an "isSeated" flag. In block (614), the process (610) may surface the flag set in block (612) to an application. For example, the application may send a request for the value of the flag to the operating system and perform a read from a database where the flag is set. Reading the flag may allow the application to modify specific dynamics to better accommodate seated users. For example, an application can change the positioning of virtual objects to move them within arm's reach, whereas if the user is standing, these adjustments may not be necessary, as the user can take measures to reach more distant objects and reach objects on the floor more easily. An example of using flags to allow applications to adjust the dynamics for sitting configurations is discussed below with respect to FIG. 7b.

[0129] FIG. 6c is a flowchart illustrating a process (620) used in some implementations to customize virtual space boundary displays in response to user posture. In some implementations, the process (620) may be performed as a sub-process of block (514) of process (500). In some cases, the process (620) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0130] In block (622), the process (620) may receive a selection of virtual space boundary display modes. In some implementations, this selection may come from a mapping of postures to boundary display modes. For example, when the system detects that the user is standing (a situation where the user is more likely to move around and make faster movements than when sitting), there is a greater likelihood that a user collision with the boundary will cause harm. Therefore, standing postures may be mapped to a boundary display mode that is more likely to draw the user's attention, such as a boundary appearing as virtual walls with a specific color (e.g., bright red, green, orange, etc.) and / or a specific pattern (e.g., grid, densely packed dots, flashing lines, etc.). However, when the system detects that the user is sitting or lying down (a situation where the user is less likely to move around and make slower movements than when standing), there is less likelihood that a user collision with the boundary will cause harm. Accordingly, sitting or lying postures can be mapped to boundary display modes that are less likely to interfere with the user's artificial reality experience, such as boundaries appearing in less bright colors (e.g., gray, tan, brown, etc.) and / or specific patterns (e.g., a grid where only +s are shown at line intersections, scattered dots, dim lines, etc.). In some implementations, for sitting or lying postures, the selected boundary display mode may show real-world objects as a passthrough to the virtual environment. In this passthrough mode, if the artificial reality system determines that the user is approaching a boundary or is about to collide with an object, the artificial reality system may show real-world objects in the virtual space, allowing the user to identify and avoid them.In some implementations, real-world objects depicted in this pass-through mode may be depicted as not bright or merely as shadows to avoid the user going too far out of her virtual space.

[0131] In block (624), the process (620) can detect a boundary display event. This may occur by detecting whether the user is within a threshold distance of the boundary, whether the user's determined trajectory is expected to intersect the boundary, and / or whether the user's body position and configuration (e.g., arm or leg width, stride length, height, motion profile, etc.) makes it likely that the user will intersect the boundary. In some implementations, a machine learning model may be trained to make this decision. For example, inputs to the machine learning model may include inertia, position, camera, and / or other sensor inputs from the headset and / or controllers of the artificial reality system, a model of the user's body (e.g., bone structure), past movement data of the user or average users, edge configuration details, etc. The machine learning model may be trained to generate a prediction of whether the user is likely to intersect the boundary (e.g., it may be trained based on previous identifications of the context when the user intersects the boundary). In some implementations, there may be other triggers for displaying the boundary, such as another person or object entering the virtual space, which may cause a change at the boundary or cause passthrough mode or current application signaling for the boundary to be displayed.

[0132] In block (626), the processor (620) may display boundaries or objects in the surrounding area using a selected virtual space boundary display mode selected in block (622) in response to a detected boundary display event of block (624) (i.e., activate a passthrough mode). For example, the processor (620) may display a virtual wall or part of a virtual wall having a specific pattern and / or color, may display objects that are within a threshold distance of the user or that the artificial reality system determines are likely to collide with the user, and may display virtual walls or objects that fade in response to the distance to the user. An example of customizing virtual space boundary displays to display boundaries in passthrough when the user is in a sitting position is discussed below with respect to FIG. 7c.

[0133] FIG. 6d is a flowchart illustrating a process (630) used in some implementations to enable posture-specific virtual experiences. In some implementations, the process (630) may be performed as a sub-process of a block (514) of the process (500). In some cases, the process (630) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0134] In block (632), the process (630) can initialize a virtual experience designed to be performed in the current posture. For example, the application can be configured to be used in a sitting-only posture, a standing-only posture, a lying-only posture, etc. As more specific examples, the company can specify a training program in which the user must sit throughout the training program, otherwise the program must pause; the game can be configured to have a first input form (e.g., a static virtual control board) when the user is sitting and a different input form (e.g., based on monitored user movements) when the user is standing; and the application developer may want to synchronize the use of her application with the most common user posture and thus collect statistics on total user postures.

[0135] In block (634), the process (630) can detect whether there has been a change in the user's posture. This may occur in a manner similar to block (602). In some implementations, only specific changes in posture will trigger the Yes (Y) branch from block (634). For example, while the process (630) can detect various changes in posture such as arm positions or torso tilt, the current application may specify that only changes from sitting to standing posture or changes in lateral movement exceeding a threshold distance should trigger the Yes branch. In various implementations, the operating system for the artificial reality system or the application driven by the artificial reality system may specify specific mappings between posture changes and response actions, which may be these mapped posture changes that trigger the Yes branch from block (634). If no such posture changes are detected (No (N) branch), the process (630) may continue to block (638). When such a change in posture is detected (e.g., branch), the process (630) can continue to block (636).

[0136] In block (636), in response to a change in posture detected in block (634), the process (630) may trigger a response action. In some implementations, there may be response actions set by the artificial reality system, such as automatically stopping or pausing the virtual experience, providing a notification to resume a previous posture or that the current posture is not recommended for this experience, logging the times that various postures are maintained, or switching to another mode (e.g., passthrough mode). In some implementations, the current application may specify one or more mappings of trigger actions to be performed for specific changes in posture. For example, the application may specify that if the user is standing from a sitting posture, a notification should be displayed to return to the sitting position and the standing time should be logged, and if the standing time exceeds a threshold, the virtual experience should be paused. As another example, a game application can specify that when the user is in a sitting position, the virtual experience should drive a virtual car, but if the user is standing, the virtual experience should change to show the user exiting the virtual car and transitioning to a virtual walking mode.

[0137] In block (638), the process (630) can determine whether the virtual experience initiated in block (632) is still in progress. While the virtual experience continues, the process (630) can monitor postures and trigger response actions. Once the virtual experience ends, the process (630) may be terminated. An example enabling posture-specific virtual experiences is discussed below with respect to FIG. 7d.

[0138] FIG. 6e is a flowchart illustrating a process (640) used in some implementations to enable a virtual sitting workspace area. In some implementations, the process (640) may be performed as a sub-process of a block (514) of the process (500). In some cases, the process (640) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0139] In block (642), the process (640) can detect a forward-bending posture while the user maintains a sitting posture. This may be a forward bend from the vertical by a threshold number of degrees, for example, 10, 15, or 20 degrees. In some implementations, other postures may be detected to trigger a workspace area, such as an arm raised for a virtual keyboard and a hand in a posture or a forearm resting on a desk.

[0140] In block (644), the process (640) may determine a workspace area. For example, this may be an area pre-established by the user; an area of ​​a set size and shape (e.g., a 3-foot × 4-foot rectangle); an area defined based on the user's characteristics (e.g., a semicircle or rectangle with a size based on the user's determined upper limb width, such as an area within the user's full reach); an area corresponding to the upper limb widths of average users having characteristics such as keys similar to the current user, or an average of areas manually set by other users; or an area based on computer vision / object recognition (e.g., an area corresponding to the top of the desk in front of the user or an area surrounding tools for the user, such as keyboards, laptops, or monitors). In various implementations, this area may be established in response to the posture detected in block (642) or may be established previously.

[0141] In block (646), the process (640) can activate a pass-through display mode for a determined workspace area. As discussed above, the pass-through mode can show parts of the real world while in an artificial reality environment. For example, images taken by external cameras can be fed into the artificial reality environment, and a part of the display can be disabled to allow light passing through the display to be seen by the user, and a part of the display can be moved to allow light to enter the user's eyes. By activating the pass-through mode for a determined workspace area when a specific posture is detected, the user can easily interact with tools and other objects in the workspace area without having to remove the headset of the artificial reality system or manually activate the pass-through mode. An example of enabling pass-through for a workspace area when the user sits and leans forward is discussed below with respect to FIG. 7e.

[0142] FIG. 6f is a flowchart illustrating a process (650) used in some implementations to automatically customize a virtual area in sitting mode. In some implementations, the process (650) may be performed as a sub-process of a block (514) of process (500). In some cases, the process (650) may be performed in response to other triggers, such as an application changing its posture mode, user selection, changes in users, or the start of a specific application.

[0143] In block (652), the process (650) can automatically determine an area for a virtual space for a user in a seated position. The process (650) may use pre-established shapes for the areas, such as a cylinder centered on the user, a half-cylinder in front of the user, a rectangular prism, or others. The dimensions of the area may be set based, for example, on settings specified by user input, determined user upper limb width, statistics on the general upper limb width of users or users identified as similar to the current user, identification of real-world objects identified by the artificial reality system to exclude from the virtual space, or identification of virtual objects identified by the artificial reality system to include in the virtual space. For example, the process (65) may determine the virtual area as a half-cylinder in front of the user having a radius equal to half the user's upper limb width (i.e., the length of one arm). As another example, the process (650) may determine the virtual area as a cube in front of the user excluding all real-world objects from the area. In some implementations, a determined virtual area may be proposed to the user, and the user may manually adjust its parameters or define different virtual areas, for example, having different sizes, shapes, or activation features.

[0144] In some implementations, the user's upper limb width can be determined by setting the upper limb width to be equal to the user's height. In some cases, the setting can be updated based on identifying the positions of the user's hands or controllers and extending the determined upper limb width if the distance between the identified hands or controllers extends beyond the determined upper limb width. In other cases, the upper limb width can be reduced if the identified hands or controllers cannot extend to the determined upper limb width within a threshold amount of time or the usage of the artificial reality system. In some implementations, instead of starting with an initial height determination, the upper limb width can be determined directly by observing these hand or controller distances. In some implementations, the determination of the upper limb width can be determined and adjusted by using a body motion model (e.g., a kinematic skeleton model) to identify the user's arms in images of the user and estimate the maximum reach.

[0145] In block (654), the process (650) can set the virtual area determined in block (652). In various implementations, this virtual area may be set globally for all instances where the current user is identified, for use when running a specific application, or for use during a specific task. An example of automatically setting virtual space dimensions is discussed below with respect to FIG. 7f.

[0146] FIG. 7a is a conceptual diagram illustrating an example (700) used in some embodiments to enable floor height customization when a user is sitting. In the example (700), the virtual space configuration system determines that the user (702) is in a sitting position. In response, the virtual space configuration system enables options for the user (702) to adjust the height of the virtual floor (704). As illustrated by the arrows (706), the user (702) can activate controls (e.g., virtual controls displayed in the artificial reality environment, controls on the controller, using her gaze, etc.) to adjust the height of the virtual floor (704). The height of the virtual floor (704) allows applications to place objects so that they are not lower than the virtual floor (704). As illustrated in example (700), by raising the virtual floor (704), the user (702) can reach any objects located on the virtual floor (704) much more easily while sitting.

[0147] FIG. 7b is a conceptual diagram illustrating an example (710) used in some implementations to use a flag to allow applications to adjust dynamics for seated use configurations. In the example (710), the user (02) is initially standing and is surrounded by objects (712A through 712E). When the virtual space configuration system detects a change in posture in which the user (702) is now sitting, the virtual space configuration system sets a sitting flag, which is surfaced to the application when controlling the objects (712). The application can adjust the position of the objects (712) so as to be within range of the seated user (702) (as illustrated by arrows (714A through 714E)).

[0148] FIG. 7c is a conceptual diagram illustrating an example (720) used in some implementations to customize virtual space boundary displays in response to user posture. The example (720) first illustrates a user (702) in a standing posture, wherein the virtual space (722) has a boundary wall having portions of a boundary wall that appear when the user (702) is within a threshold distance of the said portion. The boundary wall is configured with a virtual space boundary display mode selection that causes a red grid pattern, such as lines (724), to be displayed on the wall. In the second portion of the example (720), the user (702) adopts a sitting posture. At this time, the virtual space boundary display mode changes for the virtual space so that real-world objects within the user's (702) threshold distance are displayed in pass-through mode. In the example (720), these objects are cut off at lines (726A to 726D) to show where they are no longer within the user's (702) threshold distance.

[0149] FIG. 7d is a conceptual diagram illustrating an example (730) used in some implementations to enable sitting-only virtual experiences. In the example (730), the user (702) is initially in a sitting position while the virtual space configuration system runs the sitting-only virtual experience. When the user (702) transitions to a standing position, the virtual space configuration system pauses the sitting-only virtual experience and displays a message (732) informing the user (702) that the sitting-only virtual experience resumes when the user (702) returns to the sitting position.

[0150] FIG. 7e is a conceptual diagram illustrating an example (740) used in some implementations to enable a sitting workspace virtual area. In the example (740), the user (702) starts in a sitting, upright position. As indicated by the arrow (742), the user (702) then bends forward while remaining in a sitting position. In response to detecting this sitting, bending forward position, the virtual space configuration system displays a workspace area (744), which is an area identified by the virtual space configuration system as corresponding to the surface of a desk in front of the user, including a keyboard and a monitor.

[0151] FIG. 7f is a conceptual diagram illustrating an example (750) used in some implementations to automatically customize a virtual area in a sitting mode. The example (750) shows a first instance in which a virtual space configuration system automatically determines the size (752) of a virtual space configured to be a semi-cylindrical shape in front of the user (702) based on the determined upper limb-width of the user (702). The example (750) also illustrates a second instance in which the virtual space is a rectangular shape in front of the user (702) with length and width dimensions (754) set based on the statistical averages of areas selected by other users with user heights within the threshold of the user's height (702).

[0152] In this specification, "implementations" (e.g., "some implementations," "various implementations," "this implementation," "implementation," etc.) mean that a specific feature, structure, or characteristic described in relation to an implementation is included in at least one implementation of the present invention. The appearance of these phrases in various places in the specification does not necessarily mean that they all represent the same implementation, nor are they separate or alternative implementations mutually exclusive from other implementations. Furthermore, various features that may be exhibited by some implementations rather than by others are described. Similarly, various requirements that may be requirements for some implementations rather than for other implementations are described.

[0153] As used herein, exceeding a threshold means that the value of the item under comparison exceeds a specified other value, the item under comparison is among a specified number of items having a maximum value, or the item under comparison has a value within a specified top percentage value. As used herein, below a threshold means that the value of the item under comparison is less than a specified other value, the item under comparison is among a specified number of items having a minimum value, or the item under comparison has a value within a specified bottom percentage value. As used herein, within a threshold means that the value of the item under comparison is between two specified other values, the item under comparison is among a middle-specified number of items, or the item under comparison has a value within a middle-specified percentage range. Relative terms such as high or insignificant, unless otherwise defined, may be understood as assigning a value and determining how said value compares to an established threshold. For example, the phrase "selecting a fast connection" can be understood to mean selecting a connection with an assigned value corresponding to a connection speed exceeding a threshold.

[0154] As used herein, the word "or" indicates any possible permutation of a set of items. For example, the phrase "A, B, or C" indicates any combination of at least one of A, B, or C, or any of A; B; C; A and B; A and C; B and C; A, B, and C; or any number of any items such as A and A; B, B, and C; or A, A, B, C, and C.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological practices, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or practices described above. While specific embodiments and implementations have been described herein for illustrative purposes, various modifications may be made without departing from the scope of the embodiments and implementations. The specific features and practices described above are disclosed as exemplary forms embodying the following claims. Accordingly, embodiments and implementations are not limited except by the appended claims.

[0156] Any known patents, patent applications, and other references mentioned above are incorporated herein by reference. Aspects may be modified if necessary to utilize the systems, functions, and concepts of the various references described above to provide additional implementations. If any descriptions or subject matter in the documents incorporated by reference conflict with any descriptions or subject matter of the present application, the present application shall be controlled.

Claims

Claim 1 A method for customizing a virtual space based on a user posture, comprising: determining that the user posture corresponds to a sitting mode; and in response to the determination, a step of setting a seated customization for the virtual space, wherein the first seated customization comprises: obtaining metrics for floor height; and adjusting a system floor height based on the metrics for floor height, wherein the system floor height sets a minimum height at which virtual objects can be positioned, and the adjustment of the system floor height is such that the system floor height is higher than the height of the real-world floor; or providing a second seated customization comprises: setting a seated flag; and surface a seated flag for a first application, wherein the first application performs a first motion dynamics adjustment based on the seated flag. A method for customizing a virtual space based on a user posture, comprising: providing a second sitting customization, wherein the second application performs a second motion dynamics adjustment different from the first motion dynamics adjustment based on the sitting flag, and the second application performs a second motion dynamics adjustment different from the first motion dynamics adjustment based on the sitting flag; or providing a third sitting customization, wherein the method comprises the step of setting a sitting customization for the virtual space by providing the third sitting customization, wherein the method comprises receiving a boundary mode selection for the virtual space; detecting a boundary display event; and displaying the boundary based on the selected boundary mode, wherein the boundary is a virtual barrier specifying a physical area where the user is allowed to move, and is configured based on the selected boundary mode. Claim 2 A method for customizing a virtual space based on a user posture, wherein the step of setting the sitting customization in claim 1 includes providing the first sitting customization. Claim 3 A method for customizing a virtual space based on a user posture, wherein the metrics for the floor height in claim 1 or 2 are based on: a determined standing height of the user; a determined arm-span of the user; determined dimensions of an object on which the user is sitting; or any combination thereof. Claim 4 A method for customizing a virtual space based on a user posture, wherein the metrics for the floor height in claim 1 or 2 are determined by a machine learning model trained based on previous user selections to receive indications of sensor or camera measurements and to generate floor height metrics. Claim 5 A method for customizing a virtual space based on a user posture, wherein the metrics for the floor height are one or more user-selected values ​​indicated by one or more of: user hand gestures, user input to a controller, user voice commands, user gaze direction, or any combination thereof. Claim 6 A method for customizing a virtual space based on a user posture, wherein, in claim 1 or 2, the step of setting the sitting customization includes providing the second sitting customization. Claim 7 A method for customizing a virtual space based on a user posture, wherein, in claim 1 or 2, the step of setting the sitting customization includes providing the third sitting customization. Claim 8 A method for customizing a virtual space based on a user posture, wherein the boundary mode selection in claim 1 or 2 is based on a mapping of postures versus boundary modes provided by the current application. Claim 9 A method for customizing a virtual space based on a user posture, wherein, in claim 1 or 2, the selected boundary mode is a passthrough mode that causes the display to become a display of real-world objects in the virtual space. Claim 10 A method for customizing a virtual space based on a user posture, wherein the detection of a boundary display event in claim 1 or 2 comprises receiving one or more of inertial data, position data, camera image data, a model of the user's bone structure, or any combination thereof, and applying a machine learning model trained to generate an estimate of whether the user will cross the boundary. Claim 11 A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform actions for customizing a virtual space based on a user posture, wherein the actions include: determining that the user posture corresponds to a sitting mode; and, in response to the determination that the user posture corresponds to the sitting mode, providing the first sitting customization for the virtual space, wherein the first sitting customization includes: obtaining metrics for floor height; and adjusting the system floor height based on the metrics for floor height, wherein the system floor height sets a minimum height at which virtual objects can be positioned, and the adjustment of the system floor height is such that the system floor height is higher than the height of the real-world floor; or providing the second sitting customization, wherein the second sitting customization includes: setting a sitting flag. Providing a second sitting customization comprising: surfacing a sitting flag for a first application, wherein the first application performs a first motion dynamics adjustment based on the sitting flag; and surfacing a sitting flag for a second application, wherein the second application performs a second motion dynamics adjustment different from the first motion dynamics adjustment based on the sitting flag; or providing a third sitting customization comprising: receiving a boundary mode selection for the virtual space; and detecting a boundary display event.A computer-readable storage medium comprising setting a seating customization for the virtual space by providing the third seating customization, which includes displaying the boundary based on the selected boundary mode, the boundary being a virtual barrier specifying a physical area where a user is allowed to move, and the boundary configured based on the selected boundary mode. Claim 12 In claim 11, the setting of the sitting customization comprises: providing a fourth sitting customization, wherein while a virtual experience configured for a sitting mode is in progress: determining that a second user posture is no longer in the sitting mode; and in response thereto, triggering a response action for the virtual space; the response action comprises: automatically stopping or pausing the virtual experience; providing a notice to resume a previous posture or that the current posture is not recommended for the virtual experience; logging the times during which various postures are maintained; switching to a display mode showing real-world objects; changing input forms; or one or more of any combination thereof; and / or the response action is a computer-readable storage medium selected based on a mapping that maps trigger actions to specific posture changes provided by the current application. Claim 13 A computer-readable storage medium according to claim 11 or 12, wherein setting the sitting customization comprises: providing a fifth sitting customization, wherein: determining that a second user posture corresponds to a forward bend; determining a workspace area; and activating a display mode that displays real-world objects in the workspace area in a virtual environment; and wherein the workspace area is determined based on: an area defined based on the user's determined upper limb width; an average of workspace areas previously manually set by other users; an area corresponding to the top of a flat real-world object in front of the user; an area determined to surround one or more specified real-world tools; or any combination thereof. Claim 14 A computer-readable storage medium according to claim 11 or 12, wherein setting the sitting customization comprises: providing the sixth sitting customization, wherein the sixth sitting customization comprises: automatically determining the dimensions of the virtual space based on the physical characteristics of the user; and adjusting the virtual space based on the determined dimensions; the physical characteristics of the user include an automatically determined user upper limb width; and the user upper limb width is automatically determined by: setting the initial upper limb width to be equal to the determined user height; and updating the initial upper limb width based on identifying that the places of the user's hands or controllers extend beyond the determined upper limb width. Claim 15 A computing system for customizing a virtual space based on a user posture comprises: one or more processors; and one or more memories that store instructions for the computing system to perform operations when executed by the one or more processors, wherein the operations include: determining that the user posture corresponds to a sitting mode; and in response to the determination that the user posture corresponds to the sitting mode, providing the first sitting customization for the virtual space, wherein the first sitting customization includes: obtaining metrics for floor height; and adjusting the system floor height based on the metrics for floor height, wherein the system floor height sets a minimum height at which virtual objects can be positioned, and the adjustment of the system floor height includes adjusting the system floor height such that the system floor height becomes higher than the height of the real-world floor; or providing the second sitting customization, wherein the second sitting customization includes: setting a sitting flag. and providing a second sitting customization comprising: surfacing a sitting flag for a first application, wherein the first application performs a first motion dynamics adjustment based on the sitting flag; surfacing a sitting flag for a second application, wherein the second application performs a second motion dynamics adjustment different from the first motion dynamics adjustment based on the sitting flag; or providing a third sitting customization comprising: receiving a boundary mode selection for the virtual space; detecting a boundary display event;A computing system comprising setting a seating customization for the virtual space by one or more of providing the third seating customization, wherein the boundary is a virtual barrier specifying a physical area where a user is allowed to move, and is configured based on a selected boundary mode.

Images