Gesture-based virtual space configuration

Abstract

The embodiment of the invention relates to virtual space configuration based on gestures. A virtual space configuration system of an artificial reality system can detect user gestures and provide various corresponding customizations of the virtual space of the system. The virtual space configuration system may provide seating virtual space customizations when the user is in a seating position. In various implementations, these customizations may include: allowing adjustment of floor height; a mechanism to set a flag that can be revealed to the application to adjust the application for the seated user; customizing the display of virtual space boundaries while in the seating mode to reduce interference; providing an option to detect when the user leaves the seating mode and trigger a corresponding action; providing a delivery workspace region allowing a user to naturally interact with certain real-world objects without removing the virtual reality headset; or automatically determining a virtual space size for the seated user.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 058648, international application date November 3, 2020, entered the Chinese national phase on May 23, 2022, Chinese national application number 202080081235.1, and invention title "Pose-Based Virtual Space Configuration". Technical Field

[0002] This invention relates to the control of the configuration of virtual spaces used in artificial reality environments. Background Technology

[0003] When a user sees and interacts with “virtual objects” (i.e., computer-generated representations of objects appearing in an artificial reality environment), the user’s physical movement occurs in the real world. In some cases, an artificial reality system may prevent a user from seeing part or all of the real world, or the user may be distracted by virtual objects, causing the user to unintentionally collide with real-world objects or leave the area designated for interaction within the artificial reality environment. In other cases, the user’s movement may be limited by the user’s body posture in the real world, making interaction in the artificial reality environment somewhat difficult. For example, some virtual objects may be placed out of reach, making it difficult for the user to interact with them from their current posture. Summary of the Invention

[0004] According to a first aspect of the invention, a method for customizing a virtual space based on a user posture is provided, the method comprising: determining that the user posture corresponds to a seating mode; and, in response to the determination, setting a seating customization for the virtual space by: providing a first seating customization, including: obtaining a measurement for floor height; adjusting a system floor height based on the measurement for floor height; or providing a second seating customization, including: setting a seating indicator; and exposing the seating indicator to one or more applications, wherein the one or more applications adjust operating mechanisms based on the seating indicator; or providing a third seating customization, including: receiving a boundary mode selection for the virtual space; detecting a boundary display event; and, based on the selected boundary mode: displaying a boundary configured based on the selected boundary mode; or displaying real-world objects in the virtual space.

[0005] Setting up seating customization may preferably include providing a first seating customization.

[0006] In some embodiments, the measurement of floor height is based on one or more of the following: the user's determined standing height; the user's determined arm span; the determined dimensions of the object on which the user sits; or any combination thereof.

[0007] In some embodiments, the measurement of floor height is determined by a machine learning model trained based on previous user selections to receive indications from sensor or camera measurements and generate a floor height measurement.

[0008] In some embodiments, the measurement of floor height is one or more user-selected values ​​indicated by one or more of the following: user hand gestures, user input to the controller, user voice commands, user gaze direction, or any combination thereof.

[0009] In some embodiments, setting a seating customization includes providing a second seating customization.

[0010] In some embodiments, setting seating customization includes providing a third seating customization.

[0011] Boundary mode selection can preferably be based on the mapping from pose to boundary mode provided by the current application.

[0012] In some embodiments, the selected boundary mode is a delivery mode that makes the display a virtual space for displaying real-world objects.

[0013] Detecting boundary display events may include applying a machine learning model trained to receive one or more of inertial data, location data, camera image data, a model of the user's skeletal structure, or any combination thereof, and to produce a prediction of whether the user will intersect with the boundary.

[0014] According to a second aspect of the invention, a computer-readable storage medium is provided that stores instructions, when executed by a computing system, causing the computing system to perform operations for customizing a virtual space based on a user posture, the operations including: determining that the user posture corresponds to a seating mode; and, in response to determining that the user posture corresponds to a seating mode, setting the seating customization for the virtual space by: providing a first seating customization, including: while a virtual experience configured for the seating mode is in progress: determining that a second user posture corresponds to no longer being in a seating mode; and, in response, triggering a response action for the virtual experience; or providing a second seating customization, including: determining that the second user posture corresponds to leaning forward; determining a workspace area; and implementing a display mode that displays real-world objects located in the workspace area within the virtual environment; or providing a third seating customization, including: automatically determining the size of the virtual space based on the user's body characteristics; and adjusting the virtual space based on the determined size.

[0015] In some embodiments, setting seating customization includes providing a first seating customization.

[0016] In some embodiments, the response action includes one or more of the following: automatically stopping or pausing the virtual experience; providing a notification to resume the previous pose or to discourage the use of the current pose in the virtual experience; recording the duration for which various poses are held; switching to a display mode that shows real-world objects; changing the input modality; or any combination thereof.

[0017] In some embodiments, setting a seating customization includes providing a second seating customization.

[0018] In some embodiments, the workspace area is determined based on one or more of the following: an area defined based on the user's determined arm span; an average workspace area previously manually set by another user; an area corresponding to the top of a planar real-world object in front of the user; an area determined to contain one or more designated real-world tools; or any combination thereof.

[0019] In some embodiments, setting seating customization includes providing a third seating customization.

[0020] The user's physical characteristics may preferably include automatically determined user arm span.

[0021] The user's arm span can preferably be automatically determined by: setting an initial arm span equal to the determined user height; and updating the initial arm span based on the position of the user's hand or controller that extends beyond the determined arm span.

[0022] According to a third aspect of the invention, a computing system is provided for customizing a virtual space based on a user posture, the computing system comprising: one or more processors; one or more memories storing instructions, which, when executed by the one or more processors, cause the computing system to perform operations including: determining that a user posture corresponds to a seating mode; and, in response to determining that a user posture corresponds to a seating mode, setting a seating customization for the virtual space by one or more of the following: providing a first seating customization, including: obtaining a measurement for floor height; adjusting the system floor height based on the measurement for floor height; or providing a second seating customization, including: setting a seating indicator; and exposing the seating indicator to one or more applications, wherein the one or more applications adjust operating mechanisms based on the seating indicator; or providing a second seating customization, including: setting a seating indicator; and exposing the seating indicator to one or more applications, wherein the one or more applications adjust operating mechanisms based on the seating indicator; or providing a third seating customization. The third seating customization includes: receiving a boundary mode selection for the virtual space; detecting a boundary display event; and displaying a boundary type or object in the virtual space based on the selected boundary mode; or providing a fourth seating customization, including: while a virtual experience configured for a seating mode is in progress, determining that a second user posture corresponds to no longer being in a seating mode; and triggering a response action for the virtual experience; or providing a fifth seating customization, including: determining that a second user posture corresponds to leaning forward; determining a workspace area; and implementing a display mode that displays real-world objects in the workspace area within the virtual environment; or providing a sixth seating customization, including: automatically determining the size of the virtual space based on the user's body characteristics; and adjusting the virtual space based on the determined size. Attached Figure Description

[0023] Figure 1 This is a block diagram illustrating an overview of some implementations of this technology and the devices on which they can operate.

[0024] Figure 2A This is a wiring diagram illustrating a virtual reality headset that can be used in some implementations of this technology.

[0025] Figure 2B This is a wiring diagram of a mixed reality head-mounted device that can be used in some implementations of this technology.

[0026] Figure 3 This is a block diagram illustrating an overview of the environment in which some implementations of this technology can operate.

[0027] Figure 4 It is a block diagram illustrating components that can be used in systems employing the disclosed technology in some implementations.

[0028] Figure 5 This is a flowchart illustrating a process used in some implementations of this technology for setting up a virtual space configuration based on user gestures.

[0029] Figure 6A This is a flowchart illustrating the process used in some implementations to customize the floor height when the user is seated.

[0030] Figure 6B This is a flowchart illustrating the process used in some implementations to set flags to allow applications to adjust the seating configuration mechanism.

[0031] Figure 6C This is a flowchart illustrating the process used in some implementations to customize the display of virtual space boundaries in response to user gestures.

[0032] Figure 6D This is a flowchart illustrating the process used in some implementations to achieve a seat-only virtual experience.

[0033] Figure 6E This is a flowchart illustrating the process used in some implementations to create a virtual area for a seated workspace.

[0034] Figure 6F This is a flowchart illustrating the process used in some implementations to automatically customize virtual areas in seated mode.

[0035] Figure 7A This is a concept diagram illustrating an example of customizing the floor height when a user is seated.

[0036] Figure 7B This is a conceptual diagram illustrating an example of a mechanism that uses flags to allow applications to adjust the configuration used for seating.

[0037] Figure 7C This is a conceptual diagram illustrating an example of customizing the display of virtual space boundaries in response to user gestures.

[0038] Figure 7D This is a concept diagram illustrating an example of implementing a seated virtual experience.

[0039] Figure 7E This is a conceptual diagram illustrating an example of a virtual area for implementing a seated workspace.

[0040] Figure 7F This is a concept diagram illustrating an example of automatically customizing virtual areas in seating mode.

[0041] The technology described herein can be better understood by referring to the following detailed description and the accompanying drawings, wherein similar reference numerals indicate elements that are the same or have similar functions. Detailed Implementation

[0042] This document describes embodiments for customizing virtual spaces based on user posture. An artificial reality system can define a specific “virtual space” for the user experience, defining the range of movement a user can make during the experience, controlling how virtual objects are displayed or placed within the experience, and / or setting system actions in response to changes in posture. For example, if a user approaches the edge of a defined virtual space, the artificial reality system can provide a warning or implement a delivery mode to show the user real-world objects she might collide with. As used herein, a “posture” is the position or configuration of one or more parts of a user’s body. For example, a posture can be sitting, standing, lying down, arms outstretched, a specific hand position or posture, head orientation, torso rotation, etc. In some implementations, a posture can also include movement, such as a specific movement of one or more body parts and / or a specific movement relative to a point or object. For example, an identified first posture could be a stationary standing posture, while an identified second posture could be a standing movement (e.g., the user has made a threshold lateral movement relative to a center point).

[0043] A virtual space configuration system (which can be a subsystem of an artificial reality system) can detect user posture and provide various corresponding customizations for the system's virtual space. In some implementations, the postures that the virtual space configuration system can identify include standing (which can be categorized as standing-moving or stationary standing), sitting, lying down, etc. In various implementations, these determinations can be automatic (based on user input) or automatically determined and confirmed by the user. For example, the virtual space configuration system can determine the height of the artificial reality system's head-mounted device relative to an identified floor, and using known user heights or the average of multiple user heights, can determine whether the head-mounted device height corresponds to a standing or sitting posture. Furthermore, when the position is determined to be standing, the virtual space configuration system can determine whether the lateral position of the head-mounted device has moved beyond a threshold amount from the center point to determine whether the standing user is stationary or moving. The virtual space configuration system can provide the user with an indication of the determined posture for confirmation or modification.

[0044] When in a standing, moving posture, the virtual space configuration system can set up a virtual space that the user has defined for the current real-world environment of the artificial reality system, and / or the virtual space configuration system can automatically detect objects around the user's current location and set up virtual space to avoid collisions with those objects. When the user approaches this boundary, the virtual space configuration system can present a warning or display a grid indicating the boundary. When in a standing, stationary posture, the virtual space configuration system can define a virtual space around the user, for example, as a cylindrical region or a "wine glass" shape (i.e., a cylinder narrow at the bottom and wide at the top), to indicate that the user's legs are stationary, but a space is provided around the user's upper body in which they can move their arms. In some implementations, the diameter of the upper part of this cylinder or wine glass shape can be based on user characteristics, such as a determined arm span.

[0045] When a user is determined to be in a seated or reclining position, the virtual space configuration system can provide various other virtual space customizations. In one instance, the virtual space configuration system can obtain metrics for different floor heights to use when the user is seated. These metrics can come from, for example, machine learning models trained to predict desired floor heights, user input specifying changes in floor height (e.g., using controllers, gestures, or tracked user gazes), and / or past floor height settings with similar characteristics determined by one or more users. The virtual space configuration system can then set the floor height based on these metrics. This sets a minimum height for virtual objects associated with the user, thereby improving accessibility for the user in the virtual space by eliminating situations where the user would otherwise have to move to the edge of a chair or sofa and reach the floor.

[0046] In another example, a virtual space configuration system can facilitate adjustments to application mechanisms specific to a user's sitting or lying position. For instance, when a user sits or lies down, the application is notified to adjust the placement of virtual objects to fit within the typical or measured user's arm span. For example, a virtual object that a user would typically take a step to interact with could be automatically moved within reach. This can be in response to flags set by the virtual space configuration system for sitting and / or lying down modes, which can then be displayed to the application. Applications can be customized to have different mechanisms based on such flags.

[0047] In another scenario, the virtual space configuration system can configure boundary patterns based on the user's posture. In one case, when the user is standing, the virtual space can set boundaries, and the system will display the boundaries or issue a warning when it predicts the user may connect to them. For example, when the user is standing, the boundary could be a red grid, which would immediately draw the user's attention if displayed in the virtual space. However, because the user may move more slowly or only move her arm, collisions with the boundary are less likely to be a problem when seated. Therefore, the boundary when seated could be a less immersive pattern, such as a pattern with small gray crosses (e.g., +) markings. Alternatively, instead of displaying the boundary when seated, the system can identify and display the real-world objects around the user when it predicts they may collide with them (e.g., when they are within the user's arm span).

[0048] In another example, a virtual space configuration system can enable experiences that are available only when the user is in a specific posture or that trigger specific actions when the user changes between postures. For instance, after determining that the user is in a seated posture, the artificial reality system can initiate a "seated only" experience. The virtual space configuration system can continuously monitor the user's posture throughout the experience. If the user stands up, this can trigger the artificial reality system to take actions such as automatically stopping the seated only experience, providing the user with a notification to return to a seated position, recording the time the user stands during the experience, switching to a delivery mode that displays aspects of the real world instead of parts of the experience, and / or altering aspects of the experience, such as providing different input modalities or changing virtual objects.

[0049] Furthermore, the virtual space configuration system can provide a virtual "workspace" area that appears when the user is seated and in a specific additional posture (such as leaning forward). The workspace can be the area in front of the user, for example, based on one or more of the following: determined user arm span, general user arm length statistics, previous user settings, user drawing area, and / or an identifier of an area including specific objects (e.g., keyboard, monitor, mouse, desktop area, etc.). The virtual space configuration system can also detect that the user is leaning forward by at least a threshold amount while seated. In some implementations, this may also depend on identifying a flat workspace (such as a table) in front of the user. In making this further posture determination, the virtual space configuration system can implement a transfer mode, i.e., a mode that shows a representation of at least a portion of the real world (in this case, the determined workspace area). This allows the user to quickly and easily switch between interacting with real-world items in the workspace area and virtual objects in the virtual space.

[0050] In another scenario, the virtual space configuration system can automatically customize the size (e.g., size and / or shape) of the virtual area for a seating pattern. The system can determine the size based on context or user details such as determined user arm span, average or determined statistics of similar users, previous user settings, or the area drawn by the user, or by identifying objects in the surrounding area. The virtual space configuration system can then set the virtual area based on the determined dimensions, such as a rectangle or semicircle in front of the user, or a full circle around the user.

[0051] Embodiments of the disclosed technology may include artificial reality systems or combinations thereof. Artificial reality, or surreal (XR), is a form of reality that has been adapted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured content (e.g., photographs of the real world). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect for the viewer). Furthermore, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, for example, these are used to create content in artificial reality and / or be used in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, mobile devices or computing systems, “cave” environments or other projection systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0052] As used herein, “virtual reality” or “VR” refers to an immersive experience where the user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to a system in which users view images of the real world after they have passed through a computing system. For example, a tablet computer with a camera on its back can capture images of the real world and then display them on a screen on the side of the tablet computer opposite the camera. The tablet computer can process and adjust or “enhance” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to a system in which light entering the user’s eyes is partly generated by a computing system and partly includes light reflected from objects in the real world. For example, an MR headset can be shaped like a pair of glasses with a transmission display that allows light from the real world to pass through a waveguide while light is emitted from a projector in the MR headset, allowing the MR headset to present virtual objects that are blended with real objects that the user can see. As used herein, “artificial reality,” “hyper-reality,” or “XR” refers to any one of VR, AR, MR, or any combination or hybrid of them.

[0053] Existing XR systems provide virtual spaces. However, these systems can be difficult to use and offer limited functionality. Existing XR systems typically do not differentiate between user postures when configuring virtual spaces, requiring users to manually adjust the virtual space or operate within a virtual space that may be difficult to use, distracting, or inaccessible to certain options. For example, when a user is seated, standard XR systems do not offer the option to adjust the floor position, typically requiring the user to leave their seat to reach virtual objects placed on the floor. As another example, existing XR systems often have a single warning system for when a user is about to collide with a wall in the virtual space. However, this can be distracting and unnecessary when the user is seated, as such a collision is unlikely to cause any damage. Furthermore, existing XR systems require extensive setup of the virtual space, which may be unnecessary for seating configurations where the virtual space is likely small and unlikely to require a specific silhouette.

[0054] The virtual space configuration systems and processes described herein are intended to overcome such problems associated with traditional XR systems and to provide users with greater control over their virtual spaces. Compared to interactions in existing XR systems, the disclosed virtual space configuration systems and processes are also expected to offer greater functionality and a more natural and intuitive user experience. Despite being natural and intuitive, the virtual space configuration systems and processes described herein are rooted in computerized artificial reality systems, rather than simulations of traditional interactions. For example, these virtual space configuration systems and processes can determine when a user is seated and, in response, provide virtual space customization. One such virtual space customization could be allowing adjustment of the floor height. Another virtual space customization could be setting a sign that can be exposed to the application to adjust the application's settings. Furthermore, virtual space customization could be customizing the display of the virtual space boundaries in the seating mode to reduce distractions. Another virtual space customization could be providing an option to detect when a user leaves the seating mode and trigger a corresponding action. Yet another virtual space customization could be providing a transferable workspace area that allows users to interact naturally with certain real-world objects without removing the virtual reality headset. Yet another virtual space customization could be automatically determining the virtual space dimensions for the seated user.

[0055] Several implementations are discussed in more detail below with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating an overview of devices on which some implementations of the disclosed technologies can be operated. The devices may include hardware components of computing system 100, which can determine user posture and set corresponding virtual space customizations. In various implementations, computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) communicating via wired or wireless channels to distribute processing and share input data. In some implementations, computing system 100 may include a standalone head-mounted device capable of providing users with computer-created or enhanced experiences without requiring external processing or sensors. In other implementations, computing system 100 may include multiple computing devices, such as head-mounted devices and core processing components (such as consoles, mobile devices, or server systems), with some processing operations performed on the head-mounted device and others offloaded to the core processing components. The following is in conjunction with… Figure 2A and Figure 2B Describe an example head-mounted device. In some implementations, location and environmental data may be collected solely by sensors incorporated into the head-mounted device, while in other implementations, one or more non-head-mounted computing devices may include sensor components capable of tracking environmental or location data.

[0056] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units located in one device or distributed across multiple devices (e.g., distributed across two or more computing devices 101-103).

[0057] The computing system 100 may include one or more input devices 120 that provide input to the processor 110, informing them of actions. These actions may be mediated by a hardware controller that interprets signals received from the input devices and transmits information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, keyboard, touchscreen, touchpad, wearable input device (e.g., haptic gloves, bracelets, rings, earrings, necklaces, watches, etc.), camera (or other light-based input device, such as an infrared sensor), microphone, or other user input device.

[0058] For example, processor 110 can be coupled to other hardware devices by using internal or external buses, such as PCI buses, SCSI buses, or wireless connections. Processor 110 can communicate with the hardware controller of a device such as display 130. Display 130 can be used to display text and graphics. In some implementations, display 130 includes an input device as part of the display, such as when the input device is a touchscreen or equipped with an eye orientation monitoring system. In some implementations, the display and input device are separate. Examples of display devices include: LCD screens, LED screens, projectors, holographic or augmented reality displays (such as head-up displays or head-mounted displays), etc. Other I / O devices 140 can also be coupled to the processor, such as network chips or cards, video chips or cards, audio chips or cards, USB, FireWire or other external devices, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, etc.

[0059] The computing system 100 may include a communication device capable of wireless or wired communication with other local computing devices or network nodes. The communication device may use, for example, a TCP / IP protocol to communicate with another device or server over a network. The computing system 100 may utilize the communication device to distribute operations across multiple network devices.

[0060] Processor 110 can access memory 150, which may be contained on one of the computing devices of computing system 100 or distributed across multiple computing devices or other external devices of computing system 100. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory (such as flash memory, hard disk drive, floppy disk, CD, DVD, magnetic storage device, tape drive, etc.). Memory is not a propagating signal detached from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160 storing programs and software, such as operating system 162, virtual space configuration system 164, and other application programs 166. Memory 150 may also include data memory 170, which may include various models (e.g., pose classifier, boundary collision predictor, user height or arm span identifier, etc.), floor height settings, seating flag variables, boundary pattern variables, and associated display configurations, pose change mappings, virtual experiences, workspace area settings, virtual area settings, other configuration data, settings, user options or preferences, etc., which may be provided to program memory 160 or any element of computing system 100.

[0061] Some implementations can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be used with this technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframes, and distributed computing environments including any of the above systems or devices.

[0062] Figure 2AThis is a wiring diagram of a virtual reality head-mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. The position sensors 220, IMU 215, and computing units 230 may be internal to the HMD 200 and may not be visible to the user. In various implementations, the IMU 215, position sensors 220, and locator 225 may track the movement and position of the HMD 200 in real-world and virtual environments with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 may emit an infrared beam that produces light spots on real objects around the HMD 200. One or more cameras (not shown) integrated with the HMD 200 may detect the light spots. The computing unit 230 in HMD 200 can use the detected light spots to infer the position and movement of HMD 200, as well as identify the shape and position of real objects around HMD 200.

[0063] Electronic display 245 may be integrated with front rigid body 205 and may provide image light to the user as instructed by computing unit 230. In various embodiments, electronic display 245 may be a single electronic display or multiple electronic displays (e.g., displays for each of the user's eyes). Examples of electronic display 245 include: liquid crystal display (LCD), organic light-emitting diode (OLED) display, active matrix organic light-emitting diode display (AMOLED), display subpixels including one or more quantum dot light-emitting diodes (QOLED), projector units (e.g., microLED, LASER, etc.), some other displays, or some combinations thereof.

[0064] In some implementations, the HMD 200 may be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor the HMD 200 (e.g., via light emitted from the HMD 200), which the PC may use, in conjunction with outputs from the IMU 215 and the position sensor 220, to determine the position and movement of the HMD 200.

[0065] In some implementations, the HMD 200 can communicate with one or more other external devices, such as a controller (not shown) that the user can hold with one or both hands. The controller may have its own IMU unit, position sensor, and / or emit additional light spots. The HMD 200 or the external sensors can track these controller light spots. The computing unit 230 in the HMD 200 or the core processing component can use this tracking, combined with the IMU and position output, to monitor the user's hand position and movement. The controller may also include various buttons that the user can actuate to provide input and interact with virtual objects. In various implementations, the HMD 200 may also include additional subsystems such as an eye-tracking unit, an audio system, various network components, etc. In some implementations, instead of the controller or in addition to the controller, one or more cameras 200 included in or external to the HMD can monitor the user's hand position and posture to determine gestures and the movement of the other hand and body.

[0066] Figure 2B This is a wiring diagram of a mixed reality HMD system 250, including a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as shown in link 256. In other implementations, the mixed reality system 250 includes only a head-mounted device without external computing devices, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a delivery display 258 and a frame 260. The frame 260 can accommodate various electronic components (not shown), such as a projector (e.g., laser, LED, etc.), a camera, an eye-tracking sensor, MEMS components, networking components, etc.

[0067] A projector can be coupled to a transmission display 258, for example, via optical elements, to display media to a user. The optical elements may include one or more waveguide components, reflectors, lenses, mirrors, collimators, gratings, etc., for guiding light from the projector to the user's eyes. Image data can be transmitted from the core processing unit 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes as output light via the optical elements. The output light can be mixed with the light transmitted through the display 258, allowing the output light to present virtual objects that appear to exist in the real world.

[0068] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example in 3DoF or 6DoF, track various parts of the user (e.g., hands, feet, head or other body parts), map virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0069] Figure 3 This is a block diagram illustrating an overview of an environment 300 in which some implementations of the disclosed technology may operate. Environment 300 may include one or more client computing devices 305A-D, examples of which may include computing system 100. In some implementations, some of the client computing devices (e.g., client computing device 305B) may be HMD 200 or HMD system 250. Client computing device 305 can operate in a network environment using a logical connection via network 330 to one or more remote computers, such as server computing devices.

[0070] In some implementations, server 310 may be an edge server that receives client requests and coordinates the fulfillment of those requests through other servers such as servers 320A-C. Server computing devices 310 and 320 may include computing systems, such as computing system 100. Although each server computing device 310 and 320 is logically presented as a single server, each server computing device can be a distributed computing environment containing multiple computing devices located in geographically different or the same physical locations.

[0071] Client computing device 305 and server computing devices 310 and 320 can each act as a server or client for other server / (multiple) client devices. Server 310 can connect to database 315. Servers 320A-C can each connect to their corresponding databases 325A-C. As discussed above, each server 310 or 320 can correspond to a set of servers, and each of these servers can share a database or have its own database. Although databases 315 and 325 are logically presented as a single unit, databases 315 and 325 can each be a distributed computing environment containing multiple computing devices, which can be located within their corresponding servers or in geographically different or the same physical locations.

[0072] Network 330 can be a local area network (LAN), wide area network (WAN), mesh network, hybrid network, or other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing device 305 can connect to network 330 via a network interface, such as via wired or wireless communication. Although the connection between server 310 and server 320 is shown as a separate connection, these connections can be any kind of LAN, WAN, wired network, or wireless network, including network 330 or a separate public or private network.

[0073] Figure 4 This is a block diagram illustrating component 400 that may be used in a system employing the disclosed technology in some implementations. Component 400 may be included in a single device of computing system 100 or may be distributed across multiple devices of computing system 100. Component 400 includes hardware 410, a mediator 420, and a dedicated component 430. As discussed above, systems implementing the disclosed technology may use various hardware, including processing unit 412, working memory 414, input and output devices 416 (e.g., camera, display, IMU unit, network connection, etc.), and storage memory 418. In various implementations, storage memory 418 may be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, storage memory 418 may be one or more hard disk drives or flash drives accessible via a system bus, or may be a cloud storage provider (such as in storage 315 or 325) or other network storage accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device such as client computing device 305 or in a server computing device such as server computing device 310 or 320.

[0074] The mediator 420 may include components that mediate resources between the hardware 410 and the dedicated component 430. For example, the mediator 420 may include an operating system, services, drivers, a basic input / output system (BIOS), controller circuitry, or other hardware or software systems.

[0075] Dedicated component 430 may include software or hardware configured to perform operations for customizing the virtual space based on user posture. For example, dedicated component 430 may include a posture analysis module 434, a standing mode function 436, a seat-floor height function 438, a seat-sign configuration function 440, a seat-boundary display function 442, a seat-seating experience function 444, a seat-workspace area function 446, a seat-automated virtual area function 448, and components and APIs such as interface 432 that can be used to provide a user interface, transmit data, and control. In some implementations, component 400 may be in a computing system distributed across multiple computing devices, or it may be an interface to a server-based application that executes one or more dedicated components 430.

[0076] The posture analysis module 434 can receive sensor inputs (e.g., images from a camera, position sensor data, controller sensor inputs, etc.) and / or determined body biomechanical models (e.g., the user's kinematic skeleton model, hand position, etc.) and use these to determine the user's posture. In various implementations, posture can specify whether the user is standing, sitting, lying down, etc. In some implementations, standing posture can be categorized as standing, moving, or standing, stationary, or other movement-based postures. The following section combines... Figure 5 Block 502 describes additional details regarding the determination of the user's posture.

[0077] Some implementations may include a standing mode function 436. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a standing posture. Performing the standing mode function 436 may include receiving a user-specified boundary for a standing, moving posture, or an automatically sized cylindrical or goblet-shaped boundary (based on a determined arm span) for a standing, stationary posture. If the system predicts that the user may collide with the boundary, it may display the boundary to the user. The following is in conjunction with... Figure 5 Blocks 506 and 510 provide additional details about the stand mode functionality.

[0078] Some implementations may include a seating-floor height function 438. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing the seating-floor height function 438 may include receiving floor height metrics, such as user selection of floor height, floor height based on the determined user height, average floor height selected by other users, etc. The seating-floor height function 438 may use this metric to set the virtual floor height. The following is in conjunction with... Figure 6A and Figure 7A Provides additional details on setting the virtual floor height when the user is seated.

[0079] Some implementations may include a seating sign configuration function 440. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing the seating sign configuration function 440 may include setting a sign in response to determining that the user is in a seating posture. This sign can then be exposed to applications, allowing them to adjust the positioning of objects and other mechanisms based on whether the sign is set. The following is in conjunction with... Figure 6B and Figure 7B Additional details are provided regarding setting up and displaying seating signs to allow the application of adjustment mechanisms.

[0080] Some implementations may include a seating-boundary display function 442. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing the seating-boundary display function 442 may include determining a virtual space boundary pattern, such as a pattern, color, or type, where the type may be a virtual wall, an object displayed in a delivery pattern, a warning message, or other alarm. When the virtual space configuration system detects a boundary display event, such as predicting that the user will intersect with a boundary or a real-world object, or that a real-world object has entered the virtual space, performing the seating-boundary display function 442 may further include displaying the boundary or representation of a real-world object according to the determined virtual space boundary pattern. The following is in conjunction with... Figure 6C and Figure 7C Provides additional details on selecting the virtual space boundary mode and the corresponding display events.

[0081] Some implementations may include seat-only experience functionality 444. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing seat-only experience functionality 444 may include detecting further changes in posture while the user is engaged in the seat-only virtual experience. Using a posture-to-response action mapping, such as that provided by the seat-only virtual experience, response actions for posture changes can be determined and performed. For example, if the user stands up, the application may pause, provide a notification, record that the user is standing, change the input modality, and / or change the virtual object. The following is in conjunction with... Figure 6D and Figure 7D Provides additional details about the response action that triggers the change in posture.

[0082] Some implementations may include a seating-workspace area function 446. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing the seating-workspace area function 446 may include detecting a further forward tilting posture while the user maintains the seating posture. In response, the execution of the seating-workspace area function 446 may implement a transfer display mode for the determined workspace area, allowing the user to see a representation of real-world objects in the workspace area without removing the head-mounted device or other hardware of the artificial reality system. The workspace area, such as the top of a table or an area including various tools (such as a keyboard, mouse, and / or monitor), may be determined based on a user-predefined area, the determined user's arm span, the average of workspaces set by other users, and / or one or more of computer vision and object detection used to identify the area. In some implementations, a trained machine learning model can determine the workspace region based on the current context (e.g., user details and / or camera input), where the model is trained on similar inputs matching the user-selected workspace region or on automatically identified workspace regions determined based on object identifiers with high confidence values. The following section combines... Figure 6E and Figure 7E Provides additional details on detecting forward tilt, seating posture, and displaying the workspace area in delivery mode.

[0083] Some implementations may include a seating-automated virtual area function 448. In these implementations, the virtual space configuration system may perform these functions in response to the posture analysis module 434 identifying a seating posture. Performing the seating-automated virtual area function 448 may include automatically determining the size of the virtual area for the seating position based on one or more of the following: a user-defined area, the user's arm span, an average area set by other users, etc. In some implementations, a trained machine learning model may determine the virtual space size based on the current context (e.g., user details and / or camera input), where the model is trained based on similar inputs that match the virtual space selected by the user. The shape of the virtual area may be automatically determined based on one or more of the following: settings in the current application, user selection, the current purpose determined for the virtual space using a mapping from purpose to virtual space shape, etc. (The following is in conjunction with...) Figure 6F and Figure 7F Provides automatic determination of virtual regions.

[0084] Those skilled in the art will understand that the above Figures 1-4The components illustrated in the diagrams and those in each flowchart discussed below can be modified in various ways. For example, the order of the logic can be rearranged, sub-steps can be executed in parallel, the illustrated logic can be omitted, or other logic can be included. In some implementations, one or more of the above components can perform one or more of the processes described below.

[0085] Figure 5 This is a flowchart illustrating a process 500 used in some implementations of this technology for setting up a virtual space configuration based on user posture. In various implementations, process 500 may be performed by an artificial reality system (e.g., by a virtual space configuration subsystem) when the artificial reality system is first turned on, when the artificial reality system detects a change in the user, continuously on a periodic basis (e.g., every 1-2 seconds), or in response to a detected change in posture (e.g., posture detection of block 502 is performed periodically, or in response to input signals such as changes in the height of a head-mounted device or controller, or other movements).

[0086] At block 502, process 500 may determine the user's posture. A "posture" is the position or configuration of one or more parts of the user's body. For example, a posture may be sitting, standing, lying down, arms outstretched, a specific hand position or posture, head orientation, torso rotation or angle, etc. In some implementations, a posture may also include movement—such as a specific movement of one or more body parts and / or a specific movement relative to a point or object. For example, the identified first posture may be stationary, while the identified second posture may be standing and has undergone a threshold level of lateral movement relative to a center point. In various implementations, process 500 may automatically determine the user's posture, for example, based on the height of the head-mounted device of the determined artificial reality system, specific detected movements (e.g., of the head-mounted device, controllers, hands, legs, or other body parts), images captured by the artificial reality system, and other inputs such as location data, IMU data, etc. In some implementations, various measurements and determinations from the artificial reality system may be fed to a machine learning model trained to classify the user's current posture. Determining user poses (or "gestures") is discussed in more detail in U.S. Patent Application No. 16 / 663,141, filed October 9, 2019, entitled "Systems and Methods for Generating Dynamic Obstacle Collision Warnings Based On Detecting Poses of Users," the entire contents of which are incorporated herein by reference. In some implementations, user poses may be specified by user input, or user input may validate automatically detected poses.

[0087] At block 504, process 500 may determine whether the posture determined at block 502 corresponds to a standing-movement posture. A standing-movement posture may indicate that the user is standing and in a situation where she can move laterally (as opposed to normally standing in the same location). This may be indicated by the artificial reality system determining that the user is standing (e.g., based on the determined head-mounted device height, user posture selection input, etc.) and one or more of the following: the user has indicated a boundary area for the virtual space, the user has specified an intention to move, the current application is designed to move while standing, or it is determined that the user has moved laterally from the center point by at least a threshold amount (i.e., it is determined that they have moved from their standing location). When the posture corresponds to a standing-movement mode, process 500 may proceed to block 506, where it sets up a standing-movement virtual space customization. For example, the virtual space may be a user-defined space and / or a space defined to avoid collisions between the user and objects detected in the real-world space surrounding the user. If the artificial reality system determines that the user is at risk of colliding with the boundary, the customization may also include setting display features to indicate the boundary, such as using a very prominent red grid pattern to immediately draw the user's attention. If the posture is not a standing movement, process 500 can continue to block 508.

[0088] At block 508, process 500 can determine whether the posture determined at block 502 corresponds to a standing still posture. A standing still posture can indicate that the user is standing and unlikely to move laterally (i.e., likely standing within a few feet of the same location). Similar to a standing moving posture, a standing still posture can be indicated by the artificial reality system determining that the user is standing (e.g., based on the determined head-mounted device height, user posture selection input, etc.). However, in this case, the user may have specified an intention not to move, the current application may be designed to remain still while standing, or the artificial reality system can determine that the user has not moved from the center point by at least a threshold amount (i.e., determined that they have not moved laterally by a significant amount, such as 1-2 feet, from their current standing location). When the posture reaches a standing still posture, process 500 can proceed to block 510, where it sets a custom standing still virtual space. For example, the virtual space can be a user-defined space, or a shape around a user-defined cylinder or wine glass. If the artificial reality system determines that the user is at risk of colliding with the boundary, customization may also include setting display features to indicate the boundary, such as using a prominent red grid pattern to immediately draw the user's attention, or using a less immersive pattern of a gray cross (e.g., +) shape because such a collision is unlikely to cause damage. If the posture is not static standing, process 500 may continue to block 512.

[0089] At block 512, process 500 can determine whether the posture determined at block 502 corresponds to a sitting posture. The sitting posture can be indicated by an artificial reality system that determines a threshold distance between the system's head-mounted device and the average height of a sitting head-mounted device; it can come from user input specifying the posture; it can be determined based on a machine learning model (which takes sensor input and classifies the current posture); or it can be assumed based on the current application used while seated, or from the orientation provided to the user, or using other metrics. If the determined posture is sitting, process 500 can proceed to block 514, where any of various seating virtual space customizations can be applied. In various implementations, available seating virtual space customizations may include: allowing adjustment of the floor height (see below regarding...). Figure 6A (For additional details), you can set flags that can be exposed to the application to adjust the application's mechanisms (see below for more information). Figure 6B Additional details), customize the display of virtual space boundaries in seated mode to reduce distractions (see below for more information). Figure 6C Additional details), providing options to detect when a user leaves seated mode and trigger the corresponding action (see below for more information). Figure 6D (Further details below) provide a transfer workspace area that allows users to interact naturally with certain real-world objects without removing the artificial reality headset (see below for more information). Figure 6E Additional details), and automatically determine the virtual space size for seated users (see below for more information). Figure 6F (Additional details). If the posture is not seated, process 500 can return to block 502 to continue monitoring the user's posture for identified variations.

[0090] Figure 6A This is a flowchart illustrating a process 600 used in some implementations to customize the floor height when the user is seated. In some implementations, process 600 may be executed as a subprocess of block 514 of process 500. In some cases, process 600 may be executed in response to other triggers, such as an application changing posture mode, user selection, user changes, or the start of a specific application.

[0091] At block 602, process 600 may obtain a metric for the seated floor height. This metric may be determined based on factors such as the user's standing height, user's arm span, the dimensions of the chair, sofa, or other objects on which the user sits (e.g., determined using camera and computer vision techniques integrated with an artificial reality system), the average floor height set by other users (or users identified as similar to the current user), or other statistical data. In some implementations, one or more of these features may be used as input to a machine learning model trained to predict the desired floor height based on previous user choices. In other implementations, these features may be used to map features to floor height. In some implementations, the metric may be a value selected by the user, such as indicating floor height through hand gestures, input to a controller, voice commands, gaze, etc. In some implementations, the user may be identifiable, and the obtained metric may be based on previous choices made for that user.

[0092] At block 604, process 600 can set the floor height based on the measurement obtained at block 602. This sets a minimum height of the virtual object relative to the user, thereby improving accessibility for the user in the virtual space by eliminating situations where the user would otherwise have to move to the edge of a chair or sofa and reach the floor. In some implementations, the floor height can be set for the artificial reality system across applications, or it can be set for a specific application, with different floor heights set for other applications. The following section combines... Figure 7A Discuss examples of setting floor height.

[0093] Figure 6B This is a flowchart illustrating process 610, used in some implementations, to set flags to allow applications to adjust seating configurations. In some implementations, process 610 may be executed as a subprocess of block 514 of process 500. In some cases, process 610 may be executed in response to other triggers, such as an application changing its posture mode, user selection, user changes, or the start of a specific application.

[0094] At block 612, in response to determining that the user is in a seated posture (e.g., determined at blocks 502 and 512), process 610 may set a flag indicating the seated posture. The flag can be any type of variable, such as a binary value, a posture identifier, a posture name, etc. Setting the flag can include various types of writing to memory, such as setting program or operating system variables, writing to a database field, writing to a file, etc. For example, an operating system for an artificial reality system may maintain a set of operating condition variables, one of which could be a posture indicator or an "isSeated" flag. At block 614, process 610 may expose the flag set at block 612 to the application. For example, the application may send a request to the operating system to obtain the value of the flag, read it from a database where the flag is set, and so on. Reading the flag can allow the application to modify certain mechanisms to better accommodate a seated user. For example, the application can change the positioning of virtual objects, moving them within easy reach, whereas such adjustments might not be necessary if the user is standing, as the user could take a step to reach objects further away, or more easily reach objects on the floor, etc. (The following is a continuation of the previous paragraph.) Figure 7B Discuss examples of using flags to allow applications to adjust mechanisms used for seating configuration.

[0095] Figure 6C This is a flowchart illustrating a process 620 used in some implementations to customize the display of virtual space boundaries in response to user gestures. In some implementations, process 620 may be executed as a subprocess of block 514 of process 500. In some cases, process 620 may be executed in response to other triggers, such as an application changing its gesture mode, user selection, user changes, or the launch of a specific application.

[0096] At block 622, process 620 may receive a selection of a virtual space boundary display mode. In some implementations, this selection may be a mapping from posture to boundary display mode. For example, when the system detects that the user is standing (a situation where the user is likely to move around more frequently and faster than when seated), collisions with the boundary are more likely to cause injury. Therefore, a standing posture may be mapped to a boundary display mode that is more likely to attract the user's attention, such as virtual walls appearing as a specific color (e.g., bright red, green, orange, etc.) and / or having a specific pattern (e.g., a grid, closely packed dots, flashing lines, etc.). However, when the system detects that the user is sitting or lying down (a situation where the user is likely to move around less and slower than when standing), collisions with the boundary are less likely to cause injury. Therefore, a sitting or lying posture may be mapped to a boundary display mode that is less likely to interfere with the user's artificial reality experience, such as boundaries appearing as a less bright color (e.g., gray, tan, brown, etc.) and / or having a specific pattern (e.g., a grid showing only +s at line intersections, scattered dots, dim lines, etc.). In some implementations, for seated or reclining postures, the chosen boundary display mode can be a transitive representation of real-world objects into the virtual environment. In this transitive mode, if the AI ​​system determines that the user is approaching a boundary or about to collide with an object, the AI ​​system can display real-world objects in virtual space to allow the user to identify and avoid them. In some implementations, real-world objects displayed in this transitive mode can be minimized or shown only as shadows to avoid taking the user too far from her virtual experience.

[0097] At block 624, process 620 can detect a boundary display event. This can occur by detecting that the user is within a threshold distance of the boundary, the user's determined trajectory is expected to intersect the boundary, and / or the user's body position and configuration (e.g., arm or leg span, stride length, height, motion curve, etc.) make it highly likely that the user will intersect the boundary. In some implementations, a machine learning model can be trained to make this determination. For example, inputs to the machine learning model may include inertial, position, camera and / or other sensor inputs from the head-mounted device and / or controller of the artificial reality system, a model of the user's body (e.g., skeletal structure), past movement data of the user or average user, boundary configuration details, etc. The machine learning model can be trained to predict whether the user is likely to intersect the boundary (e.g., based on previous identification of the context when the user intersects the boundary). In some implementations, other triggers for displaying the boundary may be present, such as another person or object entering the virtual space, which may cause a change in the boundary or an implementation transmission mode or current application signaling to display the boundary.

[0098] At block 626, process 620 may, in response to the detected boundary display event at block 624, display the boundary or object in the surrounding area using the selected virtual space boundary display mode chosen at block 622 (i.e., implement the delivery mode). For example, process 620 may display a virtual wall or a portion of a virtual wall with a specified pattern and / or color, may display objects within a threshold distance of the user or objects that the artificial reality system determines the user may collide with, may display virtual walls or objects that gradually disappear as their distance from the user increases, and so on. (The following is in conjunction with...) Figure 7C This section discusses examples of customizing the display of virtual space boundaries to show the boundaries when the user is in a seated position.

[0099] Figure 6D This is a flowchart illustrating process 630 used in some implementations to achieve a gesture-specific virtual experience. In some implementations, process 630 can be executed as a subprocess of block 514 of process 500. In some cases, process 630 can be executed in response to other triggers, such as an application changing its gesture mode, user selection, user changes, or the launch of a specific application.

[0100] At block 632, process 630 can initialize a virtual experience designed to be performed in the current posture. For example, an application can be configured to be used in a seated-only posture, a standing-only posture, a lying-only posture, and so on. As more concrete examples, a company can specify a training program where the user should be seated throughout the program, or the program should be paused; a game can be configured to have a first input modality (e.g., a static virtual control panel) when the user is seated, and a different input modality (e.g., based on detected user movements) if the user is standing; application developers may want to tune their application for the most common user postures and may therefore collect statistics on aggregated user postures; and so on.

[0101] At block 634, process 630 can detect whether the user's posture has changed. This can occur in a similar manner to block 502. In some implementations, only certain specific posture changes will trigger the "Yes" (Y) branch from block 634. For example, while process 630 can detect various posture changes, such as arm position or torso tilt, the current application can specify that only changes from a seated to a standing posture or lateral movement exceeding a threshold distance will trigger the "Yes" branch. In various implementations, the operating system for the AI ​​system or the application running by the AI ​​system can specify certain mappings between posture changes and response actions, and it can be the posture changes that trigger these mappings from the "Yes" branch of block 634. If no such posture change is detected (No (N) branch), process 630 can proceed to block 638. If such a posture change is detected (Yes branch), process 630 can proceed to block 636.

[0102] At block 636, in response to the posture change detected at block 634, process 630 may trigger a response action. In some implementations, response actions may be set by the artificial reality system, such as automatically stopping or pausing the virtual experience, providing a notification to resume the previous posture or a notification that the current posture is not recommended for the experience, recording the time spent in various postures, switching to another mode (e.g., a walking mode), and so on. In some implementations, the current application may specify one or more mappings for trigger actions to be performed in response to a specific posture change. For example, the application may specify that if the user stands up from a sitting posture, a notification to return to the seated position should be displayed and the standing time should be recorded, and if the standing time exceeds a threshold, the virtual experience should be paused. As another example, a game application may specify that when the user is in a seated posture, the virtual experience should be driving a virtual car, but if the user stands up, the virtual experience should change to indicating exiting the virtual car and transitioning to a virtual walking mode.

[0103] At block 638, process 630 can determine whether the virtual experience initiated at block 632 is still in progress. Process 630 can monitor posture and trigger response actions while the virtual experience continues. Once the virtual experience ends, process 630 can also terminate. The following section combines... Figure 7D Discuss examples of achieving pose-specific virtual experiences.

[0104] Figure 6E This is a flowchart illustrating process 640 used in some implementations to create a virtual area for a seated workspace. In some implementations, process 640 may be executed as a subprocess of block 514 of process 500. In some cases, process 640 may be executed in response to other triggers, such as an application changing its posture mode, user selection, user changes, or the launch of a specific application.

[0105] At block 642, process 640 can detect a forward tilting posture while the user remains seated. This could be a tilt threshold degree from the vertical direction, such as 10, 15, or 20 degrees. In some implementations, other postures that trigger the workspace area can be detected, such as a raised arm and a hand ready for a virtual keyboard or a forearm resting on a table.

[0106] At block 644, process 640 may determine a workspace area. This could be, for example, an area pre-established by the user; an area with a pre-defined size and shape (e.g., a rectangle three feet by four feet); an area defined based on user characteristics (e.g., a semicircle or rectangle with a size determined based on the user's arm span, such as an area fully accessible to the user); an area corresponding to the arm span of an average user (a user with characteristics similar to the current user, such as height), or an average of areas manually set by other users; or an area based on computer vision / object identification (e.g., an area corresponding to the top of the table in front of the user or an area containing the user's tools (such as a keyboard, laptop, or monitor), etc. In various implementations, this area may be established in response to a posture detected at block 642 or may be pre-established.

[0107] At block 646, process 640 can implement a transfer display mode for the defined workspace area. As discussed above, the transfer mode can represent a portion of the real world within the artificial reality environment. For example, images captured by an external camera can be input into the artificial reality environment, portions of the display can be disabled to allow the user to see light passing through the display, portions of the display can be moved to allow light to enter the user's eyes, and so on. By implementing the transfer mode for the defined workspace area when a specific posture is detected, the user can easily interact with tools and other objects within the workspace area without removing the artificial reality system's head-mounted device or manually implementing the transfer mode. The following section combines... Figure 7E Discuss an example of how to implement the passing of virtual areas of the workspace when a user is seated and leans forward.

[0108] Figure 6F This is a flowchart illustrating process 650 used in some implementations for automatically customizing a virtual area in a seating pattern. In some implementations, process 650 may be executed as a subprocess of block 514 of process 500. In some cases, process 650 may be executed in response to other triggers, such as an application changing the posture pattern, user selection, user changes, or the launch of a specific application.

[0109] At block 652, process 650 can automatically determine a region for the virtual space for a user in a seated posture. Process 650 can use a pre-defined shape for this region, such as a cylinder centered on the user, a semi-cylinder in front of the user, a cuboid, or others. The size of the region can be set based on information such as settings specified by user input, the determined user arm span, statistics on the general arm span of the user or users identified as similar to the current user, identifiers of real-world objects to be excluded from the virtual space by the artificial reality system, or identifiers of virtual objects to be included in the virtual space by the artificial reality system. For example, process 650 can determine the virtual region as a semi-cylinder in front of the user with a radius equal to half the user's arm span (i.e., the length of one arm). As another example, process 650 can determine the virtual region as a cube in front of the user, excluding all real-world objects in that region. In some implementations, the determined virtual region can be suggested to the user, who can manually adjust its parameters or define different virtual regions, such as having different sizes, shapes, or activation features.

[0110] In some implementations, the user's arm span can be determined by setting it to be equal to the user's height. In some cases, this setting can be updated based on identifying the user's hand or controller position and extending the determined arm span beyond the distance of the identified hand or controller. In other cases, the arm span can be reduced if the identified hand or controller never extends to the determined arm span within a threshold time amount or the usage amount of the artificial reality system. In some implementations, instead of starting from an initial height determination, the arm span can be determined directly by observing these hand or controller distances. In some implementations, arm span determination can be determined or adjusted by identifying the user's arms in the user's image and projecting the maximum reach using a body movement model (e.g., a kinematic skeleton model).

[0111] At block 654, procedure 650 can set the virtual region defined at block 652. In various implementations, this virtual region can be set globally for all instances of the currently identified user, for use cases running a specific application, or for use cases during a specific task. The following section combines... Figure 7F This section discusses examples of automatically setting virtual space dimensions.

[0112] Figure 7AThis is a conceptual diagram illustrating Example 700, used in some implementations, to customize the floor height when a user is seated. In Example 700, a virtual space configuration system has determined that user 702 is in a seated posture. In response, the virtual space configuration system implements options for user 702 to adjust the height of virtual floor 704. As shown by arrow 706, user 702 can activate controls (e.g., virtual controls shown in an artificial reality environment, controls on a controller, using her gaze, etc.) to adjust the height of virtual floor 704. The height of virtual floor 704 allows the application to place objects at a height no lower than virtual floor 704. As shown in Example 700, by raising virtual floor 704, user 702 can more easily reach any object placed on virtual floor 704 while seated.

[0113] Figure 7B This is a conceptual diagram of example 710, illustrating a mechanism used in some implementations to allow an application to adjust its seating configuration using a flag. In example 710, user 702 is initially standing, surrounded by objects 712A-E. When the virtual space configuration system detects a change in the user 702's now seated posture, the virtual space configuration system sets a seating flag, which is exposed to the application within the control of object 712. The application can adjust the position of object 712 (as shown by arrows 714A-E) to be within reach of the seated user 702.

[0114] Figure 7C This is a conceptual diagram illustrating Example 720, used in some implementations, to customize the display of virtual space boundaries in response to user gestures. Example 720 first shows a user 702 in a standing posture, where a virtual space 722 has boundary walls that appear when the user 702 is within a threshold distance of different portions of the boundary walls. The boundary walls are configured with a virtual space boundary display mode selection that results in a red grid pattern (e.g., line 724) being displayed on the walls. In a second part of Example 720, the user 702 adopts a seated posture. At this point, the virtual space boundary display mode changes for the virtual space so that real-world objects within the user 702's threshold distance are displayed in a transitive mode. In Example 720, these objects are truncated at lines 726A-D, indicating that they are no longer within the user 702's threshold distance.

[0115] Figure 7DThis is a conceptual diagram illustrating Example 730, used in some implementations to implement a seat-only virtual experience. In Example 730, user 702 is initially in a seated position, and the virtual space configuration system executes the seat-only virtual experience. When user 702 transitions to a standing position, the virtual space configuration system pauses the seat-only virtual experience and displays message 732, notifying user 702 that the seat-only virtual experience will resume when user 702 returns to a seated position.

[0116] Figure 7E This is a conceptual diagram illustrating example 740 used in some implementations to realize a virtual area of ​​a seated workspace. In example 740, user 702 begins in a seated, upright posture. As indicated by arrow 742, user 702 then leans forward while remaining seated. In response to detecting this seated, leaning-forward posture, the virtual space configuration system displays workspace area 744, which is an area that the virtual space configuration system has identified as corresponding to the surface of the table in front of the user, containing a keyboard and monitor.

[0117] Figure 7F This is a conceptual diagram illustrating example 750 used in some implementations for automatically customizing virtual areas in a seated mode. Example 750 shows a first instance where the virtual space configuration system automatically determines the size 752 of the virtual space based on the determined arm span of user 702, which is configured as a semi-cylinder in front of user 702. Example 750 also shows a second instance where the virtual space is a cuboid in front of user 702, with its length and width dimensions 754 set based on the statistical average of areas selected by other users whose height is within a threshold of user 702's height.

[0118] In this specification, references to "implementation" (e.g., "some implementations," "various implementations," "one implementation," "implementation," etc.) indicate that a particular feature, structure, or characteristic described in connection with an implementation is included in at least one implementation of the invention. The appearance of these phrases in different places in the specification does not necessarily refer to the same implementation, nor does it mean that separate or alternative implementations are mutually exclusive with other implementations. Furthermore, various features are described that may be exhibited by some implementations but not by others. Similarly, various requirements are described that may be requirements of some implementations but not others.

[0119] As used herein, above the threshold means that the value of the item being compared is higher than any other specified value, that the item being compared is among a specified number of items with a maximum value, or that the item being compared has a value within a specified highest percentage value. As used herein, below the threshold means that the value of the item being compared is lower than any other specified value, that the item being compared is among a specified number of items with a minimum value, or that the item being compared has a value within a specified lowest percentage value. As used herein, within the threshold means that the value of the item being compared is between two other specified values, that the item being compared is among a specified number of items in the middle, or that the item being compared has a value within a specified percentage range in the middle. Relative terms, such as high or unimportant, when not otherwise defined, can be understood as assigning a value and determining how that value is compared to a given threshold. For example, the phrase “select fast connection” can be understood as meaning to select a connection with an assigned value corresponding to a connection speed above a threshold.

[0120] As used in this article, the word “or” refers to any possible permutation of a set of items. For example, the phrase “A, B or C” means at least one of A, B, C or any combination thereof, such as: A; B; C; A and B; A and C; B and C; A, B and C; or multiples of any items, such as A and A; B, B and C; A, A, B, C and C; and so on.

[0121] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of the embodiments and implementations. The specific features and actions described above are disclosed as exemplary forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited beyond the scope of the appended claims.

[0122] All of the foregoing patents, patent applications, and other references are incorporated herein by reference. If necessary, aspects may be modified to incorporate the systems, functions, and concepts of the foregoing references to provide further implementations. In the event of any conflict between statements or subject matter in the referenced documents and statements or subject matter of this application, this application shall prevail.

Claims

1. A method for providing a virtual experience to a user of an artificial reality head-mounted device, the method comprising: Detect the first posture of the user wearing the artificial reality head-mounted device; When the user is in the first posture, determine the position of one or more virtual objects around the user within the virtual experience; Detect changes in the user's posture; Based on the change in the user's posture from the first posture to the second posture, which is different from the first posture, the positions of the one or more virtual objects within the virtual experience are adjusted such that each corresponding virtual object moves from a first position within the virtual experience to a second position around the user within the virtual experience.

2. The method of claim 1, wherein each corresponding virtual object in the second position is within the reach of the user in the virtual experience.

3. The method of claim 1, wherein the first position of each corresponding virtual object is at a first distance from the user, and the second position of each corresponding virtual object is at a second distance from the user.

4. The method according to claim 3, wherein the first distance is greater than the second distance.

5. The method according to claim 1, wherein the method further comprises: Detect another change in the user's posture; as well as Based on the change in the user's posture from the second posture to a third posture, which is different from the first and second postures, the positions of one or more virtual objects within the artificial reality head-mounted device are adjusted such that each corresponding virtual object moves from the second position within the virtual experience to a third position around the user within the virtual experience.

6. The method of claim 1, wherein the method further comprises: Detect another change in the user's posture; as well as Based on the change in the user's posture from the second posture to a third posture, which is different from the first and second postures, the virtual experience is paused and an instruction to return to the first or second posture is generated for the user.

7. The method of claim 6, wherein the method further comprises: Detect any additional changes in the user's posture; as well as The virtual experience is activated when the user's posture changes from the third posture to the first or second posture based on the additional change in the determined posture.

8. The method of claim 1, wherein the method further comprises: The virtual experience is displayed from a first perspective before the change in the user's posture is detected and while the user is in the first posture; as well as After the second pose is detected, the virtual experience is displayed from a second perspective.

9. The method of claim 1, wherein the first posture and the second posture include at least a user standing or sitting.

10. An artificial reality head-mounted device configured to provide a virtual experience to a user, the artificial reality head-mounted device comprising: One or more processors; as well as One or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: Detect the first posture of the user wearing the artificial reality head-mounted device; When the user is in the first posture, determine the position of one or more virtual objects around the user within the virtual experience; Detect changes in the user's posture; as well as Based on the change in the user's posture from the first posture to the second posture, which is different from the first posture, the positions of the one or more virtual objects within the virtual experience are adjusted such that each corresponding virtual object moves from a first position within the virtual experience to a second position around the user within the virtual experience.

11. The artificial reality head-mounted device of claim 10, wherein the first position of each corresponding virtual object is at a first distance from the user, and the second position of each corresponding virtual object is at a second distance from the user.

12. The artificial reality head-mounted device of claim 10, wherein the one or more programs further include instructions for: Detect another change in the user's posture; and Based on the change in the user's posture from the second posture to a third posture, which is different from the first and second postures, the positions of one or more virtual objects within the artificial reality head-mounted device are adjusted such that each corresponding virtual object moves from the second position within the virtual experience to a third position around the user within the virtual experience.

13. The artificial reality head-mounted device of claim 10, wherein the one or more programs further include instructions for: Detect another change in the user's posture; and Based on the change in the user's posture from the second posture to a third posture, which is different from the first and second postures, the virtual experience is paused and an instruction to return to the first or second posture is generated for the user.

14. The artificial reality head-mounted device of claim 13, wherein the one or more programs further include instructions for: Detect any additional changes in the user's posture; and The virtual experience is activated when the user's posture changes from the third posture to the first or second posture based on the additional change in the user's posture.

15. The artificial reality head-mounted device of claim 10, wherein the one or more programs further include instructions for: The virtual experience is displayed from a first-person perspective before the change in the user's posture is detected and while the user is in the first posture; and After the second pose is detected, the virtual experience is displayed from a second perspective.

16. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions that, when executed by a computing device, cause the computing device to: Detect the first posture of the user wearing the artificial reality head-mounted device; When the user is in the first posture, determine the position of one or more virtual objects around the user within the virtual experience; Detect changes in the user's posture; as well as Based on the change in the user's posture from the first posture to the second posture, which is different from the first posture, the positions of the one or more virtual objects within the virtual experience are adjusted such that each corresponding virtual object moves from a first position within the virtual experience to a second position around the user within the virtual experience.

17. The non-transient computer-readable storage medium of claim 16, further comprising instructions that cause the computing device to: Detect another change in the user's posture; and Based on the change in the user's posture from the second posture to a third posture, which is different from the first and second postures, the positions of one or more virtual objects within the artificial reality head-mounted device are adjusted such that each corresponding virtual object moves from the second position within the virtual experience to a third position around the user within the virtual experience.

18. The non-transient computer-readable storage medium of claim 16, further comprising instructions that cause the computing device to: Detect another change in the user's posture; and Based on the determination that the user's posture has changed from the second posture to a third posture, which is different from the first and second postures, the virtual experience is paused and an instruction to return to the first or second posture is generated for the user.

19. The non-transient computer-readable storage medium of claim 18, further comprising instructions that cause the computing device to: Detect any additional changes in the user's posture; and The virtual experience is activated when the user's posture changes from the third posture to the first or second posture based on the additional change in the user's posture.

20. The non-transient computer-readable storage medium of claim 18, further comprising instructions that cause the computing device to: The virtual experience is displayed from a first-person perspective before the change in the user's posture is detected and while the user is in the first posture; and After the second pose is detected, the virtual experience is displayed from a second perspective.

Images