Activation of partial perspective on artificial reality devices

Abstract

Some aspects of the present disclosure relate to providing partial perspective on an artificial reality (XR) device, such as a head mounted display (HMD). Some embodiments may allow a user to open a selective see-through area on an XR device such that one or more portions of a real-world environment that physically surround the user may be seen. In some embodiments, a user may control the perspective area by specifying a percentage (e.g., 50%) of the real-world environment that he wishes to see. In some implementations, a user may control the perspective area by specifying the area of the XR device (e.g., the top of a view) that she wishes to see. In some embodiments, a user may control the perspective area by specifying a physical object (e.g., a real-world table) in the real-world environment that he wishes to see.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 380,412, entitled “Activation of Partial Pass-Through on an Artificial Reality Head-Mounted Display,” with Docket No. 3589-0197PV01, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to providing partial see-through on an artificial reality (hyper-reality (XR)) device, such as a head-mounted display (HMD). Background Art

[0004] Artificial reality (XR) devices are becoming more and more common. As they become more popular, the applications implemented on such devices are becoming more and more complex. Augmented reality (AR) applications can provide interactive 3D experiences that combine real-world images with virtual objects, while virtual reality (VR) applications can provide a completely independent 3D computer environment. For example, AR applications can be used to superimpose virtual objects on a video feed of a real scene observed by a camera. Real-world users in the scene can then make gestures captured by the camera, which can provide interactivity between real-world users and virtual objects. Mixed reality (MR) systems can allow light to enter the user's eyes: the light is partially generated by the computing system and partially includes light reflected from objects in the real world. Users can observe AR, MR, and VR (collectively referred to as XR) experiences through a head-mounted display (HMD) (e.g., glasses or a head-mounted viewer (headset)). Summary of the invention

[0005] According to a first aspect of the present invention, a method for providing partial perspective on an artificial reality device comprises:

[0006] Displaying a virtual reality experience to a user on an artificial reality device;

[0007] receiving a command to activate a see-through view on the artificial reality device, the see-through view associated with a world-locked area of ​​the user in the real-world environment;

[0008] identifying, based on the command, a first portion of a display area of ​​the artificial reality device in which the see-through view is activated, the first portion of the display area of ​​the artificial reality device corresponding to a world-locked area of ​​the real-world environment associated with the see-through view;

[0009] activating a see-through view on a first portion of a display area of ​​an artificial reality device and simultaneously displaying at least a portion of a virtual reality experience on a second portion of the display area of ​​the artificial reality device, the see-through view including a view of a world-locked area in a real-world environment; and fixing the see-through view to the world-locked area in the real-world environment by tracking movement of the artificial reality device relative to the world-locked area, and updating the display area of ​​the artificial reality device through which the first portion of the see-through view is activated based on the tracked movement.

[0010] In some embodiments, activating a see-through view on a first portion of a display area of ​​the artificial reality device includes displaying a feed of a user's real-world environment captured by one or more cameras in operable communication with the artificial reality device.

[0011] In some embodiments, the command is a voice command issued by a user of the artificial reality device, the command is collected by at least one microphone integrated with or in operable communication with the artificial reality device, and identifying a first portion of a display area of ​​the artificial reality device in which a perspective view is activated includes: based on the command, identifying a world-locked area in the real-world environment by performing voice recognition on the command and parsing the command to identify the indicated area or object.

[0012] In some embodiments, the command is a user selection of a virtual slider displayed on the artificial reality device, and the method further comprises: associating the virtual slider with a dynamic partition on the artificial reality device, the dynamic partition virtually dividing the artificial reality device into a first side and a second side;

[0013] Determining a position of the dynamic partition on the artificial reality device based on the user selection; and

[0014] Based on the position of the dynamic partition, assigning a second side of the dynamic partition to a second portion of the display area of ​​the artificial reality device in which at least a portion of the virtual reality experience is displayed, and

[0015] Among them, command-based identification of the first part of the display area of ​​the artificial reality device in which the perspective view is activated includes: based on the position of the dynamic partition, assigning the first side of the dynamic partition to the first part of the display area of ​​the artificial reality device in which the perspective view is activated.

[0016] In some embodiments, the command identifies a percentage of a display area of ​​the artificial reality device in which the see-through view is activated, wherein based on the command identifying a first portion of the display area of ​​the artificial reality device in which the see-through view is activated comprises: assigning the percentage of the display area of ​​the artificial reality device to the see-through view on the first portion of the display area of ​​the artificial reality device,

[0017] The method further comprises:

[0018] A second portion of the display area of ​​the artificial reality device is identified by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device, and wherein displaying at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device includes: cropping the virtual reality experience to correspond to the second portion of the display area of ​​the artificial reality device.

[0019] In some embodiments, the command identifies a percentage of a display area of ​​the artificial reality device in which a see-through view is activated, wherein identifying a first portion of the display area of ​​the artificial reality device in which the see-through view is activated based on the command includes: allocating the percentage of the display area of ​​the artificial reality device to the see-through view on the first portion of the display area of ​​the artificial reality device, wherein the method further includes: identifying a second portion of the display area of ​​the artificial reality device by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device, and wherein displaying at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device includes: adjusting a size of the virtual reality experience to correspond to the second portion of the display area of ​​the artificial reality device.

[0020] In some embodiments, the command identifies the world lock region as an object in the real-world environment.

[0021] In some embodiments, command-based identification of a first portion of a display area of ​​an artificial reality device in which a perspective view is activated includes identifying an object, a location of an object, or at least one of an object and a location of an object by facilitating performing at least one of object recognition, object detection, or both in a real-world environment.

[0022] In some embodiments, command-based identification of a first portion of a display area of ​​an artificial reality device in which a perspective view is activated includes identifying an object, a location of an object, or at least one of an object and a location of an object by accessing scene data, spatial anchor data, or at least one of both scene data and spatial anchor data associated with a real-world environment.

[0023] In some embodiments, the object is a moving object and the perspective view is also fixed to a world locked area in the real world environment by tracking the movement of the moving object relative to the artificial reality device.

[0024] According to a second aspect of the present invention, there is provided a computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform a process for providing partial perspective on an artificial reality device, the process comprising:

[0025] Displaying a virtual reality experience to a user on an artificial reality device;

[0026] receiving a command for activating a see-through view on the artificial reality device, the see-through view associated with a world-locked area of ​​the user in the real-world environment;

[0027] identifying, based on the command, a first portion of a display area of ​​the artificial reality device in which the see-through view is activated, the first portion of the display area of ​​the artificial reality device corresponding to a world-locked area of ​​the real-world environment associated with the see-through view; and

[0028] A see-through view is activated on a first portion of a display area of ​​an artificial reality device, and at the same time at least a portion of a virtual reality experience is displayed on a second portion of the display area of ​​the artificial reality device, the see-through view comprising a view of a world-locked area in a real-world environment, wherein the see-through view remains fixed to the world-locked area as the artificial reality device moves.

[0029] For example, a computer-readable storage medium is provided, which stores instructions that, when executed by a computing system, cause the computing system to perform a process for providing partial perspective on an artificial reality device, the process including the method of the first aspect.

[0030] In some embodiments, activating a see-through view on a first portion of a display area of ​​the artificial reality device includes displaying a feed of a user's real-world environment captured by one or more cameras integrated with or in operable communication with the artificial reality device.

[0031] In some embodiments, the command is an audible command issued by a user of the artificial reality device, the command is collected by at least one microphone integrated with or in operable communication with the artificial reality device, and identifying a first portion of a display area of ​​the artificial reality device in which a perspective view is activated includes: based on the command, identifying a world-locked area in the real-world environment by performing voice recognition on the command and parsing the command to identify the indicated area or object.

[0032] In some embodiments, the command is a user selection of a virtual slider displayed on the artificial reality device, and the method further comprises: associating the virtual slider with a dynamic partition on the artificial reality device, the dynamic partition virtually dividing the artificial reality device into a first side and a second side;

[0033] determining a location of the dynamic partition on the artificial reality device based on the user selection; and

[0034] Based on the position of the dynamic partition, assigning a second side of the dynamic partition to a second portion of the display area of ​​the artificial reality device in which at least a portion of the virtual reality experience is displayed, and

[0035] Among them, command-based identification of the first part of the display area of ​​the artificial reality device in which the perspective view is activated includes: based on the position of the dynamic partition, assigning the first side of the dynamic partition to the first part of the display area of ​​the artificial reality device in which the perspective view is activated.

[0036] In some embodiments, the command identifies a percentage of a display area of ​​the artificial reality device in which to activate the see-through view, wherein identifying a first portion of the display area of ​​the artificial reality device in which to activate the see-through view based on the command comprises: allocating the percentage of the display area of ​​the artificial reality device to the see-through view on the first portion of the display area of ​​the artificial reality device, and wherein the process further comprises:

[0037] A second portion of the display area of ​​the artificial reality device is identified by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device, and wherein displaying at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device includes: cropping the virtual reality experience to correspond to the second portion of the display area of ​​the artificial reality device.

[0038] According to a third aspect of the present invention, there is provided a computing system medium for providing partial perspective on an artificial reality device, the computing system comprising:

[0039] one or more processors; and

[0040] One or more memories storing instructions that, when executed by one or more processors, cause the computing system to perform a process comprising:

[0041] Displaying VR experiences on artificial reality devices;

[0042] receiving a command identifying a percentage of a display area of ​​the artificial reality device in which a see-through view of a user's real-world environment is activated; based on the command, identifying a first portion of the display area of ​​the artificial reality device in which the see-through view is activated by assigning the percentage of the display area of ​​the artificial reality device to the see-through view;

[0043] identifying a second portion of the display area of ​​the artificial reality device by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience; and

[0044] A see-through view is activated on a first portion of the display area and at least a portion of a virtual reality experience is simultaneously displayed on a second portion of the display area.

[0045] For example, the computing system is provided with one or more memories, which store instructions, which, when executed by the computing system, cause the computing system to perform a process including the method of the first aspect on an artificial reality device.

[0046] In some embodiments, the see-through view includes a view of a head lock area of ​​the real-world environment that corresponds to a percentage of the display area of ​​the artificial reality device.

[0047] In some embodiments, activating a see-through view on a first portion of a display area of ​​the artificial reality device includes displaying a feed of a user's real-world environment captured by one or more cameras integrated with or in operable communication with the artificial reality device.

[0048] In some embodiments, the command is an audible command issued by a user of the artificial reality device, the command is picked up by at least one microphone integrated with or in operable communication with the artificial reality device, and identifying a first portion of a display area of ​​the artificial reality device in which the see-through view is activated includes performing voice recognition on the command to identify the percentage indicated in the command.

[0049] In some embodiments, the command is a user selection of a position on a virtual slider displayed on the artificial reality device, and identifying a first portion of the display area of ​​the artificial reality device in which the perspective view is activated includes associating the position of the virtual slider with a percentage of the display area of ​​the artificial reality device. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a block diagram illustrating an overview of a number of devices upon which some embodiments of the present technology may operate.

[0051] Figure 2Ais a line diagram illustrating a virtual reality head mounted viewer that may be used in some embodiments of the present technology.

[0052] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0053] Figure 2C is a line diagram illustrating a controller that, in some embodiments, a user may hold with one or both hands to interact with an artificial reality environment.

[0054] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0055] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0056] Figure 5 is a flow chart illustrating a process for providing partial see-through on an artificial reality (XR) device (e.g., a head mounted display (HMD)) in some embodiments of the present technology.

[0057] Fig. 6A is a conceptual diagram illustrating an example view on an artificial reality (XR) device, such as a head mounted display (HMD), without a see-through view of the real world environment.

[0058] Figure 6B is a conceptual diagram illustrating an example view on an artificial reality (XR) device, such as a head-mounted display (HMD), with partial perspective of a real-world environment.

[0059] Figure 6C is a conceptual diagram illustrating an example view on an artificial reality (XR) device (e.g., a head-mounted display (HMD)) with full perspective of a real-world environment.

[0060] Figure 7 is a conceptual diagram illustrating an example view on an artificial reality (XR) device (e.g., a head-mounted display (HMD)) with partial perspective of objects in a real-world environment.

[0061] Figure 8 is a conceptual diagram illustrating an exemplary process for performing object recognition according to some embodiments of the present technology.

[0062] The technology described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to identical or functionally similar elements. DETAILED DESCRIPTION

[0063] Various aspects of the present disclosure relate to providing partial perspective on an artificial reality (XR) device (e.g., a head-mounted display (HMD)). According to some embodiments, the XR device can display a fully immersive computer-generated virtual reality (VR) experience to a user. The XR device can receive a command to activate a perspective view on the XR device, for example, through a voice command or via a selectable element (e.g., a virtual button or slider). The XR device (or other processing component in operable communication with the XR device) can identify a first portion of the display of the XR device in which the perspective view is activated based on the command. The XR device can then display the VR experience on a second portion of the display while the perspective view is activated on the first portion of the XR device. The first portion of the XR device can display the user's real-world environment, which is captured by a camera integrated with the XR device or in operable communication with the XR device (e.g., as in augmented reality), or can be seen through the display of the XR device (e.g., as in mixed reality).

[0064] Thus, some embodiments may allow a user to turn on a selective perspective area on the XR HMD so that one or more portions of the real-world environment that physically surround the user can be seen. In some embodiments, the user may control the perspective area by specifying a percentage of the real-world environment that he wishes to see (e.g., 50%). In some embodiments, the user may control the perspective area by specifying an area of ​​the field of view of the XR HMD that she wishes to see (e.g., the top of the field of view). In some embodiments, the user may control the perspective area by specifying a physical object in the real-world environment that he wishes to see (e.g., a real-world table). Such embodiments may perform object recognition on the real-world environment based on commands from the user to identify objects in the environment and select objects for perspective.

[0065] Embodiments of the disclosed technology may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user. Artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave-like" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0066] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture multiple real-world images and then display them on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially consists of light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR; AR; MR; or any combination or hybrid thereof.

[0067] Various aspects of the present disclosure provide specific improvements in the field of artificial reality technology. For example, some embodiments allow a user to select a portion of her real-world environment to be displayed simultaneously alongside the VR experience without having to exit the VR experience or remove the XR device (e.g., XR HMD). Thus, the user can continue to participate in the VR experience while also being able to see her surroundings and possibly perform other tasks. In other words, some embodiments may allow the user to multitask while enjoying the VR experience. Some embodiments may be particularly useful for preventing the user from tripping over objects in his real-world environment (e.g., when a perspective view of the floor is activated) and / or seeing movable objects (e.g., pets) in his real-world environment, thereby allowing seamless integration between reality and virtual reality. In addition, by not requiring (and not expecting or needing) to fully render the VR experience on the XR device while activating a partial perspective, computing resources such as processing power, battery power, display resources, etc. may be saved. Therefore, these resources may be allocated to perform other basic and necessary tasks on the XR device, thereby improving processing speed and latency.

[0068] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram illustrating an overview of multiple devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that may provide a partial perspective on an artificial reality (XR) head-mounted display (HMD). In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted viewer and other processing operations are transferred to the core processing component. The following is combined with Figure 2A and Figure 2B In some implementations, the location data and environmental data may be collected solely by sensors incorporated into the head mounted device, while in other implementations, one or more of the plurality of non-head mounted device computing devices may include sensor components that may track environmental data or location data.

[0069] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among computing devices 101 to 103).

[0070] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits information to the processor 110 using a communication protocol. For example, each input device 120 may include a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0071] The processor 110 can be coupled to other hardware devices, such as by using an internal bus or an external bus or a wireless connection, such as a peripheral component interconnect (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with hardware controllers of devices such as a display 130. The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices include: a liquid crystal display (LCD) display screen; a light emitting diode (LED) display screen; a projection, holographic, or augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or card, a video chip or card, an audio chip or card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0072] In some embodiments, the computing system 100 can use input from the I / O device 140 (e.g., a camera, a depth sensor, an inertial motion unit (IMU) sensor, a GPS unit, a laser radar (LiDAR) or other time-of-flight sensor, etc.) to identify and map the user's physical environment while tracking the user's position in the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., a topology, a grid, etc.) of an area (which can be a room, a building, an outdoor space, etc.) and / or obtain a map previously generated by the computing system 100 or another computing system that has mapped the area. The SLAM system can track users in the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0073] The computing system 100 may include a communication device that can communicate wirelessly or wired with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using a transmission control protocol / internet protocol (TCP / IP). The computing system 100 can use the communication device to distribute operations among multiple network devices.

[0074] Processor 110 can access memory 150, which can be included in one of the multiple computing devices of computing system 100, or can be distributed on multiple computing devices of computing system 100 or other multiple external devices. Memory includes one or more hardware devices for volatile storage or non-volatile storage, and can include both read-only memory and writable memory. For example, memory can include one or more of the following: random access memory (RAM); various cache memories; CPU registers; read-only memory (ROM); and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagation signal separated from the underlying hardware; therefore, the memory is non-transient. Memory 150 can include program memory 160, which stores programs and software, such as operating system 162, partial perspective system 164, and other applications 166. The memory 150 may also include a data memory 170, which may include, for example, virtual reality (VR) rendering data, augmented reality (AR) rendering data, real world environment rendering data, command data, natural language processing data, object recognition data, configuration data, settings, user options or preferences, etc., which may be provided to the program memory 160 or any element of the computing system 100.

[0075] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the present technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, or a distributed computing environment including any of the above systems or devices, etc.

[0076] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a band 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of locators 225, and one or more computing units 230. Position sensors 220, IMU 215, and computing units 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensors 220, and locators 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit an infrared beam that generates a light spot on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects in the environment surrounding the HMD 200.

[0077] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each user's eye). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0078] In some embodiments, the HMD 200 can be coupled to a core processing component (e.g., a personal computer (PC) (not shown)) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., through light emitted from the HMD 200), which the PC can use in combination with the output from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0079] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0080] The projector can be coupled to the perspective display 258, for example, via optical elements, to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc., for guiding the light from the projector to the user's eyes. The image data from the core processing component 254 can be transmitted to the HMD 252 via the link 256. The controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes as output light via the optical elements. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present virtual objects as follows: these virtual objects look like they exist in the real world.

[0081] Similar to the HMD 200, the HMD system 250 may also include a motion and position tracking unit, camera, light source, etc., which allows the HMD system 250 to track itself, for example, with 3DoF or 6DoF, track various parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when the HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0082] Figure 2C Controllers 270 (including controllers 276A and 276B) are shown that in some embodiments a user can hold with one or both hands to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. Controller 270 can communicate with the HMD directly or via an external device (e.g., core processing component 254). The controller can have its own IMU unit, position sensor, and / or can emit additional light points. HMD 200 or HMD 250, external sensors, or sensors in the controller can track these controller light points (e.g., to track the controller with 3DoF or 6DoF) to determine the controller position and / or orientation. The computing unit 230 or core processing component 254 in HMD 200 can use this tracking, combined with the IMU output and the position output, to monitor the user's hand position and movement. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joystick 274A and joystick 274B) that a user may actuate to provide input and interact with objects.

[0083] In various embodiments, HMD 200 or HMD 250 may also include a number of additional subsystems to monitor user interactions and indications of intent, such as an eye tracking unit, an audio system, various network components, etc. For example, in some embodiments, instead of or in addition to the controller, one or more cameras included in HMD 200 or HMD 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both of the user's eyes, and HMD 200 or HMD 250 can use eye-facing cameras to capture reflections of light to determine eye position (e.g., based on a set of reflections around the user's cornea) to model the user's eyes and determine gaze direction.

[0084] Figure 3305B may be a HMD 200 or a HMD 250. The client computing device 305 may be a logical connection to one or more remote computers (e.g., a server computing device) through a network 330 to operate in a network environment.

[0085] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include a computing system (e.g., computing system 100). Although each server computing device 310 and 320 is logically shown as a single server, multiple server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0086] Client computing device 305 and server computing devices 310 and 320 can all act as servers or clients to one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can be connected to corresponding databases 325A to 325C respectively. As discussed above, each server 310 or 320 can correspond to a group of servers, and each server in these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as single units, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or geographically different physical locations.

[0087] Network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 may be the Internet or some other public or private network. Client computing device 305 may be connected to network 330 through a network interface, for example, via wired or wireless communications. Although the connections between server 310 and server 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including network 330 or a separate public or private network.

[0088] Figure 4 4 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. The plurality of components 400 can be included in one device of the computing system 100, or can be distributed among multiple devices of the computing system 100. These components 400 include a plurality of hardware 410, middleware 420, and a plurality of dedicated components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.), and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider or other network storage device accessible via one or more communication networks (e.g., located in the storage device 315 or 325). In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0089] Middleware 420 may include components that coordinate resources between hardware 410 and dedicated components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

[0090] The specialized components 430 may include software or hardware configured to perform operations that provide partial perspective on an artificial reality (XR) device (e.g., a head mounted display (HMD)). The plurality of specialized components 430 may include: a VR experience rendering module 434; a command receiving module 436; a command interpreting module 438; a partial perspective activation module 440; and components and application programming interfaces (APIs) that may be used to provide a user interface, transmit data, and control the plurality of specialized components, such as interface 432. In some implementations, the component 400 may be located in a computing system distributed across a plurality of computing devices, or may be an interface to a server-based application that executes one or more of the plurality of specialized components 430. Although the plurality of specialized components 430 are depicted as separate components, the specialized components 430 may be logical functions or other non-physically distinct functions, and / or may be sub-modules or code blocks of one or more applications.

[0091] The VR experience rendering module 434 can facilitate displaying a virtual reality (VR) experience on an artificial reality (XR) device (e.g., a head-mounted display (HMD)). In some embodiments, the VR experience rendering module 434 can display the VR experience on a display included in the input / output device 416. In some embodiments, the VR experience rendering module 434 can facilitate displaying the VR experience on the XR device by providing data required to render and / or display the VR experience on the XR device, while the VR experience rendering module 434 itself does not display the VR experience. The VR experience can be any fully immersive VR application, game, virtual world, etc. that can be rendered on an XR device to display a computer-generated virtual environment including virtual objects. This article refers to Figure 5 Block 502 of the accompanying drawings describes more details about displaying the VR experience to the user on the XR device.

[0092] The command receiving module 436 may receive a command to activate a perspective view on the XR device. The command receiving module 436 may receive commands from one or more devices included in the input / output device 416. For example, the command receiving module 436 may receive an audible command from a microphone included in the input / output device 416. In another example, the command receiving module 436 may receive a command as a physical selection of a physical button included in the input / output device 416 (e.g., on the XR device, via a controller in operable communication with the XR device, etc.), the button being associated with activating and / or deactivating a perspective view on the XR device. In yet another example, the command receiving module 436 may receive a command as a gesture captured by a camera included in the input / output device 416, the gesture being associated with activating and / or deactivating a perspective view on the XR device, the gesture being, for example, a pointing gesture pointing to a virtual button or a virtual slider, or a circle gesture in an area visible to the XR device. References herein Figure 5 Block 504 of the flowchart will now describe further details regarding receiving a command to activate a see-through view on the XR device.

[0093] The command interpretation module 438 can identify the portion of the XR device in which the perspective view is activated based on the command. In some embodiments, when the command is audible, the command interpretation module 438 can apply speech recognition and / or natural language processing techniques to parse the command to determine the portion of the XR device in which the perspective view is activated, such as a specific area ("the lower portion of the display"), a specific percentage ("25% of the display"), or a specific object ("the table in my room"). In some embodiments, when the command is received via a virtual slider, the command interpretation module 438 can identify the corresponding percentage portion of the XR device in which the perspective view is displayed. In some embodiments, when the command is a gesture in front of the XR device, the command interpretation module 438 can identify the portion of the real-world environment that corresponds to the location where the gesture was made relative to the XR device. Reference herein Figure 5 Block 506 of the disclosure describes further details regarding identifying a portion of the XR device in which to activate a see-through view based on the command.

[0094] The partial perspective activation module 440 can activate a perspective view on a portion of the XR device identified by the command interpretation module 438. In some embodiments, the partial perspective activation module 440 can activate the perspective view by capturing an image of the real-world environment from a camera integrated with the XR device that is away from the user and pointed toward the real-world environment. The partial perspective activation module 440 can simultaneously continue to display the VR experience on the remaining portion of the XR device that has not activated the perspective. Therefore, in some embodiments, the partial perspective activation module 440 can provide an augmented reality (AR) experience. This document refers to Figure 5 Block 508 of the XR device may be used to describe further details regarding activating a see-through view on a portion of the XR device while simultaneously displaying a VR experience on the remainder of the XR device.

[0095] Those skilled in the art will recognize that the above Figures 1 to 4 And the components shown in each of the multiple flow charts discussed below can be changed in various ways. For example, the order of the logic can be rearranged, the sub-steps can be performed in parallel, the shown logic can be omitted, other logic can be included, etc. In some embodiments, one or more of the multiple components described above can perform one or more of the processes described below.

[0096] Figure 5 1 is a flow chart illustrating a process 500 for providing partial see-through on an artificial reality (XR) device (e.g., a head mounted display (HMD)) in some embodiments. In some embodiments, process 500 can be performed in response to a user request to activate a see-through view on a portion of the XR device. In some embodiments, some or all of process 500 can be performed by the XR device. In some embodiments, the XR device can be, for example, Figure 2A 2V and / or the XR HMD 252 of FIG. 2V. In some embodiments, some of process 500 may be performed by one or more other components of the XR system other than the XR device (e.g., a processing component in operable communication with the XR device). In some embodiments, some of process 500 may be performed by a server located remotely from the XR device. In some embodiments, process 500 may be performed by a server located remotely from the XR device. Figure 1 In some embodiments, process 500 may be performed by Figure 4 The dedicated component 430 in is executed.

[0097] At block 502, process 500 may display a virtual reality (VR) experience to a user on an XR device. The VR experience may be any fully immersive experience in which the user's visual input is completely controlled by the XR device. For example, the VR experience may be a VR application, VR game, virtual world, virtual environment including virtual objects, etc., created and rendered entirely by the XR device, a processing component in operable communication with the XR device, and / or a platform or other computing system hosting the VR experience.

[0098] At box 504, process 500 can receive a command to activate a see-through view on the XR device. In some embodiments, the see-through view can be associated with a world-locked area in the user's real-world environment. The world-locked area can include physical objects that do not move in response to movement of the XR device, but in some embodiments, physical objects (e.g., people, pets, balls, etc.) within the world-locked area can move independently without regard to the movement of the XR device. In some embodiments, physical objects in the world-locked area can be static and remain in the same position and orientation in the real-world environment. In some embodiments, the command can describe and / or identify the world-locked area in the real-world environment.

[0099] In some embodiments, as described herein with reference to FIG. 6A to FIG. 6C As further described, the command may be a user selection of a virtual slider displayed on the XR device. In some embodiments, process 500 may receive the user selection by selecting a physical button on the XR device or other XR device (e.g., a controller) in operable communication with the XR device. In some embodiments, process 500 may receive the user selection by a gesture detected by the XR device or other XR device (e.g., a controller having a sensor capable of detecting a gesture) in operable communication with the XR device (e.g., via a camera or other image acquisition device).

[0100] In some embodiments where the command is a user selection of a virtual slider, process 500 may also associate the virtual slider with a dynamic partition on the XR device. The dynamic partition may virtually divide the display of the XR device into a first portion and a second portion (e.g., a first side and a second side). Process 500 may determine the location of the dynamic partition on the XR device based on the user selection of the virtual slider. For example, if the user slides the virtual slider to half of its capacity, process 500 may determine that the dynamic partition should divide the display of the XR device into two halves. It is contemplated that the dynamic partition may divide the display of the XR device in any direction and / or in any shape (e.g., horizontally, vertically, diagonally, in a curved shape or any other specific shape within the display, etc.).

[0101] In some embodiments, the command may be audible and may be picked up by a microphone in operable communication with and / or integrated with the XR device. In such embodiments, process 500 may perform speech recognition and / or natural language processing techniques to parse and recognize the command. In some embodiments, the command may identify the first portion of the XR device as a percentage of the XR device (e.g., “activate perspective for the lower 50% of the display”). In some embodiments, the command may identify the first portion of the artificial reality head-mounted display as an object in the real-world environment (e.g., “show me my desk on the display” or “activate perspective of the door in my room”).

[0102] In some embodiments, process 500 may automatically generate commands. For example, a user of an XR device may activate a setting (and / or process 500 may automatically select the setting) for activating a see-through view of one or more specific objects that would otherwise come into the user's field of view in the real-world environment from within the VR experience. For example, process 500 may perform object recognition and / or object detection to identify objects that the user may want to see in see-through (e.g., physical objects on the floor that might trip the user). In another example, process 500 may perform object recognition, object detection, and / or motion detection to identify moving objects that the user may want to see in see-through (e.g., people or animals that may physically obstruct the user or may want to interact with the user while the user is in the VR experience).

[0103] At block 506, process 500 may identify a first portion of the XR device in which to activate the see-through view based on the command. In some implementations where the see-through view is associated with a world-locked area in the user's real-world environment, the first portion of the XR device may correspond to the world-locked area associated with the see-through view. In some implementations, the command received at block 504 may identify the world-locked area as an object in the real-world environment. In some implementations, the object may be a moving object (e.g., a person, a pet, a ball, a vacuum robot, etc.).

[0104] In some embodiments, process 500 can identify the object specified by the command and / or the location of the object specified by the command by performing object recognition and / or object detection in the real-world environment. For example, process 500 can use a machine learning model and / or a deep learning model to identify physical objects in the real-world environment. Process 500 can train a model based on known objects that are typically present in the user's real-world environment (e.g., windows, doors, furniture, animals, trees, etc.). In some embodiments, process 500 can identify features in known objects (e.g., corresponding to edges, corners, and other unique and / or identifiable features). Process 500 can then receive an input image captured by a camera integrated with the XR device, extract features of an object present in the image, and compare the extracted features to the identified features in the known object to identify the object with a certain degree of certainty. References herein Figure 8 to describe more details about object recognition and machine learning models.

[0105] In some embodiments, process 500 can identify an object specified by a command and / or the location of an object specified by a command by accessing scene data, spatial anchor data, or at least one of both scene data and spatial anchor data associated with the user's real-world environment. For example, as the XR device (and / or other XR devices) moves in the real-world environment, they can scan those locations and define certain anchor points (e.g., on surfaces, edges, corners, doorways, etc.). These spatial anchors can specify a map of the world around the XR device. These spatial anchors can then be stored to a local or centralized mapping service. When the same XR device or another XR device is in a similar position in the real-world environment, it can identify the spatial anchors and can also retrieve the spatial anchors for the area from a local or map building service. By aligning one or more spatial anchors detected by the XR device with corresponding spatial anchors from a map building service, the XR device can identify itself within a map defined by a larger set of anchor points from a map building service. Using this map, the XR device can track itself in the real-world environment. The XR device (or another XR device that has access to the real-world environment) can scan the area to specify object locations and types (e.g., tables, chairs, walls, floors, ceilings, doorways, etc.) within a defined scene dictionary. This scene recognition can be performed, for example, by a user manually identifying locations with corresponding object types or using a camera to capture images of physical objects in the scene and using computer vision techniques to identify the physical objects as object types. In some embodiments, the object type can be stored in association with one or more of the multiple spatial anchors defined for the area, which can be used by process 500 to identify objects and / or locations of objects in the real-world environment. In some embodiments, process 500 can access such scene data from a central system, a local storage device, and / or another XR device that has previously generated scene data.

[0106] In some implementations where the command received at block 504 identifies a percentage of the XR device in which to activate the see-through view, process 500 can allocate the percentage of the XR device to the see-through view on the first portion of the XR device. Process 500 can also identify a second portion of the XR device in which to display at least a portion of the VR experience by allocating a remaining percentage of the XR device to at least a portion of the VR experience on the second portion of the XR device.

[0107] In some implementations where the command received at block 504 is a user selection of a virtual slider displayed on the XR device, process 500 may identify a first portion of the XR device in which the see-through view is activated by assigning a first side of the dynamic partition to a first portion of the XR device in which the see-through view is activated based on the position of the dynamic partition. In some implementations, process 500 may identify the first portion of the XR device in which the see-through view is activated by associating the position of the virtual slider with a percentage of the XR device. Process 500 may also assign a second side of the dynamic partition to a second portion of the XR device in which at least a portion of the XR experience is displayed based on the position of the dynamic partition (e.g., a remaining percentage of the XR device).

[0108] At box 508, process 500 can activate a see-through view on a first portion of the artificial reality head-mounted display and simultaneously display a VR experience on a second portion of the artificial reality head-mounted display. The first portion of the XR device can display the user's real-world environment corresponding to the portion identified by the command. In some embodiments, process 500 can display a feed of the user's real-world environment captured by a camera in operable communication with the XR device and / or integrated with the XR device. Thus, in some embodiments, the XR device can partially display an augmented reality (AR) experience with see-through on the XR device and display a VR experience on the remaining area of ​​the XR device.

[0109] In some embodiments, such as when the perspective view is for a percentage of the XR device (e.g., the upper 50% of the display), the perspective view may include a view of a head-locked area in the real-world environment (i.e., a view of the real-world environment that changes as the XR device moves over the user's head). In some embodiments, the perspective view may include a view of a world-locked area in the real-world environment. In some embodiments where the perspective view is associated with the world-locked area, the perspective view may remain fixed to the world-locked area as the XR device moves. For example, in some embodiments, process 500 may also fix the perspective view to the world-locked area in the real-world environment by tracking the movement of the XR device relative to the world-locked area. In some embodiments, process 500 may also update the first portion of the XR device through which the perspective view is activated based on the tracked movement. In some embodiments, perspective may be activated for moving objects in the real-world environment. In such embodiments, process 500 may also fix the perspective view to the world-locked area by tracking the movement of the moving object relative to the XR device.

[0110] In some implementations where the command received at block 504 identifies a percentage of the XR device in which to activate the see-through view, process 500 can crop the VR experience to correspond to the second portion of the XR device. For example, if the command is to activate the see-through view on the left 25% of the display of the XR device, process 500 can crop the left 25% of the VR experience and display the remaining 75% of the VR experience on the right 75% of the display. In some implementations, process 500 can modify the size of the VR experience, reduce the VR experience, and / or change the aspect ratio of the VR experience to fit the second portion of the XR device.

[0111] Fig. 6A is a conceptual diagram illustrating an example view 600A on an artificial reality (XR) device (e.g., a head mounted display (HMD)) without a see-through view of a real world environment. View 600A includes a VR experience 602. View 600A also includes a virtual menu 604 with a virtual slider 606, which in view 600A is set to 0% see-through. Thus, view 600A includes VR experience 602 displayed on 100% of the XR device.

[0112] Figure 6B is a conceptual diagram illustrating an example view 600B on an artificial reality (XR) device (e.g., a head mounted display (HMD)) with partial perspective of a real world environment 608. View 600A includes a virtual menu 604 with a virtual slider 606, which in view 600B is set to 60% perspective. Thus, view 600B includes a VR experience 602 displayed on 40% of the XR device, and a real world environment 608 displayed on 60% of the XR device. In some implementations, the real world environment 608 can be captured by one or more cameras that are integrated with the XR device or in operable communication with the XR device.

[0113] Figure 6C is a conceptual diagram illustrating an example view 600C on an artificial reality (XR) device (e.g., a head mounted display (HMD)) with full perspective of a real world environment 608. View 600A includes a virtual menu 604 with a virtual slider 606, which in view 600C is set to 100% perspective. Thus, view 600C includes the real world environment 608 displayed on 100% of the XR device. In some implementations, the real world environment 608 can be captured by one or more cameras that are integrated with the XR device or in operable communication with the XR device.

[0114] Figure 7is a conceptual diagram illustrating an example view 700 on an artificial reality (XR) device (e.g., a head mounted display (HMD)) with partial perspective of an object 708 in a real-world environment. View 700 may include VR experience 702; in this case, VR experience 702 is a virtual car show. View 700 may also include perspective of object 708 in the user's real-world environment (as indicated by dashed lines around object 708 indicating a perspective area); in this case, object 708 is a real-world pool table. Some embodiments may activate perspective for object 708 in response to a user command (e.g., an audible command of “turn on perspective for the pool table in my room”). Some embodiments may identify object 708 by performing object recognition on captured images of the user's real-world environment and performing speech recognition and natural language processing techniques to link the command to object 708.

[0115] Figure 8 800 for performing object recognition in accordance with some embodiments of the present technology. In some embodiments, process 800 can be executed to identify specific objects in a user's real-world environment for which the user requests to partially activate perspective. The XR device 802 can capture an object image 804; in this case, the object image 804 is an image that includes a doorway. For example, one or more image capture devices (e.g., one or more cameras) that are integrated with the XR device 802 and / or in operable communication with the XR device 802 can capture an object image 804 in the field of view of the user of the XR device 802 (and in some embodiments, capture images of other real-world objects). For example, in some embodiments, the XR device 802 can capture an object image 804 that includes a doorway, a plant, a window, etc., and execute process 800 to identify a doorway from the object image 804.

[0116] Some embodiments may also receive contextual factors surrounding the object image 804, such as the location where the image was captured (e.g., a living room, a movie theater, a restaurant, an office, etc.), the time when the image was captured (e.g., morning, noon, evening, late night, holidays, weekends, etc.), audio present when the image was captured (e.g., a user discussing or announcing his surroundings, a conversation, etc.), what the user was doing when the image was captured (e.g., watching a movie, working on a computer, etc.), and / or any other contextual data that may be related to the identified object type, such as environmental factors (e.g., temperature, weather, etc.).

[0117] The XR device 802 may be, for example, an HMD, such as any of the following: Figure 2A HMD 200; Figure 2BThe object image 804 may be fed to a feature extractor 806, which may identify relevant features 808 in the object image 804. These relevant features 808 may correspond to, for example, edges, corners, shapes, curvatures, colors, or textures, or any combination thereof. The features 808 may be fed into a machine learning model 814.

[0118] The machine learning model 814 can obtain training data 810, which includes labeled object types with identified features; for example, the labeled object types are windows 812A, doorways 812B, and floors 812C. Some embodiments can train the machine learning model 814 using a collection of images with known object types and applying a feature extraction algorithm (e.g., via feature extractor 806) to manually extract features of the images, such as edge features or corner features that can be used to distinguish object types. Some embodiments can train the machine learning model 814 by analyzing a large number of training images with known object types and / or contextual factors and automatically learning inherent features of object types. Some embodiments can map the features of the training images into a classification space that identifies candidate object types associated with the features to create the training data 810. Some embodiments can also include past feedback on whether previous object type predictions within the training data 810 were correct. Some embodiments may repeat the training phase of the machine learning model 814 until a suitable level of accuracy is achieved (e.g., a level of accuracy identified by applying a loss function), such as when a sufficient amount of training data 810 has been processed and the predictions made by the machine learning model 814 do not deviate significantly from actual outcomes.

[0119] Once trained, the machine learning model 814 can use the features 808, the training data 810, and any contextual factors in some embodiments to generate an output. In some embodiments, the machine learning model 814 can use the training data 810 to map the features 808 to data points of an output vector in a classification space. The machine learning model 814 can use the output vector to compare the features 808 with the training data 810 to generate a match score between the features 808 and the training data 810 in the classification space by calculating the distance between the output vector and the candidate object types in the training data 810. The match score can be any numeric or text value or indicator, such as a statistical number or percentage.

[0120] In this case, machine learning model 814 can determine that the type of object in object image 804 has the highest match score with doorway 812B. Thus, in some implementations, feature 808 need not match all features of doorway 812B; however, machine learning model 814 can determine that feature 808 identified in object image 804 is more similar to doorway 812B than window 812A or floor 812C. In some implementations, machine learning model 814 can determine that the match score of feature 808 with doorway 812B is above a predefined threshold.

[0121] The machine learning model 814 may output object data 816. In some embodiments, the machine learning model 814 may output the object data 816 to the XR device 802 to activate partial perspective in the doorway at block 818. Some embodiments may receive feedback regarding the predicted object type, e.g., whether partial perspective was activated for the correct object. In some embodiments, the feedback may be explicit, e.g., a user audibly announcing that the predicted object type is correct or incorrect, a user audibly announcing that partial perspective was activated for an incorrect object, a user selecting a virtual button indicating partial perspective was activated for the correct object or the incorrect object, etc. In some embodiments, the feedback may be implicit, e.g., a user does not correct the partial perspective activation.

[0122] Some embodiments may update the machine learning model 814 and / or the training data 810 based on any feedback. For example, some embodiments may use metrics to evaluate the machine learning model 814 based on the feedback data. The metrics may include accuracy, precision, F1 score, mean square error, etc. If necessary, some embodiments may feed these metrics back into the machine learning model 814 to refine and update the model 814. In another example, some embodiments may use the feedback data to identify whether the predicted object type is correct or incorrect (and, if the predicted object type is incorrect, what the correct object type is if the correct object type is available), and use this information as a comparison factor to update the model and / or classification space. Some embodiments may give current training data or updated training data a greater weight than initial or past training data 810 because the later training data is considered more relevant and / or more accurate. Although shown as a single process 800, it is contemplated that the process 800 may be performed multiple times and / or repeatedly, either continuously or simultaneously, as additional object images of a particular scene are received.

[0123] Thus, some embodiments of a partial perspective system may include a machine learning component, such as a neural network, that is trained using various data, including images of known object types, past object types seen by the user or similar users, contextual factors, and / or whether the user identified the predicted object type as correct or incorrect. Some embodiments may feed input data including images of objects and / or contextual factors into the trained machine learning component, and based on the output, a predicted object type may be generated. Some embodiments provide the predicted object type to the user via a display on the XR device by activating partial perspective of the predicted object. Some embodiments receive feedback about the predicted object type to further enhance the trained model.

[0124] As used herein, "machine learning model" refers to the concept of using training data to train to predict or provide probability for new data items (whether or not the new data items are included in the training data). For example, the training data for supervised learning may include items with various parameters and specified classifications. The new data item may have parameters that the model can use to assign classifications to the new data item. As another example, the model may be a probability distribution obtained from the analysis of the training data, such as the possibility of n-grams appearing in the given language based on the analysis of a large corpus from a given language. Examples of models include: neural networks, support vector machines, decision trees, Parzen windows, Bayes, clustering, reinforcement learning, probability distributions, decision trees, decision tree forests, and others. Models may be configured for various situations, data types, sources, and output formats.

[0125] In some embodiments, the trained model can be a neural network with multiple input nodes that receive input data including an image of an object and / or any contextual factors. The input node can correspond to a function that receives an input and produces a result. These results can be provided to one or more levels of intermediate nodes, each of which produces further results based on a combination of node results at a lower level. Before the result is passed to the next layer of nodes, a weighting factor can be applied to the output of each node. In the last layer ("output layer"), one or more nodes can generate a value that classifies the input, which, once the model is trained, can be used to predict the type of object in the image. In some embodiments, this neural network, known as a deep neural network, can have multiple layers of intermediate nodes with different configurations, can be a combination of models that receive inputs from other parts of the deep neural network and / or different parts of the input, or can be convolutional or recursive-partially using the output from the previous iteration of the application model as a further input to produce the result of the current input.

[0126] A machine learning model can be trained using supervised learning, where the training data includes images of known object types and / or any contextual factors as input and a desired output (e.g., a prediction of the object type). A current image of an object can be provided to the model. The output from the model can be compared to the desired output for the object type, and based on the comparison, the model can be modified, for example, by changing the weights between nodes of a neural network or the parameters of a function used at each node in the neural network (e.g., applying a loss function). After applying each of the multiple factors in the training data and modifying the model in this manner, the model can be trained to evaluate new input data.

[0127] Some embodiments of the partial perspective system may include a deep learning component. The "deep learning model" used herein with respect to object recognition refers to a concept that is trained to learn by example to perform classification directly from an image. A deep learning model is trained by using a large set of labeled data and applying a neural network including many layers as described above. With respect to object recognition from an image, in some embodiments, the deep learning model may be a convolutional neural network (CNN) for automatically learning the inherent features of an object to recognize the object. For example, the deep learning model may be a regional CNN (R-CNN), a fast R-CNN (Fast R-CNN), or a faster RCNN (Faster-RCNN). In some embodiments, other object recognition methods can be used to perform object recognition, such as template matching, image segmentation and blob analysis, edge matching, divide-and-conquer search, grayscale matching, gradient matching, pose clustering, geometric hashing, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), region-based fully convolutional network (R-FCN), single shot detector (SSD), spatial pyramid pooling (SPP-net), etc.

[0128] References in this specification to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearance of these phrases in different places in this specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0129] As used herein, being higher than a threshold value means that the value of the compared item is higher than other specified values, the compared item is one of the items with the maximum value in a certain specified number of items, or the value of the compared item is within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of the compared item is lower than other specified values, the compared item is one of the items with the minimum value in a certain specified number of items, or the value of the compared item is within a specified minimum percentage value. As used herein, being within a threshold value means that the value of the compared item is between two other specified values, the compared item is one of the items of a specified intermediate number, or the value of the compared item is within a specified intermediate percentage range. Relative terms, such as high or unimportant, when not otherwise defined, can be understood as assigning a value and determining the comparison of the value with a set threshold value. For example, the phrase "selecting a fast connection" can be understood to mean selecting a connection with a value corresponding to a connection speed that is assigned to it that is higher than a threshold value.

[0130] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as A and A; B, B and C; A, A, B, C and C; etc.

[0131] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. For illustrative purposes, specific embodiments and implementations are described herein, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except as limited by the appended claims.

[0132] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for providing partial perspective on an artificial reality device, the method comprising: displaying a virtual reality experience to a user on the artificial reality device; receiving a command to activate a see-through view on the artificial reality device, the see-through view being associated with a world-locked area in the user's real-world environment; identifying, based on the command, a first portion of a display area of ​​the artificial reality device in which the see-through view is activated, the first portion of the display area of ​​the artificial reality device corresponding to the world-locked area of ​​the real-world environment associated with the see-through view; activating the see-through view on the first portion of the display area of ​​the artificial reality device and simultaneously displaying at least a portion of the virtual reality experience on a second portion of the display area of ​​the artificial reality device, the see-through view comprising a view of the world-locked area in the real-world environment; as well as The perspective view is fixed to the world-locked area in the real-world environment by tracking movement of the artificial reality device relative to the world-locked area, and the first portion of the display area of ​​the artificial reality device through which the perspective view is activated is updated based on the tracked movement.

2. The method according to claim 1, wherein: Activating the see-through view on the first portion of the display area of ​​the artificial reality device includes displaying a feed of the real-world environment of the user, the feed being captured by one or more cameras in operable communication with the artificial reality device.

3. The method according to claim 1 or 2, in, The command is an audible command issued by the user of the artificial reality device, wherein the command is collected by at least one microphone integrated with the artificial reality device or in operable communication with the artificial reality device, and Wherein, identifying the first portion of the display area of ​​the artificial reality device in which the perspective view is activated includes: based on the command, identifying the world locked area in the real-world environment by performing voice recognition on the command and parsing the command to identify the indicated area or object.

4. A method according to any preceding claim, in, The command is a user selection of a virtual slider displayed on the artificial reality device, Wherein, the method further comprises: associating the virtual slider with a dynamic partition on the artificial reality device, the dynamic partition virtually dividing the artificial reality device into a first side and a second side; determining a position of the dynamic partition on the artificial reality device based on the user selection; and Based on the position of the dynamic partition, assigning a second side of the dynamic partition to the second portion of the display area of ​​the artificial reality device in which at least a portion of the virtual reality experience is displayed, and Wherein, identifying the first part of the display area of ​​the artificial reality device in which the perspective view is activated based on the command includes: assigning a first side of the dynamic partition to the first part of the display area of ​​the artificial reality device in which the perspective view is activated based on the position of the dynamic partition.

5. A method according to any preceding claim, in, The command identifies a percentage of the display area of ​​the artificial reality device in which the see-through view is activated, wherein identifying, based on the command, the first portion of the display area of ​​the artificial reality device in which the see-through view is activated comprises: allocating the percentage of the display area of ​​the artificial reality device to the see-through view on the first portion of the display area of ​​the artificial reality device, and Wherein, the method further comprises: identifying the second portion of the display area of ​​the artificial reality device by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device, and Wherein, displaying at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device includes: cropping the virtual reality experience to correspond to the second portion of the display area of ​​the artificial reality device.

6. A method according to any preceding claim, in, The command identifies a percentage of the display area of ​​the artificial reality device in which the see-through view is activated, wherein identifying, based on the command, the first portion of the display area of ​​the artificial reality device in which the see-through view is activated comprises: allocating the percentage of the display area of ​​the artificial reality device to the see-through view on the first portion of the display area of ​​the artificial reality device, wherein the method further comprises: identifying the second portion of the display area of ​​the artificial reality device by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device, and Wherein, displaying at least a portion of the virtual reality experience on the second portion of the display area of ​​the artificial reality device includes: adjusting the size of the virtual reality experience to correspond to the second portion of the display area of ​​the artificial reality device.

7. A method according to any preceding claim, wherein: The command identifies the world lock region as an object in the real-world environment.

8. The method according to claim 7, wherein: Identifying the first portion of the display area of ​​the artificial reality device in which the perspective view is activated based on the command includes: identifying the object, the location of the object, or at least one of the object and the location of the object by facilitating performing object recognition, object detection, or at least one of the object recognition and the object detection in the real-world environment.

9. The method according to claim 7 or 8, wherein: Identifying the first portion of the display area of ​​the artificial reality device in which the perspective view is activated based on the command includes: identifying the object, the position of the object, or at least one of the object and the position of the object by accessing scene data, spatial anchor data, or at least one of the scene data and the spatial anchor data associated with the real-world environment.

10. The method according to one of claims 7 to 9, in, The object is a moving object, and Wherein, the perspective view is also fixed to the world locked area in the real world environment by tracking the movement of the moving object relative to the artificial reality device.

11. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for providing partial perspective on an artificial reality device, the process comprising: displaying a virtual reality experience to a user on the artificial reality device; receiving a command to activate a see-through view on the artificial reality device, the see-through view being associated with a world-locked area in the user's real-world environment; identifying, based on the command, a first portion of a display area of ​​the artificial reality device in which the see-through view is activated, the first portion of the display area of ​​the artificial reality device corresponding to the world-locked area of ​​the real-world environment associated with the see-through view; as well as activating the see-through view on the first portion of the display area of ​​the artificial reality device and simultaneously displaying at least a portion of the virtual reality experience on a second portion of the display area of ​​the artificial reality device, the see-through view comprising a view of the world-locked area in the real-world environment; Wherein, when the artificial reality device moves, the perspective view remains fixed to the world-locked area.

12. A computer-readable storage medium according to claim 11, wherein the computer-readable storage medium stores instructions that, when executed by a computing system, cause the computing system to perform a process on an artificial reality device, wherein the process includes a method according to any one of claims 2 to 10.

13. A computing system for providing partial perspective on an artificial reality device, the computing system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: displaying a virtual reality experience on the artificial reality device; receiving a command identifying a percentage of a display area of ​​the artificial reality device in which to activate a see-through view of a user's real-world environment; identifying, based on the command, a first portion of the display area of ​​the artificial reality device in which the see-through view is activated by assigning the percentage of the display area of ​​the artificial reality device to the see-through view; identifying a second portion of the display area of ​​the artificial reality device by allocating a remaining percentage of the display area of ​​the artificial reality device to at least a portion of the virtual reality experience; and The see-through view is activated on the first portion of the display area, and at least a portion of the virtual reality experience is simultaneously displayed on the second portion of the display area.

14. The computing system of claim 13, wherein: The instructions, when executed, cause the computing system to perform a process on an artificial reality device, the process comprising the method according to any one of claims 2 to 10.

Images