Head-mounted display with pass-through imaging

Abstract

To provide a system and method for ensuring a field-of-view in real world for a user wearing a head-mounted display and having an immersive experience without taking off the head-mounted display.SOLUTION: Provided is a method for presenting pass-through images using a head-mounted display (HMD). A remote computing resource transmits to the HMD a request to define an area within an environment to request that a user define an area within the environment, transmits a request to a tracking system to track the HMD within the environment and to request the user to wear the HMD and define the area, receives first image data from the tracking system, determines characteristics of the area, receives second image data from the HMD, generates a 3D model or mesh of the environment, and transmits the 3D model to the HMD and / or the tracking system.SELECTED DRAWING: Figure 5

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This is a PCT application that claims priority to U.S. Provisional Patent Application No. 16 / 848,332, filed April 14, 2020, which claims priority to U.S. Provisional Patent Application No. 62 / 837,668, filed April 23, 2019. Applications Nos. 16 / 848,332 and 62 / 837,668 are incorporated herein by reference in their entireties. [Technical field]

[0002] [Background technology] Head-mounted displays are used in a variety of applications, including engineering, medical, military, and video games. In some cases, a head-mounted display may present information or images to a user as part of a virtual reality or augmented reality environment. For example, a user may wear a head-mounted display while playing a video game to immerse the user in a virtual environment. A head-mounted display provides an immersive experience, but blocks the view of the physical or real world. As a result, a user may find it difficult to pick up objects (e.g., a controller) and / or recognize other individuals in the real world. In addition, a user may not notice physical boundaries in the real world (e.g., walls). Removing the head-mounted display may allow the user to see, but constantly taking it off and putting it back on may be cumbersome, may require the user to reorient themselves between the virtual environment and the real world, and / or may otherwise distract from the virtual reality experience. [Brief description of the drawings]

[0003] The detailed description will now be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and in which the same or similar reference numbers in different drawings indicate similar or identical items.

[0004] [Figure 1] 1 illustrates a user wearing an exemplary head mounted display in an exemplary environment, according to an exemplary embodiment of the present disclosure.

[0005] [Diagram 2] FIG. 2 is a perspective view of the head mounted display of FIG. 1 according to one embodiment of the present disclosure.

[0006] [Diagram 3] 2 illustrates a user wearing the head mounted display of FIG. 1 and the head mounted display presenting a pass-through image, according to one embodiment of the present disclosure.

[0007] [Figure 4] 2 illustrates a user wearing the head mounted display of FIG. 1 and the head mounted display presenting a pass-through image, according to one embodiment of the present disclosure.

[0008] [Diagram 5] 10A-10C illustrate an exemplary process for presenting a pass-through image using the head-mounted display of FIG. 1 according to one embodiment of the present disclosure.

[0009] [Figure 6] 10A-10C illustrate an exemplary process for presenting a pass-through image using the head-mounted display of FIG. 1 according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A head mounted display is worn by a user to view and interact with content in a virtual reality environment. To provide an immersive experience, the head mounted display may cover most or even all of the user's field of view. Thus, the head mounted display may block the user's vision from the real world, which may result in the user tripping over objects, colliding with furniture, or being unable to recognize individuals in the real world. By removing the head mounted display, the user can see the real world, but the user must reorient himself or herself between the virtual reality environment and the real world, respectively. Some head mounted displays may, for example, include a visor that opens to allow the user to peer into the real world. However, such a solution disrupts the user's immersion in the virtual reality environment.

[0011] In an attempt to overcome these deficiencies, some head-mounted displays may enable pass-through image processing that allows a respective user to view the real world without removing the user's respective head-mounted display. However, existing pass-through image processing tends to exhibit fairly rough response times and may fail to depict the real world from the user's perspective or point of view and / or may be distorted. As a result, the user may lose balance, become dizzy, disoriented, or become ill.

[0012] This application describes, in part, a head mounted display (HMD) for use in a virtual reality (VR) environment. The systems and methods described herein may determine information about the real-world environment surrounding the user, the user's location within the real-world environment, and / or the user's posture within the real-world environment. Such information may enable the HMD to display images of the real-world environment in a pass-through manner, without distracting the user from the VR environment. In some cases, the HMD may pass-through images of the real-world environment based on one or more trigger events. For example, while wearing the HMD and the user is immersed within the VR environment, the user may activate a button that allows the user to look around in the real-world environment. As an additional example, the HMD may display content associated with the real-world environment if the user hears something suspicious or of interest, if the user wishes to locate an item (e.g., a controller) within the real-world environment, and / or if a visitor enters the user's vicinity. In some cases, the content may be provided to the user in an aesthetic manner to limit distraction from the VR environment. For example, the content may be provided as an overlay to virtual content associated with the VR environment. In such an example, the user may remain immersed in the VR environment while continuing to wear the HMD. Thus, the HMD according to the present application may increase the user experience when transferring or displaying content between a real-world environment and a VR environment.

[0013] The HMD may include a front having a display worn on the face adjacent to the user's eyes. The display may output images (or other content) for the user to view. As an example, a user may wear the HMD to play games or view media content (e.g., movies).

[0014] The HMD may include a camera that captures images of the real-world environment. In some cases, the camera may be mounted on the display and / or integrated into the front of the HMD. Additionally or alternatively, the camera may face forward to capture images outside the HMD and in front of the user. Furthermore, in some cases, the camera may be separate from the HMD and placed throughout the environment or elsewhere on the user's body (e.g., on the hips).

[0015] In some cases, the cameras may be spatially separated such that the optical axes of the cameras are parallel and separated by a known distance. Thus, the cameras may capture images of the real-world environment from slightly different viewing points. The diversity of information between the viewing points can be used to calculate depth information of the real-world environment (i.e., stereo camera image processing). For example, the HMD and / or a communicatively coupled computing device (e.g., a game console, a personal computer, etc.) can use the image data captured by the cameras to generate depth information associated with the real-world environment.

[0016] For example, the camera may include a first camera and a second camera horizontally and / or vertically displaced from one another on the front of the HMD. In some cases, the first camera may be located on a first side of the HMD and the second camera may be located on a second side of the HMD. However, as noted above, the first camera and the second camera may be located elsewhere in the environment and / or above other portions of the user.

[0017] The image data captured by the first camera and / or the second camera may represent different views of a real-world environment (e.g., a room). By comparing the images (or image data) captured by the first camera and / or the second camera, the HMD (and / or another communicatively coupled computing device) may determine a difference or disparity (e.g., using a disparity mapping algorithm). The disparity may represent the difference in coordinates of corresponding image points in the two images. Because the disparity (or disparity value) is inversely proportional to the depth in the real-world environment, the HMD and / or another communicatively coupled computing device, such as a game console, may determine depth information associated with the real-world environment (or a portion thereof). In some cases, the depth information may be from the user's perspective (i.e., the user's line of sight).

[0018] Using the depth information, the HMD, and / or another communicatively coupled computing device, such as a game console, may generate a depth map or three-dimensional (3D) mesh of the real-world environment (or a portion thereof). For example, the depth map may represent the distance between the user and objects in the real-world environment (e.g., walls of a room, furniture, etc.). Additionally or alternatively, the HMD may include other sensors utilized to generate the depth map and / or 3D. For example, the HMD may include a depth sensor for determining the distance between the user and objects in the real-world environment and may determine that the user is in proximity (e.g., a predetermined proximity or threshold proximity) to an object or boundary of the real-world environment. However, the HMD or game console may additionally or alternatively use light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, structured light analysis, dot projection, particle projection, time-of-flight observations, etc., for use in generating the depth map and / or 3D mesh.

[0019] Upon generating a depth map and / or 3D mesh of the real-world environment, the HMD and / or another communicatively coupled computing device, such as a game console, may project image data onto the depth map and / or 3D mesh. In this sense, the image data may be first utilized to generate the depth map and / or 3D mesh and may be subordinately superimposed, overlaid, or projected onto the depth map and / or 3D mesh. By doing so, the HMD may display content to depict the real-world environment to the user.

[0020] In some cases, the depth map, the 3D mesh, and / or the image captured by the first camera and / or the second camera may be modified to take into account the user's pose or viewpoint. For example, the camera may not be aligned (e.g., horizontally, vertically, and in depth) with the user's eyes (i.e., the camera is not at the exact position of the user's eyes), so the depth map and / or the 3D mesh may take this discrepancy into account. In other words, the image data captured by the first camera and the second camera may represent a different viewpoint from the user's viewpoint. Failure to take this discrepancy into account may describe an incomplete real-world environment, and the user may find it difficult to pick up an object because the depth values ​​or image data are not from the user's viewpoint. For example, because the camera captures image data from a perspective and / or depth different from that of the user's eyes (i.e., the camera is not in the same horizontal, vertical, and / or depth position as the user's eyes), the 3D mesh can take this offset into account to accurately depict and present an undistorted view of the real-world environment to the user. That is, the depth map, 3D mesh, and / or image data can be corrected based at least in part on the difference (or offset) in the coordinate positions (or locations) of the first and second cameras and the user's eyes (e.g., the first and / or second eyes) or field of view. Thus, the image of the real-world environment displayed to the user can accurately represent objects in the real-world environment from the user's perspective.

[0021] Additionally or alternatively, in some cases, the HMD and / or the real-world environment may include sensors that track the user's line of sight, viewpoint, and / or field of view. For example, the HMD may include an interpupillary distance (IPD) sensor to measure the distance between the pupils of the user's eyes, and / or other sensors to detect the direction of the user's eye gaze. Such sensors can be utilized to determine the user's viewpoint so as to accurately portray an image of the real-world environment to the user.

[0022] Pass-through image processing may allow a user to interact with and view objects in a real-world environment, such as a colleague, a computer screen, a mobile device, etc. In some cases, a user may toggle or switch between a VR environment and a real-world environment, or the HMD may automatically switch between the VR environment and the real-world environment in response to one or more trigger events. That is, the HMD may include one or more modes, such as a pass-through mode in which a real-world environment is presented to the user and / or a virtual reality mode in which a VR environment is presented to the user. As an example, while wearing the HMD, a user may wish to drink water. Rather than removing the HMD, the user may actuate (e.g., double-press) a button on the HMD, which causes the real-world environment to be displayed. The display may then present an image of the real-world environment, allowing the user to locate the user's own glass of water, yet without removing the HMD. Then, after locating the glass of water, the user may actuate (e.g., press once) a button to cause the display to present the virtual content of the VR environment. Thus, a pass-through image representing the real-world environment may enable a user to move around the real-world environment and locate objects without bumping into them (e.g., furniture). As a further example, the pass-through image may represent another individual entering the user's real-world environment, where the HMD may detect the other individual and present an image on the display such that the user can recognize or bring the other individual to their attention.

[0023] In some cases, content associated with the real-world environment displayed to the user may be partially transparent to maintain the user's sense of being in the VR environment. In some cases, content associated with the real-world environment may be combined with virtual content, or only content associated with the real-world environment may be presented (e.g., 100 percent pass-through image processing). Additionally or alternatively, in some cases, images captured by the camera may be presented over the entirety of the display, or within a specific portion. Furthermore, in some cases, content associated with the real-world environment may be displayed with dotted lines to indicate to the user which content is part of the real-world environment and which content is part of the VR environment. Such presentation may allow the user to see approaching individuals, disturbances, and / or objects surrounding the user. Regardless of the particular implementation or configuration, the HMD may function to display content associated with the real-world environment to alert, detect, or otherwise recognize objects coming into the user's field of view.

[0024] In some cases, the HMD and / or other computing devices associated with the VR environment, such as a game console, may operate in conjunction with a tracking system in the real-world environment. The tracking system may include sensors that track the user's position in the real-world environment. Such tracking may be used to determine information about the real-world environment surrounding the user, such as the user's location in the environment and / or the user's posture or viewpoint while the user is immersed in the VR environment. Within the real-world environment, the tracking system may determine the user's location and / or posture. In some cases, the tracking system may determine the user's location and / or posture relative to the center of the real-world environment.

[0025] In some cases, the tracking system may include lighting elements that emit light (e.g., visible or invisible) in the real-world environment and sensors that detect the incident light. In some cases, to detect the user's location and / or posture, the HMD may include markers. When projecting light into the real-world environment, the markers may reflect the light, and the sensors may capture the incident light reflected by the markers. The captured incident light may be used to track and / or determine the location of the markers in the environment, which may be used to determine the user's location and / or posture.

[0026] In some cases, the user's location and / or pose within the real-world environment may be utilized to present warnings, instructions, or content to the user. For example, if the user approaches a wall in the real-world environment and knows the location of the user and the wall (via the tracking system), the HMD may display an image representing the wall in the real-world environment. That is, in addition to or instead of using the image captured by the camera (i.e., via the depth value) to determine that the user is approaching a wall, the tracking system may determine the user's relative location within the real-world environment. Such tracking may assist in presenting images that accurately correspond to the depth (or placement) of objects within the real-world environment.

[0027] Additionally, images or data obtained from the tracking system can be used to generate a 3D model (or mesh) of the real-world environment. For example, by knowing the location and / or pose of the user, the HMD, the tracking system, the game console, and / or another communicatively coupled computing device, the relative location of the user within the real-world environment can be determined. This location and / or pose of the user can be utilized to determine the corresponding portion of the 3D model of the real-world environment that the user is viewing (i.e., the user's field of view). For example, when a camera captures an image that is not associated with the user's viewpoint, the tracking system may determine the user's line of sight, pose, or viewpoint. Such information can be used to determine where the user is looking within the real-world environment. Knowing where the user is looking within the real-world environment can be used to correct the image captured by the camera. In doing so, the HMD may accurately display the real-world environment.

[0028] The HMD, tracking system, game console, and / or another communicatively coupled computing device may also compare a depth map and / or a 3D mesh generated using the camera image data to the 3D model to determine the user's relative location within the real-world environment. Regardless of the particular implementation, knowing the user's location and / or pose within the real-world environment, the HMD and / or another communicatively coupled computing device may convert the points of the depth map and / or the points of the 3D mesh into a 3D model of the real-world environment. The image captured by the camera may then be projected onto the 3D model corresponding to the user's viewpoint.

[0029] In view of the above, therefore, the present application contemplates an HMD that provides pass-through image processing to enhance the VR experience. Pass-through image processing may provide a relatively seamless experience when displaying content of the VR environment and content associated with the real-world environment. Such pass-through image processing provides a less intrusive and less obtrusive solution for viewing content associated with the real-world environment. In some cases, information or content associated with the real-world environment may be selectively provided to the user in response to a trigger, including, but not limited to, a movement, a sound, a gesture, a preconfigured event, a change in the user's motion, and the like. Furthermore, in some cases, the HMD according to the present application may take many forms, including helmets, visors, goggles, masks, glasses, and other head or eyewear worn on the user's head.

[0030] The present disclosure provides a general understanding of the principles of the structures, functions, devices, and systems disclosed herein. One or more examples of the present disclosure are described in the accompanying drawings. Those skilled in the art will understand that the devices and / or systems specifically described herein and illustrated in the accompanying drawings are non-limiting embodiments. Features described or described in connection with one embodiment, including between systems and methods, can be combined with features of other embodiments. Such modifications and variations are intended to be included within the scope of the appended claims.

[0031] 1 illustrates a user 100 residing in an environment 102 and wearing an HMD 104. In some cases, the user 100 can wear the HMD 104 to immerse the user 100 in the VR environment. In some cases, the user 100 can interact in the VR environment using one or more controllers 106. The HMD 104 includes a display 108 for providing virtual content and / or images to the user 100 and, in some cases, an image capture device, such as a first camera 110 and / or a second camera 112.

[0032] The first camera 110 and / or the second camera 112 can capture images of the environment 102 and pass the images of the environment 102 through to the user 100 for viewing on the display 108. That is, as discussed in detail herein, the images captured by the first camera 110 and / or the second camera 112 can be presented to the user 100 in a pass-through manner to enable the user 100 to view the environment 102 without having to leave the VR environment and / or remove the HMD 104.

[0033] In some cases, the first camera 110 and / or the second camera 112 may be disposed in or near the front of the HMD 104. In some cases, the first camera 110 and / or the second camera 112 may represent a stereo camera, an infrared (IR) camera, a depth camera, and / or any combination thereof. The images captured by the first camera 110 and / or the second camera 112 may represent the environment 102 surrounding the user 100. In some cases, the first camera 110 and / or the second camera 112 may face forward to capture images of the environment 102 in front of the user 100. In some cases, the first camera 110 and / or the second camera 112 may be spatially separated such that the optical axes of these cameras are parallel. Thus, images captured by the first camera 110 and / or the second camera 112 may represent the environment 102 from different viewing points and may be used to determine depth information associated with the environment 102 (i.e., stereo camera image processing). In some cases, however, the first camera 110 and / or the second camera 112 may be located elsewhere within the environment 102. For example, the first camera 110 and / or the second camera 112 may be located on a floor of the environment 102, on a desk within the environment 102, or the like.

[0034] As described, the HMD 104 may include processor(s) 114 that perform or otherwise implement operations associated with the HMD 104. For example, the processor(s) 114 may cause the first camera 110 and / or the second camera 112 to capture images, and then receive the images captured by the first camera 110 and / or the second camera 112, compare the images (or image data), and determine a difference between the images (or image data). If the difference is inversely proportional to the depth of an object in the environment 102 (e.g., a wall, furniture, a TV, etc.), the processor(s) 114 may determine depth information associated with the environment 102.

[0035] In some cases, using the depth information, the processor(s) 114 may generate a depth map or a 3D mesh of the environment 102. For example, as described, the HMD 104 includes a memory 116 that stores or otherwise has access to the depth map 118 of the environment 102 and / or the 3D mesh 120 of the environment 102. If the image data captured by the first camera 110 and / or the second camera 112 represents a portion of the environment 102, the depth map 118 and / or the 3D mesh 120 may represent the portion of the environment 102 accordingly. In some cases, upon generating the depth map 118 and / or the 3D mesh 120, the processor(s) 114 may store the depth map 118 and / or the 3D mesh 120 in the memory 116.

[0036] If the image data, depth map, and / or 3D mesh captured by the first camera 110 and / or the second camera 112 are not from the perspective of the user 100 (i.e., the viewpoint of the user 100), the HMD 104 (and / or another communicatively coupled computing device) may take into account the placement of the first camera 110 and / or the second camera 112 relative to the viewpoint of the user 100 (e.g., relative to the first eye and / or the second eye, respectively). That is, whether located on the HMD 104 or elsewhere in the environment, the first camera 110 and / or the second camera 112 do not capture images corresponding to the viewpoint of the user 100 and / or the viewpoint of the user 100. Thus, the image data, or points in the depth map and / or the 3D mesh may be modified or offset to take into account this displacement.

[0037] In some cases, the HMD 104 may operate in association with the tracking system 122. In some cases, the HMD 104 may be communicatively coupled to the tracking system 122 via a network 124. For example, the HMD 104 and the tracking system 122 may include one or more interfaces, such as a network interface 126 and / or a network interface 128, respectively, to facilitate wireless connection to the network 124. The network 124 represents any type of communication network, including data and / or voice networks, and may include any type of communication network, including wired infrastructures (e.g., cable, CAT5, fiber optic cable, etc.), wireless infrastructures (e.g., RF, cellular, microwave, satellite, Bluetooth, etc.), and the like. TM ), and / or other connection technologies.

[0038] The tracking system 122 may include components that determine or track the pose of the user 100, the HMD 104, the first camera 110, and / or the second camera 112 within the environment 102. In this sense, the tracking system 122 can determine the location, orientation, and / or pose of the user 100, the HMD 104, the first camera 110, and / or the second camera 112 at the time the first camera 110 and / or the second camera 112 captured an image of the environment 102 for a pass through to the user 100. For example, the tracking system 122 (and / or another computing device) can analyze and parse the image captured by the tracking system 122 to identify the user 100 and / or the pose of the user 100 within the environment 102. For example, in some cases, tracking system 122 may include projector(s) 130 and / or sensor(s) 132 that operate to determine a location, orientation, and / or posture of user 100. As shown, in some cases, tracking system 122 may be mounted to a wall of environment 102. Additionally or alternatively, tracking system 122 may be mounted elsewhere within environment 102 (e.g., a ceiling, a floor, etc.).

[0039] Projector(s) 130 are configured to generate and project light and / or images into environment 102. In some cases, the images may include images of visible light perceptible to user 100, images of visible light not perceptible to user 100, images with non-visible light, or combinations thereof. Projector(s) 130 may be implemented with any number of technologies capable of generating and projecting images into / onto environment 102. Suitable technologies include Digital Micromirror Device (DMD), Liquid Crystal on Silicon Display (LCOS), Liquid Crystal Display, 3LCD, etc.

[0040] The sensor(s) 132 may include high-resolution cameras, infrared (IR) detectors, sensors, 3D cameras, IR cameras, RGB cameras, etc. The sensor(s) 132 are configured to image the environment 102 in visible light wavelengths, non-visible light wavelengths, or both. The sensor(s) 132 may be configured to capture information to detect the depth, location, orientation, and / or posture of objects within the environment 102. For example, as the user 100 navigates around the environment 102, the sensor(s) 132 may detect the location, orientation, and / or posture of the user 100. In some cases, the sensor(s) 132 may capture some or all angles and locations within the environment 102. Alternatively, the sensor(s) 132 may focus or capture images within a defined area of ​​the environment 102.

[0041] The projector(s) 130 and / or the sensor(s) 132 may operate in conjunction with the marker(s) 134 of the HMD 104. For example, the tracking system 122 may project light into the environment 102 via the projector(s) 130, and the sensor(s) 132 may capture reflected images of the marker(s) 134. Using the captured images, the tracking system 122, such as the processor(s) 136 of the tracking system 122, may determine distance information to the marker(s) 134. Additionally or alternatively, the tracking system 122 may detect a posture (e.g., orientation) of the user 100 within the environment 102. In some cases, the marker(s) 134 may be used to determine the viewpoint of the user 100. For example, the distance between the marker(s) 134 and the eyes of the user 100 may be known. In capturing image data of the marker(s) 134, the tracking system 122 (and / or other communicatively coupled computing devices) may determine the relative viewpoint of the user 100. Thus, the tracking system 122 can utilize the marker(s) 134 of the HMD 104 to determine the relative location and / or posture of the user 100 within the environment 102.

[0042] To define or determine characteristics about the environment 102, upon initiation of the gaming application, the HMD 104 may request the user 100 to define a boundary, perimeter, or area of ​​the environment 102 within which the user 100 may navigate while immersed in the VR environment. As an example, the processor(s) 114 may cause the display 108 to present instructions to the user 100 to walk around the environment 102 and define the boundary of the environment 102 (or the area within which the user 100 may navigate while immersed in the VR environment). As the user 100 walks around the environment 102, the HMD 104 may capture images of the environment 102 via the first camera 110 and / or the second camera 112, and the tracking system 122 may track the user 100. In that regard, upon determining the boundary of the environment 102, the tracking system 122 may determine the location of the center of the area (e.g., an origin). Knowing the location of the center of the area may enable the HMD 104 to properly display the relative location of an object or scene within the environment 102. In some cases, the center location may be represented as (0,0,0) in an (X,Y,Z) Cartesian coordinate system.

[0043] In some cases, the tracking system 122 may transmit the boundary and / or center location to the HMD 104. For example, the processor(s) 114 of the HMD 104 may store the boundary and / or center origin in the memory 116 as indicated by the boundary 138. Additionally or alternatively, in some cases, images captured by the first camera 110 and / or the second camera 112 may be associated with images captured by the tracking system 122. For example, images captured by the first camera 110 and / or the second camera 112 may be used to estimate the depth of the environment 102 while defining the area. A depth map may be generated. These depth maps may accordingly be associated with a particular location / pose within the environment 102 as determined by tracking the user 100 throughout the environment 102. In some cases, these depth maps may be combined or otherwise used to generate a 3D model or mesh of the environment 102. Upon receiving subsequent image data from the HMD 104 and / or the tracking system 122, the location of the user 100 within the environment 102 may be determined, which may be useful for determining depth information within the environment 102 and from the user's viewpoint, location, or pose. For example, if the image data captured by the first camera 110 and / or the second camera 112 does not correspond to the viewpoint of the user 100, the tracking system 122 may determine the viewpoint of the user 100 via images captured from the marker(s) 134. Using this viewpoint, the image data captured by the first camera 110 and / or the second camera 112 may be modified to represent the viewpoint of the user 100.

[0044] For example, as the user 100 engages in the VR environment and maneuvers around the environment 102, the tracking system 122 may determine the relative location of the HMD 104 within the environment 102 by comparing reflected light from the marker(s) 134 to a central location. Using this information, the HMD 104, the tracking system 122, and / or another communicatively coupled computing device (e.g., a game console) may determine the distance of the HMD 104 from the central location, or the location of the HMD 104 relative to the central location. Additionally, the HMD 104, the tracking system 122, and / or another communicatively coupled computing device may determine a pose, such as a viewpoint, of the user 100 within the environment 102. For example, the tracking system 122 may determine that the user 100 is looking toward a ceiling, a wall, or a floor of the environment 102. Such information may be used to correct or otherwise take into account the position of the first camera 110 and / or the second camera 112 relative to the user's eyes or viewpoint.

[0045] In some cases, tracking system 122 may be coupled to a chassis that has a fixed orientation, or the chassis may be coupled to actuator(s) 140 such that the chassis may move. Actuators 140 may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis or components of tracking system 122, such as projector(s) 130 and / or sensor(s) 132.

[0046] The HMD 104 may additionally or alternatively operate in conjunction with a remote computing resource 142. The tracking system 122 may also be communicatively coupled to the remote computing resource 142. In some examples, the HMD 104 and / or the tracking system 122 may be communicatively coupled to the remote computing resource 142, taking into account that the remote computing resource 142 may have a computing power that far exceeds that of the HMD 104 and / or the tracking system 122. Thus, the HMD 104 and / or the tracking system 122 may utilize the remote computing resource 142 to perform a relatively complex analysis of the environment 102 and / or to generate a model (or mesh) of image data of the environment 102. For example, the first camera 110 and / or the second camera 112 may capture image data that the HMD 104 may provide to the remote computing resource 142 over the network 124 for analysis and processing. In some cases, the remote computing resource 142 may transmit content to the HMD 104 for display. For example, in response to a trigger event (e.g., a button press) that configures the HMD 104 to pass through images captured by the first camera 110 and / or the second camera 112, the remote computing resource 142 may transmit a depth map, a 3D mesh, and / or a 3D model onto which the HMD 104 projects the images captured by the first camera 110 and / or the second camera 112. Thus, images captured by the first camera 110 and / or the second camera 112 for passing to the user 100 may be transmitted to the remote computing resource 142 for processing, and the HMD 104 may receive content to be displayed on the display 108.

[0047] As described, the remote computing resource 142 includes a processor(s) 144 and a memory 146, which may store or otherwise have access to some or all of the components described with respect to the memory 116 of the HMD 104. For example, the memory 146 may have access to and utilize the depth map 118, the 3D mesh 120, and / or the boundary 138. The remote computing resource 142 may additionally or alternatively store or otherwise have access to the memory 148 of the tracking system 122.

[0048] In some cases, the remote computing resources 142 may be remote from the environment 102, and the HMD 104 may be communicatively coupled to the remote computing resources 142 via the network interface 126 and over the network 124. The remote computing resources 142 may be implemented as one or more servers, and in some cases may form part of a network-accessible computing platform implemented as a computing infrastructure of processors, storage, software, data access, etc., that is maintained and accessible over a network, such as the Internet. The remote computing resources 142 do not require end-user knowledge of the physical location and configuration of the systems that deliver the services. Common expressions associated with these remote computing resources 142 may include "on-demand computing," "software as a service (SaaS)," "platform computing," "network-accessible platform," "cloud services," "data center," and the like.

[0049] In some cases, the environment 102 may include a computer 150 (or gaming application, gaming console, gaming system) communicatively coupled to the HMD 104, the controller 106, the tracking system 122, and / or the remote computing resources 142 via the network 124 and / or wired technology. In some cases, the computer 150 may perform some or all of the processes described herein, such as those that may be performed by the HMD 104, the controller 106, the tracking system 122, and / or the remote computing resources 142. For example, the computer 150 may modify image data received from the first camera 110 and / or the second camera 112, or a depth map associated with the cameras, to take into account the viewpoint of the user 100. In such an example, the computer 150 may determine by or receive from the tracking system 122 an indication regarding the viewpoint of the user 100 and / or the origin of the area in which the user is located (e.g., the origin of the real-world environment) and / or the origin of the virtual world. Thus, the computer may modify image data received from the first camera 110 and / or the second camera 112, or a depth map associated with the camera, based on instructions received from the tracking system 122. In some embodiments, the time(s) at which the image data was captured by one or both of the cameras 110, 112 (e.g., timestamp(s)) may be used to determine the time difference between the time of capturing the image data and the time of displaying the captured image data. For example, the image data that is ultimately displayed to the user may be delayed by tens of milliseconds after the image data is captured by the camera(s) 110 and / or 112.Adjustments can be made to the image data to take this temporal parallax or mismatch into account, such as by modifying the pixel data (e.g., through rotational and / or translational adjustments) to present the image as it appears in the physical world at the time the image is displayed on the HMD, thereby avoiding the presentation of images that appear to lag behind the user's head movements. In some embodiments, the camera(s) 110 and / or 112 can be tracked and timestamp information maintained, such that the pose of the camera(s) at the time the image(s) are captured is known, and such that the pose of the camera(s) at the time the resulting image data is received and processed can be used to accurately represent the image in space, instead of relying on camera movement. Additionally, computer 150 may store or otherwise have access to some or all of the components described with respect to memory 116, 146, and / or 148.

[0050] As used herein, a processor, such as the processor(s) 114, 136, 144, and / or a processor of a computer, may include multiple processors and / or a processor having multiple cores. Furthermore, the processor(s) may comprise one or more cores of different types. For example, the processor(s) may include an application processor unit, a graphics processing unit, etc. In one implementation, the processor(s) may include a microcontroller and / or a microprocessor. The processor(s) may include a graphics processing unit (GPU), a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively, or in addition, what is functionally described herein may be implemented at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. In addition, each of the processor(s) may have its own local memory in which program components, program data, and / or one or more operating systems may be stored.

[0051] As used herein, memory, such as memory 116, 146, 148, and / or memory of computer 150, may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program components, or other data. Such memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Memory may be implemented as a computer-readable storage medium ("CRSM"), which may be any available physical medium that can be accessed by a processor(s) to execute instructions stored in the memory. In one basic implementation, the CRSM may include random access memory ("RAM") and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and that can be accessed by a processor(s).

[0052] 2 illustrates a perspective view of the HMD 104. The HMD 104 may include a front 200 and a rear 202 that are secured to the head of the user 100. For example, the HMD 104 may include a strand, cord, section, strap, band, or other member that operably connects the front 200 of the HMD 104 and the rear 202 of the HMD 104. The front 200 includes a display 108 that is positioned in front of or above the eyes of the user 100 to render images output by an application (e.g., a video game). As discussed above, the display 108 outputs images (frames) that the user 100 views, allowing the user 100 to perceive the images as if immersed in a VR environment.

[0053] In some cases, the front 200 of the HMD 104 may include a first camera 110 and / or a second camera 112. The first camera 110 and / or the second camera 112 may capture images of the exterior of the HMD 104 (i.e., of the environment 102) for the user 100 to view on the display 108. As described above, the optical axes of the first camera 110 and the second camera 112 may be parallel and separated by a predetermined distance such that images captured by the first camera 110 and / or the second camera 112 are used to generate depth information of at least a portion of the environment 102. However, in some cases, as discussed above, the HMD 104 may not include the first camera 110 and / or the second camera 112. For example, the first camera 110 and / or the second camera 112 may be located elsewhere in the environment 102.

[0054] In either scenario, the first camera 110 and / or the second camera 112 capture images that may not correspond to the viewpoint of the user 100 (i.e., when the first camera 110 and / or the second camera 112 are not at the actual position of the user's 100 eyes). However, using the techniques described herein, the image data captured by the first camera 110 and / or the second camera 112 can be modified to take into account the displacement of the first camera 110 and / or the second camera 112 relative to the viewpoint of the user 100.

[0055] For example, to determine the viewpoint of the user 100, the HMD 104 may include marker(s) 134. As shown in FIG. 2, in some cases, the marker(s) 134 may include a first marker 204(1), a second marker 204(2), a third marker 204(3), and / or a fourth marker 204(4) disposed at the corners, edges, or along the perimeter of the front 200. However, the marker(s) 134 may be located elsewhere on the HMD 104, such as along the top, sides, or rear 202. In some cases, the marker(s) 134 may include infrared elements, reflectors, digital watermarks, and / or images that are responsive to electromagnetic radiation (e.g., infrared) emitted by the projector(s) 130 of the tracking system 122. Additionally or alternatively, the marker(s) 134 may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) that is captured by the sensor(s) 132 of the tracking system 122. That is, the projector(s) 130 may project light into the environment 102 and the marker(s) 134 may reflect the light. The sensor(s) 132 may capture the incident light that the marker(s) 134 and tracking system 122 reflect, or another communicatively coupled computing device, such as a remote computing resource 142, may track and plot the location of the marker(s) 134 within the environment 102 to determine the movement, position, posture, and / or orientation of the user 100. Thus, the marker(s) 134 can be used to indicate the viewpoint of the user 100 for use in correcting, adjusting or otherwise adapting image data from the first camera 110 and / or the second camera 112 before it is displayed to the user 100.

[0056] Figure 3 illustrates a user 100 wearing an HMD 104 within an environment 300. Figure 3 illustrates the display 108 of the HMD 104, which displays virtual content 302 and interacts within the VR environment while the user 100 is wearing the HMD 104. As discussed above, the HMD 104 may display the virtual content 302 as the user 100 moves around the environment 102.

[0057] The tracking system 122 may be positioned within the environment 300 to track the user 100 from one location to another and determine the viewpoint of the user 100 within the environment. For example, the tracking system 122 may utilize the marker(s) 134 on the HMD 104 to determine the pose (e.g., location and orientation) of the user 100. In some cases, the pose of the user 100 may be relative to a central location of the environment 300. For example, the central location may have coordinates (0,0,0), and using reflected light from the marker(s) 134, the tracking system 122 (or the remote computing resource 142) may determine the pose of the user 100 in coordinate space (e.g., (X,Y,Z)). The tracking system 122 may also determine the depth of objects within the environment 300 from the perspective or viewpoint of the user 100. Additionally or alternatively, image data received from the first camera 110 and / or the second camera 112 may be used to determine the depth within the environment 300.

[0058] The HMD 104 may coordinate the display of the virtual content 302 with objects (e.g., furniture, walls, etc.) in the environment 300 and / or in and / or in front of the user 100's viewpoint. In other words, the objects of the environment 300 (or real world) displayed to the user 100 may correspond to the user's 100's viewpoint in the environment 300. For example, the HMD 104 may display objects in the environment 300 using image data (e.g., pass-through images) captured by the first camera 110 and / or the second camera 112 based at least in part on one or more trigger events. That is, the HMD 104 may be configured to display images on the display 108 at locations that correspond to the actual placement of the objects in the environment 300. Utilizing a depth map generated from image data of the first camera 110 and / or the second camera 112, as well as a 3D model or mesh of the environment 300, the pass-through image presented to the user 100 may describe objects located in their actual placement (i.e., at the appropriate depth values) within the environment 300 from the perspective of the user 100, thereby enabling the user 100 to pick up objects (e.g., a controller, a glass of water, etc.). In some cases, however, image data from the first camera 110 and / or the second camera 112 may be captured continuously, and upon detecting a trigger event, the HMD 104 may display content associated with the environment 300 to the user 100.

[0059] 3, the triggering event may include determining that the user 100 is approaching or approaching a boundary of the environment 300. For example, the user 100 may approach a corner 304 between two walls 306 and 308 of the environment 300. By knowing the location of the user 100 via the tracking system 122, the HMD 104 may display content on the display 108 to indicate that the user 100 is approaching the corner 304. For example, as shown, an indication of the corner 304 and the walls 306, 308 may be displayed as dashed or dotted lines on the display 108. In doing so, the user 100 is presented with an indication that the user 100 is about to run into a wall. In some cases, as shown in FIG. 3, instructions or content regarding the corner 304 and walls 306, 308 may be overlaid (in combination with the virtual content displayed on the display 108 of the HMD 104) or presented on (or in combination with) the virtual content 302.

[0060] As described above, image data captured by the first camera 110 and / or the second camera 112 can be modified to take into account the viewpoint of the user 100. Additionally or alternatively, the depth map, 3D mesh, or model of the environment can take into account or account for the location of the first camera 110 and / or the second camera 112 relative to the eyes or viewpoint of the user 100. Thus, if the first camera 110 and / or the second camera 112 have a viewpoint that may differ from the user 100, the image data can be modified to adjust to the user's viewpoint before presentation on the HMD 104.

[0061] While FIG. 3 illustrates a particular implementation of displaying the corner 304 and / or the walls 306, 308 on the display 108, the content may be presented in different ways, such as without modification, modified for display, embedded, merged, stacked, split, re-rendered, or otherwise manipulated, so as to be appropriately provided to the user 100 without interrupting the immersive virtual experience. For example, in some cases, the content may be displayed in a particular region of the display 108 (e.g., the top right-hand corner). Additionally or alternatively, the display 108 may present only the content associated with the corner 304 and the walls 306, 308, such that it has 100 percent pass-through. In some cases, the HMD 104 may fade in the content associated with the environment 300 on the display 108.

[0062] FIG. 4 illustrates a user 100 wearing an HMD 104 within an environment 400. In some cases, the HMD 104 may display a pass-through image based at least in part on one or more triggering events. In some cases, the triggering event may include determining that a visitor 402 has entered the environment 400 and / or a defined area within the environment 400. As shown in FIG. 4, the display 108 depicts a visitor 402 approaching a user 100 wearing an HMD 104. Here, the visitor 402 appears to be approaching the user 100, and thus the display 108 may display an image captured by the first camera 110 and / or the second camera 112. However, the image may first be modified to take into account a difference in perspective between the first camera 110 and / or the second camera 112 and the perspective of the user 100. In some cases, the HMD 104 may display the visitor 402 based at least in part on the visitor 402 being within the viewpoint of and / or in front of the user 100, as determined using the marker(s) 134 while wearing the HMD 104 or while within a threshold distance of the user 100. In some cases, the visitor 402 may be detected via a motion sensor and analyzing image data from the first camera 110 and / or the second camera 112, such as via a tracking system 122.

[0063] 5 and 6 illustrate a process according to an embodiment of the present application. The processes described herein are described as a collection of blocks in a logical flow diagram representing a sequence of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media, which, when executed by one or more processors, programs the processor to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the blocks are described should not be construed as limiting, unless otherwise stated. Any number of the described blocks may be combined in any order and / or in parallel to implement a process, or alternative processes, and not all of the blocks need to be executed. For the sake of discussion, the processes are described with reference to the environments, architectures, and systems described in the examples herein, such as those described with respect to FIGS. 1-4, although the processes may be implemented in a wide variety of other environments, architectures, and systems.

[0064] 5 illustrates an example process 500 for generating a 3D model, depth map, or mesh of the environment 102. In some cases, the process 500 may be performed by a remote computing resource 142. However, the HMD 104, the tracking system 122, and / or the computer 150 may also perform some or all of the process 500.

[0065] At 502, the process 500 may transmit a request to define an area in the environment 102. For example, the remote computing resource 142 may transmit a request to the HMD 104 requesting that the user 100 define an area of ​​the environment 102. In some cases, the area may represent an area in which the user 100 intends to move about while immersed in the VR environment. In response to receiving the request, the processor(s) 114 of the HMD 104 may present the request or information associated with the request on the display 108. For example, the request may inform the user 100 to wear the HMD 104, walk around the environment 102, and define an area in which the user intends to move about.

[0066] At 504, the process 500 may transmit a request to track the HMD 104 within the environment 102 and as the user 100 wears the HMD 104 and defines a zone. For example, the remote computing resource 142 may transmit a request to the tracking system 122 to track the user 100 throughout the environment 102 while the user 100 defines a zone. In some cases, the tracking system 122 may track the user 100 via the projector(s) 130 that project images into the environment 102 and the sensor(s) 132 that capture images of light reflected through the marker(s) 134 of the HMD 104. In some cases, the remote computing resource 142 may transmit the request of 502 and the request of 504 simultaneously or substantially simultaneously.

[0067] At 506, process 500 may receive first image data from tracking system 122. For example, remote computing resource 142 may receive the first image data from tracking system 122, where the first image data represents a location of HMD 104 and / or user 100 as user 100 walks around environment 102 to define an area. For example, the first image data may represent a location and / or a pose of marker(s) 134 within environment 102. The first image data received by remote computing resource 142 may include a timestamp of when the image was captured by tracking system 122 (or sensor(s) 132).

[0068] At 508, the process 500 may determine characteristics of the area. For example, based at least in part on receiving the first image data, the remote computing resource 142 may determine a boundary or perimeter of the area. Additionally or alternatively, the remote computing resource 142 may determine a center or origin of the area based at least in part on the first image data. In some cases, the origin may be defined in 3D space having values ​​(0, 0, 0) corresponding to X, Y, and Z coordinates, respectively, in a Cartesian coordinate system. Thus, upon receiving subsequent image data from the tracking system 122 and / or the remote computing resource 142, a relative location of the user 100 (and / or HMD 104) with respect to the origin of the area may be determined.

[0069] At 510, the process 500 may receive second image data from the HMD 104. For example, the remote computing resource 142 may receive the second image data from the HMD 104, where the second image data represents images captured by the first camera 110 and / or the second camera 112 while the user 100 defined the area of ​​the environment 102. In some cases, the second image data received by the remote computing resource 142 may include a timestamp of when the images were captured by the first camera 110 and / or the second camera 112. That is, while the first camera 110 and / or the second camera 112 capture images, another communicatively coupled computing device may determine the pose of the user 100 within the environment 102. For example, as described above, while the first camera 110 and / or the second camera 112 capture images, the processor(s) 136 of the tracking system 122 may cause the projector(s) 130 to project images onto the environment 102. The marker(s) 134 of the HMD 104 may reflect light associated with the image, and the sensor(s) 132 may capture reflected images of the marker(s) 134. Such images may be used to determine the depth, location, orientation, and / or posture of the user 100 (and / or the HMD 104) within the environment 102 and may be associated with the image data captured by the first camera 110 and / or the second camera 112.

[0070] At 512, the process 500 may generate a 3D model (or mesh) of the environment 102. For example, the remote computing resource 142 may generate the 3D model of the environment 102 based at least in part on the first image data and / or the second image data. For example, using the second image data, the remote computing resource 142 may compare images captured by the first camera 110 and / or the second camera 112, respectively, to determine disparity. The remote computing resource 142 may then determine depth information of the environment 102 for use in generating the 3D model of the environment 102. Additionally, the remote computing resource 142 may utilize the first image data received from the tracking system 122 to associate a depth map or depth values ​​of the environment 102 with areas and / or specific locations within the environment 102. In some cases, the remote computing resource 142 can compare or correlate the first and second image data (or depth maps generated from the image data) using a timestamp of when the first and / or second image data were captured. In some cases, generating the 3D model of the environment 102 can include depicting objects in the environment. For example, the remote computing resource 142 can augment the VR environment with objects (or volumes) in the environment 102, such as a chair. In some cases, the 3D model can be transmitted to the HMD 104 and / or the tracking system 122 and / or stored in memory of the HMD 104, the tracking system 122, and / or the remote computing resource 142.

[0071] 6 illustrates an example process 600 for passing through an image on an HMD, such as the HMD 104. In some cases, the process 600 may continue from 512 of the process 500 after a 3D model of the environment 102 is generated. In other words, in some cases, after the 3D model is generated, the HMD 104 can display a pass-through image to enable the user 100 to switch between the VR environment and the real-world environment (i.e., the environment 102). In some cases, the process 600 may be performed by the remote computing resource 142. However, the HMD 104, the tracking system 122, and / or the computer 150 may further perform some or all of the process 600.

[0072] At 602, the process 600 may receive first image data from the HMD 104. For example, the remote computing resource 142 may receive the first image data from the HMD 104, the first image data representing an image captured by the first camera 110 and / or the second camera 112. In some cases, the remote computing resource 142 may receive the first image data based at least in part on the HMD 104 detecting a trigger event, such as a button press by the user 100, a verbal command by the user 100, motion detected within the environment 102 (e.g., a visitor approaching the user 100), and / or the user 100 approaching or reaching within a threshold distance of a boundary of the environment 102, etc. In other examples, the remote computing resource 142 may receive the first image data continuously and may be configured to pass through these images based at least in part on the trigger event. For example, as the user 100 may be immersed in the VR environment, the user 100 may press a button on the HMD 104 to display content outside of the HMD 104. In this sense, the HMD 104 may include a pass-through mode that displays images captured by the first camera 110 and / or the second camera 112. Thus, based at least in part on detecting a triggering event, the processor(s) 114 of the HMD 104 may cause the first camera 110 and / or the second camera 112 to capture images of the environment 102.

[0073] At 604, the process 600 may generate a depth map and / or a 3D mesh based at least in part on the first image data. For example, the remote computing resource 142 may generate the depth map and / or the 3D mesh based at least in part on the first image data being received from the HMD 104 (i.e., using stereo camera image processing). If the first image data represents a portion of the environment 102, the depth map and / or the 3D mesh may also correspond to the depth map and / or the 3D mesh of the portion of the environment 102. Additionally or alternatively, the HMD 104 may generate the depth map and / or the 3D mesh. For example, upon receiving image data from the first camera 110 and / or the second camera 112, the processor(s) 114 may utilize stereoscopic camera image processing to generate the depth map. In some cases, using the depth map, the processor(s) 114 may generate a 3D mesh of the environment 102. Further, in some cases, the processor(s) 114 may store a depth map and / or a 3D mesh, such as depth map 118 and / or 3D mesh 120, in memory 116.

[0074] In some cases, the HMD 104, the remote computing resource 142, and / or the computer 150 may modify the first image data received from the first camera 110 and / or the second camera 112 before generating the depth map and / or the 3D mesh. For example, if the first image data cannot represent the viewpoint of the user 100, the first image data may be modified (e.g., repurposed, transformed, distorted, etc.) to take into account differences between the viewpoints of the first camera 110 and the second camera 112 and the viewpoint of the user 100. In some cases, the viewpoint of the user 100 may be determined using the tracking system 122 and the position of the marker(s) 134 in the environment 102. Additionally or alternatively, after generating the depth map and / or the 3D mesh, the depth map and / or the 3D mesh may be modified according to or based at least in part on the viewpoint of the user 100.

[0075] At 606, the process 600 may receive second image data forming the tracking system 122 that represents a pose of the HMD 104. For example, the remote computing resource 142 may receive the second image data corresponding to the HMD 104 in the environment 102 from the tracking system 122. As discussed above, the second image data may be captured by the sensor(s) 132 of the tracking system 122 that detects light reflected by the marker(s) 134 of the HMD 104 in response to an image projected by the projector(s) 130.

[0076] At 608, the process 600 may determine a pose of the HMD 104. For example, based at least in part on receiving the second image data, the remote computing resource 142 may determine a pose of the HMD 104 within the environment 102. In some cases, the pose may represent a location of the HMD 104 within the environment 102 and / or an orientation of the HMD 104 within the environment 102. That is, the remote computing resource 142 may analyze the first image data and / or the second image data to determine a location of the user 100 within the environment 102 relative to a center of the environment 102. Such analysis may determine a relative location, line of sight, and / or viewpoint of the user 100 relative to the center of the environment 102 or a particular area within the environment 102.

[0077] At 610, the process 600 may convert the depth map and / or the 3D mesh into points associated with a 3D model of the environment. For example, the remote computing resource 142 may utilize the pose (e.g., location and orientation) to determine the location of the user 100 within the environment 102 and / or the viewpoint of the user 100 within the environment 102. The remote computing resource 142 may convert the points of the depth map and / or the 3D mesh into points associated with the 3D model of the environment 102, thereby taking into account the viewpoint of the user 100. In other words, the processor(s) 144 of the remote computing resource 142 may locate, find, or determine depth values ​​of points in the 3D model of the environment 102 by converting the points of the depth map and / or the 3D mesh into points associated with the 3D model of the environment 102. Using the pose, the remote computing resource 142 may transfer the depth map and / or the 3D mesh generated at 604 into the 3D model of the environment 102. Such repurposing can help accurately depict the environment 102 (e.g., at the proper depth value) relative to the user 100. That is, the first image data, together with the second image data and / or a 3D model of the environment 102, can provide an absolute position and / or line of sight of the HMD 104, which can help depict the viewpoint of the user 100 at the proper depth value or depth perception.

[0078] At 612, the process 600 can project the first image data onto a portion of a 3D model to generate third image data. For example, with the pose of the user 100 known, the remote computing resource 142 can project or overlay the first image data onto a portion of a 3D model of the environment 102 to generate the third image data. In some cases, generating the third image data can include cross-blending or inputting depth and / or color values ​​for certain pixels of the third image data. For example, if the first image data cannot represent the viewpoint of the user 100, when generating the third image data depicting the viewpoint of the user 100, the third image data can have undefined depth and / or color values ​​for certain pixels. Here, pixels that do not include a color value can be assigned color values ​​from neighboring pixels, or an average value of the neighboring pixels. Additionally or alternatively, the third image data can be generated using a previous depth map or 3D mesh of the environment 100, particle system, etc.

[0079] At 614, the process 600 may transmit the third image data to the HMD 104. For example, after projecting the first image data onto the portion of the 3D model, the remote computing resource 142 may transmit the third image data to the HMD 104, where the third image data represents the first image data projected onto the portion of the 3D model.

[0080] From 614, the process 600 may loop back to 602 to receive subsequent image data. As a result, in response to a continuing trigger event (e.g., a button press, a voice command, motion detection, etc.), the HMD 104 may transition to a pass-through mode to provide a convenient way for the user 100 to leave the environment 102, without having to take off the HMD 104. For example, the remote computing resource 142 may receive an indication from the tracking system 122 that the user 100 is approaching an area boundary and / or is about to collide with a wall of the environment 102. Upon receiving this indication, the remote computing resource 142 may receive image data from the HMD 104 representing the user's viewpoint (i.e., images captured by the first camera 110 and / or the second camera 112). Upon determining the pose of the user 100, the remote computing resource 142 may project the image data onto a 3D model of the environment 102 and display the image data on the HMD 104. In this sense, images may be automatically "passed through" to the user 100, allowing the user to view their real-world environment without having to break immersion.

[0081] Although the above invention has been described with respect to specific embodiments, it should be understood that the scope of the invention is not limited to these specific embodiments. Since other modifications and variations adapted to suit specific operating requirements and environments will be apparent to those skilled in the art, the invention should not be considered as limited to the embodiments selected for purposes of disclosure, but should extend to all variations and variations that do not depart from the true concept and scope of the invention.

[0082] Although the present application describes embodiments having particular structural features and / or methodological acts, it should be understood that the claims are not necessarily limited to the particular features or acts described. Rather, the particular features and acts are merely illustrative of some embodiments that are within the scope of the present application's claims.

Claims

1. display, A first camera, and a head mounted display including a second camera; one or more processors; one or more non-transitory computer-readable media storing computer-executable instructions; the computer-executable instructions, when executed by the one or more processors, cause the one or more processors to perform actions, the actions including: capturing first image data representative of a first portion of an environment via the first camera; capturing second image data representative of a second portion of the environment via the second camera; and determining a first offset of the first camera relative to a first eye of a user and a second offset of the second camera relative to a second eye of the user; generating a depth map based at least in part on the first image data, the second image data, the first offset, and the second offset; transmitting the depth map, the first image data, and the second image data to one or more computing devices; receiving third image data from the one or more computing devices based at least in part on the depth map, the first image data, and the second image data; and displaying the third image data via the display.

2. the head mounted display further comprises a button; the first image data is captured based at least in part on detecting a depression of the button; or The system of claim 1 , wherein the second image data is at least one of: captured based at least in part on detecting the depression of the button.

3. the one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions, the actions further including receiving an indication from the one or more computing devices that the user is approaching a boundary of the environment; capturing the first image data is based at least in part on receiving the instruction; or The system of claim 1 , wherein capturing the second image data is at least one of based at least in part on receiving the instruction.

4. the users include a first user, the one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts, the acts further including receiving an indication from the one or more computing devices that a second user is within the environment; capturing the first image data is based at least in part on receiving the instruction; or The system of claim 1 , wherein capturing the second image data is at least one of based at least in part on receiving the instruction.

5. the third image data is displayed in combination with virtual content presented on the display; or The system of claim 1 , wherein the third image data is at least one of: displayed on a predetermined portion of the display.

6. 1. A method comprising: receiving first image data representative of at least a portion of an environment from a head mounted display; receiving second image data from a tracking system indicating a viewpoint of a user within the environment; generating a depth map corresponding to at least a portion of the environment based at least in part on at least one of the first image data or the second image data; Associating the depth map with a portion of a three-dimensional (3D) model of the environment; transposing the first image data to the portion of the 3D model of the environment; generating third image data representing the first image data transposed onto the portion of the 3D model of the environment; transmitting the third image data to the head mounted display.

7. transmitting a first request to the head mounted display to capture an image of the environment; 7. The method of claim 6, further comprising transmitting to the tracking system a second request to track a location of the head mounted display while the head mounted display is capturing the image of the environment.

8. receiving fourth image data from the head mounted display representative of the image of the environment; receiving fifth image data from the tracking system representative of the location of the head mounted display while capturing the image of the environment; The method of claim 7 , further comprising: generating a 3D model of the environment based at least in part on the fourth image data and the fifth image data.

9. The method of claim 6 , wherein associating the depth map with a portion of the 3D model of the environment comprises aligning individual points of the depth map with corresponding individual points of the portion of the 3D model.

10. The method of claim 6 , wherein transmitting the third image data is based at least in part on receiving an indication from the head mounted display.

11. The instructions are: A button was pressed on the head mounted display; detecting motion in the environment; detecting movement of the user within a predetermined distance; or The method of claim 10 , responsive to at least one of determining that the user is approaching a boundary of the environment.

12. 7. The method of claim 6, further comprising transmitting fourth image data to the head mounted display, the fourth image data representing virtual content associated with a virtual environment, the head mounted display being configured to display at least a portion of the third image data in combination with at least a portion of the fourth image data.

13. determining a center of the environment; determining a location of the user within the environment based at least in part on the second image data; The method of claim 6 , wherein associating the depth map with the portion of a 3D model is based at least in part on determining the location of the user relative to the center of the environment.

14. The method of claim 6 , wherein the third image data represents the viewpoint of the user within the environment.

15. 1. A system comprising: A head mounted display and A first camera; A second camera; one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions, including: receiving first image data representative of a first portion of an environment from the first camera; receiving second image data from the second camera representative of a second portion of the environment; generating a first depth map of a portion of the environment based at least in part on the first image data and the second image data; receiving data from a tracking system corresponding to a user's viewpoint within the environment; generating a second depth map based at least in part on the first depth map and the data; generating third image data based at least in part on projecting the first image data and the second image data onto the second depth map; transmitting the third image data to the head mounted display.

16. 16. The system of claim 15, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions, the actions further including receiving an indication of a trigger event, and transmitting the third image data is based at least in part on receiving the indication.

17. The instructions are: Determining that another user is in front of the user wearing the head mounted display; or The system of claim 16 , responsive to at least one of determining that the user is approaching a boundary of the environment.

18. 16. The system of claim 15, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform an action, the action further comprising transmitting fourth image data representing virtual content within a virtual reality environment.

19. The system of claim 15 , wherein the data corresponds to a location of the user within the environment relative to a central location of the environment.

20. The one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions, including: generating a 3D model of the environment; determining a portion of the 3D model that corresponds to the viewpoint of the user; The system of claim 15 , wherein generating the second depth map is based at least in part on the portion of the 3D model.

Images