Head mounted display with pass-through imaging
By integrating cameras and tracking systems in head-mounted displays, the problem of depth information and 3D grids is solved in the prior art that users find it difficult to view the real world in virtual reality environments, and high responsive pass-through imaging is achieved, improving User experience and security.
Patent Information
- Application Number
- CN202411313850.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2020-04-14
- Filing Date
- 2020-04-22
- Publication Date
- 2024-11-12
AI Technical Summary
When providing an immersive virtual reality experience, existing head-mounted displays block users' vision of the real world, making it difficult for users to pick up objects or identify individual and physical boundaries in the real world. The response time of existing direct-through imaging technologies is rough. It may cause users to be imbalanced or lose their direction.
Using a head-mounted display including a front display and a camera, the image data of the real world environment is captured, depth information and a 3D grid is generated, combined with tracking systems and sensors, providing accurate straight-through imaging from the user's viewpoint, allowing users to not remove it. Under the head-mounted monitor case to view the real world.
Achieves a relatively smooth pass-through imaging experience in a virtual reality environment, where users can view the real world without being distracted, improving the accuracy and security of the real world environment and reducing the between virtual reality and the real world. conversion interference.
Smart Images

Figure CN118938467A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application number No. 202080030592.5 filed on April 22, 2020, and the invention name is “Head-mounted display with through-imaging”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This PCT application claims priority to U.S. Patent Application Serial No. 16 / 848,332 filed on April 14, 2020, which claims priority to U.S. Provisional Patent Application Serial No. 62 / 837,668 filed on April 23, 2019. Application Serial Nos. 16 / 848,332 and 62 / 837,668 are incorporated herein by reference in their entirety. Background Art
[0004] Head-mounted displays are used in a variety of applications, including engineering, medical, military, and video games. In some cases, a head-mounted display may present information or images to a user as part of a virtual reality or augmented reality environment. For example, when playing a video game, a user may wear a head-mounted display to immerse the user in a virtual environment. Although a head-mounted display provides an immersive experience, it blocks the view of the physical or real world. Therefore, the user may find it difficult to pick up objects (e.g., controllers) and / or identify other individuals in the real world. In addition, the user may not be aware of physical boundaries (e.g., walls) in the real world. Although taking off the head-mounted display allows the user to see the real world, constantly taking off and putting on the head-mounted display again may be tedious, may require the user to reposition himself between the virtual environment and the real world, and / or may otherwise distract the user from the virtual reality experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The specific embodiments are described with reference to the accompanying drawings. In the figures, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The same or similar reference numbers in different figures represent similar or identical items.
[0006] Figure 1 A user is shown wearing an exemplary head mounted display in an exemplary environment according to an exemplary embodiment of the present disclosure.
[0007] Figure 2 According to one embodiment of the present disclosure Figure 1 A stereogram of a head-mounted display.
[0008] Figure 3 Shown is a wearing according to an embodiment of the present disclosure Figure 1 A user of a head-mounted display is provided, and the head-mounted display presents a pass-through image.
[0009] Figure 4 Shown is a wearing according to an embodiment of the present disclosure Figure 1 A user of a head-mounted display is provided, and the head-mounted display presents a pass-through image.
[0010] Figure 5 An embodiment of the present disclosure is shown using Figure 1 An exemplary process for presenting a through-image using a head-mounted display.
[0011] Figure 6 An embodiment of the present disclosure is shown using Figure 1 An exemplary process for presenting a through-image using a head-mounted display. DETAILED DESCRIPTION
[0012] A user wears a head mounted display to view and interact with content in a virtual reality environment. To provide an immersive experience, the head mounted display may cover most or even all of the user's field of view. In this way, the head mounted display may block the user's line of sight from the real world, which may cause the user to trip over objects, bump into furniture, fail to notice individuals in the real world, etc. While removing the head mounted display allows the user to see the real world, it requires the user to reposition themselves between the virtual reality environment and the real world, respectively. Some head mounted displays may include, for example, goggles that can be opened to allow the user to view the real world. However, this solution interrupts the user's sense of immersion in the virtual reality environment.
[0013] To overcome these deficiencies, some head-mounted displays may implement pass-through imaging, which allows the corresponding user to view the real world without removing their corresponding head-mounted display. However, existing pass-through imaging often exhibits rather rough response times, may not depict the real world from the user's perspective or viewpoint and / or may be distorted. As a result, the user may become unbalanced, dizzy, disoriented, or even sick.
[0014] The present application describes, in part, a head-mounted display (HMD) for a virtual reality (VR) environment. The systems and methods described herein may determine information about the real-world environment around a user, the user's location within the real-world environment, and / or the user's posture within the real-world environment. Such information may allow the HMD to display an image of the real-world environment in a direct manner without distracting the user from the VR environment. In some cases, the HMD may direct an image of the real-world environment based on one or more triggering events. For example, when a user wears the HMD and is immersed in a VR environment, the user may actuate a button that allows the user to look around in the real-world environment. As another example, if the user hears something suspicious or interesting, if the user wants to determine the location of an object (e.g., a controller) in the real-world environment and / or if a visitor enters an area near the user, the HMD may display content associated with the real-world environment. In some cases, content may be provided to the user in an aesthetically pleasing manner to limit the user's distraction from the VR environment. For example, content may be provided as an overlay of virtual content associated with the VR environment. In this case, the user may continue to wear the HMD and remain immersed in the VR environment. Therefore, the HMD according to the present application can enhance the user experience when converting or displaying content between the real world environment and the VR environment.
[0015] The HMD may include a front portion having a display that is worn on the user's face near the eyes. The display may output images (or other content) for viewing by the user. As an example, a user may wear the HMD to play games or view media content (e.g., a movie).
[0016] The HMD may include a camera that captures images of the real world environment. In some cases, the camera may be mounted to the display and / or may be housed within the front of the HMD. Additionally or alternatively, the camera may face forward to capture images outside the HMD and in front of the user. Furthermore, in some cases, the camera may be separate from the HMD and placed throughout the environment or elsewhere on the user (e.g., waist).
[0017] In some cases, the cameras may be spatially separated so that the optical axes of the cameras are parallel and separated by a known distance. Thus, the cameras may capture images of the real-world environment from slightly different viewpoints. The diversity of information between viewpoints may be used to compute depth information for the real-world environment (i.e., stereo camera imaging). For example, an HMD and / or a communicatively coupled computing device (e.g., a gaming console, a personal computer, etc.) may use image data captured by the cameras to generate depth information associated with the real-world environment.
[0018] For example, the camera may include a first camera and a second camera that are horizontally and / or vertically displaced from each other on the front of the HMD. In some cases, the first camera may be located on the front of a first side of the HMD, and the second camera may be located on the front of a second side of the HMD. However, as described above, the first camera and the second camera may be located elsewhere within the environment and / or on other parts of the user.
[0019] The image data captured by the first camera and / or the second camera may represent different views of a real-world environment (e.g., a room). By comparing the images (or image data) captured by the first camera and / or the second camera, the HMD (and / or other communicatively coupled computing device) may determine differences or disparities (e.g., using a disparity mapping algorithm). These differences may represent differences in coordinates of corresponding image points in the two images. Because these differences (or disparity values) are inversely proportional to the depth within the real-world environment, the HMD and / or other communicatively coupled computing device (such as a gaming console) may determine depth information associated with the real-world environment (or a portion thereof). In some cases, the depth information may come from the user's perspective (i.e., the user's gaze).
[0020] The HMD and / or other communicatively coupled computing devices (such as a gaming console) may use the depth information to generate a depth map or three-dimensional (3D) mesh of the real-world environment (or a portion thereof). For example, the depth map may represent the distance between the user and objects within the real-world environment (e.g., walls of a room, furniture, etc.). In addition or alternatively, the HMD may include other sensors for generating depth maps and / or 3D. For example, the HMD may include a depth sensor that determines the distance between the user and objects in the real-world environment, and may determine the proximity of the user to objects or boundaries of the real-world environment (e.g., a predetermined proximity or a threshold proximity). However, in addition or alternatively, the HMD or gaming console may use light detection and ranging (LIDAR), ultrasonic ranging, stereo ranging, structured light analysis, point projection, particle projection, time-of-flight observation, etc., to generate depth maps and / or 3D meshes.
[0021] After generating a depth map and / or 3D mesh of the real-world environment, the HMD and / or other communicatively coupled computing device (such as a gaming console) may project image data onto the depth map and / or 3D mesh. In this sense, the image data may first be used to generate the depth map and / or 3D mesh and secondly may be overlaid, superimposed, or projected onto the depth map and / or 3D mesh. In this case, the HMD may display content depicting the real-world environment to the user.
[0022] In some cases, the depth map, 3D mesh, and / or images captured by the first camera and / or the second camera may be modified to account for the user's pose or viewpoint. For example, because the camera may not be aligned with the user's eyes (e.g., horizontally, vertically, and in depth) (i.e., the camera is not in the exact location of the user's eyes), the depth map and / or 3D mesh may account for this difference. In other words, the image data captured by the first camera and the second camera may represent a viewpoint that is different from the user's viewpoint. Failure to account for this difference may show an imperfect real-world environment, and the user may find it difficult to pick up objects because the depth values or image data are not from the user's viewpoint. For example, because the camera captures image data from a different perspective and / or depth than that of the user's eyes (i.e., the camera is not in the same position as the user's eyes in the horizontal, vertical, and / or depth directions), the 3D mesh may account for this offset to accurately depict and present an undistorted view of the real-world environment to the user. That is, the depth map, 3D mesh, and / or image data may be modified based at least in part on the difference (or offset) in the coordinate position (or location) of the first camera and the second camera from the user's eyes (e.g., the first eye and / or the second eye) or viewpoint. Thus, the image of the real-world environment displayed to the user may accurately represent objects in the real-world environment from the user's viewpoint.
[0023] Additionally or alternatively, in some cases, the HMD and / or the real-world environment may include sensors that track the user's gaze, viewpoint, and / or field of view. For example, the HMD may include an interpupillary distance (IPD) sensor that measures the distance between the pupils of the user's eyes and / or other sensors that detect the direction of the user's eye gaze. Such sensors may be used to determine the user's viewpoint to accurately depict an image of the real-world environment to the user.
[0024] Through-imagery may allow a user to interact with and view objects in a real-world environment (such as coworkers, computer screens, mobile devices, etc.). In some cases, a user may switch or convert between a VR environment and a real-world environment, or the HMD may automatically convert between a VR environment and a real-world environment in response to one or more triggering events. That is, the HMD may include one or more modes, such as a through-mode that presents a real-world environment to a user and / or a virtual reality mode that presents a VR environment to a user. As an example, a user may want to drink water while wearing the HMD. Instead of taking off the HMD, the user may actuate (e.g., double-press) a button on the HMD that causes the real-world environment to be displayed. In turn, the display may present an image of the real-world environment to allow the user to locate their glass of water without removing the HMD. Later, after locating the glass of water, the user may actuate the button (e.g., single-press) to cause the display to present the virtual content of the VR environment. In this way, a through-image representing the real-world environment may allow the user to move around the real-world environment to determine the location of an object (e.g., furniture) without colliding with the object. As another example, a through-image may represent another individual entering the user's real-world environment. At this time, the HMD may detect the other individual and may present an image on the display so that the user can recognize or be aware of the other individual.
[0025] In some cases, content associated with the real-world environment displayed to the user may be partially transparent to maintain the user's sense of being in a VR environment. In some cases, content associated with the real-world environment may be combined with virtual content, or only content associated with the real-world environment may be presented (e.g., 100% through-imagery). In addition or alternatively, in some cases, the image captured by the camera may be presented on the entire display, or may be presented within a specific portion. Furthermore, in some cases, content associated with the real-world environment may be displayed with dotted lines to indicate to the user which content is part of the real-world environment and which content is part of the VR environment. Such presentation may allow the user to see approaching individuals, disturbances, and / or objects around the user. Regardless of the particular implementation or configuration, the HMD may be used to display content associated with the real-world environment to warn, detect, or otherwise identify objects that enter the user's field of view.
[0026] In some cases, the HMD and / or other computing devices associated with the VR environment (such as a gaming console) may operate in conjunction with a tracking system within the real-world environment. The tracking system may include sensors that track the position of a user within the real-world environment. When the user is immersed in the VR environment, such tracking may be used to determine information about the real-world environment surrounding the user, such as the user's position within the environment and / or the user's posture or viewpoint. Within the real-world environment, the tracking system may determine the user's position and / or posture. In some cases, the tracking system may determine the user's position and / or posture relative to the center of the real-world environment.
[0027] In some cases, the tracking system may include an illumination element that emits light (e.g., visible or invisible light) into the real-world environment and a sensor that detects the incident light. In some cases, in order to detect the position and / or posture of the user, the HMD may include a beacon. After projecting the light into the real-world environment, the beacon may reflect the light and the sensor may capture the incident light reflected by the beacon. The captured incident light can be used to track and / or determine the position of the beacon within the environment, which can be used to determine the position and / or posture of the user.
[0028] In some cases, the user's position and / or posture within the real-world environment can be used to present warnings, instructions, or content to the user. For example, if the user is approaching a wall of the real-world environment, knowing the position of the user and the wall (via the tracking system), the HMD can display an image representing the wall within the real-world environment. That is, in addition to or in lieu of using images captured by the camera to determine that the user is approaching the wall (i.e., via depth values), the tracking system can determine the relative position of the user within the real-world environment. Such tracking can help accurately present images that correspond to the depth (or placement) of objects within the real-world environment.
[0029] In addition, the images or data obtained from the tracking system can be used to generate a 3D model (or mesh) of the real-world environment. For example, knowing the position and / or posture of the user, HMD, tracking system, game console, and / or other communicatively coupled computing device can determine the relative position of the user within the real-world environment. This position and / or posture of the user can be used to determine the corresponding part of the 3D model of the real-world environment that the user is viewing (i.e., the user's field of view). For example, when the camera captures an image that is not related to the user's viewpoint, the tracking system can determine the user's gaze, posture, or viewpoint. Such information can be used to determine the location that the user is viewing within the real-world environment. Knowing the user's position in the real-world environment can be used to modify the image captured by the camera. In this case, the HMD can accurately display the real-world environment.
[0030] The HMD, tracking system, gaming machine, and / or other communicatively coupled computing device may also compare a depth map and / or 3D mesh generated using the camera's image data to the 3D model to determine the user's relative position within the real-world environment. Regardless of the particular implementation, knowing the user's position and / or pose within the real-world environment, the HMD and / or other communicatively coupled computing device may convert points of the depth map and / or 3D mesh onto the 3D model of the real-world environment. In turn, the image captured by the camera may be projected onto the 3D model corresponding to the user's viewpoint.
[0031] Therefore, in view of the above, the present application discusses an HMD that provides pass-through imaging to enhance the VR experience. Pass-through imaging can provide a relatively smooth experience when displaying the content of the VR environment and the content associated with the real-world environment. This pass-through imaging provides a less intrusive and disruptive solution for viewing content associated with the real-world environment. In some cases, in response to a trigger (including but not limited to motion, sound, gesture, preconfigured event, user movement change, etc.), information or content associated with the real-world environment may be selectively provided to the user. In addition, in some cases, the HMD according to the present application may take a variety of forms, including helmets, goggles, protective glasses, masks, glasses, and other head or eye accessories worn on the user's head.
[0032] The present disclosure provides an overall understanding of the principles of the structures, functions, devices, and systems disclosed herein. One or more examples of the present disclosure are shown in the accompanying drawings. Those of ordinary skill in the art will understand that the devices and / or systems specifically described herein and shown in the accompanying drawings are non-limiting embodiments. Features shown or described in conjunction with one embodiment may be combined with features of other embodiments, including features between systems and methods. Such modifications and variations are intended to be included within the scope of the appended claims.
[0033] Figure 1 A user 100 is shown residing within an environment 102 and wearing an HMD 104. In some cases, the user 100 may wear the HMD 104 to immerse the user 100 within the VR environment. In some cases, the user 100 may interact within the VR environment using one or more controllers 106. The HMD 104 includes a display 108 for providing virtual content and / or images to the user 100, and in some cases, an image capture device, such as a first camera 110 and / or a second camera 112.
[0034] The first camera 110 and / or the second camera 112 can capture images of the environment 102 and pass-through the images of the environment 102 to the user 100 for viewing on the display 108. That is, and as discussed in detail herein, images captured by the first camera 110 and / or the second camera 112 can be presented to the user 100 in a pass-through manner to allow the user 100 to view the environment 102 without having to leave the VR environment and / or take off the HMD 104.
[0035] In some cases, the first camera 110 and / or the second camera 112 may be disposed in or near the front of the HMD 104. In some cases, the first camera 110 and / or the second camera 112 may represent a stereo camera, an infrared (IR) camera, a depth camera, and / or any combination thereof. The images captured by the first camera 110 and / or the second camera 112 may represent the environment 102 surrounding the user 100. In some cases, the first camera 110 and / or the second camera 112 may face forward to capture images of the environment 102 in front of the user 100. In some cases, the first camera 110 and / or the second camera 112 may be spatially separated so that their optical axes are parallel. Thus, the images captured by the first camera 110 and / or the second camera 112 may represent the environment 102 from different angles and may be used to determine depth information associated with the environment 102 (i.e., stereo camera imaging). However, in some cases, the first camera 110 and / or the second camera 112 may be located elsewhere within the environment 102. For example, the first camera 110 and / or the second camera 112 may be located on the floor of the environment 102, may be located on a table within the environment 102, and so on.
[0036] As shown, the HMD 104 may include a processor 114 that performs or otherwise executes operations associated with the HMD 104. For example, the processor 114 causes the first camera 110 and / or the second camera 112 to capture images, and may subsequently receive images captured by the first camera 110 and / or the second camera 112, compare the images (or image data), and determine the difference between them. Since the difference is inversely proportional to the depth of the object within the environment 102, the processor 114 may determine depth information associated with the environment 102 (e.g., walls, furniture, television, etc.).
[0037] In some cases, processor 114 may use the depth information to generate a depth map or 3D mesh of environment 102. For example, as shown, HMD 104 includes memory 116 that stores or otherwise accesses a depth map 118 of environment 102 and / or a 3D mesh 120 of environment 102. Since the image data captured by first camera 110 and / or second camera 112 represents a portion of environment 102, depth map 118 and / or 3D mesh 120 may correspondingly represent a portion of environment 102. In some cases, after generating depth map 118 and / or 3D mesh 120, processor 114 may store depth map 118 and / or 3D mesh 120 in memory 116.
[0038] Because the image data, depth map, and / or 3D mesh captured by the first camera 110 and / or the second camera 112 are not from the perspective of the user 100 (i.e., the viewpoint of the user 100), the HMD 104 (and / or other communicatively coupled computing devices) may take into account the placement of the first camera 110 and / or the second camera 112 relative to the viewpoint of the user 100 (e.g., relative to the first eye and / or the second eye, respectively). That is, whether located on the HMD 104 or elsewhere within the environment, the first camera 110 and / or the second camera 112 do not capture images corresponding to the viewpoint of the user 100 and / or the viewpoint of the user 100. Therefore, points within the image data or depth map and / or 3D mesh may be modified or offset to account for the displacement.
[0039] In some cases, HMD 104 may operate in conjunction with tracking system 122. In some cases, HMD 104 may be communicatively coupled to tracking system 122 via network 124. For example, HMD 104 and tracking system 122 may each include one or more interfaces, such as network interface 126 and / or network interface 128, to facilitate wireless connection to network 124. Network 124 represents any type of communication network, including data and / or voice networks, and may be implemented using wired infrastructure (e.g., cable, CAT5, fiber optic cable, etc.), wireless infrastructure (e.g., RF, cellular, microwave, satellite, Bluetooth, etc.), and / or other connection technologies.
[0040] The tracking system 122 may include components that determine or track the pose of the user 100, the HMD 104, the first camera 110, and / or the second camera 112 within the environment 102. In this sense, the tracking system 122 may determine the position, orientation, and / or pose of the user 100, the HMD 104, the first camera 110, and / or the second camera 112 at the time when the first camera 110 and / or the second camera 112 captures an image of the environment 102 to pass through to the user 100. For example, the tracking system 122 (and / or other computing devices) may analyze and interpret the images captured by the tracking system 122 to identify the user 100 and / or the pose of the user 100 within the environment 102. For example, in some cases, the tracking system 122 may include a projector 130 and / or a sensor 132 that operates to determine the position, orientation, and / or pose of the user 100. As shown, and in some cases, the tracking system 122 may be mounted to a wall of the environment 102. Additionally or alternatively, tracking system 122 may be mounted elsewhere within environment 102 (eg, ceiling, floor, etc.).
[0041] Projector 130 is configured to generate and project light and / or images into environment 102. In some cases, the images may include visible light images perceptible to user 100, visible light images not perceptible to user 100, images with non-visible light, or a combination thereof. Projector 130 may be implemented with any number of technologies capable of generating and projecting images into / into environment 102. Suitable technologies include: digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, etc.
[0042] Sensors 132 may include: high-resolution cameras, infrared (IR) detectors, sensors, 3D cameras, IR cameras, RGB cameras, etc. Sensors 132 are configured to image environment 102 at visible wavelengths, non-visible wavelengths, or both. Sensors 132 may be configured to capture information for detecting depth, location, orientation, and / or posture of objects within environment 102. For example, as user 100 maneuvers around environment 102, sensor 132 may detect the position, orientation, and / or posture of user 100. In some cases, sensor 132 may capture some or all angles and positions within environment 102. Alternatively, sensor 132 may focus or capture images within a predetermined area of environment 102.
[0043] The projector 130 and / or the sensor 132 may operate in conjunction with the beacon 134 of the HMD 104. For example, the tracking system 122 may project light into the environment 102 via the projector 130, and the sensor 132 may capture an image of the reflection of the beacon 134. The tracking system 122 (such as the processor 136 of the tracking system 122) may use the captured image to determine distance information to the beacon 134. Additionally or alternatively, the tracking system 122 may detect a posture (e.g., orientation) of the user 100 within the environment 102. In some cases, the beacon 134 may be used to determine the viewpoint of the user 100. For example, the distance between the beacon 134 and the eyes of the user 100 may be known. When capturing the image data of the beacon 134, the tracking system 122 (and / or other communicatively coupled computing devices) may determine the relative viewpoint of the user 100. Thus, the tracking system 122 may utilize the beacon 134 of the HMD 104 to determine the relative position and / or pose of the user 100 within the environment 102 .
[0044] In order to define or determine characteristics about the environment 102, upon launching the gaming application, the HMD 104 may request the user 100 to define the boundaries, perimeter, or region of the environment 102 in which manipulations may be performed when the user 100 is immersed in the VR environment. As one example, the processor 114 may cause the display 108 to present instructions to the user 100 to allow the user to walk around the environment 102 and define the boundaries of the environment 102 (or region in which manipulations will be performed when the user 100 is immersed in the VR environment). As the user 100 walks around the environment 102, the HMD 104 may capture images of the environment 102 via the first camera 110 and / or the second camera 112, and the tracking system 122 may track the user 100. Among other things, after determining the boundaries of the environment 102, the tracking system 122 may determine the center location (e.g., origin) of the region. Knowing the center location of the region may allow the HMD 104 to correctly display the relative location of objects or scenes within the environment 102. In some cases, the center location may be represented as (0,0,0) in an (X,Y,Z) Cartesian coordinate system.
[0045] In some cases, the tracking system 122 may transmit the boundaries and / or center positions to the HMD 104. For example, the processor 114 of the HMD 104 may store the boundaries and / or center origin in the memory 116, the boundaries as indicated by the boundaries 138. Additionally or alternatively, in some cases, images captured by the first camera 110 and / or the second camera 112 may be associated with images captured by the tracking system 122. For example, when defining the area, using the images captured by the first camera 110 and / or the second camera 112, depth maps of the environment 102 may be generated. These depth maps may be correspondingly associated with certain positions / poses within the environment 102, as determined by tracking the user 100 throughout the environment 102. In some cases, these depth maps may be combined or otherwise used to generate a 3D model or mesh of the environment 102. Upon receiving subsequent image data from the HMD 104 and / or the tracking system 122, the position of the user 100 within the environment 102 may be determined, which may help determine depth information within the environment 102 and from the user's perspective, position, or pose. For example, because the image data captured by the first camera 110 and / or the second camera 112 does not correspond to the viewpoint of the user 100, the tracking system 122 may determine the viewpoint of the user 100 via images captured from the beacon 134. Using the viewpoint, the image data captured by the first camera 110 and / or the second camera 112 may be modified to represent the viewpoint of the user 100.
[0046] For example, when the user 100 is engaged in a VR environment and maneuvers around the environment 102, the tracking system 122 can determine the relative position of the HMD 104 within the environment 102 by comparing the reflected light from the beacon 134 to the center position. Using this information, the HMD 104, the tracking system 122, and / or other communicatively coupled computing devices (e.g., a gaming console) can determine the distance of the HMD 104 from the center position or the position of the HMD 104 relative to the center position. In addition, the HMD 104, the tracking system 122, and / or other communicatively coupled computing devices can determine a pose, such as a viewpoint of the user 100 within the environment 102. For example, the tracking system 122 can determine that the user 100 is looking at the ceiling, wall, or floor of the environment 102. Such information can be used to modify or otherwise take into account the position of the first camera 110 and / or the second camera 112 relative to the user's eyes or viewpoint.
[0047] In some cases, tracking system 122 may be coupled to a chassis having a fixed orientation, or the chassis may be coupled to actuator 140 so that the chassis is movable. Actuator 140 may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move a chassis or components of tracking system 122, such as projector 130 and / or sensor 132.
[0048] Additionally or alternatively, the HMD 104 may operate in conjunction with a remote computing resource 142. The tracking system 122 may also be communicatively coupled to the remote computing resource 142. In some examples, given that the remote computing resource 142 may have computing capabilities far exceeding those of the HMD 104 and / or the tracking system 122, the HMD 104 and / or the tracking system 122 may be communicatively coupled to the remote computing resource 142. Thus, the HMD 104 and / or the tracking system 122 may utilize the remote computing resource 142 to perform relatively complex analysis and / or generate an image data model (or mesh) of the environment 102. For example, the first camera 110 and / or the second camera 112 may capture image data, and the HMD 104 may provide the image data to the remote computing resource 142 over the network 124 for analysis and processing. In some cases, the remote computing resource 142 may transmit content to the HMD 104 for display. For example, in response to a triggering event (e.g., a button press) that configures the HMD 104 to pass through images captured by the first camera 110 and / or the second camera 112, the remote computing resource 142 may transmit a depth map, a 3D mesh, and / or a 3D model onto which the HMD 104 projects the images captured by the first camera 110 and / or the second camera 112. In this way, images captured by the first camera 110 and / or the second camera 112 for delivery to the user 100 may be transmitted to the remote computing resource 142 for processing, and the HMD 104 may receive content to be displayed on the display 108.
[0049] As shown, the remote computing resource 142 includes a processor 144 and a memory 146 that may store or otherwise access some or all of the components described with reference to the memory 116 of the HMD 104. For example, the memory 146 may access and utilize the depth map 118, the 3D mesh 120, and / or the boundary 138. Additionally or alternatively, the remote computing resource 142 may store or otherwise access the memory 148 of the tracking system 122.
[0050] In some cases, the remote computing resources 142 may be remote from the environment 102, and the HMD 104 may be communicatively coupled to the remote computing resources 142 via the network interface 126 and through the network 124. The remote computing resources 142 may be implemented as one or more servers, and in some cases may form part of a network-accessible computing platform that is implemented as a computing infrastructure of processors, memory, software, data access, etc., which may be maintained and accessed via a network (such as the Internet). The remote computing resources 142 do not require the end user to be aware of the physical location and configuration of the system providing the service. Common expressions associated with these remote computing resources 142 may include "on-demand computing", "software as a service (SaaS)", "platform computing", "network-accessible platform", "cloud service", "data center", etc.
[0051] In some cases, the environment 102 may include a computer 150 (or a gaming application, a gaming console, a gaming system) that is communicatively coupled to the HMD 104, the controller 106, the tracking system 122, and / or the remote computing resource 142 via a network 124 and / or wired technology. In some cases, the computer 150 may perform some or all of the processes described herein, such as those that may be performed by the HMD 104, the controller 106, the tracking system 122, and / or the remote computing resource 142. For example, the computer 150 may modify image data received from the first camera 110 and / or the second camera 112, or a depth map associated therewith, to account for the viewpoint of the user 100. In this case, the computer 150 may determine or receive an indication from the tracking system 122 regarding the viewpoint of the user 100 and / or the origin of the area where the user is located (e.g., the origin of the real world environment) and / or the origin of the virtual world. Thus, the computer may modify the image data received from the first camera 110 and / or the second camera 112 or the depth map associated therewith based on the instructions received from the tracking system 122. In some embodiments, the time (e.g., timestamp) at which the image data is captured by one or both of the cameras 110, 112 can be used to determine the time difference between the time when the image data is captured and the time when the captured image data is displayed. For example, the image data ultimately displayed to the user may be delayed by tens of milliseconds after the image data is captured by the camera 110 and / or 112. The image data may be adjusted to account for such time differences or inconsistencies, such as by modifying the pixel data (e.g., by rotation adjustment and / or translation adjustment) so that at the time when the image is displayed on the HMD, the images are presented as they would appear in the physical world, which avoids the situation where the presented images appear to lag behind the user's head movement. In some embodiments, the cameras 110 and / or 112 may be tracked to maintain timestamp information so that the pose of the camera at the time the image is captured is known, and so that the pose of the camera at the time the resulting image data is received and processed can be used to accurately represent the image in space without relying on camera motion. Furthermore, computer 150 may store or otherwise access some or all of the components described with reference to memories 116 , 146 , and / or 148 .
[0052] As used herein, a processor such as processors 114, 136, 144 and / or a processor of a computer may include multiple processors and / or processors with multiple cores. In addition, the processor may include one or more different types of cores. For example, the processor may include an application processor unit, a graphics processing unit, etc. In a specific implementation, the processor may include a microcontroller and / or a microprocessor. The processor may include a graphics processing unit (GPU), a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or in addition, the functions described herein may be at least partially performed by one or more hardware logic components. For example, but not limited to: exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. In addition, each processor in the processor may have its own local memory, which may also store program components, program data, and / or one or more operating systems.
[0053] As used herein, memory, such as memory 116, 146, 148 and / or memory of computer 150, may include volatile and non-volatile memory, removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program components, or other data. Such memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the required information and can be accessed by the computing device. The memory may be implemented as a computer-readable storage medium ("CRSM"), which can be any available physical medium that the processor can access to execute the instructions stored on the memory. In a basic implementation, the CRSM may include random access memory ("RAM") and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store the required information and can be accessed by the processor.
[0054] Figure 2A perspective view of an HMD 104 is shown. The HMD 104 may include a front portion 200 and a rear portion 202 that are secured to the head of the user 100. For example, the HMD 104 may include strands, cords, segments, straps, bands, or other components that are operably coupled to the front portion 200 of the HMD 104 and the rear portion 202 of the HMD 104. The front portion 200 includes a display 108 located in front of or above the user 100 to render images output by an application (e.g., a video game). As described above, the display 108 may output images (frames) viewed by the user 100 so that the user 100 perceives the images as if immersed in a VR environment.
[0055] In some cases, the front portion 200 of the HMD 104 may include the first camera 110 and / or the second camera 112. The first camera 110 and / or the second camera 112 may capture images external to the HMD 104 (i.e., of the environment 102) for the user 100 to view on the display 108. As described above, the optical axes of the first camera 110 and the second camera 112 may be parallel and separated by a predetermined distance to generate depth information of at least a portion of the environment 102 using the images captured by the first camera 110 and / or the second camera 112. However, in some cases and as described above, the HMD 104 may not include the first camera 110 and / or the second camera 112. For example, the first camera 110 and / or the second camera 112 may be located elsewhere within the environment 102.
[0056] In other cases, the first camera 110 and / or the second camera 112 capture images that may not correspond to the viewpoint of the user 100 (i.e., because the first camera 110 and / or the second camera 112 are not in the actual position of the eyes of the user 100). However, using the techniques described herein, the image data captured by the first camera 110 and / or the second camera 112 can be modified to account for the displacement of the first camera 110 and / or the second camera 112 relative to the viewpoint of the user 100.
[0057] For example, to determine the viewpoint of the user 100, the HMD 104 may include a beacon 134. Figure 2As shown, in some cases, the beacons 134 may include a first beacon 204(1), a second beacon 204(2), a third beacon 204(3), and / or a fourth beacon 204(4) disposed at a corner, an edge, or along a perimeter of the front portion 200. However, the beacons 134 may be located at other locations on the HMD 104, such as along the top, side, or back portion 202. In some cases, the beacons 134 may include infrared elements, reflectors, digital watermarks, and / or images that respond to electromagnetic radiation (e.g., infrared light) emitted by the projector 130 of the tracking system 122. Additionally or alternatively, the beacons 134 may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) that is captured by the sensor 132 of the tracking system 122. That is, the projector 130 projects light into the environment 102, and the beacons 134 may reflect the light. The sensor 132 may capture incident light reflected by the beacon 134, and the tracking system 122 or other communicatively coupled computing device (such as the remote computing resource 142) may track and map the location of the beacon 134 within the environment 102 to determine the movement, position, posture, and / or orientation of the user 100. Thus, the beacon 134 may be used to indicate the viewpoint of the user 100 for use in modifying, adjusting, or otherwise adapting image data from the first camera 110 and / or the second camera 112 before the image data is displayed to the user 100.
[0058] Figure 3 A user 100 wearing an HMD 104 within an environment 300 is shown. Figure 3 The display 108 of the HMD 104 is shown displaying virtual content 302 while the user 100 is wearing the HMD 104 and interacting within the VR environment. As described above, the HMD 104 can display the virtual content 302 while the user 100 moves around the environment 102.
[0059] The tracking system 122 may be located within the environment 300 to track the user 100 from one location to another and determine the viewpoint of the user 100 within the environment. For example, the tracking system 122 may utilize the beacon 134 on the HMD 104 to determine the pose (e.g., position and orientation) of the user 100. In some cases, the pose of the user 100 may be relative to the center location of the environment 300. For example, the center location may have coordinates (0,0,0), and the tracking system 122 (or remote computing resource 142) may use reflected light from the beacon 134 to determine the pose of the user 100 in a coordinate space (e.g., (X,Y,Z)). The tracking system 122 may also determine the depth of objects within the environment 300 from the perspective or viewpoint of the user 100. Additionally or alternatively, image data received from the first camera 110 and / or the second camera 112 may be used to determine the depth within the environment 300.
[0060] The HMD 104 may coordinate the display of the virtual content 302 with objects (e.g., furniture, walls, etc.) in the environment 300 and / or within the viewpoint of the user 100 and / or in front of the user 100. In other words, the objects of the environment 300 (or the real world) displayed to the user 100 may correspond to the viewpoint of the user 100 in the environment 300. For example, the HMD 104 may display the objects within the environment 300 using image data (e.g., a pass-through image) captured by the first camera 110 and / or the second camera 112 based at least in part on one or more triggering events. That is, the HMD 104 may be configured to display the image at a location on the display 108 that corresponds to the actual placement of the object in the environment 300. Using image data from the first camera 110 and / or the second camera 112 and a depth map generated of the 3D model or mesh of the environment 300, the pass-through image presented to the user 100 may show objects where they are actually placed within the environment 300 (i.e., at appropriate depth values) and from the user's 100 point of view, thereby allowing the user 100 to pick up an object (e.g., a controller, a glass of water, etc.). However, in some cases, image data from the first camera 110 and / or the second camera 112 may be continuously captured, and upon detecting a triggering event, the HMD 104 may display content associated with the environment 300 to the user 100.
[0061] like Figure 3 As shown, the triggering event may include determining that the user 100 is approaching or approaching a boundary of the environment 300. For example, the user 100 may be approaching a corner 304 between two walls 306, 308 of the environment 300. Knowing the location of the user 100 via the tracking system 122, the HMD 104 may display content on the display 108 to indicate that the user 100 is approaching the corner 304. For example, as shown, the indication of the corner 304 and the walls 306, 308 may be displayed on the display 108 as dashed or dotted lines. In this case, the user 100 is presented with an indication that the user 100 is about to walk into a wall. In some cases, and as shown in FIG. Figure 3 As shown, instructions or content related to the corner 304 and walls 306 , 308 may be superimposed (combined with virtual content displayed on the display 108 of the HMD 104 ) or presented on (or combined with) the virtual content 302 .
[0062] As described above, the image data captured by the first camera 110 and / or the second camera 112 may be modified to account for the viewpoint of the user 100. Additionally or alternatively, a depth map, 3D mesh, or model of the environment may take into account or factor in the positioning of the first camera 110 and / or the second camera 112 relative to the eyes or viewpoint of the user 100. As such, because the first camera 110 and / or the second camera 112 have a viewpoint that may be different from that of the user 100, the image data may be modified to adjust to the user's viewpoint before being presented on the HMD 104.
[0063] Although Figure 3 A particular implementation is shown in which the corner 304 and / or walls 306, 308 are displayed on the display 108, but the content may be presented in different ways, such as unmodified, modified for display, embedded, merged, stacked, split, re-rendered, or otherwise manipulated to be appropriately provided to the user 100 without interrupting the immersive virtual experience. For example, in some cases, the content may be displayed in a specific area of the display 108 (e.g., the upper right corner). In addition or alternatively, the display 108 may only present content associated with the corner 304 and the walls 306, 308 so as to have 100% pass-through. In some cases, the HMD 104 may fade content associated with the environment 300 on the display 108.
[0064] Figure 4 1 shows a user 100 wearing an HMD 104 within an environment 400. In some cases, the HMD 104 may display a pass-through image based at least in part on one or more triggering events. In some cases, the triggering event may include determining that a visitor 402 has entered the environment 400 and / or a predefined area within the environment 400. Figure 4 As shown, the display 108 depicts a visitor 402 approaching the user 100 wearing the HMD 104. At this point, the visitor 402 appears to be approaching the user 100, and as such, the display 108 may display images captured by the first camera 110 and / or the second camera 112. However, the images may first be modified to account for the difference in viewpoint between the first camera 110 and / or the second camera 112 and the viewpoint of the user 100. In some cases, the HMD 104 may display the visitor based at least in part on the visitor 402 being within the viewpoint of the user 100 and / or in front of the user as determined using the beacon 134 when the user 100 is wearing the HMD 104 or the visitor 402 enters within a threshold distance of the user 100. In some cases, the visitor 402 may be detected via a motion sensor, by analyzing image data from the first camera 110 and / or the second camera 112, via the tracking system 122, and the like.
[0065] Figure 5 and Figure 6A process according to an embodiment of the present application is shown. The process described herein is shown as a collection of boxes in a logical flow diagram, which represents a series of operations, some or all of which can be implemented in hardware, software, or a combination thereof. In the context of software, a box may represent a computer executable instruction stored on one or more computer readable media, which programs the processor to perform the described operations when executed by one or more processors. Typically, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. Unless otherwise specified, the order in which the boxes are described should not be understood as limiting. Any number of the described boxes can be combined in any order and / or in parallel to implement the process or an alternative process, and not all boxes need to be executed. For the purpose of discussion, reference is made to the environment, architecture, and system (such as with respect to the examples described in this article) described in the examples. Figures 1 to 4 These processes are described using the described environments, architectures, and systems, although these processes can be implemented in a variety of other environments, architectures, and systems.
[0066] Figure 5 An exemplary process 500 is shown for generating a 3D model, depth map, or mesh of an environment 102. In some cases, the process 500 may be performed by a remote computing resource 142. However, the HMD 104, the tracking system 122, and / or the computer 150 may alternatively perform some or all of the processes 500.
[0067] At 502, process 500 may transmit a request to define an area within environment 102. For example, remote computing resource 142 may transmit a request to HMD 104 requesting user 100 to define an area of environment 102. In some cases, the area may represent an area that user 100 intends to move around when immersed in the VR environment. In response to receiving the request, processor 114 of HMD 104 may present the request or information associated with the request on display 108. For example, the request may inform user 100 to wear HMD 104 and walk around environment 102 and define the area that the user intends to move around.
[0068] At 504, process 500 may transmit a request to track HMD 104 within environment 102 while user 100 is wearing HMD 104 to define an area. For example, remote computing resource 142 may transmit a request to tracking system 122 to track user 100 throughout environment 102 as user 100 defines an area. In some cases, tracking system 122 may track user 100 via projector 130 projecting images into environment 102 and sensor 132 capturing images of light reflected via beacon 134 of HMD 104. In some cases, remote computing resource 142 may transmit the request at 502 and the request at 504 at the same or substantially the same time.
[0069] At 506, process 500 can receive first image data from tracking system 122. For example, remote computing resource 142 can receive first image data from tracking system 122, where the first image data represents the location of HMD 104 and / or user 100 as user 100 walks around environment 102 to define an area. For example, the first image data can represent the location and / or pose of beacon 134 within environment 102. The first image data received by remote computing resource 142 can include a timestamp when the image was captured by tracking system 122 (or sensor 132).
[0070] At 508, process 500 may determine characteristics of the region. For example, remote computing resource 142 may determine a boundary or perimeter of the region based at least in part on receiving the first image data. Additionally or alternatively, remote computing resource 142 may determine a center or origin of the region based at least in part on the first image data. In some cases, the origin may be defined in 3D space having values (0,0,0) corresponding to X, Y, and Z coordinates in a Cartesian coordinate system, respectively. Thus, upon receiving subsequent image data from tracking system 122, remote computing resource 142 may determine a relative position of user 100 (and / or HMD 104) to the origin of the region.
[0071] At 510, the process 500 may receive second image data from the HMD 104. For example, when the user 100 defines an area of the environment 102, the remote computing resource 142 may receive the second image data from the HMD 104, where the second image data represents an image captured by the first camera 110 and / or the second camera 112. In some cases, the second image data received by the remote computing resource 142 may include a timestamp when the image was captured by the first camera 110 and / or the second camera 112. That is, when the first camera 110 and / or the second camera 112 captures the image, the other communicatively coupled computing device may determine a pose of the user 100 within the environment 102. For example, as described above, when the first camera 110 and / or the second camera 112 captures the image, the processor 136 of the tracking system 122 may cause the projector 130 to project the image into the environment 102. The beacon 134 of the HMD 104 may reflect light associated with the image, and the sensor 132 may capture the reflected image of the beacon 134. Such images may be used to determine the depth, position, orientation, and / or pose of the user 100 (and / or HMD 104 ) within the environment 102 , and may be associated with image data captured by the first camera 110 and / or the second camera 112 .
[0072] At 512, process 500 may generate a 3D model (or mesh) of environment 102. For example, remote computing resource 142 may generate a 3D model of environment 102 based at least in part on first image data and / or second image data. For example, remote computing resource 142 may use the second image data to compare images captured by first camera 110 and / or second camera 112, respectively, to determine differences. Further, remote computing resource 142 may determine depth information of environment 102 for use in generating the 3D model of environment 102. Additionally, remote computing resource 142 may utilize the first image data received from tracking system 122 to associate a depth map or depth values of environment 102 with areas and / or certain locations within environment 102. In some cases, remote computing resource 142 may compare or associate the first image data and the second image data (or a depth map generated thereby) using a timestamp when the first image data and / or the second image data were captured. In some cases, generating the 3D model of environment 102 may include depicting objects within the environment. For example, remote computing resource 142 can enhance the VR environment with objects (or volumes) in environment 102, such as a chair. In some cases, the 3D model can be transmitted to HMD 104 and / or tracking system 122, and / or stored in memory of HMD 104, tracking system 122, and / or remote computing resource 142.
[0073] Figure 6An exemplary process 600 for pass-through images on an HMD, such as HMD 104, is shown. In some cases, after generating a 3D model of environment 102, process 600 may continue from 512 of process 500. In other words, in some cases, after generating the 3D model, HMD 104 may display the pass-through images to enable user 100 to transition between the VR environment and the real world environment (i.e., environment 102). In some cases, process 600 may be performed by remote computing resource 142. However, HMD 104, tracking system 122, and / or computer 150 may additionally perform some or all of process 600.
[0074] At 602, the process 600 may receive first image data from the HMD 104. For example, the remote computing resource 142 may receive first image data from the HMD 104, wherein the first image data represents an image captured by the first camera 110 and / or the second camera 112. In some cases, the remote computing resource 142 may receive the first image data based at least in part on the HMD 104 detecting a triggering event, such as a button pressed by the user 100, a verbal command issued by the user 100, motion detected within the environment 102 (e.g., a visitor approaching the user 100), and / or the user 100 approaching or entering within a threshold distance of a boundary of the environment 102. In other cases, the remote computing resource 142 may continuously receive the first image data and may be configured to display the images based at least in part on the triggering event. For example, while the user 100 may be immersed in a VR environment, the user 100 may press a button of the HMD 104 to display content outside the HMD 104. In this sense, the HMD 104 may include a pass-through mode that displays images captured by the first camera 110 and / or the second camera 112. Thus, the processor 114 of the HMD 104 may cause the first camera 110 and / or the second camera 112 to capture an image of the environment 102 based at least in part on detecting a triggering expression.
[0075] At 604, process 600 may generate a depth map and / or 3D mesh based at least in part on the first image data. For example, remote computing resource 142 may generate a depth map and / or 3D mesh based at least in part on receiving the first image data from HMD 104 (i.e., using stereo camera imaging). Since the first image data represents a portion of environment 102, the depth map and / or 3D mesh may also correspond to a depth map and / or 3D mesh of a portion of environment 102. Additionally or alternatively, HMD 104 may generate the depth map and / or 3D mesh. For example, processor 114 may generate a depth map using stereo camera imaging after receiving image data from first camera 110 and / or second camera 112. In some cases, processor 114 may use the depth map to generate a 3D mesh of environment 102. Additionally, in some cases, processor 114 may store the depth map and / or 3D mesh (such as depth map 118 and / or 3D mesh 120) in memory 116.
[0076] In some cases, the HMD 104, the remote computing resource 142, and / or the computer 150 may modify the first image data received from the first camera 110 and / or the second camera 112 before generating the depth map and / or the 3D mesh. For example, because the first image data may not represent the viewpoint of the user 100, the first image data may be modified (e.g., translated, transformed, deflected, etc.) to account for the difference between the viewpoints of the first camera 110 and the second camera 112 and the viewpoint of the user 100. In some cases, the viewpoint of the user 100 may be determined using the location of the tracking system 122 and the beacon 134 within the environment 102. Additionally or alternatively, after the depth map and / or the 3D mesh are generated, the depth map and / or the 3D mesh may be modified according to or based at least in part on the viewpoint of the user 100.
[0077] At 606, process 600 can receive second image data representing a pose of HMD 104 from tracking system 122. For example, remote computing resource 142 can receive second image data corresponding to HMD 104 within environment 102 from tracking system 122. As described above, the second image data can be captured by sensor 132 of tracking system 122 that detects light reflected by beacon 134 of HMD 104 in response to projecting an image by projector 130.
[0078] At 608, process 600 may determine a pose of HMD 104. For example, remote computing resource 142 may determine a pose of HMD 104 within environment 102 based at least in part on receiving the second image data. In some cases, the pose may represent a position of HMD 104 within environment 102 and / or an orientation of HMD 104 within environment 102. That is, remote computing resource 142 may analyze the first image data and / or the second image data to determine a position of user 100 within environment 102 relative to a center of environment 102. Such analysis may determine a relative position, gaze, and / or viewpoint of user 100 relative to a center of environment 102 or a particular area within environment 102.
[0079] At 610, process 600 may convert the depth map and / or 3D mesh to points associated with the 3D model of the environment. For example, remote computing resource 142 may utilize the gesture (e.g., position and orientation) to determine the position of user 100 within environment 102 and / or the viewpoint of user 100 within environment 102. Remote computing resource 142 may convert points of the depth map and / or 3D mesh to points associated with the 3D model of environment 102, thereby taking into account the viewpoint of user 100. In other words, processor 144 of remote computing resource 142 may locate, look up, or determine depth values of points in the 3D model of environment 102 by converting points of the depth map and / or 3D mesh to points associated with the 3D model of environment 102. Remote computing resource 142 may translate the depth map and / or 3D mesh generated at 604 onto the 3D model of environment 102 using the gesture. Such translation may help accurately describe environment 102 to user 100 (e.g., appropriate depth values). That is, the first image data as well as the second image data and / or the 3D model of the environment 102 may provide an absolute position and / or gaze of the HMD 104 , which may help describe the viewpoint of the user 100 with an appropriate depth value or depth perception.
[0080] At 612, process 600 may project the first image data onto a portion of the 3D model to generate third image data. For example, knowing the posture of user 100, remote computing resource 142 may project or overlay the first image data onto a portion of the 3D model of environment 102 to generate third image data. In some cases, generating the third image data may include cross-blending or filling the depth values and / or color values for certain pixels of the third image data. For example, since the first image data may not represent the viewpoint of user 100, when generating the third image data describing the viewpoint of user 100, the third image data may have undefined depth values and / or color values for certain pixels. At this time, pixels without color values may be assigned color values from the color values of neighboring pixels or an average thereof. In addition or alternatively, a previous depth map or 3D mesh of environment 100, particle system, etc. may be used to generate the third image data.
[0081] At 614, process 600 can transmit the third image data to HMD 104. For example, after projecting the first image data onto a portion of the 3D model, remote computing resource 142 can transmit the third image data to HMD 104, where the third image data represents the first image data as projected onto the portion of the 3D model.
[0082] From 614, process 600 may loop to 602 to receive subsequent image data. Thus, in response to a continuous triggering event (e.g., a button press, a voice command, motion detection, etc.), HMD 104 may transition to a pass-through mode to provide a convenient way for user 100 to view environment 102 without having to remove HMD 104. For example, remote computing resource 142 may receive an indication from tracking system 122 that user 100 is approaching a boundary of an area and / or is about to hit a wall of environment 102. Upon receiving the indication, remote computing resource 142 may receive image data representing the user's viewpoint (i.e., images captured by first camera 110 and / or second camera 112) from HMD 104. After determining the pose of user 100, remote computing resource 142 may project the image data onto a 3D model of environment 102 and cause the image data to be displayed on HMD 104. In this sense, the image may be automatically "passed through" to user 100 to allow the user to see the real world environment without breaking the sense of immersion.
[0083] Although the foregoing invention has been described with reference to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. Since other modifications and variations to adapt to specific operating requirements and environments will be apparent to those skilled in the art, the invention should not be considered to be limited to the examples selected for disclosure purposes, and the invention encompasses all variations and variations that do not constitute a departure from the true spirit and scope of the invention.
[0084] Although the present application describes embodiments with specific structural features and / or method actions, it should be understood that the claims are not necessarily limited to the specific features or actions described. Instead, the specific features and actions are only examples of some embodiments falling within the scope of the claims of the present application.
Claims
1. A system, comprising: A head-mounted display, the head-mounted display comprising: monitor; First Camera; and Second camera; one or more processors; and One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions comprising: capturing, via the first camera, first image data representing a first portion of an environment; capturing, via the second camera, second image data representing a second portion of the environment; determining a first offset of the first camera relative to a first eye of a user and a second offset of the second camera relative to a second eye of the user; generating a depth map based at least in part on the first offset and the second offset; generating third image data based at least in part on the depth map, the first image data, and the second image data; and The third image data is displayed via the display.
2. The system according to claim 1, wherein: Generating the third image data includes generating the third image data based at least in part on projecting the first image data and the second image data onto the depth map.
3. The system according to claim 1, wherein: The head mounted display further includes a button, and wherein the third image data is displayed based at least in part on detecting a pressing of the button.
4. The system according to claim 1, wherein: The actions also include receiving an indication that the user is approaching a boundary of the environment, and wherein the third image data is displayed based at least in part on receiving the indication.
5. The system of claim 1, wherein the user comprises a first user, and wherein, The actions also include receiving an indication that a visitor is located within the environment, and wherein the third image data is displayed based at least in part on receiving the indication.
6. The system according to claim 1, wherein: The third image data is displayed in combination with virtual content presented on the display.
7. The system of claim 1, wherein: The first offset corresponds to at least one of: a horizontal displacement of the first camera relative to the first eye; a vertical displacement of the first camera relative to the first eye; or The first camera is displaced in depth relative to the first eye; and The second offset corresponds to at least one of: a horizontal displacement of the second camera relative to the second eye; a vertical displacement of the second camera relative to the second eye; or The second camera is displaced in depth relative to the second eye.
8. The system according to claim 1, wherein: Generating the third image data comprises: transmitting the depth map, the first image data, and the second image data to one or more computing devices; and The third image data is received from the one or more computing devices.
9. A method comprising: receiving, by a processor, first image data captured by a first camera of a head mounted display, the first image data representing a first portion of an environment; receiving, by the processor, second image data captured by a second camera of the head mounted display, the second image data representing a second portion of the environment; determining, by the processor, a first offset of the first camera relative to a first eye of a user and a second offset of the second camera relative to a second eye of the user; generating, by the processor, a depth map based at least in part on the first offset and the second offset; generating, by the processor, third image data based at least in part on the depth map, the first image data, and the second image data; as well as The third image data is displayed on the head mounted display.
10. The method according to claim 9, wherein: Generating the third image data includes generating the third image data based at least in part on projecting the first image data and the second image data onto the depth map.
11. The method of claim 9, further comprising receiving an indication of a triggering event, and wherein: The third image data is displayed based at least in part on receiving the indication.
12. The method according to claim 11, wherein: The instructions correspond to: detecting a press of a button of the head mounted display; detecting a voice command to switch the head mounted display from a virtual reality mode to a pass-through mode; detecting that the user is approaching a boundary of the environment; or A visitor is detected to be within the environment.
13. The method according to claim 9, wherein: The third image data is displayed in combination with virtual content presented on the head mounted display.
14. The method of claim 9, wherein: The first offset corresponds to at least one of: a horizontal displacement of the first camera relative to the first eye; a vertical displacement of the first camera relative to the first eye; or The first camera is displaced in depth relative to the first eye; and The second offset corresponds to at least one of: a horizontal displacement of the second camera relative to the second eye; a vertical displacement of the second camera relative to the second eye; or The second camera is displaced in depth relative to the second eye.
15. A system, comprising: Head-mounted displays; a first camera disposed in front of the head mounted display and facing forward to capture a first image of a real world environment located in front of the head mounted display; a second camera disposed in front of the head mounted display and facing forward to capture a second image of the real world environment in front of the head mounted display; one or more processors; and a memory storing computer executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions comprising: receiving first image data associated with the first image captured by the first camera, the first image data representing a first portion of the real-world environment; receiving second image data associated with the second image captured by the second camera, the second image data representing a second portion of the real-world environment; determining a first position of the first camera relative to a first eye of a user and a second position of the second camera relative to a second eye of the user; generating third image data based at least in part on the first position, the second position, the first image data, and the second image data; and The third image data is transmitted to the head mounted display.
16. The system of claim 15, wherein: The actions also include generating a 3D mesh of a portion of the real-world environment based at least in part on the first position and the second position; and Generating the third image data includes generating the third image data based at least in part on projecting the first image data and the second image data onto the 3D mesh.
17. The system of claim 15, wherein: The actions further include receiving an indication of a triggering event, and wherein transmitting the third image data is based at least in part on receiving the indication.
18. The system of claim 17, wherein: The instructions correspond to: detecting a press of a button of the head mounted display; detecting a voice command to switch the head mounted display from a virtual reality mode to a pass-through mode; detecting that the user is approaching a boundary of the real-world environment; or A visitor is detected to be within the real-world environment.
19. The system of claim 15, further comprising: transmitting data representing virtual content within the virtual reality environment to the head mounted display; The third image data is transmitted to the head mounted display for display in combination with the virtual content.
20. The system of claim 15, wherein: The first position corresponds to a displacement of the first camera relative to the first eye; and The second position corresponds to a displacement of the second camera relative to the second eye.
