System and method for providing spatial awareness in virtual reality

Abstract

In one embodiment, the method includes rendering, to one or more displays of the VR device, a first output image of the VR environment based on a field of view of the user. The VR environment has a virtual boundary corresponding to the real-world environment. The method further includes determining whether the user is approaching a threshold distance of the virtual boundary, determining a direction of movement and a field of view of the user, accessing one or more images of the real-world environment captured by one or more cameras of the VR device, and rendering, based on the accessed images, a second output image including a portion of the VR environment and a portion of a pass-through view of the real-world environment. The portion of the pass-through view is based on the determined direction of movement and the field of view of the user.

Description

Technical Field

[0001] The present disclosure generally relates to database and file management within a network environment, and more particularly to determining spatial awareness in a virtual reality (VR) setting.

Background Art

[0002] In conventional methods of spatial awareness in a VR setting, the user has to define a boundary wall that represents the outer boundary of a safe perimeter within which the user can move around. For example, the user can draw a line on the floor of a room (e.g., in the case of a room-scale VR setting) as the boundary, or (e.g., in the case of a stationary VR setting) can have a computer system that automatically defines a circular perimeter centered on the stationary sitting or standing user. When the user or the user's hand approaches the boundary, a virtual wall can appear to warn the user that they are approaching the boundary. In the case of a room-scale VR user, the user may have a small room in which the virtual wall is always displayed, disappointing the user experience and breaking the VR immersion. In the case of a stationary VR user, the user may always see the virtual wall when their head and / or hand moves, making some users feel claustrophobic and confined. Further, in some situations such as when the user is moving backward from the boundary, the virtual wall is not displayed within the user's field of view until it is too late, risking the user getting injured if they are moving too fast.

Summary of the Invention

[0003] In certain embodiments, a user of an immersive VR system may have their view of the real-world environment partially or completely obscured by the VR system, and therefore be at risk of bumping into or colliding with real-world objects while immersed in the VR environment. Furthermore, immersion in a VR environment can confuse the user regarding their position and / or orientation in the real-world environment. That is, the user may forget where they were standing or where nearby furniture or other objects are located. Thus, one technical challenge may include maintaining an immersive VR experience while conveying spatial information about the real-world environment to the user who is immersed in the VR experience. Conventional methods to keep the user safe and help the user orient themselves within the VR environment include drawing virtual boundaries, which may be lines drawn by the user that define a safe area for the user during the VR experience. When the user approaches a boundary, virtual boundary walls may appear or be activated. The system can use virtual boundary walls to warn the user where the virtual boundary is located. For example, these virtual boundary walls may have a grid-like appearance corresponding to lines drawn by the user that define the virtual boundary. However, these boundary walls can hinder immersion in the VR environment and detract from the user experience. One solution presented by the embodiments disclosed herein to address the technical challenge of conveying spatial information about the real world to the user may be to provide a “directional” passthrough view of the real-world environment within the VR environment as the user approaches the virtual boundary. The passthrough view can be thought of as “directional” in that the area and location of the passthrough view can be based on the user’s relative movement and field of view in the VR environment. While in a directional passthrough view, the user can see where the virtual boundary (e.g., a virtual line drawn by the user) is located to help the user stay in a safe zone.The technical advantages of the embodiments may include providing the user's pose (e.g., position and orientation) in a real-world environment, providing spatial information by showing the user a quick view of the real-world environment while maintaining the VR experience, and providing the user with visual information that can help the user avoid objects outside the boundary and help the user reorient themselves in the real-world environment. For example, a user walking forward may be approaching a desk that is outside the virtual boundary. Without completely breaking VR immersion, a portion of the user's field of view could transition from rendering the VR environment to rendering a directional passthrough view of the real-world environment (and thus the desk in the user's path) to help the user avoid jumping into the desk and reorient themselves in the center of the VR boundary. While this disclosure describes a method of providing spatial awareness in a VR setting using directional passthrough, this disclosure is intended to provide spatial awareness in a VR setting in any suitable way.

[0004] One aspect of the present invention provides a method comprising one or more computing systems, the method comprising rendering a first output image of a VR environment based on a user's field of view for one or more displays of a virtual reality (VR) display device, wherein the VR environment includes a virtual boundary corresponding to a real-world environment; determining whether the user has approached within a first threshold distance of the virtual boundary; determining the user's direction of movement and field of view in response to the user's approach within the first threshold distance of the virtual boundary; accessing one or more images of the real-world environment captured by one or more cameras of the VR display device; and rendering a second output image for one or more displays of the VR display device based on the accessed images, wherein the pass-through view portion is based on the user's determined direction of movement and field of view.

[0005] This method may further include rendering a third output image for one or more displays of a VR display device, which includes one or more real-world objects that cross virtual boundaries as one or more mixed reality (MR) objects.

[0006] The method may further include determining whether a user is approaching a second threshold distance of a virtual boundary, the second threshold distance being greater than a first threshold distance; determining the user's direction of movement and field of view in response to the user's approach within the second threshold distance of the virtual boundary; and accessing one or more additional images of a real-world environment, including one or more real-world objects captured by a camera of a VR display device, wherein the third output image may include one or more real-world objects in the accessed additional images.

[0007] One or more MR objects may be rendered as the outlines of one or more objects, as semi-opaque renderings of one or more objects, or as fully opaque renderings of one or more objects.

[0008] This method may further include determining the speed of the user's movement, the area of ​​the pass-through view portion may be based on the user's determined speed, and the portion may be a spherical cap of a spherical second output image.

[0009] The area of ​​the pass-through view may be relatively larger for faster determined speeds of user movement, and relatively smaller for slower determined speeds of user movement.

[0010] This method involves determining the speed of the user's movement, and may further include the fact that the sharpness of the transition from the VR environment to the pass-through view portion is based on the user's determined speed of movement, and the transition is a fade from the VR environment to the pass-through view portion.

[0011] The sharpness of the transition from the VR environment to the passthrough view portion may be relatively sharper for faster determined speeds of user movement, and relatively duller for slower determined speeds of user movement.

[0012] The rendered portion of the passthrough view can correspond to the user's direction of movement.

[0013] If the user's direction of movement is determined to be towards their field of view, the rendered portion of the passthrough view may be within the user's field of view.

[0014] If the direction of movement is determined to be perpendicular to the field of view, the rendered portion of the passthrough view may be within the user's surrounding view.

[0015] If the direction of movement is determined to be moving away from the field of view, the rendered portion of the passthrough view may be within the user's surrounding view and behind the user.

[0016] In one aspect of the present invention, one or more computer-readable non-temporary storage media are provided that, when executed, the software is operable to render a first output image of a VR environment based on the user's field of view for one or more displays of a virtual reality (VR) display device, the VR environment includes a virtual boundary corresponding to a real-world environment, determine whether the user is approaching within a first threshold distance of the virtual boundary, determine the user's direction of movement and field of view in response to the user's approach within the first threshold distance of the virtual boundary, access one or more images of the real-world environment captured by the camera of the VR display device, and render a second output image based on the accessed images for one or more displays of the VR display device, the second output image including a portion of the VR environment and a pass-through view portion of the real-world environment, the pass-through view portion being operable to be based on the user's determined direction of movement and field of view.

[0017] The software may, when executed, be further capable of rendering a third output image for one or more displays of a VR display, which includes one or more real-world objects that cross the virtual boundary as one or more mixed reality (MR) objects.

[0018] The software may, when executed, further operate to determine whether the user is approaching within a second threshold distance of the virtual boundary, and if the second threshold distance is greater than the first threshold distance, and in response to the user's approach within the second threshold distance of the virtual boundary, to determine the user's direction of movement and field of view, and to access one or more images of the real-world environment, including one or more real-world objects captured by the VR display device's camera.

[0019] One or more rendered MR objects may be rendered as one or more of the following: the outline of the object, a semi-opaque rendering of the object, or a fully opaque rendering of the object.

[0020] The software, once executed, may be further capable of determining the speed of the user's movement, and the area of ​​the pass-through view portion may be based on the speed determined by the user, and the area is an arc in the second output image.

[0021] The area of ​​the pass-through view may be relatively larger for faster determined speeds of user movement, and relatively smaller for slower determined speeds of user movement.

[0022] When the software is executed, it may determine the speed of the user's movement, and the sharpness of the transition from the VR environment to the passthrough view portion may also be based on the user's determined speed of movement, and may be further operable such that the transition is a fade from the VR environment to the passthrough view portion.

[0023] In one aspect of the present invention, a system is provided, the system comprising one or more processors and non-temporary memory coupled to the processors and containing instructions executable by the processors, wherein when the processor executes an instruction,

[0024] For one or more displays of a virtual reality (VR) display device, a first output image of a VR environment is rendered based on the user's field of view, the VR environment includes a virtual boundary corresponding to a real-world environment, it is determined whether the user is approaching within a first threshold distance of the virtual boundary, the user's direction of movement and field of view are determined in response to the user's approach within the first threshold distance of the virtual boundary, one or more images of the real-world environment captured by the VR display device's camera are accessed, and for one or more displays of the VR display device, a second output image is rendered based on the accessed images, including a portion of the VR environment and a pass-through view portion of the real-world environment, such that the pass-through view portion is based on the user's determined direction of movement and field of view.

[0025] There are certain technical challenges in determining spatial awareness in VR settings. One technical challenge can include communicating spatial information about the real-world environment and objects within the real-world environment to the user while the user is immersed in the VR experience. The solution presented by the embodiments disclosed herein to address this challenge can be to provide a quick overview through a directional pass-through view of the real-world environment so that the user can see where they are in the real-world environment. Another technical challenge can include providing the user with the visual information necessary to orient themselves towards a virtual boundary while maintaining immersion in the VR experience. The solution presented by the embodiments disclosed herein to address this challenge can be to render an opaque, translucent, or otherwise outlined rendering of real-world objects within the VR environment that can warn the user of the presence of real-world objects without significantly interrupting the VR experience.

[0026] The specific embodiments disclosed herein can provide one or more technical advantages. The technical advantages of the embodiments can include providing spatial information by providing a quick overview of the real-world environment through a directional pass-through view of the real-world environment while immersed in the VR environment, or providing a schematic rendering of real-world objects within the VR environment to warn the user of objects that may be in their path without significantly interfering with immersion in the VR experience. Another technical advantage of the embodiments can include providing spatial information by determining an optimal direction for the directional pass-through view regardless of the direction in which the user is moving. The specific embodiments disclosed herein may not provide any, some, or all of the above technical advantages. One or more other technical advantages may become readily apparent to those skilled in the art upon consideration of the drawings, description, and claims of this disclosure.

[0027] The embodiments disclosed in this specification are merely examples and the scope of the present disclosure is not limited thereto. A particular embodiment may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the present invention are disclosed in the appended claims, particularly directed to methods, storage media, systems, and computer program products, and any feature described in one claim category, for example, in a method, may also be claimed in another claim category, for example, in a system. The dependencies or references in the appended claims are merely selected for formal reasons. However, any subject matter as a result of an intentional forward reference (particularly, multiple dependencies) to any preceding claim is equally claimable, so that any combination of claims and their features is disclosed and claimable regardless of the dependencies selected in the appended claims. The claimable subject matter includes not only combinations of features described in the appended claims, but also any other combination of features in the claims, and each feature described in the claims may be combined with any other feature or combination of other features in the claims. Further, any of the embodiments and features described or shown herein may be claimed in a separate claim and / or in any combination with any of the embodiments or features described or shown herein or with any of the features of the appended claims.

Brief Description of the Drawings

[0028] [Figure 1A] FIG. Exemplary virtual reality system worn by a user according to a particular embodiment. [Figure 1B] FIG. Example of a pass-through function according to a particular embodiment. [Figure 1C] FIG. Virtual reality system in a real-world environment. [Figure 1D] FIG. Perspective view of a pass-through view of a real-world environment in a virtual reality environment. [Figure 2A]This is a top view of the passthrough view of the real-world environment within the virtual reality environment. [Figure 2B] This is a top view of the passthrough view of the real-world environment within the virtual reality environment. [Figure 2C] This is a top view of the passthrough view of the real-world environment within the virtual reality environment. [Figure 2D] This is a top view of the passthrough view of the real-world environment within the virtual reality environment. [Figure 3A] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, without any field of view adjustments. [Figure 3B] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, without any field of view adjustments. [Figure 3C] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, without any field of view adjustments. [Figure 3D] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, without any field of view adjustments. [Figure 4A] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, with field of view adjustments. [Figure 4B] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, with field of view adjustments. [Figure 4C] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, with field of view adjustments. [Figure 4D] This is a sample perspective view of a passthrough view of a real-world environment within a virtual reality environment, with field of view adjustments. [Figure 5A] This is a sample perspective view of compensation for adjusting the field of view. [Figure 5B] This graph shows the percentage compensation for the transition to the pass-through view. [Figure 5C] This graph shows the percentage compensation for the transition to the pass-through view. [Figure 6] This is a schematic sample diagram of user vision. [Figure 7]This is a user's perspective within the boundary space. [Figure 8] This is a perspective view of the contour rendering of a real-world object in a virtual reality environment. [Figure 9] This figure illustrates an exemplary method for determining spatial awareness in a VR setting using a pass-through view. [Figure 10] This figure shows an exemplary network environment associated with VR or a social networking system. [Figure 11] This is a diagram illustrating an exemplary computer system. [Modes for carrying out the invention]

[0029] In certain embodiments, a user of an immersive VR system (e.g., a head-mounted VR goggle) may have their view of the real-world environment partially or completely obscured by the VR system, and therefore be at risk of bumping into or colliding with real-world objects while immersed in the VR environment. Furthermore, immersion in a VR environment can confuse the user regarding their position and / or orientation in the real-world environment. That is, the user may forget where they were standing or where nearby furniture or other objects are located. Thus, one technical challenge may include maintaining an immersive VR experience while conveying spatial information about the real-world environment to the user who is immersed in the VR experience. Conventional methods to keep the user safe and help the user orient themselves within the VR environment include drawing virtual boundaries, which may be lines drawn by the user that define a safe area for the user during the VR experience. When the user approaches a boundary, a virtual boundary wall may appear or be activated. The system may use the virtual boundary wall to warn the user where the virtual boundary is located. For example, these virtual boundary walls may have a grid-like appearance corresponding to lines drawn by the user that define the virtual boundary. However, these boundary walls can hinder immersion in the VR environment and detract from the user's experience. One solution presented by embodiments disclosed herein to address the technical challenge of conveying spatial information about the real world to the user may be to provide a “directional” passthrough view of the real-world environment within the VR environment as the user approaches the virtual boundary. The passthrough view can be thought of as “directional” in that the area and location of the passthrough view can be based on the user’s relative movement and field of view in the VR environment. While in a directional passthrough view, the user can see where the virtual boundary (e.g., virtual lines drawn by the user) is located to help the user stay within the safe zone.The technical advantages of the embodiments may include providing the user's pose (e.g., position and orientation) in a real-world environment, providing spatial information by showing the user a quick view of the real-world environment while maintaining the VR experience, and providing the user with visual information that can help the user avoid objects outside the boundary and help the user reorient themselves in the real-world environment. For example, a user walking forward may be approaching a desk that is outside the virtual boundary. Without completely breaking VR immersion, a portion of the user's field of view could transition from rendering the VR environment to rendering a directional passthrough view of the real-world environment (and thus the desk in the user's path) to help the user avoid jumping into the desk and reorient themselves in the center of the VR boundary. While this disclosure describes a method of providing spatial awareness in a VR setting using directional passthrough, this disclosure is intended to provide spatial awareness in a VR setting in any suitable way.

[0030] Figure 1A shows an example of a virtual reality system 50 worn by user 102. In certain embodiments, the virtual reality system 50 may include a head-mounted VR display device 135, a controller 106, and a computing system 110. The VR display device 135 is worn over the user's eyes and can provide visual content to user 102 via internal displays (not shown). The VR display device 135 may have two separate internal displays, one for each eye of user 102 (a single display device is also possible). As shown in Figure 1A, the VR display device 135 can completely cover the user's field of view. The VR display device 135 achieves the goal of providing an immersive artificial reality experience by exclusively providing visual information to user 102. However, one consequence of this is that user 102 may not be able to see the physical (real-world) environment surrounding them because their field of view is obscured by the VR display device 135. Therefore, the pass-through functionality described herein may be technically advantageous for providing the user with real-time visual information about their physical surroundings.

[0031] Figure 1B shows an example of the pass-through function. User 102 may be wearing a VR display device 135 that immerses them in a virtual reality environment. Real-world objects 145 are located in the physical environment surrounding user 102. However, because the VR display device 135 obstructs user 102's vision, user 102 cannot directly see the real-world objects 145. To help the user perceive their physical environment while wearing the VR display device 135, the pass-through function captures information about the physical environment using, for example, the outward-facing cameras 105A-B described above. The captured information can then be reprojected to user 102 based on user 102's viewpoint. In a particular embodiment in which the VR display device 135 has a right display 136A for the user's right eye and a left display 136B for the user's left eye, the virtual reality system 50 can separately render (1) a reprojected view 145A of the physical environment for the right display 135A based on the viewpoint of the user's right eye, and (2) a reprojected view 145B of the physical environment for the left display 135B based on the viewpoint of the user's left eye.

[0032] Referring again to Figure 1A, the VR display device 135 may have outward-facing cameras such as the two forward-facing cameras 105A and 105B shown in Figure 1A. Although only the two forward-facing cameras 105A and 105B are shown, the VR display device 135 may have any number of cameras facing any direction (for example, an upward-facing camera to capture ceiling or room lighting, a downward-facing camera to capture the user's face and / or body parts, a backward-facing camera to capture parts of what is behind the user, and / or an inward-facing camera to capture the user's gaze for the purpose of eye-tracking). The outward-facing cameras may be configured to capture the physical environment around the user and may continuously do so to generate a series of frames (for example, as video). As previously described, the images captured by the forward-facing cameras 105A and 105B can be displayed directly to the user 102 via the VR display device 135, but even so, an accurate view of the physical environment would not be provided to the user. This is because it is not possible for cameras 105A and 105B to be physically positioned in exactly the same location as the user's eyes. Therefore, the pass-through function described herein can use a reprojection technique that generates a 3D representation of the physical environment and then renders an image based on that 3D representation from the user's perspective.

[0033] 3D representations can be generated based on depth measurements of physical objects observed by cameras 105A-105B. Depth can be measured in various ways. In certain embodiments, depth can be calculated based on stereo images. For example, two forward-facing cameras 105A-B can be configured to share overlapping fields of view and capture images simultaneously. As a result, the same physical object can be captured simultaneously by both cameras 105A-B. For example, a particular feature of an object may be represented by a single pixel p in the image captured by camera 105A. AThis may appear in another pixel p in the image captured by camera 105B. B This may appear. As long as the depth measurement system knows that two pixels correspond to the same feature, the virtual reality system 50 can use triangulation techniques to calculate the depth of the observed feature. For example, the position of camera 105A in 3D space and p relative to the field of view of camera 105A. A Based on the pixel location, from camera 105A, pixel p A It is possible to project lines through the other camera 105B to the pixels p B It is possible to project similar lines through the same area. Since both pixels will correspond to the same physical feature, these two lines should intersect. These two intersecting lines, along with the imaginary line drawn between the two cameras 105A and 105B, form a triangle, which can be used to calculate the distance of the observed feature from either camera 105A or 105B, or the point in space where the observed feature is located.

[0034] In certain embodiments, the orientation (e.g., position and orientation) of the VR display device 135 in the environment may be required. For example, in order to render an appropriate display to the user 102 while the user is moving around in the virtual environment, the virtual reality system 50 may need to determine the user's position and orientation at any time. Based on the orientation of the VR display device, the virtual reality system 50 can further determine the viewpoint of either camera 105A and 105B or the user's eyes. In certain embodiments, the VR display device 135 may be equipped with an inertial measurement unit ("IMU"). Data generated by the IMU, along with stereo images captured by outward-facing cameras 105A-B, allows the virtual reality system 50 to calculate the orientation of the VR display device 135 using, for example, SLAM (Simultaneous Localization and Mapping) or other suitable techniques.

[0035] In certain embodiments, the virtual reality system 50 may further include one or more controllers 106 that enable a user 102 to provide input. The controllers 106 can communicate with a VR display device 135 or a separate computing system 110 via a wireless or wired connection. The controllers 106 can have any number of buttons or other mechanical input mechanisms. In addition, the controllers 106 may have an IMU so that the orientation of the controllers 106 can be tracked. The controllers 106 can further be tracked based on a predetermined pattern on the controller. For example, the controllers 106 can have several infrared LEDs or other known observable features that collectively form a predetermined pattern. Using sensors or cameras, the virtual reality system 50 may be able to capture images of the predetermined pattern on the controller. Based on the observed orientation of these patterns, the system can calculate the position and orientation of the controller relative to the sensors or cameras.

[0036] The virtual reality system 50 may further include a computing system 110. The computing system 110 may be a standalone unit physically separated from the VR display device 135, or the computing system 110 may be integrated with the VR display device 135. In embodiments where the computing system 110 is a separate unit, the computing system 110 may be communicably coupled to the VR display device 135 via a wireless or wired link. The computing system 110 may be a high-performance device such as a desktop or laptop, or a resource-limited device such as a mobile phone. High-performance devices may have a dedicated GPU and a large or constant power supply. On the other hand, resource-limited devices may not have a GPU and may have a limited battery capacity. Therefore, the algorithms that can actually be used by the virtual reality system 50 depend on the capabilities of the computing system 110.

[0037] In embodiments where the computing system 110 is a high-performance device, one embodiment of the pass-through function can be designed as follows: A series of images of the surrounding physical environment can be captured via the outward-facing cameras 105A-B of the VR display device 135. However, the information captured by cameras 105A-105B may be misaligned with what the user's eyes can capture because the cameras cannot spatially match the user's eyes (for example, the cameras may be positioned some distance from the user's eyes, resulting in a different viewpoint). Therefore, simply displaying what the cameras capture to the user may not accurately represent what the user is supposed to perceive.

[0038] Therefore, the pass-through function can reproject the information captured by the outward-facing cameras 105A-B to the user, instead of simply displaying what has been captured. Using each pair of simultaneously captured stereo images, the depth of the observed features can be estimated. As described above, in order to measure depth using triangulation, the computing system 110 can find correspondences between the stereo images. For example, the computing system 110 can determine which two pixels in a pair of stereo images correspond to the same observed feature. The high-performance computing system 110 can solve the correspondence problem using GPU and optical flow techniques optimized for such tasks. The correspondence information can then be used to calculate depth using triangulation techniques. Based on the calculated depth of the observed features, the computing system 110 can determine where those features are located in 3D space (since the computing system 110 also knows where the cameras are located in that 3D space). The result can be represented by a dense 3D point cloud, where each point corresponds to an observed feature. Next, a dense point cloud can be used to generate 3D models of objects in the environment. When the system renders the scene for display, it can perform visibility tests from the user's point of view. For example, the system can project rays into 3D space from the viewpoint corresponding to each of the user's eyes. In this method, the rendered scene displayed to the user can be calculated from the viewpoint of the user's eyes, rather than from the viewpoint of the outward-facing cameras 105A-B.

[0039] However, the processes described above may not be feasible for resource-constrained computing units (for example, a mobile phone may be the primary computing unit for a VR display device). Unlike systems with powerful computing resources and abundant energy sources, mobile phones cannot rely on GPUs and computationally expensive algorithms (e.g., optical flow) to perform depth measurements and generate accurate 3D models of the environment. Therefore, optimized processes are required to provide passthrough on resource-constrained devices.

[0040] In certain embodiments, the computing device may be able to generate depth measurements using (1) a GPU and optical flow, or (2) an optimized technique using a video encoder and motion vectors, or be configured to dynamically determine at runtime whether it can generate such measurements, as will be described in more detail below. For example, if the device has a GPU and a sufficient power budget (e.g., the device is plugged into a power source, has a full battery, etc.), the device may be able to perform depth measurements using its own GPU and optical flow. However, if the device does not have a GPU or has a tight power budget, the device may be able to choose an optimized method for calculating depth.

[0041] Figure 1C shows a virtual reality system 50 within a real-world environment 100. Within the real-world environment 100, there may be a camera 105 (e.g., one or more cameras, such as a forward-facing camera on an AR / VR headset). The camera 105 can be connected to a computing system 110. The camera 105 may be worn by a user (e.g., as part of a VR headset). The camera 105 may be connected to a VR display device 135 (in certain embodiments, the camera 105 and the VR display device 135 may be separate). In certain embodiments, the computing system 110 can render a first output image of the VR environment 140 based on the user's field of view 120 for one or more displays of the VR display device 135. The VR environment 140 may have a virtual boundary 115 corresponding to the real-world environment 100. The VR environment 140 may be a VR game, a VR office, or other VR setting displayed in the user's field of view 120. The virtual boundary 115 can define or draw the edge of a safe area for the user to explore while the user is immersed in the VR environment 140. For example, in a room-scale VR environment (where the user can walk around the room during the VR experience), the virtual boundary 115 can correspond to real-world objects 145 (e.g., sofas, chairs, tables, walls, obstacles, etc.) that the user wearing the VR display device 135 would want to avoid while immersed in the VR experience. As another example, in a static VR setting (e.g., when the user is standing or sitting in place without crossing), the virtual boundary 115 can correspond to real-world objects (e.g., within a 1-meter radius around the user) that are within or just beyond the user's arm's reach.The virtual boundary 115 may be drawn by the user (for example, by having the user manually draw the virtual boundary 115 using the controller 106), determined automatically (for example, an image processor may determine a safe boundary and automatically determine boundary walls), or determined semi-automatically (for example, an image processor may determine or suggest a safe boundary and boundary walls 115, and the user may manually extend or edit the determined boundary walls 115). While this disclosure describes the use of a particular virtual reality system in a particular real-world environment, this disclosure assumes the use of any suitable virtual reality system in any suitable real-world environment.

[0042] In certain embodiments, the computing system 110 can determine whether the user is approaching within a first threshold distance of the virtual boundary 115. The computing system 110 can use sensors, accelerometers, gyroscopes, or other position sensors of the camera 105 and / or the VR display device 135 to determine whether the user is approaching within a first threshold distance of the virtual boundary 115. The first threshold distance may be a predetermined distance from the virtual boundary 115 (e.g., 1, 5, 10 meters). The first threshold distance may also be determined by the user. As an example, and not an limitation, in a room-scale VR setting, the computing system 110 can determine whether the user is approaching within a predetermined distance of the virtual boundary 115. As another example, and not an limitation, in a static VR setting, the first threshold distance may be when the user's head or hands approach the edge of a predetermined radius around the user (e.g., when the user's head or hands approach a predetermined radius of 1 meter). While this disclosure describes a specific method for determining whether a user is approaching a particular threshold distance of a virtual boundary, this disclosure assumes that a user may be approaching any suitable threshold distance of a virtual boundary in any suitable method.

[0043] Figure 1D shows a perspective view of a passthrough view 130 of the real-world environment 100 within a VR environment 140. The VR environment 140 can be rendered within the real-world environment 100. As the user approaches the virtual boundary, portions of the passthrough view 130 may appear to show the user real-world objects (e.g., real-world object 145) that the user may encounter. Thus, as the user approaches the virtual boundary, portions of the rendering of the VR environment 140 may transition to a rendering showing portions of the passthrough view 130, or appear as a rendering. The portions of the passthrough view 130ca show the user the real-world environment 100, which may have real-world objects in the user's path, in order to prevent the user from being harmed by showing the user real-world objects that may exist beyond the virtual boundary if the user continues along the path. The rendering shown in Figure 1D can be presented to the user via a VR display device 135 shown in Figure 1B. That is, portions of the passthrough view 130 may be used to capture information about real-world objects 145 within the real-world environment 100 and reproject it to the user.

[0044] Figures 2A to 2D show top views of pass-through views 130 of the real-world environment 100 within the VR environment 140. In certain embodiments, the computing system 110 can determine the user's direction of movement 125 and field of view 120 in response to a user approaching within a first threshold distance of the virtual boundary 115. The computing system 110 can determine the user's direction of movement 125 and field of view 120 using sensors, accelerometers, gyroscopes, or other position sensors of the camera 105 and / or the VR display device 135 to determine the movement and orientation of a user wearing the camera 105 and / or the VR display device 135. As an example, but not limited to, the sensor may determine that the user is moving forward (e.g., relative to direction of movement 125a) along the same direction as their field of view 120 (Figure 2A). As another example, but not limited to, the sensor may determine that the user is moving backward (e.g., relative to direction of movement 125b) in the opposite direction to their field of view 120 (Figure 2B). As an example, but not limited to, the sensor may determine that the user is moving laterally (for example, with respect to direction 125c) and perpendicular to the user's field of view 120 (Figure 2C). As an example, but not limited to, the sensor may determine that the user is moving laterally to the right (for example, with respect to direction 125d) and perpendicular to the user's field of view 120 (Figure 2D). While this disclosure describes determining the user's direction of movement and field of view, it assumes that the user's direction of movement and field of view may be determined by any suitable method.

[0045] In certain embodiments, the computing system 110 can access one or more images of the real-world environment 100 captured by one or more cameras 105 of the VR display device 135. The computing system 110 can access one or more images of the real-world environment 100 by capturing images of the user's real-world environment 100 using the cameras 105 (for example, by taking photographs or snapshots). These captured images may be partial images of the real-world environment 100 (for example, the camera captures only images of a desired orientation, such as the user's field of view or peripheral view) or complete images of the real-world environment 100 (for example, the camera captures a complete 360-degree image of the entire surroundings of the user's real world). While this disclosure describes accessing one or more images of the real-world environment in a particular manner, this disclosure assumes that one or more images of the real-world environment may be accessed in any suitable manner.

[0046] In certain embodiments, the computing system 110 can render a second output image for one or more displays of the VR display device 135, based on the accessed image, which includes a portion of the VR environment 140 and a portion of a passthrough view 130 of the real-world environment 100. The portion of the passthrough view 130 can be based on a determined direction of movement 125 and the user's field of view 120. The passthrough view 130 provides the user with a view of the real-world environment 100 beyond the VR environment 140 without dramatically compromising the immersion of the VR environment as the user approaches the virtual boundary 115. That is, as the user approaches the virtual boundary 115, the directional passthrough view 130 can be displayed on the VR display device 135 to give the user a sense of the direction in which they are moving in the real-world environment 100, while maintaining the VR environment 140 elsewhere. Therefore, the computing system can provide a solution to the technical challenge of conveying spatial information about the real-world environment 100 and real-world objects 145 within the real-world environment 100 to the user while the user is immersed in a VR experience in the VR environment 140. The solution presented herein can address this challenge by providing a quick look into the real-world environment 100 through a portion of the directional pass-through view 130, so that the user can determine where they are in the real-world environment 100. This can have the advantage of helping the user avoid objects they might encounter as they continue along a trajectory or path, and can also help the user orient themselves in the real-world environment 100. For example, the user can reposition themselves to the center of the virtual boundary 115 after looking through the directional pass-through view 130. Rather than being limited, as an example, referring to Figure 2A, if the user is moving forward along the same direction as their field of view 120 (for example, relative to the direction of movement 125a), the VR display device 135 can display the VR environment 140 and the portion of the pass-through view 130a directly in front of the user, while maintaining the VR environment 140 elsewhere.As an alternative example, not limited to this, referring to Figure 2B, if the user is moving backward in the opposite direction to their field of view 120 (for example, relative to the direction of movement 125b), the VR display device 135 can display the VR environment 140 and a portion of the pass-through view 130b that covers the user's peripheral field of view and the area behind the user, while maintaining the VR environment 140 elsewhere. As an alternative example, not limited to this, referring to Figure 2C, if the user is moving laterally (for example, relative to the direction of movement 125c) and perpendicular to the field of view 120, the VR display device 135 can display the VR environment 140 and a portion of the pass-through view 130c that covers the user's peripheral field of view on the left, while maintaining the VR environment 140 elsewhere. As an alternative example, and not an limitation, referring to Figure 2D, if the user is moving laterally to the right (for example, relative to the direction of movement 125d) and perpendicular to the field of view 120, the VR display device 135 can display the VR environment 140 and a portion of the pass-through view 130d that covers the user's peripheral field of view to the right, while maintaining the VR environment 140 elsewhere. Thus, the computing system 110 can render the pass-through view 130 to alert the user to objects in their path and help the user orient themselves within the room. Thus, embodiments can include the technical advantage of providing spatial information by providing a quick view of the real-world environment via an directional pass-through view of the real-world environment while immersed in the VR environment. While this disclosure describes rendering a particular output image in a particular way, this disclosure assumes that any suitable output image may be rendered in any suitable way.

[0047] Figures 3A to 3D show sample perspective views of portions of the passthrough view 130 of the real-world environment 100 within the VR environment 140, without adjustment of the field of view 120. Figures 4A to 4D show sample perspective views of portions of the passthrough view 130 of the real-world environment 100 within the VR environment 140, with adjustment of the field of view 120. Portions of the passthrough view 130 can be adjusted to consistently notify the user when they deviate from the center of the virtual boundary or when they approach the virtual boundary from any direction. This can be achieved by increasing the size of the arc or area of ​​the passthrough view 130 to ensure that at least a portion of the passthrough view 130 is always within the field of view 120. This can provide a technical advantage in that it provides spatial information by determining the optimal orientation of the directional passthrough view, regardless of the direction in which the user is moving. Not limited to, but as an example, referring to Figure 3A, if the user is moving forward in a direction 125 along the same direction as their field of view 120, the portion of the pass-through view 130 may be entirely within the field of view 120 and therefore may not require adjustment. Thus, referring to Figure 4A, no adjustment may be necessary for the portion of the pass-through view 130. Not limited to, but as another example, referring to Figure 3B, if the user is moving in a direction 125 away from the field of view 120, the portion of the pass-through view 130 and the field of view 120 will not completely overlap, and therefore the portion of the pass-through view 130 may require adjustment. Thus, referring to Figure 4B, an adjustment may be made to increase a portion of the pass-through view 130. For example, if the overlap between the field of view 120 and the portion of the pass-through view is 5 degrees (similar to Figure 3B), the size of the portion of the pass-through view 130 can be increased by 30 degrees so that the overlap between the field of view 120 and the portion of the pass-through view 130 can be increased to 20 degrees (similar to Figure 4B). As an alternative example, and not limited to this case, see Figure 3C. If the user is moving in a direction 125 perpendicular to their field of view 120, the portion of the pass-through view 130 and the field of view 120 may not overlap at all, and therefore the portion of the pass-through view 130 may require adjustment.Therefore, referring to Figure 4C, there may be an adjustment to increase a portion of the pass-through view 130. For example, if the overlap between the field of view 120 and the pass-through view portion is 0 degrees (similar to Figure 3C), the size of the pass-through view portion can be increased by 32 degrees so that the overlap between the field of view 120 and the pass-through view portion of 130 can be increased to 15 degrees (similar to Figure 4C). As another example, and not an limitation, referring to Figure 3D, if the user is moving in the opposite direction 125 to the field of view 120, there may be no overlap at all between the pass-through view 130 and the field of view 120, and therefore the pass-through view portion of 130 may require adjustment. Therefore, referring to Figure 4D, there may be an adjustment to increase a portion of the pass-through view 130. For example, if the overlap between the field of view 120 and the pass-through view portion is 0 degrees (similar to Figure 3D), the size of the pass-through view portion 130 can be increased by 190 degrees so that the overlap between the field of view 120 and the pass-through view portion 130 can be increased to 15 degrees (similar to Figure 4D). By performing the above field of view adjustments on the pass-through view portion 130, the computing system 110 can render a pass-through view 130 that can alert the user to objects in the path and help the user orient themselves within the room.

[0048] Figure 5A shows a sample perspective view of compensation 160 for adjusting the field of view 120. The direction of the field of view 120 can be represented by a forward vector 165. The direction of the user from the center of a virtual boundary 115 (not shown) can be represented by a direction vector 155. Based on the forward vector 165 and the direction vector 155, compensation 160 can be determined to adjust the portion of the pass-through view 130. For example, as the angle between the forward vector 165 and the direction vector 155 increases (for example, as the angle of the arc 170 decreases), compensation 160 may also increase. Compensation 160 is the angle at which the direction vector 155 may need to rotate to reach the field of view 120 in order to ensure, for example, that at least a portion of the pass-through view 130 is within the field of view 120. The increase in compensation 160 may be proportional to the decrease in the arc 170.

[0049] Figures 5B and 5C are graphs showing the percentage compensation for the transition to a pass-through view. The x-axis represents the distance between the pass-through view portion and the field of view, where x=0 represents the pass-through view portion and the field of view does not overlap, and x=1 represents the pass-through view portion and the field of view is in the opposite direction (for example, the pass-through view portion is facing away from the field of view). The y-axis represents the percentage compensation for the pass-through view portion. As shown by the percentage compensation line 161 in Figure 5B, pass-through compensation begins when the user's direction of movement goes outside the user's field of view (for example, when the percentage compensation line 161 crosses the y-axis). Because abrupt transitions to pass-through view portions can be very noticeable, abrupt transitions can interrupt the VR experience or distract from the VR experience. As shown by the percentage compensation line 162 in Figure 5C, the transition to a pass-through view may be more gradual. Compensation for the pass-through view can be gradually initiated even before the user's direction of movement leaves the user's field of view (for example, before the percentage compensation line 162 crosses the y-axis), thus smoothing the transition to the pass-through view. The advantages of having a smooth transition to the pass-through view include less disruption to the pass-through view and a more gradual introduction, which makes it less noticeable when determining when the compensation starts and ends. Furthermore, as the curve approaches x=1 (for example, when the portion of the pass-through view and the field of view are in opposite directions), compensation can increase the pass-through view to all sides of the user's vision, as viewing the pass-through view on all sides around the user makes it easier to notice objects and obstacles than simply showing the pass-through view on one side.

[0050] In certain embodiments, referring again to Figures 2A-2D, 3A-3D, and 4A-4D, the computing system 110 can determine the speed of the user's movement. The area (e.g., size) of the portion of the passthrough view 130 can be based on the user's determined speed. The portion of the passthrough view 130 may be a spherical cap of a spherical second output image (for example, the portion of the passthrough view is a spherical portion, and the sphere corresponds to a VR environment rendered on a "spherical dome" around the user). If the user is moving at a faster speed, the area of ​​the portion of the passthrough view 130 may be relatively larger for a faster determined speed of the user's movement than for a user moving at a slower determined speed. Thus, the portion of the passthrough view can occupy a larger area of ​​the output image displayed by the VR display device 135. If the user is moving at a slower speed, the area of ​​the portion of the passthrough view 130 may be relatively smaller for a user moving at a faster determined speed than for a user moving at a faster determined speed. Therefore, the pass-through view portion may occupy a smaller area of ​​the output image displayed by the VR display device 135. If the output image is rendered as a spherical dome around the user, the VR pass-through view portion may be a spherical cap within the spherical dome. However, the pass-through view portion 130 may be of any shape (for example, the shape is not limited to a circular view of the pass-through view portion 130). Not limited, but as an example, if the user is walking at a fast pace towards the virtual boundary 115, the pass-through view portion 130 may appear relatively larger than if the user were walking at a slower pace towards the virtual boundary 115. Conversely, if the user is walking slowly towards the virtual boundary 115, the pass-through view portion 130 may appear relatively smaller than if the user were walking at a faster pace towards the virtual boundary 115. While this disclosure describes determining the speed of the user's movement in a particular way, this disclosure assumes that the speed of the user's movement may be determined in any suitable way.

[0051] In certain embodiments, the sharpness of the transition from the VR environment 140 to the portion of the pass-through view 130 may be based on a determined speed of user movement. The transition may be a fade, blur, or other form of visual interruption or transition from the VR environment 140 to the portion of the pass-through view 130. That is, the transition from the VR environment 140 to the portion of the pass-through view 130 may include fading or blurring the edges where the VR environment 140 and the portion of the pass-through view 130 meet. The sharpness of the transition from the VR environment 140 to the portion of the pass-through view 130 may be sharper relative to a faster determined speed of user movement, and the sharpness of the transition from the VR environment 140 to the portion of the pass-through view 130 may be duller relative to a slower determined speed of user movement. Not limited to this, but as an example, if the user is walking quickly towards the virtual boundary 115, there will be less fading or blurring from the VR environment 140 to the passthrough view 130 (the transition from the VR environment 140 to the passthrough view 130 will be relatively abrupt). This allows the user to quickly assess potential obstacles in their path, as faster movement increases the likelihood or risk of tripping over or bumping into objects. Conversely, if the user is walking slowly towards the virtual boundary 115, there will be more fading or blurring from the virtual environment 140 to the passthrough view 130 (the transition from the VR environment 140 to the passthrough view 130 will be relatively slower). This allows the user to use the passthrough view to assess their position in the real-world environment without significantly compromising their VR experience (and thus minimizing disruption to VR immersion and the experience). While this disclosure describes determining the user's movement speed in order to determine the sharpness of a transition in a particular manner, this disclosure assumes that the user's movement speed may be determined in order to determine the sharpness of a transition in any suitable manner.

[0052] Figure 6 shows a sample schematic diagram of a user's vision 200. The user may have central vision 205, paracentral vision 210, macular vision 215, near peripheral vision 220, mid-peripheral vision 225, and far peripheral vision 230. Because human peripheral vision may only be able to detect high-contrast motion, it may be advantageous to increase sharpness in the user's peripheral vision while maintaining a decrease in sharpness near the center of the field of view (e.g., increased blurring or "feathering") to minimize disruption in the VR experience. That is, for a headset field of view 235, the sharpness 245 of the passthrough view gradient 240 in the user's peripheral vision (e.g., near peripheral vision 220, mid-peripheral vision 225, and far peripheral vision 230) can be increased to allow for greater visibility of real-world environments where only high-contrast motion may be detected. On the other hand, for the headset field of view 235 corresponding to central vision 205, paracentral vision 210, and macular vision 215, the sharpness 245 of the passthrough view gradient 240 can be reduced (for example, by increasing the blur or "feathering" gradient 240). Thus, the user can more easily detect visual objects and obstacles in the real-world environment in the user's peripheral vision (for example, their near peripheral vision 220, mid-peripheral vision 225, and far peripheral vision 230), with the added benefit of reducing the stretching of the passthrough view near central vision 205, paracentral vision 210, and macular vision 215. This allows the user to be provided with greater visual clarity of their surroundings without significantly interrupting or disrupting the VR experience.

[0053] In certain embodiments, referring to Figures 2A-2D, 3A-3D, and 4A-4D, the rendered portion of the passthrough view 130 can correspond to the user's direction of movement 125. The position of the portion of the passthrough view 130 as displayed on the VR display device 135 can correspond to the user's direction of movement 125. If the user is walking forward and straight, the portion of the passthrough view 130 can appear straight ahead of the user, centered in the user's field of view 120, along the direction of movement 125. If the user's direction of movement 125 is slightly forward and to the left, the portion of the passthrough view 130 as displayed on the VR display device 135 may appear in front of and slightly to the left of the center of the user's field of view 120, along that direction of movement 125. For example, if the user's direction of movement 125 is determined to be toward the field of view 120, the rendered portion of the passthrough view 130 may be within the user's field of view 120. As an alternative, and not limited, example, if the direction of movement 125 is determined to be perpendicular to the field of view 120, the rendered portion of the passthrough view 130 may be within the user's peripheral view. That is, the field of view may be to the left or right of the user's field of view 120 in the user's peripheral area. This may allow the user to be alerted to potential obstacles or objects that may be present in their path using their peripheral view. As another, and not limited, example, if the direction of movement 125 is determined to be away from the field of view 120, the rendered portion of the passthrough view 130 may be in the user's peripheral view and behind the user. That is, when the user is walking backward in the opposite direction of their field of view 120, the portion of the passthrough view 130 may cover both the left and right sides of the user's peripheral view, as well as the portion behind the user (beyond the user's field of view 120 and the user's peripheral view).

[0054] Figure 7 shows a perspective view of the user within the boundary space 175. The boundary space 175 may have a center 180 (which may correspond, for example, to the center of a real-world room in a real-world environment). The boundary space 175 may include a boundary 190 having one or more threshold boundaries that the user may want to customize. For example, the user may customize the boundary space 175 to include a starting threshold boundary 185 and an end boundary 195. Depending on the user's direction vector 155 (for example, as the user moves further away from or closer to the center of the boundary space 175), the computing system 110 (not shown) of the camera 105 may display a portion of the passthrough view on the VR display device 135 (not shown), increasing or decreasing its size and sharpness. As an example, when the user is at the starting boundary 185, the size of the portion of the passthrough view may be relatively smaller than when the user is at boundary 190. The sharpness of the portion of the passthrough view may be relatively duller at the starting boundary 185 than at boundary 190. As another example, if the user is at edge boundary 195, the size of the pass-through view portion may be relatively larger than if the user is at boundary 190. The sharpness of the pass-through view portion may also be relatively sharper when at edge boundary 195 than when at boundary 190.

[0055] Figure 8 shows a perspective view of a contour rendering view 150 of real-world objects in a VR environment 140. Referring to Figures 1C and 8, in certain embodiments, the computing system 110 can render a third output image for one or more displays of a VR display device 135, including one or more real-world objects 145 that cross the virtual boundary, for example, as a contour rendering view 150 of real-world objects, or as one or more mixed reality (MR) objects. The contour rendering view 150 of real-world objects can correspond to real-world objects 145 that can be located beyond the virtual boundary 115. The contour-rendered objects can be rendered as contours of one or more real-world objects 145, semi-opaque renderings of one or more real-world objects 145, fully opaque renderings of one or more real-world objects 145, or other similar renderings. The contour-rendered objects can be used to alert the user to the presence of one or more real-world objects 145 by showing the user a contour rendering view 150 of real-world objects that can correspond to the poses of the real-world objects 145 in a real-world environment. In other words, the technical advantages of the embodiment include providing spatial information by providing an outline rendering of real-world objects in the VR environment to warn the user of objects that may be in their path without significantly hindering immersion in the VR experience. Thus, the user may be given safe and subtle warnings or alerts of the presence of obstacles without disappointing or interrupting the user's VR experience. For example, if one or more real-world objects 145 (e.g., a desk) are located beyond the virtual boundary 115 (as shown in Figure 1C), then one or more real-world objects 145 may appear as outlined objects (e.g., MR objects in a contour rendering view 150 of the real-world objects) on the user's VR display device 135.While in the VR environment 140, the user can see a “ghostly” semi-opaque outline of an object 145 (e.g., a desk) without having to leave the VR environment 140. The outline-rendered view 150 of the real-world object can warn the user to avoid the desk and continue the VR experience. The outline-rendered object (e.g., the rendered outline of the desk) can fade in to the field of view, for example, if the computing system determines that the real-world object (e.g., the desk) does not pose a risk to the user, and fade out as the user moves away from the desk. Thus, the computing system 110 can provide a solution to the technical challenge of maintaining immersion in the VR experience while simultaneously providing the user with the visual information necessary to orient themselves within the virtual boundary. The solution presented herein can address this challenge by rendering an opaque, semi-transparent, or outline-rendered view 150 of the real-world object within the VR environment 140 to warn the user of the presence of the real-world object 145 without significantly interrupting the VR experience. While this disclosure describes rendering a third output image in a specific manner, it is assumed that any suitable output image may be rendered in any suitable manner.

[0056] In certain embodiments, the computing system 110 can determine whether a user is approaching a second threshold distance of a virtual boundary 115. The second threshold distance may be greater than the first threshold distance. For example, if the first threshold distance is 1 meter from the virtual boundary 115, the second threshold distance may be 2 meters from the virtual boundary 115. While this disclosure describes a particular method for determining whether a user is approaching a second threshold distance, this disclosure assumes that a user may be determined in any suitable way for determining whether a user is approaching an arbitrary threshold distance.

[0057] In certain embodiments, referring to Figures 1C, 2A-2D, and 8, the computing system 110 can determine the user's direction of movement 125 and field of view 120 in response to a user approaching within a second threshold distance of the virtual boundary 115. As described above, the computing system 110 can determine the user's direction of movement 125 and field of view 120 by using the sensors, accelerometer, gyroscope, or other position sensors of the camera 105 and / or VR display device 135 to determine the movement and orientation of a user wearing the camera 105 and / or VR display device 135. For example, but not limited to, the sensors may determine that the user is moving forward (for example, relative to the direction of movement 125a) along the same direction as their field of view 120 and is approaching within a second threshold distance of the virtual boundary 115 (Figure 2A). As an example, but not limited to, the sensor may determine that the user is moving backward (for example, with respect to direction 125b) in the opposite direction to their field of view 120 and is approaching a second threshold distance of the virtual boundary 115 (Figure 2B). As an example, but not limited to, the sensor may determine that the user is moving laterally (for example, with respect to direction 125c) perpendicular to the field of view 120 and is approaching a second threshold distance of the virtual boundary 115 (Figure 2C). As an example, but not limited to, the sensor may determine that the user is moving laterally to the right (for example, with respect to direction 125d) perpendicular to the field of view 120 and is approaching a second threshold distance of the virtual boundary 115 (Figure 2D). While this disclosure describes a particular method for determining the user's direction of movement 125 and field of view 120, this disclosure assumes that the user's direction of movement and field of view may be determined in any suitable method.

[0058] In certain embodiments, the computing system 110 can access one or more additional images of the real-world environment 100, including one or more real-world objects 145 captured by the camera 105 of the VR display device 135. A third output image may have one or more real-world objects 145 in the accessed additional images. The computing system 110 can access one or more images of the real-world environment 100 by using the camera 105 to take a photograph or snapshot (e.g., capture an image) of the user's real-world environment 100. An object detection filter or edge detection filter (e.g., a Sobel filter) can detect one or more real-world objects 145 in the vicinity of the user's real-world environment 100. For example, but not limited to, the camera 105 can be used to detect the edges of one or more real-world objects 145, such as a desk, within the user's real-world environment 100. The third output image may then include the desk captured by the camera 105. While this disclosure describes accessing one or more additional images in a particular manner, this disclosure assumes access to images in any suitable manner.

[0059] Figure 9 shows an exemplary method 900 for determining spatial awareness in a VR setting using a pass-through view. The method can begin in step 910, which may include a computing system rendering a first output image of a VR environment based on the user's field of view for one or more displays of a VR display device 135, where the VR environment includes a virtual boundary corresponding to the real-world environment. In step 920, the method may include determining whether the user is approaching within a first threshold distance of the virtual boundary. In step 930, the method may include determining the user's direction of movement and field of view in response to the user approaching within the first threshold distance of the virtual boundary. In step 940, the method may include accessing one or more images of the real-world environment captured by one or more cameras of the VR display device. In step 950, the method may include rendering a second output image for one or more displays of the VR display device based on the accessed images, which includes a portion of the VR environment and a portion of a pass-through view of the real-world environment, where the pass-through view portion is based on the determined direction of movement and the user's field of view. Specific embodiments may, where appropriate, repeat one or more steps of the method in Figure 9. While this disclosure describes and illustrates that the specific steps of the method in Figure 9 occur in a specific order, this disclosure assumes that any preferred steps of the method in Figure 9 occur in any preferred order. Furthermore, while this disclosure describes and illustrates an exemplary method for determining spatial awareness in a VR setting using a pass-through view that includes the specific steps of the method in Figure 9, this disclosure assumes any suitable method for determining spatial awareness in a VR setting using a pass-through view that includes any suitable steps, and where appropriate, does not include all, some, or any of the steps of the method in Figure 9.Furthermore, while this disclosure describes and illustrates specific components, devices, or systems that perform specific steps of the method in Figure 9, this disclosure assumes that any preferred combination of any preferred components, devices, or systems will perform any preferred step of the method in Figure 9.

[0060] Figure 10 shows an exemplary network environment 1000 associated with a VR or social networking system. The network environment 1000 includes a client system 1030, a VR or social networking system 1060, and a third-party system 1070, all connected to each other by a network 1010. While Figure 10 shows a specific arrangement of the client system 1030, the VR or social networking system 1060, the third-party system 1070, and the network 1010, this disclosure assumes any suitable arrangement of the client system 1030, the VR or social networking system 1060, the third-party system 1070, and the network 1010. Not limited to, for example, two or more of the client system 1030, the VR or social networking system 1060, and the third-party system 1070 could be directly connected to each other, bypassing the network 1010. As another example, two or more of the client system 1030, VR or social networking system 1060, and third-party system 1070 can be physically or logically located in the same place as one another, either as a whole or in part. Furthermore, although Figure 10 shows a specific number of client systems 1030, VR or social networking system 1060, third-party system 1070, and network 1010, this disclosure assumes any appropriate number of client systems 1030, VR or social networking system 1060, third-party system 1070, and network 1010. For example, but not limited to, a network environment 1000 can include multiple client systems 1030, VR or social networking system 1060, third-party system 1070, and network 1010.

[0061] This disclosure envisions any suitable network 1010. For example, but not limited to, one or more portions of network 1010 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), portion of the internet, portion of a public switched telephone network (PSTN), cellular telephone network, or two or more combinations thereof. Network 1010 may include one or more networks 1010.

[0062] Link 1050 can connect client system 1030, social networking system 1060, and third-party system 1070 to or to the communication network 1010. This disclosure envisions any suitable link 1050. In certain embodiments, one or more links 1050 include one or more wired links (e.g., digital subscriber line (DSL) or data over cable service interface specification (DOCSIS)), wireless links (e.g., Wi-Fi or worldwide interoperability for microwave access (WiMAX)), or optical links (e.g., synchronous optical network (SONET) or synchronous digital hierarchy (SDH)). In certain embodiments, one or more links 1050 each include an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, part of the internet, part of the PSTN, cellular technology-based network, satellite communication technology-based network, another link 1050, or a combination of two or more such links 1050. Link 1050 does not necessarily have to be the same throughout the entire network environment 1000. One or more first links 1050 may differ from one or more second links 1050 in one or more respects.

[0063] In certain embodiments, client system 1030 can be an electronic device including hardware, software, or embedded logic components, or a combination of two or more such components, capable of performing appropriate functionality implemented or supported by client system 1030. For example, but not limited to, client system 1030 can include computer systems such as desktop computers, notebook computers or laptop computers, netbooks, tablet computers, e-book readers, GPS devices, cameras, personal digital assistants (PDAs), handheld electronic devices, cellular phones, smartphones, augmented / virtual reality devices, or other appropriate electronic devices, or any appropriate combination thereof. This disclosure envisions any suitable client system 1030. Client system 1030 can enable network users of client system 1030 to access network 1010. Client system 1030 can enable users of client system 1030 to communicate with other users in other client systems 1030.

[0064] In certain embodiments, the client system 1030 (e.g., HMD) may include a pass-through engine 1032 for providing the pass-through functionality described herein and may have one or more add-ons, plug-ins, or other extensions. Users of the client system 1030 may connect to a specific server (such as server 1062, or a server associated with third-party system 1070). The server may accept requests and communicate with the client system 1030.

[0065] In certain embodiments, the VR or social networking system 1060 can be a network-addressable computing system capable of hosting an online virtual reality environment or social network. The VR or social networking system 1060 can generate, store, receive, and send social networking data, such as, for example, user profile data, concept profile data, social graph information, or other appropriate data related to an online social network. The social networking or VR system 1060 can be accessed directly or via the network 1010 by other components of the network environment 1000. For example, but not limited to, a client system 1030 can access the social networking or VR system 1060 directly or via the network 1010 using a web browser or a native application associated with the social networking or VR system 1060 (for example, a mobile social networking application, a messaging application, another appropriate application, or any combination thereof). In certain embodiments, the social networking or VR system 1060 can include one or more servers 1062. Each server 1062 may be a single server or a distributed server spanning multiple computers or multiple data centers. Servers 1062 may be of various types, including, but are not limited to, web servers, news servers, mail servers, message servers, advertising servers, file servers, application servers, exchange servers, database servers, proxy servers, other servers suitable for performing the functions or processes described herein, or any combination thereof.In certain embodiments, each server 1062 may include hardware, software, or embedded logical components, or combinations of two or more such components, to perform appropriate functionality implemented or supported by the server 1062. In certain embodiments, the social networking or VR system 1060 may include one or more data stores 1064. The data stores 1064 may be used to store various types of information. In certain embodiments, the information stored in the data stores 1064 may be organized according to a specific data structure. In certain embodiments, each data store 1064 may be a relational database, a column-oriented database, a correlational database, or other preferred database. While this disclosure describes or illustrates certain types of databases, this disclosure intends to cover any preferred type of database. In certain embodiments, it is possible to provide an interface that enables a client system 1030, the social networking or VR system 1060, or a third-party system 1070 to manage, retrieve, modify, add to, or delete information stored in the data stores 1064.

[0066] In certain embodiments, the social networking or VR system 1060 may store one or more social graphs in one or more data stores 1064. In certain embodiments, the social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept), and a plurality of edges connecting the nodes. The social networking or VR system 1060 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, users may join an online social network via the social networking or VR system 1060 and then add connections (e.g., relationships) to a plurality of other users of the social networking or VR system 1060 with whom they wish to connect. Hereinafter, the term “friend” may refer to any other user of the social networking or VR system 1060 with whom a user has connected, associated, or formed a relationship via the social networking or VR system 1060.

[0067] In certain embodiments, the social networking or VR system 1060 can provide the user with the ability to take action on various types of items or objects supported by the social networking or VR system 1060. These items and objects may include, but are not limited to, groups or social networks to which a user of the social networking or VR system 1060 may belong, events or calendar items of interest to the user, computer-based applications that the user may use, transactions that allow the user to buy or sell items through the service, interactions with advertisements that the user may perform, or other appropriate items or objects. The user may interact with anything that can be represented in the social networking or VR system 1060 or by an external system of the third-party system 1070 (which is separate from the social networking or VR system 1060 and connected to the social networking or VR system 1060 via the network 1010).

[0068] In certain embodiments, the social networking or VR system 1060 may be capable of linking various entities together. For example, but not limited to, the social networking or VR system 1060 may enable users to interact with each other and receive content from a third-party system 1070 or other entities, or enable users to interact with these entities through an application programming interface (API) or other communication channel.

[0069] In certain embodiments, the third-party system 1070 may include one or more types of servers, one or more data stores, one or more interfaces (including, but not limited to, APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (for example, that the servers can communicate with). The third-party system 1070 may be operated by an entity different from the entity operating the social networking or VR system 1060. However, in certain embodiments, the social networking or VR system 1060 and the third-party system 1070 may work together to provide social networking services to users of the social networking or VR system 1060 or the third-party system 1070. In this sense, the social networking or VR system 1060 may provide a platform or backbone that other systems, such as the third-party system 1070, can use to provide social networking services and functionality to users over the internet.

[0070] In certain embodiments, the third-party system 1070 may include a third-party content object provider. The third-party content object provider may include one or more sources of content objects that can be communicated to the client system 1030. As an example, but not limited to, content objects may include information about things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example, but not limited to, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0071] In certain embodiments, the social networking or VR system 1060 may also include user-generated content objects that enhance user interaction with the social networking or VR system 1060. User-generated content may include anything that the user can add, upload, transmit, or “post” to the social networking or VR system 1060. For example, but not limited to, a user communicates a post from the client system 1030 to the social networking or VR system 1060. A post may include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content may also be added to the social networking or VR system 1060 by a third party through a “communication channel” such as a news feed or stream.

[0072] In certain embodiments, the social networking or VR system 1060 may include various servers, subsystems, programs, modules, logs, and data stores. In certain embodiments, the social networking or VR system 1060 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object publication log, a guessing module, an approval / privacy server, a search module, an advertising targeting module, a user interface module, a user profile store, a connections store, a third-party content store, or a location store. The social networking or VR system 1060 may also include appropriate components, such as a network interface, a security mechanism, a load balancer, a failover server, an administration and network operations console, and other appropriate components, or any appropriate combination thereof. In certain embodiments, the social networking or VR system 1060 may include one or more user profile stores for storing user profiles. A user profile may include, for example, career information, demographic information, behavioral information, social information, or other types of descriptive information such as work history, education, hobbies or preferences, interests, affiliations, or location. Interest information may include interests related to one or more categories. Categories may be general or specific. As an example, but not an limitation, if a user "likes" an article about a shoe brand, the category may be the brand, or a general category such as "shoes" or "clothing." The connection store may be used to remember connection information about a user. Connection information may point to users who have similar or common work history, group membership, hobbies, education, or are related in any way, or who share common attributes.Connection information may also include user-defined connections between different users and content (both internal and external). A web server may be used to link the social networking or VR system 1060 to one or more client systems 1030 or one or more third-party systems 1070 via network 1010. The web server may include a mail server or other messaging functionality to receive and forward messages between the social networking or VR system 1060 and one or more client systems 1030. An API request server may enable third-party systems 1070 to access information from the social networking or VR system 1060 by calling one or more APIs. An action logger may be used to receive communications from the web server regarding user actions on or outside the social networking or VR system 1060. In conjunction with the action log, a third-party content object log may be maintained regarding user publications to third-party content objects. A notification controller may provide information about content objects to client systems 1030. Information can be pushed to the client system 1030 as a notification, or information can be retrieved from the client system 1030 in response to a request received from the client system 1030. An authorization server can be used to enforce one or more privacy settings of a user of the social networking or VR system 1060. A user's privacy settings determine how certain information associated with the user may be shared.The authorization server may allow users to opt in or opt out of having their actions recorded by the social networking or VR system 1060, or shared with other systems (e.g., third-party system 1070), for example, by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from third parties, such as third-party system 1070. A location store may be used to store location information received from client system 1030 associated with the user. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide the user with relevant advertisements in the form of notifications.

[0073] Figure 11 shows an exemplary computer system 1100. In a particular embodiment, one or more computer systems 1100 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 1100 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 1100 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more parts of one or more computer systems 1100. In this specification, references to computer systems may, where appropriate, encompass computing devices and vice versa. Furthermore, references to computer systems may, where appropriate, encompass one or more computer systems.

[0074] This disclosure envisions any number of suitable computer systems 1100. This disclosure envisions computer systems 1100 in any suitable physical form. Not as an limitation, but as an example, computer systems 1100 may be embedded computer systems, system-on-chip (SOC), single-board computer systems (SBCs) (such as computer-on-module (COM) or system-on-module (SOM)), desktop computer systems, laptop or notebook computer systems, interactive kiosks, mainframes, computer system meshes, mobile phones, personal digital assistants (PDAs), servers, tablet computer systems, augmented / virtual reality devices, or two or more of these in combination. Where appropriate, computer systems 1100 may comprise one or more computer systems 1100, be single or distributed, span multiple locations, span multiple machines, span multiple data centers, or reside in a cloud which may comprise one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein without substantial spatial or temporal limitations. As an example, but not an limitation, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein at different times or in different locations.

[0075] In a particular embodiment, the computer system 1100 includes a processor 1102, memory 1104, storage 1106, input / output (I / O) interface 1108, communication interface 1110, and bus 1112. While this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular configuration, this disclosure intends to describe any suitable computer system having any suitable number of any suitable components in any suitable configuration.

[0076] In certain embodiments, the processor 1102 includes hardware for executing instructions, such as instructions for creating computer programs. To execute instructions, the processor 1102 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1104, or storage 1106, decode and execute those instructions, and then write one or more results to internal registers, internal caches, memory 1104, or storage 1106. In certain embodiments, the processor 1102 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates a processor 1102 including any suitable number of suitable internal caches where appropriate. To certain embodiments, the processor 1102 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 1104 or storage 1106, and the instruction caches may speed up the retrieval of those instructions by the processor 1102. The data in the data cache may be a copy of data in memory 1104 or storage 1106 on which instructions executed by processor 1102 operate, the result of a previous instruction executed by processor 1102 for access by subsequent instructions executed by processor 1102, or for writing to memory 1104 or storage 1106, or other suitable data. The data cache can speed up read or write operations by processor 1102. The TLB can speed up virtual-address translation for processor 1102. In certain embodiments, processor 1102 may include one or more internal registers for data, instructions, or addresses. This disclosure intends for processor 1102 to include any suitable number of any suitable internal registers where appropriate.Where appropriate, the processor 1102 may include one or more arithmetic logic units (ALUs), be a multi-core processor, or include one or more processors 1102. While this disclosure describes and illustrates specific processors, this disclosure is intended to describe any suitable processor.

[0077] In certain embodiments, memory 1104 includes main memory for storing instructions for processor 1102 to execute or data on which processor 1102 operates. As an example, but not an limitation, computer system 1100 may load instructions into memory 1104 from storage 1106 or another source (such as another computer system 1100). Processor 602 may then load instructions from memory 604 into internal registers or internal cache. To execute the instructions, processor 602 may retrieve the instructions from the internal registers or internal cache and decode them. During or after the execution of the instructions, processor 602 may write one or more results (which may be intermediate or final results) to internal registers or internal cache. Processor 602 may then write one or more of those results to memory 604. In certain embodiments, the processor 602 executes only instructions in one or more internal registers or internal caches, or in memory 1104 (as opposed to storage 1106 or other locations), and operates only on data in one or more internal registers or internal caches, or in memory 1104 (as opposed to storage 1106 or other locations). One or more memory buses (which may each include an address bus and a data bus) can connect the processor 1102 to memory 1104. Bus 1112 may include one or more memory buses, as described below. In certain embodiments, one or more memory management units (MMUs) reside between the processor 1102 and memory 1104 to facilitate access to memory 1104 requested by the processor 1102. In certain embodiments, memory 1104 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Furthermore, where appropriate, this RAM may be single-port or multi-port RAM. This disclosure intends any suitable RAM. Memory 1104 may include one or more memory 1104, where appropriate.This disclosure describes and illustrates specific memory, but this disclosure intends to describe any suitable memory.

[0078] In certain embodiments, storage 1106 includes mass storage for data or instructions. As an example, but not an limitation, storage 1106 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or two or more combinations thereof. Storage 1106 may include removable or non-removable (or fixed) media, where appropriate. Storage 1106 may be located inside or outside the computer system 1100, where appropriate. In certain embodiments, storage 1106 is non-volatile solid-state memory. In certain embodiments, storage 1106 includes read-only memory (ROM). Where appropriate, this ROM may be a mask program ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or two or more combinations thereof. This disclosure envisions a high-capacity storage 1106 in any preferred physical form. The storage 1106 may, where appropriate, include one or more storage control units that facilitate communication between the processor 1102 and the storage 1106. Where appropriate, the storage 1106 may include one or more storage units. While this disclosure describes and illustrates specific storage, this disclosure envisions any preferred storage.

[0079] In certain embodiments, the I / O interface 1108 includes hardware, software, or both that provide one or more interfaces for communication between the computer system 1100 and one or more I / O devices. The computer system 1100 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and the computer system 1100. As an example, but not an limitation, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touchscreen, trackball, video camera, another suitable I / O device, or two or more combinations thereof. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 1108 for those I / O devices. Where appropriate, the I / O interface 1108 may include one or more device or software drivers that enable the processor 1102 to drive one or more of these I / O devices. The I / O interface 1108 may include one or more I / O interfaces 1108, where appropriate. While this disclosure describes and illustrates specific I / O interfaces, this disclosure intends to describe any suitable I / O interface.

[0080] In certain embodiments, the communication interface 1110 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1100 and one or more other computer systems 1100 or one or more networks. As an example, but not an limitation, the communication interface 1110 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi networks. This disclosure contemplates any suitable network and any suitable communication interface 1110 for that network. As an example, but not an limitation, the computer system 1100 may communicate with one or more parts of an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or two or more combinations thereof. One or more of these networks may be wired or wireless. As an example, the computer system 1100 may communicate with a wireless PAN (WPAN) (e.g., such as a Bluetooth WPAN), a Wi-Fi network, a Wi-MAX network, a cellular telephone network (e.g., such as a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or two or more combinations thereof. The computer system 1100 may, where appropriate, include any suitable communication interface 1110 for any of these networks. The communication interface 1110 may, where appropriate, include one or more communication interfaces 1110. While this disclosure describes and illustrates specific communication interfaces, this disclosure intends to include any suitable communication interfaces.

[0081] In certain embodiments, bus 1112 includes hardware, software, or both that connect components of computer system 1100 to one another. As an example, but not an limitation, bus 1112 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Frontside Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Institute Local (VLB) bus, or another suitable bus, or two or more combinations thereof. Bus 1112 may include one or more buses 1112, where appropriate. While this disclosure describes and illustrates specific buses, this disclosure intends to describe any suitable bus or interconnect.

[0082] In this specification, one or more computer-readable non-temporary storage media may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-temporary storage media, or any two or more suitable combinations thereof. The computer-readable non-temporary storage media may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0083] In this specification, “or” is inclusive and non-exclusive unless explicitly indicated otherwise or indicated by the context. Thus, in this specification, “A or B” means “A, B, or both” unless explicitly indicated otherwise or indicated by the context. Furthermore, “and” means both jointly and individually unless explicitly indicated otherwise or indicated by the context. Thus, in this specification, “A and B” means “A and B, jointly or individually” unless explicitly indicated otherwise or indicated by the context.

[0084] The scope of this disclosure includes all changes, substitutions, modifications, alterations, and modifications to the exemplary embodiments described or shown herein, as will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or shown herein. Furthermore, while this disclosure describes and shows each embodiment herein as including a particular component, element, feature, function, operation, or step, any of these embodiments may include any combination or substitution of any of the components, elements, features, functions, operations, or steps described or shown anywhere in this specification, as will be understood by those skilled in the art. Furthermore, any reference in the appended claims to an apparatus or system or component of an apparatus or system that is adapted to, arranged, capable of, configured, enabled, operable to, or operates to perform a particular function includes that apparatus, system, or component, to the extent that that apparatus, system, or component is adapted, arranged, capable, configured, enabled, operable to, or operates in such a manner, whether or not that apparatus, system, component or its particular function is activated, turned on, or unlocked. Furthermore, while this disclosure describes or indicates certain embodiments as providing certain advantages, certain embodiments may provide none of these advantages, some of them, or all of them.

Claims

1. A method comprising one or more computing systems, Rendering a first output image of a VR environment based on the user's field of view for one or more displays of a virtual reality (VR) display device, wherein the VR environment includes a virtual boundary corresponding to a real-world environment; Determining whether the user has approached within the first threshold distance of the virtual boundary, In response to the user approaching the first threshold distance of the virtual boundary, the direction of the user's movement and the field of view are determined. Accessing one or more images of the real-world environment captured by one or more cameras of the VR display device, For one or more displays of the VR display device, rendering a second output image based on the accessed image, including a portion of the VR environment and a portion of a passthrough view of the real-world environment, wherein the portion of the passthrough view is rendered based on the determined direction of movement of the user and the field of view. method.

2. The method further includes rendering a third output image for one or more displays of the VR display device, which includes one or more real-world objects beyond the virtual boundary as one or more mixed reality (MR) objects, optionally Determining whether the user is approaching a second threshold distance greater than the first threshold distance of the virtual boundary, In response to the user approaching the second threshold distance of the virtual boundary, the direction of the user's movement and the field of view are determined. Accessing one or more additional images of the real-world environment, including one or more real-world objects captured by the camera of the VR display device, It further includes, The third output image includes the one or more real-world objects in the accessed additional image. The method according to claim 1.

3. The method according to claim 2, wherein the one or more MR objects are rendered as the contours of the one or more objects, a semi-opaque rendering of the one or more objects, or a fully opaque rendering of the one or more objects.

4. Determining the speed of the user's movement, further comprising determining the speed of movement such that the area of ​​the portion of the pass-through view is based on the determined speed of the user's movement, and the portion is a spherical cap of a spherical second output image, optionally, The area of ​​the portion of the pass-through view becomes relatively larger as the user's determined movement speed increases, and the area of ​​the portion of the pass-through view becomes relatively smaller as the user's determined movement speed decreases. The method according to claim 1.

5. Determining the speed of the user's movement, further comprising determining the speed of the movement such that the sharpness of the transition from the VR environment to the portion of the passthrough view is based on the determined speed of the user's movement, and the transition is a fade from the VR environment to the portion of the passthrough view, optionally, The sharpness of the transition from the VR environment to the portion of the passthrough view becomes relatively sharper as the user's determined movement speed increases, and the sharpness of the transition from the VR environment to the portion of the passthrough view becomes relatively duller as the user's determined movement speed decreases. The method according to claim 1.

6. The method according to claim 1, wherein the rendered portion of the pass-through view corresponds to the direction of the user's movement.

7. If it is determined that the direction of the user's movement is toward the field of view, then the rendered portion of the passthrough view is within the user's field of view, or If the direction of movement is determined to be perpendicular to the field of view, then the rendered portion of the passthrough view is within the user's peripheral view, or The method according to claim 6, wherein if the direction of movement is determined to be away from the field of view, the rendered portion of the passthrough view is within the user's peripheral view and behind the user.

8. One or more computer-readable non-temporary storage media that embody software, wherein the software, when executed, Rendering a first output image of a VR environment based on the user's field of view for one or more displays of a virtual reality (VR) display device, wherein the rendering of the first output image includes a virtual boundary corresponding to a real-world environment. Determining whether the user is approaching within a first threshold distance of the virtual boundary, In response to the user approaching the first threshold distance of the virtual boundary, the direction of the user's movement and the field of view are determined. Accessing one or more images of the real-world environment captured by the camera of the VR display device, The VR display device is operable to render a second output image for one or more displays based on the accessed image, including a portion of the VR environment and a portion of a passthrough view of the real-world environment, wherein the portion of the passthrough view is operable to render a second output image based on the user's determined direction of movement and field of view. Medium.

9. When the aforementioned software is executed, The medium according to claim 8, further operable to render a third output image for one or more displays of the VR display device, the third output image including one or more real-world objects beyond the virtual boundary as one or more mixed reality (MR) objects.

10. When the aforementioned software is executed, Determining whether the user is approaching a second threshold distance greater than the first threshold distance of the virtual boundary, In response to the user approaching the second threshold distance of the virtual boundary, the direction of the user's movement and the field of view are determined. The medium according to claim 9, further operable to access one or more images of the real-world environment, including one or more real-world objects captured by the camera of the VR display device.

11. The medium according to claim 9, wherein the rendered one or more MR objects are rendered as one or more of the contours of the objects, a semi-opaque rendering of the objects, or a fully opaque rendering of the objects.

12. When the aforementioned software is executed, The medium according to claim 8, further operable to determine the speed of the user's movement, wherein the area of ​​the portion of the pass-through view is based on the determined speed of the user's movement, and the area is an arc of the second output image.

13. The medium according to claim 12, wherein the area of ​​the portion of the pass-through view becomes relatively larger as the user's determined movement speed increases, and the area of ​​the portion of the pass-through view becomes relatively smaller as the user's determined movement speed decreases.

14. When the aforementioned software is executed, The medium according to claim 8, further operable to determine the speed of the user's movement, the sharpness of the transition from the VR environment to the portion of the passthrough view is based on the determined speed of the user's movement, and the transition is a fade from the VR environment to the portion of the passthrough view.

15. A system comprising one or more processors and non-temporary memory coupled to the processors and containing instructions executable by the processors, wherein when the processor executes the instructions, Rendering a first output image of a VR environment based on the user's field of view for one or more displays of a virtual reality (VR) display device, wherein the rendering of the first output image includes a virtual boundary corresponding to a real-world environment. Determining whether the user is approaching within a first threshold distance of the virtual boundary, In response to the user approaching the first threshold distance of the virtual boundary, the direction of the user's movement and the field of view are determined. Accessing one or more images of the real-world environment captured by the camera of the VR display device, The VR display device is operable to render a second output image for one or more displays based on the accessed image, including a portion of the VR environment and a portion of a passthrough view of the real-world environment, wherein the portion of the passthrough view is operable to render a second output image based on the user's determined direction of movement and field of view. system.

Images