Generating and modifying artificial reality environments using occluding surfaces at predetermined distances

Abstract

This application relates to generating and modifying artificial reality environments using occlusion surfaces at a predetermined distance. A method includes generating a depth map of a real environment as seen from a viewpoint, the depth map including pixels having respective depth values of one or more physical objects. A two-dimensional occlusion surface is generated based on the depth map, the two-dimensional occlusion surface representing at least visible portions of the one or more physical objects located within a predetermined depth range defined relative to the viewpoint. A pose of the two-dimensional occlusion surface is set in a three-dimensional coordinate system such that the two-dimensional occlusion surface is located at a predetermined distance from the viewpoint. Visibility of a virtual object relative to the one or more physical objects is determined by comparing a model of the virtual object to the two-dimensional occlusion surface, and an output image is generated based on the visibility of the virtual object.

Description

[0001] This application is a divisional application of the application filed on June 23, 2022, with application number 202210718661.2 and invention title "Generating and Modifying an Artificial Reality Environment Using an Obscuring Surface at a Predetermined Distance".

[0002] Cross-references to related applications This application claims the benefit of priority to U.S. non-provisional application No. 17 / 356427, filed on June 23, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to augmented reality, virtual reality, mixed reality, or hybrid reality environments. Background Technology

[0004] Artificial reality is a form of reality that is adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photographs of the real world). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of these may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect for the viewer). Artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or for use in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that deliver artificial reality content can be implemented on various platforms, including head-mounted displays (HMD 104) connected to a host computer system, standalone HMD 104, mobile devices or computing systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0005] Mobile computing devices (such as smartphones, tablets, or laptops) may include features for determining their location, orientation, or direction, such as a GPS receiver, compass, gyroscope, or accelerometer. Such devices may also include features for wireless communication, such as Bluetooth, near-field communication (NFC), infrared (IR), or communication with wireless local area networks (WLANs) or cellular phone networks. These devices may also include one or more cameras, scanners, touchscreens, microphones, or speakers. Mobile computing devices can also run software applications, such as games, web browsers, or social networking applications. Using social networking applications, users can connect, communicate, and share information with other users in their social networks. Summary of the Invention

[0006] The disclosed method provides techniques for occluding two-dimensional and three-dimensional virtual objects in an AR / VR / MR environment by generating one or more two-dimensional occlusion surfaces and setting the pose of the one or more two-dimensional occlusion surfaces, which may represent one or more physical objects in a real-world environment. In a particular embodiment, a specific two-dimensional occlusion surface may be used to represent one or more physical objects located at a distance or depth range from the viewpoint of a user of the AR / VR system. At a high level, the computing system may generate a depth map of the real-world environment as seen from the user's viewpoint, the depth map including pixels with corresponding depth values ​​for one or more physical objects in the real-world environment. Based on the depth map, the computing system may generate two-dimensional occlusion surfaces representing at least the visible portion of one or more physical objects located within a predetermined depth range defined relative to the viewpoint. The disclosed method may also set the pose of the two-dimensional occlusion surfaces in a three-dimensional coordinate system such that the two-dimensional occlusion surfaces are located at a predetermined distance from the viewpoint. This pose may also be based on subsequent poses of an HMD worn by the user of the AR / VR system. The computing system can determine the visibility of a virtual object relative to one or more physical objects by comparing a model of the virtual object with a two-dimensional occluding surface, and generate an output image based on the determined visibility of the virtual object. In doing so, the computing system can accurately predict and generate a view of the immersive artificial reality environment for the user of the HMD.

[0007] In certain embodiments, the generated view may be rendered by one or more components (e.g., CPU, GPU, etc.) of a computing system associated with a device (e.g., laptop computer, mobile phone, desktop computer, wearable device). In certain embodiments, the device communicates with the computing system on the HMD, but may otherwise be physically separated from the HMD. By way of example, and not limitation, the device may be a laptop device that is wired to or wirelessly communicates with the HMD. By way of another example, and not limitation, the device may be a wearable device (e.g., a device strapped to the wrist), a handheld device (e.g., a telephone), or some other suitable device (e.g., a laptop computer, tablet computer, desktop computer) that is wired to or wirelessly communicates with the HMD. In certain embodiments, the device may send the initial view to the HMD for display. In certain embodiments, components of the device that generates the initial view may also generate one or more surfaces representing objects of interest in the view. As another example, and not a limitation, the onboard computing system of an HMD can resample and adjust the pose of one or more surfaces after receiving the initial view from a separate computing system (e.g., from the CPU or GPU of a wearable, handheld, or lap device) to account for possible movement of the HMD and / or the object of interest during the rendering of the initial image.

[0008] In certain embodiments, graphics applications (e.g., games, maps, content-providing applications, etc.) can construct a scene graph, which, along with a given viewpoint and time point, is used to generate primitives for rendering on the GPU. The scene graph can define logical and / or spatial relationships between objects in the scene. In certain embodiments, the display engine can also generate and store a scene graph that is a simplified form of the full application scene graph. The simplified scene graph can be used to specify logical and / or spatial relationships between surfaces (e.g., primitives rendered by the display engine, such as quadrilaterals or contours defined in 3D space, having corresponding textures generated based on the master frame rendered by the application). Storing the scene graph allows the display engine to render the scene across multiple display frames, adjusting each element in the scene graph for the current viewpoint (e.g., head positioning), the current object positioning (e.g., they may have moved relative to each other), and other factors that change with each display frame. Furthermore, based on the scene graph, the display engine can also adjust for geometric and color distortions introduced by the display subsystem before compositing the objects together to generate frames. Storing scene graphs allows the display engine to approximate the rendered result at the display rate, while actually rendering the assets at a significantly lower rate.

[0009] This application also relates to the following: 1) A method comprising using a computing system: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A two-dimensional occlusion surface is generated based on the depth map, the two-dimensional occlusion surface representing at least the visible portion of the one or more physical objects located within a predetermined depth range relative to the viewpoint; The pose of the two-dimensional occlusion surface is set in a three-dimensional coordinate system such that the two-dimensional occlusion surface is located at a predetermined distance from the viewpoint; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the two-dimensional occlusion surface; and An output image is generated based on the determined visibility of the virtual object.

[0010] 2) The method described in 1) further includes: A second two-dimensional occlusion surface is generated based on the depth map. This second two-dimensional occlusion surface represents at least the visible portion of one or more physical objects located within a second predetermined depth range relative to the viewpoint, where the second predetermined depth range is greater than a first depth range. The pose of the second two-dimensional occlusion surface is set in the three-dimensional coordinate system such that the second two-dimensional occlusion surface is located at a second predetermined distance from the viewpoint.

[0011] 3) The method described in 1) further includes: Based on the depth map, identify a first physical object among the one or more physical objects located at a depth less than the depth range defined relative to the viewpoint; Generate a pre-occluded 2D surface for the first physical object, wherein the pre-occluded 2D surface is based on the visibility of the first physical object from the user's viewpoint; and The pose of the pre-occluded two-dimensional surface is set in the three-dimensional coordinate system.

[0012] 4) According to the method in 1), wherein setting the pose of the two-dimensional occlusion surface includes adjusting the position or orientation of the two-dimensional occlusion surface to take into account changes in the viewpoint of the plurality of physical objects, the changes in viewpoint caused by one or more movements of a camera worn by the user after the time of generating the two-dimensional occlusion surface.

[0013] 5) The method according to 4), wherein adjusting the position or orientation of the two-dimensional occlusion surface is performed by one or more components of the computing system physically connected to the head-mounted device worn by the user.

[0014] 6) The method according to 1), wherein the depth map is calculated based on one or more images received from a camera connected to a head-mounted display worn by the user.

[0015] 7) The method according to 1), wherein the two-dimensional occlusion surface comprises a plurality of texels, wherein each texel is assigned a value that specifies the level of transparency to be assigned to the texel.

[0016] 8) The method according to 1), wherein the predetermined depth range defined relative to the viewpoint is between 1 meter and 2 meters from the viewpoint.

[0017] 9) One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A two-dimensional occlusion surface is generated based on the depth map, the two-dimensional occlusion surface representing at least the visible portion of the one or more physical objects located within a predetermined depth range relative to the viewpoint; The pose of the two-dimensional occlusion surface is set in a three-dimensional coordinate system such that the two-dimensional occlusion surface is located at a predetermined distance from the viewpoint; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the two-dimensional occlusion surface; and An output image is generated based on the determined visibility of the virtual object.

[0018] 10) According to the medium described in 9), wherein the software, when executed, is also capable of: A second two-dimensional occlusion surface is generated based on the depth map. This second two-dimensional occlusion surface represents at least the visible portion of one or more physical objects located within a second predetermined depth range relative to the viewpoint, where the second predetermined depth range is greater than a first depth range. The pose of the second two-dimensional occlusion surface is set in the three-dimensional coordinate system such that the second two-dimensional occlusion surface is located at a second predetermined distance from the viewpoint.

[0019] 11) According to the medium described in 9), wherein the software, when executed, is also capable of: Based on the depth map, identify a first physical object among the one or more physical objects located at a depth less than the depth range defined relative to the viewpoint; Generate a pre-occluded 2D surface for the first physical object, wherein the pre-occluded 2D surface is based on the visibility of the first physical object from the user's viewpoint; and The pose of the pre-occluded two-dimensional surface is set in the three-dimensional coordinate system.

[0020] 12) According to the medium described in 9), wherein setting the posture of the two-dimensional occlusion surface includes adjusting the position or orientation of the two-dimensional occlusion surface to take into account changes in the perspective of the plurality of physical objects, the changes in perspective caused by one or more movements of a user-worn camera after the time of generating the two-dimensional occlusion surface.

[0021] 13) The medium according to 12), wherein adjusting the position or orientation of the two-dimensional occlusion surface is performed by one or more components of a computing system physically connected to a head-mounted device worn by a user.

[0022] 14) The medium according to 9), wherein the depth map is calculated based on one or more images received from a camera connected to a head-mounted display worn by the user.

[0023] 15) A system comprising: One or more processors; and One or more computer-readable non-transitory storage media coupled to one or more of the processors and including instructions that, when executed by one or more of the processors, are operable to cause the system to: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A two-dimensional occlusion surface is generated based on the depth map, the two-dimensional occlusion surface representing at least the visible portion of the one or more physical objects located within a predetermined depth range relative to the viewpoint; The pose of the two-dimensional occlusion surface is set in a three-dimensional coordinate system such that the two-dimensional occlusion surface is located at a predetermined distance from the viewpoint; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the two-dimensional occlusion surface; and An output image is generated based on the determined visibility of the virtual object.

[0024] 16) The system according to 15), wherein, when the instructions are executed, the processor is also capable of operating to: A second two-dimensional occlusion surface is generated based on the depth map. This second two-dimensional occlusion surface represents at least the visible portion of one or more physical objects located within a second predetermined depth range relative to the viewpoint, where the second predetermined depth range is greater than a first depth range. The pose of the second two-dimensional occlusion surface is set in the three-dimensional coordinate system such that the second two-dimensional occlusion surface is located at a second predetermined distance from the viewpoint.

[0025] 17) The system according to 15), wherein, when the instructions are executed, the processor is also capable of: Based on the depth map, identify a first physical object among the one or more physical objects located at a depth less than the depth range defined relative to the viewpoint; Generate a pre-occluded 2D surface for the first physical object, wherein the pre-occluded 2D surface is based on the visibility of the first physical object from the user's viewpoint; and The pose of the pre-occluded two-dimensional surface is set in the three-dimensional coordinate system.

[0026] 18) The system according to 15), wherein setting the pose of the two-dimensional occlusion surface includes adjusting the position or orientation of the two-dimensional occlusion surface to take into account changes in the viewpoint of the plurality of physical objects, the changes in viewpoint caused by one or more movements of a user-worn camera after the time of generating the two-dimensional occlusion surface.

[0027] 19) The system according to 18), wherein adjusting the position or orientation of the two-dimensional occlusion surface is performed by one or more components of a computing system physically connected to a head-mounted device worn by a user.

[0028] 20) The system according to 15), wherein the depth map is calculated based on one or more images received from a camera connected to a head-mounted display worn by the user.

[0029] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited thereto. Specific embodiments may include all, some, or exclude all of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the invention are specifically disclosed in the appended claims relating to methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim class (e.g., methods) may also be claimed in another claim class (e.g., systems). Dependencies or backreferences in the appended claims are chosen only for formal reasons. However, any subject matter arising from intentional backreferences (particularly multiple dependencies) to any prior claim may also be claimed, thereby disclosing and claiming any combination of claims and their features, regardless of the dependencies chosen in the appended claims. Claimable subject matter includes not only combinations of features set forth in the appended claims but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature of the appended claims. Attached Figure Description

[0030] Figure 1A An example artificial reality system and user are shown.

[0031] Figure 1B An example augmented reality system is shown.

[0032] Figure 2A This illustrates a realistic environment containing one or more physical objects, as seen from the viewpoint of a user of an artificial reality system.

[0033] Figure 2B This shows a real-world environment containing one or more physical objects as seen from an overhead viewpoint (e.g., an aerial viewpoint).

[0034] Figure 3 The view from above shows the real environment, which has been divided into one or more depth regions.

[0035] Figure 4A and Figure 4B A two-dimensional occlusion surface is shown, representing at least the visible portion of one or more physical objects located within a predetermined depth region.

[0036] Figure 5The output image shows a view of an artificial reality environment that includes one or more two-dimensional occluding surfaces and one or more virtual objects.

[0037] Figure 6 An example method for generating an output image based on the visibility of a defined virtual object is shown.

[0038] Figure 7 A system diagram of the display engine is shown.

[0039] Figure 8 An example network environment associated with a social networking system is shown.

[0040] Figure 9 An example computer system is shown. Detailed Implementation

[0041] Figure 1A An example artificial reality system 100 and a user 102 are illustrated. In a particular embodiment, the artificial reality system 100 may include a head-mounted device 104, a controller 106, and a computing system 108. The user 102 may wear the head-mounted device 104, which may display visual artificial reality content to the user 102. The HMD 104 may include audio devices that can provide audio artificial reality content to the user 102. The head-mounted device 104 may include an eye-tracking system for determining the user 102's vergence distance. The vergence distance may be the distance from the user's eyes to an object that the user's eyes converge on (e.g., a real-world object or a virtual object in virtual space). The head-mounted device 104 may be referred to as a head-mounted display (HMD). One or more controllers 106 may be paired with the artificial reality system 100. In certain embodiments, one or more controllers 106 may be equipped with at least one inertial measurement unit (IMU) and infrared (IR) light-emitting diodes (LEDs) for the artificial reality system 100 to estimate the controller's posture and / or track its position, allowing the user 102 to perform specific functions via the controllers 106. In certain embodiments, one or more controllers 106 may be equipped with one or more trackable markers assigned to be tracked by the computing system 108. One or more controllers 106 may include a touchpad and one or more buttons. One or more controllers 106 may receive input from the user 102 and relay the input to the computing system 108. One or more controllers 106 may also provide haptic feedback to the user 102. The computing system 108 may be connected to the head-mounted device 104 and one or more controllers 106 via cable or wireless connection. One or more controllers 106 may include combinations of hardware, software, and / or firmware not explicitly shown herein to avoid obscuring other aspects of this disclosure.

[0042] The artificial reality system 100 may also include a computer unit 108. The computer unit may be a physically separate, independent unit from the HMD, or it may be integrated with the HMD. In embodiments where the computer 108 is a separate unit, it may be communicatively coupled to the HMD via a wireless or wired link. The computer 108 may be a high-performance device such as a desktop or laptop computer, or a resource-constrained device such as a mobile phone. High-performance devices may have a dedicated GPU and a large-capacity or constant power supply. On the other hand, resource-constrained devices may not have a GPU and may have a limited battery capacity. Therefore, the algorithms that can be actually used by the artificial reality system 100 depend on the capabilities of its computer unit 108.

[0043] Figure 1B An example augmented reality system 100B is illustrated. Augmented reality system 100B may include an augmented reality head-mounted display (AR HMD) 110 (e.g., glasses), which includes a frame 112, one or more displays 114, and a computing system 120. Displays 114 may be transparent or semi-transparent, allowing a user wearing AR HMD 110 to view the real world through displays 114, while simultaneously displaying visual artificial reality content to the user. AR HMD 110 may include audio devices capable of providing audio artificial reality content to the user. AR HMD 110 may include one or more cameras capable of capturing images and video of the environment. AR HMD 110 may include an eye-tracking system to track the convergence motion of a user wearing AR HMD 110. Unless otherwise specified throughout this application, the term "HMD" may be used to refer to either HMD 104 (which may obscure a user's view of the real environment) or AR HMD 110 (which allows a user to see the real world and simultaneously displays visual artificial reality content to the user).

[0044] The augmented reality system 100B may also include a controller comprising a touchpad and one or more buttons. The controller can receive input from the user and relay the input to the computing system 120. The controller can also provide haptic feedback to the user. The computing system 120 can be connected to the AR HMD 110 and the controller via a cable or wireless connection. The computing system 120 can control the AR HMD 110 and the controller to provide augmented reality content to the user and receive input from the user. The computing system 120 may be a standalone host computer system, an onboard computer system integrated with the AR HMD 110, a mobile device, or any other hardware platform capable of providing augmented reality content to the user and receiving input from the user.

[0045] HMDs can have externally facing cameras, such as... Figure 1A The two forward-facing cameras 105A and 105B are shown. Although only two forward-facing cameras 105A-105B are shown, an HMD can have any number of cameras facing any direction (e.g., an upward-facing camera for capturing ceiling or room lighting, a downward-facing camera for capturing a portion of the user's face and / or body, a rearward-facing camera for capturing a portion of the content behind the user, and / or an internal camera for capturing the user's eye gaze for eye-tracking purposes). The externally facing cameras 105A and 105B are configured to capture the physical environment around the user and can do so continuously to generate a sequence of frames (e.g., as video).

[0046] In certain embodiments, the posture (e.g., position and orientation) of the HMD in the environment may be required. For example, in order to render an appropriate display for user 102 as he moves around in a virtual or augmented reality environment, system 100 will need to determine his position and orientation at any given moment. Based on the posture of the HMD, system 100 can also determine the viewpoint of either of cameras 105A and 105B or either of the user's two eyes. In certain embodiments, the HMD may be equipped with an inertial measurement unit (“IMU”). Data generated by the IMU, along with stereo images captured by the externally facing cameras 105A-105B, allows system 100 to calculate the posture of the HMD using techniques such as SLAM (Simultaneous Localization and Mapping) or other suitable techniques.

[0047] Since their inception, artificial reality (AVR), virtual reality (VR), and mixed reality (MR) technologies have been plagued by latency issues when rendering AR / VR / MR representations of virtual objects in response to sudden changes in the user's perspective. Traditional 3D artificial reality environment reconstruction techniques and algorithms can gradually integrate depth information about the real environment over time to create a 3D world mesh, which can be used to re-render the environment as the user's perspective changes. However, these methods are inherently too slow to account for the constant changes in perspective when the user experiences the artificial reality environment. For example, a user may suddenly move their head while viewing a scene, and the rendered image may need to respond immediately to the changing perspective by adjusting the view of one or more virtual representations presented to the user. Using traditional methods, re-rendering the entire view to account for changes in perspective can only be done at a relatively low frame rate (e.g., 60 Hz, or once every 1 / 60th of a second). Therefore, it may not be feasible to modify the view by re-rendering the entire view at a sufficiently fast speed (e.g., 200 Hz, once every 1 / 200 second) to account for changes in perspective, in order to prevent users from perceiving latency and thus avoid or adequately reduce sensory dissonance.

[0048] Furthermore, traditional artificial reality environment reconstruction techniques and algorithms can require expensive computational resources, limiting the ability to reconstruct artificial reality environments using components that are compact enough to be housed in an HMD (especially an AR HMD with a small shape factor). However, the timescales involved are often small enough that it may be infeasible for a more powerful system (e.g., a standalone laptop or wearable device) physically separate from the HMD to perform modifications, as the HMD must transmit captured images along with information about its current location and orientation, wait for a separate system to detect real (e.g., physical) objects in the images and render a new view, and then receive the new view from the separate system.

[0049] One solution to these traditional rendering methods involves generating and resampling "surfaces." A surface can be one or more primitives rendered by a display engine, such as quadrilaterals or contours defined in 3D space, having a corresponding texture generated based on the master frame rendered by the application. In a particular embodiment, one or more surfaces can represent a specific view of objects (e.g., real or virtual objects) within an artificial reality environment, where the surface corresponds to one or more objects that are expected to move / translate, tilt, scale, distort, or otherwise change their appearance together as a whole due to changes in viewpoint. This method can essentially be an efficient shortcut for adjusting the view in response to user head movements and can significantly reduce the processing required, thus ensuring that the view is updated quickly enough to adequately reduce latency. A method for generating and modifying object representations in an augmented reality or virtual reality environment is disclosed in U.S. Patent Application No. 16 / 586,590, filed September 27, 2019, which is incorporated herein by reference. In certain embodiments, instead of re-rendering the entire view, resampling the surface in response to a sudden movement may be efficient enough to modify the view within an allocated timeframe using the relatively limited processing power of the HMD's (especially AR HMD) computing system, which can be designed with a small shape factor, much like a traditional pair of glasses. By simply resampling the surface, the modification can be performed entirely on the HMD, thus speeding up the process.

[0050] The disclosed invention allows a two-level system to replace traditional occlusion techniques by rendering 2D surfaces at different depths in an AR scene. Each surface represents an occlusion surface of one or more physical objects located within a certain depth range from the user in the real environment (e.g., generating a first 2D occlusion surface using all physical objects within 0-1 meter of the user's viewpoint or pose, a second 2D occlusion surface using all objects between 1-2 meters of the user's viewpoint or pose, and so on). This technique provides occlusion rendering that scales to any number of physical objects in the scene, rather than generating a separate occlusion surface for each individual object. This significantly reduces computational resources and latency. Furthermore, by combining multiple physical objects into a single occlusion mask based on depth, the need for re-rendering is reduced because the computational system can reuse the mask surface as the user moves through the environment (e.g., for occlusion masks located further away from the user, the change in the user's viewpoint is smaller). This is particularly true for objects located further away from the user's viewpoint or pose, as slight movements of the user can cause imperceptible changes in the viewpoint of physical objects further away from the user's viewpoint or pose.

[0051] Specific embodiments of the invention utilizing one or more surfaces disclosed herein may be implemented using an artificial reality system or in combination with an artificial reality system. In specific embodiments, the processing tasks involved in rendering the artificial reality environment and generating and modifying its surfaces may be divided among two or more computing systems. By way of example and not limitation, a depth map of a real environment as seen from a user's viewpoint may initially be generated by a first computing system (e.g., a laptop computer, mobile phone, desktop computer, wearable device). The rendered result may be used to generate one or more surfaces for the view and / or one or more surfaces representing at least the visible portion of one or more physical objects in the real environment located within a predetermined depth range defined relative to the viewpoint. In addition to texture, color, and transparency information, surfaces may also include information about their location in the artificial reality environment. These surfaces may be passed to a second computing system (e.g., an onboard computing system on an HMD). The HMD may set the pose (e.g., position and orientation) of the surfaces in a three-dimensional coordinate system such that two-dimensional occluded surfaces are located at a predetermined distance from the user's viewpoint or pose. Setting the pose of these occluding surfaces can be further based on the current viewpoint or pose of the user wearing the HMD (e.g., the current viewpoint or pose determined by the position and orientation of the HMD), such that, for example rather than by constraint, the surface normals of the surface point toward the user's viewpoint or pose. Any changes in the user's viewpoint can be tracked via received image data and sensors on the HMD, and the surfaces in the view can be resampled to render a time-adjusted viewpoint. Due to the adjustment of the viewpoint, the surfaces may translate / move, tilt, scale, distort, or otherwise change their appearance when resampled. Since the viewpoint is not re-rendered from scratch, it can be modified relatively quickly (e.g., at 200 Hz). In certain embodiments, the first computing system may be relatively powerful when compared to a second computing system because the second computing system (e.g., the HMD) may have limited system resources that do not increase significantly without causing excessive weight, size, and / or heat for user comfort. The computing system can use a posed surface to determine the visibility of a virtual object relative to one or more physical objects by comparing a model or surface representing the virtual object with a two-dimensional occluding surface and generating an output image based on the determined visibility of the virtual object. In addition to saving resources and time through this method, an additional advantage provided by the disclosed embodiments is the ability of the artificial reality system to render its assets without any inherent knowledge of the user's environment. Therefore, users can still experience immersive artificial reality experiences without sharing details of their personal space (e.g., their home) with third-party applications. The disclosed embodiments thus simply improve security and privacy by reducing the amount of personal information users must provide to enable third-party applications, thereby reducing the risk of third-party applications leaking user data.

[0052] Figure 2A A realistic environment containing one or more physical objects is shown as seen from the viewpoint of a user of the artificial reality system. In a particular embodiment, the realistic environment 200 may include one or more physical objects posed in the realistic environment, such as, but not limited to, a plant 210A, a table 210B, a chair 210C, a laptop computer 210D, a chair 210E, and a trash can 210F. In a particular embodiment, the physical objects may be located at different depths from the viewpoint or pose of the user of the artificial reality system. Figure 2B This illustrates a real-world environment containing one or more physical objects as seen from an overhead viewpoint (e.g., an aerial viewpoint). Figure 2B As shown, in the real environment 200, the table 210B, chair 210C, laptop computer 210D, and trash can 210F can be located at a shallower depth from the user 102. As another example, the plant 210A and chair 210E can be located at a deeper depth from the user 102.

[0053] In a particular embodiment, the computing system can generate a depth map or point cloud of a real-world environment. The depth map or point cloud may include multiple pixels, each corresponding to a location or coordinate in the real-world environment. In a particular embodiment, one or more pixels may have a corresponding distance or depth value, which may represent the distance (e.g., a signed distance field) of one or more physical objects in the real-world environment to a specific coordinate. In a particular embodiment, the distance or depth value may represent the distance from a specific coordinate to a specific physical object or a specific portion of a physical object. For example, returning to... Figure 2B The distance from coordinate 220 to trash can 210F can be measured relative to one or more specific coordinates, corners, edges, surfaces, centroids, or multiple points.

[0054] In certain embodiments, the computing system may receive one or more depth measurements of a real-world environment. In certain embodiments, one or more depth measurements may be determined based on sensor or image data, such as images captured by one or more cameras worn by a user and connected to a head-mounted display, images captured by LiDAR, pre-generated and stored depth maps of the real-world environment, etc. In certain embodiments, the depth measurements may be sparse depth measurements. In certain embodiments, the computing system may generate a depth map of the real-world environment based on one or more depth measurements. Although this disclosure describes specific techniques for generating depth maps or point clouds of a real-world environment, it should be understood that any suitable techniques for determining depth measurements using image data or sensor data from an artificial reality system can be utilized.

[0055] In certain embodiments, the computing system can continuously update the depth map as the user moves throughout the real-world environment. For example, the computing system can continuously receive image or sensor data indicating that the current depth map is inaccurate or outdated, due to, for example, the relocation of one or more physical objects in the real-world environment since the depth map was generated or last updated. In certain embodiments, the computing system can generate a 3D mesh of the environment based on the depth map or point cloud.

[0056] In a particular embodiment, the computing system can generate one or more depth regions that divide a real environment. Figure 3 The view presents a realistic environment from an overhead perspective, which has been divided into one or more depth regions. In a particular embodiment, a depth region may represent a range of distances from specific coordinates in the realistic environment (e.g., coordinates representing a specific viewpoint or pose of the user or HMD). Figure 3 As shown, a depth region can represent the area or volume of the real environment located within a specific distance range from the viewpoint or posture of user 102. For example, depth region 310 can represent the area of ​​the real environment located between 0 meters and 1 meter from the viewpoint of user 102, depth region 320 can represent the area of ​​the real environment located between 1 meter and 3 meters from the viewpoint of user 102, and depth region 330 can represent the area of ​​the real environment located between 3 meters and 5 meters from the viewpoint of user 102. Although Figure 3 Depth regions with varying widths are shown, but it should be understood that in certain non-limiting embodiments, the size of each depth region may be constant (e.g., each depth region is a constant width of 1 meter), or the size of each depth region may vary based on any number of factors (e.g., the size of the real environment 200, the application the user of the artificial reality system is interacting with, or the current speed, acceleration, etc. of user 102). It should also be understood that in certain non-limiting embodiments, the number of depth regions dividing the real environment may vary based on any number of factors, such as the size of the real environment 200 (e.g., a larger environment may require more depth regions), the application the user of the artificial reality system is interacting with (e.g., a gaming application may require more depth regions than a video conferencing application), or the current speed, acceleration, etc. of user 102 (e.g., a user moving at a faster pace may require more depth regions to avoid latency when the user moves throughout the real environment). It should also be understood that, although... Figure 3Depth regions are shown as having a specific shape or volume (e.g., rectangle or rectangular prism), but in certain embodiments, one or more depth regions can be represented by any suitable shape, including, for example, a circle, semicircle, arc, cone, square, trapezoid, etc. In certain embodiments, depth regions can be centered on specific coordinates representing the user's viewpoint or posture (e.g., concentric circles radiating outward from the user's posture or viewpoint).

[0057] In certain embodiments, the computing system may use depth maps or image data to determine the distance of one or more physical objects or pixels from a user's viewpoint or pose in a real-world environment. The computing system may use, for example, the current pose of the HMD, the viewpoint of one or more cameras on the HMD, or the viewpoint of one or more of the user's eyes to determine the user's pose or viewpoint in the real-world environment. Based on the user's current pose or viewpoint in the real-world environment, the computing system may determine the distance of one or more physical objects or pixels from the user's viewpoint or pose. In certain embodiments, the distance from user 102 to a particular physical object may be measured using the methods described herein from the average depth of a specific coordinate, corner, edge, surface, centroid, or multiple points of the real-world object to the average depth representing the specific coordinate, corner, edge, surface, centroid, or multiple points of user 102 (e.g., coordinates representing the user's eye pose or viewpoint).

[0058] In certain embodiments, the computing system may assign one or more pixels from the depth map to one or more predefined depth regions. For example, the computing system may assign all pixels within 0-1 meter of the user's viewpoint or pose to depth region 310, and all pixels within 1-2 meters of the user's viewpoint or pose to depth region 320, and so on. This can result in objects (e.g., objects crossing the boundary between depth regions 310 and 320) being separated across two or more depth regions. Thus, in some embodiments, a particular physical object may be represented by two or more two-dimensional occlusion masks (e.g., the occlusion mask for region 310 may represent a portion of the physical object within 0-1 meter of the user's viewpoint or pose, while the occlusion mask for region 320 may represent different portions of the same physical object between 1 and 2 meters of the user's viewpoint or pose). Such a method may be computationally efficient, but may produce visual artifacts or other inaccuracies.

[0059] In a particular embodiment, the computing system can assign one or more physical objects to one or more predefined depth regions. Using the depth map and corresponding image data, the computing system can use known image segmentation techniques (e.g., using neural networks to generate segmentation masks, etc.) to determine and group one or more pixels in the image corresponding to a specific object. For example, all pixels corresponding to table 210B can form a single segmentation mask. Return to Figure 3 The computing system can determine that table 210B, chair 210C, laptop computer 210D and trash can 210F are located within depth region 320, while plant 210A and chair 210E are located within depth region 330.

[0060] Despite Figure 3 As not shown, but in certain embodiments, a physical object may intersect with the boundaries of depth regions, such that a portion of the physical object lies within two or more depth regions. In this case, the computing system can use a depth map and a segmentation mask to assign a specific physical object to a single depth region. For example, although different pixels representing table 210B may have multiple depth measurements as described above, the computing system can receive all depth measurements corresponding to each pixel of a specific physical object and select a specific value (e.g., the average, median, maximum, minimum, etc. of the depth values ​​received for the specific physical object). In some embodiments, the computing system may further remove one or more outliers from the received depth measurements before selecting the specific value. In certain embodiments, selecting the minimum depth value for a specific physical object (resulting in the physical object being placed in the nearest depth region among the multiple depth regions that the physical object might occupy) can provide the user with a more immersive and realistic artificial reality environment.

[0061] Figure 4A and Figure 4B A two-dimensional occlusion surface is shown, representing at least the visible portion of one or more physical objects located within a predetermined depth region. In a particular embodiment, a computing system may use a depth map to generate one or more two-dimensional occlusion surfaces representing at least the visible portion of one or more physical objects located within a predetermined depth region, which will be displayed within a view of an artificial reality environment. For example, Figure 4A A two-dimensional occlusion surface 410 for depth region 320 may be shown. The two-dimensional occlusion surface 410 includes at least a portion of one or more physical objects located within depth region 320 (e.g., table 210B, chair 210C, laptop computer 210D, and trash can 210F). Figure 4BA two-dimensional occlusion surface 420 may be shown for depth region 330, the two-dimensional occlusion surface 420 including at least a portion of one or more physical objects (e.g., plant 210A and chair 210E) located within depth region 330. In a particular embodiment, a two-dimensional occlusion surface may be generated for each of one or more depth regions representing a real environment.

[0062] In certain embodiments, a two-dimensional occlusion mask can be generated to represent a user's viewing frustum from a specific viewpoint or pose. The two-dimensional occlusion surface can represent one or more physical objects when they should appear from a specific viewpoint, and thus, the user's perspective of the objects in a specific temporal view can be considered. The texture data of the two-dimensional occlusion surface can consist of one or more sub-parts (referred to herein as "texels"). These texels can be blocks (e.g., rectangular blocks) aggregated together to create an array of texels constituting the two-dimensional occlusion surface. By way of example and not limitation, they can be contiguous blocks constituting the surface. For illustrative purposes, the texels of a surface can be conceptualized as pixels similar to those of an image. The two-dimensional occlusion surface can be generated by any suitable device. In certain embodiments, the surface can encode the visual information (RGBA) of one or more of its texels (e.g., as texture). The α component can be a value specifying a level of transparency to be consistent with the texel. As an example and not a limitation, an α value of 0 can indicate that a texel is completely transparent, an α value of 1 can indicate that a texel is opaque, and α values ​​between 0 and 1 can indicate a level of transparency between completely transparent and opaque (the exact level of transparency is determined by this value). The 2D occlusion surface can support any suitable image format. To conserve resources, the 2D occlusion surface can be transmitted as an alpha mask representing the occlusion and blending (e.g., transparency) of each pixel in the segmentation mask. The alpha mask can be a low-resolution texture (64x64 pixels), which reduces power consumption, provides a blurred (but aesthetically pleasing) boundary when overlaid on the occluded object 210 of interest, reduces latency due to the lower resolution, and provides better scalability. In a particular embodiment, a signed distance field can be used to define the contours of the edges or boundaries of each physical object represented in the 2D occlusion mask. Applying a signed distance field can provide an aesthetic modification that results in hard edges for each occlusion mask.

[0063] In certain embodiments, the pose of a 2D occluded surface can be set in a three-dimensional coordinate system corresponding to the real environment (e.g., known world-locked coordinates (x, y, z)) based on a depth region corresponding to a specific 2D occluded surface. The world coordinates of the 2D surface can be based on absolute coordinates in the artificial reality environment (e.g., at specific x, y, coordinates), or the world coordinates of the surface can be determined relative to the pose of the user, HMD, one or more other surfaces, or virtual objects in the artificial reality environment (e.g., a pose set relative to coordinates of a wall or virtual coffee table in the artificial reality environment). This allows the 2D occluded surface to be located at a predetermined distance from the user's viewpoint or pose. For example, in certain embodiments, the pose of the 2D occluded surface can be set in a three-dimensional coordinate system at the maximum, minimum, or average depth of a specific depth region. For example, for a two-dimensional occlusion surface 410 corresponding to a depth region 320 (e.g., between 1 meter and 3 meters from user 102), the pose of the two-dimensional occlusion surface 410 can be set at the maximum depth (e.g., 3 meters from user 102), minimum depth (e.g., 1 meter from user 102), or average depth (e.g., 2 meters from user 102) of the depth region. In some non-limiting embodiments, the computing system can set the pose of the two-dimensional occlusion surface at any appropriate depth within a specific depth region, for example, and not limited to, at the depth of a specific physical object within the depth region (e.g., setting the pose of the two-dimensional occlusion surface 410 at a specific depth at the edge of table 210B). In a particular embodiment, the computing system can set the pose of the two-dimensional occlusion surface such that the surface normal of the particular two-dimensional occlusion surface points towards the viewpoint or pose of the user or HMD.

[0064] In a particular embodiment, the computing system may generate an initial output image of a viewpoint of an artificial reality environment to be displayed to a user. This initial output image may include, for example, one or more two-dimensional occluding surfaces and one or more virtual objects. By way of example, and not limitation, the initial output image of the view may be a view of an artificial reality environment comprising a set of virtual objects (e.g., a virtual bear) and one or more two-dimensional occluding surfaces representing real objects in a real environment. The computing system may utilize ray casting or other rendering processes such as ray tracing to determine the visibility of one or more virtual objects relative to one or more physical objects by comparing models or surfaces representing virtual objects with one or more two-dimensional occluding surfaces within the view of the artificial reality environment.

[0065] In a particular embodiment, a first computing system (e.g., a laptop computer, mobile phone, desktop computer, wearable device) may perform the first ray casting process to render the viewpoint of a scene in an artificial reality environment. The “viewpoint” of the artificial reality environment may refer to the user’s perspective within the artificial reality environment, which may be determined, for example, based on the current position and orientation of the HMD. This use of the ray casting process may be referred to herein as a “visibility test” because it can be used to determine the visibility of a virtual object 510 relative to one or more physical objects by comparing a model of the virtual object to one or more two-dimensional occluding surfaces. For example, a predetermined depth of one or more two-dimensional occluding surfaces allows the computing system to determine and render appropriate occlusion of the virtual object relative to one or more physical objects in the real environment, for example, by occluding a portion of a surface representing the virtual object in the scene based on the pose of one or more two-dimensional occluding surfaces in a three-dimensional coordinate system. The ray casting process can ultimately be used to associate pixels of the screen with intersection points on any object that will be visible to the view of the artificial reality environment.

[0066] In certain embodiments, the generated view can be rendered by one or more components (e.g., CPU, GPU, etc.) of a computing system physically connected to the HMD. However, the HMD may have limited system resources and power, and these limitations are not significantly reduced without resulting in excessive weight, size, and / or heat for user comfort. Therefore, it may be impractical for the HMD to unilaterally handle all the processing tasks involved in rendering the view. In certain embodiments, one or more components may be associated with a device (e.g., a laptop computer, mobile phone, desktop computer, wearable device) that can be used to render the initial view (e.g., perform a ray casting process). In certain embodiments, the device communicates with the computing system on the HMD but may otherwise be physically decoupled from the HMD. By way of example, and not limitation, the device may be a laptop device that is wired to or wirelessly communicates with the HMD. By way of another example, and not limitation, the device may be a wearable device (e.g., a device strapped to the wrist), a handheld device (e.g., a telephone), or some other suitable device (e.g., a laptop computer, tablet computer, desktop computer) that is wired to or wirelessly communicates with the HMD. In a particular embodiment, the device may send the initial view to the HMD for display.

[0067] In a particular embodiment, components of the device generating the initial view may also generate one or more two-dimensional occluding surfaces representing physical objects in the artificial reality environment. As another example, and not a limitation, the onboard computing system of the HMD may generate one or more two-dimensional occluding surfaces after receiving the initial view from a separate computing system (e.g., from the CPU or GPU of a wearable, handheld, or laptop computer). In a particular embodiment, for efficiency purposes, a predetermined maximum number of surfaces (e.g., 16 surfaces) may be generated for the view. While this disclosure focuses on displaying a view to a user on an AR HMD, it envisions displaying a scene to a user on a VR display or any other suitable device.

[0068] After setting the pose of the 2D occluded surfaces and generating an initial view of the artificial reality environment, the computing system may need to make one or more adjustments to the view to accurately show the current pose of one or more 2D occluded surfaces to the user. These adjustments are unique for displaying artificial reality via, for example, an AR HMD 110 that allows users to simultaneously see through and perceive their surrounding real-world environment. For instance, because the user is experiencing the artificial reality environment in real time, after executing the process described herein to render an image of the artificial reality environment for display, the computing system may, approximately 100 milliseconds later, be in the current pose of the user's head (e.g., the HMD pose), resulting in a slightly different perspective from the initially rendered view. Therefore, if there is a significant delay in updating the view to reflect this change in perspective regarding the artificial reality environment that allows the user to perceive their surrounding real-world environment, the user can perceive this delay, resulting in sensory dissonance. To prevent this dissonance, the content displayed to the user may need to account for changes in the user's perspective at a very fast rate. However, re-rendering the entire artificial reality environment to account for these changes in perspective takes time. As an example, and not a limitation, even with powerful computing systems, frames can only be rendered at 60 Hz in this manner. This means that any changes in perspective that occur within 1 / 60th of a second (e.g., due to rapid head movements) will not be captured for rendering and display. Furthermore, re-rendering the entire view is a resource-intensive task that may involve performing numerous "visibility tests" to obtain visual information (e.g., color and transparency information) to render the view.

[0069] To remedy any changes in perspective that might result from user head movement during the delay period, instead of re-rendering the entire view, the computing system can adjust the pose of one or more 2D occluding surfaces for a specific view to process the rendered view quickly and efficiently. This adjustment can take into account changes in the user's perspective in the artificial reality environment during the time the view is rendered (e.g., due to the HMD on the user's head moving to different positions and / or orientations).

[0070] In certain embodiments, such a shortcut for approximating view changes may include resampling a two-dimensional occluding surface within the view (instead of re-rendering the entire view). In certain embodiments, resampling may include performing a second ray-casting process to determine an approximation of how the two-dimensional occluding surface may appear from an adjusted viewpoint, which may be caused by one or more movements of the user's head (and the HMD and camera) during the delay period. The adjusted viewpoint may be based on the latest viewpoint, position, and orientation information from the HMD, which the computing system may receive continuously or semi-continuously (e.g., using inertial sensors, optical sensors, depth sensors, and / or other sensors on the HMD or on a remote device tracking the HMD) to determine the user's viewpoint at any given time while wearing the HMD. In certain embodiments, the computing system may also continuously or semi-continuously track the user's eye positioning (e.g., to adjust for distortions caused by the HMD's lenses based on the user's gaze). In certain embodiments, the latest viewpoint may be further determined based on (e.g., as determined by eye tracking) the user's gaze.

[0071] By focusing on resampling only a limited number of surfaces within the view (e.g., 16 surfaces), the view can be modified quickly, fast enough to prevent or reduce the user's perception of latency. As an example, and not a limitation, a further second ray casting process can be performed every 1 / 200th of a second to account for possible changes in viewing angle due to variations in the HMD's positioning or orientation. In a particular embodiment, the HMD's onboard computing system can resample one or more surfaces by performing a ray casting process to reflect the user's latest viewing angle. If the viewing angle has changed since the previous ray casting process, hypothetical rays might intersect different points on the surfaces accordingly. Modifying the view based on resampling in this way may only be an approximation of how the view should change with changes in viewing angle, and this approximation may only apply to relatively small changes in viewing angle. But this may be all that is needed, as the aim is to account for changes occurring within relatively short time intervals (e.g., on the order of a hundredth of a second) between frames that generate a fully rendered image. Essentially, it can be a temporary fix until the view can be re-rendered (e.g., by a more powerful computing system, such as a wearable device). As an example, and not a limitation, a more powerful computing system that can be decoupled from the HMD (e.g., a wearable device attached to a user's belt or waist) can re-render the view from scratch every 1 / 60th of a second and can send these re-rendered views to the HMD device during rendering. In this example, in the time between receiving the re-rendered view every 1 / 60th of a second, the HMD device can modify the surface itself (e.g., every 1 / 200th of a second) to account for rapid changes in the user's perspective.

[0072] In a particular embodiment, the ray casting process may be performed individually for each of the user's eyes to account for minute changes in viewpoint (e.g., based on the relative positioning of the eyes). As an example, and not a limitation, the first and second ray casting processes may be performed on a screen associated with the left eye and a screen associated with the right eye. These processes may operate independently of each other, or even asynchronously.

[0073] Figure 5 An output image of an artificial reality environment, including one or more two-dimensional occluding surfaces and one or more virtual objects (e.g., a virtual bear 510), is shown. The time-adjusted resampled pose of the two-dimensional occluding surfaces can be transferred to the HMD for display. The computing system can generate the output image based on the determined visibility of the virtual objects. Generating the output image can be done on the computing system's GPU for display by rendering surfaces as viewed from the user's current viewing position. Based on this information, the computing system can display the view on the HMD accordingly, based on the relative occlusion determined according to the latest user head pose, and the transparency of the two-dimensional occluding surfaces and / or virtual objects in the rendered view. Two-dimensional occluding surfaces (which can be alpha masks) can be used to occlude virtual objects that should be occluded by physical objects in the real environment. For example, if an opaque portion of a two-dimensional occluding surface 410 lies in front of a portion of a virtual object 510 in the environment, that portion of the virtual object 510 will not be displayed, such as... Figure 5 As shown. Then, an immersive artificial reality environment can be displayed to the user accordingly.

[0074] While the 2D occlusion surfaces described herein are well-suited for certain artificial reality applications, in some applications, a hybrid approach combining 2D occlusion surfaces with pre-occlusion surfaces representing virtual objects may be advantageous for generating viewpoints in artificial reality environments. For example, because they are close to the user, the user's perspective on nearby physical and virtual objects can change dramatically even with small movements or changes in viewpoint or user pose. Therefore, using 2D occlusion surfaces on nearby physical or virtual objects when rendering objects close to the user can lead to visual artifacts or inaccuracies. Using pre-occlusion surfaces on nearby objects, while potentially more resource-intensive than using 2D occlusion surfaces, can provide a more aesthetically pleasing and immersive artificial reality experience for the user. Specific embodiments employ a hybrid approach where the computational system generates and displays pre-occlusion surfaces for virtual objects located near the user and generates 2D occlusion surfaces for physical objects located at a distance greater than a minimum distance from the user's viewpoint or pose. These two techniques can be used individually or in combination to generate viewpoints in artificial reality environments.

[0075] In certain embodiments, this hybrid approach can be employed based on the depth of one or more physical objects in the real environment as viewed from the user's viewpoint or pose. The computing system can identify one or more physical or virtual objects in the real environment that lie within a specific predefined depth range or within a minimum predefined distance from the user's current pose or viewpoint (e.g., within 1 meter of the user). Using a 3D mesh of the real environment, the computing system can determine the depth of each physical object within the minimum predefined distance, or the depth of each pixel within the view frustum. The computing system can then compare each of these depths to the depth of a posed virtual object to determine which physical objects or pixels are in front of and behind the virtual object. This allows the computing system to determine the visibility of the virtual object from the user's current viewpoint. Based on the visibility determination, the computing system can then render a pre-occluded 2D surface representing the virtual object as viewed from the user's specific viewpoint. The rendered pre-occluded 2D surface of the virtual object can be utilized like any other surface in this disclosure (e.g., for posing in a 3D coordinate system, for user-based movement adjustments, for visibility testing, etc.). For example, a computing system can use one or more pre-occluded 2D surfaces to represent areas within a specific distance from the user's pose or within a certain depth region (e.g., return). Figure 3 Any physical or virtual object within depth region 310 (between 0 and 1 meter from the user) and using a two-dimensional occlusion surface to represent any physical object located outside that distance (e.g., within depth regions 320 and 330 (e.g., more than 1 meter from the user)).

[0076] Figure 6 An example method 600 for generating an output image based on the visibility of determined virtual objects is shown. The method may begin at step 610, whereby the computing system can generate a depth map of the real environment as seen from the user's viewpoint, the depth map comprising pixels with corresponding depth values ​​for one or more physical objects in the real environment.

[0077] In step 620, the computing system may generate a two-dimensional occlusion surface based on the depth map, the two-dimensional occlusion surface representing at least the visible portion of one or more physical objects located within a predetermined depth range defined relative to the viewpoint.

[0078] In step 630, the calculation system can set the pose of the two-dimensional occlusion surface in the three-dimensional coordinate system, so that the two-dimensional occlusion surface is located at a predetermined distance from the viewpoint.

[0079] In step 640, the computing system can determine the visibility of the virtual object relative to one or more physical objects by comparing the model of the virtual object with a two-dimensional occlusion surface.

[0080] In step 650, the computing system can generate an output image based on the visibility of the determined virtual objects.

[0081] Where appropriate, certain embodiments may be repeated. Figure 6 One or more steps of the method. Although this disclosure describes and illustrates steps occurring in a particular order. Figure 6 The method involves specific steps, but this disclosure contemplates that they may occur in any suitable order. Figure 6 Any suitable steps of the method. Furthermore, although this disclosure describes and illustrates example methods for generating output images based on the visibility of determined virtual objects (which include...) Figure 6 This disclosure contemplates any suitable method for generating an output image based on the visibility of a determined virtual object, including any appropriate steps, where suitable, it may include... Figure 6 All steps, some steps, or none of the steps of the method Figure 6 Any steps of the method. Furthermore, although this disclosure describes and illustrates the implementation... Figure 6 The method may refer to a specific component, device, or system of a particular step, but this disclosure contemplates the execution of... Figure 6 Any suitable step of the method, any suitable component, device or system, or any suitable combination thereof.

[0082] Figure 7 A system diagram of the display engine 700 is shown. The display engine 700 may include four types of top-level blocks. For example... Figure 7 As shown, these blocks may include control block 710, transformation blocks 720a and 720b, pixel blocks 730a and 730b, and display blocks 740a and 740b. One or more components of the display engine 700 may be configured to communicate via one or more high-speed buses, shared memory, or any other suitable method. Figure 7 As shown, the control block 710 of the display engine 700 can be configured to communicate with the transformation blocks 720a and 720b and the pixel blocks 730a and 730b. The display blocks 740a and 740b can be configured to communicate with the control block 710. As explained in further detail herein, this communication may include data as well as control signals, interrupts, and other instructions.

[0083] In a particular embodiment, control block 710 may receive input data stream 760 from the main rendering component and initialize the pipeline in display engine 700 to complete rendering for display. In a particular embodiment, input data stream 760 may include data and control packets from the main rendering component. The data and control packets may include information, such as one or more surfaces including texture data and position data, and additional rendering instructions. Control block 710 may distribute data to one or more other blocks of display engine 700 as needed. Control block 710 may initiate pipeline processing for one or more frames to be displayed. In a particular embodiment, HMD may include multiple display engines 700, and each display engine may include its own control block 710.

[0084] In a particular embodiment, transform blocks 720a and 720b can determine initial visibility information of surfaces to be displayed in a view of an artificial reality environment. Generally, transform blocks (e.g., transform blocks 720a and 720b) ​​can project light from pixel locations on the screen and generate filtering commands (e.g., filtering based on bilinear or other types of interpolation techniques) to be sent to pixel blocks 730a and 730b. Transform blocks 720a and 720b can project light from the user's current viewpoint (e.g., determined using an inertial measurement unit, eye tracker, and / or any suitable tracking / localization algorithm, such as Simultaneous Localization and Mapping (SLAM)) onto the artificial scene where surfaces are located, and can produce results to be sent to the corresponding pixel blocks 730a and 730b.

[0085] Generally, according to a specific embodiment, transform blocks 720a and 720b may each include a four-stage pipeline. The stages of the transform blocks can be configured as follows: A ray projector can emit ray beams corresponding to an array of one or more aligned pixels, called tiles (e.g., each tile may include 16×16 aligned pixels). The ray beams may be distorted before entering the artificial reality environment according to one or more distortion grids. The distortion grids can be configured to correct geometric distortion effects originating at least from the displays 750a and 750b of the HMD. Transform blocks 720a and 720b can determine whether each ray beam intersects a surface in the artificial reality environment by comparing the bounding box of each tile with the bounding box of each surface. If a ray beam does not intersect an object, it may be discarded. Tile surface intersections are detected, and the corresponding tile surface pairs 725a and 725b are passed to pixel blocks 730a and 730b.

[0086] Generally, according to a specific embodiment, pixel blocks 730a and 730b determine color values ​​from tile surface pairs 725a and 725b to generate pixel color values. The color value of each pixel is sampled from the texture data of the surface received and stored by control block 710 (e.g., as part of input data stream 760). Pixel blocks 730a and 730b receive tile surface pairs 725a and 725b from transform blocks 720a and 720b, respectively, and schedule bilinear filtering. For each tile surface pair 725a and 725b, pixel blocks 730a and 730b can sample the color information of the pixels within the tile using the color values ​​corresponding to the locations where the projected tiles intersect with the surfaces. In a specific embodiment, pixel blocks 730a and 730b can process the red, green, and blue components separately for each pixel. Pixel blocks 730a and 730b can then output pixel color values ​​735a and 735b to display blocks 740a and 740b, respectively.

[0087] Typically, display blocks 740a and 740b can receive pixel color values ​​735a and 735b from pixel blocks 730a and 730b, convert the data format to a scanline output more suitable for the display, apply one or more brightness corrections to the pixel color values ​​735a and 735b, and prepare the pixel color values ​​735a and 735b for output to displays 750a and 750b. Display blocks 740a and 740b can convert the tile-order pixel color values ​​735a and 735b generated by pixel blocks 730a and 730b into scanline or line-order data that displays 750a and 750b may require. Brightness correction can include any desired brightness correction, gamma mapping, and dithering. Display blocks 740a and 740b can directly provide pixel outputs 745a and 745b to displays 750a and 750b, such as corrected pixel color values, or they can provide pixel outputs 745a and 745b to blocks outside the display engine 700 in various formats. For example, an HMD may include additional hardware or software to further customize back-end color processing, support wider interfaces for displays, or optimize display speed or fidelity.

[0088] Figure 8 An example network environment 800 associated with a social networking system is shown. Network environment 800 includes client systems 830, a social networking system 860, and a third-party system 870 connected to each other via network 810. Although Figure 8A specific arrangement of client system 830, social networking system 860, third-party system 870, and network 810 is shown; however, this disclosure contemplates any suitable arrangement of client system 830, social networking system 860, third-party system 870, and network 810. By way of example and not limitation, two or more of client system 830, social networking system 860, and third-party system 870 may bypass network 810 and be directly connected to each other. As another example, two or more of client system 830, social networking system 860, and third-party system 870 may be physically or logically located in the same place as each other, wholly or partially. Furthermore, although... Figure 8 A specific number of client systems 830, social networking systems 860, third-party systems 870, and networks 810 are shown, but this disclosure contemplates any suitable number of client systems 830, social networking systems 860, third-party systems 870, and networks 810. As an example and not as a limitation, network environment 800 may include multiple client systems 830, social networking systems 860, third-party systems 870, and networks 810.

[0089] This disclosure contemplates any suitable network 810. By way of example and not limitation, one or more portions of network 810 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 810 may include one or more networks 810.

[0090] Link 850 can connect client system 830, social networking system 860, and third-party system 870 to communication network 810 or to each other. This disclosure contemplates any suitable link 850. In a particular embodiment, one or more links 850 include one or more wired (such as, for example, Digital Subscriber Line (DSL) or Coaxial Cable Data Service Interface Specification (DOCSIS)) links, wireless (such as, for example, Wi-Fi or Global Microwave Access Interoperability (WiMAX)) links, or optical (such as, for example, Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In a particular embodiment, one or more links 850 each include an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, a portion of the Internet, a portion of the PSTN, a cellular-based network, a satellite-based network, another link 850, or a combination of two or more such links 850. Links 850 need not be identical throughout the network environment 800. One or more first links 850 may differ from one or more second links 850 in one or more aspects.

[0091] In a particular embodiment, client system 830 may be an electronic device comprising hardware, software, or embedded logic components, or a combination of two or more such components, and capable of performing appropriate functions implemented or supported by client system 830. By way of example and not limitation, client system 830 may include computer systems such as desktop computers, laptop or notebook computers, netbooks, tablet computers, e-book readers, GPS devices, cameras, personal digital assistants (PDAs), handheld electronic devices, cellular phones, smartphones, augmented / virtual reality devices, other suitable electronic devices, or any suitable combination thereof. This disclosure contemplates any suitable client system 830. Client system 830 enables network users at client system 830 to access network 810. Client system 830 enables its users to communicate with other users at other client systems 830.

[0092] In a particular embodiment, client system 830 may include web browser 832 and may have one or more add-ons, plugins, or other extensions. A user at client system 830 may enter a Uniform Resource Locator (URL) or another address that directs web browser 832 to a specific server (e.g., server 862 or a server associated with third-party system 870), and web browser 832 may generate a Hypertext Transfer Protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and, in response, pass one or more Hypertext Markup Language (HTML) files to client system 830. Client system 830 may render a webpage based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable webpage file. By way of example and not limitation, webpages may be rendered from HTML files, Extensible Hypertext Markup Language (XHTML) files, or Extensible Markup Language (XML) files as needed. Such pages may also execute scripts, markup languages, and combinations thereof. Throughout this document, references to a webpage include one or more corresponding webpage files (which a browser may use to render the webpage), and vice versa, where appropriate.

[0093] In a particular embodiment, the social networking system 860 may be a network-addressable computing system capable of hosting online social networks. The social networking system 860 may generate, store, receive, and transmit social network data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to the online social network). The social networking system 860 may be accessed directly or via network 810 by other components of the network environment 800. By way of example and not limitation, the client system 830 may access the social networking system 860 directly or via network 810 using a web browser 832 or a native application associated with the social networking system 860 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In a particular embodiment, the social networking system 860 may include one or more servers 862. Each server 862 may be a unitary server or a distributed server spanning multiple computers or multiple data centers. Server 862 can be of various types, such as, and without limitation, a web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 862 may include hardware, software, or embedded logic components, or a combination of two or more such components for performing suitable functions implemented or supported by server 862. In a particular embodiment, social networking system 860 may include one or more data stores 864. Data stores 864 can be used to store various types of information. In a particular embodiment, the information stored in data stores 864 may be organized according to a particular data structure. In a particular embodiment, each data store 864 may be a relational database, a columnar database, a correlation database, or other suitable database. Although this disclosure describes or illustrates specific types of databases, this disclosure contemplates any suitable type of database. Specific embodiments may provide an interface that enables client system 830, social network system 860, or third-party system 870 to manage, retrieve, modify, add, or delete information stored in data storage 864.

[0094] In a particular embodiment, the social network system 860 may store one or more social graphs in one or more data stores 864. In a particular embodiment, the social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a specific user) or multiple concept nodes (each corresponding to a specific concept)—and multiple edges connecting the nodes. The social network system 860 may provide users of the online social network with the ability to communicate and interact with other users. In a particular embodiment, a user may join an online social network via the social network system 860 and then add connections (e.g., relationships) to multiple other users they wish to connect with in the social network system 860. In this document, the term "friend" may refer to any other user of the social network system 860 with whom the user forms a connection, association, or relationship via the social network system 860.

[0095] In a particular embodiment, the social networking system 860 may provide users with the ability to take actions on various types of items or objects supported by the social networking system 860. By way of example and not limitation, items and objects may include groups or social networks to which the user of the social networking system 860 belongs, events or calendar entries that the user may be interested in, computer-based applications that the user can use, transactions that allow the user to buy or sell items via services, interactions with advertisements that the user can perform, or other suitable items or objects. Users may interact with anything that can be represented in the social networking system 860 or by an external system represented by a third-party system 870, which is decoupled from and coupled to the social networking system 860 via a network 810.

[0096] In a particular embodiment, the social networking system 860 can link various entities. By way of example and not limitation, the social networking system 860 can enable users to interact with each other and receive content from third-party systems 870 or other entities, or allow users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0097] In certain embodiments, third-party system 870 may include one or more types of servers, one or more data storage devices, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., servers may communicate with these components). Third-party system 870 may be operated by an entity different from the entity operating social networking system 860. However, in certain embodiments, social networking system 860 and third-party system 870 may operate in combination to provide social networking services to users of either social networking system 860 or third-party system 870. In this sense, social networking system 860 may provide a platform or backbone that other systems (e.g., third-party system 870) can use to provide social networking services and functionality to users across the Internet.

[0098] In a particular embodiment, third-party system 870 may include a third-party content object provider. The third-party content object provider may include one or more sources of content objects that can be delivered to client system 830. As an example, and not a limitation, the content object may include information about things or activities that the user is interested in, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example, and not a limitation, the content object may include incentive content objects (e.g., coupons, discount vouchers, gift certificates, or other suitable incentives).

[0099] In a particular embodiment, the social networking system 860 also includes user-generated content objects that can enhance user interaction with the social networking system 860. User-generated content can include any content that a user can add, upload, send, or "post" to the social networking system 860. As an example, and not as a limitation, a user posts a message from client system 830 to social networking system 860. A post can include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the social networking system 860 by third parties via "communication channels" (e.g., newsfeeds or streams).

[0100] In certain embodiments, the social networking system 860 may include various servers, subsystems, programs, modules, logs, and data stores. In certain embodiments, the social networking system 860 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertising-targeting module, a user interface module, a user profile store, a connection store, a third-party content store, or a location store. The social networking system 860 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 860 may include one or more user profile stores for storing user profiles. User profiles may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinity, or location). Interest information may include interests related to one or more categories. Categories may be general or specific. As an example, and not a limitation, if a user “likes” an article about a brand of shoes, the category may be the brand, or the general category of “shoes” or “clothing.” A relationship store may be used to store relationship information about a user. Relationship information may indicate users who have similar or shared work experience, group memberships, hobbies, educational history, or are related or share common attributes in any way. Relationship information may also include user-defined relationships between different users and content (internal and external). A web server may be used to link the social networking system 860 to one or more client systems 830 or one or more third-party systems 870 via network 810. The web server may include a mail server or other messaging functionality for receiving and sending messages along prescribed routes between the social networking system 860 and one or more client systems 830. An API request server may allow third-party systems 870 to access information from the social networking system 860 by calling one or more APIs. An action recorder may be used to receive communications from the web server regarding user actions on or outside the social networking system 860. By combining action logs, third-party content object logs can be maintained, recording user exposure to third-party content objects. The notification controller can provide client system 830 with information about the content objects.Information may be pushed to client system 830 as a notification, or it may be retrieved from client system 830 in response to a request received from client system 830. An authorization server may be used to implement one or more privacy settings for users of social networking system 860. A user's privacy settings determine how specific information associated with that user can be shared. The authorization server may, for example, allow users to opt in or out by setting appropriate privacy settings, enabling their actions to be recorded by social networking system 860 or shared with other systems (e.g., third-party system 870). A third-party content object store may be used to store content objects received from third parties (e.g., third-party system 870). A location store may be used to store location information received from client system 830 associated with the user. An advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0101] Figure 9 An example computer system 900 is illustrated. In a particular embodiment, one or more computer systems 900 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 900 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 900 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. Specific embodiments include one or more portions of one or more computer systems 900. Herein, references to computer systems may include computing devices, and references to computing devices may include computer systems, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0102] This disclosure contemplates any suitable number of computer systems 900. The computer systems 900 are contemplated to take any suitable physical form. By way of example and not limitation, the computer system 900 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system mesh, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these systems. Where appropriate, the computer system 900 may include one or more computer systems 900; may be monolithic or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 900 may perform one or more steps of the methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 900 may execute one or more steps of the methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 900 may execute one or more steps of the methods described or illustrated herein at different times or at different locations.

[0103] In a particular embodiment, computer system 900 includes processor 902, memory 904, storage device 906, input / output (I / O) interface 908, communication interface 910, and bus 912. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0104] In a particular embodiment, processor 902 includes hardware for executing instructions, such as those constituting a computer program. By way of example, and not limitation, to execute instructions, processor 902 may retrieve (or fetch) instructions from internal registers, internal caches, memory 904, or storage device 906; decode and execute them; and then write one or more results to internal registers, internal caches, memory 904, or storage device 906. In a particular embodiment, processor 902 may include one or more internal caches for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 902 may include any suitable number of suitable internal caches. By way of example, and not limitation, processor 902 may include one or more instruction caches, one or more data caches, and one or more translation lookup buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 904 or storage device 906, and the instruction cache may accelerate the retrieval of those instructions by processor 902. The data in the data cache may be: a copy of data in memory 904 or storage device 906 for operating instructions executed at processor 902; the result of a previous instruction executed at processor 902 for access by a subsequent instruction executed at processor 902 or for writing to memory 904 or storage device 906; or other suitable data. The data cache can accelerate read or write operations performed by processor 902. The TLB can accelerate virtual address translation with respect to processor 902. In a particular embodiment, processor 902 may include one or more internal registers for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 902 may include any suitable number of suitable internal registers. Where appropriate, processor 902 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 902. Although this disclosure describes and illustrates particular processors, this disclosure contemplates any suitable processor.

[0105] In a particular embodiment, memory 904 includes main memory for storing instructions for causing processor 902 to execute or data for causing processor 902 to operate. As an example, and not as a limitation, computer system 900 may load instructions from storage device 906 or another source (e.g., another computer system 900) into memory 904. Processor 902 may then load instructions from memory 904 into internal registers or internal caches. To execute instructions, processor 902 may retrieve instructions from internal registers or internal caches and decode them. During or after the execution of instructions, processor 902 may write one or more results (which may be intermediate or final results) to internal registers or internal caches. Processor 902 may then write one or more of these results to memory 904. In a particular embodiment, processor 902 executes only instructions in one or more internal registers or internal caches or in memory 904 (not storage device 906 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 904 (not storage device 906 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 902 to memory 904. As described below, bus 912 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between processor 902 and memory 904 and facilitate access to memory 904 requested by processor 902. In a particular embodiment, memory 904 includes random access memory (RAM). Where appropriate, the RAM may be volatile memory. Where appropriate, the RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, the RAM may be single-port RAM or multi-port RAM. This disclosure contemplates any suitable RAM. Where appropriate, memory 904 may include one or more memories 904. Although this disclosure describes and illustrates specific memories, this disclosure contemplates any suitable memories.

[0106] In a particular embodiment, storage device 906 includes a mass storage device for data or instructions. By way of example and not limitation, storage device 906 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk drive, magneto-optical disk drive, magnetic tape drive, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 906 may include removable or non-removable (or fixed) media. Where appropriate, storage device 906 may be internal or external to computer system 900. In a particular embodiment, storage device 906 is a non-volatile solid-state memory. In a particular embodiment, storage device 906 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage device 906 in any suitable physical form. Where appropriate, storage device 906 may include one or more storage device control units that facilitate communication between processor 902 and storage device 906. Where appropriate, storage device 906 may include one or more storage devices 906. Although this disclosure describes and illustrates specific storage devices, any suitable storage device is contemplated herein.

[0107] In a particular embodiment, I / O interface 908 includes hardware, software, or both that provide one or more interfaces for communication between computer system 900 and one or more I / O devices. Where appropriate, computer system 900 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and computer system 900. By way of example and not limitation, I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touchscreen, trackball, video camera, another suitable I / O device, or a combination of two or more of these. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 908 for them. Where appropriate, I / O interface 908 may include one or more device or software drivers that enable processor 902 to drive one or more of these I / O devices. Where appropriate, I / O interface 908 may include one or more I / O interfaces 908. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure contemplates any suitable I / O interface.

[0108] In a particular embodiment, communication interface 910 includes hardware, software, or both providing one or more interfaces for communication (e.g., packet-based communication) between computer system 900 and one or more other computer systems 900 or one or more networks. By way of example and not limitation, communication interface 910 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired-based networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 910 for it. By way of example and not limitation, computer system 900 may communicate with one or more portions of an ad hoc network, personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of these networks may be wired or wireless. As an example, computer system 900 may communicate with a wireless PAN (WPAN) (e.g., Bluetooth WPAN), a Wi-Fi network, a Wi-Fi Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, computer system 900 may include any suitable communication interface 910 for any of these networks. Where appropriate, communication interface 910 may include one or more communication interfaces 910. Although specific communication interfaces are described and shown in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0109] In a particular embodiment, bus 912 includes hardware, software, or both, that couples components of computer system 900 to each other. By way of example and not limitation, bus 912 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Where appropriate, bus 912 may include one or more buses 912. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.

[0110] In this document, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0111] In this document, unless otherwise expressly indicated or indicated by the context, "or" is inclusive rather than exclusive. Therefore, in this document, unless otherwise expressly indicated or indicated by the context, "A or B" means "A, B, or both." Furthermore, unless otherwise expressly indicated or indicated by the context, "and" is both joint and separate. Therefore, in this document, unless otherwise expressly indicated or indicated by the context, "A and B" means "A and B, jointly or separately."

[0112] The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although this disclosure describes and illustrates corresponding embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or substitution of any component, element, feature, function, operation, or step described or illustrated anywhere herein that will be understood by those skilled in the art. Furthermore, references in the appended claims to a means or system or a component of a means or system suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform a particular function include that means, system, or component, whether or not it or that particular function is activated, turned on, or unlocked, provided that the means, system, or component is so adapted, arranged, enabled, configured, implemented, operable, or operated. Furthermore, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide some, all, or none of these advantages.

Claims

1. A method comprising using a computing system: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A first two-dimensional occlusion mask is generated based on the depth map. The first two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a first predetermined depth range defined relative to the viewpoint. A second two-dimensional occlusion mask is generated based on the depth map. The second two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a second predetermined depth range defined relative to the viewpoint, wherein the minimum depth of the second predetermined depth range is greater than the maximum depth of the first predetermined depth range. Set the poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask in a three-dimensional coordinate system; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the first two-dimensional occlusion mask and the second two-dimensional occlusion mask; and An output image is generated based on the determined visibility of the virtual object.

2. The method according to claim 1, wherein, The poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are set such that the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are located at a first predetermined distance and a second predetermined distance from the viewpoint, respectively.

3. The method according to claim 1, further comprising: A third two-dimensional occlusion mask is generated based on the depth map. The third two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a third predetermined depth range relative to the viewpoint, wherein the minimum depth of the third predetermined depth range is greater than the maximum depth of the second depth range. and The pose of the third two-dimensional occlusion mask is set in the three-dimensional coordinate system such that the third two-dimensional occlusion mask is located at a third predetermined distance from the viewpoint.

4. The method according to claim 3, wherein, The third predetermined depth range defined relative to the viewpoint is between 1) greater than 3 meters and 2) 4 meters from the viewpoint.

5. The method according to claim 1, wherein, Setting the pose of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask includes adjusting the position or orientation of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask to take into account changes in the viewpoint of the one or more physical objects, which are caused by one or more movements of the user-worn camera after the time the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are generated.

6. The method according to claim 5, wherein, The adjustment of the position or orientation of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask is performed by one or more components of the computing system physically connected to the head-mounted display worn by the user.

7. The method according to claim 1, wherein, The depth map is calculated based on one or more images received from a camera attached to a head-mounted display worn by the user.

8. The method according to claim 1, wherein, The first two-dimensional occlusion mask and the second two-dimensional occlusion mask include a plurality of texels, wherein each texel is assigned a value that specifies the level of transparency to be assigned to the texel.

9. The method according to claim 1, wherein, The first predetermined depth range defined relative to the viewpoint is between 1) 1 meter and 2) 2 meters from the viewpoint.

10. The method according to claim 9, wherein, The second predetermined depth range defined relative to the viewpoint is between 1) greater than 2 meters and 2) 3 meters from the viewpoint.

11. The method according to claim 1, wherein, The first predetermined depth range and the second predetermined depth range have the same width.

12. The method according to claim 11, wherein, The widths of the first predetermined depth range and the second predetermined depth range are based on one or more of the following: The size of the real environment; The applications that the user of the computing system is interacting with; and The user's velocity or acceleration when the first two-dimensional occlusion mask is generated.

13. The method according to claim 1, wherein, The computing system includes one or more computing devices, wherein one or more computing devices include a head-mounted display, and one or more computing devices include individual computing devices.

14. The method of claim 13, further comprising the step of allocating the method between the computing device of the head-mounted display and a separate computing device based on one or more metrics of available computing resources.

15. One or more computer-readable non-transitory storage media comprising software, which, when executed, is operable to: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A first two-dimensional occlusion mask is generated based on the depth map. The first two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a first predetermined depth range defined relative to the viewpoint. A second two-dimensional occlusion mask is generated based on the depth map. The second two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a second predetermined depth range defined relative to the viewpoint, wherein the minimum depth of the second predetermined depth range is greater than the maximum depth of the first predetermined depth range. Set the poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask in a three-dimensional coordinate system; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the first two-dimensional occlusion mask and the second two-dimensional occlusion mask; and An output image is generated based on the determined visibility of the virtual object.

16. The medium according to claim 15, wherein, The poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are set such that the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are located at a first predetermined distance and a second predetermined distance from the viewpoint, respectively.

17. The medium according to claim 15, wherein, Setting the pose of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask includes adjusting the position or orientation of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask to take into account changes in the viewpoint of the one or more physical objects, which are caused by one or more movements of the user-worn camera after the time the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are generated.

18. A system comprising: One or more processors; and One or more computer-readable storage media coupled to one or more of the processors and including instructions that, when executed by one or more of the processors, are operable to cause the system to: Generate a depth map of the real environment as seen from the user's viewpoint, the depth map including pixels having corresponding depth values ​​for one or more physical objects in the real environment; A first two-dimensional occlusion mask is generated based on the depth map. The first two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a first predetermined depth range defined relative to the viewpoint. A second two-dimensional occlusion mask is generated based on the depth map. The second two-dimensional occlusion mask represents at least the visible portion of the one or more physical objects located within a second predetermined depth range defined relative to the viewpoint, wherein the minimum depth of the second predetermined depth range is greater than the maximum depth of the first predetermined depth range. Set the poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask in a three-dimensional coordinate system; The visibility of the virtual object relative to the one or more physical objects is determined by comparing the model of the virtual object with the first two-dimensional occlusion mask and the second two-dimensional occlusion mask; and An output image is generated based on the determined visibility of the virtual object.

19. The system according to claim 18, wherein, The poses of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are set such that the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are located at a first predetermined distance and a second predetermined distance from the viewpoint, respectively.

20. The system according to claim 18, wherein, Setting the pose of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask includes adjusting the position or orientation of the first two-dimensional occlusion mask and the second two-dimensional occlusion mask to take into account changes in the viewpoint of the one or more physical objects, which are caused by one or more movements of the user-worn camera after the time the first two-dimensional occlusion mask and the second two-dimensional occlusion mask are generated.

Images