View synthesis pipeline for rendering unvarnished images

Abstract

A processor accesses a depth map and a first image of a scene, the depth map and the first image being generated using one or more sensors of an artificial reality device. The processor generates a plurality of segmentation masks based on the first image, the plurality of segmentation masks being respectively associated with a plurality of object types. The segmentation masks segment the depth map into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the object types. The processor generates a plurality of grids using the segmented depth maps, respectively. For each eye of the user, the processor acquires a second image and generates segmentation information based on the second image. The processor distorts the plurality of meshes to generate a distorted mesh for the eye, and then generates an eye-specific mesh for the eye by compositing the distorted mesh according to the segmentation information. The processor renders an output image for the eye using the second image and the eye-specific grid.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 63 / 379,035, filed on October 11, 2022, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to computer graphics and, more particularly, to mixed reality rendering techniques. Background Art

[0004] Artificial reality is a form of reality that has been adjusted in some way before being presented to a user. Artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content (e.g., mixed reality images) may include fully generated content or generated content combined with captured content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). Artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0005] "Passthrough" is a feature that allows a user to see their physical environment while wearing an artificial reality system. By having the headset of the artificial reality system display information captured by the outward-facing camera of the headset, information about the user's physical environment is visually "passed through" to the user. This visual information, which can be referred to as "passthrough", allows the user to see their physical environment while wearing the HMD. By having the HMD display information captured by the outward-facing camera of the headset, information about the user's physical environment is visually "passed through" to the user. However, simply displaying the captured image does not work as expected. Since the position of the camera is inconsistent with the position of the user's eyes, the image captured by the camera does not accurately reflect the user's perspective. In addition, since the image does not have depth, simply displaying the image will not provide the user with the appropriate parallax effect if the user moves away from the location where the image was taken. Incorrect parallax coupled with user motion may cause motion sickness. Therefore, depth information is an important part of the passthrough rendering process. Summary of the invention

[0006] Specific embodiments described herein relate to an improved view synthesis pipeline or view synthesis architecture for rendering depth-accurate pass-through images for the left and right eyes of a user. In the context of MR, a series of frames of pass-through images are rendered to provide an immersive experience. Since each pass-through image is generated based on a corresponding depth map, the temporal consistency of each depth map plays an important role in ensuring a consistent and smooth pass-through experience.

[0007] Each specific embodiment provides a technique for improving the temporal consistency and smoothness of a depth map used to render a pass-through image. The original depth map of the current frame output by one or more sensors can be temporally smoothed using optical flow data generated from an image of the scene. The original depth map that may be associated with time t may be noisy and lack temporal consistency with the previous depth map associated with time t-1. It may be necessary to temporally smooth the original depth map before using the original depth map to generate a pass-through image. Preferably, the original depth map is temporally smoothed using optical flow data generated from a source different from the depth map, because such optical flow data provides independent information for correcting the depth map. In one example, the optical flow data can be generated using an image of the scene. The image can be a brightness image (e.g., a monochrome image that encodes the brightness of the scene), which can be generated as a byproduct when generating a depth map. For example, the brightness image can be generated by a time-of-flight sensor, or the brightness image can be one of the images in a stereo pair of images used to calculate depth. In certain embodiments, a computing system may generate optical flow data to represent motion between a previous luminance image (e.g., associated with time t-1) and a current luminance image (e.g., associated with time t). Optical flow may be calculated using conventional techniques, including finding pixel correspondences between pixels in the two images and encoding the correspondences using motion vectors. The optical flow data may then be used to temporally smooth the original depth map. For example, the computing system may apply the optical flow data to a previous depth map (e.g., associated with time t-1) to generate a predicted depth map associated with the same time instance as the original depth map (e.g., associated with time t). The predicted depth map may then be used to improve temporal smoothness in the original depth map and reduce noise in the original depth map. For example, the predicted depth map and the original depth map may be averaged to generate a depth map (e.g., an updated / adjusted depth map) for use by the rest of the pipeline to generate a pass-through image for the current frame.

[0008] As previously mentioned, the transparent image is generated by reprojecting the transparent image captured by the camera of the artificial reality device to the user's eye position using the depth measurement results of the scene. Since the left and right transparent images are captured from two different viewpoints, it would be preferable to use two depth sensors (one for each eye) to obtain the depth measurements of these viewpoints. However, due to system constraints and costs, it may not be feasible to equip a power-limited head-mounted device with two depth sensors. It is more practical to measure depth with a single depth sensor, and the measured results can be distorted to the viewpoint of the transparent camera.

[0009] In one embodiment, the depth measurements from the depth sensor can generate a single grid representing the entire scene. This may have computational advantages, but because the depth grid needs to be warped to the viewpoints of the two pass-through cameras, it may cause inaccuracies and blurred boundaries, especially between background objects and foreground objects. Therefore, in certain embodiments, it is preferred to use multiple depth maps and corresponding depth grids to represent different types of objects in the scene. One advantage of using multiple depth maps compared to using a single depth map is that these depth maps can more accurately represent the depth in the scene and (especially after warping) maintain clearer boundaries between the grids. In addition, certain embodiments can group depth measurements by object type so that each depth map in each depth map can be associated with a specific predetermined object type. Additional knowledge of the object type associated with each group of depth measurements also allows the use of known geometric constraints or other characteristics of known object types to refine the depth measurements. Examples of object types include, but are not limited to, static background scenes, planes, user bodies, other people in the scene, pets, other dynamic objects, etc. Another benefit of having K depth maps and grids is that each depth map and grid can be interpolated and extrapolated separately in areas where direct depth measurements are not performed. Any suitable method such as a Laplacian solver / filter may be used to fill in the missing information so that the user's entire field of view has depth information. The end result is that these K grids will have no holes (ie, each grid covers the user's entire field of view).

[0010] In a particular embodiment, first, a processing pipeline may process an image of a scene (e.g., a brightness image) using a machine learning (ML)-based semantic segmentation model to generate K segmentation masks, each of which identifies pixels in the image corresponding to a particular type of object. These K segmentation masks may then segment the depth map into K separate parts. For example, for a given segmentation mask associated with a user's body (e.g., a visible portion of a user's arm), the pixels in the mask identify which pixels in the brightness image correspond to the user's body. Those identified pixels may then identify pixels in a depth map (e.g., a depth map that has been temporally smoothed using the above-described techniques) corresponding to the user's body. The identified pixels in the depth map may then be extracted to form a segmented depth map containing only depth measurements corresponding to the user's body. By applying the K segmentation masks to the depth map, corresponding K segmented depth maps may be generated.

[0011] The segmented depth maps may have missing depth information, which may be due to the sparsity of the depth measurements and the fact that the segmented depth maps only contain a portion of the information in the source depth map. For example, a segmented depth map for the background may have missing depth information corresponding to the locations where foreground objects exist in the source depth map. In certain embodiments, the missing depth information in each segmented depth map may be filled before the segmented depth maps are used to generate a mesh. Any suitable method such as a Laplacian solver or other filter may be used to fill the missing information so that the final segmented depth map is filled and does not have missing information. The filled segmented depth map may then be converted into a corresponding mesh.

[0012] In certain embodiments, each segmented depth map or its corresponding mesh associated with a particular object type may be refined using information known about that object type. For example, a plane should be flat, a user's body should be related to physical human structure constraints, a background scene or static object should be stable in time, and so on. In certain embodiments, a known model of each object type may be used to generate a mesh that best fits the observed depth measurements. For example, one or more service components / modules may be configured to track and generate models of a particular type of object present in a scene. For example, one service component may be configured to detect and generate a three-dimensional (3D) mesh of a user's hand; another service component may be configured to detect and generate a 3D mesh of a person; and yet another service component may be configured to generate a 3D mesh of a static object and / or plane in a scene. These components may generate a 3D mesh of a particular type of object by fitting the 3D model to specific geometric constraints of the observed depth measurements. For example, a hand service component for generating a mesh of a hand may first detect a hand in a scene and identify its key points (e.g., joints or other predetermined features). These key points may then be used to configure the pose of the 3D model of the hand. The optimization algorithm can adjust the parameters of the 3D model so that the 3D model best fits the observed depth information while still remaining within the geometric constraints or other constraints associated with the hand (e.g., the amount that a finger can bend is physically limited). Once the hand service component generates a final hand model, the view synthesis pipeline can use the final hand model to refine its hand mesh or the corresponding segmented depth map of the hand. For example, the view synthesis pipeline can improve the mesh generated by the view synthesis pipeline from the segmented depth map of the user's hand by replacing any corresponding part of the mesh with the 3D model output by the hand service component. In other embodiments, the view synthesis pipeline can refine the segmented depth map by comparing the segmented depth map with the 3D model output by the hand service component. For example, depth measurements in the segmented depth map that fall outside the 3D model can be treated as noise and filtered out of the segmented depth map. The K filtered segmented depth maps can then be converted into K meshes.

[0013] The K meshes represent objects within a scene observed at a first time instance from the viewpoint of a depth sensor (e.g., a time-of-flight sensor, a stereo depth sensor, etc.). In a particular embodiment, before the K meshes are used to render an output transparent image, the K meshes are distorted to represent objects within the scene observed at a second time instance from the viewpoint of a transparent camera. As background, an artificial reality device may have additional transparent cameras that are used to capture images that will be used to generate transparent images. These additional transparent cameras may be color cameras and may be different from the depth sensor. For example, an artificial reality device may have: a left transparent camera that is used to capture images that will be "transmitted" to a display for the user's left eye; and a right transparent camera that is used to capture images that will similarly be "transmitted" to a display for the user's right eye. Since these transparent cameras are not located at the same location as the depth sensor, the depth of the scene seen by these cameras will be different from the depth measured by the depth sensor. In order to account for viewpoint differences, the computing system may distort the K meshes so that they represent the depth of the scene as seen from the perspective of the transparent camera. 3D distortion can also take into account changes in the posture of the artificial reality device. The depth sensor measurements used as the underlying data for generating the K grids may have been collected at a different time from the image captured by the transparent camera. During this time difference, the user and / or object in the scene may have moved. The 3D distortion process can distort the K grids to account for the movement based on the change in the posture of the artificial reality device and the non-intrinsic parameters and intrinsic parameters of the transparent camera. The change in the posture of the artificial reality device is tracked using traditional tracking methods (such as simultaneous localization and mapping (SLAM)). In a specific embodiment, the resulting K distorted grids for the left eye represent: at the moment when the left transparent camera captures the left transparent image, as K grids observed from the perspective of the left transparent camera. Similarly, another set of K distorted grids for the right eye represent: at the moment when the right transparent camera captures the right transparent image, as K grids observed from the perspective of the right transparent camera.

[0014] Once K warped meshes have been generated for each eye, the pass-through view synthesis pipeline can use these warped meshes to synthesize a final depth mesh corresponding to the pass-through image. For the left eye, the pass-through view synthesis pipeline can use any conventional machine learning segmentation technique to process the pass-through image captured by the pass-through camera of the left eye to generate segmentation information, which identifies pixels in the pass-through image corresponding to the aforementioned multiple object types. The pipeline can then generate an eye-specific mesh for the left eye by synthesizing multiple warped meshes for the left eye based on the segmentation information of the pass-through image of the left eye. For example, pixels associated with the user's body in the pass-through image of the left eye will correspond to a portion of the warped mesh of the left eye associated with the user's body. Similarly, pixels associated with the background in the pass-through image of the left eye will correspond to a portion of the warped mesh of the left eye associated with the background. Those portions of the warped mesh corresponding to the segmented pixels can be combined to form a complete eye-specific mesh for the left eye. The eye-specific mesh represents the depth information as observed from the viewpoint of the camera of the left eye. In other embodiments, a depth map can be used instead of a mesh to represent the eye-specific depth information. The above process for generating a complete eye-specific mesh for the left eye is similar to the process for generating a complete eye-specific mesh for the right eye, except that the transmission images used will be captured by the transmission camera for the right eye, and the warped meshes used will be those created for the right eye. Once the eye-specific meshes for the left eye and the eye-specific meshes for the right eye have been created, the eye-specific meshes for the left eye and the eye-specific meshes for the right eye will be used to reproject the left transmission image and the right transmission image, respectively, to generate the final output transmission image for the left eye and the final output transmission image for the right eye.

[0015] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. A specific embodiment may include all, some, or not include the components, elements, features, functions, operations, or steps in the embodiments disclosed herein. The embodiments according to the present invention are specifically disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentional reference to any previous claim (especially multiple dependencies) may also be claimed, so that any combination of multiple claims and multiple features thereof are disclosed and may be claimed regardless of the dependencies selected in the attached claims. The subject matter that can be claimed includes not only multiple combinations of multiple features stated in the attached claims, but also any other combination of multiple features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiment and any feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or any feature described or depicted herein or in any combination with any feature in the appended claims.

[0016] Particular embodiments use a computing system to: access a depth map and a first image of a scene, the depth map and the first image generated using one or more sensors of an artificial reality device; generate a plurality of segmentation masks based on the first image, the plurality of segmentation masks being respectively associated with a plurality of object types, wherein each segmentation mask identifies pixels in the first image corresponding to an object type associated with the segmentation mask; use the plurality of segmentation masks to segment the depth map into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the plurality of object types; and generate a plurality of meshes using the plurality of segmented depth maps, respectively. Then, for each eye of the user, the computing system: captures a second image of the scene; generates segmentation information based on the second image, the segmentation information identifying pixels in the second image corresponding to the plurality of object types; warps the plurality of meshes to generate a plurality of warped meshes for the eyes; generates eye-specific meshes for the eyes by synthesizing the plurality of warped meshes for the eyes based on the segmentation information of the second image; and renders an output image for the eyes using the second image and the eye-specific meshes.

[0017] In certain embodiments, the depth map is generated by temporally smoothing an original depth map output by one or more sensors. In certain embodiments, temporally smoothing the original depth map to generate the depth map includes: generating optical flow data to represent motion between a first image and a previous image acquired by the one or more sensors; generating a predicted depth map associated with the same time instance as the original depth map by applying the optical flow data to the previous depth map; and generating the depth map based on the original depth map and the predicted depth map. In certain embodiments, the depth map is generated by averaging the original depth map and the predicted depth map.

[0018] In a particular embodiment, the one or more sensors include a time-of-flight sensor, and the first image is an output of the time-of-flight sensor.

[0019] In certain embodiments, the one or more sensors include a pair of stereo cameras, and the first image is output by one of the pair of stereo cameras.

[0020] In a particular embodiment, the computing system fills missing depth information in at least one of the plurality of segmented depth maps using a filter before generating the plurality of meshes, wherein the plurality of meshes are generated using the plurality of depth maps after filling in the missing information.

[0021] In certain embodiments, generating the plurality of meshes further comprises using one or more 3D models of the plurality of object types.

[0022] In certain embodiments, at least one of the plurality of meshes is generated by: identifying an object type associated with the mesh, the object type being selected from a plurality of object types; generating one or more 3D models of the identified object type, the one or more 3D models of the identified object type fitting observed features of one or more objects of the identified object type present in the scene; and using the one or more 3D models to refine a mesh generated from an associated segmented depth map. In certain embodiments, the identified object type is at least one of a plane, a person, or a static object observed in the scene over a period of time.

[0023] In particular embodiments, the plurality of warped grids, eye-specific grids, and output images generated for the user's left eye are different than the plurality of warped grids, eye-specific grids, and output images generated for the user's right eye.

[0024] In a particular embodiment, the plurality of warped meshes for the eyes are generated by warping the plurality of meshes based on a position of a camera of the artificial reality device for acquiring the second image. In a particular embodiment, the plurality of warped meshes for the eyes are generated by warping the plurality of meshes based on an updated pose of the artificial reality device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with one or more color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0026] Figure 1 An example artificial reality system worn by a user is shown in accordance with certain embodiments.

[0027] Figure 2 An example of a pass-through function according to a specific embodiment is shown.

[0028] Figure 3 Shows the difference between images of the same scene captured by two different cameras.

[0029] Figure 4 An illustration of 3D-passthrough rendering based on a 3D model of an environment is provided.

[0030] Figure 5 An example problem when rendering a pass-through image based on inaccurate depth measurements is shown.

[0031] Figure 6 Another example problem when rendering a pass-through image based on inaccurate depth measurements is shown.

[0032] Figure 7 An example block diagram of an improved view synthesis architecture or view synthesis pipeline for rendering depth-accurate pass-through images is shown.

[0033] Figure 8 An example source image of a visual scene including various objects and an example depth mesh that may be created for one or more particular objects of the visual scene are shown.

[0034] Fig. 9 An example method for rendering depth-accurate pass-through images using the improved view synthesis pipeline discussed herein is shown in accordance with certain embodiments.

[0035] Fig.10 An example network environment associated with an artificial reality system is shown.

[0036] Fig.11 An example computer system is shown. DETAILED DESCRIPTION

[0037] Artificial reality is a form of reality that has been adjusted in some way before being presented to a user. Artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content (e.g., mixed reality images) may include fully generated content or generated content combined with captured content (e.g., real-world photos). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted devices (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers. At least in Figure 1 An example artificial reality system is shown in FIG. 4 and is discussed below.

[0038] “Transparent transmission” is a feature that allows users to see their physical environment while wearing an artificial reality system, such as Figure 1 Artificial reality system 100 is shown. By having a head-mounted view device (headset) of the artificial reality system display information collected by the outward-facing camera of the head-mounted view device, information about the user's physical environment is visually "transmitted" to the user. This visual information, which can be referred to as "transmitted", allows the user to see his physical environment while wearing the HMD. By having the HMD display information collected by the outward-facing camera of the head-mounted view device, information about the user's physical environment is visually "transmitted" to the user. However, simply displaying the collected image does not work as expected. Since the position of the camera is inconsistent with the position of the user's eyes, the image collected by the camera cannot accurately reflect the user's perspective. In addition, since the image has no depth, if the user moves away from the position where the image is taken, simply displaying the image will not provide the user with a proper parallax effect. Incorrect parallax plus user motion may cause motion sickness. Therefore, in order to generate correct parallax instead of just displaying the collected image, accurate depth information of the visual scene and combining the accurate depth information with the collected image are also required to reconstruct a depth-accurate transmittal image that represents the visual scene from the user's current viewpoint.

[0039] Figure 1An example of an artificial reality system 100 worn by a user 102 is shown. In certain embodiments, the artificial reality system 100 may include a head mounted device (“HMD”) 104, a controller 106, and a computing system 108. The HMD 104 may be worn over the user's eyes and provide visual content to the user 102 via an internal display (not shown). The HMD 104 may have two separate internal displays, one for each eye of the user 102. Figure 1 As shown, HMD 104 can completely cover the user's field of view. By being the only provider of visual information to user 102, HMD 104 achieves the goal of providing an immersive artificial reality experience. However, one result of this is that user 102 will not be able to see the physical environment around him / her because his / her field of view is blocked by HMD 104. Therefore, the transparent transmission function described in this article is needed to provide the user with real-time visual information about his / her physical surroundings.

[0040] The HMD 104 may have an outward-facing camera, such as Figure 1 Two forward-facing cameras 105A and 105B are shown. In a particular embodiment, camera 105A can be used to capture images that will be "transmitted" to the user's right eye, and camera 105B can be used to capture images that will be "transmitted" to the user's left eye. Although only two forward-facing cameras 105A and camera 105B are shown, HMD 104 can have any number of cameras facing in any direction (e.g., an upward-facing camera for capturing a ceiling or room light, a downward-facing camera for capturing a portion of the user's face and / or a portion of the body, a rear-facing camera for capturing a portion of an object behind the user, and / or an internal camera for capturing the user's eye gaze for eye tracking purposes). The outward-facing cameras are configured to capture the physical environment around the user, and can do so continuously to generate a series of frames (e.g., as a video). As previously explained, although the images captured by the forward-facing cameras 105A and 105B can be displayed directly to the user 102 via the HMD 104, doing so will not provide the user with an accurate view of the physical environment because the cameras 105A and 105B cannot be physically located at the same location as the user's eyes. In addition, because the images do not have depth, simply displaying the images will not provide the user with an appropriate parallax effect if the user moves away from the location where the images were captured. Incorrect parallax combined with user motion may cause motion sickness. Therefore, accurate depth information of the visual scene or the user's physical environment is also required when rendering pass-through images. The present disclosure describes an improved view synthesis pipeline for rendering depth-accurate pass-through images. At least reference will be made later to Figure 7 and Fig. 9 The improved view synthesis pipeline is discussed in detail.

[0041] A three-dimensional (3D) representation may be generated based on depth measurements of a physical object observed by a depth sensor. Depth may be measured in a variety of ways. In a particular embodiment, depth may be measured using a depth sensor (not shown), which may be a time-of-flight (ToF) sensor. A ToF sensor may determine the depth of an object within its field of view by measuring the amount of time it takes for photons to reflect back from an object in the scene. The ToF sensor may output a depth map that specifies the depth measurements within the scene and a brightness image that specifies the brightness in the scene. In a particular embodiment, depth may alternatively be calculated based on a stereo image. For example, two forward-facing stereo cameras may share overlapping fields of view and may be configured to capture images simultaneously. Thus, the same physical object may be captured simultaneously by two cameras. For example, a particular feature of an object may appear in a pixel p in an image captured by one camera. A and the same feature can appear at another pixel p in an image captured by another camera. B As long as the depth measurement system knows that the two pixels correspond to the same feature, the depth measurement system can use triangulation techniques to calculate the depth of the observed feature. For example, based on the position of the camera in 3D space, and p A Relative to the pixel position of the camera's field of view, a line can be projected from the camera through pixel p A A line can be projected from another camera through pixel p B Since it is assumed that the two pixels correspond to the same physical feature, the two lines should intersect. The two intersecting lines, as well as an imaginary line drawn between the two cameras, form a triangle that can be used to calculate the distance of the observed feature from the camera or from the point in space where the observed feature is located. The resulting depth information can be stored using a depth map, which can be represented using a matrix of pixels, where each pixel encodes the depth of the object observed through that pixel.

[0042] In certain embodiments, the pose (e.g., position and orientation) of the HMD 104 within the environment may be desired. For example, in order to render an appropriate display to the user 102 as the user moves around in the virtual environment, the system 100 will need to determine the user's position and orientation at any time. Based on the pose of the HMD, the system 100 may also determine the viewpoint of either camera 105A and 105B or the viewpoint of either eye of the user. In certain embodiments, the HMD 104 may be equipped with an inertial-measurement unit ("IMU"). The data generated by the IMU and the stereo images captured by the outward-facing cameras 105A and 105B allow the system 100 to calculate the pose of the HMD 104, for example, using simultaneous localization and mapping (SLAM) or other suitable techniques.

[0043] In certain embodiments, the artificial reality system 100 may also have one or more controllers 106 that enable the user 102 to provide input. The controller 106 may communicate with the HMD 104 or a separate computing unit 108 via a wireless connection or a wired connection. The controller 106 may have any number of buttons or other mechanical input mechanisms. In addition, the controller 106 may have an IMU so that the position of the controller 106 can be tracked. The controller 106 may also be tracked based on a predetermined pattern on the controller. For example, the controller 106 may have several infrared light emitting diodes (LEDs) or other known observable features that together form a predetermined pattern. The system 100 may be able to use a sensor or camera to capture an image of a predetermined pattern on the controller. Based on the observed orientation of these patterns, the system can calculate the position and orientation of the controller relative to the sensor or camera.

[0044] The artificial reality system 100 may also include a computer unit 108. The computer unit may be a stand-alone unit physically separated from the HMD 104, or may be integrated with the HMD 104. In an embodiment where the computer 108 is a separate unit, the computer may be communicatively coupled to the HMD 104 via a wireless link or a wired link. The computer 108 may be a high-performance device (e.g., a desktop or laptop computer), or a resource-constrained device (e.g., a mobile phone). High-performance devices may have a dedicated graphics processing unit (GPU) and a high-capacity power supply or a constant power supply. On the other hand, resource-constrained devices may not have a GPU and may have a limited battery capacity. Therefore, the algorithms that can actually be used by the artificial reality system 100 depend on the capabilities of its computer unit 108.

[0045] Figure 2 An example of a pass-through function is shown. User 102 may be wearing HMD 104 and immersed in a virtual reality environment. A physical table 250 is in the physical environment surrounding user 202. However, because user 102's field of view is blocked by HMD 104, the user cannot directly see table 250. To help the user perceive their physical environment while wearing HMD 104, the pass-through function uses, for example, outward-facing cameras 105A and 105B of HMD 104 to collect information about the physical environment. The collected information may then be reprojected to user 102 based on the user's viewpoint. In a particular embodiment where HMD 104 has a right display 260A for the user's right eye and a left display 260B for the user's left eye, computing system 108 may separately render: (1) a reprojected view 250A of the physical environment for right display 260A based on the viewpoint of the user's right eye; and (2) a reprojected view 250B of the physical environment for left display 260B based on the viewpoint of the user's left eye.

[0046] The reprojection is performed using the depth information of the scene. At a high level, the depth information encoded using a depth map or depth mesh can be viewed as the 3D geometry of the scene. These images are viewed as textures of the 3D geometry. Conceptually, an image captured by a camera with one viewpoint can be reprojected to another viewpoint by rendering the scene using the 3D geometry and the corresponding textures.

[0047] Various methods can be used to collect the depth information of the scene. For example, in the case of a stereo depth sensor that collects stereo images, a high-performance computing unit can use a GPU and optical flow technology to solve the correspondence problem, which is optimized for determining the correspondence. Then, the correspondence information between the stereo images can be used to calculate the depth using triangulation technology. Based on the calculated depth of the observed features, the computing unit can determine where those features are located in the 3D space (because the computing unit also knows where the camera is located in the 3D space). The result can be represented by a dense 3D point cloud, where each point corresponds to an observed feature. Then, a dense point cloud can be used to generate a 3D model (e.g., a 3D depth grid or depth map) of an object in the environment. When the system renders a scene for display, the system can perform visibility testing from the perspective of the user's eyes. For example, the system can project rays into the 3D space from the viewpoint corresponding to each eye of the user. In this way, the rendered scene displayed to the user will be calculated according to the perspective of the user's eyes, rather than the perspective of the camera facing outward.

[0048] Once the computing device has generated a point cloud that can be encoded as a depth map based on the depth measurement results, the depth information can be used to generate a 3D mesh representation of the observed environment. For high-performance devices, accurate models of objects in the environment can be generated (for example, each object such as a table or chair can have its own 3D model). However, for resource-constrained devices, the cost of generating such a model and / or the cost of the underlying depth measurement results used to generate the model may be too high. Therefore, in certain embodiments, the 3D mesh representation of the environment can be a rough approximation of the overall outline of the objects in the environment. In certain embodiments, a single 3D mesh can be used to approximate all observed objects. Conceptually, the 3D mesh is similar to a blanket or sheet covering the entire observable surface in the environment. In certain embodiments, the mesh can be initialized to be equidistant from the viewer or camera (for example, 1 meter, 2 meters, 2.5 meters, or 3 meters). Since the 3D mesh is equidistant from the viewer, the 3D mesh forms a hemisphere around the user. The 3D mesh can be deformed according to the depth measurement results of the observed physical objects to model the outline of the environment. In certain embodiments, a 3D mesh may be deformed based on a point cloud representation of a viewer's position and depth measurements. To determine which portion of the 3D mesh corresponds to each point in the point cloud, a computing device may cast a conceptual ray from the viewer's position to that point. Each ray will intersect a primitive (e.g., a triangle or other polygon) of the 3D mesh. Thus, intersection points on the mesh are deformed based on the depth value associated with the point through which the ray was cast. For example, if the depth measurement for the point is 2.2 meters from the viewer, an initial 2 meter depth value associated with an intersection point on the mesh may become 2.2 meters. Once this process is completed for each point in the point cloud, the resulting deformed mesh will represent the outline of the physical environment as viewed by the viewer.

[0049] Using a single mesh to represent the entire scene may be computationally efficient, but the single mesh representation sacrifices depth accuracy, especially at the boundaries between distant foreground and background objects. Figure 3The difference between an image 300 and an image 310 of the same scene captured by two different cameras is shown. As shown, even though the same scene is captured by a left pass-through camera and a right pass-through camera, the image 300 and the image 310 are obviously different due to the difference in the viewpoints of the cameras. The difference is even more obvious when a foreground object occludes a background object. As can be seen, in the left image 300, the center of the laptop computer 302 is occluded by the user's hand 304, while in the right image 310, only the left portion of the laptop computer 302 is occluded by the hand 304. In addition, in the left image 300, the three fingers 303a, 303b, 303c of the user's hand 304 are clearly visible, while in the right image 310, only the thumb 303c and the index finger 303b are clearly visible. This example shows that the difference between what is observed by the left pass-through camera and what is observed by the right pass-through camera may be obvious. Therefore, the grid used to reproject the images captured by the two cameras should be generated for their respective perspectives. This is the rationale for generating a set of K grids for the left eye and another set of K grids for the right eye.

[0050] Figure 4An illustration of 3D-transparent rendering based on a 3D model of an environment is provided. In a particular embodiment, the rendering system may determine the current viewing position of the user 102 relative to the environment. In a particular embodiment, the system may use SLAM or other suitable techniques to calculate the posture of the HMD 104. Then, based on the known mechanical structure of the HMD 104, the system may use an offset from the posture of the HMD 104 to estimate the viewpoints of the user's eyes 400A and 400B. Then, the system may render a transparent image for each of the user's eyes 400A and 400B. For example, in order to render a transparent image for the user's right eye 400A, the system may project a ray 420 through each pixel of the virtual screen space 410A from the estimated viewpoint of the right eye 400A to see which part of the 3D model the ray 420 will intersect. This ray casting process may be referred to as a visibility test because the purpose is to determine what can be seen from the selected viewpoint 400A. In the particular example shown, ray 420 projected through a particular pixel 422 intersects a particular point 421 on the 3D model 480. This indicates that the intersecting point 421 will be displayed by the pixel 422. Once the intersecting point 421 is found, the rendering system can sample the corresponding point in the texture image that maps to the intersecting point 421. In a particular embodiment, the images captured by the cameras 105A and 105B of the HMD 104 can be used to generate the texture of the 3D model 480. Doing so allows the rendered image to look more like an actual physical object. In a similar manner, the rendering system can render a pass-through image for the user's left eye 400B. In the example shown, ray 430 can be projected from the left eye viewpoint 400B through the pixel 432 of the left screen space 410B. The ray 430 intersects the 3D model 490 at the position 431. Then, the rendering system can sample the texture image at the texture position corresponding to the position 431 on the model 490 and calculate the appropriate color to be displayed by the pixel 432. Since the pass-through images are re-rendered from the user's viewpoint 400A and 400B, the images will look natural and will provide the appropriate parallax effect.

[0051] As mentioned above, depth information is an important part of the pass-through rendering process. Depth information can be measured based on the correspondence between stereo images. Alternatively, depth information can be obtained by one or more dedicated depth sensors such as time-of-flight sensors, which can be integrated into artificial reality systems such as artificial reality system 100. However, depth sensors have measurement errors, low resolution, and do not work on certain object types. In addition, the depth calculated based on stereo images is prone to errors and mistakes. Therefore, it is generally difficult to measure the scene geometry of artificial realities such as mixed reality and is not very accurate. This is a key issue for artificial reality devices (e.g., mixed reality devices), and overcoming this key issue largely determines the commercial success of AR / VR devices.

[0052] As mentioned previously, one way to represent depth information within a scene is by using a single overlay grid. Although using a single grid may have computational advantages, it is a crude approximation of the depth in the scene and contains inaccuracies, especially at the boundaries between foreground and background objects. Figure 5 and Figure 6 shows an example problem that arises when rendering a pass-through image based on such inaccurate depth measurements. In particular, Figure 5 An example problem of using a single mesh to represent depth in a scene is shown. For example, the left image 500 shows that the background 502 around the user's hand 504 is deformed. This rendering artifact is due to an inaccurate depth representation between the portion of the mesh corresponding to the hand 504 and the portion corresponding to the background 502. These two portions of the mesh are connected in a single mesh depth representation. The right image 510 shows a mismatch between a pass-through image 512 of the user's hand and the position 514 of the user's hand determined by the hand tracking algorithm. The observed mismatch can also be attributed to inaccuracies in the mesh.

[0053] Figure 6 Another example problem when rendering a pass-through image based on inaccurate depth measurements is shown. Close objects present a challenge because they move faster, and inaccuracies in the depth measurements will result in an inaccurate mesh representation of the entire scene. For example, Figure 6 The left image 600 in shows a gap between the user's arm 602 and the background 604. This is an undesirable artifact that occurs because the overlay grid representing the scene has inaccuracies around the edges of the user's arm 602. As shown in the right image 610, one solution is to fill the gap area with a neutral color. Testing has shown that good results can be achieved if these transition areas are filled with a distorted texture. This simplifies rendering and avoids sudden changes in brightness that can be overly irritating to the eye.

[0054] Therefore, when rendering a pass-through image for a user immersed in an artificial reality via the user's artificial reality system, accurate and / or reliable depth measurements need to be generated. Figure 7 An example block diagram 700 of an improved view synthesis architecture / pipeline for rendering depth-accurate pass-through images is shown. It should be noted that the operations associated with each block 702 to block 752 of the improved view synthesis pipeline can be performed by the computer unit 108 or the computer system 1100 of the artificial reality system 100. Although the present disclosure describes and illustrates Figure 7 The specific blocks of the improved view synthesis pipeline of occur in a specific order, but the present disclosure contemplates Figure 7 Any suitable blocks of the improved view synthesis pipeline may occur in any suitable order.

[0055] In certain embodiments, view synthesis is part of a mixed reality (MR) pipeline. Old pipelines need to be updated to address the pass-through rendering problem, e.g., as in Figure 5 and Figure 6 An improved pipeline may be particularly needed for rendering certain objects (e.g., hands), which have been identified as key to providing an accurate MR experience.

[0056] As depicted, raw depth measurements 702 and images 704 may be obtained as inputs to the pipeline. For example, one or more sensors associated with the artificial reality system 100 may generate depth measurements 702 and images 704. The one or more sensors may be depth sensors capable of generating depth measurements and brightness images, such as time-of-flight sensors. For example, for each pixel, the depth sensor may generate a depth value and an amplitude signal indicating a measure of the brightness of the pixel. In some embodiments, the depth measurements 702 and images 704 (e.g., brightness images) may be generated by a time-of-flight (ToF) sensor. For example, a ToF sensor may emit a signal and then determine how long it takes to receive a reflection from an object. Based on this, the ToF sensor may generate a depth measurement 702 and a corresponding brightness image 704, both of which are associated with the current time. In some other embodiments, a stereoscopic image may be acquired by a stereo camera (e.g., cameras 105A and 105B) of the artificial reality system 100, and then the depth measurement 702 may be acquired by comparing the stereo images and calculating the depth using triangulation techniques. In the case where a stereo camera is used, one of the stereo images associated with one eye (eg, the left eye or the right eye) may simply be used as the luminance image 704 .

[0057] The raw depth measurement 702 may be represented as a depth map, which may be implemented as a two-dimensional matrix of pixels, where each pixel holds a depth measurement. As discussed elsewhere herein, the depth measurement 702 may contain noise and may be inaccurate, and the depth measurement 702 from frame to frame may be collected independently and lack temporal consistency. In other words, two consecutive depth maps (e.g., depth maps corresponding to image frame N-1 and frame N, respectively) may be temporally inconsistent with each other and / or may be temporally unstable. Rendering a pass-through image based on such a temporally unstable depth map may result in a lack of temporal smoothness between frames.

[0058] In certain embodiments, to align the depth maps in time, a series of images including image 704 is used to calculate a first optical flow (e.g., as indicated by block 706). Optical flow is a technique for representing motion (e.g., object motion) between a series of images or a sequence of images. In certain embodiments, calculating the optical flow may include determining a correspondence between a first image (e.g., image frame N-1) and a second image (e.g., image frame N) in the series of images, and calculating a motion vector based on the correspondence. The motion vector may be an optical flow vector as discussed herein. Once the optical flow is calculated and the optical flow vector (or motion vector) is obtained, depth temporal noise reduction (depth TNR) 710 is performed using the optical flow to improve the temporal consistency of each depth map obtained based on the depth measurement result 702. In certain embodiments, optical flow data specifying the pixel-level motion of an object in a scene from frame N-1 to frame N may be applied to a previous depth map associated with frame N-1 to generate a predicted depth map for frame N. The predicted depth map for frame N may then be used to update and denoise a currently acquired depth map obtained based on the depth measurements 702 for frame N. In one example, the predicted depth map and the current depth map (e.g., the depth map based on the depth measurements 702) may be combined (e.g., averaged) to generate an updated or adjusted depth map 712 for use by the rest of the pipeline. The depth map 712 produced by the TNR block 710 is more stable in time and has reduced temporal inconsistencies.

[0059] As previously mentioned, another purpose of the present pipeline 700 is to use different grids to represent different types of objects in the scene. Using multiple grids helps improve depth accuracy and edge accuracy, and grouping / classifying depth measurements based on objects allows us to further refine the depth measurements using known geometric constraints about these object types. To this end, a first segmentation 714 can be performed on an image 704 (e.g., a luminance image) associated with a current frame N to decompose the visual scene represented by the image 704 into a plurality of layers 716 (e.g., a plurality of segmentation masks) corresponding to different predetermined object types. Each segmentation mask identifies a plurality of pixels within the image 704 corresponding to one or more objects of a predetermined object type or object class in the visual scene. For example, if the visual scene includes one or more body parts of the user (e.g., hands, legs), background static objects (e.g., tables, chairs, murals, paintings, etc.), and other people in the scene, then the first segmentation 714 can generate 3 layers / masks, including a first layer / mask corresponding to the user's own body parts (e.g., hands, legs), a second layer / mask corresponding to the background static objects, and a third layer / mask corresponding to other people in the scene.

[0060] In certain embodiments, the segmentation 714 discussed herein may be performed using machine learning (ML) techniques. In other words, the segmentation may be ML-based segmentation or may use an ML model to perform the segmentation discussed herein. For example, an ML model may be trained to recognize different categories / types of objects in an image and generate image layers / masks corresponding to these different categories / types of objects. A computing system (e.g., computer 108 or computer system 1100) may use such an ML model to perform a first segmentation 714 to generate a plurality of segmentation layers / masks 716 corresponding to a plurality of predetermined object types in a visual scene.

[0061] In certain embodiments, the plurality of segmentation masks 716 (e.g., segmentation layers) may be used to create K depth grids 718 from the temporally aligned depth maps 712 (or, alternatively, the raw depth maps output by the depth sensor if TNR is not performed). Each depth grid 718 may correspond to one or more objects of a particular type in the visual scene. In certain embodiments, to create a particular grid 718 corresponding to a particular type of object (e.g., a user's body), a segmentation layer / mask 716 associated with that object type may be used to extract depth measurements / points from the depth map 712. The extracted portion of the depth map represents a depth measurement that may correspond to the object type of interest. A 3D grid is then created based on the extracted depth points or point clouds corresponding to one or more objects of the desired type (e.g., a hand). This process may be repeated for each segmentation mask 716 to generate multiple grids corresponding to different object types. As an example and not limitation, a first mesh may be made based only on points corresponding to certain body parts of the user (e.g., hands), a second mesh may be made based only on points corresponding to planes, and a third mesh may be made based on all other depth points. In other words, in this example, three meshes may be created, where mesh #1 includes only the user's arms and other observable body parts, mesh #2 includes visible planes in the scene, and mesh #3 includes all other objects in the scene. As another example, if there are K segmentation masks 716 generated based on the first segmentation 714 as described above, there may be K depth meshes generated corresponding to the K segmentation masks, where a first depth mesh may correspond to one or more body parts (e.g., hands, legs) of a user wearing an artificial reality device, a second depth mesh may correspond to a plane in the visual scene, and a third depth mesh may correspond to a background static object (e.g., a table, a chair, a mural, a painting, etc.).

[0062] Figure 8 An example source image 800 ( Figure 7804 ). A segmentation mask may be generated for the user's body and another segmentation mask may be generated for the background. A source image 800 may be acquired using an outward-facing camera (e.g., camera 105A or 105B) of the artificial reality system 100. As depicted, image 800 displays user body parts such as hand 801 and leg 802 and other objects in the scene such as wall 803, table 804, and whiteboard 805. Pixels corresponding to hand 801 and leg 802 may be identified by a first segmentation mask, and pixels corresponding to the background including wall 802, table 804, and whiteboard 805 may be identified by a second segmentation mask. Image 810 conceptually illustrates two segmentation masks applied to a depth map. A first segmentation mask associated with the user's body may be overlaid on the depth map to identify depth measurements / points corresponding to the user's hand 801 and leg 802. Similarly, a second segmentation mask may identify other depth measurements / points corresponding to the background environment.

[0063] In certain embodiments, the depth measurements extracted from the depth map 712 using the K segmentation masks 716 may be refined / filtered before generating the K grids 718. Figure 7 , represented by a refinement or filtering process at block 720. In some embodiments, depth refinement 720 may include filling missing gaps or depth information in a sparse depth map. For example, a segmented depth map associated with a particular object type may include missing depth information because some portions of the field of view of the depth map are occluded or occupied by other types of objects. For example, one or more foreground objects may occlude one or more background objects and / or deform one or more background objects. As a result, the segmented depth map of the background will have missing depth information that was previously occupied by the depth measurements of the foreground objects. In this case, the computing system discussed herein may use any suitable filtering technique to fill these missing gaps or information fragments to increase the depth map density. In a particular embodiment, a Laplacian filter may be used to fill these missing gaps or information fragments. The segmented depth map with increased density (i.e., all pixels within the field of view of the map have depth values) may then be used to generate a corresponding grid.

[0064] As previously described, one advantage of segmenting depth measurements based on known object types is that the known geometric constraints of those object types can be used to refine the depth measurements. In one embodiment, depth refinement 720 may include refining the depth measurements using one or more components 722a, 722b, ..., 722n (individually and / or collectively 722) that provide 3D modeling constraints or geometric constraints for depth filtering. These components 722 may be modules or services that detect or track certain object types of interest and generate 3D meshes for them. By way of example and not limitation, the one or more components 722 may include: (1) a first component 722a, which may be a plane component that includes information about two-dimensional (2D) planes in a visual scene; (2) a second component 722b, which may be a human component that includes information about different humans in the scene; and (3) a third component 722n, which may be a three-dimensional reconstruction (3DR) component that includes information about observed depth measurements or geometries of static objects in the scene accumulated over a period of time. In particular embodiments, the object types supported by component 722 may correspond to the object types associated with segmentation mask 716 .

[0065] Each of the components 722 may include a prior (e.g., a 3D model or law) that constrains the geometry of the associated object type. For example, if component 722a is associated with a plane, the geometric constraint will be a 2D plane detected by component 722a. As an example, the computing system may use a plane component (e.g., component 722a) to refine the depth measurements within a segmented depth map associated with the plane (i.e., a reference segmented depth map may be generated by extracting the depth measurements from the depth map 712 using a segmentation mask 716 associated with the plane). In a particular embodiment, an optimization algorithm may be used to find an arrangement of planes that best fit the observed depth measurements associated with the plane in the segmented depth map. Depth measurements that do not match the fitted planes in the segmented depth map may be filtered out. In another embodiment, component 722a may have independently detected planes in the scene and corresponding meshes are created to represent these planes. If so, refinement 720 may occur at the mesh level rather than at the depth map level. For example, after a mesh of a plane has been created using the segmented depth map of the plane, portions of the mesh of the plane may be replaced by a 3D model of the plane independently generated by component 722a.

[0066] As another example, the system can use a people component (e.g., component 722b) to refine depth measurements corresponding to people in a visual scene. In certain embodiments, component 722b may include a human body model that constrains the possible geometry of a human body. An optimization algorithm can be used to find the poses of one or more human bodies that best fit the observed depth measurements in a segmented depth map associated with a person (i.e., a reference segmented depth map generated using a segmentation mask associated with a person). The 3D fitted model of the person can be used to filter out depth measurements in the segmented depth map that are outliers relative to the 3D fitted model of the person. After filtering, the resulting segmented depth map can be used to generate a single mesh to represent any number of people in the scene. In other embodiments, depth refinement 720 may include replacing portions of a depth mesh of a person generated from a corresponding segmented depth map using a 3D mesh of a person independently generated by component 722b.

[0067] Once one or more depth refinements 720 are applied, the resulting refined / filtered K grids 724 can be saved and kept separate until they are combined at rendering time. At rendering time, a left pass-through image 742a and a right pass-through image 742b can be acquired from the sensors of the artificial reality system. For example, the left pass-through image and the right pass-through image can be acquired by external cameras 105A and 105B of the artificial reality system. These acquired images 742a and images 742b can represent the visual scene or the user's physical environment at the current time instance (or the second time instance). However, these acquired images 742a and images 742b do not include depth. It may be necessary to acquire accurate depth information for these acquired images 742a and images 742b in order to render depth-accurate pass-through images for both eyes from the user's perspective.

[0068] In certain embodiments, the images 742a and 742b captured at the current time instance (or second time instance) associated with the pass-through image generation may be different from the image 704 captured by one or more sensors at the previous time instance (or first time instance) associated with the depth generation. For example, there may be some time delay between the image 704 captured at the previous time instance and the image captured at the current time instance (e.g., the left pass-through image 742a or the right pass-through image 742b), and therefore, objects in the visual scene at the current time instance (e.g., the user's hand, other people, etc.) may be in different positions relative to the objects in the image at the previous time instance. Due to the time delay and / or different positioning of the objects in the visual scene, the refined K grids 724 may need to be warped and then combined to render the output image from the user's current eye perspective. The process for rendering a depth-accurate pass-through image for each eye is discussed in detail below.

[0069] In order to render a depth-accurate pass-through image for each eye, a second segmentation 744a and 744b (e.g., ML segmentation) may be performed on each of the left-eye pass-through image 742a and the right-eye pass-through image 742b. For example, a second segmentation 744a may be performed on the left pass-through image 742a at the current time instance or the second time instance to decompose the left pass-through image 742a into a plurality of segmentation masks / layers 746a. Each of these masks 746a may identify pixels corresponding to a particular object type of interest (similar to the type identified by the first segmentation process 714) in the left pass-through image 742a. ​​Similarly, a second segmentation 744b may be performed on the right pass-through image 742b at the current time instance or the second time instance to decompose the right pass-through image 742b into a plurality of masks / layers 746b. Each of the plurality of masks 746b may correspond to one or more objects of a particular type in the visual scene at the second time instance. As previously described, the position of an object in the scene at the second time instance may be different from the position of an object at the previous time instance or the first time instance. This may be due to the time delay between the acquisition of segmented image 742a and segmented image 742b and brightness image 704. In addition, the scene at the second time instance may change due to a change in the user's eye position or head position (for example, the user is now looking at a slightly different angle than before). In addition, the sensor used to acquire the through-images 742a and 742b may be different from the sensor used to acquire the brightness image 704. Therefore, the second segmentation 744a and 744b process requires processing the through-images 742a and 742b to identify the position of the object of interest within those images 742a and 742b.

[0070] The transparent images 742a and 742b will be reprojected to the user's eye position. In order to do this, depth information corresponding to the geometry of the scene is also required. In a specific embodiment, it may be necessary to distort the refined K grids 724 for each eye. Specifically, in block 730a, the K grids 724 generated based on the depth information obtained from the perspective of the depth sensor are distorted so that they represent the depth information observed from the perspective of the left transparent camera at the second time instance. The distortion process will take into account the changes in the head posture of the artificial reality device and the non-intrinsic parameters and intrinsic parameters of the left transparent camera. The resulting distorted grid 748a will be left eye specific. Similarly, in block 730b, K grids 724 are distorted to represent the depth information observed from the perspective of the right transparent camera at the second time instance. The resulting distorted grid 748b will be right eye specific. In doing so, the distorted grids 748a and 748b will provide appropriate depth information aligned with the transparent images 742a and 742b, respectively.

[0071] Once the meshes are warped for each eye in blocks 730a and 730b, an output image may be rendered for each eye. For example, in the rendering block 750a for the left eye, the computing system may synthesize K warped meshes 748a for the left eye to generate a single mesh ready for rendering. Portions of the K warped meshes 748a may be combined according to the segmentation mask 746a associated with the left eye to form a single final mesh. Similarly, in the rendering block 750b for the right eye, portions of the K warped meshes 748b may be combined according to the segmentation mask 746b associated with the right eye to form another final mesh for the right eye. In certain embodiments, the rendering in blocks 750a and 750b may include mapping, associating, correlating, and / or matching each warped mesh 748a and 748b with its corresponding segmentation information 746a and 746b. As an example and not limitation, the warped mesh 748a corresponding to a body part of the user is mapped to a segmentation layer / mask 746a associated with the body part of the user. As another example, the warped mesh 748b corresponding to a background static object is mapped to a segmentation layer / mask 746b associated with the background static object. The computing system can generate a synthetic final mesh for rendering an output image for the left eye and a synthetic final mesh for rendering an output image for the right eye based on this mapping or correspondence between the warped meshes 730a and 730b and their corresponding segmentation layers / masks 746a and 746b. For example, the K segmentation masks 746a for the left eye will be used to extract and combine the parts of the corresponding warped mesh 748a to form a final eye-specific mesh for the left eye. The final eye-specific mesh will be used as the geometry information associated with the left pass-through image 742a. ​​Then, the geometry information and the pass-through image 742a can be used by the rendering engine to render the final output image 752a for the left eye. The viewpoint used to render the output image 752a can be the predicted viewpoint of the user's left eye. In a similar manner, the final eye-specific grid for the right eye and the right see-through image 742b can be used to render the output image 752b for the right eye. These output images 752a and 752b can then be output on the left eye display and right eye display of the artificial reality device, respectively, to give the user a "see-through" view of the physical environment.

[0072] In some embodiments, during rendering in blocks 750a and 750b, only meshes corresponding to objects currently visible in the left pass-through image 742a and the right pass-through image 742b may be used. For example, if three meshes are created, where mesh #1 includes only the user's exposed hands, mesh #2 includes all other objects in the scene except the user's exposed hands, and mesh #3 includes all objects in the scene, then rendering may be performed in the following rendering modes. In the example first rendering mode, if the user's hands are not visible to both the left and right eyes, only mesh #3 is used. In the example second rendering mode, if the user's hands are visible and there are no objects or clothes on the hands, separate meshes are made for the left and right eyes. To this end, mesh #1 and mesh #2 may be combined separately for each eye using separate segmentation masks for each eye. In the example third rendering mode, if the user's hands are visible but with objects or clothes, a combination of the above first and second rendering modes may be used. That is, areas where there are objects or clothing in the hand are marked with a fallback behavior mask and rendered therein as in the first rendering mode. In those parts of the hand away from clothing and objects, rendering occurs as in the second rendering mode. The fallback behavior mask can have blurred edges to smooth the transition between modes. The fallback behavior mask can be stabilized similar to deep TNR, that is: a) the mask can be represented as an image in the range 0 to 1; b) motion vectors can be used to overlay the fallback behavior mask of this frame and the previous frame; and c) the previous fallback behavior mask can be averaged with the current fallback behavior mask.

[0073] In certain embodiments, the output images 752a and 752b generated from the rendering blocks 750a and 750b, respectively, are depth accurate pass-through images. Each of the output images 752a and 752b can be rendered for display on a display component of an artificial reality device (e.g., HMD 104 of artificial reality system 100).

[0074] Fig. 9 An example method 900 for rendering a depth-accurate pass-through image using the improved view synthesis pipeline discussed herein is shown according to certain embodiments. It should be noted that steps 910 to 990 in method 900 correspond to generating a depth-accurate pass-through image for one eye (e.g., the left eye) of a user wearing an artificial reality device. The same or similar steps may be repeated to generate a depth-accurate pass-through image for the other eye (e.g., the right eye) of the user.

[0075] Method 900 may begin at step 910, in which a computing system (e.g., computer 108 or computer system 1100) associated with an artificial reality system (e.g., artificial reality system 100) may access a depth map and a first image of a scene, the depth map and the first image being generated using one or more sensors of an artificial reality device. The artificial reality device may be a mixed reality head mounted viewer. In some embodiments, the one or more sensors discussed herein are time-of-flight sensors, and the first image is the output of the time-of-flight sensor. In some embodiments, the one or more sensors are depth sensors capable of generating depth measurements and brightness images. In other embodiments, the one or more sensors may be a pair of stereo cameras (e.g., cameras 105A and 105B), and the first image may be output by one of the pair of stereo cameras. When the sensor is a stereo camera, the depth measurement may be obtained by comparing the stereo images.

[0076] In certain embodiments, an adjusted depth map (e.g., updated / adjusted depth map 712) is generated by temporally smoothing an original depth map output by the one or more sensors, such as the depth map accessed in step 910 or a depth map acquired based on the depth measurement result 702. Temporally smoothing the original depth map to generate the adjusted depth map may include: generating optical flow data to represent motion between a first image and a previous image acquired by the one or more sensors; generating a predicted depth map associated with the same time instance as the original depth map by applying the optical flow data to the previous depth map; and generating the adjusted depth map based on the original depth map and the predicted depth map. In certain embodiments, the adjusted depth map (e.g., adjusted depth map 712) is generated by averaging the original depth map and the predicted depth map.

[0077] At step 920, the computing system can generate a plurality of segmentation masks (e.g., the segmentation mask 716 at the first time instance) based on the first image, the plurality of segmentation masks being respectively associated with a plurality of object types. Each segmentation mask or layer identifies pixels in the first image corresponding to an object type associated with the segmentation mask. At step 930, the computing system can use the plurality of segmentation masks to segment a depth map (e.g., the adjusted depth map 712) into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the plurality of object types.

[0078] At step 940, the computing system may use the plurality of segmented depth maps to generate a plurality of grids (e.g., K grids 718), respectively. In certain embodiments, the computing system may also be configured to fill missing depth information in at least one of the plurality of segmented depth maps using a filter (e.g., a Laplacian filter) before generating the plurality of grids. After filling the missing information, the plurality of depth maps are used to generate the plurality of grids.

[0079] In some embodiments, as shown in reference Figure 7 As discussed, the plurality of meshes in step 940 are generated using one or more 3D models of the plurality of object types. For example, at least one of the plurality of meshes may be generated by: (1) identifying an object type associated with the mesh, the object type selected from a plurality of object types; (2) generating one or more 3D models of the identified object type, the one or more 3D models of the identified object type fitting observed features of one or more objects of the identified object type present in the scene; and (3) using the one or more 3D models to refine a mesh generated from an associated segmented depth map. The identified object type is at least one of a plane, a person, or a static object observed in the scene over a period of time.

[0080] In steps 950 to 990, an output image (e.g., a depth-accurate transparent image) may be rendered for each eye of a user wearing an artificial reality device. For example, in step 950, for each eye, the computing system may capture a second image of the scene. For example, one of the external cameras 105A and 105B may capture a second image (e.g., a left transparent image 742a or a right transparent image 742b) that represents a scene around the user at a current time instance or at a second time instance. The image representing the visual scene at the second time instance may be different from the image captured by one or more sensors at a previous time instance or at the first time instance. For example, there may be some time delay between the image captured at the first time instance and the image captured at the second time instance, and therefore, objects in the scene at the second time instance (e.g., the user's hand, other people) may be in a different position relative to the objects in the image at the first time instance.

[0081] At step 960, the computing system may generate segmentation information based on the second image, the segmentation information identifying pixels in the second image corresponding to the plurality of object types. For example, the computing system may perform a second segmentation on the second image (e.g., a luminance image at a second time instance) to decompose the second image into a plurality of segmentation masks / layers (e.g., segmentation masks / layers 746a or 746b) at the second time instance. In certain embodiments, as discussed elsewhere herein, the second segmentation performed at step 960 is an ML-based segmentation.

[0082] In step 970, as shown in reference Figure 7 As discussed in block 730a or block 730b in , the computing system can warp the multiple grids (e.g., K grids 724) to generate multiple warped grids for the eyes (e.g., warped grids 748a or 748b). In some embodiments, the multiple warped grids for the eyes are generated by warping the multiple grids based on the position of the camera of the artificial reality device used to capture the second image. In other embodiments, the multiple warped grids for the eyes are generated by warping the multiple grids based on the updated posture of the artificial reality device.

[0083] In step 980, the computing system may generate an eye-specific mesh for an eye by synthesizing the plurality of warped meshes for an eye (e.g., a left eye or a right eye) based on the segmentation information of the second image. In some embodiments, generating an eye-specific mesh may include mapping, associating, correlating, and / or matching each warped mesh corresponding to one or more objects in the scene with a segmentation mask / layer corresponding to the one or more objects at a second time instance. More specifically, once a mesh has been warped, the mesh is mapped to a segmentation mask / layer corresponding to the mesh (e.g., obtained after segmenting the second image at the second time instance). By way of example and not limitation, meshes corresponding to body parts of a user are mapped to a segmentation mask / layer that includes those body parts. As another example, meshes corresponding to background static objects are mapped to a segmentation mask / layer that includes those background static objects. Once each warped depth mesh is mapped to its corresponding segmentation mask / layer, the mapped warped meshes are combined together to generate an eye-specific mesh.

[0084] At step 990, the computing system may use the second image and the eye-specific grid to render an output image for the eye (e.g., output image 752a for the left eye or output image 752b for the right eye). Figure 4In certain embodiments, the plurality of warped meshes, eye-specific meshes, and output images generated for the user's left eye are different from the plurality of warped meshes, eye-specific meshes, and output images generated for the user's right eye.

[0085] Once the output image is rendered, the computing system can present the output image for display on a display component of an artificial reality device (e.g., HMD 104 of artificial reality system 100). In a particular embodiment, the output image is a depth-accurate see-through image generated for one eye of a user wearing the artificial reality device. For example, the output image is output image 752a for the left eye or output image 752b for the right eye. Similarly, method 900 can use steps 950 to 990 discussed herein to render an output image for the other eye. Once the images for both eyes are rendered and displayed, method 900 ends.

[0086] Where appropriate, specific examples may be repeated Fig. 9 One or more steps of the method. Fig. 9 The various specific steps of the method are described and shown as being performed in a particular order, but the present disclosure contemplates Fig. 9 In addition, where appropriate, although the present disclosure describes and illustrates a method for rendering a depth-accurate pass-through image using an improved view synthesis pipeline, including Fig. 9 However, the present disclosure contemplates an example method for rendering a depth-accurate pass-through image using an improved view synthesis pipeline, including any suitable steps (which may include Fig. 9 In addition, although the present disclosure describes and illustrates implementation Fig. 9 The present disclosure contemplates specific components, devices or systems for each specific step of the method. Fig. 9 Any suitable combination of any suitable components, devices, or systems for any suitable steps of the method.

[0087] Fig.10 An example network environment 1000 associated with an artificial reality system is shown. Fig.10 Although illustrated with a virtual reality system, the example network environment 1000 may include one or more other artificial reality systems, such as a mixed reality system, an augmented reality system, etc. The network environment 1000 includes a user 1001 interacting with a client system 1030, a social network system 1060, and a third-party system 1070 connected to each other via a network 1010. Fig.10A particular arrangement of user 1001, client system 1030, social networking system 1060, third-party system 1070, and network 1010 is shown, but the present disclosure contemplates any suitable arrangement of user 1001, client system 1030, social networking system 1060, third-party system 1070, and network 1010. By way of example and not limitation, two or more of user 1001, client system 1030, social networking system 1060, and third-party system 1070 may be directly connected to each other, bypassing network 1010. As another example, two or more of client system 1030, social networking system 1060, and third-party system 1070 may be physically or logically co-located, in whole or in part, with each other. Furthermore, although Fig.10 A particular number of users 1001, client systems 1030, social networking systems 1060, third-party systems 1070, and networks 1010 are shown, but the present disclosure contemplates any suitable number of client systems 1030, social networking systems 1060, third-party systems 1070, and networks 1010. By way of example and not limitation, network environment 1000 may include a plurality of users 1001, a plurality of client systems 1030, a plurality of social networking systems 1060, a plurality of third-party systems 1070, and a plurality of networks 1010.

[0088] The present disclosure contemplates any suitable network 1010. By way of example and not limitation, one or more portions of network 1010 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 1010 may include one or more networks 1010.

[0089] Links 1050 may connect client systems 1030, social networking systems 1060, and third-party systems 1070 to communication network 1010 or to each other. The present disclosure contemplates any suitable links 1050. In particular embodiments, one or more links 1050 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In certain embodiments, one or more links 1050 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 1050, or a combination of two or more such links 1050. The multiple links 1050 are not necessarily all the same throughout the network environment 1000. In one or more aspects, the one or more first links 1050 can be different from the one or more second links 1050.

[0090] In certain embodiments, client system 1030 may be an electronic device that includes hardware, software, or embedded logic components or a combination of two or more such components and is capable of performing appropriate functions implemented or supported by client system 1030. By way of example and not limitation, client system 1030 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality head-mounted device and controller or a mixed reality head-mounted device and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 1030. Client system 1030 may enable network users at client system 1030 to access network 1010. Client system 1030 may enable its users to communicate with other users at other client systems 1030. Client system 1030 may generate a virtual reality environment or a mixed reality environment for users to interact with content.

[0091] In certain embodiments, the client system 1030 may include a virtual reality (or augmented reality or mixed reality) head mounted device 1032, and one or more virtual reality input devices 1034 (e.g., virtual reality controllers). A user at the client system 1030 may wear the virtual reality head mounted view device 1032 and use the one or more virtual reality input devices to interact with a virtual reality environment 1036 generated by the virtual reality head mounted view device 1032. Although not shown, the client system 1030 may also include a separate processing computer, and / or any other components of the virtual reality system. The virtual reality head mounted view device 1032 may generate a virtual reality environment 1036, which may include system content 1038 (including but not limited to an operating system), such as software or firmware updates, and the virtual reality environment may also include third-party content 1040, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted view device 1032 may include one or more sensors 1042 (e.g., accelerometers, gyroscopes, magnetometers) to generate sensor data that tracks the position of the head mounted view device 1032. The head-mounted view device 1032 may also include an eye tracker for tracking the position of the user's eyes or the direction in which they are viewed. The client system 1030 may use data from the one or more sensors 1042 to determine the speed, orientation, and gravity relative to the head-mounted view device. One or more virtual reality input devices 1034 may include one or more sensors 1044 (e.g., accelerometers, gyroscopes, magnetometers, and touch sensors) to generate sensor data that tracks the position of the input device 1034 and the position of the user's fingers. The client system 1030 may use outside-in tracking, in which a tracking camera (not shown) is placed outside the virtual reality head-mounted view device 1032 and within the line of sight of the virtual reality head-mounted view device 1032. In outside-in tracking, the tracking camera may track the position of the virtual reality head-mounted view device 1032 (e.g., by tracking one or more infrared LED markers on the virtual reality head-mounted view device 1032). Alternatively or additionally, the client system 1030 can use inside-out tracking, in which a tracking camera (not shown) can be placed on the virtual reality headset 1032 or inside the virtual reality headset 1032 itself. In inside-out tracking, the tracking camera can capture images of its surroundings in the real world, and can use the changing perspective of the real world to determine its own position in space.

[0092] In certain embodiments, the client system 1030 (e.g., HMD) may include a transparent transmission engine 1046 for providing the transparent transmission functions described herein, and the client system may have one or more add-ons, plug-ins, or other extensions. A user at the client system 1030 may connect to a specific server (e.g., server 1062 or a server associated with a third-party system 1070). The server may accept requests and communicate with the client system 1030.

[0093] The third-party content 1040 may include a web browser and may have one or more add-ons, plug-ins or other extensions. A user at the client system 1030 may enter a Uniform Resource Locator (URL) or other address to direct the web browser to a specific server (e.g., server 1062 or a server associated with the third-party system 1070), and the web browser may generate a Hypertext Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may accept the HTTP request and transmit one or more Hypertext Markup Language (HTML) files to the client system 1030 in response to the HTTP request. The client system 1030 may render a web page based on an HTML file from a server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web page interface may be rendered according to an HTML file, an Extensible Hyper Text Markup Language (XHTML) file or an Extensible Markup Language (XML) file, according to specific needs. Such an interface may also execute scripts, such as, but not limited to, a combination of markup language and scripts, etc. In this document, references to a web page interface include one or more corresponding source files (which a browser may use to render the web page interface), and vice versa, where appropriate.

[0094] In certain embodiments, social networking system 1060 may be a network addressable computing system that may host an online social network. Social networking system 1060 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 1060 may be accessed directly by other components in network environment 1000 or through network 1010. By way of example and not limitation, client system 1030 may access social networking system 1060 directly or via network 1010 using a web browser of third-party content 1040 or a local application associated with social networking system 1060 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 1060 may include one or more servers 1062. Each server 1062 may be a single server or a distributed server across multiple computers or multiple data centers. The server 1062 may be of various types, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 1062 may include hardware, software, or embedded logic components, or a combination of two or more of these components, for performing the appropriate functions implemented or supported by the server 1062. In a particular embodiment, the social networking system 1060 may include one or more data repositories 1064. The data repository 1064 may be used to store various types of information. In a particular embodiment, the information stored in the data repository 1064 may be organized according to a particular data structure. In a particular embodiment, each data repository 1064 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 1030 , social networking system 1060 , or third-party system 1070 to manage, retrieve, modify, add, or delete information stored in data repository 1064 .

[0095] In certain embodiments, the social networking system 1060 may store one or more social graphs in one or more data stores 1064. In certain embodiments, the social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept) and a plurality of edges connecting the plurality of nodes. The social networking system 1060 may provide users of the online social network with the ability to communicate and interact with other users. In certain embodiments, users may join an online social network through the social networking system 1060 and may subsequently add connections (e.g., relationships) with a plurality of other users in the social networking system 1060 to whom they want to be connected. As used herein, the term "friend" may refer to any other user in the social networking system 1060 with whom the user has formed a connection, association, or relationship through the social networking system 1060.

[0096] In certain embodiments, social networking system 1060 may provide users with the ability to take actions on various types of items or objects supported by social networking system 1060. By way of example and not limitation, these items and objects may include groups or social networks to which users of social networking system 1060 may belong, events or calendar entries that may be of interest to the user, computer-based applications that the user may use, transactions that allow users to purchase or sell items through the service, interactions with advertisements that the user may act on, or other suitable items or objects. Users may interact with anything that can be represented in social networking system 1060 or represented by an external system of third-party system 1070 that is separate from social networking system 1060 and coupled to social networking system 1060 via network 1010.

[0097] In certain embodiments, social networking system 1060 may be able to link various entities. By way of example and not limitation, social networking system 1060 may enable users to interact with each other and receive content from third-party systems 1070 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0098] In certain embodiments, third-party system 1070 may include one or more types of servers, one or more data repositories, one or more interfaces (including, but not limited to, APIs), one or more web services, one or more content sources, one or more networks, or any other suitable component with which, for example, a server can communicate. Third-party system 1070 may be operated by a different entity than the entity that operates social networking system 1060. However, in certain embodiments, social networking system 1060 and third-party system 1070 may operate in conjunction with each other to provide social networking services to users of social networking system 1060 or users of third-party system 1070. In this sense, social networking system 1060 may provide a platform or backbone that other systems (e.g., third-party system 1070) may use to provide social networking services and functionality to users on the Internet.

[0099] In certain embodiments, third-party system 1070 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 1030. By way of example and not limitation, content objects may include information related to things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0100] In certain embodiments, social networking system 1060 also includes user-generated content objects that can enhance a user's interaction with social networking system 1060. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 1060. By way of example and not limitation, a user transmits a post from client system 1030 to social networking system 1060. Posts can include data such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 1060 by third parties through "communication channels" (e.g., news feeds or streams).

[0101] In certain embodiments, the social networking system 1060 may include various servers, subsystems, programs, modules, logs, and data repositories. In certain embodiments, the social networking system 1060 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object disclosure log, an inference module, an authorization / privacy server, a search module, an advertising positioning module, a user interface module, a user profile repository, a contact repository, a third-party content repository, or a location repository. The social networking system 1060 may also include suitable components, such as a web interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 1060 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biometric information, personal background information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, in-law relationships, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be the brand, or may be the general category "shoes" or "clothing". A contact repository may be used to store contact information about a user. The contact information may indicate users with similar or common work experience, group membership, hobbies, educational history, or users who are related or share common attributes in any way. The contact information may also include user-defined connections between different users and content (both internal and external). A web server may be used to link the social networking system 1060 to one or more client systems 1030 or one or more third-party systems 1070 via the network 1010. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 1060 and one or more client systems 1030. The API request server may allow a third-party system 1070 to access information from the social networking system 1060 by calling one or more APIs. An action logger may be used to receive information from a web server about a user's actions to log in or out of the social networking system 1060. In conjunction with the action log, a third-party content object log that a user has disclosed to a third-party content object may be maintained. The notification controller may provide information about the content object to the client system 1030. The information may be pushed to the client system 1030 as a notification, or may be pulled from the client system 1030 in response to a request received from the client system 1030. The authorization server may be used to enforce one or more privacy settings of users of the social networking system 1060.The user's privacy settings determine how specific information associated with the user can be shared. The authorization server can allow users to choose to have their actions recorded by the social networking system 1060 or shared with other systems (e.g., third-party systems 1070), for example, by setting appropriate privacy settings. The third-party content object repository can be used to store content objects received from third parties (e.g., third-party systems 1070). The location repository can be used to store location information received from client systems 1030 associated with users. The advertising pricing module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0102] Fig.11 An example computer system 1100 is shown. In particular embodiments, one or more computer systems 1100 perform one or more steps in one or more processes, algorithms, techniques, or methods described or shown herein. In particular embodiments, one or more computer systems 1100 provide functionality described or shown herein. In particular embodiments, software running on one or more computer systems 1100 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. Particular embodiments include one or more portions of one or more computer systems 1100. In this document, references to computer systems may include computing devices, and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0103] The present disclosure contemplates any suitable number of computer systems 1100. The present disclosure contemplates computer systems 1100 in any suitable physical form. By way of example and not limitation, computer system 1100 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented reality / virtual reality device, or a combination of two or more of these systems. Where appropriate, computer system 1100 may include one or more computer systems 1100; may be single or distributed; across multiple locations; across multiple machines; across multiple data centers; or located in a cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein in real time or in batch processing mode. Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0104] In a particular embodiment, computer system 1100 includes a processor 1102, memory 1104, storage 1106, an input / output (I / O) interface 1108, a communication interface 1110, and a bus 1112. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0105] In a particular embodiment, the processor 1102 includes hardware for executing a plurality of instructions, such as those that constitute a computer program. By way of example and not limitation, to execute a plurality of instructions, the processor 1102 may retrieve (or fetch) the instructions from an internal register, an internal cache, a memory 1104, or a storage 1106; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, a memory 1104, or a storage 1106. In a particular embodiment, the processor 1102 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1102 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1102 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1104 or the storage 1106, and the instruction cache may speed up the retrieval of the instructions by the processor 1102. The data in the data cache may be a copy of the data in the memory 1104 or storage 1106 for the instruction operation executed at the processor 1102; may be the result of the previous instruction executed at the processor 1102, for access by the subsequent instruction executed at the processor 1102, or for writing to the memory 1104 or storage 1106; or may be other suitable data. The data cache may speed up the read operation or write operation of the processor 1102. The TLB may speed up the virtual address translation of the processor 1102. In a particular embodiment, the processor 1102 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1102 including any suitable number of any suitable internal registers. Where appropriate, the processor 1102 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1102. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0106] In certain embodiments, the memory 1104 includes a main memory for storing instructions for execution by the processor 1102 or data for operation by the processor 1102. As an example and not limitation, the computer system 1100 can load instructions from the memory 1106 or another source (e.g., another computer system 1100) to the memory 1104. The processor 1102 can then load these instructions from the memory 1104 to an internal register or an internal cache memory. In order to execute these instructions, the processor 1102 can retrieve these instructions from the internal register or the internal cache memory and decode them. During or after executing these instructions, the processor 1102 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache memory. The processor 1102 can then write one or more of these results to the memory 1104. In a particular embodiment, the processor 1102 executes only instructions in one or more internal registers or in one or more internal cache memories, or in memory 1104 (different from memory 1106 or other locations), and operates only on data in one or more internal registers or in internal cache memories, or in memory 1104 (different from memory 1106 or other locations). One or more memory buses (each memory bus may include an address bus and a data bus) can couple the processor 1102 to the memory 1104. As described below, the bus 1112 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 1102 and the memory 1104 and facilitate access to the memory 1104 requested by the processor 1102. In a particular embodiment, the memory 1104 includes random access memory (RAM). Where appropriate, the RAM is a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1104 may include one or more memories 1104. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0107] In a particular embodiment, the memory 1106 includes a large capacity memory for data or instructions. As an example and not limitation, the memory 1106 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these memories. Where appropriate, the memory 1106 may include a removable medium or a non-removable (or fixed) medium. Where appropriate, the memory 1106 may be inside or outside the computer system 1100. In a particular embodiment, the memory 1106 is a non-volatile solid-state memory. In a particular embodiment, the memory 1106 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage 1106 taking any suitable physical form. Where appropriate, storage 1106 may include one or more storage control units that facilitate communication between processor 1102 and storage 1106. Where appropriate, storage 1106 may include one or more memories 1106. Although the present disclosure describes and illustrates particular memories, the present disclosure contemplates any suitable memories.

[0108] In a particular embodiment, the I / O interface 1108 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 1100 and one or more I / O devices. Where appropriate, the computer system 1100 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1100. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1108 for the I / O device. Where appropriate, the I / O interface 1108 may include one or more devices or software drivers that enable the processor 1102 to drive one or more of these I / O devices. Where appropriate, I / O interface 1108 may include one or more I / O interfaces 1108. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0109] In a particular embodiment, the communication interface 1110 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1100 and one or more other computer systems 1100 or one or more networks. By way of example and not limitation, the communication interface 1110 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other line-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1110 for the network. By way of example and not limitation, the computer system 1100 may communicate with one or more portions of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these networks. One or more portions of one or more of these networks may be wired or wireless. As an example, the computer system 1100 can communicate with the following networks: wireless PAN (wireless PAN, WPAN) (e.g., BLUETOOTH WPAN), WI-FI network, WI-MAX network, cellular phone network (e.g., Global System for Mobile Communication (GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 1100 may include any suitable communication interface 1110 for any of these networks. Where appropriate, the communication interface 1110 may include one or more communication interfaces 1110. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0110] In particular embodiments, bus 1112 includes hardware, software, or both that couples the various components of computer system 1100 to each other. By way of example and not limitation, bus 1112 may include: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, front-side bus (FSB), Hypertransport (HT) interconnect, Industry Standard Architecture (ISA) bus, InfiniBAND interconnect, low-pin-count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCIe) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these buses. Where appropriate, bus 1112 may include one or more buses 1112. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0111] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards (SECURE DIGITAL cards) or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0112] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is joint and several, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0113] The scope of the present disclosure includes all changes, substitutions, changes, transformations and modifications to the example embodiments described or shown herein, which will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific parts, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the parts, elements, features, functions, operations, or steps described or shown anywhere herein as will be understood by those of ordinary skill in the art. In addition, references to devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated to perform a specific function, or components in a device or system in the appended claims include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated. Additionally, although this disclosure may describe or illustrate a particular embodiment as providing particular advantages, that particular embodiment may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A method comprising: accessing a depth map and a first image of a scene, the depth map and the first image generated using one or more sensors of an artificial reality device; A plurality of segmentation masks are generated based on the first image, wherein the plurality of segmentation masks are respectively associated with a plurality of object types, wherein: Each segmentation mask identifies pixels in the first image corresponding to an object type associated with the segmentation mask; Using the plurality of segmentation masks, segment the depth map into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the plurality of object types; Generating a plurality of meshes using the plurality of segmented depth maps respectively; For each eye of the user: capturing a second image of the scene; generating segmentation information based on the second image, the segmentation information identifying pixels in the second image corresponding to the plurality of object types; warping the plurality of meshes to generate a plurality of warped meshes for the eyes; generating an eye-specific mesh for the eye by synthesizing the plurality of warped meshes for the eye based on the segmentation information of the second image; as well as An output image for the eye is rendered using the second image and the eye-specific mesh.

2. The method according to claim 1, wherein: The depth map is generated by temporally smoothing a raw depth map output by the one or more sensors.

3. The method according to claim 2, wherein: Temporally smoothing the original depth map to generate the depth map includes: generating optical flow data to represent motion between the first image and previous images acquired by the one or more sensors; generating a predicted depth map associated with the same time instance as the original depth map by applying the optical flow data to a previous depth map; and The depth map is generated based on the original depth map and the predicted depth map.

4. The method according to claim 3, wherein: The depth map is generated by averaging the original depth map and the predicted depth map.

5. The method according to claim 1, wherein: The one or more sensors include a time-of-flight sensor, and the first image is an output of the time-of-flight sensor.

6. The method according to claim 1, wherein: The one or more sensors include a pair of stereo cameras, and the first image is output by one of the pair of stereo cameras.

7. The method according to claim 1, further comprising: Prior to generating the plurality of meshes, missing depth information is filled in at least one of the plurality of segmented depth maps using a filter, wherein the plurality of meshes are generated using the plurality of depth maps after the missing information is filled.

8. The method according to claim 1, wherein: Generating the plurality of meshes further includes using one or more 3D models of the plurality of object types.

9. The method according to claim 8, wherein: At least one of the plurality of grids is generated by: identifying an object type associated with the grid, the object type selected from the plurality of object types; generating one or more 3D models of the identified object type, the one or more 3D models of the identified object type fitting observed features of one or more objects of the identified object type present in the scene; as well as The one or more 3D models are used to refine the mesh generated from the associated segmented depth map.

10. The method according to claim 9, wherein: The object type identified is at least one of a plane, a person, or a static object observed in the scene over a period of time.

11. The method according to claim 1, wherein: The plurality of warped grids, the eye-specific grids, and the output image generated for the left eye of the user are different from the plurality of warped grids, the eye-specific grids, and the output image generated for the right eye of the user.

12. The method according to claim 1, wherein: The multiple distorted meshes for the eye are generated by distorting the multiple meshes based on the position of a camera of the artificial reality device used to capture the second image.

13. The method according to claim 11, wherein: The plurality of warped meshes for the eyes are generated by warping the plurality of meshes based on an updated pose of the artificial reality device.

14. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: accessing a depth map and a first image of a scene, the depth map and the first image generated using one or more sensors of an artificial reality device; A plurality of segmentation masks are generated based on the first image, wherein the plurality of segmentation masks are respectively associated with a plurality of object types, wherein: Each segmentation mask identifies pixels in the first image corresponding to an object type associated with the segmentation mask; Using the plurality of segmentation masks, segment the depth map into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the plurality of object types; Generating a plurality of meshes using the plurality of segmented depth maps respectively; For each eye of the user: capturing a second image of the scene; generating segmentation information based on the second image, the segmentation information identifying pixels in the second image corresponding to the plurality of object types; warping the plurality of meshes to generate a plurality of warped meshes for the eyes; generating an eye-specific mesh for the eye by synthesizing the plurality of warped meshes for the eye based on the segmentation information of the second image; as well as An output image for the eye is rendered using the second image and the eye-specific mesh.

15. One or more computer-readable non-transitory storage media according to claim 14, wherein: The depth map is generated by temporally smoothing a raw depth map output by the one or more sensors.

16. One or more computer-readable non-transitory storage media according to claim 15, wherein: Temporally smoothing the original depth map to generate the depth map includes: generating optical flow data to represent motion between the first image and previous images acquired by the one or more sensors; generating a predicted depth map associated with the same time instance as the original depth map by applying the optical flow data to a previous depth map; and The depth map is generated based on the original depth map and the predicted depth map.

17. The one or more computer-readable non-transitory storage media of claim 14, wherein: Generating the plurality of meshes further includes using one or more 3D models of the plurality of object types.

18. An artificial reality device comprising: one or more sensors; at least one display component; one or more processors; as well as one or more computer-readable non-transitory storage media coupled to one or more of the one or more processors and comprising instructions that, when executed by one or more of the one or more processors, are operable to cause the artificial reality device to: accessing a depth map and a first image of a scene, the depth map and the first image generated using one or more sensors of an artificial reality device; generating a plurality of segmentation masks based on the first image, the plurality of segmentation masks being respectively associated with a plurality of object types, wherein each segmentation mask identifies pixels in the first image corresponding to the object type associated with the segmentation mask; Using the plurality of segmentation masks, segment the depth map into a plurality of segmented depth maps, the plurality of segmented depth maps being respectively associated with the plurality of object types; Generating a plurality of meshes using the plurality of segmented depth maps respectively; For each eye of the user: capturing a second image of the scene; generating segmentation information based on the second image, the segmentation information identifying pixels in the second image corresponding to the plurality of object types; warping the plurality of meshes to generate a plurality of warped meshes for the eyes; generating an eye-specific mesh for the eye by synthesizing the plurality of warped meshes for the eye based on the segmentation information of the second image; and An output image for the eye is rendered using the second image and the eye-specific mesh.

19. The artificial reality device of claim 18, wherein: The depth map is generated by temporally smoothing a raw depth map output by the one or more sensors.

20. The artificial reality device of claim 19, wherein: Temporally smoothing the original depth map to generate the depth map includes: generating optical flow data to represent motion between the first image and previous images acquired by the one or more sensors; generating a predicted depth map associated with the same time instance as the original depth map by applying the optical flow data to a previous depth map; and The depth map is generated based on the original depth map and the predicted depth map.

Images