Temporally stable viewing angle correction

Abstract

In one embodiment, a method of performing viewing angle correction is performed by a device including an image sensor, a display, one or more processors, and a non-transitory memory. The method includes capturing an image of a physical environment using the image sensor. The method includes acquiring a depth map including a plurality of depths respectively associated with a plurality of pixels of an image of the physical environment, for a particular pixel representing a dynamic object in the physical environment at a particular pixel location, the depth map includes a particular depth corresponding to a distance between the image sensor and a static object behind the dynamic object in the physical environment. The method includes transforming, using the one or more processors, an image of the physical environment based on the depth map. The method includes displaying the transformed image on the display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 403,938, filed on September 6, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and apparatus for performing temporally stable perspective correction. Background Art

[0004] In various embodiments, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures an image of the physical environment (e.g., a scene) in which a user is present and a display that displays the image to the user. In some cases, the image or a portion thereof may be combined with one or more virtual objects to present an XR experience to the user. In other cases, the HMD may operate in a pass-through mode in which an image or a portion thereof is presented to the user without adding virtual objects. Ideally, the image of the physical environment presented to the user is substantially similar to what the user would see if the HMD were not present. However, due to the different positioning of the eyes, display, and camera in space, this may not occur, resulting in impaired distance perception, disorientation, and poor hand-eye coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order that the present disclosure may be understood by those skilled in the art, a more particular description will be given with reference to some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1 is a block diagram of an example operating environment according to some implementations.

[0007] Figure 2 Example scenarios related to capturing images of a physical environment and displaying the captured images according to some implementations are illustrated.

[0008] Figure 3 An image of the physical environment captured by an image sensor from a specific perspective.

[0009] Figure 4 yes Figure 3 A top-down perspective view of the physical environment.

[0010] Figure 5A This illustrates what the user's left eye sees at the first time when the user is not wearing the HMD. Figure 3 A first view of the physical environment.

[0011] Figure 5BThe left image sensor of the HMD captures the image for the first time. Figure 3 A first image of the physical environment.

[0012] Fig. 6A Illustrated Figure 5B The first depth map of the first image is a center row of the first depth map.

[0013] Figure 6B Illustrated Figure 5B The first static depth map of the first image is a center row of the first static depth map.

[0014] Fig. 7A Illustrated based on Figure 5B The first image and Fig. 6A The first transformed first image of the first depth map.

[0015] Figure 7B Illustrated based on Figure 5B The first image and Figure 6B The static depth map of the second transformed first image.

[0016] Fig. 8A Illustrate what would be seen by the user's left eye at the second time if the user was not wearing the HMD Figure 3 A second view of the physical environment.

[0017] Figure 8B exemplifies the image captured by the left image sensor of the HMD at the second time Figure 3 A second image of the physical environment.

[0018] Fig. 9A Illustrated Figure 8B The second depth map of the second image is a center row of the second depth map.

[0019] Fig. 9B Illustrated based on Figure 8B The second image and Fig. 9A The second depth map is a first transformed second image.

[0020] Fig. 10A Illustrated based on Figure 8B The second image and Fig. 6A A second image of a second transformation of the first depth map.

[0021] Fig. 10B Illustrated based on Figure 8B The second image and Figure 6B A third transformed second image of the first static depth map.

[0022] Fig.11 is a flowchart representation of a method for performing perspective correction according to some specific implementations.

[0023] Fig.12 is a block diagram of an example controller according to some specific implementations.

[0024] Fig.13 is a block diagram of an example electronic device according to some implementations.

[0025] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0026] Various embodiments disclosed herein include devices, systems, and methods for performing perspective correction. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of a physical environment using the image sensor. The method includes acquiring a depth map including multiple depths associated with multiple pixels of the image of the physical environment, respectively, wherein for a specific pixel representing a dynamic object in the physical environment at a specific pixel position, the depth map includes a specific depth corresponding to the distance between the image sensor and a static object behind the dynamic object in the physical environment. The method includes using the one or more processors to transform the image of the physical environment based on the depth map. The method includes displaying the transformed image on the display.

[0027] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0028] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0029] As described above, in an HMD with a display and a scene camera, if the HMD does not exist due to the different positions of the eyes, display, and camera in space, the image of the physical environment presented to the user on the display may not always reflect what the user sees. In various cases, this leads to, for example, poor distance perception, disorientation and discomfort (e.g., motion sickness) for the user when interacting with the physical environment, and poor hand-eye coordination. Therefore, in various specific implementations, the images from the scene camera are transformed so that they appear to be captured at the location of the user's eyes using a depth map that represents, for each pixel of the image, the distance from the camera to the object represented by the pixel. In various specific implementations, the images from the scene camera are partially transformed so that they appear to be captured at a location that is closer to the location of the user's eyes in one or more dimensions than the location of the scene camera.

[0030] In various implementations, the depth map is changed to reduce artifacts. For example, in various implementations, the depth map is smoothed to avoid holes in the transformed image. In various implementations, the depth map is clamped to reduce large movements of pixels during the transformation. In various implementations, the depth map is made static so that dynamic objects do not contribute to the depth map. For example, in various implementations, the depth map value at the location of the dynamic object is determined by interpolating the depth map using positions around the location of the dynamic object. In various implementations, the depth map value at the location of the dynamic object is determined based on the depth map value determined when the dynamic object is not at the location. In various implementations, the depth map is determined using a three-dimensional model of the physical environment without dynamic objects. Using a static depth map can increase spatial artifacts, such as objects not being displayed at their true locations. However, using a static depth map can reduce temporal artifacts, such as flicker.

[0031] Figure 1 1 is a block diagram of an example operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0032] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Fig.12 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by and / or combined with electronic device 120.

[0033] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. Reference is made below. Fig.13 The electronic device 120 is described in more detail.

[0034] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .

[0035] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.

[0036] Figure 2 An example scenario 200 is illustrated involving capturing an image of an environment and displaying the captured image according to some implementations. A user wears a device (e.g., Figure 1 The image sensor 230 captures an image of the physical environment, and the display 210 displays the image of the physical environment to the user's eye 220. The image sensor 230 has a viewing angle that is vertically offset from the user's viewing angle (e.g., where the user's eye 220 is located) by a vertical offset 241. Additionally, the viewing angle of the image sensor 230 is longitudinally offset from the user's viewing angle by a longitudinal offset 242. Additionally, in various implementations, the viewing angle of the image sensor 230 is laterally offset from the user's viewing angle by a lateral offset (e.g., toward Figure 2 within or offset outside of the page).

[0037] Figure 3 is an image 300 of a physical environment 301 captured by an image sensor from a specific viewing angle. The physical environment 301 includes a structure 310 having a first surface 311 closer to the image sensor, a second surface 312 farther from the image sensor, and a third surface 313 connecting the first surface 311 and the second surface 312. The first surface 311 has letters A, B, and C painted thereon, the third surface 313 has letters D painted thereon, and the second surface 312 has letters E, F, and G painted thereon.

[0038] From a particular viewing angle, image 300 includes all of the letters painted on structure 310. However, from other viewing angles, as described below, the captured image may not include all of the letters painted on structure 310.

[0039] Figure 4 yes Figure 3 4 is a top-down perspective view of a physical environment 301 of FIG. 4. The physical environment 301 includes a structure 310 and a user 410 wearing an HMD 420. The user 410 has a left eye 411a at a left eye position, thereby providing a left eye perspective. The user 410 has a right eye 411b at a right eye position, thereby providing a right eye perspective. The HMD 420 includes a left image sensor 421a at a left image sensor position that provides a left image sensor perspective. The HMD 420 includes a right image sensor 421b at a right image sensor position that provides a right image sensor perspective. Since the left eye 411a of the user 410 and the left image sensor 421a of the HMD 420 are at different positions, they each provide a different perspective of the physical environment.

[0040] Figure 5A A first view 501 of the physical environment 301 seen at the first time by the left eye 411a of the user 410 when the user 410 is not wearing the HMD 420 is illustrated. In the first view 501, there are the first surface 311 and the second surface 312, but there is no third surface 313. On the first surface 311, the letters B and C can be seen, while the letter A is not in the field of view of the left eye 411a. Similarly, on the second surface 312, the letters E, F, and G can be seen. The view 501 also includes a person 599 in front of the first surface 311 between the letters B and C.

[0041] Figure 5B A first image 502 of the physical environment 301 captured by the left image sensor 421a at a first time is illustrated. As with the first view 501, in the first image 502, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are at least partially visible, while the letter A is not in the field of view of the left image sensor 421a. Similarly, on the second surface 312, the letters F and G are visible, while the letter E is not in the field of view of the left image sensor 421a. It is noteworthy that in the first image 502, compared to the first view 501, the letter E is not present on the second surface 312. Therefore, the letter E is in the field of view of the left eye 411a, but not in the field of view of the left image sensor 421a. The first image 502 also includes a person 599 in front of the first surface 311 that partially blocks the letter C.

[0042] In various implementations, the HMD 420 transforms the first image 502 so that it appears as if it is captured from a left eye perspective rather than from a left image sensor perspective, for example, to look like the first view 501. In various implementations, the transformation is a perspective transformation. In various implementations, the HMD 420 transforms the first image 502 based on a depth value associated with the first image 502 and a difference between the left image sensor perspective and the left eye perspective. In various implementations, the difference between the left image sensor perspective and the left eye perspective is determined during a calibration process. In various implementations, the depth value of a pixel of the first image 502 represents a distance from the left image sensor 421a to an object represented by the pixel in the physical environment 301. In various implementations, the depth value is used to generate a depth map that includes a corresponding depth value for each pixel of the first image 502.

[0043] In various implementations, the resulting transformed image includes holes, such as pixel positions of the transformed image where there are no corresponding pixel positions of the first image 502. Such holes may be filled via interpolation or using additional images, such as using another image from a different perspective (e.g., from the right image sensor 421b or from the left image sensor 421a at a different time).

[0044] In various implementations, the resulting transformed image includes blur, such as pixel locations of the transformed image that exist at multiple corresponding pixel locations of the first image 502. An averaging or consensus algorithm may be used to remove such blur.

[0045] In various implementations, the depth map does not include dynamic objects and / or only includes static objects. In various implementations, the depth map does not include movable objects and / or only includes fixed objects. In various implementations, the depth map does not include temporary objects and / or only includes permanent objects. Therefore, in various implementations, the depth map does not include personnel 599. For each pixel position representing personnel 599 in the first image 502, a depth value is determined by ignoring the distance from the left image sensor 421a to the personnel 599. Instead, the depth value represents the distance from the left image sensor 421a to the static object behind the personnel 599. In various implementations, the depth value is determined by interpolating the depth values ​​of the pixels around the pixel position representing the personnel 599. In various implementations, the depth value is determined based on the depth value of the pixel position when the personnel 599 is not at the pixel position. In various implementations, the depth value is determined using a three-dimensional model of the physical environment 301 that does not include personnel 599.

[0046] For example, in various specific implementations, the HMD 420 generates a three-dimensional model including a three-dimensional mesh. The three-dimensional mesh includes a set of triangles connected by their common edges and / or vertices. Each vertex is located at a three-dimensional position in the three-dimensional coordinate system of the physical environment 301. In order to generate the three-dimensional model, the HMD 420 uses a depth sensor, stereo matching, or any other method to determine the three-dimensional position of the vertex. However, in various specific implementations, the vertex is added only when it is determined that the corresponding object is a static object. For example, when using a depth sensor, the vertex can only be added if the number of times the vertex is detected in a sufficiently long period of time reaches at least a threshold number of times. As another example, when using stereo matching, if the pixel in the stereo image corresponds to a static object in the image determined via semantic segmentation, the vertex can be added. For example, in a first image of the physical environment and a second image of the physical environment, a neural network can be used to classify each pixel as one of a plurality of object types. For pixels classified as walls or tables in the first image and corresponding pixels with similar classifications in the second image, the vertices corresponding to the pixels can be added to the three-dimensional model. For pixels in the first image that are classified as a person, hand, dog, or vehicle and corresponding pixels in the second image that have similar classifications, vertices corresponding to the pixels may be added to the three-dimensional model.

[0047] Using a static depth map may increase spatial artifacts, such as person 599 not being displayed at their true location in physical environment 301. However, using a static depth map may reduce temporal artifacts, such as flickering or distortion. In addition, another advantage of using a three-dimensional model to generate a depth map is that a static depth map can be rendered from any viewpoint where the room geometry is visible.

[0048] Fig. 6A A first depth map 600 of a center row of a first depth map of a first image 502 is illustrated. The first depth map 600 includes a left first portion 601A corresponding to distances between the left image sensor 421A and various points on the first surface 311 to the left of the person 599, and a right first portion 601B corresponding to distances between the left image sensor 421A and various points on the first surface 311 to the right of the person 599. The first depth map 600 includes a second portion 602 corresponding to distances between the left image sensor 421A and various points on the second surface 312. The first depth map 600 includes a third portion 603 corresponding to distances between the left image sensor 421A and various points on the person 599.

[0049] Figure 6BThe first static depth map 610 of the center row of the first static depth map of the first image 502 is illustrated. In various implementations, the first static depth map is generated by rasterizing the three-dimensional model or performing ray tracing based on the three-dimensional model. Therefore, each pixel in the static depth map indicates the distance from the left image sensor 421A to the static object in the physical environment 301 through the pixel of the first image 502 in the camera plane.

[0050] The first static depth map 610 includes a first portion 611 corresponding to the distances between the left image sensor 421A and various points on the first surface 311, and a second portion 602 corresponding to the distances between the left image sensor 421A and various points on the second surface 312. It is worth noting that the first static depth map 610 does not include a third portion 603 corresponding to the distances between the left image sensor 421A and various points on the person 599 because the person 599 is not a static object.

[0051] Fig. 7A A first transformed first image 701 generated by transforming the first image 502 based on the first depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective is illustrated.

[0052] In the first transformed first image 701, the first surface 311 and the second surface 312 exist, but the third surface 313 does not exist. On the first surface 311, the letter B and the letter C can be seen. Similarly, on the second surface 312, the letter E, the letter F, and the letter G can be seen. The first transformed first image 701 also includes the person 599 in front of the first surface 311 between the letter B and the letter C. Therefore, the first transformed first image 701 is substantially the same as the first view 501. Although the left half of the letter C and the entire letter E are not visible in the first image 502, the left half of the letter C and the entire letter E can be seen in the first transformed first image 701 by using other images for hole filling (rather than interpolation).

[0053] Figure 7B A second transformed first image 702 generated by transforming the first image 502 based on the first static depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective is illustrated.

[0054] In the second transformed first image 701, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are at least partially visible. Similarly, on the second surface 312, the letters E, F, and G are visible. The second transformed first image 701 also includes a person 599 in front of the first surface 311 that partially occludes the letter C. Therefore, although the first surface 311 and the second surface 312 have been correctly transformed and the letters have the same size and position as in the first view 501, the person 599 is not between the letters B and C (as shown in the first view 501), but partially occludes the letter C (as shown in the first image 502). The transformation based on the first static depth map regards the person 599 as being located at the depth of the first surface 311 (e.g., painted on the first surface 311), rather than being located in front of the first surface 311. Therefore, the portion of the first image 502 corresponding to the person 599 is mapped to the same position as the portion of the first image corresponding to the letter C. Although the letter E is not visible in the first image 502 , by using other images for hole filling (rather than interpolation), the letter E can be seen in the second transformed first image 702 .

[0055] Fig. 8A A second view 801 of the physical environment 301 seen by the left eye 411a of the user 410 at a second time after the first time when the user 410 is not wearing the HMD 420 is illustrated. In the second view 801, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are visible, while the letter A is not in the field of view of the left eye 411a. Similarly, on the second surface 312, the letters E, F, and G are visible. The second view 801 also includes a person 599 that has moved in front of the second surface 312, blocking the letter E.

[0056] Figure 8B A second image 802 of the physical environment 301 captured by the left image sensor 421a at a second time is illustrated. In the second image 802, as in the second view 801, the first surface 311 of the structure 310 and the second surface 312 of the structure 310 are present, but the third surface 313 is absent. On the first surface 311, the letters B and C are visible, while the letter A is not in the field of view of the left image sensor 421a. Similarly, on the second surface 312, the letters F and G are at least partially visible, while the letter E is not in the field of view of the left image sensor 421a. The second image 802 also includes a person 599 in front of the first surface 311 that partially obscures the letter F.

[0057] In various implementations, the HMD 420 transforms the second image 802 so that it appears as if it is captured from a left eye perspective rather than from a left image sensor perspective, e.g., appears like the second view 801. In various implementations, the HMD 420 transforms the second image 802 based on a depth value associated with the second image 802 and the difference between the left image sensor perspective and the left eye perspective. In various implementations, the depth value is used to generate a depth map that includes a corresponding depth value for each pixel of the second image 802.

[0058] Fig. 9A A second depth map 900 of a center row of a second depth map of the second image 802 is illustrated. The second depth map 900 includes a first portion 901 corresponding to distances between the left image sensor 421A and various points on the first surface 311. The second depth map 900 includes a left second portion 902A corresponding to distances between the left image sensor 421A and various points on the second surface 312 to the left of the person 599, and a right second portion 902B corresponding to distances between the left image sensor 421A and various points on the second surface 312 to the right of the person 599. The second depth map 900 includes a third portion 903 corresponding to distances between the left image sensor 421A and various points on the person 599.

[0059] Fig. 9B A first transformed second image 911 generated by transforming the second image 802 based on the second depth map of the second image 802 and the difference between the left scene camera perspective and the left eye perspective is illustrated.

[0060] In the first transformed second image 911, the first surface 311 and the second surface 312 are present, but the third surface 313 is absent. On the first surface 311, the letter B and the letter C are visible. Similarly, on the second surface 312, the letter F and the letter G are visible. The first transformed second image 911 also includes the person 599 in front of the second surface 312, which blocks the letter E. Therefore, the first transformed second image 911 is substantially the same as the second view 801. Although a portion of the letter F is not visible in the second image 802, the entire letter F can be seen in the first transformed second image 911 by using other images for hole filling (rather than interpolation).

[0061] In various implementations, generating a depth map may be time consuming. Therefore, at the second time, the second depth map may be unavailable, incomplete, or inaccurate. For example, at the second time, only the first depth map (corresponding to the first time) may be available.

[0062] Fig. 10AA second transformed second image 1001 generated by transforming the second image 802 based on the first depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective is illustrated.

[0063] In the second transformed second image 1001, the first surface 311 and the second surface 312 are present, but the third surface 313 is absent. On the first surface 311, two instances of the letter C and the letter B can be seen. The first instance (on the left) is generated by mapping the letter C in the second image 802 to a new position based on the first depth map. The second instance (on the right) is generated by hole filling the hole generated by transforming based on the first depth map in which the person 599 is in front of the letter C.

[0064] On the second surface 312 , the letter E, the letter F and the letter G are at least partially visible. As in the second image 802 , the second transformed second image 1001 also includes the person 599 partially obscuring the letter F.

[0065] Thus, in the second transformed second image 1001, person 599 is in the wrong position compared to the second view 801 and a second instance of the letter C is introduced between letters B and C. To avoid artifacts, such as repeated letters C or distortions around dynamic objects, a static depth map may be used. Because the perspectives of the first image 502 and the second image 802 are the same, the second static depth map of the second image 802 is consistent with the second static depth map having the same depth map as the first image 502 and the second image 802. Figure 6B The first static depth map 610 illustrated in FIG. 5 is the same as the first static depth map of the first image 502 .

[0066] Fig. 10B A third transformed second image 1002 generated by transforming the second image 802 based on the second static depth map of the second image 802 and the difference between the left scene camera perspective and the left eye perspective is illustrated.

[0067] In the third transformed second image 1002, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are visible. Similarly, on the second surface 312, the letters E, F, and G are at least partially visible. The third transformed second image 1002 also includes the person 599 in front of the second surface 311, partially occluding the letter F. Therefore, although the first surface 311 and the second surface 312 are correctly transformed, where the letters have the same size in the third transformed second image 1002 and the second view 801, the person 599 does not occlude the letter E (as in the second view 801), but partially occludes the letter F (as in the second image 802). The third transformed second image 1002 treats the person 599 as if the person 599 is located at the depth of the second surface 312 (e.g., painted on the second surface 312), rather than in front of the second surface 312. Therefore, the portion of second image 802 corresponding to person 599 is mapped to the same position as the portion of second image 802 corresponding to letter F. Although the letter E is not seen in second image 802, the letter E is seen in the third transformed second image 1002 by performing hole filling (rather than interpolation) using images from other perspectives.

[0068] Fig.11 11 is a flowchart representation of a method for performing image perspective correction according to some specific implementations. In various specific implementations, method 1100 is performed by a device having an image sensor, a display, one or more processors, and a non-volatile memory (e.g., Figure 1 In some implementations, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1100 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0069] The method 1100 begins in block 1110 where a device captures an image of a physical environment using an image sensor.

[0070] Method 1100 continues in box 1120, where the device acquires a depth map including multiple depths respectively associated with multiple pixels of the image of the physical environment, wherein for a particular pixel representing a dynamic object in the physical environment at a particular pixel position, the depth map includes a specific depth corresponding to a distance between the image sensor and a static object in the physical environment behind the dynamic object.

[0071] In various implementations, the depth map includes a dense depth map that represents, for each pixel of the image, an estimated distance between the image sensor and a static object represented by the pixel or a static object behind a dynamic object represented by the pixel. In various implementations, the depth map includes a sparse depth map that represents, for each subset of pixels of the image, an estimated distance between the image sensor and a static object represented by the pixel or a static object behind a dynamic object represented by the pixel. In various implementations, the device generates the sparse depth map from the dense depth map by sampling the dense depth map, such as selecting a single pixel in each N×N pixel block.

[0072] In various implementations, the device obtains at least one of the multiple depths from a depth sensor. In various implementations, the device uses stereo matching to obtain at least one of the multiple depths, for example, using an image of the physical environment captured by a left scene camera and another image of the physical environment captured by a right scene camera. In various implementations, the device obtains the multiple depths by eye tracking, for example, the intersection of the gaze directions of the two eyes of the user indicates the depth of the object the user is looking at. In various implementations, as further discussed below, the device obtains the multiple depths based on a three-dimensional model of the physical environment that does not include dynamic objects.

[0073] In various implementations, obtaining the depth map includes determining a specific depth by interpolating depths at locations surrounding the specific pixel. For example, in various implementations, to generate Figure 6B The first portion 611 of the static depth map 600, the HMD 420 Figure 6B Interpolation is performed between the left first portion 601a and the right first portion 601b of the first depth map 610.

[0074] In various implementations, obtaining the depth map includes determining a specific depth when the dynamic object is not represented at the pixel location. For example, in various implementations, to generate a second static depth map (which appears to be the same as Figure 6B When the person 599 is not at the right part, the HMD 420 uses Fig. 6A In various implementations, method 1100 includes obtaining a three-dimensional model of a physical environment that does not include dynamic objects. In various implementations, method 1100 includes obtaining a three-dimensional model of the physical environment based on objects determined to be static in the physical environment. In various implementations, obtaining the three-dimensional model includes retrieving the three-dimensional model from a device or a non-volatile memory remote from the device. In various implementations, obtaining the three-dimensional model includes generating the three-dimensional model.

[0075] In various implementations, generating the three-dimensional model includes adding one or more points to the three-dimensional model at one or more locations in a three-dimensional coordinate system of the physical environment corresponding to one or more static objects in the physical environment. In various implementations, the one or more points are vertices of mesh triangles. In various implementations, adding the one or more points to the three-dimensional model includes determining that the one or more points correspond to the one or more static objects in the physical environment (and / or adding the points after determining that this is the case).

[0076] In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes performing semantic segmentation on one or more images. For example, pixels classified as a table or a wall may form part of a key point pair in stereo matching to generate a point to be added to a three-dimensional model, while pixels classified as a person or a vehicle may not form part of a key point pair in stereo matching to generate a point to be added to a three-dimensional model. In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes detecting the particular point at the same location at least a threshold number of times over a period of time. For example, if a feature is detected at the same location in the physical environment on the following two or more days, then adding a point determined to correspond to a static object at that location. In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes determining that surrounding points correspond to one or more static objects. For example, if points of a wall are detected at three non-linear coplanar locations, points on a specified plane between these points may be determined to be points corresponding to one or more static objects.

[0077] Thus, in various implementations, the three-dimensional model is based on historical data acquired at one or more times before the image of the physical environment is captured. For example, the three-dimensional model can be based on data acquired one second, several seconds, one minute, several minutes, one hour, one day, or any other time before the image of the physical environment is captured.

[0078] In various implementations, determining the specific depth based on the three-dimensional model includes rasterizing the three-dimensional model. In various implementations, determining the specific depth based on the three-dimensional model includes ray tracing based on the three-dimensional model.

[0079] In various implementations, acquiring the depth map includes acquiring a temporally stabilized depth map that includes depths based on distances between the image sensor and static objects and does not include depths based on distances between the image sensor and dynamic objects.

[0080] The method 1100 continues at block 1130 where the device, using one or more processors, transforms the image of the physical environment based on the depth map. In various implementations, the device transforms the image of the physical environment at an image pixel level, an image tile level, or a combination thereof.

[0081] In various implementations, the device transforms the image of the physical environment based on a difference between a first perspective and a second perspective of the image sensor. In various implementations, the second perspective is a perspective of a user, such as a perspective of an eye of the user. In various implementations, the second perspective is a perspective at a location closer to an eye of the user in one or more directions.

[0082] In various implementations, the device performs a projective transformation based on the depth map and a difference between a first perspective and a second perspective of the image sensor.

[0083] In various implementations, the projective transformation is a forward mapping, wherein for each pixel of the image of the physical environment at the pixel position in the untransformed space, a new pixel position is determined in the transformed space of the transformed image. In various implementations, the projective transformation is a reverse mapping, wherein for each pixel of the transformed image at the pixel position in the transformed space, a source pixel position is determined in the untransformed space of the image of the physical environment.

[0084] In various implementations, the source pixel position is determined according to the following equation, where x1 and y1 are the pixel positions in the untransformed space, x2 and y2 are the pixel positions in the transformed space, P2 is the 4×4 view projection matrix of the second perspective, P1 is the 4×4 view projection matrix of the first perspective of the image sensor, and d is the depth map value at the pixel position:

[0085]

[0086] In various implementations, for each pixel in the image of the physical environment, the source pixel position is determined using the above equation. In various implementations, for each pixel in an image that is smaller than the physical environment, the source pixel position is determined using the above equation.

[0087] In various implementations, the device determines a view projection matrix for the second perspective and a view projection matrix for the first perspective during calibration, and stores data indicating the view projection matrices (or their product) in a non-transitory memory. The product of the view projection matrices is a transformation matrix representing the difference between the first perspective and the second perspective of the image sensor.

[0088] Thus, in various implementations, transforming the image of the physical environment includes, for a plurality of pixels of the transformed image having corresponding pixel positions, determining a corresponding plurality of source pixel positions. In various implementations, determining the corresponding plurality of source pixel positions includes, for each of the plurality of pixels of the transformed image, multiplying a vector including the corresponding pixel position and a multiplicative inverse of a corresponding element of the smoothed depth map by a transformation matrix representing a difference between a first perspective and a second perspective of the image sensor.

[0089] Using the source pixel positions in the untransformed space and the pixel values ​​of the pixels of the image of the physical environment, the device generates a pixel value for each pixel position of the transformed image using interpolation or other techniques.

[0090] In various implementations, the device modifies the depth map before performing the transformation using the depth map. For example, in various implementations, transforming the image of the physical environment based on the depth map also includes smoothing the depth map. For example, in various implementations, the device applies a two-dimensional low-pass filter to the depth map. As another example, in various implementations, transforming the image of the physical environment based on the depth map also includes clamping the depth map. For example, in various implementations, any values ​​in the depth map below a clamping threshold are replaced by the clamping threshold.

[0091] Method 1100 continues in box 1140, where the device displays the transformed image on a display. In various embodiments, the transformed image includes XR content. In some embodiments, the XR content is added to the current image of the physical environment before the transformation (at box 1130). In some embodiments, the XR content is added to the transformed image. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on metadata indicating the addition of the XR content to the physical environment. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on the amount of XR content (e.g., the percentage of the image of the physical environment that contains XR content).

[0092] In various implementations, the device determines whether to add the XR content to the image of the environment before or after the transformation based on metadata indicating the depth of the XR content. Accordingly, in various implementations, method 1100 includes receiving XR content and XR content metadata, selecting an image of the physical environment or a transformed image based on the XR content metadata, and adding the XR content to the selected image.

[0093] Fig.12is a block of an example of a controller 110 according to some implementations. While certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 1202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 1206, one or more communication interfaces 1208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1210, a memory 1220, and one or more communication buses 1204 for interconnecting these components and various other components.

[0094] In some implementations, the one or more communication buses 1204 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 1206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0095] Memory 1220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, memory 1220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1220 optionally includes one or more storage devices away from one or more processing units 1202. Memory 1220 includes non-transitory computer-readable storage media. In some specific implementations, memory 1220 or non-transitory computer-readable storage media of memory 1220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1230 and an XR experience module 1240.

[0096] The operating system 1230 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 1240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 1240 includes a data acquisition unit 1242, a tracking unit 1244, a coordination unit 1246, and a data transmission unit 1248.

[0097] In some specific implementations, the data acquisition unit 1242 is configured to at least Figure 1 The electronic device 120 acquires data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various specific implementations, the data acquisition unit 1242 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0098] In some implementations, the tracking unit 1244 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various implementations, the tracking unit 1244 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0099] In some implementations, the coordination unit 1246 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 1246 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0100] In some implementations, the data transfer unit 1248 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transfer unit 1248 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0101] Although the data acquisition unit 1242, the tracking unit 1244, the coordination unit 1246, and the data transmission unit 1248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 1242, the tracking unit 1244, the coordination unit 1246, and the data transmission unit 1248 may be located in separate computing devices.

[0102] also, Fig.12It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig.12 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0103] Fig.13 is a block diagram of an example of an electronic device 120 according to some implementations. While certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 1302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1306, one or more communication interfaces 1308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1310, one or more XR displays 1312, one or more optional internal-facing and / or external-facing image sensors 1314, a memory 1320, and one or more communication buses 1304 for interconnecting these and various other components.

[0104] In some implementations, the one or more communication buses 1304 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0105] In some implementations, one or more XR displays 1312 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 1312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 1312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. In another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 1312 can present MR and VR content.

[0106] In some implementations, the one or more image sensors 1314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 1314 are configured to face forward so as to acquire image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 1314 may include one or more RGB cameras (e.g., having a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0107] Memory 1320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, memory 1320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1320 optionally includes one or more storage devices away from one or more processing units 1302. Memory 1320 includes non-transitory computer-readable storage media. In some specific implementations, memory 1320 or non-transitory computer-readable storage media of memory 1320 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1330 and an XR rendering module 1340.

[0108] The operating system 1330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 1340 is configured to present XR content to a user via one or more XR displays 1312. To this end, in various implementations, the XR rendering module 1340 includes a data acquisition unit 1342, a perspective transformation unit 1344, an XR rendering unit 1346, and a data transmission unit 1348.

[0109] In some specific implementations, the data acquisition unit 1342 is configured to at least Figure 1 The controller 110 acquires data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various implementations, the data acquisition unit 1342 includes instructions and / or logic components for instructions as well as heuristics and metadata for heuristics.

[0110] In some implementations, the perspective transformation unit 1344 is configured to perform perspective correction based on static objects (rather than dynamic objects) using a temporally stable depth map. To this end, in various implementations, the perspective transformation unit 1344 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0111] In some implementations, the XR rendering unit 1346 is configured to display the transformed image via one or more XR displays 1312. To this end, in various implementations, the XR rendering unit 1346 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0112] In some implementations, the data transmission unit 1348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 1348 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 1348 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0113] Although the data acquisition unit 1342, the perspective transformation unit 1344, the XR rendering unit 1346, and the data transmission unit 1348 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 1342, the perspective transformation unit 1344, the XR rendering unit 1346, and the data transmission unit 1348 may be located in separate computing devices.

[0114] also, Fig.13More as a functional description of various features that may be present in a particular implementation, as opposed to the structural schematic diagrams of the implementations described herein. As will be appreciated by those of ordinary skill in the art, the items shown separately may be combined, and some items may be separated. For example, some of the functional modules shown separately in Figure 10 may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the implementation, and in some implementations, depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0115] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, a device and / or a method can be implemented using any number of aspects set forth herein. In addition, in addition to or different from one or more aspects set forth herein, such a device and / or such a method can be implemented using other structures and / or functions.

[0116] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0117] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their grouping.

[0118] As used herein, the term "if" may be interpreted to mean "when the antecedent is true" or "when the antecedent is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" is interpreted to mean "upon determining that the antecedent is true" or "in response to determining" or "upon determining" that the antecedent is true or "when detecting that the antecedent is true" or "in response to detecting" that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At a device including an image sensor, a display, one or more processors, and a non-transitory memory: capturing an image of a physical environment using the image sensor; acquiring a depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein for a particular pixel at a particular pixel position representing a dynamic object in the physical environment, the depth map includes a particular depth corresponding to a distance between the image sensor and a static object in the physical environment behind the dynamic object; transforming, using the one or more processors, the image of the physical environment based on the depth map; as well as The transformed image is displayed on the display. 2 . The method of claim 1 , wherein obtaining the depth map comprises determining the specific depth via interpolation using depths of locations surrounding the specific pixel location. 3 . The method according to claim 1 , wherein obtaining the depth map comprises determining the specific depth when the dynamic object is not represented at the specific pixel position. 4 . The method according to claim 1 , wherein acquiring the depth map comprises determining the specific depth based on a three-dimensional model of the physical environment excluding the dynamic object. The method according to claim 4 , further comprising generating the three-dimensional model.

6. The method of claim 5, wherein generating the three-dimensional model comprises adding one or more points to the three-dimensional model at one or more locations in the three-dimensional coordinate system of the physical environment corresponding to one or more static objects in the physical environment. 7 . The method of claim 6 , wherein adding the one or more points to the three-dimensional model comprises determining that the one or more points correspond to the one or more static objects in the physical environment. 8 . The method of claim 7 , wherein determining that a particular point of the one or more points corresponds to the one or more static objects in the physical environment comprises performing semantic segmentation on one or more images.

9. The method of claim 7 or 8, wherein determining that a particular point among the one or more points corresponds to the one or more static objects in the physical environment comprises detecting the particular point at the same location at least a threshold number of times over a period of time.

10. The method according to any one of claims 7 to 9, wherein determining that a specific point among the one or more points corresponds to the one or more static objects in the physical environment includes determining that surrounding points correspond to the one or more static objects in the physical environment.

11. The method according to any one of claims 4 to 10, wherein determining the specific depth based on a three-dimensional model comprises rasterizing the three-dimensional model. 12 . The method according to claim 4 , wherein determining the specific depth based on the three-dimensional model comprises performing ray tracing based on the three-dimensional model.

13. The method of any one of claims 1 to 12, wherein transforming the image of the physical environment based on the depth map further comprises smoothing the depth map.

14. The method of any one of claims 1 to 13, wherein transforming the image of the physical environment based on the depth map further comprises clamping the depth map.

15. A device, comprising: Image sensor; monitor; non-transitory memory; and One or more processors configured to: capturing an image of a physical environment using the image sensor; Acquire a three-dimensional model of the physical environment; acquiring, based on the three-dimensional model, a depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment; transforming, using the one or more processors, the image of the physical environment based on the depth map; as well as The transformed image is displayed on the display.

16. The apparatus of claim 15, wherein the three-dimensional model comprises a three-dimensional mesh.

17. An apparatus according to claim 15 or 16, wherein the three-dimensional model is based on objects in the physical environment that are determined to be static.

18. A non-transitory computer readable memory having encoded thereon instructions that, when executed by one or more processors of a device including an image sensor and a display, cause the device to: capturing an image of a physical environment using the image sensor; acquiring a temporally stable depth map, the temporally stable depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the temporally stable depth map does not include a depth based on a distance between the image sensor and a dynamic object; transforming, using the one or more processors, the image of the physical environment based on the temporally stable depth map; and The transformed image is displayed on the display.

19. A device, comprising: one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods of claims 1 to 14.

20. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device, cause the device to perform any of the methods of claims 1 to 14.

21. A device, comprising: one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods according to claims 1 to 14.

Images