Method and apparatus for refining depth map
By capturing and transforming the depth map of physical environment images in HMD, the user experience problems caused by different device locations are solved, distance perception and hand-eye coordination are improved, and the authenticity of the XR experience is enhanced.
Patent Information
- Application Number
- CN202410692923.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In head mounted devices (HMDs), due to the different positions of the eyes, displays, and cameras in space, the physical environment images seen by the user are not in line with reality, resulting in poor distance perception, disorders in orientation and poor hand-eye coordination.
Capture the physical environment image through the image sensor, acquire multiple depth maps, adjust the depth map based on the control signal, generate a transformed image, make it look like it was captured from the position of the user's eyes, transform the image using the processor, and display it on the display .
Improves user distance perception and hand-eye coordination, reduces artifacts and time artifacts, and improves the authenticity of the XR experience.
Smart Images

Figure CN119071460A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to systems, methods, and devices for performing perspective correction. Background Art
[0002] In various embodiments, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures an image of the physical environment (e.g., a scene) in which the user is located and a display that displays the image to the user. In some cases, the image or a portion thereof may be combined with one or more virtual objects to present an XR experience to the user. In other cases, the HMD may operate in a pass-through mode in which the image or a portion thereof is presented to the user without adding virtual objects. Ideally, the image of the physical environment presented to the user is substantially similar to what the user would see if the HMD were not present. However, since the eyes, display, and camera are in different positions in space, this may not occur, resulting in user discomfort. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] So that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be had with reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.
[0004] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.
[0005] Figure 2 Example scenarios related to capturing images of a physical environment and displaying the captured images are shown according to some implementations.
[0006] Figure 3 An image of the physical environment captured by an image sensor from a specific perspective.
[0007] Figure 4 yes Figure 3 A top-down perspective view of the physical environment.
[0008] Figure 5A shows what the user's left eye sees at the first time when the user is not wearing the HMD Figure 3 A first view of the physical environment.
[0009] Figure 5B shows the first time captured by the left image sensor of the HMD Figure 3 A first image of the physical environment.
[0010] Fig. 6A Shows Figure 5B The first depth map of the first image is a center row of the first depth map.
[0011] Figure 6B Shows Figure 5B The first static depth map of the first image is a center row of the first static depth map.
[0012] Fig. 7A Shown based on Figure 5B The first image and Fig. 6A The first depth map is a first transformation of the first image.
[0013] Figure 7B Shown based on Figure 5B The first image and Figure 6B A second transformation of the static depth map of the first image.
[0014] FIG. 8A to FIG. 8C , Fig. 8E and Figure 8F is a block diagram of an exemplary perspective correction architecture according to some specific implementations.
[0015] Fig.8D is a block diagram of an exemplary edge analytics engine according to some specific implementations.
[0016] 9A to 9F is a block diagram of an exemplary perspective correction architecture according to some specific implementations.
[0017] Fig.10 is a flowchart representation of a method for performing perspective correction according to some specific implementations.
[0018] Fig.11 is a flowchart representation of another method for performing perspective correction according to some specific implementations.
[0019] Fig.12 is a block diagram of an example controller according to some implementations.
[0020] Fig.13 is a block diagram of an exemplary electronic device according to some implementations.
[0021] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention
[0022] Various embodiments disclosed herein include devices, systems, and methods for performing perspective correction. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of a physical environment using the image sensor; acquiring a first depth map including multiple depths associated with multiple pixels of the image of the physical environment, respectively; adjusting one or more portions of the first depth map based on a control signal associated with the image of the physical environment to generate a second depth map; using one or more processors to transform the image of the physical environment based on the second depth map; and displaying the transformed image via the display.
[0023] Various embodiments disclosed herein include devices, systems, and methods for performing perspective correction. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of a physical environment using the image sensor; acquiring a first depth map including multiple depths associated with multiple pixels of the image of the physical environment, respectively; selecting one or more portions of the first depth map based on a control signal associated with the image of the physical environment; generating a second depth map by refining one or more portions of the first depth map; using one or more processors to transform the image of the physical environment based on the second depth map; and displaying the transformed image via the display.
[0024] According to some specific implementations, a device includes a display, an image sensor, one or more processors, a non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of a device with a display and an image sensor, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: a display, an image sensor, one or more processors, a non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION
[0025] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.
[0026] As described above, in an HMD with a display and a scene camera, if the HMD does not exist due to the different positions of the eyes, display, and camera in space, the image of the physical environment presented to the user on the display may not always reflect what the user sees. In various cases, this leads to, for example, poor distance perception, disorientation of the user, and poor hand-eye coordination when interacting with the physical environment. Therefore, in various specific implementations, the images from the scene camera are transformed so that they appear to be captured at or approximately near the location of the user's eyes using a depth map that represents, for each pixel of the image, the distance from the camera to the object represented by the pixel. In various specific implementations, the images from the scene camera are partially transformed so that they appear to be captured at a location that is closer to the location of the user's eyes in one or more dimensions than the location of the scene camera.
[0027] In various implementations, the depth map is changed to reduce artifacts. For example, in various implementations, the depth map is smoothed to avoid holes in the transformed image. In various implementations, the depth map is clamped to reduce large movements of pixels during the transformation. In various implementations, the depth map has temporal stability, such as remaining static, so that dynamic objects are substantially excluded from the depth map. For example, in various implementations, the depth map value at the location of the dynamic object is determined by interpolating the depth map using the locations around the location of the dynamic object. In various implementations, the depth map value at the location of the dynamic object is determined based on the depth map value determined when the dynamic object is not at the location. In various implementations, the depth map is determined using a three-dimensional model of the physical environment without dynamic objects. Using a static depth map (sometimes also referred to as a "temporally stable depth map" in this article) may increase spatial artifacts, such as objects not being displayed at their true locations. However, using a static depth map can reduce temporal artifacts, such as flicker.
[0028] In various implementations, at least some portions of the static depth map may be further refined to improve the resulting transformed image. As one example, one or more portions of the static depth map may be regularized or adjusted based on specific luminous intensities, colors, or textures (or changes / transitions thereof) within the image from the scene camera. As another example, one or more portions of the static depth map associated with a particular object may be regularized or amplified. As another example, one or more portions of the static depth map associated with a corner or edge may be regularized or amplified. As yet another example, one or more portions of the static depth map associated with a corner or edge of a particular object may be regularized or amplified. Similarly, in some implementations, at least some portions of the warped mesh used to transform the image from the scene camera may be further refined to improve the resulting transformed image. Thus, the resulting transformed image may be improved while saving power and computing resources by refining selected portions of the static depth map that are most noticeable to the user (such as specific objects, edges, luminous intensity / color / texture transitions, etc.).
[0029] In various embodiments, the depth map is refined to improve the resulting transformed image. As an example, one or more portions of the depth map may be regularized or adjusted based on specific luminous intensities, colors, or textures (or changes / transitions thereof) within the image from the scene camera. As another example, one or more portions of the depth map associated with a particular object may be regularized or amplified. As another example, one or more portions of the depth map associated with a corner or edge may be regularized or amplified. As yet another example, one or more portions of the depth map associated with a corner or edge of a particular object may be regularized or amplified. Thus, the resulting transformed image may be improved while saving power and computing resources by refining selected portions of the depth map that are most easily noticed by the user (such as specific objects, edges, luminous intensity / color / texture transitions, etc.).
[0030] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.
[0031] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Fig.12Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by electronic device 120 and / or is combined with the electronic device.
[0032] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. Reference is made below. Fig.13 The electronic device 120 is described in more detail. According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105.
[0033] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device 120 includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or tablet) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.
[0034] Figure 2 An exemplary scenario 200 is shown involving capturing an image of an environment and displaying the captured image according to some implementations. A user wears a device (e.g., Figure 1 The image sensor 230 captures an image of the physical environment, and the display 210 displays the image of the physical environment to the user's eye 220. The image sensor 230 has a viewing angle that is vertically offset from the user's viewing angle (e.g., where the user's eye 220 is located) by a vertical offset 241. Additionally, the viewing angle of the image sensor 230 is longitudinally offset from the user's viewing angle by a longitudinal offset 242. Additionally, in various implementations, the viewing angle of the image sensor 230 is laterally offset from the user's viewing angle by a lateral offset (e.g., toward Figure 2 within or offset outside of the page).
[0035] Figure 3 is an image 300 of a physical environment 301 captured by an image sensor from a specific viewing angle. The physical environment 301 includes a structure 310 having a first surface 311 closer to the image sensor, a second surface 312 farther from the image sensor, and a third surface 313 connecting the first surface 311 and the second surface 312. The first surface 311 has letters A, B, and C painted thereon, the third surface 313 has letters D painted thereon, and the second surface 312 has letters E, F, and G painted thereon.
[0036] From a particular viewing angle, image 300 includes all of the letters painted on structure 310. However, from other viewing angles, as described below, the captured image may not include all of the letters painted on structure 310.
[0037] Figure 4 yes Figure 3 4 is a top-down perspective view of a physical environment 301 of FIG. 4. The physical environment 301 includes a structure 310 and a user 410 wearing an HMD 420. The user 410 has a left eye 411a at a left eye position, thereby providing a left eye perspective. The user 410 has a right eye 411b at a right eye position, thereby providing a right eye perspective. The HMD 420 includes a left image sensor 421a at a left image sensor position that provides a left image sensor perspective. The HMD 420 includes a right image sensor 421b at a right image sensor position that provides a right image sensor perspective. Since the left eye 411a of the user 410 and the left image sensor 421a of the HMD 420 are at different positions, they each provide a different perspective of the physical environment.
[0038] Figure 5A A first view 501 of the physical environment 301 seen by the left eye 411a of the user 410 at the first time when the user 410 is not wearing the HMD 420 is shown. In the first view 501, there are the first surface 311 and the second surface 312, but there is no third surface 313. On the first surface 311, the letters B and C can be seen, while the letter A is not in the field of view of the left eye 411a. Similarly, on the second surface 312, the letters E, F, and G can be seen. The view 501 also includes a person 599 in front of the first surface 311 between the letters B and C.
[0039] Figure 5B A first image 502 of the physical environment 301 captured by the left image sensor 421a at a first time is shown. As with the first view 501, in the first image 502, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are at least partially visible, while the letter A is not in the field of view of the left image sensor 421a. Similarly, on the second surface 312, the letters F and G are visible, while the letter E is not in the field of view of the left image sensor 421a. It is worth noting that in the first image 502, compared to the first view 501, the letter E is not present on the second surface 312. Therefore, the letter E is in the field of view of the left eye 411a, but not in the field of view of the left image sensor 421a. The first image 502 also includes a person 599 in front of the first surface 311 that partially obscures the letter C.
[0040] In various implementations, the HMD 420 transforms the first image 502 so that it appears as if it is captured from a left eye perspective rather than from a left image sensor perspective, for example, it appears as if the first view 501. In various implementations, the transformation is a perspective transformation. In various implementations, the HMD 420 transforms the first image 502 based on a depth value associated with the first image 502 and a difference between the left image sensor perspective and the left eye perspective. In various implementations, the difference between the left image sensor perspective and the left eye perspective is determined during a calibration procedure. In various implementations, the depth value of a pixel of the first image 502 represents a distance from the left image sensor 421a to an object represented by the pixel in the physical environment 301. In various implementations, the depth value is used to generate a depth map that includes a corresponding depth value for each pixel of the first image 502.
[0041] In various implementations, the resulting transformed image includes holes, such as pixel positions of the transformed image where there are no corresponding pixel positions of the first image 502. Such holes may be filled via interpolation or using additional images, such as using another image from a different perspective (e.g., from the right image sensor 421b or from the left image sensor 421a at a different time).
[0042] In various implementations, the resulting transformed image includes blur, such as pixel locations of the transformed image that exist at multiple corresponding pixel locations of the first image 502. An averaging or consensus algorithm may be used to remove such blur.
[0043] In various implementations, the depth map does not include dynamic objects and / or only includes static objects. In various implementations, the depth map does not include movable objects and / or only includes fixed objects. In various implementations, the depth map does not include temporary objects and / or only includes permanent objects. Therefore, in various implementations, the depth map does not include personnel 599. For each pixel position representing personnel 599 in the first image 502, a depth value is determined by ignoring the distance from the left image sensor 421a to the personnel 599. Instead, the depth value represents the distance from the left image sensor 421a to the static object behind the personnel 599. In various implementations, the depth value is determined by interpolating the depth values of the pixels around the pixel position representing the personnel 599. In various implementations, the depth value is determined based on the depth value of the pixel position when the personnel 599 is not at the pixel position. In various implementations, the depth value is determined using a three-dimensional model of the physical environment 301 that does not include personnel 599.
[0044] For example, in various specific implementations, the HMD 420 generates a three-dimensional model including a three-dimensional mesh. The three-dimensional mesh includes a set of triangles connected by their common edges and / or vertices. Each vertex is located at a three-dimensional position in the three-dimensional coordinate system of the physical environment 301. In order to generate the three-dimensional model, the HMD 420 uses a depth sensor, stereo matching, or any other method to determine the three-dimensional position of the vertex. However, in various specific implementations, the vertex is added only when it is determined that the corresponding object is a static object. For example, when using a depth sensor, the vertex can only be added if the number of times the vertex is detected in a sufficiently long period of time reaches at least a threshold number of times. As another example, when using stereo matching, if the pixel in the stereo image corresponds to a static object in the image determined via semantic segmentation, the vertex can be added. For example, in a first image of the physical environment and a second image of the physical environment, a neural network can be used to classify each pixel as one of a plurality of object types. For pixels classified as walls or tables in the first image and corresponding pixels with similar classifications in the second image, the vertices corresponding to the pixels can be added to the three-dimensional model. For pixels in the first image that are classified as a person, hand, dog, or vehicle and corresponding pixels in the second image that have similar classifications, vertices corresponding to the pixels may be added to the three-dimensional model.
[0045] Using a static depth map may increase spatial artifacts, such as person 599 not being displayed at their true location in physical environment 301. However, using a static depth map may reduce temporal artifacts, such as flickering or distortion. In addition, another advantage of using a three-dimensional model to generate a depth map is that a static depth map can be rendered from any viewpoint where the room geometry is visible.
[0046] Fig. 6A A first depth map 600 of a center row of a first depth map for a first image 502 is shown. The first depth map 600 includes a left first portion 601A corresponding to distances between the left image sensor 421A and points on the first surface 311 to the left of the person 599, and a right first portion 601B corresponding to distances between the left image sensor 421A and points on the first surface 311 to the right of the person 599. The first depth map 600 includes a second portion 602 corresponding to distances between the left image sensor 421A and points on the second surface 312. The first depth map 600 includes a third portion 603 corresponding to distances between the left image sensor 421A and points on the person 599.
[0047] Figure 6BThe first static depth map 610 of the center row of the first static depth map of the first image 502 is shown. In various implementations, the first static depth map is generated by rasterizing the three-dimensional model or performing ray tracing based on the three-dimensional model. Therefore, each pixel in the static depth map indicates the distance from the left image sensor 421A to the static object in the physical environment 301 through the pixel of the first image 502 in the camera plane.
[0048] The first static depth map 610 includes a first portion 611 corresponding to the distances between the left image sensor 421A and various points on the first surface 311, and a second portion 602 corresponding to the distances between the left image sensor 421A and various points on the second surface 312. It is worth noting that the first static depth map 610 does not include a third portion 603 corresponding to the distances between the left image sensor 421A and various points on the person 599 because the person 599 is not a static object.
[0049] Fig. 7A A first transformed first image 701 generated by transforming the first image 502 based on the first depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective is shown.
[0050] In the first transformed first image 701, the first surface 311 and the second surface 312 exist, but the third surface 313 does not exist. On the first surface 311, the letter B and the letter C can be seen. Similarly, on the second surface 312, the letter E, the letter F, and the letter G can be seen. The first transformed first image 701 also includes the person 599 in front of the first surface 311 between the letter B and the letter C. Therefore, the first transformed first image 701 is substantially the same as the first view 501. Although the left half of the letter C and the entire letter E are not visible in the first image 502, the left half of the letter C and the entire letter E can be seen in the first transformed first image 701 by using other images for hole filling (rather than interpolation).
[0051] Figure 7B A second transformed first image 702 generated by transforming the first image 502 based on the first static depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective is shown.
[0052] In the second transformed first image 701, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are at least partially visible. Similarly, on the second surface 312, the letters E, F, and G are visible. The second transformed first image 701 also includes a person 599 in front of the first surface 311 that partially occludes the letter C. Therefore, although the first surface 311 and the second surface 312 have been correctly transformed and the letters have the same size and position as in the first view 501, the person 599 is not between the letters B and C (as shown in the first view 501), but partially occludes the letter C (as shown in the first image 502). The transformation based on the first static depth map regards the person 599 as being located at the depth of the first surface 311 (e.g., painted on the first surface 311), rather than being located in front of the first surface 311. Therefore, the portion of the first image 502 corresponding to the person 599 is mapped to the same position as the portion of the first image corresponding to the letter C. Although the letter E is not visible in the first image 502 , the letter E can be seen in the second transformed first image 702 by using other images for hole filling (rather than interpolation).
[0053] Fig. 8A 8 is a block diagram of an exemplary perspective correction architecture 800A according to some implementations. Although some specific features are shown, those skilled in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 1 and Fig.12 The controller 110, Figure 1 and Fig.13 The electronic device 120 or a suitable combination thereof includes the perspective correction architecture 800A or provides its functionality.
[0054] like Fig. 8A As shown, the image sensor 801A captures an unprocessed image 803 associated with the physical environment 105, the depth sensor 801B captures depth values associated with the physical environment 105, and the depth map generator 808 uses these depth values to generate an unprocessed depth map 805 including a corresponding depth value for each pixel in the unprocessed image 803.
[0055] exist Fig. 8A 8, image buffer 802A stores a plurality of unprocessed images captured by image sensor 801A at a plurality of different times and at a plurality of different spatial perspectives of physical environment 105. Similarly, depth buffer 802B stores a plurality of unprocessed depth maps from a plurality of different times and at a plurality of different spatial perspectives of physical environment 105.
[0056] like Fig. 8AAs shown, the scene reconstruction engine 812 generates a three-dimensional model of the physical environment 105 based on the image buffer 802A and the depth buffer 802B. As described above, the three-dimensional model may correspond to a three-dimensional mesh.
[0057] In various implementations, generating the three-dimensional model includes adding one or more points to the three-dimensional model at one or more locations in the three-dimensional coordinate system of the physical environment 105 corresponding to one or more static objects in the physical environment 105. In various implementations, the one or more points are vertices of mesh triangles. In various implementations, adding the one or more points to the three-dimensional model includes determining that the one or more points correspond to the one or more static objects in the physical environment (and / or adding the points after determining this).
[0058] In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes performing semantic segmentation on one or more images. For example, pixels classified as a table or a wall may form part of a key point pair in stereo matching to generate a point to be added to a three-dimensional model, while pixels classified as a person or a vehicle may not form part of a key point pair in stereo matching to generate a point to be added to a three-dimensional model. In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes detecting the particular point at the same location at least a threshold number of times over a period of time. For example, if a feature is detected at the same location in the physical environment within the next two minutes or more or two seconds or more, then adding the point determined to correspond to the static object at that location. In various embodiments, determining that a particular point in the one or more points corresponds to one or more static objects includes determining that surrounding points correspond to one or more static objects. For example, if points of a wall are detected at three non-linear coplanar locations, points on a specified plane between these points may be determined to be points corresponding to one or more static objects.
[0059] Thus, in various implementations, the three-dimensional model is based on historical data acquired at one or more times prior to capturing the unprocessed image 803 of the physical environment 105. For example, the three-dimensional model may be based on data acquired one second, several seconds, one minute, several minutes, one hour, one day, or any other time prior to capturing the unprocessed image 803 of the physical environment 105.
[0060] exist Fig. 8A8, the depth map processor 814 generates a processed depth map 815 based on the three-dimensional model of the physical environment 105 generated by the scene reconstruction engine 812 (and optionally based on the unprocessed depth map 805). In some implementations, the depth map processor 814 smoothes the processed depth map 815 to avoid holes in the transformed image 823. In various implementations, the depth map processor 814 clamps objects (e.g., depth map values) within the processed depth map 815 to a maximum depth in order to reduce large movements of pixels during transformation.
[0061] In various implementations, the depth map handler 814 also causes the processed depth map 815 to remain static (e.g., based on the three-dimensional model of the physical environment 105) and substantially excludes dynamic objects from the processed depth map 815. For example, in various implementations, depth map values associated with the processed depth map 815 at the location of the dynamic object are determined by interpolating the processed depth map 815 using locations surrounding the location of the dynamic object. In various implementations, depth map values associated with the processed depth map 815 at the location of the dynamic object are determined based on depth map values determined when the dynamic object is not at the location (e.g., based on the depth buffer 802B or the three-dimensional model of the physical environment 105). In various implementations, at least some of the depth map values associated with the processed depth map 815 are determined using the three-dimensional model of the physical environment 105 without the dynamic object.
[0062] In various implementations, the processed depth map 815 includes a dense depth map that represents, for each pixel of the image, an estimated distance between the image sensor 801A and a static object represented by the pixel or a static object behind a dynamic object represented by the pixel. In various implementations, the processed depth map 815 includes a sparse depth map that represents, for each subset of pixels of the image, an estimated distance between the image sensor 801A and a static object represented by the pixel or a static object behind a dynamic object represented by the pixel. In various implementations, the perspective correction architecture 800A generates the sparse depth map from the dense depth map by sampling the dense depth map, for example, selecting a single pixel in every N×N pixel block.
[0063] In various implementations, the perspective correction architecture 800A obtains at least one depth map value from the depth sensor 801B among a plurality of depth map values associated with the processed depth map 815. In various implementations, the perspective correction architecture 800A uses stereo matching (e.g., using a depth map obtained by Figure 4 The image of the physical environment captured by the left scene camera 421a in Figure 4The processing unit 800A may further include a perspective correction framework 800A and a perspective correction framework 800A. The perspective correction framework ...
[0064] like Fig. 8A As shown, the transformation engine 822 transforms the unprocessed image 803 based on the processed depth map 815 so that the resulting transformed image 823 appears to be captured at or substantially near the location of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0065] Figure 8B 800B according to some implementations. Although some specific features are shown, those skilled in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 8B The perspective correction architecture 800B in is similar to and adapted from Fig. 8A To this end, Fig. 8A and Figure 8B Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0066] like Figure 8B As shown, the depth map refinement engine 842 refines and / or regularizes one or more portions of the processed depth map 815 based on the control signal 841 and the unprocessed image 803 to generate a refined depth map 843A. For example, the depth map refinement engine 842 increases the detail level and / or resolution of one or more portions of the processed depth map 815 based on the control signal 841 and the unprocessed image 803.
[0067] According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the processed depth map 815 based on a specific luminous intensity (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the processed depth map 815 based on a specific color (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the processed depth map 815 based on a specific texture (or a change / transition thereof) within the unprocessed image 803. As an example, one or more portions of the processed depth map 815 may correspond to one or more corners and / or edges within the unprocessed image 803. As another example, one or more portions of the processed depth map 815 may correspond to one or more specific objects within the unprocessed image 803. As yet another example, one or more portions of the processed depth map 815 may correspond to one or more corners and / or edges of one or more specific objects within the unprocessed image 803.
[0068] According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the processed depth map 815 associated with corners and / or edges within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the processed depth map 815 associated with specific objects within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the processed depth map 815 associated with corners and / or edges of specific objects within the unprocessed image 803.
[0069] In some implementations, the depth map refinement engine 842 can dynamically switch between refinement types based on an updated control signal, etc. For example, the depth map refinement engine 842 can switch from refining one or more portions of a processed depth map by making adjustments based on texture in an unprocessed image (e.g., using a first guided filter or a joint bilateral filter) to refining one or more portions of a processed depth map by making adjustments based on color in the unprocessed image (e.g., using a second guided filter or a joint bilateral filter). As another example, the depth map refinement engine 842 can switch from refining one or more portions of a processed depth map by making adjustments based on color to semantic refinement or edge refinement when a particular object (e.g., a monitor, a television, a phone, a laptop, etc.) is detected.
[0070] In some specific implementations, the depth map refinement engine 842 corresponds to a guided filter, a joint bilateral filter, a machine learning module, etc. For example, the machine learning module corresponds to a neural network, a convolutional neural network, a deep neural network, a recursive neural network, a state vector machine, etc.
[0071] In various implementations, the optional pre-processing engine 832 performs one or more pre-processing operations on the processed depth map 815 prior to the above-described operations (such as smoothing, clamping, etc.) performed by the depth map refinement engine 842 .
[0072] like Figure 8B As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 843A so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0073] Figure 8C 800C according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 8C The perspective correction architecture in 800C is similar to and adapted from Fig. 8A The perspective correction architecture in 800A and Figure 8B To this end, Fig. 8A , Figure 8B and Figure 8C Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0074] like Figure 8C As shown, the edge analysis engine 844 determines a temporally stable edge 845 associated with the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. Fig.8D The edge analytics engine 844 is described in more detail.
[0075] Fig.8D is a block diagram of an edge analytics engine 844 according to some specific implementations. Fig.8D As shown, the edge detection module 852 generates edge points 853 (e.g., three-dimensional coordinates relative to the physical environment 105) for edges within the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. The point refinement module 854 uses a stereo snake algorithm (e.g., relative to the three-dimensional coordinates from Figure 4 The left image sensor 421a and Figure 4The method further comprises: arranging the edge points 853 to generate edge segments 857 based on the refined edge points 855. The edge filtering module 858 discards at least some edge segments 857 based on one or more filters and / or visibility indicators to generate filtered edge segments 859. The parameter fitting module 862 generates a fitting model 863 for each filtered edge segment 859 using a parameter fitting technique. The tracking module 864 generates a temporally stable edge 845 by tracking the fitting model 863 across time and across spatial changes in the perspective of the physical environment 105 and / or spatial changes in objects within the physical environment 105. A person of ordinary skill in the art will appreciate that Fig.8D The illustrated edge analysis engine 844 is an example that can be altered in numerous ways, such as removing or replacing the tracking module 864, the edge filtering module 858, and / or the point refinement module 854 in various other implementations.
[0076] like Figure 8C As shown, the depth map refinement engine 842 generates a refined depth map 843B by regularizing one or more portions of the processed depth map 815 corresponding to the temporally stable edge 845. In various implementations, the depth map refinement engine 842 enlarges one or more portions of the processed depth map 815 associated with the temporally stable edge 845 based on the unprocessed image 803 and the unprocessed depth map 805.
[0077] According to some implementations, the depth map refinement engine 842 adjusts one or more portions of the processed depth map 815 not associated with the temporally stable edge 845 based on luminous intensity, color, texture, etc. associated with the unprocessed image 803 (e.g., using a joint bilateral filter).
[0078] According to some implementations, the depth map refinement engine 842 may be removed, and instead, the transformation engine may include a warp mesh refinement sub-engine that refines at least some portions of the warp mesh used to transform the unprocessed image 803 based on the temporally stable edges 845 to improve the transformed image 823. A person of ordinary skill in the art will appreciate that the following references to Fig. 8E The described correction architecture 800D is similarly modified to refine at least some portions of the warping grid used to transform the unprocessed image 803 based on the filtered pixel mask from the semantic filter 884, thereby improving the transformed image 823.
[0079] like Figure 8CAs shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 843B so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0080] Fig. 8E 800D according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig. 8E The perspective correction architecture in the 800D is similar to and adapted from Fig. 8A The perspective correction architecture in 800A and Figure 8B To this end, Fig. 8A , Figure 8B and Fig. 8E Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0081] like Fig. 8E As shown, the semantic analysis engine 882 determines relevant semantic information 883 within the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. According to some specific implementations, the semantic analysis engine 882 performs semantic segmentation. In some specific implementations, the semantic analysis engine 882 includes a semantic classifier or machine learning module trained for semantic separation and recognition of objects. In various specific implementations, the semantic information 883 includes a semantic label and a pixel mask for each object identified in the physical environment 105.
[0082] The semantic filter 884 discards semantic information 883 associated with unwanted objects. Therefore, the depth map refinement engine 842 generates a refined depth map 843C by regularizing one or more portions of the processed depth map 815 corresponding to a specific object (e.g., a filtered pixel mask). In various implementations, the depth map refinement engine 842 enlarges one or more portions of the processed depth map 815 associated with a specific object based on the unprocessed image 803 and the unprocessed depth map 805.
[0083] According to some implementations, the depth map refinement engine 842 adjusts one or more portions of the processed depth map 815 that are not associated with a particular object based on luminous intensity, color, texture, etc. associated with the unprocessed image 803 (eg, using a joint bilateral filter).
[0084] like Fig. 8EAs shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 843C so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0085] Figure 8F 800E is a block diagram of an exemplary perspective correction architecture 800E according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 8F The perspective correction architecture in 800E is similar to and adapted from Fig. 8A The perspective correction architecture 800A in Figure 8B The perspective correction architecture 800B and Fig. 8E To this end, Fig. 8A , Figure 8B , Fig. 8E and Figure 8F Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0086] like Figure 8F As shown, the semantic separator 886 generates a semantically separated depth map 887 that separates specific objects within the unprocessed depth map 805 based on the filtered pixel mask from the semantic filter 884. Therefore, the semantic separator 886 separates some parts of the unprocessed depth map 805 based on the filtered pixel mask from the semantic filter 884. Figure 8F , the depth map blending engine 872 generates a blended depth map 873 by blending the semantically separated depth map 887 with the processed depth map 815. In some implementations, when a conflict occurs between the semantically separated depth map 887 and the processed depth map 815, the depth map blending engine 872 uses the depth map value from the semantically separated depth map 887. The depth map refinement engine 842 refines the blended depth map 873 based on the control signal 841 to generate a refined depth map 843D.
[0087] According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the blended depth map 873 based on a particular luminous intensity (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the blended depth map 873 based on a particular color (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the blended depth map 873 based on a particular texture (or a change / transition thereof) within the unprocessed image 803.
[0088] like Figure 8F As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 843D so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0089] Fig.9A 9 is a block diagram of an exemplary perspective correction architecture 900A according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig.9A The perspective correction architecture 900A in is similar to and adapted from Fig. 8A To this end, Fig. 8A and Fig.9A Similar reference numerals are used in .
[0090] like Fig.9A As shown, the transformation engine 822 transforms the unprocessed image 803 based on the unprocessed depth map 805 so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0091] Fig. 9B 900B is a block diagram of an exemplary perspective correction architecture 900B according to some implementations. Although some specific features are shown, those skilled in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig. 9B The perspective correction architecture 900B in is similar to and adapted from Figure 8B The perspective correction architecture in 800B and Fig.9ATo this end, Fig.9A and Fig. 9B Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0092] like Fig. 9B As shown, the depth map refinement engine 842 refines and / or normalizes one or more portions of the unprocessed depth map 805 based on the control signal 841 and the unprocessed image 803 to generate a refined depth map 915A. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the unprocessed depth map 805 based on a specific luminous intensity (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the unprocessed depth map 805 based on a specific color (or a change / transition thereof) within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to adjust one or more portions of the unprocessed depth map 805 based on a specific texture (or a change / transition thereof) within the unprocessed image 803.
[0093] As one example, one or more portions of the unprocessed depth map 805 may correspond to one or more corners and / or edges within the unprocessed image 803. As another example, one or more portions of the unprocessed depth map 805 may correspond to one or more specific objects within the unprocessed image 803. As yet another example, one or more portions of the unprocessed depth map 805 may correspond to one or more corners and / or edges of one or more specific objects within the unprocessed image 803.
[0094] According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the unprocessed depth map 805 associated with corners and / or edges within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the unprocessed depth map 805 associated with specific objects within the unprocessed image 803. According to some implementations, the control signal 841 may instruct the depth map refinement engine 842 to regularize one or more portions of the unprocessed depth map 805 associated with corners and / or edges of specific objects within the unprocessed image 803.
[0095] In some specific implementations, the depth map refinement engine 842 corresponds to a guided filter, a joint bilateral filter, a machine learning module, etc. For example, the machine learning module corresponds to a neural network, a convolutional neural network, a deep neural network, a recursive neural network, a state vector machine, etc.
[0096] like Fig. 9BAs shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 915A so that the resulting transformed image 823 appears to be captured at or substantially near the location of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0097] Fig. 9C 900C according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig. 9C The perspective correction architecture 900C in is similar to and adapted from Figure 8C The perspective correction architecture in 800C and Fig. 9B To this end, Figure 8C , Fig. 9B and Fig. 9C Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0098] like Fig. 9C As shown, the edge analysis engine 844 determines a temporally stable edge 845 associated with the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. Fig.8D The edge analytics engine 844 is described in more detail.
[0099] like Fig. 9C As shown, the depth map refinement engine 842 generates a refined depth map 915B by regularizing one or more portions of the unprocessed depth map 805 corresponding to the temporally stable edge 845. In various implementations, the depth map refinement engine 842 enlarges one or more portions of the unprocessed depth map 805 associated with the temporally stable edge 845 based on the unprocessed image 803 and the unprocessed depth map 805.
[0100] like Fig. 9C As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 915B so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0101] Fig.9D900D according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig.9D The perspective correction architecture in the 900D is similar to and adapted from Fig. 8E The perspective correction architecture in the 800D and Fig. 9B To this end, Fig. 8E , Fig. 9B and Fig.9D Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0102] like Fig.9D As shown, the semantic analysis engine 882 determines semantic information 883 associated with the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. According to some specific implementations, the semantic analysis engine 882 performs semantic segmentation. In some specific implementations, the semantic analysis engine 882 includes a semantic classifier or machine learning module trained for semantic recognition of objects. In various specific implementations, the semantic information 883 includes a semantic label and a pixel mask for each object identified within the physical environment 105.
[0103] The semantic filter 884 discards the semantic information 883 associated with the unwanted objects. Therefore, the depth map refinement engine 842 generates a refined depth map 915C by regularizing one or more portions of the unprocessed depth map 805 corresponding to the specific object. In various implementations, the depth map refinement engine 842 enlarges one or more portions of the unprocessed depth map 805 associated with the specific object based on the unprocessed image 803 and the unprocessed depth map 805.
[0104] like Fig.9D As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 915C so that the resulting transformed image 823 appears to be captured at or substantially near the location of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0105] Fig.9E 9 is a block diagram of an exemplary perspective correction architecture 900E according to some implementations. Although some specific features are shown, those skilled in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig.9E The perspective correction architecture in 900E is similar to and adapted from Fig. 9CThe perspective correction architecture in 900C and Fig.9D To this end, FIG. 9C to FIG. 9E Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0106] like Fig.9E As shown, the semantic filter 884 discards the semantic information 883 associated with the unwanted objects. Therefore, the edge analysis engine 844 determines the temporally stable edge 845 associated with the specific object within the physical environment 105 based on the unprocessed image 803 and the unprocessed depth map 805. Fig.8D The edge analytics engine 844 is described in more detail.
[0107] like Fig.9E As shown, the depth map refinement engine 842 generates a refined depth map 915D by regularizing one or more portions of the unprocessed depth map 805 corresponding to the temporally stable edge 845 of the specific object. In various specific implementations, the depth map refinement engine 842 enlarges one or more portions of the unprocessed depth map 805 associated with the temporally stable edge 845 of the specific object based on the unprocessed image 803 and the unprocessed depth map 805.
[0108] like Fig.9E As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 915D so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0109] Fig.9F 900F according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Fig.9F The perspective correction architecture in 900F is similar to and adapted from Fig. 9B The perspective correction architecture 900B in FIG. Fig. 9B and Fig.9F Similar reference numerals are used and for the sake of brevity only the differences between them will be discussed.
[0110] like Fig.9F As shown, the grayscale converter 932 can convert the unprocessed image 803 into a grayscale image. Fig.9F, the depth map refinement engine 842 regularizes one or more portions of the raw depth map 805 based on luminous intensity values or transitions thereof in the grayscale image to generate a refined depth map 915E.
[0111] like Fig.9F As shown, the transformation engine 822 transforms the unprocessed image 803 based on the refined depth map 915E so that the resulting transformed image 823 appears to be captured at or substantially near the position of the user's eyes. In some specific implementations, the transformed image 823 can be composited with rendered content / graphics before being displayed to the user.
[0112] Fig.10 1 is a flowchart representation of a method for performing perspective correction of an image according to some specific implementations. In various specific implementations, method 1000 is performed by a device (e.g., Figure 1 and Fig.13 The electronic device 100, Figure 1 and Fig.12 In some implementations, the method 1000 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1000 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).
[0113] Method 1000 begins in block 1010 where a device captures an image of a physical environment using an image sensor.
[0114] Method 1000 continues at block 1020 where the device acquires a first depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment.
[0115] In some implementations, for a particular pixel at a particular pixel position representing a dynamic object in the physical environment, the first depth map includes a specific depth corresponding to a distance between the image sensor and a static object in the physical environment behind the dynamic object. In some implementations, acquiring the first depth map includes determining the specific depth by interpolating using depths at positions around the specific pixel position. In some implementations, acquiring the first depth map includes determining the specific depth when the dynamic object is not shown at the specific pixel position. In some implementations, acquiring the first depth map includes determining the specific depth based on a three-dimensional model of the physical environment. In some implementations, the three-dimensional model of the physical environment corresponds to a time-stable three-dimensional model of the physical environment. In some implementations, the three-dimensional model of the physical environment does not include the dynamic object. In various implementations, determining the specific depth based on the three-dimensional model includes rasterizing the three-dimensional model. In various implementations, determining the specific depth based on the three-dimensional model includes ray tracing based on the three-dimensional model.
[0116] In some implementations, the first depth map corresponds to a temporally stable depth map that includes a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the temporally stable depth map does not include a depth based on a distance between the image sensor and a dynamic object.
[0117] In some implementations, obtaining the first depth map includes obtaining a three-dimensional model of the physical environment and generating a first depth map based on the three-dimensional model, the first depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment. In various implementations, the three-dimensional model includes a three-dimensional mesh. In various implementations, the three-dimensional model is based on objects in the physical environment that are determined to be static. In some implementations, method 1000 also includes a device for generating the three-dimensional model, including: determining that one or more points correspond to one or more static objects in the physical environment; and adding the one or more points to the three-dimensional model at one or more locations in the three-dimensional coordinate system of the physical environment corresponding to the one or more static objects in the physical environment.
[0118] In some implementations, the method 1000 further includes smoothing the first depth map to fill one or more holes. In some implementations, the method 1000 further includes clamping depth map values associated with the first depth map to a predefined depth value.
[0119] The method 1000 continues at block 1030, where the device adjusts one or more portions of the first depth map based on a control signal associated with the image of the physical environment to generate a second depth map. In some implementations, the second depth map is derived from the first depth map having at least some updated and / or different depth map values. According to some implementations, when adjusting the one or more portions of the first depth map based on the control signal, at least some depth map values of the one or more portions of the first depth map are updated. In various implementations, the one or more portions of the first depth map are adjusted using a guided filter or a joint bilateral filter.
[0120] As an example, see the reference above Figure 8B 800B, where the depth map refinement engine 842 adjusts the processed depth map 815 based on the control signal 841 (e.g., color, texture, luminous intensity, etc.). As another example, see the above reference Fig. 9B 800B, where the depth map refinement engine 842 adjusts the raw depth map 805 based on a control signal 841 (e.g., color, texture, luminous intensity, etc. adjustments).
[0121] In some implementations, method 1000 further includes obtaining a control signal associated with texture adjustment, wherein generating the second depth map includes adjusting one or more portions of the first depth map based on at least one texture or texture transition within the image of the physical environment. In some implementations, method 1000 further includes obtaining a control signal associated with color adjustment, wherein generating the second depth map includes adjusting one or more portions of the first depth map based on at least one color or color transition within the image of the physical environment. In some implementations, method 1000 further includes obtaining a control signal associated with luminous intensity adjustment, wherein generating the second depth map includes adjusting one or more portions of the first depth map based on at least one luminous intensity or luminous intensity transition within the image of the physical environment.
[0122] In some implementations, the control signal indicates one or more specific objects within the image of the physical environment, and wherein one or more portions of the first depth map correspond to the one or more specific objects. In some implementations, the control signal indicates one or more edges within the image of the physical environment, and wherein one or more portions of the first depth map correspond to the one or more edges.
[0123] The method 1000 continues at block 1040 where the device transforms the image of the physical environment based on the second depth map using one or more processors. In various implementations, the device transforms the image of the physical environment at an image pixel level, an image tile level, or a combination thereof.
[0124] In various implementations, the device transforms the image of the physical environment based on a difference between a first perspective and a second perspective of the image sensor. In various implementations, the second perspective is a perspective of a user, such as a perspective of an eye of the user. In various implementations, the second perspective is a perspective at a location closer to an eye of the user in one or more directions.
[0125] In various implementations, the device performs a projective transformation based on the second depth map and a difference between a first perspective and a second perspective of the image sensor.
[0126] In various implementations, the projective transformation is a forward mapping, wherein for each pixel of the image of the physical environment at the pixel position in the untransformed space, a new pixel position is determined in the transformed space of the transformed image. In various implementations, the projective transformation is a reverse mapping, wherein for each pixel of the transformed image at the pixel position in the transformed space, a source pixel position is determined in the untransformed space of the image of the physical environment.
[0127] In various implementations, the source pixel position is determined according to the following equation, where x1 and y1 are the pixel positions in the untransformed space, x2 and y2 are the pixel positions in the transformed space, P2 is the 4×4 view projection matrix of the second perspective, P1 is the 4×4 view projection matrix of the first perspective of the image sensor, and d is the depth map value at the pixel position:
[0128]
[0129] In various implementations, for each pixel in the image of the physical environment, the source pixel position is determined using the above equation. In various implementations, for each pixel in an image that is smaller than the physical environment, the source pixel position is determined using the above equation.
[0130] In various implementations, the device determines a view projection matrix for the second perspective and a view projection matrix for the first perspective during calibration, and stores data indicating the view projection matrices (or their product) in a non-transitory memory. The product of the view projection matrices is a transformation matrix representing the difference between the first perspective and the second perspective of the image sensor.
[0131] Thus, in various implementations, transforming the image of the physical environment includes, for a plurality of pixels of the transformed image having corresponding pixel positions, determining a corresponding plurality of source pixel positions. In various implementations, determining the corresponding plurality of source pixel positions includes, for each of the plurality of pixels of the transformed image, multiplying a vector including the corresponding pixel position and a multiplicative inverse element of a corresponding element of the smoothed depth map by a transformation matrix representing a difference between a first perspective and a second perspective of the image sensor.
[0132] Using the pixel values of the pixels of the image of the physical environment and the source pixel positions in the untransformed space, the device generates a pixel value for each pixel position of the transformed image using interpolation or other techniques.
[0133] In various implementations, the device modifies the second depth map before using the second depth map to perform the transformation. For example, in various implementations, transforming the image of the physical environment based on the second depth map also includes smoothing the second depth map to fill holes. For example, in various implementations, the device applies a two-dimensional low-pass filter to the second depth map. As another example, in various implementations, when transforming the image of the physical environment based on the second depth map, it is also necessary to clamp the second depth map. For example, in various implementations, any values in the second depth map that are below a clamping threshold are replaced by the clamping threshold.
[0134] Method 1000 continues in box 1050, where the device displays the transformed image on the display. In various embodiments, the transformed image includes XR content. In some embodiments, the XR content is added to the current image of the physical environment before the transformation (in box 1040). In some embodiments, the XR content is added to the transformed image. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on metadata indicating the addition of the XR content to the physical environment. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on the amount of XR content (e.g., the percentage of the image of the physical environment that contains XR content).
[0135] In various implementations, the device determines whether to add the XR content to the image of the environment before or after the transformation based on metadata indicating the depth of the XR content. Accordingly, in various implementations, method 1000 includes receiving XR content and XR content metadata, selecting an image or a transformed image of the physical environment based on the XR content metadata, and adding the XR content to the selected image.
[0136] Fig.11 11 is a flowchart representation of a method for performing image perspective correction according to some specific implementations. In various specific implementations, method 1100 is performed by a device (e.g., Figure 1 and Fig.13 The electronic device 100, Figure 1 and Fig.12In some implementations, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1100 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).
[0137] The method 1100 begins in block 1110 where a device captures an image of a physical environment using an image sensor.
[0138] The method 1100 continues at block 1120 where the device acquires a first depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment.
[0139] In some implementations, for a particular pixel at a particular pixel position representing a dynamic object in the physical environment, the first depth map includes a specific depth corresponding to a distance between the image sensor and a static object in the physical environment behind the dynamic object. In some implementations, acquiring the first depth map includes determining the specific depth by interpolating using depths at positions around the specific pixel position. In some implementations, acquiring the first depth map includes determining the specific depth when the dynamic object is not shown at the specific pixel position. In some implementations, acquiring the first depth map includes determining the specific depth based on a three-dimensional model of the physical environment. In some implementations, the three-dimensional model of the physical environment corresponds to a time-stable three-dimensional model of the physical environment. In some implementations, the three-dimensional model of the physical environment does not include the dynamic object. In various implementations, determining the specific depth based on the three-dimensional model includes rasterizing the three-dimensional model. In various implementations, determining the specific depth based on the three-dimensional model includes ray tracing based on the three-dimensional model.
[0140] In some implementations, the first depth map corresponds to a temporally stable depth map that includes a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the temporally stable depth map does not include a depth based on a distance between the image sensor and a dynamic object.
[0141] In some implementations, obtaining a first depth map includes obtaining a three-dimensional model of the physical environment and generating a first depth map based on the three-dimensional model, the first depth map including a plurality of depths respectively associated with a plurality of pixels of an image of the physical environment. In various implementations, the three-dimensional model includes a three-dimensional mesh. In various implementations, the three-dimensional model is based on objects in the physical environment that are determined to be static. In some implementations, method 1100 also includes a device for generating the three-dimensional model, including: determining that one or more points correspond to one or more static objects in the physical environment; and adding one or more points to the three-dimensional model at one or more locations in the three-dimensional coordinate system of the physical environment corresponding to the one or more static objects in the physical environment.
[0142] In some implementations, the method 1100 further includes smoothing the first depth map to fill one or more holes. In some implementations, the method 1100 further includes clamping depth map values associated with the first depth map to a predefined depth value.
[0143] The method 1100 continues at block 1130, where the device selects one or more portions of the first depth map based on a control signal associated with the image of the physical environment. In some implementations, the control signal indicates one or more specific objects within the image of the physical environment, and where the one or more portions of the first depth map correspond to the one or more specific objects. In some implementations, the control signal indicates one or more edges within the image of the physical environment, and where the one or more portions of the first depth map correspond to the one or more edges. In some implementations, the control signal indicates one or more edges of the one or more specific objects within the image of the physical environment, and where the one or more portions of the first depth map correspond to the one or more edges of the one or more specific objects.
[0144] The method 1100 continues in block 1140, where the device generates a second depth map by refining one or more portions of the first depth map. In some implementations, the second depth map is derived from the first depth map having at least some updated and / or different depth map values. According to some implementations, when refining one or more portions of the first depth map, at least some depth map values of the one or more portions of the first depth map are updated or amplified. In various implementations, the one or more portions of the first depth map are refined using a guided filter, a joint bilateral filter, a machine learning module, and the like. For example, the machine learning module corresponds to a neural network, a convolutional neural network, a deep neural network, a recursive neural network, a state vector machine, and the like.
[0145] As an example, see the reference above Figure 8C800C, wherein the depth map refinement engine 842 regularizes one or more portions of the processed depth map 815 associated with the temporally stable edge 845. As another example, see the above reference Fig. 8E 800D, in which the depth map refinement engine 842 regularizes one or more portions of the processed depth map 815 associated with a particular object.
[0146] As an example, see the reference above Fig. 9C , wherein the depth map refinement engine 842 regularizes one or more portions of the unprocessed depth map 805 associated with the temporally stable edge 845. As another example, see the above reference Fig.9D 900D, where the depth map refinement engine 842 regularizes one or more portions of the processed depth map 815 associated with a particular object. As yet another example, see the above reference Fig.9E 8. Description of the perspective correction architecture 900E, in which the depth map refinement engine 842 regularizes one or more portions of the processed depth map 815 associated with the edges of particular objects.
[0147] In some implementations, one or more portions of the first depth map are refined by upscaling depth map values associated with the one or more portions of the first depth map. In some implementations, one or more portions of the first depth map are refined by normalizing depth map values associated with the one or more portions of the first depth map.
[0148] The method 1100 continues at block 1150 where the device, using one or more processors, transforms the image of the physical environment based on the second depth map. In various implementations, the device transforms the image of the physical environment at an image pixel level, an image tile level, or a combination thereof.
[0149] In various implementations, the device transforms the image of the physical environment based on a difference between a first perspective and a second perspective of the image sensor. In various implementations, the second perspective is a perspective of a user, such as a perspective of an eye of the user. In various implementations, the second perspective is a perspective at a location closer to an eye of the user in one or more directions.
[0150] In various implementations, the device performs a projective transformation based on the second depth map and a difference between a first perspective and a second perspective of the image sensor.
[0151] In various implementations, the projective transformation is a forward mapping, wherein for each pixel of the image of the physical environment at the pixel position in the untransformed space, a new pixel position is determined in the transformed space of the transformed image. In various implementations, the projective transformation is a reverse mapping, wherein for each pixel of the transformed image at the pixel position in the transformed space, a source pixel position is determined in the untransformed space of the image of the physical environment.
[0152] In various implementations, the source pixel position is determined according to the following equation, where x1 and y1 are the pixel positions in the untransformed space, x2 and y2 are the pixel positions in the transformed space, P2 is the 4×4 view projection matrix of the second perspective, P1 is the 4×4 view projection matrix of the first perspective of the image sensor, and d is the depth map value at the pixel position:
[0153]
[0154] In various implementations, for each pixel in the image of the physical environment, the source pixel position is determined using the above equation. In various implementations, for each pixel in an image that is smaller than the physical environment, the source pixel position is determined using the above equation.
[0155] In various implementations, the device determines a view projection matrix for the second perspective and a view projection matrix for the first perspective during calibration, and stores data indicating the view projection matrices (or their product) in a non-transitory memory. The product of the view projection matrices is a transformation matrix representing the difference between the first perspective and the second perspective of the image sensor.
[0156] Thus, in various implementations, transforming the image of the physical environment includes, for a plurality of pixels of the transformed image having corresponding pixel positions, determining a corresponding plurality of source pixel positions. In various implementations, determining the corresponding plurality of source pixel positions includes, for each of the plurality of pixels of the transformed image, multiplying a vector including the corresponding pixel position and a multiplicative inverse element of a corresponding element of the smoothed depth map by a transformation matrix representing a difference between a first perspective and a second perspective of the image sensor.
[0157] Using the pixel values of the pixels of the image of the physical environment and the source pixel positions in the untransformed space, the device generates a pixel value for each pixel position of the transformed image using interpolation or other techniques.
[0158] In various implementations, the device modifies the second depth map before using the second depth map to perform the transformation. For example, in various implementations, transforming the image of the physical environment based on the second depth map also includes smoothing the second depth map to fill holes. For example, in various implementations, the device applies a two-dimensional low-pass filter to the second depth map. As another example, in various implementations, when transforming the image of the physical environment based on the second depth map, it is also necessary to clamp the second depth map. For example, in various implementations, any values in the second depth map that are below a clamping threshold are replaced by the clamping threshold.
[0159] Method 1100 continues in box 1160, where the device displays the transformed image on the display. In various embodiments, the transformed image includes XR content. In some embodiments, the XR content is added to the current image of the physical environment before the transformation (in box 1150). In some embodiments, the XR content is added to the transformed image. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on metadata indicating the addition of the XR content to the physical environment. In various embodiments, the device determines whether to add the XR content to the image of the physical environment before or after the transformation based on the amount of XR content (e.g., the percentage of the image of the physical environment that contains XR content).
[0160] In various implementations, the device determines whether to add the XR content to the image of the environment before or after the transformation based on metadata indicating the depth of the XR content. Accordingly, in various implementations, method 1100 includes receiving XR content and XR content metadata, selecting an image or a transformed image of the physical environment based on the XR content metadata, and adding the XR content to the selected image.
[0161] Fig.12is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 1202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 1206, one or more communication interfaces 1208 (e.g., a universal serial bus (USB), FIREWIRE, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1210, a memory 1220, and one or more communication buses 1204 for interconnecting these components and various other components.
[0162] In some implementations, the one or more communication buses 1204 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 1206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.
[0163] Memory 1220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, memory 1220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1220 optionally includes one or more storage devices away from one or more processing units 1202. Memory 1220 includes non-transitory computer-readable storage media. In some specific implementations, memory 1220 or non-transitory computer-readable storage media of memory 1220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1230 and an XR experience module 1240.
[0164] The operating system 1230 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 1240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 1240 includes a data acquisition unit 1242, a tracking unit 1244, a coordination unit 1246, and a data transmission unit 1248.
[0165] In some specific implementations, the data acquisition unit 1242 is configured to at least Figure 1 The electronic device 120 acquires data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various specific implementations, the data acquisition unit 1242 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0166] In some implementations, the tracking unit 1244 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various implementations, the tracking unit 1244 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0167] In some implementations, the coordination unit 1246 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 1246 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0168] In some implementations, the data transfer unit 1248 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transfer unit 1248 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.
[0169] Although the data acquisition unit 1242, the tracking unit 1244, the coordination unit 1246, and the data transmission unit 1248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 1242, the tracking unit 1244, the coordination unit 1246, and the data transmission unit 1248 may be located in separate computing devices.
[0170] also, Fig.12It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig.12 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.
[0171] Fig.13 is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 1302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1306, one or more communication interfaces 1308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1310, one or more XR displays 1312, one or more optional internal-facing and / or external-facing image sensors 1314, a memory 1320, and one or more communication buses 1304 for interconnecting these components and various other components.
[0172] In some implementations, the one or more communication buses 1304 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.
[0173] In some implementations, one or more XR displays 1312 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 1312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 1312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 1312 can present MR and VR content.
[0174] In some implementations, the one or more image sensors 1314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 1314 are configured to face forward so as to acquire image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 1314 may include one or more RGB cameras (e.g., having a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.
[0175] Memory 1320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, memory 1320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1320 optionally includes one or more storage devices away from one or more processing units 1302. Memory 1320 includes non-transitory computer-readable storage media. In some specific implementations, memory 1320 or non-transitory computer-readable storage media of memory 1320 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1330 and an XR rendering module 1340.
[0176] The operating system 1330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 1340 is configured to present XR content to a user via one or more XR displays 1312. To this end, in various implementations, the XR rendering module 1340 includes a data acquisition unit 1342, a perspective transformation unit 1344, an XR rendering unit 1346, and a data transmission unit 1348.
[0177] In some specific implementations, the data acquisition unit 1342 is configured to at least Figure 1 The controller 110 acquires data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various implementations, the data acquisition unit 1342 includes instructions and / or logic components for instructions as well as heuristics and metadata for heuristics.
[0178] In some implementations, the perspective transform unit 1344 is configured to use the refined depth map (e.g. FIG. 8B to FIG. 8F The refined depth maps 843A to 843D in FIG. 9B to FIG. 9F To this end, in various implementations, the perspective transformation unit 1344 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.
[0179] In some implementations, the XR rendering unit 1346 is configured to display the transformed image via one or more XR displays 1312. To this end, in various implementations, the XR rendering unit 1346 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.
[0180] In some implementations, the data transmission unit 1348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 1348 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 1348 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0181] Although the data acquisition unit 1342, the perspective transformation unit 1344, the XR rendering unit 1346, and the data transmission unit 1348 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 1342, the perspective transformation unit 1344, the XR rendering unit 1346, and the data transmission unit 1348 may be located in separate computing devices.
[0182] also, Fig.13It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig.13 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.
[0183] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.
[0184] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0185] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.
[0186] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
Claims
1. A method comprising: At a device including an image sensor, a display, one or more processors, and a non-transitory memory: capturing an image of a physical environment using the image sensor; acquiring a first depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment; generating a second depth map by adjusting one or more portions of the first depth map based on a control signal associated with the image of the physical environment; transforming, using the one or more processors, the image of the physical environment based on the second depth map; as well as The transformed image is displayed via the display.
2. The method according to claim 1, further comprising: The control signal associated with texture alignment is obtained, wherein generating the second depth map includes adjusting the one or more portions of the first depth map based on at least one texture or texture transition within the image of the physical environment.
3. The method according to claim 1, further comprising: The control signal associated with color alignment is obtained, wherein generating the second depth map includes adjusting the one or more portions of the first depth map based on at least one color or color transition within the image of the physical environment.
4. The method according to claim 1, further comprising: The control signal associated with luminous intensity alignment is obtained, wherein generating the second depth map includes adjusting the one or more portions of the first depth map based on at least one luminous intensity or luminous intensity transition within the image of the physical environment. 5 . The method according to claim 1 , wherein the one or more portions of the first depth map are adjusted using a joint bilateral filter.
6. The method according to any one of claims 1 to 5, wherein the control signal indicates one or more specific objects within the image of the physical environment, and wherein the one or more portions of the first depth map correspond to the one or more specific objects.
7. The method of any one of claims 1 to 5, wherein the control signal indicates one or more edges within the image of the physical environment, and wherein the one or more portions of the first depth map correspond to the one or more edges.
8. A method according to any one of claims 1 to 7, wherein, for a specific pixel at a specific pixel position representing a dynamic object in the physical environment, the first depth map includes a specific depth corresponding to the distance between the image sensor and a static object in the physical environment behind the dynamic object. 9 . The method of claim 8 , wherein obtaining the first depth map comprises determining the specific depth by interpolating using depths of locations surrounding the specific pixel location.
10. The method of claim 8 or 9, wherein obtaining the first depth map comprises determining the specific depth when the dynamic object is not represented at the specific pixel position.
11. The method according to any one of claims 8 to 10, wherein acquiring the first depth map comprises determining the specific depth based on a three-dimensional model of the physical environment. 12 . The method of claim 11 , wherein determining the specific depth based on a three-dimensional model comprises rasterizing the three-dimensional model. 13 . The method of claim 11 , wherein determining the specific depth based on the three-dimensional model comprises performing ray tracing based on the three-dimensional model.
14. The method according to any one of claims 11 to 13, wherein the three-dimensional model of the physical environment corresponds to a time-stable three-dimensional model of the physical environment.
15. The method of any one of claims 11 to 13, wherein the three-dimensional model of the physical environment does not include dynamic objects.
16. A method according to any one of claims 1 to 7, wherein the first depth map corresponds to a temporally stable depth map, the temporally stable depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the temporally stable depth map does not include a depth based on a distance between the image sensor and a dynamic object.
17. The method according to any one of claims 1 to 7, wherein obtaining the first depth map comprises obtaining a three-dimensional model of the physical environment and generating the first depth map based on the three-dimensional model, the first depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment. The method of claim 17 , wherein the three-dimensional model comprises a three-dimensional mesh.
19. The method of claim 17 or 18, wherein the three-dimensional model is based on objects in the physical environment that are determined to be static.
20. The method according to any one of claims 17 to 19, further comprising: Generating the three-dimensional model includes: determining that one or more points correspond to one or more static objects in the physical environment; and The one or more points are added to the three-dimensional model at one or more locations in a three-dimensional coordinate system of the physical environment corresponding to the one or more static objects in the physical environment.
21. The method according to any one of claims 1 to 20, further comprising: At least one of the first depth map or the second depth map is smoothed to fill one or more holes.
22. The method according to any one of claims 1 to 21, further comprising: A depth map value associated with at least one of the first depth map or the second depth map is clamped to a predefined depth value.
23. A device comprising: monitor; Image sensor; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods of claims 1 to 22.
24. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having a display and an image sensor, cause the device to perform any of the methods of claims 1 to 22.
25. An apparatus comprising: monitor; Image sensor; one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods according to claims 1 to 22.
26. A method comprising: At a device including an image sensor, a display, one or more processors, and a non-transitory memory: capturing an image of a physical environment using the image sensor; acquiring a first depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment; selecting one or more portions of the first depth map based on a control signal associated with the image of the physical environment; generating a second depth map by refining the one or more portions of the first depth map; transforming, using the one or more processors, the image of the physical environment based on the second depth map; as well as The transformed image is displayed via the display.
27. The method of claim 26, wherein the one or more portions of the first depth map are refined by enlarging depth map values associated with the one or more portions of the first depth map.
28. The method of claim 26, wherein one or more portions of the first depth map are refined by regularizing depth map values associated with the one or more portions of the first depth map.
29. The method of any one of claims 26 to 28, wherein the one or more portions of the first depth map are refined using a joint bilateral filter or a machine learning module.
30. The method of any one of claims 26 to 29, wherein the control signal indicates one or more specific objects within the image of the physical environment, and wherein the one or more portions of the first depth map correspond to the one or more specific objects.
31. A method according to any one of claims 26 to 29, wherein the control signal indicates one or more edges within the image of the physical environment, and wherein the one or more portions of the first depth map correspond to the one or more edges.
32. A method according to any one of claims 26 to 29, wherein the control signal indicates one or more edges of one or more specific objects within the image of the physical environment, and wherein the one or more portions of the first depth map correspond to the one or more edges of the one or more specific objects.
33. A method according to any one of claims 26 to 32, wherein, for a specific pixel at a specific pixel position representing a dynamic object in the physical environment, the first depth map includes a specific depth corresponding to the distance between the image sensor and a static object in the physical environment behind the dynamic object.
34. The method of claim 33, wherein obtaining the first depth map comprises determining the specific depth by interpolating using depths of locations surrounding the specific pixel location.
35. The method of claim 33 or 34, wherein obtaining the first depth map comprises determining the specific depth when the dynamic object is not represented at the specific pixel position.
36. The method of any one of claims 33 to 35, wherein acquiring the first depth map comprises determining the specific depth based on a three-dimensional model of the physical environment.
37. The method of claim 36, wherein determining the specific depth based on a three-dimensional model comprises rasterizing the three-dimensional model.
38. The method of claim 36, wherein determining the specific depth based on the three-dimensional model comprises performing ray tracing based on the three-dimensional model.
39. A method according to any one of claims 36 to 38, wherein the three-dimensional model of the physical environment corresponds to a time-stable three-dimensional model of the physical environment.
40. A method according to any one of claims 36 to 38, wherein the three-dimensional model of the physical environment does not include dynamic objects.
41. A method according to any one of claims 26 to 40, wherein the first depth map corresponds to a temporally stable depth map, the temporally stable depth map comprising a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the temporally stable depth map does not include a depth based on a distance between the image sensor and a dynamic object.
42. A method according to any one of claims 26 to 40, wherein obtaining the first depth map includes obtaining a three-dimensional model of the physical environment and generating the first depth map based on the three-dimensional model, the first depth map including multiple depths respectively associated with multiple pixels of the image of the physical environment.
43. The method of claim 42, wherein the three-dimensional model comprises a three-dimensional mesh.
44. A method according to claim 42 or 43, wherein the three-dimensional model is based on objects in the physical environment that are determined to be static.
45. The method according to any one of claims 42 to 44, further comprising: Generating the three-dimensional model includes: determining that one or more points correspond to one or more static objects in the physical environment; and The one or more points are added to the three-dimensional model at one or more locations in a three-dimensional coordinate system of the physical environment corresponding to the one or more static objects in the physical environment.
46. The method according to any one of claims 26 to 45, further comprising: At least one of the first depth map or the second depth map is smoothed to fill one or more holes.
47. The method according to any one of claims 26 to 46, further comprising: A depth map value associated with at least one of the first depth map or the second depth map is clamped to a predefined depth value.
48. An apparatus comprising: monitor; Image sensor; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, when executed by the one or more processors, cause the device to perform any of the methods according to claims 26 to 47.
49. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having a display and an image sensor, cause the device to perform any of the methods of claims 26 to 47.
50. An apparatus comprising: monitor; Image sensor; one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods of claims 26 to 47.
