Remote depth buffer compression

Abstract

A high dynamic range (HDR) depth buffer is received at a remote computer. The HDR depth buffer is formed into a plurality of tiles. For each tile, a respective maximum value and a minimum value of the HDR depth buffer in an area greater than a width of the respective tile are determined. An initial pair of piecewise bilinear bounding functions for the HDR depth buffer is determined using the determined maximum depth values and minimum depth values. For each tile, the initial pair of piecewise bilinear bounding functions is iteratively adjusted to bring the respective minimum depth value and maximum depth value of each tile closer to the HDR depth buffer, wherein no adjacent tiles are adjusted in the same iteration. Using the adjusted pair of piecewise bilinear bounding functions, a low dynamic range (LDR) depth buffer and tile data are generated and encoded using a video encoder and a lossless compressor, respectively.

Description

Background Technology

[0001] Mixed reality (MR) systems and devices include virtual reality (VR) systems and augmented reality (AR) systems. Conventional VR systems create a fully immersive experience by limiting the user's field of vision to virtual images rendered only within the VR scene / environment. Conventional AR systems create AR experiences by visually presenting virtual images placed in or interacting with the real world. As used herein, VR and AR systems are described and referred to interchangeably using the phrase "MR system." Furthermore, as used herein, the terms "virtual image," "virtual content," "color image," and "hologram" refer to any type of digital image rendered by an MR system. Additionally, it should be noted that head-mounted displays (HMDs) typically provide a display that the user uses to view and / or interact with holograms provided within the MR scene / environment.

[0002] Remote rendering is typically performed where the hologram is initially generated or prepared at a remote system (e.g., at a remote cloud service with a dedicated graphics processing unit (GPU)) and subsequently transmitted to the HMD over a network. As used herein, a remote computer or system includes a computer or system separate from the HMD, such as one located near the HMD or situated in the cloud. Remote rendering is advantageous because it helps reduce the amount of processing performed locally on the HMD.

[0003] In remote rendering, a computer or cloud server generates color and depth images, which are transmitted over a network to a head-mounted display (HMD). The depth image is typically used for positional correction of the color image, also known as post-reprojection, and for compositing with local content. This makes transmitting the depth image with sufficiently high accuracy a crucial aspect of remote rendering.

[0004] The embodiments described below are not intended to be limited to solutions to any or all the drawbacks of environments such as those described above. Rather, this background information is provided merely to illustrate an exemplary technical field in which some of the embodiments described herein may be implemented. Summary of the Invention

[0005] The following is a simplified overview of this disclosure to provide the reader with a basic understanding. This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Its sole purpose is to present, in a simplified form, a set of concepts disclosed herein as a prelude to the more detailed description that follows.

[0006] The examples disclosed herein work to improve the accuracy of depth images transmitted between a remote rendering device and an HMD. By following the disclosed principles, substantial benefits, advantages, and computational efficiency can be achieved, thereby improving the performance of the computing architecture and the user experience of using the HMD.

[0007] In some examples, a High Dynamic Range (HDR) depth buffer is received at a remote computer. The HDR depth buffer is decomposed into a Low Dynamic Range (LDR) depth buffer and tile data. Decomposing the HDR depth buffer involves: dividing the HDR depth buffer into multiple tiles; for each of the multiple tiles, determining the corresponding maximum and minimum values ​​of the HDR depth buffer in a region larger than the width of the corresponding tile; using the determined maximum and minimum depth values ​​of the HDR depth buffer in the region larger than the width of the corresponding tile, determining an initial pair of piecewise bilinear bounding functions for the HDR depth buffer; for each tile, the initial pair of piecewise bilinear bounding functions is iteratively adjusted so that the corresponding minimum and maximum depth values ​​of each tile move closer to the HDR depth buffer, where no adjacent tiles are adjusted in the same iteration; using the adjusted pair of piecewise bilinear bounding functions, generating the LDR depth buffer and tile data; encoding the generated LDR depth buffer using a video encoder, and compressing the tile data using a lossless compressor. The encoded LDR depth buffer and compressed tile data are transmitted to the head-mounted device (HMD) via a data network.

[0008] In some embodiments, the head-mounted device (HMD) receives an encoded low dynamic range (LDR) depth buffer and compressed tile data. The compressed LDR depth buffer is decoded using a video decoder, and the compressed tile data is decompressed using a decompressor. Using a piecewise bilinear interpolation function, the HDR depth buffer is reconstructed from the decoded LDR depth buffer and the decoded tile data.

[0009] As these features become better understood through the following detailed description taken in conjunction with reference to the accompanying drawings, many accompanying features will be more readily grasped. Attached Figure Description

[0010] This description will be better understood from the following detailed description, which is read in conjunction with the accompanying drawings, in which: Figure 1 This illustrates an example architecture used to perform remote rendering; Figure 2 This illustrates an example architecture used to perform remote rendering; Figure 3 A flowchart illustrating an example technique for remote rendering of depth images; Figure 4 Example graphs are shown depicting various example bounding functions for depth images according to known techniques; Figure 5 Example graphs depicting the limit functions for depth images according to this technique are shown; Figure 6 An example method for generating a boundary function for depth images according to the present technology is shown; Figure 7 Example graphs depicting the limit functions for depth images according to this technique are shown; Figure 8 Example graphs depicting the limit functions for depth images according to this technique are shown; Figure 9 This shows a depth image that has been renormalized according to an example of a known technique; Figure 10 This shows an example of a renormalized depth image according to this technique; Figure 11 A flowchart illustrating an example technique for reconstructing depth images on an HMD is shown; and Figure 12 An exemplary computation-based device in which remote rendering is implemented is shown. The same reference numerals are used to indicate the same parts in the figures. Detailed Implementation

[0011] The detailed description provided below, in conjunction with the accompanying drawings, is intended as a description of this example and is not intended to represent the only form in which this example can be constructed or utilized. This description illustrates the functionality of the example and the sequence of operations used to construct and operate the example. However, the same or equivalent functionality and sequences may be implemented in different examples.

[0012] In remote rendering scenarios, powerful computers or cloud servers generate color and depth images, which are transmitted to the HMD (Head-up Display) via a network. For each pixel, the depth image includes a value indicating how far away a surface in the world depicted by that pixel is from the camera. Depth images can be used for many purposes, such as position correction of the corresponding shading images (post-projection) and compositing with local content, making transmission with sufficiently high accuracy crucial. Therefore, to ensure adequate quality for post-projection and / or compositing, it is essential that the depth image be transmitted over the network in high quality.

[0013] Regarding color images, commercial video encoders can easily encode color images with sufficiently high quality. For example, existing video codec hardware implementing H.264, H.265, or AV1 is well-suited for compressing low dynamic range (LDR) color images with low latency to reduce network bandwidth requirements.

[0014] Video codec hardware typically supports a dynamic range of 8 bits per pixel, or 10-12 bits per pixel when High Dynamic Range (HDR) support is available. However, depth images typically have a dynamic range of 16, 24, or 32 bits per pixel. This makes it impossible to directly encode depth images using standard video codecs. Therefore, additional processing can be performed to downscale HDR images to LDR images that can be transmitted over the network with minimal loss in order to transmit depth images over a network. However, existing methods of processing HDR depth images before encoding can introduce discontinuities into LDR depth images, which in turn introduce pseudo-high-frequency components into the LDR depth images sent to the video encoder.

[0015] Addressing discontinuities introduced into LDR depth images requires additional processing for encoding and adds significant latency to the transmission of the depth image to the HMD. Therefore, reducing the amount of depth discontinuities in a depth image while maintaining depth quality offers numerous benefits.

[0016] Figure 1 An example architecture 100 for performing remote rendering is shown. Figure 1 As shown, architecture 100 includes a scene involving a head-mounted display (HMD) 140. The HMD includes hardware such as a graphics processing unit (GPU) 145. Architecture 100 also includes a remote computer 110, which includes hardware such as a GPU 112 and a depth rendering engine 114. The remote computer 110 processes HDR depth images into LDR depth images, which are sent to the HMD 140 via a network 130 as part of a depth buffer stream 120. The remote computer can also send a color image video stream 125 to the HMD 140 via the network 130. The network 130 and / or the remote computer 110 may operate in a cloud environment 135.

[0017] In various examples, the functionality in the remote computer is able to determine a pair of piecewise bilinear bounding functions using a heuristic process for solving millions of inequalities in real time. The pair of piecewise bilinear bounding functions is determined in approximately one millisecond, rather than several seconds, significantly reducing latency in HDR depth image transmission. Furthermore, the determined pair of piecewise bilinear bounding functions fits tightly around the HDR depth buffer, reducing overall quantization error. The determined functions are also determined with minimal computation and are easily encoded (e.g., via tile data) and can be transmitted using a small amount of bandwidth.

[0018] Because the bounding function of this technique fits tightly around the HDR depth values, the overall quantization error is reduced. Furthermore, due to the tight fit of the piecewise bilinear bounding function and the lack of discontinuity in the tiles within the LDR depth buffer, the reconstruction of the HDR depth image is performed on the HMD with minimal computation.

[0019] Figure 2 An example system 200 for performing remote rendering is shown. Example system 200 includes a remote computer 210, a network 220, and an HMD 230. In some cases, the remote computer 210 is a server or compute node in a data center. At the remote computer 210, a high dynamic range (HDR) depth buffer 212 is received (e.g., from a renderer that renders a depth buffer from a 3D model). As used herein, the terms depth buffer and depth image may be used interchangeably. Depth normalization 214 is performed on the HDR depth buffer 212 to obtain a low dynamic range (LDR) depth buffer 216 and tile data 218. In some examples, the tile data 218 includes metadata about the normalization 214 of the HDR depth buffer 212. The LDR depth buffer 216 is compressed using a video encoder 216a to obtain a depth video 222, which is transmitted over the network 220 to the HMD 230. Tile data 218 is compressed using lossless compressor 218a, and the compressed tile data 224 is sent to HMD 230 via network 220.

[0020] At HMD 230, video decoder 226 is used to decode the compressed LDR depth buffer 222 into LDR depth buffer 232. Also at HMD 230, decompressor 228 is used to decompress the compressed tile data 224 into tile data 234. LDR depth buffer 232 and tile data 234 are reconstructed 236 to obtain HDR depth buffer 238 at HMD.

[0021] Figure 3 Exemplary methods are shown that are performed on a remote server or remote computer, such as remote computer 210. Figure 2 As depicted, a remote computer receives a 302 HDR depth buffer, such as from a video game, a renderer rendering a depth buffer from a 3D model, or from memory storing the HDR depth buffer. To perform depth renormalization, the HDR depth buffer is divided 304 into multiple tiles. The HDR depth buffer can be divided in several ways. For example, the HDR depth buffer can be divided into NxN square tiles with predetermined dimensions. However, the tiles do not need to be square. In this example, the tiles are rectangular.

[0022] For each of the multiple tiles, the corresponding maximum and minimum values ​​of the HDR depth buffer in the region greater than the width of the corresponding tile are determined 306. An initial pair of piecewise bilinear bounding functions for the HDR depth buffer are determined using the determined maximum and minimum depth values ​​308.

[0023] For each tile, the initial pair of piecewise bilinear bounding functions is iteratively adjusted by 310 so that the corresponding minimum and maximum depth values ​​of the initial pair of piecewise bilinear bounding functions for each tile move closer to the HDR depth buffer, wherein the depth values ​​of the initial pair of piecewise bilinear bounding functions for adjacent tiles are not adjusted in the same iteration.

[0024] Using a pair of adjusted piecewise bilinear bounding functions, the LDR depth buffer and tile data are generated as 312. The tile data includes renormalized metadata about the HDR depth buffer. For example, the tile data may include metadata about the values ​​of the piecewise bilinear bounding functions. This can take the form of a pair of minimum and maximum depth values ​​per tile. For example, in the case of a 64x64 tile size, the transmitted tile data represents less than 1% bandwidth overhead.

[0025] The LDR depth buffer is compressed using a video encoder 314, and the tile data is compressed using a lossless compressor 316. The compressed LDR depth buffer and the compressed tile data are transmitted over the network 318 to the HMD.

[0026] As described above, the renormalization scheme for depth encoding involves finding a pair of parameterized bounding functions lo(x, y) and hi(x, y) on either side of the depth value d(x, y) at each pixel location (x, y) in the depth buffer. As used herein, lo(x, y) and hi(x, y) represent the minimum bounding function and the maximum bounding function, respectively. This yields the following pair of parameterized bounding functions:

[0027] These functions enclose depth values. The luminance value Y(x, y) can be calculated by a remote computer. The luminance value is generated by renormalization. The luminance value is quantized to n-bit precision (e.g., n=8, 10, 12) that can be supported by standard video codec hardware.

[0028] The following equation, or other methods known in the art for renormalizing depth images, can be used to generate LDR depth buffers. For example, to generate an 8-bit renormalized depth buffer (with values ​​ranging from 0 to 255), the minimum depth value in a tile can be mapped to 0, and the maximum depth value in a tile can be mapped to 255. Any value between the maximum and minimum depth values ​​can be scaled between 0 and 255. Similarly, in cases where the video codec supports 10 or 12-bit precision, remapping occurs at 0-1023 and 0-4095, respectively.

[0029] When attempting to solve the inequalities for piecewise bilinear bounds of a depth buffer, millions of constraints exist. Therefore, solving these constraints using known methods to find the optimal pair of bounds for an HDR depth buffer typically takes several seconds. The time spent solving these inequalities using known methods adds significant latency to data transmission and can impact the user experience.

[0030] A solution is provided for solving millions of inequalities at a speed several orders of magnitude faster than known methods by using a pair of piecewise bilinear bounding functions determined using the techniques disclosed herein. The pair of piecewise bilinear bounding functions is determined in approximately one millisecond, rather than several seconds, thus significantly reducing latency in HDR depth image transmission. Furthermore, the determined pair of piecewise bilinear bounding functions fits tightly around the HDR depth buffer, thereby reducing overall quantization error. The determined functions are also determined with minimal computation and are easily encoded (e.g., via tile data) and can be transmitted using minimal bandwidth.

[0031] refer to Figure 4 The graph depicts a depth of 410 relative to tile coordinates 420. The solid black line represents the HDR depth buffer 430, and the dashed lines 442, dappled lines 444, and dotted lines 446 represent several of the infinite possibilities of the maximum bounding function, such that the depth buffer is always less than or equal to the maximum bounding functions 442, 444, and 446. Each "x" on the graph represents the position within the corresponding tile. For simplicity, Figure 4 The "x" in the diagram is chosen as the corresponding center of each tile. For example, in... Figure 4 As can be seen, the optimal maximum (or minimum) depth value for each tile cannot be selected independently of other tiles. For example, if the depth value 442a at dashed line 442 is selected as close as possible to the valley of depth buffer 430 so that the boundary function does not intersect with the depth buffer, the depth values ​​at adjacent tiles are selected at a greater distance above the HDR depth buffer 430, which is not optimal.

[0032] Similarly, for the maximum bounding functions of dotted-dash line 444 and dotted line 446, the selected depth values ​​for the corresponding tiles are more appropriate, but they do not fit the "valley" of the HDR depth buffer 430 as well. The same principle applies to the infinite possibilities of the piecewise bilinear bounding function, because optimizing a single tile to be as close as possible to the HDR depth buffer will lead to globally suboptimal results.

[0033] Figure 5 This is a diagram illustrating how a remote computer determines an initial pair of piecewise bilinear bounding functions in one dimension. Figure 5 The diagram depicts a plot showing depth 510 relative to tile coordinates 520. Figure 5 In the diagram, the coordinates of the tile's center are marked with a dotted line at 560. Although the graph is drawn relative to the tile's center, different points on the tile can be used as reference points. Similar to... Figure 4 ,exist Figure 5 In the diagram, the HDR depth buffer is depicted using a solid black line 530. To determine the maximum piecewise bilinear bounding equation (i.e., a bounding equation for each tile in the HDR depth buffer with a depth value larger than the HDR depth buffer itself), an initial maximum piecewise bilinear bounding function 540 is determined. To obtain the initial piecewise bilinear maximum bounding function represented by the dashed line 540, for each tile, the maximum depth value of the HDR depth buffer 530 in a region larger than the width of the tile is determined. The determined maximum depth values ​​are concatenated to obtain the initial maximum piecewise bilinear bounding function 540. As an example, in... Figure 5 In this context, the width of the tile is denoted as "T", and the tile depth value used for the initial maximum piecewise bilinear bounding function 540 is selected as the maximum depth value in the 2T region 545 around the center of each tile in the HDR depth buffer 530.

[0034] A similar process is followed to obtain the initial minimum piecewise bilinear bounding function, represented by the dashed line 550. For each tile, a minimum depth value for the HDR depth buffer 530 in a region greater than the tile width is determined. The determined minimum depth values ​​are concatenated to obtain the initial minimum piecewise bilinear bounding function 550. Similar to the maximum piecewise bilinear bounding function, the tile width is T, and the tile depth value used for the initial minimum piecewise bilinear bounding function 550 is chosen as the minimum depth value of the HDR depth buffer 530 in a 2T region 555 around the center of each tile. For the same example in two dimensions (2D), the tile is chosen to have a width W and a height H. In 2D, the corresponding initial region is a rectangle with a width of 2W and a height of 2H.

[0035] Therefore, when the initial piecewise bilinear bounding function is found, the HDR depth buffer is scanned individually for each tile to find the corresponding maximum and minimum depth values ​​in a rectangular region of 2T centered around the center of each tile. Since the range of the scanned region for each tile is larger than the width of the tile, the scanned regions for adjacent tiles will overlap. This ensures that the interpolated values ​​between adjacent tiles do not violate any bounding function inequalities (i.e., the maximum bounding function is greater than or equal to the HDR depth buffer at all tile locations, and the minimum bounding function is less than or equal to the HDR depth buffer at all tile locations).

[0036] By having overlapping regions for each adjacent tile when determining the corresponding minimum and maximum values ​​of the HDR depth buffer, discontinuities in the tile boundaries are absent when compared with existing piecewise constant functions. This results in a sufficiently smooth function that does not introduce pseudo-high frequencies into the video texture. The avoidance of tile boundary discontinuities due to the use of a piecewise bilinear bounding function reduces the amount of additional processing required to generate a uniform image at the HMD. Furthermore, by using a piecewise bilinear bounding function, each tile can be processed independently of other tiles, allowing for the parallelization of the initial piecewise bilinear bounding function, for example, in a parallel computing processor. This parallelization also allows each tile to be processed simultaneously, resulting in a balanced fit that does not require a locally optimal fit at the expense of the overall fit.

[0037] Although the initial pair of piecewise bilinear bounding functions is not the optimal fit, the distance between the initial piecewise bilinear bounding functions and the HDR depth buffer does not change significantly, which is consistent with... Figure 4 The example scenario depicted in the text is the opposite. Furthermore, as in... Figure 5 As can be seen, the upper bound (maximum piecewise bilinear bounding function) is exactly horizontal around the peak of the HDR depth buffer. Subsequent rounds of the disclosed technique will maintain this invariant and will keep the tile values ​​adjacent to this peak unchanged, thus preventing the optimization of a tile's fit at the expense of degrading the fit of another tile.

[0038] Figure 6 A sample two-dimensional visualization of method 600 for optimizing an initial pair of piecewise bilinear bounding functions is shown. In this example, for simplicity, the HDR depth buffer is depicted as multiple rows and columns divided into square tiles. However, it is noted that the tiles can have different shapes, and the HDR depth buffer can be divided in different ways beyond regular rows and columns.

[0039] Parallel adjustment of two adjacent tiles requires data communication between parallel processors optimizing those adjacent tiles to avoid violating bounded function constraints. This inter-processor data communication between adjacent tiles introduces significant latency to the processing of the depth buffer. By processing a tile while keeping its nearest neighbor untouched (i.e., adjusting only adjacent tiles), the surrounding environment of adjacent tiles can be ignored without requiring any inter-process or inter-thread communication. By ensuring that no adjacent tiles are optimized in a single pass, the piecewise bilinear fit of one tile is not optimized at the expense of another. As referred to herein, a “pass” is equivalent to one iteration of the method. Figure 6 In order to avoid adjusting the depth values ​​of the initial pair of bilinear piecewise bounding functions for adjacent tiles, in the first round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted only for every tile in every other column (which is also equivalent to every tile in every other row). Figure 6 As depicted, in the first round (“Round 1”), only the odd-numbered tiles in each odd-numbered column are adjusted. Then, in the second round (“Round 2”), every odd-numbered tile in each even-numbered column is adjusted. In the third round (“Round 3”), every even-numbered tile in each odd-numbered column is adjusted, and in the fourth round (“Round 4”), every even-numbered tile in each even-numbered column is adjusted.

[0040] It is important to note that the order in which the depth values ​​of the tiles are adjusted in the corresponding rounds is not important. For example, instead of adjusting the depth value of each odd tile in each odd column, the depth value of each even tile in each odd column can be adjusted in the first round, and so on.

[0041] In this way, optimization is complete once each tile has had its corresponding depth value adjusted so that the maximum and minimum depth values ​​of each tile's pair of piecewise bilinear bounding functions move closer to the HDR depth buffer distribution. For example, the corresponding minimum and maximum depth values ​​of a tile are adjusted until the interpolated values ​​between adjacent tiles become tangent to the HDR depth buffer distribution. By tightening only a subset of tiles in each round, so that no pair of adjacent tiles is modified in a single round, the technique disclosed herein can be executed on a parallel processor.

[0042] Figure 7 Example graphs depict the bounding functions of a one-dimensional deep buffer after the first optimization round of an initial pair of piecewise bilinear bounding functions. Figure 7 The diagram depicts a plot showing depth 710 relative to tile coordinates 720. Figure 7 In, similar to Figure 5The coordinates of the tile center are marked by a dotted line 760, and the HDR depth buffer is depicted by a solid line 730. A pair of piecewise bilinear bounding functions 740 and 750 are similar to... Figure 5 The functions shown in the diagram undergo a round of adjusting the depth values, as depicted by arrows 745 and 755. Figure 7 In the example shown, a pair of piecewise bilinear bounding functions is adjusted only for even-indexed tiles, while the depth values ​​for odd-indexed tiles remain unchanged. Up and down arrows 745 and 755 indicate the relationship between... Figure 5 The initial pair of piecewise bilinear bounding functions shown in the figure are compared with the improved pair of piecewise bilinear bounding functions.

[0043] exist Figure 7 In the process, when the depth value of a tile is adjusted, the corresponding maximum and minimum depth values ​​for that tile are moved closer to the depth value of the HDR depth buffer 730 until the interpolated line between adjacent tiles becomes tangent to the HDR depth value. This is marked with an "x" when the interpolated line is tangent to the HDR depth buffer distribution 730. Tightening of even-indexed tiles is followed by tightening of odd-indexed tiles, without further tightening of even-indexed tiles. This is in Figure 8 It is depicted in the middle.

[0044] Figure 8 Example graphs depict the bounding functions of a one-dimensional deep buffer after the second optimization round, based on an initial pair of piecewise bilinear bounding functions. Figure 8 The diagram depicts a plot showing depth 810 relative to tile coordinates 820. Figure 8 In, similar to Figure 5 and Figure 7 The coordinates of the tile center are marked by dotted line 860, and the HDR depth buffer is depicted by solid line 830. A pair of piecewise bilinear bounding functions 840 and 850 are similar to... Figure 5 and Figure 7 The functions shown in the diagram undergo a second round of adjustment of the depth values, as depicted by arrows 845 and 855. Figure 8 In the example shown, after the first round, a pair of piecewise bilinear bounding functions is adjusted only for odd-indexed tiles, while the depth values ​​for even-indexed tiles remain unchanged. Up and down arrows 845 and 855 indicate the positions of the tiles in the example. Figure 5 and Figure 7 The improvement of the initial pair of piecewise bilinear bounding functions shown in the figure compared to the pair of piecewise bilinear bounding functions adjusted in the first round.

[0045] exist Figure 8 In the process of adjusting the depth value of a tile, the corresponding maximum and minimum depth values ​​for that tile are moved closer to the depth value of the HDR depth buffer 830 until the interpolated line between adjacent tiles becomes tangent to the HDR depth value. When the interpolated line is tangent to the HDR depth buffer distribution 830, the interpolated line is marked with an "x".

[0046] Although Figure 7 and Figure 8 The initial pair of piecewise bilinear bounding functions is shown in one dimension and optimized using two rounds. However, it is noted that in two dimensions, there will be four rounds of tightening (e.g., ...). Figure 6 (As depicted in the image) to cover all the tiles.

[0047] The techniques presented in this paper provide a localization scheme for optimizing piecewise bilinear bounding functions for HDR depth buffers. Since the results of each round can be generated independently for each tile, massively parallel computing devices are able to execute each round, resulting in an overall computation time of less than one millisecond – an improvement of over 1000 times compared to general-purpose linear constraint solvers of the prior art.

[0048] Refer again Figure 2 The LDR depth buffer 216 and tile data 218 are compressed using a video encoder and a lossless compressor, respectively, and transmitted to the HMD 230 via data network 220. At the HMD, the LDR depth buffer and tile data are used to reconstruct the HDR depth buffer. The reconstructed HDR depth buffer can then be used to perform post-projection, or to composite the HDR depth buffer with local content or other content.

[0049] Figure 9 An example of an LDR depth buffer 900 obtained from a known method using a piecewise constant bounding function is shown. Because the known method renormalizes the HDR depth buffer by directly mapping depth values ​​to, for example, a 0-255 range for an 8-bit depth image, discontinuities around tile boundaries are introduced into the LDR depth image sent to the video encoder via the network to the HMD. The tile structure in... Figure 9 As clearly visible, each edge between tiles represents a discontinuity that introduces pseudo-high-frequency components into the image sent to the video encoder. The tile boundaries of individual tiles in the LDR depth buffer 900 are delineated by squares 910. This requires additional bits to encode these edges, even though these edges do not represent any features of the original HDR depth image.

[0050] Figure 10An example of an LDR depth buffer 1000 obtained from this technique is shown. By using a piecewise bilinear function, the obtained LDR depth buffer has no discontinuities between tiles, and therefore requires less bandwidth when compressed using a video encoder.

[0051] Example flow of methods executed by HMD in Figure 11 It is shown in the middle. Figure 11 In the process, the encoded LDR depth buffer and compressed tile data are received at the HMD 1102. The HMD's video decoder decodes 1104 the encoded LDR depth buffer, and the decompressor decompresses 1104 the compressed tile data. The HMD uses a piecewise bilinear reconstruction function to reconstruct 1106 the HDR depth buffer from the decoded LDR depth buffer and the decompressed tile data. For example, the hardware sampling unit on the HMD's GPU can be used to perform piecewise bilinear interpolation of the tile boundaries in the LDR depth buffer to obtain the HDR depth buffer. This can be accomplished using a pair of texture acquisition and simple arithmetic instructions, as shown below in the High-Level Shading Language (HLSL): / / Load the bounds from a texture containing the minimum and maximum values ​​per tile float2 hiLo = bounds Texture.Sample(nearestNeighborSampler, uv); / / Load the depth value from the decoded depth video texture float depthValue = depthVideoTexture.Sample(depthSampler, uv); / / Rescale to full range depthValue=lerp(depthValue, hiLo.x, hiLo.y);

[0052] For example, when reconstructing the original HDR depth image from LDR image and tile data, the sampled values ​​from the decoded LDR depth buffer can be remapped such that, for example, 0 is mapped to the minimum depth value for each tile, and 255 is mapped to the maximum depth value for each tile. The remaining depth values ​​are scaled accordingly.

[0053] As described above, the renormalization scheme for depth encoding involves finding a pair of parameterized bounding functions lo(x, y) and hi(x, y) on either side of the depth value d(x, y) at each pixel location (x, y) in the depth buffer. As used herein, lo(x, y) and hi(x, y) represent the minimum bounding function and the maximum bounding function, respectively. This yields the following pair of parameterized bounding functions:

[0054] These functions enclose depth values. Luminance values ​​Y(x, y), such as luminance values ​​calculated on a remote computer, are computed. The luminance values ​​are quantized to n-bit precision (e.g., n=8, 10, 12) that can be supported by standard video codec hardware.

[0055] On an HMD, the HDR depth buffer can be reconstructed by interpolating between the lo(x, y) bound and the hi(x, y) bound. This can be done according to the following equation:

[0056] The quality of depth reconstruction depends heavily on the choice of the bounding function used for reconstruction. Because the bounding function of this technique fits tightly around the HDR depth values, the overall quantization error is reduced. The tighter the fit of the piecewise bilinear bounding function, the fewer quantization errors are introduced through rounding during renormalization. Furthermore, due to the tight fit of the bounding function and the lack of tile discontinuities in the LDR depth image, the reconstruction of the HDR depth image is performed on the HMD with minimal computation.

[0057] The piecewise bilinear bounding function of this disclosure is determined in an unconventional manner to achieve the bandwidth reduction and significant improvement in processing speed. By determining a piecewise bilinear bounding function that eliminates discontinuities in the renormalized depth buffer while maintaining high accuracy of the HDR depth buffer, the operation of the underlying computing device is improved.

[0058] Figure 12 Various components of an exemplary computer system 1200 are shown, which are implemented as any form of computing and / or electronic device, and in some examples therein, Figures 1 to 10 Implementations are as follows. The computer 1200 can take various forms. For example, in cases where the computer system 1200 is used to deploy on a remote computer described herein, the computer system 1200 can be a server, a computing node, or a distributed computing function. In some examples, the computer system 1200 is an HMD (Host Device Management).

[0059] The computer system 1200 can also be a distributed system, which includes one or more connected computing components / devices that communicate with the computer system 1200.

[0060] In its most basic configuration, the computer system 1200 includes a variety of different components. Figure 12 The computer system 1200 shown includes one or more processors 1205A (also referred to as "hardware processing units"), multiple GPUs 1205B, a buffer engine 1210, and memory 1220.

[0061] Regarding the multiple processors 1205A and multiple GPUs 1205B, it will be appreciated that the functions described herein can be performed at least in part by one or more hardware logic components (e.g., (multiple) processors 1205A or (multiple) GPUs 1205B). Illustrative types of hardware logic components / processors that can be used include, for example, but not limited to, field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), application-specific standard products (“ASSPs”), system-on-a-chip systems (“SOCs”), complex programmable logic devices (“CPLDs”), central processing units (“CPUs”), graphics / graphics processing units (“GPUs”), or any other type of programmable hardware.

[0062] The buffer engine 1210 may be implemented to perform any operations relating to the creation and modification of depth buffers. Additionally or alternatively, processor(s) 1205A or GPU(s) 1205B may also be involved. The buffer engine 1210 may be implemented as a specific processing unit (e.g., a dedicated processing unit as previously described) configured to perform one or more specialized operations for computer system 1000. As used herein, the terms “executable module,” “executable component,” “component,” “module,” or “engine” may refer to a hardware processing unit or a software object, routine, or method that may be executed on computer system 1200. The various components, modules, engines, and services described herein may be implemented as objects or processors (e.g., as separate threads) that execute on computer system 1200.

[0063] Memory 1220 may be physical system memory, which may be volatile, non-volatile, or a combination of both. The term "memory" may also be used herein to refer to non-volatile mass storage, such as physical storage media. If computer system 1000 is distributed, processing, memory, and / or storage capacity may also be distributed.

[0064] Memory 1220 is shown as including executable instructions (i.e., code 1225). Executable instructions refer to instructions that can be executed by the processor(s) 1205A of computer system 1200 (or possibly even by the GPU(s) 1205B or buffer engine 1210) to perform the disclosed operations, such as those described in the various methods.

[0065] The disclosed embodiments may include or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such as processor(s) 1205A) and system memory (such as memory 1220), as discussed in more detail below. Embodiments also include physical and other computer-readable storage media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable storage media may be any available media accessible by a general-purpose or special-purpose computer system. A computer-readable storage medium that stores computer-executable instructions in data form is a “physical computer storage medium” or “hardware storage device.” A computer-readable storage medium that carries computer-executable instructions is a “transmission medium.” In contrast, a communication medium carries computer-readable instructions, data structures, program modules, or similar objects in a modulated data signal, such as a carrier wave, or other transmission mechanism. As defined herein, a computer-readable storage medium does not include a communication medium. Therefore, a computer storage medium does not include a signal in transit. Thus, by way of example and not limitation, the present embodiments may include at least two distinctly different computer-readable media: a computer storage medium and a transmission medium.

[0066] Computer storage media (also known as “hardware storage devices”) are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, RAM-based solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”) or other types of memory, or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions, data or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0067] Computer system 1200 can also be connected (via wired or wireless connection) to external sensors (e.g., one or more remote cameras) or devices via network 1230. For example, computer system 1200 can communicate with any number of devices or cloud services to acquire or process data. In some cases, network 1230 itself can be a cloud network. Furthermore, computer system 1200 can also be connected via one or more wired or wireless networks 1230 to remote / separate(multiple) computer systems configured to perform any of the processes described regarding computer system 1200.

[0068] The computer system 1200 may also include an input / output controller arranged to output display information to a display device that may be separate from or integrated with the computing-based device 1200. The display information may provide a graphical user interface. The input / output controller is also arranged to receive and process input from one or more devices, such as user input devices (e.g., a mouse, keyboard, camera, microphone, or other sensors). In some examples, the user input device detects user gestures or other user actions and provides a natural user interface (NUI). This user input may be used to obtain HDR depth images.

[0069] Any of the input / output controller, display device, and user input device may include NUI technology that enables users to interact with the computer system in a natural manner, free from human constraints imposed by input devices such as mice, keyboards, remote controls, and the like. Examples of NUI technologies provided in some examples include, but are not limited to, those relying on on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence. Other examples of NUI technologies used in some examples include intent and goal understanding systems, motion gesture detection systems using depth cameras (such as stereo camera systems, infrared camera systems, red-green-blue (RGB) camera systems, and combinations thereof), motion gesture detection using accelerometers / gyroscopes, facial recognition, three-dimensional (3D) displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, and technologies using electric field sensing electrodes to sense brain activity (electroencephalography (EEG) and related methods).

[0070] A “network” (such as network 1230) is defined as one or more data links and / or data switches that enable the transmission of electronic data between computer systems, modules, and / or other electronic devices. When information is transmitted or provided to a computer via a network (hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately considers the connection as a transmission medium. Computer system 1200 will include one or more communication channels used to communicate with network 1230. Transmission media include networks that can be used to carry data or desired program code in the form of computer-executable instructions or data structures. Furthermore, these computer-executable instructions can be accessed by general-purpose or special-purpose computers. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0071] Upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in the RAM within a network interface module (e.g., a network interface card or "NIC") and subsequently transferred to the computer system RAM and / or a less volatile computer storage medium at the computer system. Therefore, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0072] Computer-executable (or computer-interpretable) instructions include, for example, instructions that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or set of functions. Computer-executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0073] Those skilled in the art will recognize that the embodiments can be implemented in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics devices, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, and the like. The embodiments can also be implemented in distributed system environments, where local and remote computer systems linked via a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) each perform tasks (e.g., cloud computing, cloud services, and the like). In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0074] In place of or appended to other examples described herein, the examples include any combination of the following:

[0075] Clause A. A computer-implemented method comprising: receiving a high dynamic range (HDR) depth buffer; decomposing the HDR depth buffer into a low dynamic range (LVR) depth image and tile data, wherein decomposing the HDR depth buffer comprises: segmenting the HDR depth buffer into a first tile and a second tile; for each of the first and second tiles, determining a corresponding maximum and minimum value of the HDR depth buffer in a region greater than the width of the corresponding tile; and using the determined maximum and minimum depth values ​​of the HDR depth buffer in the region greater than the width of the corresponding tile to determine a depth for the HDR depth buffer. An initial pair of piecewise bilinear bounding functions; for each tile, iteratively adjusting the initial pair of piecewise bilinear bounding functions so that the corresponding minimum and maximum depth values ​​of each tile move closer to the high dynamic range depth buffer, wherein no adjacent tiles are adjusted in the same iteration; and using the adjusted pair of piecewise bilinear bounding functions, generating a low dynamic range depth buffer and tile data; compressing the low dynamic range depth buffer using a video encoder; compressing the tile data using a lossless compressor; and transmitting the compressed low dynamic range depth buffer and compressed tile data to a head-mounted device (HMD) via a data network.

[0076] Clause B. According to the method of Clause A, wherein the initial pair of piecewise bilinear bounding functions comprises a maximum bounding function and a minimum bounding function, wherein: determining the maximum bounding function comprises selecting a depth at the coordinates of each tile, the depth being at least a determined maximum depth value for the corresponding tile in the region; and determining the minimum bounding function comprises selecting a depth at the coordinates of each tile being at most a determined minimum depth value for the corresponding tile in the region.

[0077] Clause C. According to the method of Clause B, where the coordinates selected for depth for each tile in the corresponding maximum and minimum bounding functions are the corresponding center points of each tile.

[0078] Clause D. According to the method of any of the preceding clauses, the area whose width is greater than the width of the corresponding tile is twice the width of the tile.

[0079] Clause E. According to the method of any of the preceding clauses, wherein the high dynamic range depth buffer is divided into multiple rows and columns, and wherein iteratively adjusting a determined pair of piecewise bilinear bounding functions comprises: in a first round, for every tile in every other column of the multiple columns, adjusting the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth value of the high dynamic range depth buffer of the corresponding tile; in a second round, for each row in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the first round, for tiles adjacent to the tiles whose depth values ​​were adjusted in the first round, adjusting the depth values ​​of the initial pair of piecewise bilinear bounding functions so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the high dynamic range of the corresponding tile. The depth value of the depth buffer; in the third round, for each column in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the first round, for tiles adjacent to the tiles whose depth values ​​were adjusted in the first round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth value of the high dynamic range depth buffer of the corresponding tile; and in the fourth round, for each row in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the third round, for tiles adjacent to the tiles whose depth values ​​were adjusted in the third round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth value of the high dynamic range depth buffer of the corresponding tile.

[0080] Clause F. According to the method of Clause E, wherein adjusting the depth values ​​of an initial pair of piecewise bilinear bounding functions for a tile to make the depth values ​​closer to the depth values ​​of the high dynamic range depth buffer comprises: adjusting the depth values ​​of the initial pair of piecewise bilinear bounding functions until the interpolated line between adjacent tiles is tangent to the high dynamic range depth buffer.

[0081] Clause G. The method according to any of the preceding clauses, wherein the tile data includes metadata comprising a pair of maximum depth values ​​and minimum depth values ​​per tile.

[0082] Clause H. The method according to any of the preceding clauses, wherein generating the low dynamic range depth buffer comprises: for each tile, mapping the minimum depth value in the tile to the minimum value of the n-bit precision LDR depth buffer, and mapping the maximum depth value in the tile to the maximum value of the n-bit precision LDR depth buffer.

[0083] Clause I. The method according to Clause H, wherein generating the low dynamic range depth buffer further includes: scaling the depth values ​​between the minimum and maximum depth values ​​for each tile between the minimum and maximum values ​​of the n-bit precision LDR depth buffer.

[0084] Clause J. According to the method of Clause H or I, wherein the n-bit precision LDR depth buffer is an 8-bit LDR depth buffer, and the minimum value of the 8-bit LDR depth buffer is 0 and the maximum value of the 8-bit LDR depth buffer is 255.

[0085] Clause K. A computer-implemented method executed on a head-mounted device (HMD), comprising: receiving a compressed low dynamic range depth buffer and compressed tile data; decoding the compressed low dynamic range depth buffer using a video decoder and decoding the compressed tile data using a decompressor; and reconstructing a high dynamic range depth buffer from the decoded low dynamic range depth buffer and the decoded tile data using a piecewise bilinear interpolation function.

[0086] Clause L. According to the method of Clause K, the reconstructed high dynamic range depth buffer is used for the later reprojection of the color image.

[0087] Clause M. According to the method of Clause K, the reconstructed high dynamic range depth buffer is used for compositing HDR depth images with local content.

[0088] Clause N. The method according to Clauses K to M, wherein the HMD includes a hardware sampling unit on the GPU configured to perform a piecewise bilinear interpolation function.

[0089] Clause O. According to the methods of Clauses K to N, the piecewise bilinear interpolation function is performed using texture acquisition and arithmetic instructions.

[0090] Clause P. According to the method of Clauses K to O, the piecewise bilinear interpolation function is performed using texture acquisition and arithmetic instructions.

[0091] Clause Q. According to the method of Clause P, wherein reconstructing the high dynamic range depth buffer from the decoded low dynamic range depth buffer and the decompressed tile data comprises: for each tile, remapping the minimum depth value of the low dynamic range depth buffer to the minimum depth value in the tile data, and remapping the maximum value of the low dynamic range depth buffer to the maximum depth value in the tile data for the corresponding tile.

[0092] Clause R. According to the method of Clause Q, wherein for each tile, the depth value between the minimum and maximum depth values ​​of the low dynamic range depth buffer is scaled proportionally between the minimum and maximum depth values ​​included in the tile data.

[0093] Clause S. A system comprising: a remote computing device configured to: receive a high dynamic range depth buffer; decompose the high dynamic range depth buffer into low dynamic range depth images and tile data, wherein decomposing the high dynamic range depth buffer includes the remote computing device configured to: segment the high dynamic range depth buffer into a plurality of tiles; for each of the plurality of tiles, determine a corresponding maximum and minimum value of the high dynamic range depth buffer in a region greater than the width of the corresponding tile; using the determined maximum and minimum values ​​of the high dynamic range depth buffer in the region greater than the width of the corresponding tile, determine an initial pair of piecewise bilinear bounding functions for the high dynamic range depth buffer; and iteratively adjust the initial pair of piecewise bilinear bounding functions for each tile such that the corresponding minimum value and minimum value of each tile are... The maximum value is moved closer to the high dynamic range depth buffer, where no adjacent tiles are adjusted in the same iteration; and a low dynamic range depth buffer and tile data are generated using a pair of adjusted piecewise bilinear bounding functions; the low dynamic range depth buffer is compressed using a video encoder; the compressed low dynamic range depth buffer and compressed tile data are transmitted to a head-mounted device (HMD) via a data network; and the HMD is configured to: receive the compressed low dynamic range depth buffer and compressed tile data; decode the compressed low dynamic range depth buffer using a video decoder, and decode the compressed tile data using a decompressor; and reconstruct the high dynamic range depth buffer from the decoded low dynamic range depth buffer and decoded tile data using a piecewise bilinear interpolation function.

[0094] Clause T. The method according to Clause S, wherein the HMD includes a hardware sampling unit on the GPU configured to perform a piecewise bilinear interpolation function.

[0095] The terms “computer” or “computation-based device” are used herein to refer to any device having processing power to execute instructions. Those skilled in the art will recognize that such processing power is incorporated into many different devices, and therefore, the terms “computer” and “computation-based device” each include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices.

[0096] The methods described herein are, in some examples, executed by software in a machine-readable form on a tangible storage medium, for example, in the form of a computer program comprising computer program code means adapted to perform all operations of one or more methods described herein when the program is run on a computer, wherein the computer program may be specifically implemented on a computer-readable medium. The software is adapted to be executed on a parallel or serial processor, such that the method operations can be executed in any suitable order or simultaneously.

[0097] Those skilled in the art will recognize that the storage devices used to store program instructions can optionally be distributed across a network. For example, a remote computer can store an example of a process described as software. A local computer or terminal computer can access the remote computer and download part or all of the software to run the program. Alternatively, the local computer can download parts of the software as needed, or execute some software instructions at the local terminal and others at the remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to them, all or part of the software instructions can be executed by dedicated circuitry such as a digital signal processor (DSP), a programmable logic array, etc.

[0098] Any range or device value given herein may be extended or altered without losing the desired effect, as will be apparent to those skilled in the art.

[0099] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims.

[0100] It will be understood that the benefits and advantages described above may relate to one embodiment or several embodiments. The embodiments are not limited to those that solve any or all of the stated problems, nor are they limited to those that have any or all of the stated benefits and advantages. It will also be understood that a reference to “one” means one or more of those items.

[0101] The operations of the methods described herein can be performed in any suitable order, or concurrently where appropriate. Furthermore, individual blocks can be removed from any method without departing from the scope of the subject matter described herein. Aspects of any example described above can be combined with aspects of any other example described above to form further examples without losing the desired effect.

[0102] The term "subset" refers to at least one item in a set, but not all items in the set. The term "subset" does not include the empty set.

[0103] The term “comprising” is used herein to mean including the identified method block or element, but such block or element does not constitute an exclusive list, and the method or apparatus may include additional blocks or elements.

[0104] It will be understood that the above description is given by way of example only, and various modifications can be made by those skilled in the art. The foregoing specification, examples, and data provide a complete description of the structure and usage of exemplary embodiments. Although various embodiments have been described above with some degree of specificity or by reference to one or more independent embodiments, those skilled in the art can make many changes to the disclosed embodiments without departing from the scope of this specification.

Claims

1. A computer-implemented method, comprising: Receive (302) high dynamic range depth buffer (530, 730, 830); The high dynamic range depth buffers (530, 730, 830) are decomposed into low dynamic range depth buffers (216) and tile data (218), wherein the decomposition of the high dynamic range depth buffers (530, 730, 830) includes: The high dynamic range depth buffer (530, 730, 830) is divided (304) into a first tile and a second tile; For each of the first and second tiles, determine (306) the corresponding maximum and minimum values ​​of the high dynamic range depth buffer (530, 730, 830) in a region greater than the width of the corresponding tile; Using the determined maximum and minimum depth values ​​of the high dynamic range depth buffer (540, 740, 840) in the region greater than the width of the corresponding tile, determine (308) an initial pair of piecewise bilinear limit functions (540, 550) for the high dynamic range depth buffer. For each tile, the initial pair of piecewise bilinear bounding functions (310) are iteratively adjusted such that the corresponding minimum depth values ​​(540, 740, 840) and maximum depth values ​​(550, 750, 850) for each tile move closer to the high dynamic range depth buffer (530, 730, 830), wherein no adjacent tiles are adjusted in the same iteration; and The adjusted pair of piecewise bilinear bounding functions are used to generate (312) a low dynamic range depth buffer (216) and tile data (218). The low dynamic range depth buffer (216) is compressed (314) using a video encoder (216A). The tile data (316) is compressed using a lossless compressor (218A); and The compressed low dynamic range depth buffer (222) and compressed tile data (224) are transmitted (318) to the head-mounted device HMD (140, 230) via the data network (130, 220).

2. The method of claim 1, wherein the initial pair of piecewise bilinear bounding functions comprises a maximum bounding function and a minimum bounding function, wherein: Determining the maximum limit function includes: selecting a depth at the coordinates of each tile, the depth being at least a determined maximum depth value for the corresponding tile in the region; and Determining the minimum bound function includes selecting a depth at the coordinates of each tile, the depth being at most a determined minimum depth value for the corresponding tile in the region.

3. The method of claim 2, wherein the coordinates are the corresponding center point of each tile, and the depth at the coordinates is selected for each tile in the corresponding maximum bound function and the minimum bound function.

4. The method according to any of the preceding claims, wherein the region with a width greater than the width of the corresponding tile is twice the width of the tile.

5. The method according to any of the preceding claims, wherein the high dynamic range depth buffer is divided into a plurality of rows and columns, and wherein iteratively adjusting the determined pair of piecewise bilinear bounding functions comprises: In the first round, for every tile in every other column of the plurality of columns, the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth value of the high dynamic range depth buffer of the corresponding tile. In the second round, for each row in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the first round, for the tiles adjacent to the tiles whose depth values ​​were adjusted in the first round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth values ​​of the high dynamic range depth buffer of the corresponding tile; In the third round, for each column in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the first round, for the tiles adjacent to the tiles whose depth values ​​were adjusted in the first round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth values ​​of the high dynamic range depth buffer of the corresponding tile; as well as In the fourth round, for each row in which the depth values ​​of the initial pair of piecewise bilinear bounding functions were adjusted in the third round, for the tiles adjacent to the tiles whose depth values ​​were adjusted in the third round, the depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted so that the corresponding depth values ​​of the initial pair of piecewise bilinear bounding functions are closer to the depth values ​​of the high dynamic range depth buffer of the corresponding tile.

6. The method of claim 5, wherein adjusting the depth values ​​of the initial pair of piecewise bilinear bounding functions for the tile to bring the depth values ​​closer to the depth values ​​of the high dynamic range depth buffer comprises: The depth values ​​of the initial pair of piecewise bilinear bounding functions are adjusted until the interpolated line between adjacent tiles is tangent to the high dynamic range depth buffer.

7. The method according to any of the preceding claims, wherein the tile data includes metadata, the metadata including a pair of maximum depth values ​​and minimum depth values ​​per tile.

8. The method according to any of the preceding claims, wherein generating the low dynamic range depth buffer comprises: For each tile, the minimum depth value in the tile is mapped to the minimum value of the n-bit precision low dynamic range (LDR) depth buffer, and the maximum depth value in the tile is mapped to the maximum value of the n-bit precision LDR depth buffer.

9. The method of claim 8, wherein generating the low dynamic range depth buffer further comprises: The depth values ​​between the minimum and maximum depth values ​​for each tile are scaled proportionally between the minimum and maximum values ​​in the n-bit precision LDR depth buffer.

10. The method of claim 8 or 9, wherein the n-bit precision LDR depth buffer is an 8-bit LDR depth buffer, and the minimum value of the 8-bit LDR depth buffer is 0, and the maximum value of the 8-bit LDR depth buffer is 255.

11. A computer-implemented method, said computer-implemented method being performed on a head-mounted device (HMD) (140, 230), comprising: Receive (1102) a compressed low dynamic range depth buffer (222) and compressed tile data (224). The compressed low dynamic range depth buffer (222) is decoded (1104) using a video decoder (226) and the compressed tile data (224) is decoded (1104) using a decompressor (228); and The high dynamic range depth buffer (238) is reconstructed (236, 1106) from the decoded low dynamic range depth buffer (232) and the decoded tile data (234) using a piecewise bilinear interpolation function.

12. The method of claim 11, wherein the reconstructed high dynamic range depth buffer is used for post-projection of the color image.

13. The method of claim 11, wherein the reconstructed high dynamic range depth buffer is used for compositing the HDR depth image with local content.

14. The method according to any one of claims 11 to 13, wherein the HMD includes a hardware sampling unit on a graphics processing unit (GPU) configured to perform the piecewise bilinear interpolation function.

15. The method of claim 14, wherein the piecewise bilinear interpolation function is executed using texture acquisition and arithmetic instructions.

16. The method according to any one of claims 11 to 15, wherein the tile data includes metadata, the metadata including a pair of maximum depth values ​​and minimum depth values ​​per tile.

17. The method of claim 16, wherein reconstructing the high dynamic range depth buffer from the decoded low dynamic range depth buffer and the decompressed tile data comprises: For each tile, the minimum depth value of the low dynamic range depth buffer is remapped to the minimum depth value in the tile data, and the maximum value of the low dynamic range depth buffer is remapped to the maximum depth value in the tile data for the corresponding tile.

18. The method of claim 17, wherein for each tile, the depth value between the minimum depth value and the maximum depth value of the low dynamic range depth buffer is scaled proportionally between the minimum depth value and the maximum depth value included in the tile data.

19. A system (100, 200) comprising: A remote computer (110, 210, 1200), wherein the remote computer (110, 210, 1200) includes a processor (1205A) and a memory (1220), the memory (1220) storing instructions that, when executed on the processor (1205A), perform the following operations: Receive (302) high dynamic range depth buffer (530, 730, 830); The high dynamic range depth buffers (530, 730, 830) are decomposed into low dynamic range depth buffers (216) and tile data (218), wherein the decomposition of the high dynamic range depth buffers (530, 730, 830) includes the remote computers (110, 210, 1200) being configured as follows: The high dynamic range depth buffer is divided (304) into multiple tiles; For each of the plurality of tiles, determine (306) the corresponding maximum and minimum values ​​of the high dynamic range depth buffer in a region greater than the width of the corresponding tile; Using the determined maximum and minimum values ​​of the high dynamic range depth buffer in the region greater than the width of the corresponding tile, determine (308) an initial pair of piecewise bilinear bounding functions (540, 550) for the high dynamic range depth buffer (530, 730, 830). For each tile, the initial pair of piecewise bilinear bounding functions (310) are iteratively adjusted so that the corresponding minimum (540, 740, 840) and maximum (550, 750, 850) of each tile move closer to the high dynamic range depth buffer (530, 730, 830), wherein no adjacent tiles are adjusted in the same iteration; as well as The adjusted pair of piecewise bilinear bounding functions are used to generate (312) a low dynamic range depth buffer (216) and tile data (218). The low dynamic range depth buffer (216) is compressed (314) using a video encoder (216A). The tile data (218) is compressed (316) using a lossless compressor (218A); as well as The compressed low dynamic range depth buffer (222) and compressed tile data (224) are transmitted (318) to the head-mounted device HMD (140, 230) via data networks (130, 220); and The HMD (140, 230), wherein the HMD (140, 230) has a processor (1205A) and a memory (1220), the memory (1220) storing instructions that, when executed on the processor (1205A) of the HMD (140, 230), perform the following operations: Receive (1102) the compressed low dynamic depth buffer (222) and the compressed tile data (224). The compressed low dynamic range depth buffer (222) is decoded (1104) using a video decoder (226), and the compressed tile data (224) is decoded (1104) using a decompressor (228); and The high dynamic range depth buffer (238) is reconstructed from the decoded low dynamic range depth buffer and the decoded tile data using a piecewise bilinear interpolation function (236, 1106).

20. The system of claim 19, wherein the HMD includes a hardware sampling unit on the GPU configured to perform the piecewise bilinear interpolation function.

Images