Integrated circuit, device, computer-implemented method for estimating depth map and computer-readable medium.

Abstract

An integrated circuit and computer-implemented method are provided for estimating an image depth map using a bilateral joint filter with reduced computational complexity. For this purpose, the image data is also accessed as depth data from a template depth map. A bilateral joint filter is then applied to the template depth map using the image data as a range term in the bilateral joint filter, thus obtaining an adapted image depth map as output.The application of the joint bilateral filter involves initializing a weighted depth sum volume and a weight sum volume as respective empty data structures in memory, performing a splatting operation to populate said volumes, performing a slicing operation to obtain an adapted image depth volume, and performing an interpolation operation to obtain an adapted image depth value from the adapted image depth map for each pixel in the image. Compared to known methods of estimating depth maps from an image using a joint bilateral filter, reduced computational complexity is achieved.

Description

1 / 32 Integrated circuit, device, computer-implemented method for estimating depth map and computer-readable medium. Field of Invention

[0001] The invention relates to an integrated circuit configured to estimate a depth map from an image using a joint bilateral filter. The invention further relates to a method for estimating a depth map from an image using a joint bilateral filter, and to a computer-readable means comprising transient or non-transient data representing instructions arranged to cause a processor system to perform the method. Previous Technique

[0002] Display devices, such as televisions, tablets, and smartphones, may comprise a 3D display to provide a user with a perception of depth when viewing content on such a device. For this purpose, such 3D displays may, alone or in conjunction with glasses worn by the user, provide the user with different images in each eye, so as to provide the user with a perception of depth based on stereoscopy.

[0003] 3D monitors typically require content that contains depth information. Depth information can be provided implicitly in 3D content. For example, in the case of stereoscopic content, also referred to in short as stereo content, depth information is provided by the differences between a left and a right image. Depth information can also be provided explicitly in 3D content. For example, in 3D content encoded in the so-called image+ format. Petition 870250033491, dated 04 / 28 / 2025, page 11 / 93 2 / 32 Depth: Depth information is provided by a depth map that may include depth values, disparity values, and / or parallactic shift values, with each of said values ​​indicating the distance that objects in the image have from the camera direction.

[0004] A large amount of content that is currently available, for example, films, television programs, images, etc., is 2D content. Such content needs to be converted to 3D in order to allow its presentation in 3D on a 3D monitor. 3D conversion may involve generating a depth map for a 2D image, for example, for each 2D image in a 2D video. In general, this process is referred to as 2D-to-3D conversion, and may involve creating the depth map manually. For example, a software tool running on a workstation may offer a professional user the possibility of adding depth to 2D images by drawing depth maps using a digital pen. The depth map can also be generated automatically. For example, a device can estimate the distance between objects in the 2D image and the camera and, based on this, generate a depth map for the 2D image.

[0005] An example of automatic depth map generation is known from US 8,447,141, which describes a method for generating a depth map for an image using monocular information. The method comprises generating a first depth map for the image that provides a global depth profile, which may be a simple generic template, such as a slope. Furthermore, a second depth map is generated based on the depth values ​​of the first depth map and the color and / or luminance values ​​of the image. A Petition 870250033491, dated 04 / 28 / 2025, page 12 / 93 3 / 32 Generation of the second depth map may involve applying a joint bilateral filter to the first depth map, using image banding information. It should be noted that as a result, objects will become more distinct from the overall depth profile.

[0006] Indeed, US 8,447,141 uses the joint bilateral filter to adapt the generic template provided by the depth map to the actual image content.

[0007] However, the implementation of a joint bilateral filter is computationally complex. The publication “A fast approximation of the bilateral filter using a signal processing approach”, by Paris et al., International Journal of Computer Vision 81.1, 2009, pp. 24-52 describes an approximation of the joint bilateral filter that is considered less computationally complex. Disadvantageously, the described approximation of the bilateral filter is still relatively computationally complex, and, therefore, is not well suited for economical implementation in consumer devices, for example, in an integrated circuit. Summary of the Invention

[0008] One of the objectives of the invention is to obtain an integrated circuit configured to estimate a depth map of an image using a joint bilateral filter, which is computationally less complex than the approximation described by Paris et al.

[0009] A first aspect of the invention provides an integrated circuit configured to estimate a depth map of an image, wherein the integrated circuit is defined by claim 1. A further aspect of the invention provides a computer-implemented method for estimating a depth map from an image, wherein the method is defined by claim 14.

[00010] Essentially, the integrated circuit implements the filter Petition 870250033491, dated 04 / 28 / 2025, page 13 / 93 The bilateral 4 / 32 set is a more approximate solution, but in a different way from Paris et al. One of the advantageous differences is that Paris et al. first perform an interpolation and then a slicing operation, with the latter involving division. The division in Paris et al. is thus performed on the data having been interpolated back to full resolution. The integrated circuit as claimed performs a slicing operation on the reduced volumes, with very reduced resolution. For example, if volumes of, for example, 18 x 12 x 18 (height x width x stripe) are used, only 3,888 division operations are performed, for example, on a 480 x 270 image (129,600 division operations). The reduction in quality when performing the slicing operation on the reduced volumes has been found to be very limited.However, since division operations are computationally complex to implement, the computational weight in performing joint bilateral filtering is very low. This allows the union operation to be performed in software, which also results in flexibility regarding its implementation.

[00011] It is noted that the depth map, which is used as input, can be a template depth map, and thus correspond to a predetermined generic depth profile, such as a slope, or a generic depth profile that is selected from a list of different generic depth profiles that best match the image content. However, in general, the depth map can be any depth map that benefits from being further adapted to the image.

[00012] It is also noted that volumes can be three-dimensional volumes, for example, consisting of reduced two-dimensional spatial versions of the image and a version Petition 870250033491, dated 04 / 28 / 2025, page 14 / 93 5 / 32 reduced from the band dimension of one of the image components, for example, the luminance component. Alternatively, volumes may comprise two or more band dimensions, which may correspond to two or more of the image components, for example, the luminance components and one or more chrominance components of a YUV image, or the individual color components of an RGB image, etc.

[00013] Optionally, the processing subsystem comprises an application-specific hardware circuit and a software-configurable microprocessor, where: - The application-specific hardware circuit is configured to perform the splatting operation and the interpolation operation; and The microprocessor is configured by the software to perform the slicing operation during the integrated circuit's operation.

[00014] This aspect of the invention is based on the view that, while the splatting operation and the interpolation operation are both still relatively complex in computationally, the slicing operation has been significantly reduced in complexity compared to Paris et al., by operating only on reduced volumes. As such, the slicing operation can be performed in software, resulting in flexibility, whereas the splatting operation and the interpolation operation can be performed in hardware in view of the generally greater efficiency of a hardware implementation versus software.

[00015] Optionally, the application-specific hardware circuit comprises a filter table to store the splatting weights used in the splatting operation and / or the interpolation weights used in the interpolation operation. In the case of a Petition 870250033491, dated 04 / 28 / 2025, page 15 / 93 6 / 32 Since the splatting and interpolation operations are hardware operations, said weights can be stored in a filter table of the hardware circuit, which can be a read-only memory or a random access memory. This allows said weights to be readily accessible by the hardware of the respective operations.

[00016] Optionally, the filter table is loaded with the splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation before performing the respective operation. In the case of different weights being used for the splatting operation instead of for the interpolation operation, these weights can be located within the filter table before the respective operation is performed, for example, by the microprocessor. In this way, the filter table hardware is reused and not implemented separately for both the splatting operation hardware and the interpolation operation hardware. It is noted that in addition to or instead of reusing the filter table hardware between the splatting and interpolation operations, there may also generally be hardware reuse to determine the relative position of a sample within a compartment between the splatting and interpolation operations.

[00017] Optionally, the splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation are the same. This aspect of the invention is based on the view that the same weights can be used to perform the splatting operation as well as to perform the interpolation operation, since both are essentially similar operations that, in the present context, also operate, on the one hand, on the image, and on the other hand, on a multidimensional volume that has Petition 870250033491, dated 04 / 28 / 2025, page 16 / 93 7 / 32 reduced dimensions compared to the two special dimensions and the image strip dimension. By using the same weights, the implementation of both operations is considerably more efficient.

[00018] Optionally, the splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation represent linear interpolation with respect to the image coordinate system. The weights are thus chosen so that linear interpolation is applied to the data in the respective volumes of the splatting operation and the interpolation operation along each of the volume dimensions. For example, for a three-dimensional volume, trilinear interpolation can be used, while for a four-dimensional volume, for example, having two strip dimensions, tetralinear interpolation can be used. Linear interpolation was considered well suited for both the splatting and interpolation operations. Alternatively, higher-order interpolation can be used in each dimension, including but not limited to cubic interpolation.

[00019] Optionally, the microprocessor is configured by software to apply temporal filtering to the weighted depth sum volume and the weight sum volume during integrated circuit operation before performing the slicing operation. Temporal filtering is frequently applied to depth maps to ensure temporal stability. Instead of, or in addition to, applying such temporal filtering to the depth map, the volumes used to generate the depth map itself can be temporally filtered. This has been responsible for yielding a more temporally stable depth map, while Petition 870250033491, dated 04 / 28 / 2025, page 17 / 93 8 / 32 computationally efficient to implement on the relatively small size of volumes, compared to an image. For example, a typical 18 x 12 x 18 volume contains 3,888 data values ​​to be filtered, whereas a 480 x 270 image contains 129,600 data values ​​to be filtered. Due to the filtering specifically applied to volumes, a software implementation is possible. Advantageously, it has been found that applying an Infinite Impulse Response (IIR) filter to the weighted depth sum volume improves the temporal stability of the resulting depth map in the case of so-called shot-cuts, without the need for dedicated shot-cut processing, for example, involving dedicated shot-cut detectors. Temporal filtering can, for example, be a higher-order infinite impulse response filter, and can be implemented as part of other (non-linear) operations on the volume data.

[00020] Optionally, the microprocessor is configured by the software to apply temporal filtering to the adapted image depth volume during integrated circuit operation. Such temporal filtering offers comparable reductions in computational complexity compared to filtering a real image, as described with respect to weighted depth sum volume filtering or weight sum volume filtering. The temporal filtering can be of the same type as that described with respect to weighted depth sum volume filtering or weight sum volume filtering.

[00021] Optionally, the processing subsystem is configured to, after performing the splatting operation, perform convolution of the weighted depth sum volume with a Gaussian kernel. Such convolution has been found to improve the quality of the depth map. Petition 870250033491, dated 04 / 28 / 2025, page 18 / 93 9 / 32

[00022] Optionally, the splatter depth map has a reduced spatial resolution compared to the image, for example, having two spatial dimensions that correspond to two spatial dimensions of the volume of the sum of the weighted depths and the volume of the sum of the weights. The splatter depth map can then be interpolated using the same weights as those used in the splatting operation.

[00023] Optionally, the joint bilateral filter is applied only to the luminescence data of the image. Optionally, the joint bilateral filter is applied to both the luminescence and chrominance data of the image. In the latter case, the volumes can be five-dimensional volumes possessing three band dimensions, for example, a Y, U, and V dimension.

[00024] Optionally, the integrated circuit is, or forms part of, a field-programmable gate array or a system-on-a-chip. The integrated circuit may be part of a device, such as a display device or a decoder box, but also other devices in which it may be used to convert 2D video into 3D video by estimating the depth map.

[00025] A further aspect of the invention provides a computer-readable means comprising transient or non-transient data representing instructions arranged to cause a processor system to perform the method.

[00026] Those skilled in the art will appreciate that two or more of the embodiments, implementations and / or aspects of the invention mentioned above can be combined in any way deemed useful.

[00027] Modifications and variations of the method that correspond to the described modifications and variations of the integrated circuit may be carried out by those skilled in the art based on the present description. Petition 870250033491, dated 04 / 28 / 2025, page 19 / 93 10 / 32 Brief Description of the Drawings

[00028] These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described later herein. In the drawings,

[00029] Figure 1 schematically illustrates an integrated circuit that is configured to estimate a depth map of an image in a computationally efficient manner;

[00030] Figure 2 illustrates the relationship between an image and a volume that can be used to calculate weighted weight and depth values ​​during a splatting operation and that can maintain depth values ​​for an interpolation operation;

[00031] Figure 3A illustrates a simplified example of a splatting operation;

[00032] Figure 3B illustrates a detailed example showing compartment addressing and associated weights for both the splatting operation and the interpolation operation;

[00033] Figure 4 illustrates various aspects of the splatting operation;

[00034] Figures 5 to 7 illustrate various aspects of the interpolation operation;

[00035] Figure 8 illustrates a method for estimating a depth map of an image; and

[00036] Figure 9 illustrates a computer-readable medium comprising instructions for causing a processor system to perform the method.

[00037] It should be noted that items that have the same numerical references in different figures have the same structural characteristics and the same functions, or are the same symbols. Where the function and / or structure of such an item has been explained, there is no need to reproduce the explanation in the detailed description. Petition 870250033491, dated 04 / 28 / 2025, p. 20 / 93 11 / 32 List of references and abbreviations

[00038] The following list of references and abbreviations is provided to facilitate the interpretation of the drawings and should not be considered part of the claims. Image data, depth volume input data, depth output data, gauge depth data, interpolated gauge depth data, sum of weighted volume data, sum of weighted depth volume data, weight and volume index data, weight data, integrated circuit processing subsystem, image data input interface, depth volume data input interface, depth data output interface, volume data output interface, splatting block, weighting block, interpolation block, 2D interpolation block, control logic, image horizontal dimension, vertical dimension, image light part mapping, image dark background mapping, image volume representation horizontal dimension. Petition 870250033491, dated 04 / 28 / 2025, page 21 / 93 12 / 32 320 vertical dimension 330 strip dimension 400 dimension (horizontal, vertical or strip) 410 series of depth samples 420 splat accumulation interval 430 weight function 440 compartment interval 442 edge compartment 444 non-border compartment 500 method for estimating the depth map from the image 510 Access image data 520 Access depth data 530 apply joint bilateral filter 540 initialize volumes 550 splatting operation 560 slicing operation 570 interpolation operation 600 computer-readable media 610 instructions representing non-transient data. Detailed Description of the Modalities

[00039] Figure 1 schematically illustrates a processing subsystem 100 of an integrated circuit configured to computationally estimate a depth map from an image. The processing subsystem 100 is illustrated as a functional block diagram and may consist of components such as a microprocessor, an application-specific hardware circuit, and one or more local memories. The integrated circuit may comprise other components not illustrated in Figure 1, including but not limited to... Petition 870250033491, dated 04 / 28 / 2025, page 22 / 93 13 / 32 limited to other microprocessors, a bus system, other memories, etc. In general, the integrated circuit can be implemented as, or as part of a field-programmable gate array (FPGA), or a System on a Chip (SoC), or in any other suitable way.

[00040] The processing subsystem 100 is illustrated to comprise an image data input interface 110 through which image data 010 can be read from a memory, for example, through Direct Memory Access (DMA) communication. For example, the image data can be Yin luminescence input data. In this regard, it is noted that the image data input interface 110 and other interfaces of the processing subsystem 100 can comprise or can be connected to a local memory that acts as the store, labeled Buf throughout Figure 1.

[00041] The processing subsystem 100 is further illustrated to comprise a depth volume data input interface 120 through which depth volume data 020 can be read from memory, and a depth data output interface 122 through which depth data 022 can be written to memory. The processing subsystem 100 is further illustrated to comprise respective volume data output interfaces 130, 132 for writing the weighted volume data sum 030 and the weighted depth data sum 032 to memory.

[00042] Other functional blocks of the processing subsystem 100 include a splatting block 140 that communicates with the volume data output interfaces 130, 132, an interpolation block 160 that communicates with the depth data interfaces 120, 122, a weighting block 150 that communicates Petition 870250033491, dated 04 / 28 / 2025, p. 23 / 93 14 / 32 with image data input interface 110 and provides weight and volume index data 052 for splatting block 140 and interpolation block 160, a 2D interpolation block 170 that receives gauge depth data 024 from control logic 180 and weight data 054 from weighting block 150 and provides interpolated gauge depth data 026 for splatting block 140.

[00043] In one embodiment, the processing subsystem 100, as illustrated in Figure 1, can be implemented as an application-specific hardware circuit. Alternatively, a separate microprocessor (not illustrated in Figure 1) can be provided in the integrated circuit that is configured to perform a slicing operation, as discussed further. For this purpose, the microprocessor can have access to memory, for example, via DMA. Alternatively, the slicing operation can also be implemented by the application-specific hardware circuit.

[00044] The operation of processing subsystem 100 and its functional blocks will be explained further with reference to figures 2 to 7. In this regard, it is noted that in figure 1 * denotes a local storage or an interface that is associated with a volumetric multidimensional data representation, as explained with reference to figure 2.

[00045] Figure 2 illustrates the relationship between a 200 image and a 300 volume, with this type of volume being used to accumulate weight values ​​and weighted depth values ​​during a splatting operation and maintaining the pixel depth value of the 200 image for an interpolation operation. The 300 volume is illustrated as having three dimensions. The X dimension 310 can correspond to the horizontal geometric axis 210 of the 200 image. The Y dimension 320 can correspond to the vertical geometric axis 220 of the Petition 870250033491, dated 04 / 28 / 2025, p. 24 / 93 15 / 32 image 200. The I 330 dimension may correspond to the image's band dimension, for example, of an image component of image 200, such as, but not limited to, a luminance component. The 300 volume may have a size that is subsampled with respect to the spatial dimensions and band dimensions of image 200. For example, while image 200 may have spatial dimensions of 480 pixels by 270 lines (X, Y) and an 8-bit band dimension (I) that defines a band of 256 values, the 300 volume may have dimensions of 18 x 12 x 18 (X, Y, I), that is, 18 x 12 (X, Y) for the spatial dimensions 310, 320 and 18 (I) for the band dimension 330.

[00046] The cells of volume 300 can represent compartments. During a splatting operation, such compartments can define accumulation intervals for the accumulation of depth information from image 200. Here, the compartments used in accumulating a weight or weighted depth value associated with a particular pixel are selected as a function of the pixel's spatial coordinate and its range value. For example, the luminance values ​​of image 200 can determine the compartment coordinate along dimension I 330 of volume 300, since the depth information of dark image content can be accumulated in the lower compartments of volume 300, as illustrated by arrow 260, while the depth information of light image content can be accumulated in higher compartments of volume 300, as illustrated by arrow 250.Furthermore, the spatial location of the image content can define the compartment's location along the spatial dimensions 310, 320 of the volume 300. Accordingly, the combination of the spatial coordinate and the range value of a pixel in the image 200, that is, the coordinate of pixel no. Petition 870250033491, dated 04 / 28 / 2025, page 25 / 93 16 / 32 3D coordinate system (at least) of the image, can determine in which of the 300 volume bins a weight value or a weighted depth value is accumulated during a splatting operation, or which bins hold the depth value of a pixel for interpolation by an interpolation operation.

[00047] This determination of compartments can essentially involve mapping the spatial coordinate and pixel range value to a coordinate in volume coordinate space and based on the relative position in the volume identification compartments to be used during splatting and interpolation. Effectively, during splatting, adjacent compartments can be identified to determine which compartments the pixel contributes to based on its splatting footprint (with the contribution being the accumulation of weight value or weighted depth value), while during interpolation, adjacent compartments can be identified to determine between which compartments the interpolation should be performed based on the pixel's relative position within a volume.For this purpose, a mapping function can be used by mapping the pixel's spatial coordinate and range value to a coordinate in the volume coordinate space, with the latter coordinate then directly indicating the adjacent compartments.

[00048] Due to undersampling, multiple pixels of the 200 image can contribute to a single 300 volume compartment, since their depth values ​​can, at least in part, be accumulated in the singular compartment. Conversely, a single pixel can contribute to multiple 300 volume compartments since its coordinates in the volume coordinate system can lie between several cells of the 300 volume. As such, Petition 870250033491, dated 04 / 28 / 2025, p. 26 / 93 17 / 32 when accumulating pixel depth value in volume 300, the depth value may need to be weighted to compensate for the contribution to multiple cells in volume 300. This is also referred to as splatting.

[00049] Figure 3A illustrates such a splatting operation along a single dimension K 400. This dimension K can be the horizontal (X), vertical (Y), or luminance (l) dimension. For this explanation, the dimension K 400 is considered to represent the luminance dimension l. First, this dimension is divided into bins 440, for example, by subsampling the luminance dimension l, for example, from 256 values ​​(8 bit, 0 to 255) down to sixteen bins of width 16. As is known per se, each bin is a storage element that can hold a particular value or a sum of accumulated values. It is noted that for illustrative purposes, and in particular to improve the visibility of the figures, in figure 3A, and below, a compartmentalization is illustrated and involves only twelve compartments in total: ten normal compartments [1] to

[10] and two so-called edge compartments [0] and

[11] will be explained with further reference to figure 3B.

[00050] The following illustrates the splatting operation with reference to a histogram operation. A conventional histogram can be obtained as follows: for each pixel of an image, it can be determined within which singular compartment its luminance value lies. Then, the value of that compartment can be incremented, for example, by 1. As a result, the relative position of the luminance value, with respect to the luminance range associated with the compartment, can be irrelevant. For example, if a compartment defines a luminance range of [0 7] for accumulation, all values ​​of Petition 870250033491, dated 04 / 28 / 2025, page 27 / 93 18 / 32 luminescence values ​​found within this compartment can cause the same increase, that is, by 1, regardless of whether the luminescence value is located within the center of the compartment (for example, luminescence values ​​3 and 4) or on an edge of the compartment (for example, luminescence values ​​0 and 7).

[00051] Splatting techniques can be used to obtain a better, for example, more accurate histogram representation. That is, the relative position of a luminescence value within a bin can be taken into account by weighting. In such splatting techniques, the contribution of the pixel being splatted can be determined by the explicit or implicit assignment of a footprint to the pixel's coordinate along the luminescence dimension, for example, for the luminescence value. An accumulation by splatting can be performed as follows: for a pixel's luminescence value, it is determined to which adjacent bins the pixel contributes, with contribution referring to a bin that lies at least partly within the pixel's footprint. The values ​​in the adjacent bins can then be incremented by the respective weights that depend on the relative position of the luminescence value with respect to the two bins.For example, when the luminescence value falls centrally within a given compartment, the contribution to that compartment may be high, while the contribution to the preceding (lower) or next (upper) compartment may be low. Similarly, when a luminescence value lies between two compartments, the distribution for each compartment may be half the high value.

[00052] Position-dependent weighting mentioned above Petition 870250033491, dated 04 / 28 / 2025, page 28 / 93 19 / 32 can substantiate such a contribution based on footprint for the compartments. It is noted that since weighting only defines a contribution to a compartment within a particular range, this range can also be considered as representing the accumulation range of the particular compartment. For example, the accumulation range of the present (or current) compartment can be considered to include the present compartment while also extending halfway into the previous and next compartments. In this way, starting halfway into the previous compartment, the contribution to the present compartment can slowly increase from zero to a maximum in a centralized manner within the present compartment and then slowly decrease to zero halfway into the next compartment.

[00053] As a result of using the splatting operation, the accumulated values ​​in the bins can provide a more accurate representation of a histogram.

[00054] In a specific and efficient implementation of the splatting operation, the footprint is considered to contribute at most to two adjacent compartments, for example, with a size that corresponds to the size of one compartment or smaller. In this case, a pixel contributes at most to compartment [n] and compartment [n + 1], with n being a compartment index or coordinate in the volume's coordinate system. The first compartment can then be referred to as the present compartment and the second compartment can be referred to as the next compartment.

[00055] A particularly specific and efficient implementation defines the accumulation intervals of the bins, such as the contribution of a pixel within the present bin serves Petition 870250033491, dated 04 / 28 / 2025, page 29 / 93 20 / 32 only for the current compartment, the next compartment. This implementation, however, can be considered as having a mid-compartment offset, where it would be intuitively understood that the pixel is located. That is, in this specific implementation, a maximum contribution to a compartment is not obtained in the middle of the compartment, but at its lowest limit. One reason for defining the compartments in this way is to allow greater hardware reuse between the splatting operation and the interpolation operation, for example, when considering the calculation and storage of weights and / or the calculation of relative positions with respect to a compartment.

[00056] As an example, consider in Figure 3A the case of a luminescence value p0, which is a luminescence value on the left side of compartment [5]. This relative position can contribute a high weight to compartment [5], according to the weighting function 430 (dashed line being at / near maximum) and a low weight to compartment [6] (dashed line being at / near zero). Next, consider the case of a luminescence value p1, which is in the middle of compartment [5]. This relative position can contribute an equal weight (half the high weight) to compartments [5] and [6]. Finally, consider the case of a luminescence value p2 on the right side of compartment [5]. This relative position can contribute a low weight to compartment [5] and a high weight to compartment [6].Accordingly, the relative position of the luminescence value can determine the weight by which the luminescence value is accumulated in the current and nearby compartments. The example in Figure 3A thus illustrates a linear fading from a maximum contribution to compartment [5] to a maximum contribution to compartment [6], depending on the value. Petition 870250033491, dated 04 / 28 / 2025, page 30 / 93 21 / 32 of luminescence.

[00057] It will be appreciated that in this specific and efficient implementation in which a pixel, possessing a luminance value within the present compartment, contributes only to the current compartment and the next compartment, the accumulation interval associated with compartment [5] can be the interval that encompasses compartment [5] and its previous compartment, for example, the interval that corresponds to the dashed line covering compartments [4] and [5] in Figure 3A. Similarly, the accumulation interval associated with compartment [6] can be the interval that encompasses compartment [6] and its previous compartment, for example, the interval that corresponds to the dashed line covering compartments [5] and [6] in Figure 3A.

[00058] Figure 3B illustrates more details of a similar modality, but applied to the accumulation of weighted depth values ​​and weights in the respective volumes, that is, the volume of weighted depth sum and the volume of the sum of the weights mentioned above.

[00059] In this example, both volumes have a fixed maximum size of 18 x 12 x 18 compartments (X, Y, I) regardless of the image size, while the actual number of compartments can vary. That is, a sizeBinK parameter can be used in the splatting operation to define the size of a non-border 444 compartment and thus determine how many compartments are used. This size can be a power of 2 to reduce implementation complexity. The size of the two compartments at the edge of an edgeBinK dimension can vary to allow any dimension size value. Figure 3B illustrates a bottom edge 442 compartment. For example, if the image width is 480 and sizeBinX is Petition 870250033491, dated 04 / 28 / 2025, page 31 / 93 If 22 / 32 is selected as equal to 32, there can be 14 non-border bins and 2 border bins, each with a width of (480 - 14*32) / 2 = 16. In another example, if the image height is 270 and sizeBinY is 32, the width of each border bin can be (270 - 8*32) / 2 = 7. Therefore, using border bins allows non-border bins to have a size equal to a power of 2, thus simplifying implementation complexity. That is, normalization in fixed-point arithmetic can be performed by a simple shift operation. Furthermore, using two border bins (at the top and bottom) has shown better filtering results since the border bins coincide with the image edges. It may, therefore, be desirable to intentionally have border bins of a specific size.For example, it has been found that edge compartments that are half the size of a regular non-edge compartment may be preferable, provided they are at least ¼ the size of a regular compartment and at most ¾ the size of a regular compartment.

[00060] Figure 3B further illustrates an example of compartment addressing and associated weights for both splatting and depth interpolation. For ease of understanding, Figure 3B illustrates a single dimension only. This dimension can be the X, Y of the image position, or the range dimension. Specifically, Figure 3B illustrates the position 410 of the depth samples at the indicated index positions [0]...

[12] , for example, corresponding to the sample positions along a line on the input depth map, for example, as obtained from the 024 gauge depth data in Figure 1. For each index position, a corresponding splatting range 420, a weight Petition 870250033491, dated 04 / 28 / 2025, p. 32 / 93 23 / 32 determined by a weight function 430 for splatting and depth profile interpolation (indicated by numerical reference 170 in Figure 1), and a compartment interval 440 are illustrated. Figure 3B should be interpreted as follows. For a given index position, for example, index position [7], an accumulation interval is defined, denoted by the same numerical reference [7], in addition to a corresponding weight function 430 which is illustrated in the same line style as the accumulation interval [7]. The weight function 430 represents a weight f that transitions linearly from 0 from the edges of the accumulation interval [7] to 1 at the center of the accumulation interval [7].Within the same accumulation interval [7], also a weight 1-f is illustrated which, in the left half of the accumulation interval [7], represents a weight for the depth sample [6] in its half-overlap accumulation interval [6], and in the right half of the accumulation interval [7] represents a weight for the depth sample [8] in its half-overlap accumulation interval [8].

[00061] It can be observed that the accumulation intervals and weights are selected so that, for a given position p along the illustrated dimension, the weight sum volume compartments can be accumulated according to SW[x] + = f and SW[x+1] + = (1-f), while the weight sum depth volume compartments can be accumulated according to SWD[x] + = f*dp and SWD[x+1] + = (1-f)*dp. Here, the position p determines the compartment x, being [6] in this example, and the depth value dp is obtained by interpolating the depth profile (170 in Figure 1) from the depth map D according to dp = (1f) * D[x] + f*D[x+1]. It is noted that this is only for a single dimension; typically, the addressing of the compartments is based on spatial dimensions X, Y and one or more dimensions of Petition 870250033491, dated 04 / 28 / 2025, page 33 / 93 24 / 32 image band I, and, therefore, it typically also depends on the image data itself.

[00062] It is noted that the weights fe (1-f) can be computed as fixed-point values ​​with normalized expressions. For example, in the case of 3 bits after the binary point, the value of 8 represents 1 and, in this way, f is a fixed [0..8]. In a specific example, if it is assumed that dp = (f-1)*D[x] + f* D[x+1] must be computed, with D[x] being equal to 24, D[x+1] being equal to 8 and f being equal to 6, dp can be computed as ((8-6)*24 + 6*8) / 8 = 96 / 8 -12. The division by 8 is a normalization step. This example is also illustrated in the table in Figure 5 at position Y 10. In this and other figures and throughout the text, wny and wpy represent the fixed-point values ​​of fe (1-f) before this normalization.

[00063] In a specific and efficient embodiment, the maximum weight may correspond to the size of the compartments, for example, 8 for a compartment size equal to 8. Accordingly, each step in the luminescence value results in a step in weight for each of the two adjacent compartments. Similarly, each step in the (x or y) position results in a step in weight for each of the two respective adjacent compartments.

[00064] Figure 4 illustrates the splatting operation supplying the sum of weighted depths (SWD) and sum of weights (SW) along a single dimension. In this example, the vertical geometric axis Y is chosen, with the bin size being equal to 8 (sizeBinY) and the first bin having a size equal to 4 (edgeBinY). positionY is simply the row number. The table in Figure 4 illustrates the interpolation with respect to the rows in binY with index 1. The factorY is the relative position of a row within a bin. Based on the value of factorY, two Petition 870250033491, dated 04 / 28 / 2025, page 34 / 93 25 / 32 wpy and wny weights are derived, and relate to the complementary weights fe 1-f, as described for Figure 3B, but now expressed as a wny weight for the next bin and as a wpy weight for the present bin, with next and present referring to two consecutive and spatially adjacent bins. Splatting is essentially a sampling reduction function whereby a high-resolution input is transformed into a low-resolution output. In this example, all lines within binY number 1 have an input depth value equal to 20. Depending on the line position, the weight gradually changes from bin 1 to bin 2. As such, at line number 4 (which is the first line in bin 1), the full value is accumulated in bin 1 and no value is added to bin 2, which is illustrated in Figure 4 by the two arrows.In line number 8 (halfway through the compartment), 50% of the value is added to compartment 1 and 50% of the value is added to compartment 2. In this way, a linear fade between the compartments is achieved. Using weights, the cells of the SW volume (sum of weights) only accumulate the applied weights, while the cells of the SWD volume (sum of weighted depths) accumulate the depth value multiplied by the weight (weighted depth).

[00065] In short, the splatting operation can involve, for each pixel in the image, determining a pixel coordinate in the image coordinate system, for example, in the form of a (X, Y, I) coordinate, a (X, Y, R, G, B) coordinate, or a (X, Y, I, U, V) coordinate. In the latter, (I, U, V) refers to the components of a YUV signal, with the Y (luminescence) component being referred to as I (Intensity) to distinguish it from the dimension. Petition 870250033491, dated 04 / 28 / 2025, page 35 / 93 26 / 32 spatial Y. It can then be determined which adjacent cells in the weighted sum volume represent accumulation intervals associated with the pixel. A pixel depth value can be obtained from the depth map, possibly using interpolation if the depth map has a lower spatial resolution than the image. For each of the adjacent cells, a weight can be obtained to weight the depth value. The weight can be (pre)calculated based on a relative position of the pixel with respect to the accumulation interval of a respective cell as indicated by the coordinate. The depth value can then be weighted by the weight and accumulated in the respective cell of the weighted depth sum volume, with the weight itself being accumulated in a corresponding cell of the weighted sum volume.

[00066] It is noted that for a single dimension, a linear interpolation requires 2 values. Similarly, for two dimensions, a bilinear interpolation requires 4 values. For a 3-dimensional volume, a trilinear interpolation uses 8 values. The weights can be pre-calculated values ​​as a function of the relative position of a sample within a compartment. In the case where the depth map has a reduced spatial resolution with respect to the image, and in particular, the same reduced spatial resolution as the volume of the sum of weighted depths and the volume of the sum of weights, the depth map can be interpolated before the splatting operation at the image resolution using the same weights as those used in the splatting operation. This is illustrated in Figure 1 by the weighting block 150 which provides weight data 054 for the 2D interpolation block 170 which interpolates the depth template to the image resolution.

[00067] Having performed the splatting operation, an operation of Petition 870250033491, dated 04 / 28 / 2025, page 36 / 93 27 / 32 Slicing can be performed to obtain an adapted image depth volume. This operation can be performed by a software-configured microprocessor, which is not explicitly shown in Figure 1 but may have access to memory, i.e., via DMA communication. The slicing operation may involve, for each cell of the weighted depth sum volume and the corresponding cell of the weight sum, dividing the accumulated weighted depth values ​​by the accumulated weights. As a result of this division, each new bin contains an adapted image depth value with the overall volume representing an adapted image depth volume.

[00068] Having performed the slicing operation, an interpolation operation can be performed to obtain an adapted image depth value from the adapted image depth map for each pixel in the image. This interpolation operation may comprise identifying adjacent cells in the adapted image depth volume based on said cells representing accumulation intervals for the pixel in the weighted depth sum volume based on the pixel coordinate, and applying an interpolation filter to the adjacent cells of the adapted image depth volume, where the interpolation filter comprises, for each of the cells, a weight that is determined based on the relative position of the pixel with respect to the accumulation interval of a respective cell, as indicated by the coordinate.In other words, the position of a pixel, as determined by its spatial coordinate and range values, can determine the bins to be used in interpolation, while the relative position of a pixel within a bin can determine the interpolation weights. The weights. Petition 870250033491, dated 04 / 28 / 2025, page 37 / 93 28 / 32 can be pre-calculated. In particular, the weights can be the same weights used in the splatting operation, and / or the same hardware circuit can be used to store or calculate the weights.

[00069] Figure 5 illustrates the interpolation operation in a manner similar to that of Figure 4 for the splatting operation. Here, weights are used to interpolate an output depth value from the adapted image depth volume. As can be observed when comparing Figure 5 with Figure 4, the same heights are used for a particular relative position within a compartment. This is illustrated in Figure 1, by the weighting block 150 which provides the same weight and volume index data 052 for the interpolation block 160 as for the splatting block 140. The right side of the table illustrates how the weights are applied to input depth values ​​24 and 8. For example, interpolation at Y position 7 results in ((5 x 24 + 3 x 8)) / 8 = 18.

[00070] Figure 6 graphically illustrates the two heights used in Figures 4 and 5, namely the value of wpy and wny on the vertical geometric axis, as a function of the Y row number on the horizontal geometric axis. The wpy weight is applied to the present value, while the wny height is applied to the next value. Figure 7 corresponds to Figure 6, but also illustrates the interpolation output as obtained between the input depth values ​​24 and 8 as illustrated in Figure 5 – it can be observed that the weights are calculated in order to provide a linear fade between the input depth values. In addition to linear fade, higher-order interpolation functions can also be used, for example, cubic or join interpolations.

[00071] In general, it is noted that the size of the compartments in the X or Y dimension of the described volumes can always be a Petition 870250033491, dated 04 / 28 / 2025, p. 38 / 93 29 / 32 power of 2, since, in this case, the interpolation, being for example a trilinear fixed-point interpolation, can use a shift operation for normalization, which results in a substantial reduction in hardware cost. This can result in a variable number of bins being needed depending on the image size. With the change in bin size, the filter performance can be influenced. However, experiments show that this does not significantly impact visual performance. Furthermore, using a variable number of bins instead of a fixed number of bins does not significantly affect the hardware design. The size of a bin in the X or Y dimension can be specified by a hardware parameter, while the analysis of which value to select can be left, for example, to microprocessor software.

[00072] It will be appreciated that, in general, the processing subsystem can be provided separately from the integrated circuit described, for example, in another type of SoC.

[00073] Data can be provided without a computer-readable medium that defines the processing subsystem in the form of netlists and / or synthesizable RTLs. The computer-readable medium, and thus the data stored therein, can be transient or non-transient. For example, the processing subsystem can be provided as a synthesizable core, for example, in a hardware description language such as Verilog or VHDL, or as generic port-level netlists providing a Boolean algebraic representation of the IP RTC block logic function implemented as generic ports or process-specific standard cells.

[00074] The term map refers to data arranged in rows and Petition 870250033491, dated 04 / 28 / 2025, page 39 / 93 30 / 32 columns. Furthermore, the adjective depth should be understood as indicating the depth of parts of a camera image. Therefore, the depth map can be constituted by depth values, but also, for example, by disparity values ​​or parallactic shift values. Essentially, the depth map can thus constitute a disparity map or a parallactic shift map. Here, the term disparity refers to a difference in the position of an object when perceived with the user's left eye or right eye. The term parallactic shift refers to a displacement of the object between two views in order to provide said disparity to the user. Disparity and parallactic shift are generally negatively correlated with distance or depth. Devices and methods for conversion between all the above types of maps and / or values ​​are known.

[00075] Figure 8 illustrates a computer-implemented method 500 for estimating a depth map of an image. The method 500 is illustrated comprising, in an operation titled ACCESSING IMAGE DATA, accessing 510 image data from the image, in an operation titled ACCESSING DEPTH DATA, accessing 520 the depth data of a template depth map, in an operation titled APPLYING JOINT BILATERAL FILTER, applying 530 a joint bilateral filter to the template depth map using the image data as a range term in the joint bilateral filter, thus obtaining an adapted image depth map as output, where applying the joint bilateral filter comprises, in an operation titled INITIALIZING VOLUMES, initializing 540 a weighted depth sum volume and a weight sum volume as respective empty data structures in a memory, in a Petition 870250033491, dated 04 / 28 / 2025, page 40 / 93 31 / 32 operation entitled SPLATTING OPERATION, perform 550 a splatting operation to populate said volumes, in an operation entitled SLICING OPERATION, perform 560 a slicing operation to obtain an adapted image depth volume and in an operation entitled INTERPOLATION OPERATION, perform 570 an interpolation operation to obtain an adapted image depth value from the adapted image depth map for each pixel in the image. It will be appreciated that the above operation may be performed in any suitable order, for example, consecutively, simultaneously, or a combination of both, subject to, where applicable, a particular order required, for example, by input / output relationships. For example, operations 510 and 520 may be performed in parallel or sequentially.

[00076] The 500 method can be implemented in a processor system, for example, in a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. As also illustrated in Figure 9, instructions for the computer, for example, executable code, can be stored in a computer-readable medium 600, for example, in the form of a series 610 of machine-readable physical marks and / or as a series of elements possessing different electrical, for example, magnetic or optical properties or values. The executable code can be stored in a transient or non-transient form. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 9 illustrates an optical disc 600.

[00077] It should be noted that the embodiments mentioned above illustrate rather than limit the invention, and that these skilled persons Petition 870250033491, dated 04 / 28 / 2025, page 41 / 93 32 / 32 in the technique could lead to the design of many alternative modalities.

[00078] In claims, any reference signs located within parentheses should not be considered as limiting the claim. The use of the verb "to comprise" and its conjugations does not exclude the presence of elements or steps beyond those mentioned in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claim of a device enumerating several means, several of these means may be embodied by one and the same hardware item. The mere fact that certain measurements are mentioned in mutually different dependent claims does not indicate that a combination of these measurements cannot be used advantageously. Petition 870250033491, dated 04 / 28 / 2025, page 42 / 93

Claims

1 / 7 CLAIMS 1. Integrated circuit configured to estimate a depth map of an image (200), the integrated circuit comprising or being connected to a memory, the integrated circuit being characterized in that it comprises: an image data interface configured to access the image data of the image; a depth data interface configured to access the depth data of a template depth map, the template depth map representing a template that must be adapted to the image data; a processing subsystem configured to apply a joint bilateral filter (530) to the template depth map using image data as a range term in the joint bilateral filter (530), thereby obtaining an adapted image depth map as output;where the processing subsystem is configured to implement the bilateral filter (530) set by: initialization (540) of a weighted depth sum volume and a weight sum volume as respective empty data structures in memory, each of said volumes (300) comprising: two spatial dimensions (310, 320) representing a reduced two-dimensional spatial version of the image data; and at least one band dimension (330) representing a reduced band dimension version of an image component of the image data; wherein the cells of said volumes define compartments in an image coordinate system that is defined with respect to two spatial dimensions of the image and the band dimension of the image data;perform a splatting operation (550) to populate said volumes, wherein the splatting operation comprises, for each pixel in the image: identifying adjacent compartments in the weighted depth sum volume to which the pixel contributes to the splatting operation based on a pixel coordinate in the image coordinate system, the coordinate indicating a relative position of the pixel with respect to the compartments of each of said volumes; obtaining a pixel depth value from the gauge depth map; for each of the adjacent compartments; obtaining a splatting weight to weight the depth value, wherein the splatting weight determines a pixel's contribution to a respective compartment and is determined based on the pixel's relative position with respect to the respective compartment; weighting the depth value by the splatting weight;accumulate the weighted depth value in the respective compartment of the weighted depth sum volume, and accumulate the splatting weight in a corresponding compartment of the weight sum volume; perform a slicing operation (560) to obtain an adapted image depth volume, for each compartment of the weighted depth sum volume and the corresponding compartment of the weight sum volume, by dividing the accumulated weighted depth values ​​by the accumulated weights;perform an interpolation operation (570) to obtain a Petition 870250033491, dated 04 / 28 / 2025, page 44 / 93 3 / 7 adapted image depth value from the adapted image depth map for each pixel in the image, wherein the interpolation operation comprises: based on the pixel coordinate in the image coordinate system, identifying adjacent compartments in the adapted image depth volume, based on the pixel contributing to the corresponding compartments of the weighted depth sum volume during the splatting operation; applying an interpolation filter to the adjacent compartments of the adapted image depth volume, wherein the interpolation filter comprises, for each of the adjacent compartments, an interpolation weight that is determined based on the relative position of the pixel with respect to the respective compartment.; 2. Integrated circuit, according to claim 1, characterized in that the processing subsystem comprises an application-specific hardware circuit and a software-configured microprocessor, wherein: the application-specific hardware circuit is configured to perform the splatting operation and the interpolation operation; and the microprocessor is software-configured to perform the slicing operation during the operation of the integrated circuit.

3. Integrated circuit, according to claim 2, characterized in that the application-specific hardware circuit comprises a filter table for storing the splatting weights used in the splatting operation and / or the interpolation weights used in the interpolation operation.

4. Integrated circuit, according to claim 3, characterized in that the filter table is loaded with splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation before performing the respective operation.

5. Integrated circuit, according to any one of claims 1 to 3, characterized in that the splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation are equal.

6. Integrated circuit, according to any one of claims 1 to 5, characterized in that the splatting weights used in the splatting operation and the interpolation weights used in the interpolation operation represent a linear interpolation with respect to the image coordinate system.

7. Integrated circuit, according to any one of claims 2 to 6, dependent on claim 2, characterized in that the microprocessor is configured by software to, during the operation of the integrated circuit, apply temporal filtering to the volume of the weighted depth sum and to the volume of the weight sum before performing the slicing operation.

8. Integrated circuit according to claim 7, characterized in that the time filtering is a first finite impulse response filter or a higher-order finite impulse response filter.

9. Integrated circuit, according to any one of claims 1 to 8, characterized in that the processing subsystem is configured to, after performing the splatting operation, perform the convolution of the weighted depth sum volume with a Gaussian kernel.

10. Integrated circuit, according to any one of claims 1 to 9, characterized in that the integrated circuit is Petition 870250033491, dated 04 / 28 / 2025, page 46 / 93 5 / 7 or forms part of a field-programmable gate assembly.

11. Integrated circuit, according to any one of claims 1 to 9, characterized in that the integrated circuit is or forms part of a system-on-a-chip.

12. Device, characterized by comprising the integrated circuit, as defined in any one of claims 1 to 11.

13. Device according to claim 12, characterized in that it is a display device or a decoding box.

14. A computer-implemented method for estimating a depth map from an image, the method being characterized by comprising: accessing image data from the image; accessing depth data from a template depth map, the template depth map representing a template that must be adapted to the image data; applying a joint bilateral filter (530) to the template depth map using the image data as a range term in the joint bilateral filter (530), thus obtaining an adapted image depth map as output, where applying the joint bilateral filter (530) comprises: initializing (540) a weighted depth sum volume and a weight sum volume as respective empty data structures in memory, each of said volumes comprising: two spatial dimensions (310, 320) representing a reduced two-dimensional spatial version of the image data;and Petition 870250033491, of 04 / 28 / 2025, page 47 / 93 6 / 7 at least one band dimension (330) representing a reduced version of a band dimension of an image component of the image data; wherein the cells of said volumes define compartments in an image coordinate system that is defined with respect to two spatial dimensions of the image and the image data band dimension; perform a splatting operation (550) to populate said volumes, wherein the splatting operation comprises, for each pixel in the image: identifying the adjacent compartments in the weighted depth sum volume to which the pixel contributes in the splatting operation based on the pixel coordinate in the image coordinate system, the coordinate being indicative of the pixel's relative position with respect to the compartments of each of said volumes; obtaining a pixel depth value from the template depth map;for each of the adjacent compartments: obtain a splatting weight to weight the depth value, where the splatting weight determines a pixel's contribution to a respective compartment and is determined based on the pixel's relative position with respect to the respective compartment; weight the depth value by the splatting weight; accumulate the weighted depth value in the respective compartment of the weighted depth sum volume and accumulate the splatting weight in a corresponding compartment of the weight sum volume; perform a slicing operation (560) to obtain an adapted image depth volume, for each Petition 870250033491, 04 / 28 / 2025, p. 48 / 93 7 / 7 compartment of the weighted depth sum volume and corresponding compartment of the weight sum volume, dividing the accumulated weighted depth values ​​by the accumulated weights;perform an interpolation operation (570) to obtain an adapted image depth value from the adapted image depth map for each pixel in the image, wherein the interpolation operation comprises: based on the pixel coordinate in the image coordinate system, identifying adjacent compartments in the adapted image depth volume based on the pixel contributing to the corresponding compartments of the weighted sum of depths volume during the splatting operation; applying an interpolation filter to the adjacent compartments of the adapted image depth volume, wherein the interpolation filter comprises, for each of the adjacent compartments, an interpolation weight that is determined based on the relative position of the pixel with respect to the respective compartment.

15. Computer-readable medium characterized in that it comprises transient or non-transient data representing instructions arranged to cause a processor system to perform the method, as defined in claim 14. Petition 870250033491, dated April 28, 2025, p. 49 / 93

Images